Multi-user super-resolution processing method and device, equipment and medium

Abstract

The invention provides a multi-user super-resolution processing method and device, equipment and a medium, and belongs to the field of image and video processing. The super-resolution processing method comprises the following steps: combining a plurality of to-be-processed images from a plurality of users to generate batch image combinations; inputting the batch image combination into a pre-constructed model pool; for any to-be-processed image in the batch image combination, at least one target super-resolution processing model corresponding to the to-be-processed image is determined from a model pool; inputting the to-be-processed image into at least one target super-resolution processing model; respectively outputting at least one prediction image corresponding to the to-be-processed image by at least one target super-resolution processing model; generating an output image based on a predetermined weight combination and the at least one prediction image; and outputting the output image to a user corresponding to the to-be-processed image. According to the method, different preferences of different users can be met by using as few model resources as possible.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image video processing, and in particular to a super-resolution processing method and device for multi-user personalized needs, a computing device, a medium and a program product. BACKGROUND

[0002] Super-resolution (SR) is a process that uses image processing techniques to restore details and data information of an image by increasing its resolution. It can improve the clarity and detail performance of an image. In practical applications, super-resolution is usually modeled as an inverse problem of reconstructing a high-resolution image from a low-resolution observation. Image super-resolution is a typical "ill-posed problem", that is, a low-resolution image corresponds to numerous reasonable high-resolution images. This "one-to-many" characteristic determines that there is no unique and objective "best" super-resolution result. What is "good" effect highly depends on the subjective quality preference of users, which is closely related to image content, application scenarios and personal aesthetics. The evaluation criteria of super-resolution effect are diversified, subjective and biased.

[0003] The existing SR mainstream models include convolution-based networks and self-attention-based networks, and the commonly used training targets cover pixel error and perception-based indicators. However, these SR models are not suitable for being used in multi-user or multi-business scenarios with different needs. SUMMARY

[0004] The present application aims to at least solve the technical problem that the image video processing manner in the background art cannot well meet the subjective preferences of different users. To this end, one purpose of the present application is to provide a super-resolution processing method for multi-user personalized needs, so as to meet the differentiated quality preferences of multi-users on the basis of reusing model computing resources through the decoupling mechanism of model pool sharing and personalized weight fusion.

[0005] Embodiments of the first aspect of the present application provide a multi-user super-resolution processing method, comprising: combining a plurality of to-be-processed images respectively from a plurality of users to generate a batch image combination; inputting the batch image combination into a pre-constructed model pool, the model pool comprising a plurality of super-resolution processing models; for any to-be-processed image in the batch image combination: determining at least one target super-resolution processing model corresponding to the to-be-processed image from the model pool; inputting the to-be-processed image into the at least one target super-resolution processing model; outputting at least one predicted image corresponding to the to-be-processed image by the at least one target super-resolution processing model respectively; generating an output image corresponding to the to-be-processed image based on a pre-determined weight combination and the at least one predicted image; and outputting the output image to a user corresponding to the to-be-processed image.

[0006] In the technical solution of the embodiments of the present application, for images from different users, a super-resolution processing model corresponding to the user is selected from the model pool, and after obtaining the model processing result, a weight combination conforming to the user demand is further used to obtain the final output image, so that the output image can meet the different subjective preferences of each user, and for multiple users, the same model pool can be used, without the need to train and deploy a dedicated model for each user, thereby significantly reducing the marginal cost of model deployment and memory occupation, while realizing flexible response to the differentiated subjective preferences of multiple users.

[0007] In some embodiments, the super-resolution processing method further comprises: for any one of the multiple users: determining at least one target super-resolution processing model corresponding to the user from the model pool based on a time delay budget and / or a style preference of the user, the time delay budget being used to indicate the time delay requirement of the user for super-resolution processing, and the style preference being used to indicate the image style of the user; determining at least one weight corresponding to the at least one target super-resolution processing model respectively; and determining a weight combination based on the at least one weight. Determining the super-resolution processing model and the weight combination suitable for the user according to the time delay budget and / or the style preference of the different user can make the finally obtained output image meet the differentiated needs of the user as much as possible.

[0008] In some embodiments, determining at least one target super-resolution processing model corresponding to the to-be-processed image from the model pool comprises: determining the at least one target super-resolution processing model from the model pool according to the number of the user corresponding to the to-be-processed image. Selecting the corresponding target super-resolution processing model according to the user number can accurately determine the model suitable for each user from all the models in the model pool, and this way realizes logical isolation of multi-user configuration, can quickly route to the preset algorithm combination according to the user identity, and ensures the accuracy of policy matching in a multi-user scenario.

[0009] In some embodiments, determining at least one weight corresponding to the at least one target super-resolution processing model respectively comprises: performing weight scanning using the at least one target super-resolution processing model based on a public sample set to obtain at least one initial weight; and determining the at least one weight based on a user sample set and the at least one initial weight. On the basis of determining the initial weight, the weight for each user is further optimized in combination with the sample set of each user, so that the weight combination of each user can conform to the user demand as much as possible, and personalized super-resolution processing effect is obtained.

[0010] In some embodiments, the determining, based on the public sample set, the at least one initial weight using the at least one target super-resolution processing model includes: determining an evaluation condition corresponding to the style preference of the user; determining a scanning weight vector corresponding to the at least one target super-resolution processing model, the scanning weight vector including at least one scanning weight corresponding to the at least one target super-resolution processing model respectively; determining a comprehensive score corresponding to the scanning weight vector based on the public sample set and the scanning weight vector; and in response to the comprehensive score satisfying the evaluation condition, determining the at least one scanning weight in the scanning weight vector as the at least one initial weight. By constructing a comprehensive score mechanism including the scanning weight vector, quantitative evaluation and automatic screening of different style preferences are realized, and an interpretable preferred initialization parameter is provided for subsequent personalized configuration.

[0011] In some embodiments, the determining, based on the time delay budget and / or the style preference of the user, the at least one target super-resolution processing model corresponding to the user from the model pool includes: determining at least one super-resolution processing model from the model pool; determining a super-resolution processing time delay of the at least one super-resolution processing model; and in response to the super-resolution processing time delay of the at least one super-resolution processing model satisfying the time delay budget, determining the at least one super-resolution processing model as one target super-resolution processing model corresponding to the user. When determining the target super-resolution processing model corresponding to each user, whether different super-resolution processing model combinations satisfy the time delay budget of the user is determined, so that when facing models with different calculation complexities, the response can be completed within the time window specified by the user, and the risk of service timeout or lag due to improper model selection is reduced.

[0012] In some embodiments, the inputting the to-be-processed image into the at least one target super-resolution processing model includes: for any target super-resolution processing model in the at least one target super-resolution processing model: in response to the calculation resource occupied by the target super-resolution processing model satisfying a pre-set resource condition, inputting the to-be-processed image into the target super-resolution processing model. The risk of blocking the system due to a sudden surge in requests for individual popular models is reduced, thereby improving the throughput and stability of the overall system in a multi-user high-concurrency scenario.

[0013] Embodiments of the second aspect of the present application provide a multi-user super-resolution processing apparatus, comprising: a combination module configured to combine a plurality of to-be-processed images respectively from a plurality of users to generate a batch image combination; an input module configured to input the batch image combination into a pre-constructed model pool, the model pool comprising a plurality of super-resolution processing models; a generation module configured to, for any to-be-processed image in the batch image combination: determine at least one target super-resolution processing model corresponding to the to-be-processed image from the model pool; input the to-be-processed image into the at least one target super-resolution processing model; output at least one predicted image corresponding to the to-be-processed image respectively by the at least one target super-resolution processing model; generate an output image corresponding to the to-be-processed image based on a pre-determined weight combination and the at least one predicted image; and an output module configured to output the output image to a user corresponding to the to-be-processed image.

[0014] Embodiments of the third aspect of the present application provide a computing device, comprising: at least one processor; and at least one memory communicatively connected with the at least one processor, the at least one memory storing instructions which, when executed by the at least one processor alone or jointly, cause the computing device to perform the super-resolution processing method in the above embodiments.

[0015] Embodiments of the fourth aspect of the present application provide a computer-readable storage medium storing instructions which, when executed by one or more processors of a computing device alone or jointly, cause the computing device to perform the super-resolution processing method in the above embodiments.

[0016] Embodiments of the fifth aspect of the present application provide a computer program product comprising instructions which, when executed by one or more processors of a computing device alone or jointly, cause the computing device to perform the super-resolution processing method in the above embodiments.

[0017] The above description is only a summary of the technical solutions of the present application. In order to enable a clearer understanding of the technical means of the present application, the above description can be implemented according to the content of the specification, and in order to enable the above and other purposes, features and advantages of the present application to be more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0018] In the drawings, like reference numerals refer to same or similar functionalities throughout the several views. The drawings are not necessarily to scale. It is to be understood that the drawings only depict several embodiments in accordance with the present disclosure and should not be considered as limiting the scope of the present application.

[0019] Figure 1 Flowchart of the multi-user super-resolution processing method of some embodiments of the present application; Figure 2 Flowchart of a super-resolution processing method for some embodiments of the present application; Figure 3 Flowchart of determining weight combination for some embodiments of the present application; Figure 4 Flowchart of determining weight for some embodiments of the present application; Figure 5 Flowchart of determining initial weight for some embodiments of the present application; Figure 6 Flowchart of determining target super-resolution processing model for some embodiments of the present application; Figure 7 Flowchart of a super-resolution processing method for some embodiments of the present application; Figure 8 Schematic diagram of a multi-user super-resolution processing device for some embodiments of the present application; Figure 9 Schematic diagram of a computing device for some embodiments of the present application. DETAILED DESCRIPTION

[0020] The embodiments of the technical solutions of the present application will be described in detail below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and therefore only serve as examples, and cannot limit the protection scope of the present application.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "include" and "have" and any variations thereof in the specification and claims of the present application and the above description of drawings are intended to cover non-exclusive inclusion.

[0022] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly and specifically limited.

[0023] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily independent or alternative embodiments to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0024] In the description of the embodiments of the present application, the term "and / or" is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A existing alone, A and B existing together, and B existing alone. In addition, the character " / " herein generally represents that the front and rear associated objects are in an "or" relationship.

[0025] In the description of the embodiments of the present application, the term "a plurality of" refers to two or more (including two), and similarly, "a plurality of groups" refers to two or more groups (including two groups), and "a plurality of pieces" refers to two or more pieces (including two pieces).

[0026] In the description of the embodiments of the present application, the technical terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the embodiments of the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the embodiments of the present application.

[0027] In the description of the embodiments of the present application, unless otherwise explicitly specified and limited, the technical terms "mounting", "connection", "connection", "fixing" and the like should be understood in a broad sense, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrated; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication or interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0028] Super-resolution (Super-Resolution, abbreviated as SR) is a process of restoring the details and data information of an image by increasing the resolution of the image using image processing technology. It can improve the clarity and detail performance of the image. Image super-resolution is a typical "ill-posed problem", that is, a low-resolution image corresponds to a large number of reasonable high-resolution images. This "one-to-many" characteristic determines that there is no unique and objective "best" super-resolution result. What is "good" effect is highly dependent on the user's subjective quality preference, which is closely related to image content, application scenarios and personal aesthetics. The evaluation criteria for super-resolution effect are diversified, subjective and biased.

[0029] The existing mainstream SR model includes a convolution-based network and a self-attention-based network, and the commonly used training targets cover pixel error and perception-based indicators. However, these SR models are not suitable for being used in a multi-user or multi-business line scenario with different needs. In a service for multiple users, it is usually necessary to deploy independent models for different user preferences, and at the same time, multiple sets of model versions need to be maintained, which is prone to engineering problems such as an increase in the number of versions, complex instance maintenance, and a decline in computing power utilization.

[0030] To solve the above technical problems, an embodiment of the present application proposes a super-resolution processing method based on a shared model pool and personalized weight fusion. The core idea is that instead of allocating a fixed dedicated model for each user, a shared model pool containing multiple heterogeneous super-resolution processing models is set. For different user needs, the model subset in the model pool that matches each user's needs is dynamically scheduled for parallel processing, and the outputs of these models are weighted and fused using pre-determined weights that meet the user's preferences to obtain the final output image that meets the user's needs.

[0031] Through the above method, the general "feature calculation" and the personalized "style rendering" can be decoupled. Since it is not necessary to maintain separate model files and deployment instances for each user, this scheme can significantly eliminate model deployment redundancy in a multi-user scenario, reduce memory usage and version maintenance costs. At the same time, by performing unified batch processing on the shared model pool, hardware resource utilization optimization can be achieved under the premise of meeting the latency constraints of each user.

[0032] Based on the above technical principles, an embodiment of the present application provides a multi-user super-resolution processing method. Referring to Figure 1 , the multi-user super-resolution processing method 100 includes steps 110 to 170.

[0033] Step 110, combining multiple images to be processed respectively from multiple users to generate a batch image combination.

[0034] Step 120, inputting the batch image combination into a pre-constructed model pool. The model pool includes multiple super-resolution processing models.

[0035] Step 130, for any one of the images to be processed in the batch image combination: determining at least one target super-resolution processing model corresponding to the image to be processed from the model pool.

[0036] Step 140, inputting the image to be processed into the at least one target super-resolution processing model.

[0037] Step 150, outputting at least one predicted image corresponding to the image to be processed by the at least one target super-resolution processing model, respectively.

[0038] At step 160, an output image corresponding to the to-be-processed image is generated based on the pre-determined weight combination and the at least one predicted image.

[0039] At step 170, the output image is output to a user corresponding to the to-be-processed image.

[0040] In embodiments of the present application, a "user" can refer to a minimum unit sharing the same model pool and service architecture but having independent and exclusive preferences or requirements. The preferences or requirements of different users are independent of each other, rather than being independent of each other in terms of physical resources. In some embodiments, different users can be different customer companies (multi-tenant SaaS service), for example, on a public cloud, company A (e-commerce platform) and company B (social app) are two different users, they share the computing power pool and model pool of the cloud service provider, but each has its own business optimized picture quality preference formula. In other embodiments, different users can also be different business lines or channels within the same company, for example, a large live streaming platform can define "game live streaming channel", "show live streaming channel" and "on-demand movie channel" as three different users, they share the private deployment system within the company, but the game channel uses "text sharp" preference, the show live streaming channel uses "portrait beautification" preference, and the on-demand movie channel uses "film grain" preference. In other embodiments, different users can also be sub-types of content or user groups within the same business, for example, within the "game live streaming" channel, if it is hoped to provide more refined picture quality for "FPS games" and "MOBA games", then FPS games and MOBA games can be regarded as further divided sub-users. In embodiments of the present application, different users can be distinguished by assigning different user numbers (Client ID) to different users. For example, a unique Client ID can be assigned to each user, and different users can be routed and matched by the unique Client ID. In embodiments of the present application, the "to-be-processed image" is a picture with visual effects, which can have different sources and forms of expression, for example, it can come from a single picture, a continuous image data stream, a video, etc. In one example, if the to-be-processed image comes from a single picture, then the to-be-processed image is the picture itself; if the to-be-processed image comes from a non-video image data stream, then the to-be-processed image can be an image in the data stream; if the to-be-processed image comes from a video, the to-be-processed image can be a to-be-processed frame in the video. Embodiments of the present application do not limit the specific source of the to-be-processed image. In the following, a to-be-processed frame in a video will be taken as an example for illustration.

[0041] In some embodiments, at a certain moment, image super-resolution processing requests from different users need to be processed in parallel, for example, for a certain live broadcast platform, there will be multiple channels simultaneously live broadcast. Therefore, in step 110, the image requests from different users, which can also be understood as each to-be-processed image (which can also be understood as to-be-processed frame) in the video of different users, are combined, and the to-be-processed images are combined into a batch to obtain a batch image combination.

[0042] By combining the to-be-processed images into a batch, the parallel computing advantages of hardware accelerators such as graphics processing units (GPUs) can be fully utilized, and the overall throughput and computing resource utilization efficiency of the system can be significantly improved.

[0043] The batch image combination will be input into a pre-constructed model pool. The model pool includes a plurality of super-resolution processing models. These super-resolution processing models can select different models with heterogeneity, for example, a convolutional neural network (CNN) model, a deep learning (Transformer) model, a model trained using different loss functions, a specific scene model, and the like. Among them, the CNN model has an inductive bias that focuses more on modeling local textures, captures the relevance of local images through deep convolution stacking, is usually robust in restoring regular textures (such as building facades, fabrics, etc.) and removing compression artifacts, and the output is often smoother and more natural (depending on the training strategy). The Transformer model can model the overall structure and long-distance pixel dependence through the self-attention mechanism, better understand the overall structure of the image and the dependence of distant pixels, and usually perform better in restoring complex structures (such as the symmetry of facial features, the details of distant buildings, etc.) and edge sharpness. The state space model is good at capturing long sequence dependencies and is computationally efficient, and its effect is between CNN and Transformer, while maintaining efficient inference, it can better balance local details and global structure. For models trained using different loss functions, models trained using the mean absolute error L1 Loss have higher fidelity (PSNR), but may appear smooth and lack sharpness; and models trained using perceptual loss (such as LPIPS / VGG) or adversarial loss (GAN) have higher quality in human visual perception, richer texture, and sharper edges, but may change the original color or introduce generated artifacts, etc. Existing specific scene models can be, for example, STISR models for text, face SR models for faces, etc.

[0044] Each user corresponds to a set of super-resolution processing models, which are also known as the target super-resolution processing models for each user. In some embodiments, the target super-resolution processing model for each user can be determined in advance based on factors such as the user's latency budget. Typically, a user has no more than five target super-resolution processing models. The process of determining the target super-resolution processing model for each user will be described in detail below.

[0045] In some embodiments of this application, step 130 includes: determining at least one target super-resolution processing model from the model pool based on the user ID corresponding to the image to be processed.

[0046] As mentioned above, each user is assigned a unique user ID, or Client ID. Therefore, for images from different users within a batch of images, the Client ID will be used to input the corresponding target super-resolution processing model for that user. Selecting the appropriate target super-resolution processing model based on the user ID accurately identifies the model suitable for each user from the model pool. This approach achieves logical isolation in multi-user configurations, enabling rapid routing to the pre-defined algorithm combination based on user identity and ensuring accurate policy matching in multi-user scenarios.

[0047] In step 150, each target super-resolution processing model will process the image to be processed separately to obtain a predicted image, which is the image obtained after super-resolution reconstruction of the image to be processed. In other words, for a user's image to be processed, each target super-resolution processing model will obtain a predicted image.

[0048] Each user will also have a corresponding set of weight combinations. In one example, the weight combination for each user can be determined based on the user ID (Client ID). For each target super-resolution processing model, a weight will correspond to a weight in the weight combination. In step 160, based on the predicted image generated by each target super-resolution processing model and the weight combination, the predicted image is weighted using the weights in the weight combination to obtain the final output image corresponding to that user.

[0049] In one example, for each user, the weight combination corresponding to that user will be determined from the weight lookup table based on the Client ID, and a weighted summation operation will be performed: I_final = w1×I_model1+w2×I_model2+……+wn×I_modeln, where I_final represents the final output image, I_model1, I_model2……I_modeln are the predicted images obtained by each target super-resolution processing model, and w1, w2……wn are the weights in the weight combination corresponding to each target super-resolution processing model.

[0050] In other embodiments, other weighting schemes can be used. For example, to address the structural ghosting and detail artifacts that may occur during the fusion of heterogeneous models, two robust weighted fusion mechanisms can be introduced: one is a decoupling strategy based on frequency domain separation, which locks the low-frequency components (image skeleton) of each model's output and applies personalized weight combinations only to the high-frequency components (texture details), thereby physically reducing macroscopic structural misalignment; the other is a dynamic circuit breaker strategy based on consistency checks, which generates an uncertainty heatmap by calculating the prediction variance at the pixel or region level, and forces the weight combination to collapse to the most stable baseline model in high-risk areas (such as complex edges) where there are significant discrepancies in the outputs of various models. Using these two weighted fusion mechanisms, the output image can maintain geometric stability and image purity while responding to the user's personalized style preferences.

[0051] For the first decoupling strategy based on frequency domain separation, in order to improve the geometric consistency of the main structure of the image while satisfying users' personalized preferences for texture details (such as sharpness and graininess), a fusion strategy based on frequency domain separation can be adopted to generate an output image corresponding to the image to be processed based on a pre-determined weight combination and at least one prediction image. Specifically, the output image of the super-resolution model usually contains low-frequency components and high-frequency components. Among them, the low-frequency components mainly represent the overall structure, contours, and color block distribution of the image. Different models usually make relatively consistent predictions for this part and should remain stable. The high-frequency components mainly represent the edge details, texture noise, and sharpness features of the image. This is the key area reflecting "style preferences" and also the part where there are the greatest differences among various heterogeneous super-resolution processing models. For each prediction image, frequency domain decomposition will be performed separately. For example, filtering techniques such as Gaussian Blur, Laplacian Pyramid, or Wavelet Transform can be used to decompose each predicted image into a structural layer (low-frequency components) and a detail layer (high-frequency components). During the fusion stage, different processing logics are applied to the low-frequency and high-frequency components. For the low-frequency components (structural layer), strategies such as Average Fusion or Anchor Model Retention can be used to calculate the average of the low-frequency components of all predicted images, or the low-frequency components of the predicted image obtained from the pre-defined, structurally most stable benchmark super-resolution processing model (such as the MSE-based model) can be directly selected as the final low-frequency basis. This processing ensures that the geometric skeleton of the image remains aligned and stable regardless of weight changes, preventing ghosting. For the high-frequency components (detail layer), a pre-determined weight combination can be applied for linear weighting, and the high-frequency components of the predicted images obtained from each model can be weighted and combined according to the user's style preferences (such as "sharp text" or "film grain"). Finally, the fused low-frequency components are inversely transformed or superimposed with the weighted fused high-frequency components to reconstruct the final output image. This method decouples "image quality structure" and "image quality style" from a physical perspective, so that the adjustment of weights only affects the texture detail level, thereby reducing ghosting problems caused by structural misalignment.

[0052] For the second dynamic circuit breaker strategy based on consistency checks, to further improve the structural stability of the output image and eliminate ghosting or artifacts caused by minor spatial misalignment or style differences between heterogeneous models, generating an output image corresponding to the image to be processed based on a pre-determined weight combination and at least one predicted image may further include: performing dynamic weight adjustment based on consistency checks. Specifically, since different super-resolution processing models (e.g., CNN models and Transformer models) differ in inference mechanisms and receptive fields, for high-frequency complex regions in the image (such as edges, densely textured areas, etc.), the outputs of different models may exhibit pixel-level spatial inconsistencies. Therefore, before performing weighted fusion, a consistency index at the pixel / patch level for at least one predicted image can be calculated. In a specific example, the variance or standard deviation of multiple predicted images at each pixel location (or only on the luminance channel Y channel) can be calculated to generate a corresponding uncertainty heatmap. This uncertainty heatmap can reflect the degree of "disagreement" between models in local regions of the image: in flat areas, the outputs of each model converge, and the variance is low; in edge or textured complex areas, if there is an alignment deviation between models, the variance will increase significantly. Furthermore, a spatially adaptive weight mask can be generated based on the uncertainty heatmap. For example, the uncertainty heatmap can be mapped using a preset threshold function or a nonlinear activation function (such as the sigmoid function), so that the values ​​corresponding to high-variance regions in the mask approach 1 (indicating high risk), while low-variance regions approach 0 (indicating low risk). Based on the spatially adaptive weight mask, a dynamic weight collapse strategy can be further implemented. In one example, the specific fusion formula can be as follows: .

[0053] in, For the final output image; For spatially adaptive weighted masks; This is the conventional weighted result calculated based on user style preferences (i.e., the aforementioned weight combination); The output image for a pre-specified anchor model. The anchor model is usually selected from the model pool as the model with the highest structural stability and the best fidelity index (e.g., a CNN model trained based on MSE loss).

[0054] Through the aforementioned dynamic collapse mechanism, in flat or low-frequency regions of the image ( It retains the user-customized stylized weight combination to meet the user's personalized preferences (such as graininess and sharpness); while in high-frequency edge areas where there is a potential risk of ghosting ( This forces the weights to "collapse" to the most stable baseline anchor point model, thereby effectively reducing double edges or blur artifacts that may be generated by the fusion of heterogeneous models while maintaining the overall style of the image, achieving the best balance between stylization and structural stability.

[0055] The three weighting schemes mentioned in the above embodiments can be selected according to usage requirements. In one example, the same weighting scheme is used in determining the corresponding weight combination for each user and optimizing the weight combination, as well as in weighting the predicted images obtained from each model.

[0056] In step 170, the final output image will be sent to the corresponding user.

[0057] For images from different users, a super-resolution processing model corresponding to that user is selected from the model pool. After obtaining the model processing results, a weight combination that meets the user's needs is further used to obtain the final output image. This allows the output image to satisfy the different subjective preferences of each user. Furthermore, the same model pool can be used for multiple users, eliminating the need to build different super-resolution processing models for each user, thus reducing model computational resources while meeting user needs. Using this super-resolution processing method, personalized "model computation" can be transformed into unified, batch-capable "model computation" and lightweight, parallel "weight computation," effectively resolving the contradiction between personalization and processing efficiency in super-resolution processing.

[0058] Figure 2 The image illustrates an example of a super-resolution processing method. For example... Figure 2 As shown, taking the images to be processed as frames from multiple videos as an example, image requests from multiple users, i.e., multiple images to be processed at the same time, will be combined by a frame batch combiner to obtain batch image combinations. These batch image combinations will be input into the model pool. Based on the Client ID corresponding to each user, each image to be processed will be input into the super-resolution processing model corresponding to that user. Each super-resolution processing model will output corresponding predicted images for output caching. These predicted images will then be processed by a personalized fusion engine and the weight combination corresponding to each user will be looked up in the weight lookup table (LUT). Figure 2 The three users illustrated are user A, user B, and user C. Each user applies a weighted combination to the predicted image to obtain the final output image. After passing through the result distributor, the output image corresponding to user A will be output to user A, the output image corresponding to user B will be output to user B, and the output image corresponding to user C will be output to user C.

[0059] By using a model pool for unified batch processing and a parallel processing workflow that integrates personalized processing for each user, hardware utilization efficiency and personalized services are balanced. When adding new users, there is no need to retrain or deploy models, reducing the resource waste of deploying model instances individually for each user. Computational resources do not swell significantly as the user base expands, achieving overall cost control and a significant improvement in operational efficiency while meeting user needs.

[0060] In some embodiments of this application, the super-resolution processing method further includes a first process 300. (See reference...) Figure 3 The first process 300 includes steps 310 to 330.

[0061] For any one of the multiple users: Step 310: Based on the user's latency budget and / or style preference, determine at least one target super-resolution processing model corresponding to the user from the model pool. The latency budget indicates the user's latency requirements for super-resolution processing. The style preference indicates the user's image style.

[0062] Step 320: Determine at least one weight corresponding to at least one target super-resolution processing model.

[0063] Step 330: Determine the weight combination based on at least one weight.

[0064] For each user, before performing super-resolution processing, the target super-resolution processing model and weight combination corresponding to that user can be pre-determined and bound to the user ID. During subsequent super-resolution processing, based on the user ID, the target super-resolution processing model matching that user can be determined from the model pool. After obtaining the predicted image, the appropriate weight combination is used for weighting to obtain the final output image.

[0065] As mentioned above, different types of super-resolution processing models have different biases, such as producing smoother, more natural outputs or sharper edges. Therefore, a super-resolution processing model can be determined based on the user's style preferences to match that user. Furthermore, super-resolution processing has high latency requirements in certain applications, and these requirements will vary depending on the user's specific scenario. For example, latency requirements are relatively lenient in offline or low-real-time image enhancement scenarios (such as photo editing and batch image super-resolution processing); second-level latency is acceptable for live streaming or video-on-demand scenarios; while latency requirements are more stringent for extremely low-latency streaming or highly interactive scenarios (such as cloud gaming and interactive live streaming). Therefore, it's also necessary to determine whether the processing time of these super-resolution processing models meets the user's latency budget; only if it does should they be selected as the target super-resolution processing model for that user.

[0066] In some embodiments of this application, reference is made to Figure 6 Step 310 includes steps 610 to 630.

[0067] Step 610: Determine at least one super-resolution processing model from the model pool.

[0068] Step 620: Determine the super-resolution processing delay of at least one super-resolution processing model.

[0069] Step 630: In response to the super-resolution processing latency of at least one super-resolution processing model satisfying the latency budget, at least one super-resolution processing model is determined as at least one target super-resolution processing model corresponding to the user.

[0070] In step 610, user samples can be used to determine some super-resolution processing models from the model pool that better match the user's style preferences. In one example, a set of super-resolution processing models can be determined from the model pool based on the user's usage scenario. Typically, there are no more than 5 models. For example, models suitable for common style preferences can be selected, such as sharper, cleaner, grainier, and more natural.

[0071] In one example, a super-resolution processing model corresponding to a specific style preference can be determined from the model pool based on quality feature vectors and style feature vectors. For quality feature vectors, a set of normalized quality metrics will be calculated for the outputs of each super-resolution processing model, forming the quality feature vector. Quality feature vectors can include, but are not limited to: distortion metrics (such as measures based on pixel error or structural similarity), perceptual metrics (such as subjective quality estimation based on depth features), and scene-specific metrics (e.g., for text scenes, character recognition accuracy, edge preservation, etc.). For style feature vectors, a unified image encoding network (e.g., based on convolutional features or a pre-trained visual backbone network, such as VGG) will be used to extract and statistically summarize features from high-resolution reference images and model output images, encoding style-related information such as texture details, overall contrast, noise particles, and color distribution into style feature vectors. For each style preference, a "style prototype vector" for that style will be pre-calculated from publicly available sample materials. For each super-resolution processing model, the degree of fit to the style preference is measured by the distance to the style prototype vector. For each style preference, a comprehensive score can be calculated for each super-resolution processing model by combining the quality feature vector and the style feature vector. Models are then ranked from highest to lowest score, and those with the highest scores and most stable performance are prioritized as the super-resolution processing models that match the style preference. It should be understood that the calculation of the comprehensive score and the selection of super-resolution processing models based on it can be a one-time event. Preference labels can be assigned to each candidate super-resolution processing model to provide a reference for model combinations corresponding to different style preferences. Furthermore, a complementarity constraint is introduced when determining at least one super-resolution processing model from the model pool: once the first model is determined, subsequent selected models must maintain a certain degree of difference in style feature space. For example, two models with significantly different style performance can be identified in the style feature space, one focusing on edge sharpening and detail generation, and the other on noise suppression and structural smoothing. This combination allows for a better transition between the two styles in the output image by adjusting the weights. The core advantage of introducing the complementarity constraint is that it maximizes the coverage of the style space with as few models as possible. This means that by running only a few "baseline models" in parallel, a rich variety of style variants can be fitted through mathematical combinations of weight vectors, thus adapting to the personalized preferences of more users. This mechanism greatly reduces the loading frequency of different models, significantly improves model reusability and parallel inference throughput, and is more friendly to large-scale scenarios. In one example, when determining at least one super-resolution processing model, the selection range of models can be limited to train them on the same or compatible downsampling / blurring degradation models, thereby improving the consistency of reconstructed geometry.

[0072] To reduce resource consumption during model processing, a "heterogeneous model of varying sizes" approach can be used to select at least one super-resolution processing model from the model pool. While meeting user needs, the model pool can include optimized lightweight models. For example, for game streaming users, a mainstream high-performance model can be selected as the primary model, paired with a lightweight model as an auxiliary. This allows for the introduction of model diversity while effectively controlling the overall computational load of the model pool.

[0073] In step 620, the time required for this part of the super-resolution processing model to perform super-resolution processing will be determined, i.e., the super-resolution processing latency. In one example, the super-resolution processing latency can be the end-to-end time of a single processing step, i.e., the processing time from the user's output image request to the user receiving the final output image. Specifically, it can include queuing time, model inference time, output fusion time, and frame overhead time. Queuing time can refer to the time an image request waits in the queue before being assigned to a super-resolution processing model. For example, it can include waiting time due to computational resource contention, waiting time to be combined with other images to be processed, etc. Model inference time can refer to the time from the super-resolution processing model receiving input information to the completion of all super-resolution processing model outputs. For example, it can include the input image copying time (from the Central Processing Unit, or CPU, to the Graphics Processing Unit, or GPU), the time required for all super-resolution processing models corresponding to the user to complete processing, and the output image copying time (from GPU to CPU), etc. Output fusion time can include the time for weighting the predicted images obtained from each super-resolution processing model. Framework overhead time may include other overhead incurred by the service framework, such as the time required for parsing and encoding / decoding, request routing, parameter validation, etc. In some embodiments, the single-processing time required by these super-resolution processing models for a single user's input image to be processed, as well as the total queuing time after batching the images to be processed from multiple users, may also be determined separately.

[0074] In step 630, it will be determined whether the super-resolution processing latency of these super-resolution processing models meets the user's latency budget. If it meets the user's latency budget, these super-resolution processing models will be identified as the target super-resolution processing models corresponding to that user. If not, other super-resolution processing models will be re-identified. For example, if a larger super-resolution processing model exceeds the latency limit, it will be replaced with a similar but faster super-resolution processing model. Alternatively, in some embodiments, the user may be consulted regarding whether the resolution or frame rate can be reduced, and the target super-resolution processing model will be re-identified based on the reduced latency budget.

[0075] After determining the target super-resolution processing model for each user, it will be bound to the user's ID number, and will only be re-evaluated and confirmed when the user requests a change.

[0076] In some embodiments, when a new user joins, the target super-resolution processing model corresponding to the new user can be determined from the model pool in the manner described above, without affecting the super-resolution processing of the existing user.

[0077] When determining the target super-resolution processing model for each user, the determination is made based on whether different combinations of super-resolution processing models meet the user's latency budget, thereby improving the super-resolution processing efficiency for each user and meeting the user's latency requirements.

[0078] In some embodiments, for a given user, multiple sets of target super-resolution processing models can be determined from a model pool. Each set of target super-resolution processing models includes at least one target super-resolution processing model, and several weight combinations corresponding to different preferences can be determined. When processing the image to be processed from the user, the user ID can be mapped to a specific set of target super-resolution processing models and a specific set of weight combinations. Furthermore, the weight combinations can be fine-tuned based on the user's sample set to achieve preference alignment. If the final output image obtained by the set of target super-resolution processing models and the set of weight combinations does not meet the user's requirements and / or the super-resolution processing latency does not meet the latency budget, a reselection mechanism is triggered. This allows for the re-determining of a target super-resolution processing model for the user from the model pool, or the re-determining of a weight combination. Further fine-tuning of the re-determined weight combinations is also possible. This approach is more suitable for large-scale scenarios (such as live streaming platforms) and facilitates the rapid determination of a superior set of target super-resolution processing models and weight combinations for each user, quickly meeting user needs.

[0079] In some embodiments of this application, reference is made to Figure 4 Step 320 includes steps 410 to 420.

[0080] Step 410: Based on the public sample set, perform a weight scan using at least one target super-resolution processing model to obtain at least one initial weight.

[0081] Step 420: Determine at least one weight based on the user sample set and at least one initial weight.

[0082] After selecting a set of target super-resolution processing models for the user, in step 410, this set of target super-resolution processing models can be used to perform a weight scan on the public sample set to obtain initial weights. The public sample set can be a pre-constructed standardized, neutral image quality evaluation dataset. The sample materials in it can be used to simulate and cover various mainstream image quality preferences that users may raise in the future (such as sharp text, smooth portraits, etc.), and are all taken from publicly available resources without copyright disputes.

[0083] Initial weights can cover different style preferences. For example, a set of initial weights can be obtained for common style preferences such as "sharp text," "smooth portraits," "film grain," and "neutral." In some embodiments, a fine-tuning range of ±5% can be set for each initial weight. By presetting the initial weights and fine-tuning ranges corresponding to standardized styles, the weight combination corresponding to a user can be determined quickly.

[0084] Based on the initial weights, the weights for each user are further optimized by combining the sample sets of each user. This allows the weight combination for each user to better meet the user's needs and improve the super-resolution processing effect.

[0085] In some embodiments of this application, reference is made to Figure 5 Step 410 includes steps 510 to 540.

[0086] Step 510: Determine the evaluation criteria corresponding to the user's style preferences.

[0087] Step 520: Determine the scan weight vector corresponding to at least one target super-resolution processing model. The scan weight vector includes at least one scan weight corresponding to each of the at least one target super-resolution processing models.

[0088] Step 530: Based on the public sample set and the scan weight vector, determine the comprehensive score corresponding to the scan weight vector.

[0089] Step 540: If the overall response score meets the evaluation criteria, at least one scan weight in the scan weight vector is determined as at least one initial weight.

[0090] During the weighted scanning process, a supervised search method can be used to determine optimal weight combinations for some typical style preferences. In step 510, for different style preferences, the attribute most relevant to that style preference can be determined as the evaluation condition. For example, for animation scenes, the corresponding evaluation condition can be determined based on Gradient Magnitude Similarity Deviation (GMSD), while for scenes with text, the corresponding evaluation condition can be determined based on Optical Character Recognition (OCR) accuracy. In one example, the evaluation condition can be a set of comprehensive indicators aimed at quantifying the overall quality of the generated image under the current weight combination. Specifically, the comprehensive evaluation score can be calculated based on multiple sub-indicators of different dimensions and their corresponding weighting coefficients. For example, it can include sub-indicators such as distortion, structure, and perceptual quality, as well as other optional sub-indicators related to style preferences, such as OCR accuracy for text, and the indicator weights can be adjusted according to style preferences. In one example, the evaluation condition can be determined according to the following formula: .

[0091] in: The score represents the overall evaluation score; a higher score indicates a better overall generation effect. λ1, λ2, and λ3 are the weighting coefficients for each indicator, all of which are non-negative real numbers. In one embodiment, λ1 + λ2 + λ3 = 1. A pixel-level distortion metric is used to quantify the difference in pixel values ​​between a generated image and a reference image. For example, it may include mean squared error (MSE) or mean absolute error (MAE). This refers to structural similarity metrics used to evaluate the degree to which images retain structural information such as texture and contour. For example, it can include the structural similarity index (SSIM). The perceptual quality index is used to measure the naturalness of an image in deep feature space or human vision. Examples include Learning Perceptual Patch Similarity (LPIPS). N(·) represents the normalization and orientation alignment function, which maps each original index I to a unified value range (e.g., [0,1]) and unifies its directionality (i.e., a larger value indicates better quality). Specifically, for indices where smaller values ​​are better (e.g., MSE, LPIPS), N(I) = 1 - Norm(I); for indices where larger values ​​are better (e.g., SSIM), N(I) = Norm(I), where Norm(·) is the range normalization operation. Furthermore, to adapt to the evaluation needs of specific scenarios, the index weights can be adjusted according to style preferences. For example, the formula can be extended to include a fourth term λ4· ,in The accuracy of text recognition (e.g., optical character recognition accuracy) is used to ensure that the readability of the text content is included in the comprehensive evaluation, satisfying λ1 + λ2 + λ3 + λ4 = 1.

[0092] In step 520, a scan weight vector is determined for each target super-resolution processing model. In one example, for n target super-resolution processing models, a scan weight vector can be defined as: W = [w1, w2, ..., wn], ∑wi = 1, wi ≥ 0, i = 1, 2…n. Here, w1 to wn represent the scan weights corresponding to the n target super-resolution processing models. Thus, based on a finite number of discrete target models, this application constructs a continuous, multi-dimensional visual style space. This mechanism breaks the limitation of a fixed style for a single model, enabling the coverage of fine-grained preference differences between different users through extremely low-cost linear adjustment, while simultaneously achieving complementary advantages of heterogeneous models in terms of structural fidelity and texture richness.

[0093] In step 530, the weight vector can be scanned based on a public sample set to determine the comprehensive score. Scanning methods can include grid search, random search, and Bayesian optimization. Grid search performs an exhaustive scan within the weight space (e.g., the weights of each model range from 0 to 1 with a step size of 0.1); random search randomly samples a large number of weight combinations within the weight space for evaluation; Bayesian optimization builds a probabilistic model based on the effects of tried combinations to predict which untried regions might be better, and proactively selects the next point with the "greatest potential" for evaluation.

[0094] In step 540, after scanning, if the comprehensive score of a certain set of scan weight vectors can meet the previously determined evaluation conditions, then each scan weight in the scan weight vector can be determined as the initial weight corresponding to each target super-resolution processing model.

[0095] For different style preferences, a set of initial weights will be obtained for each style preference. Therefore, users can choose an initial weight corresponding to a style as the initial value for weight adjustment. In step 420, the target super-resolution processing model can be used to predict the user sample set to obtain the corresponding predicted image. The initial weights are then used to weight the predicted image. Based on the output image obtained from the final weighted combination, the weights are adjusted and optimized on the basis of the initial weights. By gradually adjusting each weight in the weight combination, the output image will be gradually optimized. When the final output image meets the user's requirements, the adjusted weights can be used as a set of weights corresponding to that user, and a weight combination corresponding to that user can be obtained based on this set of weights.

[0096] In some embodiments, the weight combination can be fine-tuned and aligned according to user preferences. For example, in step 420, the adjustment and optimization process of the weight combination can be configured into two implementation modes: supervised fitting and weakly supervised guidance, depending on the data completeness of the user sample set (e.g., whether high-resolution ground truth images exist). If the user sample set contains pairs of ground truth images, supervised optimization based on an objective function can be performed. By constructing a combined loss function of the pixel domain and the perceptual domain, the weights are iteratively adjusted using a gradient descent algorithm so that the model's output mathematically directly fits the user's ideal image quality. If the user sample set lacks ground truth (e.g., in blind super-resolution scenarios), external guidance information can be introduced to perform style alignment. On the one hand, distribution alignment based on reference maps can be used, that is, calculating and minimizing the distance between the predicted image and the style reference map provided by the user (which is neither the ground truth corresponding to the input image nor the public sample set described above, but can be understood as a set of ideal effect images selected by the user) on the feature statistical distribution (such as the Gram matrix) to simulate the texture of the target. On the other hand, preference learning based on interactive feedback can be used to transform the user's selection or ranking behavior of multiple candidate images into sparse feedback signals, and use pairwise ranking loss or derivativeless optimization algorithms to update the weights in reverse, thereby approximating the user's subjective aesthetic preferences through iteration in the absence of absolute ground truth.

[0097] Specifically, in some embodiments, when the user sample set contains high-resolution ground truth images, supervised learning-based weight optimization can be used. That is, if the user sample set contains pairs of low-resolution images and high-resolution ground truth images, supervised optimization based on objective function regression can be performed. Specifically, an image quality loss function can be constructed, including but not limited to pixel domain loss (such as L1 norm, L2 norm, etc., used to constrain color and brightness consistency) and perceptual domain loss (such as LPIPS, VGG Feature Loss, etc., used to constrain texture and structure consistency). Starting with initial weights, the weighted sum of the outputs of multiple target super-resolution processing models is used as the prediction result. Gradient descent or least squares methods are used to iteratively update the weight values ​​in the weight combination until the combined loss function value between the prediction result and the high-resolution ground truth image is minimized. This achieves a direct mathematical fit to the user's ideal image quality.

[0098] In other embodiments, for user sample sets containing only low-resolution images (e.g., blind super-resolution scenarios), weight optimization based on weak supervision or unsupervised methods can be used. That is, if the user sample set contains only low-resolution images (blind super-resolution scenarios), style alignment optimization can be performed using external guidance information. Depending on the source of the external guidance information, two implementation methods can be specifically included. In the first method, reference-based style matching can be used. Fitting based on feature statistical distributions is performed using one or more style reference images uploaded by the user. The style reference images are used to represent the texture desired by the user (e.g., film graininess, sharpness distribution, etc.) and do not need to correspond in content with the image to be processed, nor do they need to be images from a publicly available sample set. Specifically, high-dimensional style feature vectors can be extracted from the weighted fused output image and the style reference images using a feature extraction network (e.g., a pre-trained convolutional neural network). The style feature vectors can be constructed based on statistics such as the Gram matrix, mean, and variance of the feature maps. Further adjustments can be made to the weight combination to minimize the distance between the output image and the style reference image in the style feature vector space (such as Kullback-Leibler divergence or Euclidean distance), thereby making the final output image approximate the user-provided reference paradigm in terms of texture style distribution. In the second approach, interactive preference learning can be used. Based on the user's selection of candidate images, iterative optimization based on learning-to-rank or reinforcement learning can be performed. Specifically, multiple sets of perturbation weights can be generated in the parameter space based on the current weight combination, and corresponding candidate image sets can be generated in parallel and displayed to the user, receiving user preference feedback signals for the candidate images (e.g., clicking on a "better" image, or performing pairwise comparisons of images). Based on the accumulated preference feedback signals, a pairwise ranking loss function can be constructed to maximize the predicted score of the user-selected image and minimize the score of the unselected image; alternatively, the user's selection behavior can be treated as a sparse reward, and the search direction of the weight combination can be updated using derivative-free optimization algorithms (such as Bayesian optimization or evolutionary strategies). Through multiple rounds of "display-selection-update" iterations, the weight combination gradually converges to the optimal solution perceived by the user's subjective visual perception.

[0099] Determining the initial weights based on the comprehensive score of each scan weight vector in the super-resolution processing process allows different initial weight combinations to have specific style biases, facilitating further adjustments when determining the weight combinations for each user.

[0100] Determining the appropriate super-resolution processing model and weight combination based on different users' latency budgets and / or style preferences can ensure that the final output image meets the user's needs as much as possible.

[0101] Using the above scheme, the super-resolution quality of the output image obtained after fusion can be greatly reduced through a three-layer protection approach: the selection constraints of the model group, the comprehensive evaluation index of the weight scan, and the weighting scheme.

[0102] In some embodiments of this application, step 140 includes: For any one of the at least one target super-resolution processing models: In response to the fact that the computing resources occupied by the target super-resolution processing model meet the preset resource conditions, the image to be processed is input into the target super-resolution processing model.

[0103] As mentioned above, images from multiple users will be batched and then input into their respective target super-resolution processing models. This means that parallel processing is possible for a single super-resolution processing model. Since different users use different target super-resolution processing models, the request load on each model may vary, potentially leading to some models being busier and experiencing longer queuing times. Therefore, resource conditions for each super-resolution processing model can be set individually, with each model maintaining independent queuing and concurrency limits. Only when the computing resources used by a super-resolution processing model meet the pre-set resource conditions (e.g., the current number of concurrent tasks has not reached the limit) will the image be input into that model for processing. This reduces the risk of one model's popularity slowing down other models and affecting other users. Configure independent resource conditions for each super-resolution processing model to implement a resource isolation mechanism. For example, limit the number of GPU execution slots (parallel tasks) occupied by a certain super-resolution processing model at one time. This ensures that even if there is a peak in requests, the computing power occupied by the super-resolution processing model is limited to a preset range, thereby preventing congestion from spilling over to other users who do not depend on the super-resolution processing model.

[0104] In some embodiments, for significantly busy super-resolution processing models, they can be deployed simultaneously on multiple machines to distribute queuing pressure. In some embodiments, during busy periods, such as when a large number of users are processing simultaneously, the maximum waiting time for combining images to be processed can be controlled to reduce the additional queuing time caused by batch processing. In addition, if a super-resolution processing model processes slowly during busy periods and repeatedly approaches the user's latency budget, a capacity alarm can be issued, prompting the need to increase the computing power of the super-resolution processing model or to negotiate with the relevant users to adjust processing specifications (such as frame rate, resolution, etc.).

[0105] Using a super-resolution processing model only when its computational resources are sufficient can reduce the probability of prolonged processing times due to insufficient resources for a particular model, thereby improving the efficiency of parallel super-resolution processing. Furthermore, mechanisms such as batch timeouts and priority queues can be introduced to ensure timely processing of requests from high-priority users or those with high real-time requirements.

[0106] A specific embodiment of this application is described below. It should be understood that this specific embodiment is described for illustrative purposes only and should not be construed as limiting the scope of this application.

[0107] like Figure 7 As shown, there are three users (live streams): an English teaching channel, a gaming channel, and an outdoor channel. The three users have different style preferences: the English teaching channel prioritizes text clarity, especially small fonts; the gaming channel prioritizes UI sharpness and edge clarity; and the outdoor channel prioritizes low noise processing and natural skin tone transitions. The model pool includes three super-resolution processing models: Super-resolution Processing Model 1, Super-resolution Processing Model 2, and Super-resolution Processing Model 3. Based on the latency budget and style preferences of different users, the super-resolution processing models are first selected to determine if their latency meets the latency budget of each user. For each user, the latency of the corresponding target super-resolution processing model group is tested to ensure it meets their latency budget. Then, based on whether each super-resolution processing model matches the style preferences of different users, a set of target super-resolution processing models is obtained for each live stream. The English teaching channel corresponds to Super-resolution Processing Model 1 and Super-resolution Processing Model 2, the gaming channel corresponds to Super-resolution Processing Model 2 and Super-resolution Processing Model 3, and the outdoor channel corresponds to Super-resolution Processing Model 1 and Super-resolution Processing Model 3. Further, weight combinations corresponding to each user are determined. The English teaching channel corresponds to the text sharpness weight combination W1, the game channel corresponds to the UI sharpness weight combination W2, and the outdoor channel corresponds to the low noise and natural weight combination W3. The weight combination corresponding to each user is written into the weight lookup table LUT.

[0108] When three live streams are broadcasting simultaneously, the images to be processed from the three live streams (i.e., the frames to be processed in the live video) are batched. Each image carries its live stream ID, resulting in a batch of images, which are then input into the model pool. Based on the live stream ID, the target super-resolution processing model corresponding to each live stream is determined, and the image to be processed for that live stream is input into the corresponding target super-resolution processing model. Specifically, English teaching frames are input into super-resolution processing model 1 and super-resolution processing model 2, game frames into super-resolution processing model 2 and super-resolution processing model 3, and outdoor frames into super-resolution processing model 1 and super-resolution processing model 3. For each image to be processed, the target super-resolution processing model obtains a corresponding predicted image, i.e., a predicted frame for each frame to be processed. The predicted frames are then weighted and combined using the formula I_final(x,y;c)=∑wi·I_i(x,y;c) to obtain the final output SR frame. In the formula, wi≥0,∑wi=1, I_final(x,y;c) represents the final output frame, I_i(x,y;c) represents the predicted frame obtained by each target super-resolution processing model, and wi represents the weights in the weight combination corresponding to each target super-resolution processing model.

[0109] like Figure 7 As shown, for English teaching frames, the predicted frames obtained by super-resolution processing model 1 and super-resolution processing model 2 are weighted according to the text sharpness weight combination W1 to obtain English teaching SR frames; for game frames, the predicted frames obtained by super-resolution processing model 2 and super-resolution processing model 3 are weighted according to the UI sharpness weight combination W2 to obtain game SR frames; for outdoor frames, the predicted frames obtained by super-resolution processing model 1 and super-resolution processing model 3 are weighted according to the low-noise naturalness weight combination W3 to obtain outdoor SR frames. English teaching SR frames will be output to the English teaching channel, game SR frames will be output to the game channel, and outdoor SR frames will be output to the outdoor channel.

[0110] In addition, queues will be maintained separately for each super-resolution processing model. Concurrency limits and batch waiting restrictions will also be implemented.

[0111] Based on the same technical concept, embodiments of this application provide a multi-user super-resolution processing apparatus. Embodiments of the multi-user super-resolution processing apparatus can be referred to those of the multi-user super-resolution processing method, and repeated details will not be repeated. Reference Figure 8 The multi-user super-resolution processing device 800 includes a combination module 810, an input module 820, a generation module 830, and an output module 840.

[0112] The combination module 810 is used to combine multiple images to be processed from multiple users to generate a batch image combination.

[0113] The input module 820 is used to combine batch images into a pre-built model pool. The model pool includes multiple super-resolution processing models.

[0114] The generation module 830 is used for any image to be processed in a batch of images: determining at least one target super-resolution processing model corresponding to the image to be processed from the model pool; inputting the image to be processed into the at least one target super-resolution processing model; outputting at least one predicted image corresponding to the image to be processed by the at least one target super-resolution processing model; and generating an output image corresponding to the image to be processed based on a predetermined weight combination and at least one predicted image.

[0115] The output module 840 is used to output the output image to the user corresponding to the image to be processed.

[0116] The combination module 810, input module 820, generation module 830, and output module 840 in the multi-user super-resolution processing apparatus 800 can correspond to steps 110 to 170 in the multi-user super-resolution processing method 100, and will not be described in detail here for the sake of brevity. It should be understood that, corresponding to the embodiments of the multi-user super-resolution processing method 100, the embodiments of the multi-user super-resolution processing apparatus 800 may also include more modules.

[0117] It should be noted that the functions of the modules discussed herein can be divided into multiple modules, and / or at least some functions of multiple modules can be combined into a single module. The specific actions performed by a particular module discussed herein include the specific module itself performing the action, or alternatively, the specific module calling or otherwise accessing another component or module that performs the action (or performs the action in conjunction with the specific module). Therefore, a specific module performing an action can include the specific module performing the action itself and / or another module that performs the action, called or otherwise accessed by the specific module.

[0118] It should also be understood that this article can describe various technologies in the general context of software and hardware components or program modules. The above regarding... Figure 8The described modules can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these modules can be implemented as computer program code / instructions configured to execute in one or more processors and stored in a computer-readable storage medium. Alternatively, these modules can be implemented as hardware logic / circuit. Hardware logic / circuit may include integrated circuit chips (which include processors (e.g., Central Processing Unit (CPU), microcontrollers, microprocessors, digital signal processors (DSPs), etc.), memory, one or more communication interfaces, and / or one or more components in other circuitry), and may optionally execute received program code and / or include embedded firmware to perform functions.

[0119] This application provides a computing device 900, such as... Figure 9 As shown. Figure 9 An example configuration of a computing device 900 that can be used to implement the multi-user super-resolution processing method 100 described herein is shown. For example, the multi-user super-resolution processing apparatus 800 described above may be implemented wholly or at least partially by the computing device 900 or a similar device or system.

[0120] The computing device 900 may include at least one processor 905 capable of communicating with each other, such as via a bus 904 or other suitable connection, a memory 907, multiple communication interfaces 902, a display device 901, other input / output (I / O) devices 903, and one or more mass storage devices 906. Instructions are stored on the memory 907 that, when executed by the processor 905, cause the processor 905 to perform the super-resolution processing method as described in the above embodiments.

[0121] The computing device 900 can be a variety of different types of devices. Examples of the computing device 900 include, but are not limited to: desktop computers, server computers, laptop or netbook computers, mobile devices (e.g., tablets, cellular or other wireless phones (e.g., smartphones), notebook computers, mobile stations), wearable devices (e.g., glasses, watches), entertainment devices (e.g., entertainment appliances, set-top boxes communicatively coupled to a display device, game consoles), televisions or other display devices, automotive computers, and so on.

[0122] Processor 905 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 905 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, processor 905 may be configured to fetch and execute computer-readable instructions stored in memory 907, mass storage device 906, or other computer-readable media, such as program code of operating system 908, program code of application program 909, program code of other program 910, etc.

[0123] Memory 907 and mass storage device 906 are examples of computer-readable storage media for storing instructions that are executed by processor 905 to perform the various functions described above. For example, memory 907 can generally include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, mass storage device 906 can generally include hard disk drives, solid-state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Both memory 907 and mass storage device 906 can be collectively referred to herein as memory or computer-readable storage media, and can be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which can be executed by processor 905 as a specific machine configured to perform the operations and functions described in the examples herein.

[0124] Multiple programs can be stored on mass storage device 906. These programs include operating system 908, one or more application programs 909, other programs 910, and program data 911, and they can be loaded into memory 907 for execution. Examples of such application programs or program modules may include, for example, computer program logic (e.g., computer program code or instructions) for implementing components / functions such as: multi-user super-resolution processing apparatus 800 (including combination module 810, input module 820, generation module 830, and output module 840), multi-user super-resolution processing method 100 (including any suitable steps of super-resolution processing method 100), and / or other embodiments described herein.

[0125] Although Figure 9The data is illustrated as being stored in memory 907 of computing device 900, but operating system 908, application program 909, other programs 910 and program data 911 or portions thereof may be implemented using any form of computer-readable medium accessible by computing device 900.

[0126] One or more communication interfaces 902 are used for exchanging data with other devices, such as via a network, direct connection, etc. Such communication interfaces can be one or more of the following: any type of network interface (e.g., a network interface card, or NIC), wired or wireless (such as IEEE 802.11 Wireless LAN, or WLAN) wireless interface, Wi-MAX interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth™ interface, Near Field Communication (NFC) interface, etc. Communication interface 902 can facilitate communication across various network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, etc. Communication interface 902 can also provide communication with external storage devices (not shown), such as storage arrays, network-attached storage, storage area networks, etc.

[0127] In some examples, a display device 901, such as a monitor, may be included for displaying information and images to the user. Other I / O devices 903 may be devices that receive various inputs from the user and provide various outputs to the user, and may include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, and so on.

[0128] The technologies described herein can be supported by these various configurations of computing device 900, and are not limited to specific examples of the technologies described herein. For example, the functionality can also be implemented wholly or partially on a “cloud” using a distributed system. A cloud includes and / or represents a platform for resources. The platform abstracts the underlying functionality of the cloud’s hardware (e.g., servers) and software resources. Resources may include applications and / or data that can be used when performing computational processing on servers remote from computing device 900. Resources may also include services provided via the Internet and / or via subscriber networks such as cellular or Wi-Fi networks. The platform can abstract resources and functionality to connect computing device 900 to other computing devices. Therefore, the implementation of the functionality described herein can be distributed throughout the cloud. For example, the functionality can be implemented partly on computing device 900 and partly through a platform that abstracts the functionality of the cloud.

[0129] This application also provides a computer-readable storage medium storing instructions that, when executed individually or jointly by one or more processors of a computing device, cause the computing device to perform the methods described in any of the above embodiments.

[0130] Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, Digital Universal Disc (DVD) or other optical storage devices, magnetic cassettes, magnetic tapes, disk storage devices or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by computer equipment.

[0131] This application also provides a computer program product including instructions that, when executed individually or jointly by one or more processors of a computing device, cause the computing device to perform the methods as described in any of the above embodiments.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application, and they should all be covered within the scope of the claims and specification of this application. In particular, as long as there is no structural conflict, the various technical features mentioned in the embodiments can be combined in any way. This application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A multi-user super-resolution processing method, characterized by, The method comprises: combining a plurality of to-be-processed images respectively from a plurality of users to generate a batch image combination; inputting the batch image combination into a pre-constructed model pool, the model pool comprising a plurality of super-resolution processing models; for any to-be-processed image in the batch image combination: determining at least one target super-resolution processing model corresponding to the to-be-processed image from the model pool; inputting the to-be-processed image into the at least one target super-resolution processing model; outputting at least one predicted image corresponding to the to-be-processed image respectively by the at least one target super-resolution processing model; generating an output image corresponding to the to-be-processed image based on a pre-determined weight combination and the at least one predicted image; and outputting the output image to a user corresponding to the to-be-processed image. The super-resolution processing method further comprises:

2. The super-resolution processing method of claim 1, wherein, for any user in the plurality of users: determining at least one target super-resolution processing model corresponding to the user from the model pool based on a time delay budget and / or a style preference of the user, the time delay budget being used to indicate a time delay requirement of the user for super-resolution processing, and the style preference being used to indicate an image style of the user; determining at least one weight corresponding to the at least one target super-resolution processing model respectively; and determining the weight combination based on the at least one weight. The determining at least one target super-resolution processing model corresponding to the to-be-processed image from the model pool comprises:

3. The super-resolution processing method of claim 2, wherein, determining the at least one target super-resolution processing model from the model pool according to a number of a user corresponding to the to-be-processed image. The determining at least one weight corresponding to the at least one target super-resolution processing model respectively comprises:

4. The super-resolution processing method of claim 2, wherein, performing weight scanning using the at least one target super-resolution processing model based on a public sample set to obtain at least one initial weight; and determining the at least one weight based on a user sample set and the at least one initial weight. The performing weight scanning using the at least one target super-resolution processing model based on a public sample set to obtain at least one initial weight comprises:

5. The super-resolution processing method of claim 4, wherein, determining an evaluation condition corresponding to the style preference of the user; determining a scanning weight vector corresponding to the at least one target super-resolution processing model, the scanning weight vector comprising at least one scanning weight corresponding to the at least one target super-resolution processing model respectively; determining a comprehensive score corresponding to the scanning weight vector based on the public sample set and the scanning weight vector; and in response to the comprehensive score satisfying the evaluation condition, determining at least one scanning weight in the scanning weight vector as the at least one initial weight. The determining at least one target super-resolution processing model corresponding to the user from the model pool based on a time delay budget and / or a style preference of the user comprises:

6. The super-resolution processing method of any one of claims 2-5, wherein, determining at least one super-resolution processing model from the model pool; determining a super-resolution processing time delay of the at least one super-resolution processing model; and determining a super-resolution processing time delay of the at least one super-resolution processing model; and In response to the super-resolution processing time delay of the at least one super-resolution processing model satisfying the time delay budget, the at least one super-resolution processing model is determined as the at least one target super-resolution processing model corresponding to the user.

7. The super-resolution processing method of any one of claims 1-5, wherein, The inputting of the to-be-processed image into the at least one target super-resolution processing model comprises: For any target super-resolution processing model in the at least one target super-resolution processing model: In response to the computing resource occupied by the target super-resolution processing model satisfying a preset resource condition, the to-be-processed image is input into the target super-resolution processing model.

8. A multi-user super-resolution processing apparatus, characterized by comprising: Comprise: The combination module is configured to combine a plurality of to-be-processed images respectively from a plurality of users to generate a batch image combination; The input module is configured to input the batch image combination into a model pool pre-constructed, the model pool comprising a plurality of super-resolution processing models; The generation module is configured to, for any to-be-processed image in the batch image combination: determine at least one target super-resolution processing model corresponding to the to-be-processed image from the model pool; input the to-be-processed image into the at least one target super-resolution processing model; Output at least one predicted image corresponding to the to-be-processed image by the at least one target super-resolution processing model respectively; generate an output image corresponding to the to-be-processed image based on a pre-determined weight combination and the at least one predicted image; and And The output module is configured to output the output image to a user corresponding to the to-be-processed image.

9. A computing device, comprising: Comprise: At least one processor; And At least one memory connected with the at least one processor in communication, the at least one memory storing instructions, the instructions, when executed by the at least one processor alone or collectively, causing the computing device to perform the super-resolution processing method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The instructions, when executed by one or more processors of a computing device alone or collectively, cause the computing device to perform the super-resolution processing method of any one of claims 1 to 7.

11. A computer program product, characterised in that, The instructions, when executed by one or more processors of a computing device alone or collectively, cause the computing device to perform the super-resolution processing method of any one of claims 1 to 7.

Images