Super-division model training method, super-division method, equipment and storage medium

Abstract

The invention discloses a super-division model training method, a super-division method, equipment and a storage medium, and belongs to the technical field of image processing. The method comprises the following steps: acquiring a multi-scale residual fusion network model with residual connection introduced across network layers in a plurality of network layers as a to-be-trained super-division model; a training set comprising data sources of various applications is obtained, the data source of each application comprises a plurality of image pairs corresponding to the applications, and each image pair comprises a pair of low-resolution images and high-resolution images with the same content; and training a to-be-trained super-division model according to the training set to obtain a target super-division model for super-division of multiple applications. Residual connection is introduced across network layers in the network structure of the model, that is, the multi-scale residual fusion network model is introduced to serve as the to-be-trained super-division model, and multiple compatible data sources are adopted for model training, so that the convergence effect after model training is better, and the model is more accurate.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a super-resolution model training method, super-resolution method, device and storage medium. Background Technology

[0002] With the rapid development of terminal technology and gaming applications, players' demands for game graphics quality are also increasing. However, high graphics quality is often accompanied by high power consumption, leading to problems such as overheating and battery life. To reduce power consumption while providing high graphics quality, manufacturers have introduced game super-resolution technology. The principle is to first render a low-resolution image using a graphics processing unit (GPU), and then use a super-resolution model to perform super-resolution processing on the low-resolution image (such as upscaling, sharpening, and anti-aliasing) to obtain an image close to the native resolution, thereby restoring image details.

[0003] Currently, in the field of super-resolution, considering the issue of high-quality image processing, the data sources for super-resolution models corresponding to different games all come from the game's own visual data. In other words, for a single game, the model is trained based on a single data source. For example, a super-resolution model for game application A is trained based on the data source for game application A, and a super-resolution model for game application B is trained based on the data source for game application B. Therefore, considering model generalization, there is an urgent need to train a universal super-resolution model to super-resolution multiple games. However, simply training multiple data sources together will result in training failure to converge, and the model's accuracy cannot be guaranteed. Summary of the Invention

[0004] This application provides a super-resolution model training method, super-resolution method, device, and storage medium, which can be compatible with multiple data sources for model training and improve the convergence performance of the trained model, thereby improving the accuracy of the model. The technical solution is as follows:

[0005] Firstly, a super-resolution model training method is provided, the method comprising:

[0006] Obtain the super-resolution model to be trained, which is a multi-scale residual fusion network model. The multi-scale residual fusion network model includes multiple network layers with residual connections introduced across network layers. Obtain the training set, which includes data sources from various applications. Each application's data source includes multiple image pairs corresponding to the application. Each image pair includes a pair of low-resolution images and a high-resolution image with the same content. Based on the training set, train the super-resolution model to be trained to obtain the target super-resolution model. The target super-resolution model is used to perform super-resolution processing on application images from various applications.

[0007] In this context, a high-resolution image has a higher resolution than a low-resolution image. The terms "low-resolution image" and "high-resolution image" in this application are relative; each image pair includes two images with the same content but different resolutions. The image with the lower resolution is called the low-resolution image, and the image with the higher resolution is called the high-resolution image.

[0008] In this embodiment, by introducing residual connections across network layers into the model's network structure—that is, introducing a multi-scale residual fusion network model as the super-resolution model to be trained—on the one hand, the output of the previous layer can be directly added to the input of the next layer, which helps the gradient to propagate directly to earlier layers during backpropagation, thereby alleviating the problem of zero directional gradient propagation during model training convergence. On the other hand, the introduced residual connections allow the gradient to bypass certain layers and propagate directly to shallower layers during backpropagation, enabling the gradient to propagate better within the network layers, reducing gradient decay and accumulation during propagation, and thus reducing the possibility of gradient explosion. Therefore, by using a multi-scale residual fusion network model with residual connections across network layers as the super-resolution model to be trained, and using compatible multiple data sources for model training, the problem of zero directional gradient propagation or gradient explosion during model training convergence can be alleviated, thereby improving training convergence performance, resulting in better convergence effect and faster convergence speed, and thus obtaining a more accurate general super-resolution model capable of super-resolution for multiple applications (such as multiple game applications).

[0009] In one embodiment, a multi-scale residual fusion network model can be obtained by introducing residual connections across network layers in the network structure of the original super-resolution model. That is, the super-resolution model to be trained can be constructed by introducing residual connections across network layers in the already constructed network structure of the original super-resolution model. Alternatively, depending on the network structure requirements of the super-resolution model to be trained, multiple network layers can be constructed first, and then residual connections can be added across these multiple network layers to construct the super-resolution model to be trained. This application does not limit the method of constructing the super-resolution model to be trained.

[0010] In this context, the original super-resolution model refers to a super-resolution model currently designed for a single application, i.e., a super-resolution model used for super-resolution processing of a single application. The original super-resolution model can be a convolutional neural network model, etc. For example, in a game scenario, the original super-resolution model refers to a super-resolution model specific to a single game application.

[0011] In one embodiment, residual connections are introduced between two network layers every T layers, starting from a specific initial layer. These residual connections, introduced every T layers, are used to pass the output features of the preceding network layer to the next network layer, allowing the output features of the preceding layer to be fused with the original input features of the next network layer, which then becomes the new input features for that next network layer.

[0012] The specific starting layer can be preset, for example, it can be the first network layer or the second network layer, etc., and this application embodiment does not limit this. T is a positive integer and can be set as needed. T is usually a small value so as to introduce residual connections across a small number of network layers. For example, T is 1 or 2, etc., and this application embodiment does not limit this.

[0013] As an example, a residual network is introduced between two network layers, starting from a specific starting layer, every other network layer in a multi-layer network. The specific starting layer can be the first network layer or the second network layer.

[0014] In one embodiment, the multi-scale residual fusion network model includes N network layers. A residual connection is introduced between the i-th layer and the (i+2)-th layer. The residual connection between the i-th layer and the (i+2)-th layer is used to pass the output features of the i-th layer to the (i+2)-th layer, so that the output features of the i-th layer can be fused with the output features of the (i+1)-th layer and used as the input features of the (i+2)-th layer. Here, i is an odd or even number greater than or equal to 1 and less than or equal to N-2.

[0015] In one embodiment, residual connections are introduced between the first and last layers of multiple network layers. The residual connections between the first and last layers are used to pass the output features of the first layer to the last layer, so that the output features of the first layer can be fused with the output features of the previous layer of the last layer and used as the input features of the last layer.

[0016] As an example, the aforementioned multiple applications can be different applications within the same category of applications, or applications from different categories, as described in this embodiment. For instance, in a game scenario, to train a target super-resolution model capable of super-resolution for multiple game applications, a training set including data sources from multiple game applications can be obtained. That is, the aforementioned multiple applications can be different game applications within the game category, and the data source for each game application includes multiple image pairs corresponding to the game application. Each image pair can include a pair of low-resolution and high-resolution game frames from the corresponding game application. It should be understood that the aforementioned multiple applications can be different game applications from other categories of applications, such as different applications in video applications, different applications in instant messaging applications, or different applications in message aggregation applications, etc., and this embodiment does not limit this.

[0017] In one embodiment, the various application data sources in the training set (hereinafter referred to as multiple data sources) are in a balanced state in terms of feature dimensions, that is, the number of image features from the multiple data sources is in a balanced state. The number of image features from each data source refers to the total number of features contained in all images of each data source. The balanced state of the number of image features from multiple data sources means that the number of image features from the multiple data sources is evenly distributed, and the difference between the number of image features from any two data sources is less than a preset threshold, that is, the number of image features from various data sources is not significantly different.

[0018] By acquiring multiple data sources with balanced feature dimensions as training sets, the samples of different categories can maintain a certain balance in the feature dimensions. That is, the distribution of samples of each category in the feature space is relatively uniform, and no one category has an absolute advantage in a certain feature. Such feature-balanced training samples can reduce prediction bias, improve generalization ability, and avoid overfitting and underfitting, thereby improving the accuracy of the trained model.

[0019] In this embodiment, when the features of samples from different categories in the training samples are balanced, the model can learn the features of each category more comprehensively, reducing prediction bias caused by data bias. Furthermore, a balanced data distribution helps the model form a more impartial and unbiased decision boundary during training, thereby improving prediction accuracy. Secondly, feature-balanced training samples help the model learn more generalized feature representations, enabling the model to make more accurate predictions when faced with new data (such as various unknown or unseen data), thus improving the model's generalization ability. In addition, feature imbalance often leads to overfitting to one category and underfitting to others. By adjusting the sample features in the training samples to achieve balance, this situation can be avoided to some extent.

[0020] In one embodiment, the operation of obtaining a training set includes: obtaining an initial training set, which includes initial data sources for multiple applications, and each initial data source for an application includes several initial image pairs for each application; and augmenting the initial training set with a feature enhancement algorithm to obtain a training set, wherein the amount of data from each application's data source in the training set is greater than the amount of data from the initial data source of the corresponding application in the initial training set, and the data sources for multiple applications in the training set are balanced in terms of feature dimensions.

[0021] Thus, before training the model, the initial training set can be augmented using feature enhancement algorithms to obtain a training set that is balanced in terms of feature dimensions, thereby improving the accuracy of the super-resolution model to be trained.

[0022] In one embodiment, the operation of augmenting the initial training set with a feature enhancement algorithm to obtain a training set includes: obtaining a training set to be augmented based on the initial training set, wherein the training set to be augmented is either the initial training set or obtained by augmenting the initial training set with an image transformation algorithm; and performing augmentation processing on the training set to be augmented using a feature enhancement algorithm to obtain the training set.

[0023] In other words, the initial training set can be directly augmented using a feature enhancement algorithm to obtain the training set; or, an image transformation algorithm can be used to augment the initial training set to obtain an intermediate training set, and then a feature enhancement algorithm can be used to augment the intermediate training set to obtain the training set.

[0024] Image transformation refers to the process of converting an image from one representation to another to change attributes such as size, shape, orientation, or color. Image transformation algorithms can include geometric transformations, pixel value transformations, affine transformations, Fourier transforms, etc., and this application does not limit the image transformation algorithms. Geometric transformations can change the coordinate positions of pixels in an image. For example, geometric transformations can include translation, rotation, scaling, and mirroring. Pixel value transformations mainly change attributes such as color or brightness of pixels in an image without changing the coordinate positions of the pixels. For example, pixel value transformations can include brightness adjustment, contrast adjustment, color adjustment, and sharpening.

[0025] Feature enhancement algorithms are used to modify images along the feature dimension. In one embodiment, the operation of using a feature enhancement algorithm to expand the training set to obtain the training set includes: expanding each image in a first image pair along the feature dimension to obtain M expanded images corresponding one-to-one with M features in each image, where the first image pair is any one of all or part of the image pairs in the training set to be expanded, and M is a positive integer; generating M image pairs based on the M expanded images corresponding to each image in the first image pair, and the training set includes M image pairs.

[0026] In other words, for each image pair in the initial training set or a portion of the training set to be expanded, the above method can be used to expand one image pair into multiple image pairs in the feature dimension. For example, if both images in an image pair include 3 features, then by performing a feature enhancement algorithm to expand this image pair once, 3 image pairs corresponding one-to-one with these 3 features can be obtained. Performing a feature enhancement algorithm to expand this image pair twice will yield 6 image pairs. The more times the expansion process is performed, the more images are obtained. Moreover, since image expansion is performed based on the features contained in the images in each image pair, feature balance among multiple data sources can be ensured by controlling the balance of data volume from various data sources.

[0027] In one embodiment, the operation of augmenting each image in the first image pair along the feature dimension to obtain M augmented images corresponding one-to-one with the M features in each image includes: performing convolution on the first image to obtain M feature maps corresponding one-to-one with the M features in the first image, where the first image is any image in the first image pair; adjusting each of the M feature maps to obtain M new feature maps; and performing deconvolution on each of the M new feature maps to obtain M new feature maps. Figure 1 One corresponding M augmented images.

[0028] In this way, the features of the original image can be changed in the model convolution process, thus preserving more features.

[0029] In one embodiment, the operation of adjusting each feature map in the M feature maps includes: randomly adjusting the first feature map, such as randomly perturbing the first feature map to obtain a new first feature map, wherein the first feature map is any one of the M feature maps.

[0030] In one embodiment, training the super-resolution model to be trained based on the training set may include the following steps: using the training set as input to the super-resolution model to be trained, outputting prediction results through the super-resolution model to be trained, calculating a loss function based on the prediction results, and adjusting the model parameters of the super-resolution model to be trained based on the loss function.

[0031] As an example, the super-resolution model can be trained multiple times using the training set. For each training cycle, the above training process can be followed. That is, for each training cycle, the following operations can be performed: use the training set as input to the super-resolution model to be trained, use the super-resolution model to output prediction results, calculate the loss function based on the prediction results, and adjust the model parameters of the super-resolution model to be trained based on the loss function.

[0032] In one embodiment, the loss function of the super-resolution model to be trained can be a cross-entropy loss function or a focal loss function, etc., and this application embodiment does not limit it.

[0033] In one embodiment, during model training, the training data can be adaptively adjusted. For example, the number of training iterations for difficult samples in the training set can be increased, or the number of training iterations for difficult samples in the training set can be increased while the number of training iterations for easy samples can be decreased. This allows the model to deepen its learning of difficult samples during training, avoiding the problem of insufficient learning of difficult samples.

[0034] As an example, during the training of the super-resolution model to be trained based on the training set, the training weight of the first part of the training data is increased. The first part of the training data refers to the training data in the training set whose loss function is greater than or equal to a first preset threshold during the training process. The training weight is used to indicate the number of training times for the corresponding training data.

[0035] By increasing the training weights of the first part of the training data, the number of training iterations for difficult samples can be increased, thereby deepening the model's learning of difficult samples during the model training process and avoiding the problem of insufficient learning of difficult samples.

[0036] As another example, during the training of the super-resolution model to be trained based on the training set, the training weight of the second part of the training data is reduced. The second part of the training data refers to the training data in the training set whose loss function is less than a second preset threshold during the training process.

[0037] By reducing the training weights of the second part of the training data, the number of training iterations for simple samples can be reduced, thus avoiding the problem of continuing to learn from simple samples even after they have been learned well.

[0038] In one embodiment, the loss function of the super-resolution model to be trained includes a regularization term, the value of the regularization parameter of the regularization term is greater than a preset threshold, and the regularization parameter is used to control the weight of the regularization term in the loss function.

[0039] A regularization parameter value greater than the preset threshold indicates that the regularization parameter is set relatively large. By setting a larger regularization parameter, the network can be prevented from getting stuck in local optima, the learning speed of the model can be accelerated, and the training efficiency of the model can be improved.

[0040] In one embodiment, the multi-scale residual fusion network model is obtained by introducing residual connections across network layers in the network structure of the original super-resolution model, where the original super-resolution model is a convolutional neural network model.

[0041] In one embodiment, the multiple applications are multiple game applications, and the target super-resolution model is used to perform super-resolution processing on the game screens of the multiple game applications.

[0042] Secondly, a super-resolution model training device is provided, which has the function of implementing the super-resolution model training method described in the first aspect above. The super-resolution model training device includes at least one module, which is used to implement the super-resolution model training method provided in the first aspect above.

[0043] Thirdly, a super-resolution model training apparatus is provided. The apparatus includes a processor and a memory. The memory stores programs that support the super-resolution model training apparatus in executing the super-resolution model training method provided in the first aspect, and stores data related to implementing the super-resolution model training method described in the first aspect. The processor is configured to execute the programs stored in the memory. The super-resolution model training apparatus may further include a communication bus for establishing a connection between the processor and the memory.

[0044] Fourthly, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the super-resolution model training method described in the first aspect.

[0045] Fifthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to execute the super-resolution model training method described in the first aspect.

[0046] The technical effects achieved by the second, third, fourth, and fifth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect mentioned above, and will not be repeated here.

[0047] Sixthly, a super-resolution method is provided, which is applied in an electronic device. The electronic device includes a first application, a graphics processing unit (GPU), and a target super-resolution model. The target super-resolution model is used to perform super-resolution processing on the application screens of multiple applications. The first application is any one of the multiple applications. The target super-resolution model is a multi-scale residual fusion network model, which includes multiple network layers with residual connections introduced across network layers. The method includes: after the first application is started, rendering the first application screen of the first application through the GPU; performing super-resolution processing on the first application screen through the target super-resolution model to obtain a second application screen corresponding to the first application screen, wherein the resolution of the second application screen is greater than that of the first application screen; and displaying the second application screen.

[0048] In this embodiment, by configuring a target super-resolution model in the electronic device with a network structure of multi-scale residual fusion network and capable of super-resolution of multiple applications, after any application in the multiple applications is launched, the electronic device can first render the low-resolution application screen through the GPU, and then perform super-resolution processing on the low-resolution application screen through the target super-resolution model to obtain a high-resolution image screen, and then display the high-resolution image screen to the user.

[0049] In this way, high-quality application visuals can be provided to users while saving device power consumption. Moreover, the target super-resolution model configured in electronic devices has high versatility and can be applied to a variety of applications without the need to configure different super-resolution models for different applications. This saves storage space and device power consumption, thereby improving device performance.

[0050] In one embodiment, the multiple applications are multiple game applications, and the target super-resolution model is used to perform super-resolution processing on the game screens of the multiple game applications. The first application is any one of the multiple game applications.

[0051] In a seventh aspect, a super-resolution model training apparatus is provided, which has the function of implementing the super-resolution model training method described in the sixth aspect above. The super-resolution model training apparatus includes at least one module, which is used to implement the super-resolution model training method provided in the sixth aspect above.

[0052] Eighthly, a super-resolution model training apparatus is provided, comprising a processor and a memory. The memory stores programs that support the super-resolution model training apparatus in executing the super-resolution model training method provided in the sixth aspect, and stores data related to implementing the super-resolution model training method described in the sixth aspect. The processor is configured to execute the programs stored in the memory. The super-resolution model training apparatus may further include a communication bus for establishing a connection between the processor and the memory.

[0053] In a ninth aspect, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the super-resolution model training method described in the sixth aspect above.

[0054] In a ninth aspect, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to execute the super-resolution model training method described in the sixth aspect above.

[0055] The technical effects achieved by the seventh, eighth, ninth and ninth aspects mentioned above are similar to the technical effects achieved by the corresponding technical means in the sixth aspect mentioned above, and will not be repeated here. Attached Figure Description

[0056] Figure 1 This is a schematic diagram illustrating an application scenario of a game super-resolution technology provided in an embodiment of this application;

[0057] Figure 2 This is a schematic diagram illustrating the training of a super-resolution model for a single game based on a data source from a single game, as provided in an embodiment of this application.

[0058] Figure 3 This is a comparative schematic diagram of the convergence curves of the loss function for model training based on different data sources, provided in an embodiment of this application.

[0059] Figure 4 This is a schematic diagram of a super-resolution model training method provided in an embodiment of this application;

[0060] Figure 5 This is a schematic diagram of a multi-scale residual fusion network provided in an embodiment of this application;

[0061] Figure 6 This is a schematic diagram of a multi-scale residual fusion network provided in an embodiment of this application;

[0062] Figure 7 This is a schematic diagram illustrating the feature distribution of multiple data sources in an original training set and a feature-balanced training set, as provided in an embodiment of this application.

[0063] Figure 8 This is a schematic diagram illustrating an image augmentation process using a feature enhancement algorithm, as provided in an embodiment of this application.

[0064] Figure 9 This is a schematic diagram illustrating a method for training a super-resolution model based on multiple data sources, as provided in this application.

[0065] Figure 10 This is a schematic diagram illustrating an adaptive adjustment of training data provided in an embodiment of this application;

[0066] Figure 11 This is a schematic diagram of another super-resolution model training method provided in the embodiments of this application;

[0067] Figure 12 This is a comparative schematic diagram of the convergence curves of the loss function for model training using the super-resolution model training method provided in the embodiments of this application based on different data sources;

[0068] Figure 13 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application;

[0069] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0070] Figure 15 This is a block diagram of a software system for an electronic device provided in an embodiment of this application;

[0071] Figure 16 This is a flowchart of a super-resolution method provided in an embodiment of this application. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0073] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limiting purposes, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details.

[0074] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0075] It should also be understood that in the embodiments of this application, "one or more" refers to one, two, or more; "and / or" describes the relationship between the associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0076] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," "fourth," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0077] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0078] To facilitate understanding, the application scenarios involved in the embodiments of this application will be explained first.

[0079] As the performance demands of mobile games continue to rise, especially with the trend towards thinner and lighter designs in smartphones and other mobile devices, high power consumption and overheating have become obstacles affecting the player experience. To address the issues of overheating and battery life caused by high power consumption and to break through the limitations of GPU performance, game artificial intelligence super resolution (Game AI-SR) technology has begun to attract attention, hereinafter referred to as game super-resolution technology.

[0080] The principle behind game super-resolution technology is to first render a lower-resolution image using the GPU, and then use an accelerated processing unit (APU) or specific algorithms to generate an image close to the original resolution, restoring image details. Its technical principle is equivalent to "subtraction before addition." Game super-resolution technology has two main applications: firstly, it reduces chip power consumption while maintaining near-native image quality. With this "subtraction before addition" super-resolution technology, players can enjoy the game without overheating or battery life issues even when playing in high-quality modes; secondly, it provides users with higher-resolution game visuals, enhancing the immersive gaming experience.

[0081] The specific algorithms used in game super-resolution technology are generally artificial intelligence (AI) super-resolution algorithms, such as super-resolution models. Super-resolution models (SR models) are an important research direction in computer vision and image processing. They aim to restore low-resolution (LR) images to high-resolution (HR) images, thereby improving the visual quality of the images. The core of a super-resolution model lies in learning the mapping relationship between low-resolution and high-resolution images. This typically involves complex image processing techniques and machine learning algorithms. By learning from a large number of low-resolution and high-resolution image pairs, the model can grasp the detailed information in the images and apply this information to new low-resolution images during the inference phase to generate high-resolution images. In other words, a super-resolution model can perform super-resolution processing on low-resolution images to obtain the corresponding high-resolution images.

[0082] Please refer to Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of a game super-resolution technology provided in an embodiment of this application. For example... Figure 1As shown, after the game application starts, the low-resolution game screen can be rendered by the GPU first, and then the low-resolution game screen can be super-resolved by the super-resolution model to restore the high-resolution game screen. The high-resolution game screen is then displayed to the user through the display screen.

[0083] Currently in the field of super-resolution, considering the issue of high-quality image effects, the data sources used for training super-resolution models for different game applications (hereinafter referred to as games) are all from their respective game data. The super-resolution models trained for different games are used to super-resolution different games. That is, for a single game, a super-resolution model specifically designed for super-resolution of that game is trained by using a single data source for that game (i.e., the training samples of that game, such as low-resolution and high-resolution image pairs of that game).

[0084] For example, please refer to Figure 2 , Figure 2 This is a schematic diagram illustrating the training of a super-resolution model for a single game based on a data source provided in this application. For example... Figure 2 As shown, a super-resolution model can be trained based on the data source of game A to generate a super-resolution model specifically for game A; similarly, a super-resolution model can be trained based on the data source of game B to generate a super-resolution model specifically for game B; and similarly, a super-resolution model can be trained based on the data source of game C to generate a super-resolution model specifically for game C. The game described in this embodiment can be any game, and this embodiment does not limit it; for example, the game could be Genshin Impact. R Peace Elite R Honor of Kings R wait.

[0085] However, current model training methods can only train super-resolution models for a single game, failing to meet the need for model generalization and thus having certain limitations. For example, when an electronic device has multiple game applications installed, it needs to be configured with multiple super-resolution models to perform super-resolution processing for different game applications. This will occupy a large amount of device storage space, increase device power consumption, and affect device performance.

[0086] Therefore, there is an urgent need to train a highly generalized super-resolution model capable of super-resolution processing for various applications, such as super-resolution of multiple game applications using a single model. However, when training such a generalized super-resolution model, simply training multiple data sources together may result in the model failing to converge during training, and the model's accuracy cannot be guaranteed.

[0087] Model convergence refers to the process where, as the number of training iterations increases, the loss function of the model gradually decreases and tends to a small and stable value. Conversely, model non-convergence during training means that, as the number of training iterations increases, the loss function of the model fails to gradually decrease to a small and stable value, or decreases at a slow rate.

[0088] Please refer to Figure 3 , Figure 3 This is a comparative schematic diagram of the convergence curves of the loss function for model training based on different data sources, provided in an embodiment of this application. Figure 3 The convergence curves of the loss function for training models using current model training methods based on a single data source, two data sources, and three data sources are shown respectively. Figure 3 As shown, the convergence curve of the loss function for model training based on a single data source ( Figure 3 As shown by the solid convergence curve in the graph, during model training based on a single data source, the loss value of the model's loss function can gradually decrease to near zero, indicating good convergence performance. The convergence curves of the loss function for model training based on two data sources are also shown. Figure 3 As shown in the dotted-line convergence curve, during model training based on two data sources, the loss value of the model's loss function can gradually decrease to near 0.6, but cannot decrease to near 0, indicating low convergence performance. The convergence curves of the loss function for model training based on three data sources (…) Figure 3 As shown by the dashed convergence curve in the figure, during model training based on three data sources, the loss value of the model's loss function can only decrease to approximately 1.2, and the loss value increases again after decreasing, fluctuating and unable to decrease smoothly, further reducing the convergence performance. Figure 3 A comparison of the three convergence curves shows that as the number of data sources increases, the difficulty of model training convergence increases, and the difficulty of training an accurate model also increases.

[0089] To address the issues of training convergence failure and model accuracy jeopardization caused by training with multiple data sources simultaneously in the process of training a general super-resolution model, this application proposes a super-resolution model training method. In this method, a multi-scale residual fusion network model with residual connections introduced across network layers in the model's network structure can be used as the super-resolution model to be trained. For example, residual connections can be introduced across network layers in the original super-resolution model's network structure, and the resulting multi-scale residual fusion network model can be used as the super-resolution model to be trained. Then, this super-resolution model is trained based on data sources from various applications to obtain a general super-resolution model capable of super-resolution for multiple applications.

[0090] In this context, the data source for multiple applications refers to the training samples for multiple applications. The training samples for each application include multiple image pairs corresponding to that application. Each image pair includes a pair of low-resolution and high-resolution images with identical content, such as a pair of low-resolution and high-resolution application screenshots for that application. For example, if the multiple applications are multiple game applications, the data source for each game application includes multiple image pairs corresponding to that game application. Each image pair includes a pair of low-resolution and high-resolution game screenshots for that game application.

[0091] By introducing residual connections across network layers in the model's network structure, i.e., introducing a multi-scale residual fusion network model as the super-resolution model to be trained, and using compatible multiple data sources for model training, the problem of zero-valued or gradient explosion in the directional gradient propagation during model training convergence can be alleviated, thereby improving training convergence performance and making the model training converge better. This results in a more accurate general super-resolution model that can perform super-resolution for multiple applications (such as multiple game applications).

[0092] On the one hand, as the number of network layers increases, the gradient undergoes multiple multiplication operations during backpropagation, causing the gradient value to gradually decrease. During model training, if the directional gradient is zero, it usually signifies gradient vanishing. Gradient vanishing refers to the phenomenon where, due to the chain rule's multiplicative effect, the gradient value gradually decreases as it propagates layer by layer during backpropagation, eventually approaching zero. This leads to slow or stagnant updates of weights in network layers closer to the input layer, affecting the model's training performance. By adding residual connections across network layers in the model's network structure, introducing a multi-scale residual fusion network, the output of the previous layer can be directly added to the input of the next layer. This helps the gradient propagate directly to earlier layers during backpropagation, thus alleviating the problem of the directional gradient becoming zero during model training convergence.

[0093] On the other hand, as the number of network layers increases, such as in deep networks, the gradients of the error may accumulate and multiply during updates. If the gradient values ​​between network layers are too large, repeated multiplication can lead to an exponential increase in the gradient. During model training, if the value of the directional gradient is too large, this usually means gradient explosion. Gradient explosion refers to the situation where, during backpropagation, the gradient value becomes extremely large for some reason, thus having a huge impact on parameter updates. Gradient explosion may cause the model to fail to converge or converge too slowly. By introducing residual connections across network layers in the model's network structure, these residual connections allow the gradient to bypass some layers and propagate directly to shallower layers during backpropagation. This cross-layer connection design provides a direct propagation path for the gradient, making it easier for the gradient to flow to the shallower layers of the network, thus improving propagation. This reduces the decay and accumulation of gradients during propagation, thereby reducing the possibility of gradient explosion and mitigating the gradient explosion problem caused by the multiplication effect in deep networks.

[0094] Next, the super-resolution model training method provided in the embodiments of this application will be described in detail with reference to the accompanying drawings.

[0095] Figure 4 This is a schematic diagram of a super-resolution model training method provided in an embodiment of this application. The method is applied to a computer device, which is a model training device capable of model training, also known as an electronic device. For example, the computer device can be a large model training all-in-one machine, such as a full-stack artificial intelligence device, etc. This embodiment of the application does not limit this. Figure 4 As shown, the method includes the following steps:

[0096] Step 401: The computer device acquires the super-resolution model to be trained. The super-resolution model to be trained is a multi-scale residual fusion network model. The multi-scale residual fusion network model includes multiple network layers, in which residual connections are introduced across network layers.

[0097] The multi-scale residual fusion network model comprises multiple network layers, with residual connections introduced across these layers. In this embodiment, a multi-scale residual fusion network model with residual connections introduced across network layers can be used as a super-resolution model to be trained.

[0098] By adding residual connections across network layers in the model's network structure, a multi-scale residual fusion network is introduced. On the one hand, the output of the previous layer can be directly added to the input of the next layer, which helps the gradient propagate directly to earlier layers during backpropagation, thus alleviating the problem of zero directional gradient propagation during model training convergence. On the other hand, the introduced residual connections allow the gradient to bypass some layers and propagate directly to shallower layers during backpropagation, enabling better gradient propagation within the network layers and reducing gradient decay and accumulation during propagation, thereby reducing the possibility of gradient explosion. Therefore, by using a multi-scale residual fusion network model with residual connections across network layers as the super-resolution model to be trained, and using compatible multiple data sources for model training, the problem of zero directional gradient propagation or gradient explosion during model training convergence can be alleviated, thereby improving training convergence performance and resulting in a more accurate general super-resolution model capable of super-resolution for various applications.

[0099] In one embodiment, a multi-scale residual fusion network model can be obtained by introducing residual connections across network layers in the network structure of the original super-resolution model. That is, the super-resolution model to be trained can be constructed by introducing residual connections across network layers in the already constructed network structure of the original super-resolution model. Alternatively, depending on the network structure requirements of the super-resolution model to be trained, multiple network layers can be constructed first, and then residual connections can be added across these multiple network layers to construct the super-resolution model to be trained. This application does not limit the method of constructing the super-resolution model to be trained.

[0100] In this context, the original super-resolution model refers to a super-resolution model currently designed for a single application, i.e., a super-resolution model used for super-resolution processing of a single application. The original super-resolution model can be a convolutional neural network model, etc. For example, in a game scenario, the original super-resolution model refers to a super-resolution model specific to a single game application.

[0101] In one embodiment, residual connections are introduced between two network layers every T layers, starting from a specific initial layer. These residual connections, introduced every T layers, are used to pass the output features of the preceding network layer to the next network layer, allowing the output features of the preceding layer to be fused with the original input features of the next network layer, which then becomes the new input features for that next network layer.

[0102] The specific starting layer can be any one of the multiple network layers, typically one of the earlier layers. The specific starting layer can be preset as needed. For example, it could be the first or second network layer, etc., but this embodiment does not limit this. T is a positive integer and can be set as needed. T is usually a small value to introduce residual connections across a smaller number of network layers. For example, T can be 1, 2, or 3, etc., but this embodiment does not limit this.

[0103] As an example, a residual network is introduced between two network layers, starting from a specific initial layer and alternating every other network layer in a multi-layer network. The specific initial layer can be the first or second network layer in the multi-layer network.

[0104] For example, the number of network layers is N, meaning the multi-scale residual fusion network model includes N network layers. A residual connection is introduced between the i-th layer and the (i+2)-th layer. This residual connection is used to pass the output features of the i-th layer to the (i+2)-th layer, allowing the output features of the i-th layer to be fused with the output features of the (i+1)-th layer and used as the input features of the (i+2)-th layer. In other words, the output features of the i-th layer can be fused with the original input features of the (i+1)-th layer, and the fused features become the new input features of the (i+2)-th layer. Here, i is an odd or even positive integer greater than or equal to 1 and less than or equal to N-2. That is, i is an odd or even number in the range [1, N-2].

[0105] For example, assuming N=7, then in the 7 network layers: residual connections are introduced between layers 1 and 3, between layers 3 and 5, and between layers 5 and 7. Thus, the output features of layer 1 can be passed across network layers to layer 3, allowing them to be fused with the output features of layer 2 to become the input features of layer 3. The output features of layer 3 can be passed across network layers to layer 5, allowing them to be fused with the output features of layer 4 to become the input features of layer 5. The output features of layer 5 can be passed across network layers to layer 7, allowing them to be fused with the output features of layer 6 to become the input features of layer 7.

[0106] In addition, residual connections can be introduced between the first layer and the last layer in multiple network layers, so that the output features of the first layer can be passed across network layers to the last layer. In this way, the output features of the first layer can be combined with the output features of the previous layer of the last layer to form the input features of the last layer. For example, the output features of the first layer can be fused with the output features of the previous layer of the last layer and used as the input features of the last layer.

[0107] Please refer to Figure 5 , Figure 5 This is a schematic diagram of a multi-scale residual fusion network provided in an embodiment of this application. Figure 5As shown, this multi-scale residual fusion network comprises seven layers. The output features of each of these seven layers can be sequentially passed to the next layer as input features. Furthermore, residual connections are introduced between every other layer, starting from layer 1. For example, a residual connection is introduced between layers 1 and 3, between layers 3 and 5, and between layers 5 and 7. Additionally, a residual connection is introduced between layers 1 and 7, i.e., between the first and last layers. Thus, the output features of layer 1 can be passed across layers to layer 3, layer 3 to layer 5, and layer 5 to layer 7. Moreover, the output features of the first layer can also be passed across layers to the last layer, meaning the output features of layer 1 can also be passed across layers to layer 7.

[0108] Please refer to Figure 6 , Figure 6 This is a schematic diagram of a multi-scale residual fusion network provided in an embodiment of this application. Figure 5 As shown, this multi-scale residual fusion network comprises eight network layers. The output features of each of these eight layers can be sequentially passed to the next network layer as input features. Furthermore, residual connections are introduced between every other network layer, starting from layer 1. For example, a residual connection is introduced between layers 1 and 3, between layers 3 and 5, and between layers 5 and 7. Additionally, a residual connection is introduced between layers 1 and 8, i.e., between the first and last layers. Thus, the output features of layer 1 can be passed across network layers to layer 3, layer 3 to layer 5, and layer 5 to layer 7. Moreover, the output features of the first layer can also be passed across network layers to the last layer, meaning the output features of layer 1 can also be passed across network layers to layer 8.

[0109] Step 402: The computer device acquires a training set, which includes data sources for various applications, and each application's data source includes multiple image pairs for the corresponding application.

[0110] Each image pair consists of a pair of low-resolution and high-resolution images with identical content, such as a pair of low-resolution and high-resolution application screens for the application. The high-resolution image has a higher resolution than the low-resolution image.

[0111] It should be noted that the low-resolution image and high-resolution image mentioned in the embodiments of this application are relative. Each image pair includes two images with the same content but different resolutions. The image with the lower resolution is called the low-resolution image, and the image with the higher resolution is called the high-resolution image.

[0112] As an example, the aforementioned multiple applications can be different applications within the same category of applications, or applications from different categories, as described in this embodiment. For instance, in a game scenario, to train a target super-resolution model capable of super-resolution for multiple game applications, a training set including data sources from multiple game applications can be obtained. That is, the aforementioned multiple applications can be different game applications within the game category, and the data source for each game application includes multiple image pairs corresponding to the game application. Each image pair can include a pair of low-resolution and high-resolution game frames from the corresponding game application. It should be understood that the aforementioned multiple applications can be different game applications from other categories of applications, such as different applications in video applications, different applications in instant messaging applications, or different applications in message aggregation applications, etc., and this embodiment does not limit this.

[0113] As an example, the various application data sources in the training set (referred to as multiple data sources) are in a balanced state in terms of feature dimensions, that is, the number of image features from the various data sources is in a balanced state. The number of image features from each data source refers to the total number of features contained in all images from each data source. The balanced number of image features from multiple data sources means that the number of image features from the various data sources is evenly distributed, and the difference between the number of image features from any two data sources is less than a preset threshold, that is, the number of image features from various data sources is not significantly different.

[0114] By acquiring multiple data sources with balanced feature dimensions as training sets, the samples of different categories can maintain a certain balance in the feature dimensions. That is, the distribution of samples of each category in the feature space is relatively uniform, and no one category has an absolute advantage in a certain feature. Such feature-balanced training samples can reduce prediction bias, improve generalization ability, and avoid overfitting and underfitting, thereby improving the accuracy of the trained model.

[0115] In this embodiment, when the features of samples from different categories in the training samples are balanced, the model can learn the features of each category more comprehensively, reducing prediction bias caused by data bias. Furthermore, a balanced data distribution helps the model form a more impartial and unbiased decision boundary during training, thereby improving prediction accuracy. Secondly, feature-balanced training samples help the model learn more generalized feature representations, enabling the model to make more accurate predictions when faced with new data (such as various unknown or unseen data), thus improving the model's generalization ability. In addition, feature imbalance often leads to overfitting to one category and underfitting to others. By adjusting the sample features in the training samples to achieve balance, this situation can be avoided to some extent.

[0116] As an example, obtaining the training set can be achieved by following these steps 1)-2):

[0117] 1) Obtain the initial training set, which includes initial data sources for various applications. Each initial data source for an application includes multiple initial image pairs for that application.

[0118] Each initial image pair consists of a pair of identical low-resolution and high-resolution images, such as a pair of low-resolution and high-resolution game screenshots for a given application. The initial data source for each application refers to the initial training samples for that application, which can be obtained through sample annotation.

[0119] Because applications differ, the number of features contained in the initial data sources may vary. For example, some simple games have relatively simple game screens with fewer features, resulting in a relatively small total number of features in the initial data source. Conversely, some complex games have more complex game screens with more features, leading to a larger total number of features in the initial data source. This uneven distribution of features across different initial data sources in the initial training set—a feature imbalance in the training samples—can cause prediction bias, low generalization ability, and overfitting or underfitting during model training, ultimately affecting the model's accuracy.

[0120] In this embodiment of the application, in order to improve the accuracy of the super-resolution model to be trained, the initial training set can be augmented with a feature enhancement algorithm before model training to obtain a training set that is balanced in the feature dimension.

[0121] 2) The initial training set is augmented with a feature enhancement algorithm to obtain a training set. The amount of data from the data source of each game in the training set is greater than the amount of data from the initial data source of the corresponding game in the initial training set, and the data sources of the various games in the training set are in a balanced state in terms of feature dimensions.

[0122] By augmenting the initial training set with feature enhancement algorithms, a feature-balanced training set can be obtained. Training the model based on the feature-balanced training set can improve phenomena such as prediction bias, low generalization ability, overfitting or underfitting that occur during the model training process, thereby improving the accuracy of the model.

[0123] Please refer to Figure 7 , Figure 7 This is a schematic diagram illustrating the feature distribution of multiple data sources in an original training set and a feature-balanced training set, as provided in an embodiment of this application. The feature-balanced training set is obtained by augmenting the original training set using a feature enhancement algorithm. For example... Figure 7 As shown, both the original training set and the training set expanded using the feature enhancement algorithm include three data sources: Data Source 1, Data Source 2, and Data Source 3. In the original training set, the number of image features in each of the three data sources varies significantly and is unevenly distributed. For example, Data Source 1 has approximately 1800 image features, Data Source 2 has approximately 2400, while Data Source 3 has only approximately 500, a considerable difference from the first two. Using this initial training set for model training may lead to prediction bias, overfitting, and underfitting during the training process. However, in the feature-balanced training set obtained after expanding the original training set using the feature enhancement algorithm, the number of image features in Data Source 1, Data Source 2, and Data Source 3 are all around 2500. The number of image features in these three data sources is evenly distributed, indicating a balance in feature dimensions. Therefore, training the model based on this training set can improve prediction bias, low generalization ability, and overfitting or underfitting issues that occur during model training, thereby improving the model's accuracy.

[0124] As an example, augmenting the initial training set with feature enhancement algorithms can be implemented in two possible ways:

[0125] The first possible implementation is to use a feature enhancement algorithm to expand the initial training set to obtain a new training set.

[0126] In other words, feature enhancement algorithms are directly used to expand the initial training set, so as to increase the quantity and diversity of training data and ensure that the training set obtained after expansion is balanced in terms of feature dimensions.

[0127] The second possible implementation is to first use an image transformation algorithm to augment the initial training set, obtaining the training set to be augmented. Then, a feature enhancement algorithm is used to augment the training set to be augmented, resulting in the final training set.

[0128] In other words, an image transformation algorithm is first used to augment the initial training set, increasing the amount and diversity of training samples. Then, a feature enhancement algorithm is used to augment the result of the first augmentation, further increasing the quantity and diversity of training samples while ensuring that the augmented training set maintains a balanced feature dimension. This approach increases the amount and diversity of training samples while ensuring feature balance, thereby further improving the model's accuracy.

[0129] As an example, the ratio of the amount of data augmented by the image transformation algorithm to the amount of data augmented by the feature enhancement algorithm is a preset ratio, which can be pre-set, such as 3:7. For example, the amount of data augmented by the image transformation algorithm accounts for 30% of the total augmented data, and the amount of data augmented by the feature enhancement algorithm accounts for 70% of the total augmented data.

[0130] Expanding the initial training set using image transformation algorithms can include: performing image transformations on all or part of the initial image pairs in the initial training set to obtain new initial image pairs, and constructing the training set to be expanded based on the new initial image pairs and the original initial image pairs in the initial training set.

[0131] Image transformation refers to the process of converting an image from one representation to another to change attributes such as size, shape, orientation, or color. Image transformation algorithms can include geometric transformations, pixel value transformations, affine transformations, Fourier transforms, etc., and this application does not limit the image transformation algorithms. Geometric transformations can change the coordinate positions of pixels in an image. For example, geometric transformations can include translation, rotation, scaling, and mirroring. Pixel value transformations mainly change attributes such as color or brightness of pixels in an image without changing the coordinate positions of the pixels. For example, pixel value transformations can include brightness adjustment, contrast adjustment, color adjustment, and sharpening.

[0132] Feature augmentation algorithms are used to modify images along the feature dimension. As an example, using a feature augmentation algorithm to augment a training set can involve the following steps: Augmenting each image in a first image pair along the feature dimension to obtain M augmented images corresponding one-to-one with the M features in each image. The first image pair can be any one of all or some image pairs in the initial training set or the training set to be augmented, where M is a positive integer. Based on the M augmented images corresponding to each image in the first image pair, M image pairs are generated, and the training set includes M image pairs. Each image in the first image pair contains M features, where M is a positive integer.

[0133] In other words, for each image pair in the initial training set or a portion of the training set to be expanded, the above method can be used to expand one image pair into multiple image pairs in the feature dimension. For example, if both images in an image pair include 3 features, then by performing a feature enhancement algorithm to expand this image pair once, 3 image pairs corresponding one-to-one with these 3 features can be obtained. Performing a feature enhancement algorithm to expand this image pair twice will yield 6 image pairs. The more times the expansion process is performed, the more images are obtained. Moreover, since image expansion is performed based on the features contained in the images in each image pair, feature balance among multiple data sources can be ensured by controlling the balance of data volume from various data sources.

[0134] In one embodiment, the operation of augmenting each image in the first image pair in terms of feature dimensions to obtain M augmented images corresponding one-to-one with the M features in each image may include the following steps 1)-3):

[0135] 1) Perform convolution processing on the first image to obtain M feature maps.

[0136] The first image is any image in the first image pair. The first image contains M features, where M is the number of features contained in the first image. There is a one-to-one correspondence between the M feature maps and the M features.

[0137] For example, assuming the first image includes 4 features, by performing convolution processing on the first image, 4 feature maps corresponding one-to-one with these 4 features can be obtained.

[0138] As an example, the first image can be processed with five convolutional layers to obtain M feature maps.

[0139] 2) Adjust each of the M feature maps to obtain M new feature maps.

[0140] As an example, each feature map can be randomly adjusted to obtain a new feature map corresponding to each feature map. For instance, the first feature map can be randomly perturbed to obtain a new first feature map, which can be any one of the M feature maps.

[0141] 3) Perform deconvolution on each of the M new feature maps to obtain the M new feature maps. Figure 1 One corresponding M augmented images.

[0142] In this way, the features of the original image can be changed in the model convolution process, thus preserving more features.

[0143] Please refer to Figure 8 , Figure 8 This is a schematic diagram illustrating an image augmentation process using a feature enhancement algorithm, as provided in an embodiment of this application. Figure 8 As shown, for the image to be processed, it can first be processed through 5 convolutional layers to obtain feature maps. If the image includes multiple features, multiple feature maps can be obtained. Then, random perturbation is added to each feature map to adjust it, resulting in a new feature map corresponding to each feature map. After that, deconvolution processing is performed on each new feature map to obtain a new image corresponding to each new feature map, which is the augmented image.

[0144] It should be noted that the feature-balanced training set can be obtained by the computer device or from other devices, and this application embodiment does not limit this. For example, the initial training set can be processed by other devices to obtain a feature-balanced training set, and then the feature-balanced training set can be sent to the computer device.

[0145] Step 403: The computer device trains the super-resolution model to be trained based on the training set to obtain the target super-resolution model, which is used to perform super-resolution processing on game screens for various applications.

[0146] In other words, computer equipment can train the introduced multi-size residual fusion network model based on multiple application data sources (hereinafter referred to as multiple data sources) to obtain a general super-resolution model that can perform super-resolution for multiple applications, namely the target super-resolution model.

[0147] Please refer to Figure 9 , Figure 9 This is a schematic diagram illustrating a method for training a super-resolution model based on multiple data sources, as provided in this application. For example... Figure 9As shown, multiple data sources can be input into the super-resolution model to be trained, resulting in a general super-resolution model (i.e., the target super-resolution model) for super-resolution of various applications. These multiple data sources can include data sources from game A, game B, and game C, etc.

[0148] In one embodiment, training the super-resolution model to be trained based on the training set may include the following steps: using the training set as input to the super-resolution model to be trained, outputting prediction results through the super-resolution model to be trained, calculating a loss function based on the prediction results, and adjusting the model parameters of the super-resolution model to be trained based on the loss function.

[0149] As an example, the super-resolution model can be trained multiple times using the training set. For each training cycle, the above training process can be followed. That is, for each training cycle, the following operations can be performed: use the training set as input to the super-resolution model to be trained, use the super-resolution model to output prediction results, calculate the loss function based on the prediction results, and adjust the model parameters of the super-resolution model to be trained based on the loss function.

[0150] In one embodiment, the loss function of the super-resolution model to be trained can be a cross-entropy loss function or a focal loss function, etc., and this application embodiment does not limit it.

[0151] Considering that after a model has been trained to a certain extent, especially when it has learned well on some samples (such as easy samples) but is still underlearned on others (such as difficult samples), the model parameters to be adjusted may no longer change, leading to a situation where model performance may no longer improve after a certain training period, i.e., the so-called "overfitting" or "stagnation" problem. In one embodiment, during model training, the training data can be adaptively adjusted. For example, during model training, the number of training iterations for difficult samples in the training set can be adaptively increased, or the number of training iterations for difficult samples in the training set can be increased while the number of training iterations for easy samples is decreased. In this way, the model's learning degree on difficult samples can be deepened during model training, avoiding the problem of underlearning on difficult samples.

[0152] In this context, "hard samples" can be training samples whose loss function during training is greater than or equal to a first preset threshold, and "easy samples" can be training samples whose loss function during training is less than a second preset threshold. That is, hard samples are training samples with relatively large loss functions, and easy samples are training samples with relatively small loss functions. Both the first and second preset thresholds can be set according to actual needs. For example, the first and second preset thresholds can be the same or different, such as both being 0.5. This embodiment does not limit this.

[0153] As an example, during the training of the super-resolution model based on the training set, the training weights of the first part of the training data can be increased to increase the number of training iterations for the first part of the training data. Here, the first part of the training data refers to the training data in the training set whose loss function is greater than or equal to a first preset threshold during the training process, and the training weights are used to indicate the number of training iterations for the corresponding training data.

[0154] By increasing the training weights of the first part of the training data, the number of training iterations for difficult samples can be increased, thereby deepening the model's learning of difficult samples during the model training process and avoiding the problem of insufficient learning of difficult samples.

[0155] As another example, during the training of the super-resolution model based on the training set, in addition to increasing the training weight of the first part of the training data, the training weight of the second part of the training data can be decreased to reduce the number of training iterations for the second part of the training data. Here, the second part of the training data refers to the training data in the training set whose loss function is less than a second preset threshold during the training process.

[0156] By reducing the training weights of the second part of the training data, the number of training iterations for simple samples can be reduced, thus avoiding the problem of continuing to learn from simple samples even after they have been learned well.

[0157] As an example, during training, for each training epoch, after calculating the loss function for each training data point in that epoch, the training data with the larger loss function and the training data with the smaller loss function can be determined from the training set of that epoch. The training weights of the first and second training data points are then updated. Based on these weights, the training set for the next training epoch is updated; for example, the amount of data in the first training data point and the amount of data in the second training data point are increased in the next training epoch. This allows for increasing the number of training iterations for the first training data point and decreasing the number of training iterations for the second training data point.

[0158] Please refer to Figure 10 , Figure 10 This is a schematic diagram illustrating an adaptive adjustment of training data provided in an embodiment of this application. For example... Figure 10 As shown, assuming the model requires N training epochs, after each training epoch, based on the loss function of each data point in the training set used in that epoch, the first part of the training data (hard samples) and the second part of the training data (easy samples) can be determined from the training set. The training weights of the first and second parts of the training data are then updated, for example, by increasing the training weight of the first part and decreasing the training weight of the second part. Then, the training set is updated according to the training weights. For example, the amount of data in the first part of the training data with larger training weights can be increased, such as by copying the data to obtain more data; and / or the amount of data in the second part of the training data with smaller training weights can be decreased, such as by deleting all or part of the data in the second part of the training data with smaller training weights. Afterwards, the super-resolution model is trained for the next training epoch based on the updated training set. This round-trip training process achieves adaptive adjustment of the training data in the training set.

[0159] In one embodiment, the loss function of the super-resolution model to be trained includes a regularization term, wherein the value of the regularization parameter of the regularization term is greater than a preset threshold. The regularization parameter controls the weight of the regularization term in the loss function.

[0160] A regularization parameter value greater than the preset threshold indicates that the regularization parameter is set relatively large. By setting a larger regularization parameter, the network can be prevented from getting stuck in local optima, the learning speed of the model can be accelerated, and the training efficiency of the model can be improved.

[0161] Next, the super-resolution model training method provided in this application will be described in detail with specific examples.

[0162] Please refer to Figure 11 , Figure 11 This is a schematic diagram of another super-resolution model training method provided in an embodiment of this application, which is applied in a computer device. Figure 11 As shown, the method includes the following steps:

[0163] Step 1101: Construct a multi-scale residual fusion network model based on the original super-resolution model. Use the multi-scale residual fusion network model as the super-resolution model to be trained. The scale residual fusion network model introduces residual connections across network layers in the network structure of the original super-resolution model.

[0164] Step 1102: Obtain the original training set. The initial training set includes multiple initial data sources, and each initial data source includes multiple initial image pairs.

[0165] In this context, multiple data sources can refer to data sources from multiple applications, such as the initial data sources for multiple game applications. Each application's initial data source includes multiple initial image pairs corresponding to that application.

[0166] As an example, the original training set can be obtained through image annotation.

[0167] Step 1103: Use an image transformation algorithm to augment the initial training set to obtain an intermediate training set.

[0168] Step 1104: Use a feature enhancement algorithm to expand the intermediate training set to obtain a training set. This training set includes multiple data sources, and the multiple data sources are in a balanced state in terms of feature dimensions.

[0169] In this context, "multiple data sources" refers to data sources from multiple applications, such as data sources from multiple game applications. Each application's data source includes multiple image pairs corresponding to that application.

[0170] Step 1105: Train the super-resolution model to be trained based on the training set.

[0171] As an example, the super-resolution model to be trained can be trained based on the training set and the preset training period. The preset training period can be set as needed, for example, the preset training period can be N, where N is a positive integer.

[0172] During the training process, for each training cycle, the training set of the current training cycle can be used as the input of the super-resolution model to be trained. The super-resolution model to be trained outputs the prediction result, the loss function is calculated based on the prediction result, and the model parameters of the super-resolution model to be trained are adjusted based on the loss function.

[0173] Step 1106: During the training process, increase the number of training iterations for the first part of the training data in the training set and decrease the number of training iterations for the second part of the training data.

[0174] The first part of the training data refers to the training data in which the loss function during the training process is greater than or equal to a first preset threshold, and the second part of the training data refers to the training data in which the loss function during the training process is less than a second preset threshold.

[0175] As an example, during the training process, for each training cycle, the first part of the training data with the larger loss function can be determined from the training set based on the loss function of the training data in that training cycle. Before the next training cycle arrives, the first part of the training data is used to train the super-resolution model to be trained, and then the training set is used to train the model in the next training cycle.

[0176] In this way, training on the first part of the training data can be added between two training cycles, thereby increasing the number of training times for the first part of the training data and reducing the number of training times for the second part of the training data throughout the entire model training process.

[0177] As another example, during training, for each training epoch, based on the loss function of the training data in that epoch, a first part of the training data with a larger loss function and a second part of the training data with a smaller loss function can be determined from the training set of that epoch. The training weights of the first and second parts of the training data are then updated. Based on these weights, the training set for the next training epoch is updated; for example, the amount of data in the first part of the training set is increased, and the amount of data in the second part is decreased. Then, the model is trained for the next training epoch based on the updated training set. This increases the number of training iterations for the first part of the training data and decreases the number of training iterations for the second part.

[0178] In one embodiment, the loss function of the super-resolution model to be trained includes a regularization term, wherein the value of the regularization parameter of the regularization term is greater than a preset threshold. The regularization parameter controls the weight of the regularization term in the loss function.

[0179] A regularization parameter value greater than the preset threshold indicates that the regularization parameter is set relatively large. By setting a larger regularization parameter, the network can be prevented from getting stuck in local optima, the learning speed of the model can be accelerated, and the training efficiency of the model can be improved.

[0180] Step 1107: End training when the preset termination condition is met.

[0181] The preset termination conditions can be set in advance as needed. For example, the preset termination conditions may include one or more of the following conditions: the training of the super-resolution model to be trained has reached a preset number of iterations (i.e., the number of times the entire training set has been traversed), the loss value of the loss function of the super-resolution model to be trained has reached a preset threshold, the super-resolution model to be trained has met the convergence condition, and specific business indicators have been met, etc.

[0182] Step 1108: Use the trained super-resolution model to be trained as the target super-resolution model.

[0183] In this embodiment, by introducing residual connections across network layers in the network structure of the original super-resolution model, a multi-scale residual fusion network model is introduced as the super-resolution model to be trained. The original training set is expanded using a feature enhancement algorithm to obtain a feature-balanced dataset including multiple data sources. Then, the model is trained using multiple feature-balanced data sources, and the training data is adaptively adjusted during the model training process. This results in better convergence performance, faster convergence speed, and more accurate model after training.

[0184] Please refer to Figure 12 , Figure 12 This is a comparative schematic diagram of the convergence curves of the loss function for model training using the super-resolution model training method provided in the embodiments of this application based on different data sources. Figure 12 The convergence curves of the loss function for model training based on a single data source are shown respectively. Figure 12 The solid-line convergence curve in the image), and the convergence curve of the loss function for model training based on two data sources (…). Figure 12 The dotted-line convergence curve in the image), and the convergence curve of the loss function for model training based on three data sources (…). Figure 12 The convergence curves (dashed lines in the diagram) show that, as can be seen from the convergence curves of the loss functions corresponding to these three data sources, when using the super-resolution model training method provided in this application for model training, regardless of whether the model is trained using a single data source, two data sources, or three data sources, the loss value of the model's loss function can gradually decrease to near zero during the training process. In other words, when using the super-resolution model training method provided in this application for model training, increasing the number of data source types does not significantly affect the convergence effect of model training. With an increase in the number of data source types, the convergence effect and convergence speed after model training can still be guaranteed, thereby ensuring the accuracy of the model.

[0185] Next, examples of the computer devices provided in the embodiments of this application will be given.

[0186] Figure 13 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. This computer device is the aforementioned device for model training. See also... Figure 13 The computer device includes at least one processor 201, a communication bus 202, a memory 203, and at least one communication interface 204.

[0187] The processor 201 may be a microprocessor (including a central processing unit (CPU) or the like), an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application.

[0188] The communication bus 202 may include a path for transmitting information between the aforementioned components.

[0189] The memory 203 may be a read-only memory (ROM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), optical disc (including compact disc read-only memory (CD-ROM), compressed optical disc, laser disc, digital versatile optical disc, Blu-ray disc, etc.), magnetic disk storage medium, or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but not limited thereto. The memory 203 may exist independently and be connected to the processor 201 via the communication bus 202. The memory 203 may also be integrated with the processor 201.

[0190] Communication interface 204 uses any transceiver-like device to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0191] In a specific implementation, as one example, the processor 201 may include one or more CPUs, such as... Figure 2 CPU0 and CPU1 are shown in the diagram.

[0192] In a specific implementation, as one example, a computer device may include multiple processors, such as... Figure 2 The processors 201 and 205 are shown. Each of these processors may be a single-core processor or a multi-core processor. Here, "processor" may refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0193] In a specific implementation, as one embodiment, the computer device may further include an output device 206 and an input device 207. The output device 206 communicates with the processor 201 and can display information in various ways. For example, the output device 206 may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device 207 communicates with the processor 201 and can receive user input in various ways. For example, the input device 207 may be a mouse, keyboard, touchscreen device, or sensing device, etc.

[0194] The aforementioned computer device can be a general-purpose computer device or a special-purpose computer device. In specific implementations, the computer device can be a desktop computer, a portable computer, a network server, a handheld computer, a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. The embodiments of this application do not limit the type of computer device.

[0195] The memory 203 stores program code 210 for executing the scheme of this application, and the processor 201 executes the program code 210 stored in the memory 203. This computer device can implement the above-mentioned scheme through the processor 201 and the program code 210 in the memory 203. Figure 4 and Figure 11 The super-resolution model training method provided in the embodiment.

[0196] After training a general target super-resolution model capable of super-resolution for various applications using the above model training method, the target super-resolution model can be configured into electronic devices so that the electronic devices can use the target super-resolution model to super-resolution any application among various applications.

[0197] As an example, the electronic device includes a first application, a GPU, and a target super-resolution model. The first application can be any one of multiple applications. After the first application is launched, the electronic device first renders the first application screen using the GPU, and then performs super-resolution processing on the first application screen using the target super-resolution model to obtain a second application screen corresponding to the first application screen. The resolution of the second application screen is greater than that of the first application screen.

[0198] The target super-resolution model can be applied to game scenarios, as well as other scenarios such as video (e.g., short videos), instant messaging, and message aggregation. This application does not limit its application to these scenarios. For example, multiple applications can be multiple game applications within a game application category; thus, a single target super-resolution model can be used to super-resolution the game footage of multiple game applications. Similarly, multiple applications can be multiple video applications within a video application category; thus, a single target super-resolution model can be used to super-resolution the video footage of multiple video applications.

[0199] In this application, the electronic device configured with the target super-resolution model can also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The electronic device can be a mobile phone, smart TV, wearable device, tablet computer, computer with wireless transceiver capabilities, virtual reality (VR) electronic device, augmented reality (AR) electronic device, wireless terminal in industrial control, wireless terminal in self-driving, wireless terminal in remote medical surgery, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, etc. The embodiments of this application do not limit the specific technology or form of the electronic device.

[0200] Before providing a detailed explanation of the super-resolution method provided in the embodiments of this application, the electronic equipment involved in the embodiments of this application will be described first.

[0201] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. See also... Figure 14The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0202] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0203] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0204] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0205] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0206] In some embodiments, the processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0207] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0208] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Mobile communication module 150 can provide solutions for wireless communication applications in electronic device 100, including 2G / 3G / 4G / 5G. Wireless communication module 160 can provide solutions for wireless communication applications in electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), and other wireless communication technologies.

[0209] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0210] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is an integer greater than 1.

[0211] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0212] The ISP is used to process data fed back from the camera 193. The camera 193 is used to capture still images or videos. An object passes through the lens, generating an optical image that is projected onto a photosensitive element. The photosensitive element converts the light signal into an electrical signal, which is then passed to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into image signals in standard formats such as RGB and YUV. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is an integer greater than 1. The digital signal processor is used to process digital signals, and in addition to processing digital image signals, it can also process other digital signals.

[0213] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0214] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0215] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions, such as saving music, video, and other files on the external memory card.

[0216] Internal memory 121 can be used to store computer-executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created by electronic device 100 during use (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0217] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D and application processor.

[0218] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the touch operation intensity based on pressure sensor 180A. Electronic device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example, when a touch operation with an intensity less than the pressure threshold is applied to the SMS application icon, a command to view an SMS message is executed. When a touch operation with an intensity greater than or equal to the pressure threshold is applied to the SMS application icon, a command to create a new SMS message is executed.

[0219] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch display." Touch sensor 180K detects touch operations applied to or near it. Touch sensor 180K can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0220] The software system of electronic device 100 will be described next.

[0221] The electronic device provided in this application embodiment can be user equipment (UE), such as a mobile terminal (e.g., a user's mobile phone), tablet computer, desktop computer, laptop computer, handheld computer, netbook, personal digital assistant (PDA), etc.

[0222] In addition, an operating system runs on top of these components. Examples include Apple's iOS, Google's Android, and Microsoft's Windows. Applications can be installed and run on this operating system.

[0223] The operating system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.

[0224] Figure 15 This is a block diagram of a software system for an electronic device 100 provided in an embodiment of this application. See also... Figure 15 A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the native layer, the user-space driver layer, and the driver layer.

[0225] The application layer can include a series of application packages. For example... Figure 15As shown, the application package may include multiple applications, such as game applications A, B, and C. In addition to the aforementioned game applications, the application package may also include applications for camera, gallery, calendar, calling, map, navigation, WLAN, Bluetooth, music, video, and SMS, etc. This application embodiment does not limit the scope of such applications.

[0226] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example... Figure 15 As shown, the application framework layer can include a window manager, content providers, a view system, a phone manager, a resource manager, and a notification manager. The window manager manages window programs. It can obtain the screen size, determine if a status bar is present, lock the screen, and capture the screen. The content provider stores and retrieves data, making this data accessible to the application. This data can include videos, images, audio, made and received phone calls, browsing history and bookmarks, and phone books. The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build the application's display interface, which can consist of one or more views, such as a view displaying SMS notification icons, a view displaying text, and a view displaying images. The phone manager provides communication functions for the electronic device 100, such as managing call status (including connection and disconnection). The resource manager provides the application with various resources, such as localized strings, icons, images, layout files, and video files. The notification manager allows the application to display notification information in the status bar, which can be used to convey informational messages and can disappear automatically after a short pause without user interaction. For example, the notification manager is used to notify users of download completions and message alerts. The notification manager can also display notifications as icons or scrolling text in the system's top status bar, such as notifications from background applications. Furthermore, the notification manager can appear as dialog boxes on the screen, such as displaying text messages in the status bar, emitting sounds, causing electronic devices to vibrate, or flashing indicator lights.

[0227] The native layer typically refers to the system runtime layer, located between the application framework layer and the driver layer, serving as a bridge connecting the application layer and the hardware layer. For example... Figure 15As shown, the local layer includes a game engine, which supports the operation of game applications. For example, the game engine includes a target super-resolution model, which is used to perform super-resolution on various game applications. It should be understood that the game engine may also include other processing modules, such as an image enhancement module, etc., but this embodiment does not limit this. The image enhancement module is used to enhance the image quality of the image to be processed.

[0228] The user-mode driver layer is located between the local layer and the driver layer. For example... Figure 15 As shown, the driver layer may include a hardware composer (HWC) for layer compositing. Additionally, the user-mode driver layer may also include a graphics system layer (GSL), etc., but this embodiment does not limit the scope of the application.

[0229] The driver layer resides at the lowest level of the software system and interacts directly with the hardware. It provides interfaces to various hardware devices and low-level libraries, enabling upper-layer applications to easily access and control these devices. For example... Figure 15 As shown, the driver layer can include display drivers, GPU drivers, etc. Display drivers are used to drive the display in the hardware layer. GPU drivers are used to drive the GPU.

[0230] It should be noted that, Figure 15 The layers in the illustrated software structure and the components contained in each layer do not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more layers than illustrated. Furthermore, each layer may include more or fewer components than illustrated, which is not limited herein.

[0231] It is understood that, in order to implement the super-resolution method in the embodiments of this application, the electronic device includes hardware and / or software modules that perform various functions. Based on the algorithmic steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments.

[0232] It should be noted that although the embodiments of this application are described using the Android system as an example, the basic principles are also applicable to electronic devices based on operating systems such as iOS or Windows.

[0233] The following example, using a game scenario, illustrates the workflow of the software and hardware of electronic device 100.

[0234] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the driver layer. The driver layer processes the touch operation into a raw input event (including touch coordinates, touch operation timestamp, etc.). The raw input event is stored in the driver layer. The application framework layer retrieves the raw input event from the driver layer and identifies the control corresponding to the raw input event. Taking a single click as an example, where the corresponding control is the game application icon, the game application calls the driver layer to start the GPU driver, rendering the low-resolution game screen using the GPU. Then, it calls the target super-resolution model in the local layer's game engine to super-resolution the low-resolution game screen into a high-resolution screen. Finally, it calls the HWC in the user-mode driver layer to composite and display the high-resolution screen on the monitor.

[0235] The super-resolution method provided in this application can be implemented by the aforementioned electronic device, or by a functional module and / or entity within that electronic device capable of implementing the video processing method. Furthermore, the solution in this application can be implemented through hardware and / or software, the specific implementation depending on actual usage requirements, and this application does not impose any limitations. The following description uses an electronic device as an example, along with the accompanying drawings, to exemplarily illustrate the video processing method provided in this application.

[0236] Next, taking a game scenario as an example, the super-resolution method provided in the embodiments of this application will be illustrated with reference to the accompanying drawings.

[0237] Please refer to Figure 16 , Figure 16 This is a flowchart of a super-resolution method provided in an embodiment of this application. The method is applied in an electronic device, which includes a first application, a GPU, and a target super-resolution model. The target super-resolution model is a multi-scale residual fusion network model used for super-resolution of multiple applications. The first application can be any one of the multiple applications. This embodiment uses the target super-resolution model to perform super-resolution processing on game screens of multiple game applications, with game A (i.e., game application A) as the first application for illustration. Figure 1 As shown, the method includes the following steps:

[0238] Step 1601: The user clicks the application icon of game A.

[0239] This embodiment of the application is only an example of a user opening game A by clicking the application icon of game A. In other embodiments, the user may also open game A in other ways, such as by voice operation, gesture operation or button operation. This embodiment of the application does not limit this.

[0240] Step 1602: In response to the user's action, game A starts.

[0241] Step 1603: Game A starts the rendering thread.

[0242] Game A can launch a rendering thread to render the game's visuals. For example, the rendering thread can call upon the target super-resolution model in the GPU and game engine to complete the rendering process.

[0243] Step 1604: The rendering thread calls the GPU.

[0244] For example, the rendering thread can send a first call instruction to the GPU, which instructs the GPU to render a low-resolution game screen of game A, i.e., game screen 1.

[0245] Step 1605: GPU renders game screen 1 of game A.

[0246] Game screen 1 can be a low-resolution game screen.

[0247] Step 1606: The GPU sends game screen 1 to the rendering thread.

[0248] Step 1607: The rendering thread calls the target super-resolution model in the game engine based on game screen 1.

[0249] For example, the rendering thread can send a second invocation instruction to the target super-resolution model, which instructs the target super-resolution model to perform super-resolution processing on game screen 1. The second invocation instruction may carry game screen 1 or instruction information for game screen 1. This instruction information is used to instruct game screen 1, such as an identifier or storage path information for game screen 1, etc., and this embodiment does not limit this.

[0250] Step 1608: The target super-resolution model performs super-resolution processing on game screen 1 to obtain game screen 2 corresponding to game screen 1. The resolution of game screen 2 is greater than that of game screen 1.

[0251] In other words, game screen 2 is a game screen with the same content as game screen 1 but at a higher resolution. A higher resolution game screen can be obtained through a target super-resolution model.

[0252] Step 1609: The target super-resolution model sends game screen 2 to the rendering thread.

[0253] Step 1610: The rendering thread sends game screen 2 to HWC.

[0254] Step 1611: HWC composites the game screen 1.

[0255] HWC can composite one or more images rendered by the rendering thread. For example, HWC can composite game screen 1 with other images sent by the rendering thread.

[0256] Step 1612: HWC displays the synthesized game graphics.

[0257] Step 1613: The screen displays the synthesized game screen.

[0258] This application also provides a chip coupled to a memory, which is used to read and execute computer programs or instructions stored in the memory to perform the methods in the above embodiments.

[0259] This application also provides an electronic device including a chip for reading and executing computer programs or instructions stored in a memory, causing the methods in the various embodiments to be performed.

[0260] This embodiment also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the video processing method described above.

[0261] This embodiment also provides a computer program product. The computer-readable storage medium stores program code. When the computer program product is run on a computer, it causes the computer to perform the above-mentioned related steps to implement the video processing method in the above embodiment.

[0262] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component, or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the video processing methods in the above-described method embodiments.

[0263] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0264] This application does not specifically limit the structure of the execution subject of the method provided in this application embodiment. As long as a program containing the code of the method provided in this application embodiment can be run to perform video processing according to the method provided in this application embodiment, it is acceptable. For example, the execution subject of the method provided in this application embodiment can be an electronic device, or a functional module in an electronic device that can call and execute a program.

[0265] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0266] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0267] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0268] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium and includes several instructions that cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0269] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A super-resolution model training method, characterized in that, The method includes: Obtain a super-resolution model to be trained, wherein the super-resolution model to be trained is a multi-scale residual fusion network model, and the multi-scale residual fusion network model includes multiple network layers in which residual connections are introduced across network layers; Obtain a training set, which includes data sources from various applications. Each data source for each application includes multiple image pairs corresponding to the application. Each image pair includes a pair of low-resolution images and a high-resolution image with the same content, wherein the resolution of the high-resolution image is greater than that of the low-resolution image. Based on the training set, the super-resolution model to be trained is trained to obtain the target super-resolution model, which is used to perform super-resolution processing on the application screens of the various applications.

2. The method as described in claim 1, characterized in that, The multi-scale residual fusion network model includes N network layers. A residual connection is introduced between the i-th layer and the (i+2)-th layer. The residual connection between the i-th layer and the (i+2)-th layer is used to pass the output features of the i-th layer to the (i+2)-th layer, so that the output features of the i-th layer can be fused with the output features of the (i+1)-th layer and used as the input features of the (i+2)-th layer. Here, i is an odd or even positive integer greater than or equal to 1 and less than or equal to N-2.

3. The method as described in claim 1 or 2, characterized in that, A residual connection is introduced between the first and last layers of the multiple network layers. The residual connection between the first and last layers is used to pass the output features of the first layer to the last layer, so that the output features of the first layer can be fused with the output features of the previous layer of the last layer and used as the input features of the last layer.

4. The method as described in claim 1, characterized in that, The acquisition of the training set includes: Obtain an initial training set, which includes initial data sources for the various applications, and the initial data source for each application includes several initial image pairs for each application; The initial training set is augmented using a feature enhancement algorithm to obtain the training set. The amount of data from each application's data source in the training set is greater than the amount of data from the initial data source of the corresponding application in the initial training set, and the data sources of the various applications in the training set are balanced in terms of feature dimensions.

5. The method as described in claim 4, characterized in that, The process of augmenting the initial training set using a feature enhancement algorithm to obtain the training set includes: Based on the initial training set, an expansion training set is obtained, wherein the expansion training set is the initial training set or is obtained by expanding the initial training set using an image transformation algorithm; The feature enhancement algorithm is used to expand the training set to be expanded, thereby obtaining the training set.

6. The method as described in claim 5, characterized in that, The feature enhancement algorithm is used to expand the training set to obtain the training set, which includes: Each image in the first image pair is augmented in the feature dimension to obtain M augmented images that correspond one-to-one with the M features in each image. The first image pair is any one of all or part of the image pairs in the training set to be augmented, and M is a positive integer. Based on the M augmented images corresponding to each image in the first image pair, M image pairs are generated, and the training set includes the M image pairs.

7. The method as described in claim 6, characterized in that, The process of augmenting each image in the first image pair along the feature dimension yields M augmented images corresponding one-to-one with the M features in each image, including: The first image is convolved to obtain M feature maps that correspond one-to-one with the M features in the first image, where the first image is any one of the first image pairs. Each of the M feature maps is adjusted to obtain M new feature maps; The M new feature maps are deconvolved to obtain M augmented images that correspond one-to-one with the M new feature maps.

8. The method as described in claim 7, characterized in that, The adjustment of each feature map in the M feature maps includes: The first feature map is randomly perturbed to obtain a new first feature map, which is any one of the M feature maps.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: During the training of the super-resolution model to be trained based on the training set, the training weight of the first part of the training data is increased. The first part of the training data refers to the training data in the training set whose loss function is greater than or equal to a first preset threshold during the training process. The training weight is used to indicate the number of training times for the corresponding training data.

10. The method as described in claim 9, characterized in that, The method further includes: During the training of the super-resolution model to be trained based on the training set, the training weight of the second part of the training data is reduced. The second part of the training data refers to the training data in the training set whose loss function is less than a second preset threshold during the training process.

11. The method according to any one of claims 1-10, characterized in that, The loss function of the super-resolution model to be trained includes a regularization term. The value of the regularization parameter of the regularization term is greater than a preset threshold. The regularization parameter is used to control the weight of the regularization term in the loss function.

12. The method according to any one of claims 1-11, characterized in that, The multi-scale residual fusion network model is obtained by introducing residual connections across network layers in the network structure of the original super-resolution model, which is a convolutional neural network model.

13. The method according to any one of claims 1-12, characterized in that, The various applications refer to various game applications, and the target super-resolution model is used to perform super-resolution processing on the game screens of the various game applications.

14. A super-resolution method, characterized in that, The method is applied in an electronic device, which includes a first application, a graphics processing unit (GPU), and a target super-resolution model. The target super-resolution model is used to perform super-resolution processing on the application screens of multiple applications. The first application is any one of the multiple applications. The target super-resolution model is a multi-scale residual fusion network model, in which residual connections are introduced across multiple network layers. The method includes: After the first application is launched, the first application screen is rendered by the GPU; The first application screen is super-resolution processed by the target super-resolution model to obtain a second application screen corresponding to the first application screen, wherein the resolution of the second application screen is greater than that of the first application screen. The second application screen will be displayed.

15. The method as described in claim 14, characterized in that, The multiple applications refer to multiple game applications, and the target super-resolution model is used to perform super-resolution processing on the game screens of the multiple game applications. The first application is any one of the multiple game applications.

16. An electronic device, characterized in that, The electronic device includes: one or more processors, and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as claimed in any one of claims 1 to 13 or 14-15.

17. A computer program product containing instructions, characterized in that, When it is run on a computer, it causes the computer to perform the method as claimed in any one of claims 1 to 13 or claims 14-15.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on a computer, cause the computer to perform the method as claimed in any one of claims 1 to 13 or 14-15.

Images