Image enhancement model training method, image enhancement method and related device

Abstract

The present application discloses an image enhancement model training method, an image enhancement method and related devices, wherein the image enhancement model training method includes: obtaining a clear sample image and a degraded image corresponding to the clear sample image; extracting features from the degraded image to obtain a first image feature of the degraded image; mining the first image feature through multiple cascaded residual networks to obtain a second image feature; enhancing the degraded image based on the second image feature to obtain a predicted image; and training the image enhancement model using the difference between the predicted image and the clear sample image. The above scheme can improve the image enhancement effect of the image enhancement model.

Description

Technical Field

[0001] The present application relates to the technical field of model training, and in particular to an image enhancement model training method, an image enhancement method, and related devices. Background Art

[0002] With the rapid development of Internet technology, the live broadcast industry has appeared more and more frequently in people's vision, providing people with a new form of entertainment. At present, in the live broadcast industry, the requirements for high frame rate and high resolution live broadcast images are getting higher and higher to improve the user's perception and experience. At present, the frame rate of mainstream live broadcast videos is generally 60fps, and the resolution is 1080x1920.

[0003] However, in various specific live broadcast scenarios, the host's behavior and the equipment itself often have certain uncontrollable phenomena or hidden dangers. For example, the network environment is poor and high-definition video cannot be transmitted in real time, or the host's equipment is poor, and noise occurs when rendering the live broadcast picture, which can easily lead to uneven quality of the live video.

[0004] In the industry, the BM3D denoising algorithm based on block matching is generally used to process live broadcast images, but this algorithm is only effective in one dimension of denoising and cannot enhance details. The GRDB algorithm based on deep convolutional neural networks and attention mechanisms has obvious denoising effects, but it takes a long time to run, cannot achieve 60 frames of real-time effects, and cannot enhance detailed textures. Summary of the invention

[0005] The present application provides an image enhancement model training method, an image enhancement method and related devices.

[0006] The present application provides an image enhancement model training method, comprising: obtaining a clear sample image and a degraded image corresponding to the clear sample image; performing feature extraction on the degraded image to obtain a first image feature of the degraded image; performing feature mining on the first image feature through multiple cascaded residual networks to obtain a second image feature; performing image enhancement on the degraded image based on the second image feature to obtain a predicted image; and training an image enhancement model using the difference between the predicted image and the clear sample image.

[0007] Among them, the step of performing feature mining on the first image feature through multiple cascaded residual networks to obtain the second image feature includes: performing feature encoding on the first image feature through multiple cascaded first residual networks to obtain encoded image features; performing feature decoding on the encoded image features through multiple cascaded second residual networks to obtain decoded image features matching the size of the degraded image, and determining the decoded image features as the second image features.

[0008] Among them, the step of encoding the first image feature through multiple cascaded first residual networks to obtain the encoded image feature of the first image feature includes: downsampling the first image feature using a convolutional layer with a step size of 2 through at least four sequentially cascaded first residual networks to obtain the encoded image feature; wherein each first residual network also includes 4 densely connected convolutional layers with a kernel size of 3x3 and one convolutional layer with a kernel size of 1x1.

[0009] Among them, the steps of decoding the encoded image features through multiple cascaded second residual networks to obtain decoded image features that match the size of the degraded image, and determining the decoded image features as the second image features include: upsampling the first image features using pixel reorganization through at least four sequentially cascaded second residual networks to obtain decoded image features that match the size of the degraded image; wherein each second residual network also includes 4 densely connected deconvolution layers with a kernel size of 3x3 and one deconvolution layer with a kernel size of 1x1.

[0010] Among them, the step of performing image enhancement on the degraded image based on the second image feature to obtain a predicted image includes: processing the second image feature through a convolutional layer with a kernel size of 3x3 to obtain high-frequency residual information; and superimposing the high-frequency residual information with the degraded image to obtain a predicted image.

[0011] Among them, the step of training the image enhancement model using the difference between the predicted image and the clear sample image includes: obtaining the loss function of the image enhancement model through the difference between the predicted image and the clear sample image; adjusting the parameters of the image enhancement model through the loss function to train the image enhancement model.

[0012] The loss function includes at least one of a reconstruction loss function, a perception loss function, and an adversarial loss function.

[0013] Among them, when the loss function includes a reconstruction loss function, the steps of obtaining the loss function of the image enhancement model by the difference between the predicted image and the clear sample image include: calculating the Manhattan distance between the predicted image and the clear sample image based on the predicted image and the clear sample image; performing Gaussian blurring on the clear sample image with a Gaussian kernel of a set value to obtain a blurred image of the clear sample image; subtracting the clear sample image from the blurred image to obtain a texture residual of the clear sample image; normalizing the texture residual to obtain the spatial weight of each position of the clear sample image; multiplying the spatial weight by the Manhattan distance to obtain the reconstruction loss function of the image enhancement model.

[0014] The step of obtaining a clear sample image and a corresponding degraded image includes: obtaining a clear sample image, wherein a resolution of the clear sample image exceeds a resolution threshold; and performing transcoding degradation and noise degradation on the clear sample image to obtain a degraded image.

[0015] Among them, the image enhancement model training method also includes: judging whether the convolution kernel in each convolution layer in the image enhancement model training is important through the L1 paradigm; if the convolution kernel is not important, removing the convolution kernel based on the pruning ratio, and adjusting the model parameters on the image enhancement model.

[0016] The present application also provides an image enhancement model training device, including: an acquisition module, used to acquire a clear sample image and a degraded image corresponding to the clear sample image; a feature extraction module, used to extract features from the degraded image to obtain a first image feature of the degraded image; a mining module, used to perform feature mining on the first image feature through multiple cascaded residual networks to obtain a second image feature; an enhancement module, used to perform image enhancement on the degraded image based on the second image feature to obtain a predicted image; and a training module, used to train the image enhancement model using the difference between the predicted image and the clear sample image.

[0017] The present application also provides an image enhancement method, comprising: obtaining an image to be enhanced and an image enhancement model, wherein the image enhancement model is trained by any of the above-mentioned image enhancement model training methods; performing feature extraction on the image to be enhanced using the image enhancement model to obtain a first image feature of the image to be enhanced; performing feature mining on the first image feature through multiple cascaded residual networks in the image enhancement model to obtain a second image feature of the image to be enhanced; performing image enhancement on the image to be enhanced based on the second image feature through the image enhancement model to obtain an enhanced image.

[0018] The present application also provides an image enhancement device, including: an acquisition module, used to acquire an image to be enhanced and an image enhancement model, wherein the image enhancement model is trained by any of the image enhancement model training methods described above; a feature extraction module, used to perform feature extraction on the image to be enhanced using the image enhancement model to obtain a first image feature of the image to be enhanced; a mining module, used to perform feature mining on the first image feature through multiple cascaded residual networks in the image enhancement model to obtain a second image feature of the image to be enhanced; and an enhancement module, used to perform image enhancement on the image to be enhanced based on the second image feature through the image enhancement model to obtain an enhanced image.

[0019] The present application also provides an electronic device, comprising a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned image enhancement model training method or image enhancement method.

[0020] The present application also provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the above-mentioned image enhancement model training method or image enhancement method is implemented.

[0021] The above scheme first extracts features from the degraded image to obtain the first image features of the degraded image, then performs feature mining on the first image features through multiple cascaded residual networks to obtain the second image features, and after performing image enhancement on the degraded image based on the second image features to obtain the predicted image, the image enhancement model is trained using the difference between the predicted image and the clear sample image. This embodiment can perform feature mining on the first image features through multiple cascaded residual networks, thereby improving the semantic expression ability of the first image features to obtain the second image features, and finally train the image enhancement model using the difference between the predicted image obtained through the second image features and the clear sample image, thereby fully improving the enhancement effect of the image enhancement model, so that the output of the image enhancement model can be close to the clear sample image to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flowchart of an embodiment of the image enhancement model training method of the present application;

[0023] Figure 2 It is a flowchart of another embodiment of the image enhancement model training method of the present application;

[0024] Figure 3 yes Figure 2 A schematic diagram of a process for obtaining a reconstruction loss function according to an embodiment;

[0025] Figure 4 yes Figure 2 Schematic diagram of the structure of an embodiment of an image enhancement model;

[0026] Figure 5 yes Figure 2 Schematic diagram of the local structure of an embodiment of a feature encoding subnetwork;

[0027] Figure 6 yes Figure 2 Schematic diagram of the local structure of an embodiment of a feature decoding subnetwork;

[0028] Figure 7 It is a flowchart of an embodiment of the image enhancement method of the present application;

[0029] Figure 8 It is a schematic diagram of the framework of an embodiment of the image enhancement model training device of the present application;

[0030] Fig. 9 It is a schematic diagram of the framework of an embodiment of the image enhancement device of the present application;

[0031] Fig.10 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0032] Fig.11 A schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0033] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0034] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0035] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of the associated objects. There can be three kinds of relationships. For example, A and / or B can be: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally means that the associated objects before and after are in an "or" relationship. In addition, "more" in this article means two or more than two.

[0036] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the image enhancement model training method of the present application. Specifically, it may include the following steps:

[0037] Step S11: obtaining a clear sample image and a degraded image corresponding to the clear sample image.

[0038] Before training the image enhancement model, prepare the training data set first.

[0039] In a specific application scenario, a GPU graphics card or a high-definition camera with super rendering capabilities can be used to record lossless ultra-high-definition sample videos, and in order to remove redundancy in the sample content, multiple frames of images in the sample video can be extracted at a preset frequency to obtain clear sample images. The preset frequency can be 100 frames / extraction, 200 frames / extraction, etc. The specific preset frequency can be determined based on actual conditions and is not limited here.

[0040] Among them, the sample videos can be original pictures of game live broadcasts, dance recording videos, green screen videos of outdoor activities, etc., and there is no limitation here.

[0041] After obtaining a clear sample image, the clear sample image is degraded to obtain a degraded image corresponding to the clear sample image. The degradation process of the clear sample image can simulate the degradation phenomenon of various types of sample videos in actual applications, so that the training data of the training data set can be enhanced based on the targeted sample type, thereby improving the image enhancement effect.

[0042] Step S12: extracting features from the degraded image to obtain first image features of the degraded image.

[0043] The image enhancement model is used to extract features of each degraded image to obtain first image features corresponding to each degraded image.

[0044] In a specific application scenario, the degraded image can be feature extracted through a convolution layer to obtain the first image feature of the degraded image. In a specific application scenario, convolution processing can be performed based on the three channels (RGB mode) of the degraded image to obtain the multi-channel features of the degraded image, thereby obtaining the first image features of each degraded image.

[0045] In a specific application scenario, the degraded image may also be subjected to feature extraction through a tree model, L1, L2 penalty values ​​or a recursive feature elimination method, thereby obtaining the first image feature of the degraded image. This is not limited here.

[0046] Step S13: performing feature mining on the first image features through multiple cascaded residual networks to obtain second image features.

[0047] The first image feature is subjected to feature mining through multiple cascaded residual networks to obtain the second image feature. The first image feature is subjected to feature mining through multiple cascaded residual networks to enrich the image feature of the degraded image to obtain the second image feature.

[0048] In a specific application scenario, the first image feature may be subjected to multiple convolution processes, thereby realizing feature mining of the first image feature and improving the semantic expression capability of the first image feature.

[0049] Step S14: performing image enhancement on the degraded image based on the second image feature to obtain a predicted image.

[0050] After the second image feature is acquired, image enhancement processing is performed on the degraded image based on the second image feature to obtain an output of the image enhancement model, that is, a predicted image.

[0051] Step S15: train the image enhancement model using the difference between the predicted image and the clear sample image.

[0052] After obtaining the predicted image, the predicted image is compared with the clear sample image to obtain the difference between the predicted image and the clear sample image. The image enhancement model is trained based on the difference between the predicted image and the clear sample image. In a specific application scenario, a difference threshold can be set, and it can be determined whether the difference between the predicted image and the clear sample image is less than the difference threshold. When the difference between the predicted image and the clear sample image is not less than the difference threshold, the relevant parameters of the image enhancement training model are modified based on the difference between the predicted image and the clear sample image. When the difference between the predicted image and the clear sample image is less than the difference threshold, the image enhancement training model training is completed.

[0053] Through the above-mentioned manner, the image enhancement model training method of this embodiment first extracts features from the degraded image to obtain the first image features of the degraded image, then performs feature mining on the first image features through multiple cascaded residual networks to obtain the second image features, and after performing image enhancement on the degraded image based on the second image features to obtain the predicted image, the image enhancement model is trained using the difference between the predicted image and the clear sample image. This embodiment can perform feature mining on the first image features through multiple cascaded residual networks, thereby improving the semantic expression ability of the first image features, obtaining the second image features, and finally using the difference between the predicted image obtained through the second image features and the clear sample image to train the image enhancement model, thereby being able to fully improve the enhancement effect of the image enhancement model, so that the output of the image enhancement model can be close to the clear sample image to a certain extent.

[0054] See also Figure 2 , Figure 2 It is a flowchart of another embodiment of the image enhancement model training method of the present application. In this embodiment, the training direction of the image enhancement model is trained using the original game live broadcast as a training sample. In other embodiments, other training directions may also be used, such as dance dynamic pictures, outdoor pictures, etc., or multiple directions may be trained at the same time, and the subsequent degradation processing may also be based on the actual degradation phenomenon of the training direction, which is not limited here.

[0055] Step S21: obtaining a clear sample image, wherein the resolution of the clear sample image exceeds a resolution threshold, performing transcoding degradation and noise degradation on the clear sample image to obtain a degraded image.

[0056] In game live broadcasting, the types of video quality degradation are diverse. First, the game engine rendering capabilities of each anchor are limited, which may cause noise in the original picture of the live video, that is, the original game picture, and blurred local details during the rendering process. Some anchors have poor network environments and cannot transmit high-definition videos in real time, resulting in excessive compression of the video before transmission, resulting in severe compression noise.

[0057] Therefore, in this step, the lossless ultra-high-definition game original painting can be recorded by a GPU graphics card with super rendering capabilities to obtain a sample video, and the sample video is framed to obtain a clear sample image. In a specific application scenario, the resolution of the clear sample image needs to exceed the resolution threshold to ensure the training effect of the final model, where the resolution threshold can be set based on the actual application and is not limited here.

[0058] In a specific application scenario, a GPU graphics card with super rendering capabilities can be used to record lossless ultra-high-definition game original pictures for 10 hours, including a variety of popular games. In order to remove redundancy in image content, one frame of image is extracted every 100 frames of video, for a total of 21,600 frames of image.

[0059] The clear sample image is transcoded and degraded to obtain a degraded image. Since this embodiment refers to training in the direction of game live broadcast, the transcoding degradation and noise degradation in this step need to simulate the degradation phenomenon of actual game live broadcast. In a specific application scenario, a variety of bit rates, a variety of transcoding methods, and a variety of quantization parameters can be used to transcode and degrade the clear sample image, and then JPEG compression, Gaussian noise, Poisson noise, etc. are randomly added to simulate the various degradation phenomena that occur in the game rendering and transmission transcoding of the live stream. In a specific application scenario, 5 bit rates (1M, 2M, 4M, 6M and 8M) and 2 transcoding methods (h264 and h265) and 10 quantization parameters (QP values) (10, 15, 24, 28, 32, 36, 38, 40, 44, 48) can be used, and a total of 100 transcoding configurations can be used to transcode the clear sample images to generate low-quality data. Then, by randomly adding JPEG compression, Gaussian noise and Poisson noise, the various degradation phenomena that occur in the game rendering and transmission transcoding of the live stream are simulated to obtain the degraded images corresponding to each clear sample image. After the above three steps, a total of 2160000 degraded images can be generated for 21600 frames of clear sample images. The samples contain multi-dimensional degradation effects such as transcoding degradation, noise degradation and compression degradation. Among them, one frame of clear sample image can correspond to multiple frames of degraded images with different degradation phenomena, so as to simulate the picture degradation in the actual game live broadcast as much as possible and improve the robustness of the image enhancement model.

[0060] Step S22: extracting features from the degraded image to obtain first image features of the degraded image.

[0061] The feature extraction subnetwork of the image enhancement model is used to extract features of the degraded image to obtain the first image feature of the degraded image.

[0062] In a specific application scenario, the degraded image can be feature extracted through a two-dimensional convolution layer to obtain a multi-channel first image feature of the degraded image. In a specific application scenario, the three channels of the image (RGB mode) can be mapped to 64 channels through a two-dimensional 5x5 convolution, thereby obtaining a first image feature with 64 channel features. In other embodiments, first image features of other numbers of channels can also be obtained, which is not limited here.

[0063] Step S23: encoding the first image features through multiple cascaded first residual networks to obtain encoded image features.

[0064] The first image feature is feature encoded by the feature encoding subnetwork of the image enhancement model to obtain the encoded image feature. Specifically, the feature encoding subnetwork of the image enhancement model may include multiple cascaded first residual networks. In a specific application scenario, the feature encoding subnetwork of the image enhancement model may include four cascaded first residual networks.

[0065] In a specific application scenario, the first image features can be downsampled using four first residual networks cascaded in sequence using a convolution layer with a step size of 2 to obtain encoded image features. In a specific application scenario, four first residual networks cascaded in sequence can also be used to perform 2x downsampling using a 3x3 convolution with a step size of 2, and the number of channels is increased by 2 times each time the sampling is downsampled to enhance the semantic expression ability of the first image features. Each first residual network also includes 4 densely connected convolution layers with a kernel size of 3x3 and one convolution layer with a kernel size of 1x1. In the downsampling process, although the semantic expression ability of the features will be enhanced, the size of the encoded image features will also be changed to adapt them to the display area for easy feature mining.

[0066] In a specific application scenario, when the first image feature of 64 channels is obtained, it is input into the 3x3 convolution layer with a step size of 2 in the first residual network for convolution processing, and then sequentially input into 4 convolution layers with a kernel size of 3x3 and a convolution layer with a kernel size of 1x1 in the first residual network for convolution processing, and the output result of the first convolution layer with a kernel size of 1x1 is input into the 3x3 convolution layer with a step size of 2 in the second first residual network for convolution processing, and then sequentially input into 4 convolution layers with a kernel size of 3x3 and a convolution layer with a kernel size of 1x1 in the second first residual network for convolution processing. The output result of the second convolutional layer with a kernel size of 1x1 is input into the 3x3 convolutional layer with a step size of 2 in the third first residual network for convolution processing, and then sequentially input into the 4 convolutional layers with a kernel size of 3x3 and one convolutional layer with a kernel size of 1x1 in the third first residual network for convolution processing. The output result of the third convolutional layer with a kernel size of 1x1 is input into the 3x3 convolutional layer with a step size of 2 in the fourth first residual network for convolution processing, and then sequentially input into the 4 convolutional layers with a kernel size of 3x3 and one convolutional layer with a kernel size of 1x1 in the fourth first residual network for convolution processing, and finally obtain the encoded image features. Among them, the number of channels of the encoded image features is 64. 4 , thereby improving the semantic expression ability of the encoded image features. When there are multiple residual networks in other embodiments, the processing method is similar to the above application scenario and will not be repeated here.

[0067] Step S24: feature decoding is performed on the encoded image features through multiple cascaded second residual networks to obtain decoded image features that match the size of the degraded image, and the decoded image features are determined as second image features.

[0068] The encoded image features are feature decoded by the feature decoding subnetwork of the image enhancement model to obtain decoded image features that match the size of the degraded image, and the decoded image features are determined as second image features. Specifically, the feature encoding subnetwork of the image enhancement model may include multiple cascaded second residual networks. In a specific application scenario, the feature encoding subnetwork of the image enhancement model may include four cascaded second residual networks.

[0069] In a specific application scenario, the first image features can be upsampled by a factor of 2 using pixel reorganization through at least four sequentially cascaded second residual networks, and the number of channels is reduced by half after upsampling to obtain decoded image features that match the size of the degraded image; wherein each second residual network also includes four densely connected deconvolution layers with a kernel size of 3x3 and one deconvolution layer with a kernel size of 1x1.

[0070] Specifically, when 64 4After encoding the image features of the channel, it is input into the first second residual network for 2x upsampling through pixel reorganization (PixelShuffle), and then sequentially input into the first second residual network for 4 deconvolution layers with a kernel size of 3x3 and one deconvolution layer with a kernel size of 1x1 for deconvolution. The output result of the first deconvolution layer with a kernel size of 1x1 is input into the second second residual network for 2x upsampling through pixel reorganization, and then sequentially input into the second second residual network for 4 deconvolution layers with a kernel size of 3x3 and one deconvolution layer with a kernel size of 1x1 for deconvolution. The output results of the two convolution deconvolution layers with a kernel size of 1x1 are input into the third second residual network for 2x upsampling by pixel reorganization, and then sequentially input into the third residual network for 4 deconvolution layers with a kernel size of 3x3 and one deconvolution layer with a kernel size of 1x1 for deconvolution processing, and the output results of the third deconvolution layer with a kernel size of 1x1 are input into the fourth second residual network for 2x upsampling by pixel reorganization, and then sequentially input into the fourth residual network for 4 deconvolution layers with a kernel size of 3x3 and one deconvolution layer with a kernel size of 1x1 for deconvolution processing, and finally the decoded image features are obtained. Among them, the number of channels of the decoded image features is 64, so that the size of the decoded image features is adapted to the degraded image, and the decoded image features are determined as the second image features, that is, the deep convolution features of the degraded image. When there are multiple residual networks in other embodiments, the processing method is similar to the above application scenario, which will not be repeated here.

[0071] Common upsampling methods include bilinear interpolation, transposed convolution, upsampling (unsampling) and up-pooling (unpooling). This step does not limit the specific upsampling method.

[0072] Step S25: Process the second image feature through a convolutional layer with a kernel size of 3x3 to obtain high-frequency residual information, and superimpose the high-frequency residual information with the degraded image to obtain a predicted image.

[0073] The second image feature is processed by the image reconstruction subnetwork of the image enhancement model to achieve image enhancement of the degraded image.

[0074] In a specific application scenario, the second image feature is processed by a convolutional layer with a kernel size of 3x3 to obtain high-frequency residual information. Since the size of the second image feature is adjusted to fit the degraded image during feature decoding, the high-frequency residual information of this step is also adapted to the degraded image. The high-frequency residual information is superimposed on the degraded image to obtain a predicted image. The predicted image is the output of the image enhancement model during the training process.

[0075] Step S26: obtaining the loss function of the image enhancement model by calculating the difference between the predicted image and the clear sample image, and adjusting the parameters of the image enhancement model by using the loss function to train the image enhancement model.

[0076] The loss function of the image enhancement model is obtained by calculating the difference between the predicted image and the clear sample image. The parameters of the image enhancement model are adjusted through the loss function to train the image enhancement model.

[0077] In a specific application scenario, the loss function includes at least one of a reconstruction loss function, a perception loss function, and an adversarial loss function. For example, the loss function may include a reconstruction loss function and a perception loss function, or the loss function may include a reconstruction loss function, a perception loss function, and an adversarial loss function, which are not limited here.

[0078] See also Figure 3 , Figure 3 yes Figure 2 A schematic diagram of the process of obtaining the reconstruction loss function of an embodiment.

[0079] Step S31: Calculate the Manhattan distance between the predicted image and the clear sample image based on the predicted image and the clear sample image.

[0080] This embodiment uses Manhattan distance as the basic reconstruction loss function and performs spatial content adaptive weighting on it. First, the Manhattan distance (L1 distance) between the predicted image and the clear sample image is calculated.

[0081] Step S32: performing Gaussian blurring with a Gaussian kernel of a set value on the clear sample image to obtain a blurred image of the clear sample image.

[0082] The clear sample image is Gaussian blurred with a Gaussian kernel of a set value to obtain a blurred image of the clear sample after Gaussian blurring. In a specific application scenario, the clear sample image can be Gaussian blurred with a Gaussian kernel of 5 to obtain a blurred image corresponding to the clear sample image.

[0083] Step S33: subtract the clear sample image from the blurred image to obtain a texture residual of the clear sample image.

[0084] The clear sample image is subtracted from the blurred image to obtain the texture residual of the clear sample image, where the texture residual reflects the position where the texture difference is large in the clear sample image.

[0085] Step S34: normalize the texture residual to obtain the spatial weight of each position of the clear sample image.

[0086] The texture residual is normalized to obtain the spatial weight of each position of the clear sample image.

[0087] Step S35: Multiply the spatial weight by the Manhattan distance to obtain the reconstruction loss function of the image enhancement model.

[0088] The spatial weights of each position of the clear sample image obtained in the previous step and the Manhattan distance between the predicted image and the clear sample image obtained in step S31 are multiplied, and the Manhattan distance is adjusted to obtain the reconstruction loss function of the image enhancement model. Thus, the content-aware weighted reconstruction loss function can well restore the high-frequency area of ​​the degraded image and make the output degraded image clearer.

[0089] The method for obtaining the perceptual loss function includes: first, using the VGG deep convolutional network to extract multi-layer convolution features of the predicted image and the clear sample image, the multi-layer convolution features include both low-layer texture features and high-layer semantic features, and then respectively calculating the cosine distances between the multi-layer convolution feature maps, and finally performing weighted summation of the cosine distances of different layers to obtain the perceptual loss value.

[0090] The adversarial loss function uses a generative adversarial network to generate loss values ​​through the discriminative network while learning.

[0091] Based on the above three loss functions, the output of the image enhancement model is fed back, so that the relevant model parameters of the image enhancement model are modified based on the clear sample image to complete the training of the image enhancement model.

[0092] In a specific application scenario, during the training process of an image enhancement model, in order to improve the running speed of the image enhancement model, the network pruning method can be used to prune the image enhancement model. Pruning is a technique of deep learning, and its goal is to develop smaller and more efficient neural networks. This is a model optimization technique that includes removing redundant values ​​from weight tensors. The pruned image enhancement model can run faster and also reduce the computational cost of the image enhancement model. Specifically, the L1 paradigm is used to determine whether the convolution kernels in each convolution layer in the image enhancement model training are important; if the convolution kernels are not important, the convolution kernels are removed based on the pruning ratio, and the model parameters on the image enhancement model are adjusted.

[0093] In a specific application scenario, first, in each convolution layer of the image enhancement model, the L1 norm value of the convolution kernel is used to determine whether the convolution kernel is important. The pruning ratio is set to directly remove the unimportant convolution kernels of the layer, and fine-tune the training. When determining the pruning ratio, it is assumed that each convolution layer is independent of each other, and it is pruned at different pruning ratios, and the performance of the model in the validation set of PSNR value (peak signal-to-noise ratio) and VMAF value (a video quality indicator) is evaluated, and a sensitivity analysis is performed to determine a reasonable pruning ratio. Among them, when pruning, if a convolution kernel in a convolution layer is removed, the corresponding channel in its output feature map will be missing, so the parameters of the corresponding channels of the subsequent normalization layer and convolution layer should be adjusted accordingly to completely remove the convolution kernel and realize network pruning.

[0094] In a specific application scenario, after the training of the image enhancement training model is completed, before the image enhancement model is applied, the deployment hardware GPU of the image enhancement model can be deployed for quantization acceleration. Specifically, during the training process of the image enhancement model, FP32 precision can be used for training, and during the deployment of the image enhancement model, low-precision data INT8 can be used for deployment. Among them, the process of reducing FP32 precision to INT8 precision is equivalent to information re-encoding, which converts the original use of 32 bits to represent a tensor into using 8 bits to represent a tensor.

[0095] In a specific application scenario, linear mapping (or linear quantization) can be used to map the ±|T| of the training data to ±127, and directly map the value beyond the threshold ±|T| to the threshold ±127. This mapping relationship is called saturation. By statistically analyzing the distribution of activation values ​​in the output results of each convolutional layer in the image enhancement model, a suitable threshold is selected to map the scattered and larger activation values ​​to 127, so that the accuracy loss is not reduced too much. Specifically, the T value can be calculated by the KL divergence value (Kullback–Leibler divergence, also known as KL distance). When the activation value is greater than ±|T|, the activation value is mapped to 127 or -127 based on the positive and negative values, that is, when the activation value is negative, it is mapped to -127, and when the activation value is positive, it is mapped to 127. When the activation value is less than ±|T|, it is mapped to ±127 based on the linear mapping rule, thereby completing the deployment quantization acceleration of the image enhancement model, thereby improving the running speed of the image enhancement model, and significantly improving the reasoning ability of the model with only a slight decrease in the PSNR and VMAF indicators, greatly reducing the cost of large-scale deployment.

[0096] Through the above method, the image enhancement model training method of this embodiment first prepares a degraded image that simulates the actual degradation phenomenon of the game live broadcast, thereby ensuring to a certain extent that the image enhancement model can be trained based on the direction of the game live broadcast, thereby improving the pertinence and image enhancement effect of the image enhancement model. Specifically, the image enhancement model training method sequentially performs feature extraction, feature encoding, feature decoding and image reconstruction on the degraded image, wherein both feature extraction and feature encoding are subjected to feature mining through multiple cascaded residual networks, so that the semantic expression ability of the image features can be fully enriched to a certain extent through the residual network, so that the image enhancement can be enhanced based on the enriched image features to directly improve the image enhancement effect of the image enhancement model. The image enhancement model is optimized through three loss functions to maintain the spatial structure of the degraded image, while improving the visual quality of the predicted image, and using the labeled image to enrich the texture, thereby ensuring the training effect of the image enhancement model. In addition, with the improvement of the live broadcast image quality, it provides greater bit rate saving space for subsequent transcoding services, which can further reduce bandwidth costs.

[0097] See also Figure 4-6 , Figure 4 yes Figure 2 Schematic diagram of the structure of an embodiment of an image enhancement model. Figure 5 yes Figure 2 Schematic diagram of the local structure of the first embodiment of the feature encoding subnetwork, Figure 6 yes Figure 2 Schematic diagram of the local structure of an embodiment of a feature decoding subnetwork.

[0098] The image enhancement model 40 of this embodiment includes a feature extraction subnetwork 41, a feature encoding subnetwork 42, a feature decoding subnetwork 43, an image reconstruction subnetwork 44, and a loss function subnetwork 45. The image enhancement model 40 of this embodiment can use the U-net network (image semantic segmentation network) as a basic network for training. In other embodiments, other image enhancement networks can also be used as basic networks for training.

[0099] The degraded image is input into the feature extraction subnetwork 41 for feature extraction to obtain the first image feature, the first image feature is input into the feature encoding subnetwork 42 for feature encoding to obtain the encoded image feature, the encoded image feature is input into the feature decoding subnetwork 43 for feature decoding to obtain the decoded image feature, i.e. the second image feature, the second image feature is input into the image reconstruction subnetwork 44 for image reconstruction to obtain the predicted image. Feedback processing is performed based on the predicted image through the loss function subnetwork 45 to modify the relevant parameters of the image enhancement model 40 and complete the training of the image enhancement model 40.

[0100] The feature extraction sub-network 41 includes a two-dimensional 5x5 convolutional layer to extract features from the degraded graph.

[0101] The feature encoding subnetwork 42 includes a plurality of first residual networks 421, wherein the first residual network 421 includes: a 3x3 convolution RDB with a step size of 2, a convolution layer conv1 with a kernel size of 3x3, a convolution layer conv2 with a kernel size of 3x3, a convolution layer conv3 with a kernel size of 3x3, a convolution layer conv4 with a kernel size of 3x3, and a convolution conv5 with a kernel size of 1x1. The first image feature is input into the 3x3 convolution RDB with a step size of 2 for processing, and then sequentially input into each convolution layer for feature encoding, and finally the encoded image feature is obtained.

[0102] The feature decoding subnetwork 43 includes a plurality of second residual networks 431, wherein the second residual network 431 includes: a pixel reorganization subnetwork 4311, a deconvolution layer 4312 with a kernel size of 3x3, a deconvolution layer 4313 with a kernel size of 3x3, a deconvolution layer 4314 with a kernel size of 3x3, a deconvolution layer 4315 with a kernel size of 3x3, and a deconvolution layer 4316 with a kernel size of 1x1. The encoded image features are input into the pixel reorganization subnetwork 4311 for processing, and then sequentially input into each deconvolution layer for feature decoding, and finally the decoded image features are obtained. Among them, the feature encoding subnetwork 42 is symmetrical with the feature decoding subnetwork 43.

[0103] The image reconstruction subnetwork 44 includes a convolutional layer with a kernel size of 3x3 to process the second image features to obtain high-frequency residual information.

[0104] Through the above structure, the image enhancement model of this embodiment can quickly remove the degradation phenomenon on the degraded image, thereby achieving image enhancement of the degraded image.

[0105] See also Figure 7 , Figure 7 It is a flowchart of an embodiment of the image enhancement method of the present application.

[0106] Step S51: Obtain the image to be enhanced and the image enhancement model.

[0107] The image to be enhanced and the image enhancement model are obtained. In a specific application scenario, the image to be enhanced may be a live game video, a live dance video, or other types of live videos or other images that require image enhancement. The image enhancement model may be an image enhancement model trained based on the type of the image to be enhanced. Specifically, the training method of the image enhancement model of this embodiment is similar to the above-mentioned Figure 1 Example or Figure 2 The image enhancement model training method of the embodiment is the same, please refer to the previous text and will not be repeated here.

[0108] In a specific application scenario, when the image to be enhanced is a live game video in a live broadcast scenario, the image enhancement model can be an image enhancement model trained based on the original game live broadcast image as a training sample, thereby improving the adaptability between the image enhancement model and the image to be enhanced.

[0109] In a specific application scenario, the image enhancement model may include a feature extraction subnetwork, a feature encoding subnetwork, a feature decoding subnetwork and an image reconstruction subnetwork, and the feature extraction subnetwork may include a two-dimensional 5x5 convolution layer. The feature encoding subnetwork may include multiple first residual networks, wherein the first residual network may include: a 3x3 convolution layer with a step size of 2, 4 convolution layers with a kernel size of 3x3 and a convolution layer with a kernel size of 1x1. The feature decoding subnetwork may include multiple second residual networks, wherein the second residual network may include: a pixel reorganization subnetwork, 4 deconvolution layers with a kernel size of 3x3 and a deconvolution layer with a kernel size of 1x1. The image reconstruction subnetwork may include a convolution layer with a kernel size of 3x3.

[0110] Step S52: extracting features of the image to be enhanced by using the image enhancement model to obtain first image features of the image to be enhanced.

[0111] The image enhancement model is used to extract features of the image to be enhanced, so as to obtain first image features corresponding to the image to be enhanced.

[0112] In a specific application scenario, the image to be enhanced can be feature extracted through the convolution layer of the image enhancement model to obtain the first image feature of the image to be enhanced. In a specific application scenario, convolution processing can be performed based on the three channels (RGB mode) of the image to be enhanced to obtain the multi-channel features of the image to be enhanced, thereby obtaining the first image feature of the image to be enhanced.

[0113] In a specific application scenario, the two-dimensional 5x5 convolutional layer in the feature extraction subnetwork of the image enhancement model can be used to extract features of the image to be enhanced, thereby obtaining the first image features of the image to be enhanced.

[0114] Step S53: performing feature mining on the first image feature through multiple cascaded residual networks in the image enhancement model to obtain the second image feature of the image to be enhanced.

[0115] The first image feature is feature mined by multiple cascaded residual networks in the image enhancement model to obtain the second image feature. The first image feature is feature mined by multiple cascaded residual networks to enrich the image feature of the degraded image to obtain the second image feature.

[0116] In a specific application scenario, the first image feature may be subjected to multiple convolution processes, thereby realizing feature mining of the first image feature and improving the semantic expression capability of the first image feature.

[0117] In a specific application scenario, the first image features may be sequentially encoded by multiple first residual networks of the feature encoding subnetwork of the image enhancement model to obtain the encoded image features of the image to be enhanced. The encoded image features are then feature decoded by the feature decoding subnetwork of the image enhancement model to obtain the decoded image features, i.e., the second image features of the image to be enhanced.

[0118] Step S54: performing image enhancement on the image to be enhanced based on the second image feature by using the image enhancement model to obtain an enhanced image.

[0119] After acquiring the second image feature, the image enhancement model is used to perform image enhancement processing on the image to be enhanced based on the second image feature to obtain the output of the image enhancement model, that is, the enhanced image.

[0120] In a specific application scenario, the second image feature can be processed by the image reconstruction subnetwork of the image enhancement model to obtain high-frequency residual information, and the high-frequency residual information is superimposed on the image to be enhanced to obtain an enhanced image.

[0121] Through the above method, the image enhancement method of this embodiment uses the image enhancement model to perform feature extraction, feature encoding, feature decoding and image reconstruction on the image to be enhanced in sequence, wherein the feature extraction and feature encoding are both subjected to feature mining through multiple cascaded residual networks, so that the semantic expression ability of the image features can be fully enriched to a certain extent through the residual network, so that the image enhancement can be performed based on the enriched image features, so as to directly improve the image enhancement effect of the image enhancement model and improve the visual quality of the enhanced image.

[0122] See also Figure 8 , Figure 8 It is a schematic diagram of the framework of an embodiment of the image enhancement model training device of the present application. The image enhancement model training 80 includes an acquisition module 81, a feature extraction module 82, a mining module 83, an enhancement module 84 and a training module 85. The acquisition module 81 is used to acquire a clear sample image and a degraded image corresponding to the clear sample image; the feature extraction module 82 is used to extract features from the degraded image to obtain a first image feature of the degraded image; the mining module 83 is used to perform feature mining on the first image feature through multiple cascaded residual networks to obtain a second image feature; the enhancement module 84 is used to enhance the degraded image based on the second image feature to obtain a predicted image; the training module 85 is used to train the image enhancement model using the difference between the predicted image and the clear sample image.

[0123] The acquisition module 81 is further used to acquire a clear sample image, wherein the resolution of the clear sample image exceeds a resolution threshold; and perform transcoding degradation and noise degradation on the clear sample image to obtain a degraded image.

[0124] The mining module 83 is also used to perform feature encoding on the first image features through multiple cascaded first residual networks to obtain encoded image features; and perform feature decoding on the encoded image features through multiple cascaded second residual networks to obtain decoded image features that match the size of the degraded image, and determine the decoded image features as second image features.

[0125] The enhancement module 84 is further used to process the second image feature through a convolution layer with a kernel size of 3×3 to obtain high-frequency residual information; and to superimpose the high-frequency residual information with the degraded image to obtain a predicted image.

[0126] The training module 85 is also used to obtain the loss function of the image enhancement model by calculating the difference between the predicted image and the clear sample image; and adjust the parameters of the image enhancement model by using the loss function to train the image enhancement model.

[0127] The above scheme can enhance the image enhancement effect of the image enhancement model.

[0128] See also Fig. 9 , Fig. 9 It is a schematic diagram of the framework of an embodiment of the image enhancement device of the present application.

[0129] The image enhancement device 90 includes an acquisition module 91 for acquiring an image to be enhanced and an image enhancement model, wherein the image enhancement model is obtained by Figure 1 Example or Figure 2 The image enhancement model training method of the embodiment is used for training; a feature extraction module 92 is used to use the image enhancement model to extract features of the image to be enhanced, so as to obtain the first image features of the image to be enhanced; a mining module 93 is used to perform feature mining on the first image features through multiple cascaded residual networks in the image enhancement model, so as to obtain the second image features of the image to be enhanced; an enhancement module 94 is used to perform image enhancement on the image to be enhanced based on the second image features through the image enhancement model, so as to obtain an enhanced image.

[0130] The above solution can improve the image quality of the image to be enhanced.

[0131] See also Fig.10 , Fig.10It is a schematic diagram of the framework of an embodiment of an electronic device of the present application. The electronic device 100 includes a memory 101 and a processor 102 coupled to each other, and the processor 102 is used to execute program instructions stored in the memory 101 to implement the steps of any of the above-mentioned image enhancement model training method embodiments or the steps of any of the above-mentioned image enhancement method embodiments. In a specific implementation scenario, the electronic device 100 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 100 may also include mobile devices such as laptops and tablet computers, which are not limited here.

[0132] Specifically, the processor 102 is used to control itself and the memory 101 to implement the steps of any of the above-mentioned image enhancement model training method embodiments. The processor 102 may also be referred to as a CPU (Central Processing Unit). The processor 102 may be an integrated circuit chip having signal processing capabilities. The processor 102 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. In addition, the processor 102 may be implemented by an integrated circuit chip.

[0133] The above scheme can enhance the image enhancement effect of the image enhancement model and improve the image quality of the image to be enhanced.

[0134] See also Fig.11 , Fig.11 The computer-readable storage medium 110 stores program instructions 1101 that can be executed by a processor, and the program instructions 1101 are used to implement the steps of any of the above-mentioned image enhancement model training method embodiments or the steps of any of the above-mentioned image enhancement method embodiments.

[0135] The above scheme can enhance the image enhancement effect of the image enhancement model and improve the image quality of the image to be enhanced.

[0136] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0137] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0138] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0139] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A method for training an image enhancement model, characterized in that: The image enhancement model training method comprises: A clear sample image and a degraded image corresponding to the clear sample image are obtained; wherein the clear sample image is transcoded and degraded by using a variety of bit rates, a variety of transcoding methods, and a variety of quantization parameters and randomly adding JPEG compression, Gaussian noise, and Poisson noise to obtain the degraded image; Performing feature extraction on the degraded image to obtain a first image feature of the degraded image; Performing feature mining on the first image feature through multiple cascaded residual networks to obtain a second image feature; performing image enhancement on the degraded image based on the second image feature to obtain a predicted image; Training the image enhancement model using the difference between the predicted image and the clear sample image; The step of performing feature mining on the first image features through multiple cascaded residual networks to obtain the second image features includes: The first image feature is feature encoded through multiple cascaded first residual networks to obtain an encoded image feature; the encoded image feature is feature decoded through multiple cascaded second residual networks to obtain a decoded image feature matching the size of the degraded image, and the decoded image feature is determined as the second image feature.

2. The image enhancement model training method according to claim 1, characterized in that: The step of encoding the first image feature through a plurality of cascaded first residual networks to obtain an encoded image feature of the first image feature comprises: Down-sampling the first image feature by using at least four first residual networks cascaded in sequence using a convolutional layer with a step size of 2 to obtain the encoded image feature; Each of the first residual networks also includes four densely connected convolutional layers with a kernel size of 3x3 and one convolutional layer with a kernel size of 1x1.

3. The image enhancement model training method according to claim 1, characterized in that: The step of decoding the encoded image features through a plurality of cascaded second residual networks to obtain decoded image features matching the size of the degraded image, and determining the decoded image features as the second image features comprises: Upsampling the first image feature by using pixel reorganization through at least four second residual networks cascaded in sequence to obtain a decoded image feature matching the size of the degraded image; Each of the second residual networks further includes four densely connected deconvolution layers with a kernel size of 3x3 and one deconvolution layer with a kernel size of 1x1.

4. The image enhancement model training method according to claim 1, characterized in that: The step of performing image enhancement on the degraded image based on the second image feature to obtain a predicted image comprises: Processing the second image feature through a convolutional layer with a kernel size of 3x3 to obtain high-frequency residual information; The high-frequency residual information is superimposed on the degraded image to obtain the predicted image.

5. The image enhancement model training method according to claim 1, characterized in that: The step of training the image enhancement model by using the difference between the predicted image and the clear sample image comprises: Obtaining a loss function of the image enhancement model through a difference between the predicted image and the clear sample image; The parameters of the image enhancement model are adjusted by using the loss function to train the image enhancement model.

6. The image enhancement model training method according to claim 5, characterized in that: The loss function includes at least one of a reconstruction loss function, a perception loss function, and an adversarial loss function.

7. The image enhancement model training method according to claim 6, characterized in that: When the loss function includes a reconstruction loss function, the step of obtaining the loss function of the image enhancement model by the difference between the predicted image and the clear sample image includes: The Manhattan distance between the predicted image and the clear sample image is calculated based on the predicted image and the clear sample image; Performing Gaussian blurring with a Gaussian kernel of a set value on the clear sample image to obtain a blurred image of the clear sample image; Subtracting the clear sample image from the blurred image to obtain a texture residual of the clear sample image; Normalizing the texture residual to obtain the spatial weight of each position of the clear sample image; The spatial weight is multiplied by the Manhattan distance to obtain a reconstruction loss function of the image enhancement model.

8. The image enhancement model training method according to claim 1, characterized in that: The step of acquiring a clear sample image and a corresponding degraded image comprises: Acquire the clear sample image, wherein the resolution of the clear sample image exceeds a resolution threshold; The clear sample image is subjected to transcoding degradation and noise degradation to obtain the degraded image.

9. The image enhancement model training method according to claim 1, characterized in that: The image enhancement model training method also includes: Determine whether the convolution kernels in each convolution layer in the image enhancement model training are important by using the L1 paradigm; If the convolution kernel is not important, the convolution kernel is removed based on the pruning ratio, and the model parameters on the image enhancement model are adjusted.

10. An image enhancement model training device, characterized in that: include: An acquisition module is used to acquire a clear sample image and a degraded image corresponding to the clear sample image, wherein the clear sample image is transcoded and degraded by adopting multiple bit rates, multiple transcoding modes, and multiple quantization parameters and randomly adding JPEG compression, Gaussian noise, and Poisson noise to obtain the degraded image; A feature extraction module, used for performing feature extraction on the degraded image to obtain a first image feature of the degraded image; A mining module, configured to perform feature mining on the first image feature through a plurality of cascaded residual networks to obtain a second image feature, wherein the first image feature is feature encoded through a plurality of cascaded first residual networks to obtain an encoded image feature; the encoded image feature is feature decoded through a plurality of cascaded second residual networks to obtain a decoded image feature matching the size of the degraded image, and the decoded image feature is determined as the second image feature; an enhancement module, configured to perform image enhancement on the degraded image based on the second image feature to obtain a predicted image; A training module is used to train the image enhancement model using the difference between the predicted image and the clear sample image.

11. An image enhancement method, characterized in that: The image enhancement method comprises: An image to be enhanced and an image enhancement model are obtained, wherein the image enhancement model is trained by the image enhancement model training method according to any one of claims 1 to 9; Using the image enhancement model to perform feature extraction on the image to be enhanced, to obtain a first image feature of the image to be enhanced; Performing feature mining on the first image feature through a plurality of cascaded residual networks in the image enhancement model to obtain a second image feature of the image to be enhanced; The image to be enhanced is enhanced by using the image enhancement model based on the second image feature to obtain an enhanced image.

12. An image enhancement device, characterized in that: The image enhancement device comprises: An acquisition module, used to acquire an image to be enhanced and an image enhancement model, wherein the image enhancement model is trained by the image enhancement model training method according to any one of claims 1 to 9; A feature extraction module, used to extract features of the image to be enhanced by using the image enhancement model to obtain a first image feature of the image to be enhanced; A mining module, used for performing feature mining on the first image feature through a plurality of cascaded residual networks in the image enhancement model to obtain a second image feature of the image to be enhanced; An enhancement module is used to perform image enhancement on the image to be enhanced based on the second image feature through the image enhancement model to obtain an enhanced image.

13. An electronic device, characterized in that: It includes a memory and a processor coupled to each other, and the processor is used to execute program instructions stored in the memory to implement the image enhancement model training method as described in any one of claims 1 to 9 or the image enhancement method as described in claim 11.

14. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by the processor, the image enhancement model training method as described in any one of claims 1 to 9 or the image enhancement method as described in claim 11 is implemented.

Images