Enhancement model training method, processing method, device, equipment and program product

Abstract

The invention relates to an image enhancement model training method and device, an image processing method and device, electronic equipment and a computer program product, and belongs to the technical field of image processing. The training method of the image enhancement model comprises the following steps: acquiring an input sample image and a corresponding enhanced sample image; inputting the input sample image into a first enhancement model to obtain a first output sample image, and inputting the input sample image into a second enhancement model to obtain a second output sample image; obtaining a model output quality difference according to the first output sample image, the second output sample image and the enhanced sample image; obtaining a multi-order joint variance according to the input sample image and the first output sample image, and obtaining quality constraint loss according to the multi-order joint variance and the output quality difference; and updating model parameters of the first enhancement model based on the quality constraint loss to obtain an image enhancement model. By calculating the quality constraint loss, the image enhancement quality can be effectively improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a training method for an image enhancement model, a training device for an image enhancement model, an image processing method, an image processing device, an electronic device, and a computer program product. Background Art

[0002] In recent years, image processing technology has evolved from traditional methods to neural network methods, and deep learning technology has greatly improved the upper limit of image processing technology. Among them, image enhancement and restoration technology can directly improve the quality of image content and enhance user experience, and has wide applications in video image processing and other fields.

[0003] With the development of video image media, video images will face a series of quality problems in the process of creation, processing, compression, transmission and distribution, which will greatly affect the quality of video image content. However, current image enhancement and restoration technology is difficult to solve the above problems.

[0004] In view of this, there is an urgent need in this field for a method that can effectively improve the quality of image enhancement and restoration.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The purpose of the present disclosure is to provide a training method for an image enhancement model, a training device for an image enhancement model, an image processing method, an image processing device, an electronic device and a computer program product, thereby effectively improving the quality of image enhancement and restoration at least to a certain extent.

[0007] According to a first aspect of the present disclosure, a method for training an image enhancement model is provided, comprising:

[0008] Acquire an input sample image of a first quality and an enhanced sample image of a second quality corresponding to the input sample image, wherein the second quality is higher than the first quality;

[0009] Inputting the input sample image into a pre-trained first enhancement model to obtain a first output sample image, and inputting the input sample image into a pre-trained second enhancement model to obtain a second output sample image, wherein the first enhancement model is trained on the basis of the second enhancement model;

[0010] Obtaining an output quality difference between the first enhancement model and the second enhancement model according to the first output sample image, the second output sample image, and the enhanced sample image;

[0011] Obtaining a multi-order joint variance of the first enhancement model according to the input sample image and the first output sample image, and obtaining a quality constraint loss according to the multi-order joint variance and the output quality difference;

[0012] The model parameters of the first enhancement model are updated based on the quality constraint loss, and an image enhancement model is obtained according to the updated first enhancement model.

[0013] In an exemplary embodiment of the present disclosure, obtaining the output quality difference between the first enhancement model and the second enhancement model according to the first output sample image, the second output sample image and the enhanced sample image includes:

[0014] Obtaining a first quality difference according to the first output sample image and the second output sample image;

[0015] Obtaining a second quality difference according to the second output sample image and the enhanced sample image;

[0016] An output quality difference between the first enhancement model and the second enhancement model is obtained according to the first quality difference and the second quality difference.

[0017] In an exemplary embodiment of the present disclosure, obtaining a first quality difference according to the first output sample image and the second output sample image includes:

[0018] Obtain a first sample variance according to the first output sample image, and obtain a second sample variance according to the second output sample image;

[0019] The first quality difference is obtained according to the first sample variance and the second sample variance.

[0020] In an exemplary embodiment of the present disclosure, obtaining a second quality difference according to the second output sample image and the enhanced sample image includes:

[0021] Obtain a second sample variance according to the second output sample image, and obtain an enhanced sample variance according to the enhanced sample image;

[0022] The second quality difference is obtained according to the second sample variance and the enhanced sample variance.

[0023] In an exemplary embodiment of the present disclosure, obtaining a multi-order joint variance of the first enhancement model according to the input sample image and the first output sample image includes:

[0024] Calculating the first-order variance and the second-order variance of the input sample image, and calculating the first-order variance and the second-order variance of the first output sample image;

[0025] A multi-order joint variance of the first enhancement model is obtained according to the first-order variance and the second-order variance of the input sample image and the first-order variance and the second-order variance of the first output sample image.

[0026] In an exemplary embodiment of the present disclosure, the calculating the first-order variance and the second-order variance of the input sample image includes:

[0027] Obtaining an input image edge of the input sample image by an edge operator, and calculating the variance of the input image edge according to a calculation window of a preset size to obtain a first-order variance of the input sample image;

[0028] The second-order gradient of the input sample image is calculated by a second-order differential operator, and the variance of the second-order gradient of the input sample image is calculated according to the calculation window to obtain the second-order variance of the input sample image.

[0029] In an exemplary embodiment of the present disclosure, the calculating the first-order variance and the second-order variance of the first output sample image includes:

[0030] Obtaining an output image edge of the first output sample image by using an edge operator, and calculating a variance of the output image edge according to a calculation window of a preset size to obtain a first-order variance of the first output sample image;

[0031] The second-order gradient of the first output sample image is calculated by a second-order differential operator, and the variance of the second-order gradient of the first output sample image is obtained according to the calculation window to obtain the second-order variance of the first output sample image.

[0032] In an exemplary embodiment of the present disclosure, obtaining the multi-order joint variance of the first enhancement model according to the first-order variance and the second-order variance of the input sample image and the first-order variance and the second-order variance of the first output sample image includes:

[0033] Obtaining a first-order sample variance ratio according to a ratio of a first-order variance of the first output sample image to a first-order variance of the input sample image;

[0034] Obtaining a second-order sample variance ratio according to a ratio of the second-order variance of the first output sample image to the second-order variance of the input sample image;

[0035] The multi-order joint variance of the first enhanced model is obtained according to the first-order sample variance ratio and the second-order sample variance ratio.

[0036] According to a second aspect of the present disclosure, there is provided an image processing method, comprising:

[0037] Acquire an original image to be processed, and input the original image into a pre-trained image enhancement model, wherein the image enhancement model is obtained by the training method of the image enhancement model as described above;

[0038] The original image is subjected to image enhancement processing by using the image enhancement model to obtain an enhanced image corresponding to the original image.

[0039] According to a third aspect of the present disclosure, a training device for an image enhancement model is provided, comprising:

[0040] A sample image acquisition module is configured to acquire an input sample image of a first quality and an enhanced sample image of a second quality corresponding to the input sample image, wherein the second quality is higher than the first quality;

[0041] A sample image input module is configured to input the input sample image into a pre-trained first enhancement model to obtain a first output sample image, and input the input sample image into a pre-trained second enhancement model to obtain a second output sample image, wherein the first enhancement model is trained on the basis of the second enhancement model;

[0042] a quality difference determination module, configured to obtain an output quality difference between the first enhancement model and the second enhancement model according to the first output sample image, the second output sample image and the enhancement sample image;

[0043] a constraint loss determination module, configured to obtain a multi-order joint variance of the first enhancement model according to the input sample image and the first output sample image, and obtain a quality constraint loss according to the multi-order joint variance and the output quality difference;

[0044] The model parameter updating module is configured to update the model parameters of the first enhancement model based on the quality constraint loss, and obtain an image enhancement model according to the updated first enhancement model.

[0045] In an exemplary embodiment of the present disclosure, the quality difference determination module includes:

[0046] A first quality difference determining unit is configured to obtain a first quality difference according to the first output sample image and the second output sample image;

[0047] A second quality difference determining unit is configured to obtain a second quality difference according to the second output sample image and the enhanced sample image;

[0048] The model output quality difference determining unit is configured to obtain an output quality difference between the first enhanced model and the second enhanced model according to the first quality difference and the second quality difference.

[0049] In an exemplary embodiment of the present disclosure, the first quality difference determining unit includes:

[0050] A first sample variance calculation unit is configured to obtain a first sample variance according to the first output sample image and obtain a second sample variance according to the second output sample image;

[0051] A first quality difference calculation unit is configured to obtain the first quality difference according to the first sample variance and the second sample variance.

[0052] In an exemplary embodiment of the present disclosure, the second quality difference determining unit includes:

[0053] A second sample variance calculation unit is configured to obtain a second sample variance according to the second output sample image and obtain an enhanced sample variance according to the enhanced sample image;

[0054] The second quality difference calculation unit is configured to obtain the second quality difference according to the second sample variance and the enhanced sample variance.

[0055] In an exemplary embodiment of the present disclosure, the constraint loss determination module includes:

[0056] A multi-order variance calculation unit, configured to calculate the first-order variance and the second-order variance of the input sample image, and calculate the first-order variance and the second-order variance of the first output sample image;

[0057] The multi-order joint variance determining unit is configured to obtain the multi-order joint variance of the first enhancement model according to the first-order variance and the second-order variance of the input sample image and the first-order variance and the second-order variance of the first output sample image.

[0058] In an exemplary embodiment of the present disclosure, the multi-order variance calculation unit includes:

[0059] An input sample first-order variance calculation unit is configured to obtain an input image edge of the input sample image by executing an edge operator, and obtain a variance of the input image edge according to a calculation window of a preset size to obtain a first-order variance of the input sample image;

[0060] The input sample second-order variance calculation unit is configured to calculate the second-order gradient of the input sample image through a second-order differential operator, and obtain the variance of the second-order gradient of the input sample image according to the calculation window to obtain the second-order variance of the input sample image.

[0061] In an exemplary embodiment of the present disclosure, the multi-order variance calculation unit further includes:

[0062] an output sample first-order variance calculation unit, configured to obtain an output image edge of the first output sample image by executing an edge operator, and obtain a variance of the output image edge according to a calculation window of a preset size to obtain a first-order variance of the first output sample image;

[0063] The output sample second-order variance calculation unit is configured to calculate the second-order gradient of the first output sample image through a second-order differential operator, and obtain the variance of the second-order gradient of the first output sample image according to the calculation window to obtain the second-order variance of the first output sample image.

[0064] In an exemplary embodiment of the present disclosure, the multi-order joint variance determination unit includes:

[0065] a first-order sample variance ratio calculation unit, configured to obtain a first-order sample variance ratio according to a ratio of a first-order variance of the first output sample image to a first-order variance of the input sample image;

[0066] A second-order sample variance ratio calculation unit is configured to obtain a second-order sample variance ratio according to a ratio of the second-order variance of the first output sample image to the second-order variance of the input sample image;

[0067] The multi-order joint variance calculation unit is configured to obtain the multi-order joint variance of the first enhanced model according to the first-order sample variance ratio and the second-order sample variance ratio.

[0068] According to a fourth aspect of the present disclosure, there is provided an image processing apparatus, comprising:

[0069] Acquire an original image to be processed, and input the original image into a pre-trained image enhancement model, wherein the image enhancement model is obtained by the training method of the image enhancement model as described above;

[0070] The original image is subjected to image enhancement processing by using the image enhancement model to obtain an enhanced image corresponding to the original image.

[0071] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the training method of the image enhancement model described in any one of the above.

[0072] According to a sixth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the training method of the image enhancement model described in any one of the above items is implemented.

[0073] The exemplary embodiments of the present disclosure may have the following beneficial effects:

[0074] In the training method of the image enhancement model of the example implementation mode of the present disclosure, on the one hand, based on the difference between the input and output images of the first enhancement model, the multi-order joint variance of the first enhancement model is obtained, which can effectively measure the degree of texture change and clarity change of the output image of the first enhancement model; on the other hand, by calculating the output quality difference between the first enhancement model and the second enhancement model, and combining the multi-order joint variance and the output quality difference to obtain the quality constraint loss, it is possible to combine the advantages of the outputs of the first enhancement model and the second enhancement model and avoid the disadvantages, thereby effectively improving the subjective quality, avoiding the generation of image artifacts, and enhancing the texture details and clarity of the image, repairing low-quality signals in the image such as compression artifacts, blur, noise, etc., so that the output image effect is more natural and in line with the subjective perception of the human eye.

[0075] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0077] Figure 1 A schematic diagram showing a flow chart of a method for training an image enhancement model according to an exemplary embodiment of the present disclosure;

[0078] Figure 2 A schematic diagram showing a process of determining output quality differences of a model according to an exemplary embodiment of the present disclosure;

[0079] Figure 3 A schematic diagram of a process for determining a multi-order joint variance of a first enhanced model according to an exemplary embodiment of the present disclosure is shown;

[0080] Figure 4 A schematic diagram showing a flow chart of calculating the first-order variance and the second-order variance of an input sample image according to an exemplary embodiment of the present disclosure;

[0081] Figure 5 A schematic diagram showing a flow chart of calculating the first-order variance and the second-order variance of the first output sample image according to an exemplary embodiment of the present disclosure;

[0082] Figure 6 A schematic diagram of a process of calculating multi-order joint variances according to variances of various orders according to an exemplary embodiment of the present disclosure is shown;

[0083] Figure 7 A schematic diagram showing a process of calculating the output quality difference between an MSE-guided model and a GAN-guided model in a specific embodiment of the present disclosure;

[0084] Figure 8 A schematic diagram of a process for performing subjective quality constraints on an image enhancement model in a specific embodiment of the present disclosure is shown;

[0085] Fig. 9 A schematic diagram showing a flow chart of an image processing method according to an exemplary embodiment of the present disclosure;

[0086] Fig.10 A block diagram of a training apparatus for an image enhancement model according to an exemplary embodiment of the present disclosure is shown;

[0087] Fig.11 A block diagram showing an image processing apparatus according to an exemplary embodiment of the present disclosure;

[0088] Fig.12 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0089] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0090] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein.

[0091] The following example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0092] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0093] In some related embodiments, the image quality can be enhanced while achieving high fidelity through CNN (Convolutional Neural Networks), Transformer and other solutions.

[0094] For example, Real-ESRGAN (Real-Enhanced Super-Resolution Generative Adversarial Networks) can simulate the multi-link problems in the image processing chain and enable the network to repair low-quality signals in images and enhance clarity through a large amount of supervised data learning. However, due to over-reliance on data degradation, the network generation results often face the problem of over-smoothing or artifacts.

[0095] In addition, applying the Transformer architecture to low-level image processing tasks can achieve good results. However, although this method greatly improves the quality of network generation, it also faces the problem of artifacts and textures being difficult to distinguish.

[0096] In addition to the above methods, the LDL (Locally Discriminative Learning) scheme focuses on distinguishing artifacts / texture details of network-generated results and improving the naturalness of deep network output results. This scheme calculates the variance of the network output results by region and assigns larger penalty items to regions with larger variances, thereby eliminating image artifacts and retaining texture details. However, distinguishing only by variance does not have good robustness, and it is still difficult to avoid such problems in practical applications.

[0097] Based on the above problems, this example implementation first provides a training method for an image enhancement model. Figure 1 As shown, the training method of the above image enhancement model may include the following steps:

[0098] Step S110: Obtain an input sample image of a first quality and an enhanced sample image of a second quality corresponding to the input sample image, wherein the second quality is higher than the first quality.

[0099] Step S120: Input the input sample image into a pre-trained first enhancement model to obtain a first output sample image, and input the input sample image into a pre-trained second enhancement model to obtain a second output sample image.

[0100] The first enhanced model is trained on the basis of the second enhanced model.

[0101] Step S130. Obtain the output quality difference between the first enhancement model and the second enhancement model according to the first output sample image, the second output sample image and the enhanced sample image.

[0102] Step S140: Obtain a multi-order joint variance of the first enhancement model according to the input sample image and the first output sample image, and obtain a quality constraint loss according to the multi-order joint variance and the output quality difference.

[0103] Step S150: Update the model parameters of the first enhancement model based on the quality constraint loss, and obtain the image enhancement model according to the updated first enhancement model.

[0104] In the training method of the image enhancement model of the example implementation mode of the present disclosure, on the one hand, based on the difference between the input and output images of the first enhancement model, the multi-order joint variance of the first enhancement model is obtained, which can effectively measure the degree of texture change and clarity change of the output image of the first enhancement model; on the other hand, by calculating the output quality difference between the first enhancement model and the second enhancement model, and combining the multi-order joint variance and the output quality difference to obtain the quality constraint loss, it is possible to combine the advantages of the outputs of the first enhancement model and the second enhancement model and avoid the disadvantages, thereby effectively improving the subjective quality, avoiding the generation of image artifacts, and enhancing the texture details and clarity of the image, repairing low-quality signals in the image such as compression artifacts, blur, noise, etc., so that the output image effect is more natural and in line with the subjective perception of the human eye.

[0105] Next, combine Figures 2 to 6 The above steps of this exemplary embodiment are described in more detail.

[0106] In step S110, an input sample image of a first quality and an enhanced sample image of a second quality corresponding to the input sample image are obtained, where the second quality is higher than the first quality.

[0107] In this example implementation, the input sample image of the first quality is a low-resolution or low-definition low-quality image, and the enhanced sample image of the second quality is a high-quality image corresponding to the low-quality image. The low-quality image and its corresponding high-quality image can be used as sample images for model training.

[0108] In step S120, the input sample image is input into a pre-trained first enhancement model to obtain a first output sample image, and the input sample image is input into a pre-trained second enhancement model to obtain a second output sample image.

[0109] In this example implementation, the first enhancement model and the second enhancement model are enhancement models obtained by pre-training based on different model losses, wherein the first enhancement model is trained on the basis of the second enhancement model, and the first enhancement model is trained again on the basis of the first enhancement model to obtain the final image enhancement model.

[0110] For example, the first enhancement model may be a GAN-Oriented network, and the second enhancement model may be an MSE-Oriented network, wherein the GAN-Oriented network is further trained from the MSE-Oriented network. The MSE (Mean-Square Error) output result may be obtained through the MSE-Oriented network, and the GAN (Generative Adversarial Networks) output result may be obtained through the GAN-Oriented network.

[0111] In step S130, the output quality difference between the first enhancement model and the second enhancement model is obtained according to the first output sample image, the second output sample image and the enhanced sample image.

[0112] Based on the first output sample image, the second output sample image and the enhanced sample image, the output quality difference between the first enhancement model and the second enhancement model can be calculated as a means of improving subjective quality.

[0113] In this example implementation, Figure 2 As shown, according to the first output sample image, the second output sample image and the enhanced sample image, the output quality difference between the first enhanced model and the second enhanced model is obtained, which may specifically include the following steps:

[0114] Step S210: Obtain a first quality difference according to the first output sample image and the second output sample image.

[0115] In this example implementation, a first sample variance may be obtained according to the first output sample image, a second sample variance may be obtained according to the second output sample image, and a first quality difference may be obtained according to the first sample variance and the second sample variance.

[0116] The first quality difference is used to measure the difference between the output result of the first enhancement model and the output result of the second enhancement model. For example, the first quality difference GAN_Map can be used to measure the difference between the GAN output result and the MSE output result.

[0117] The variances of the MSE output result and the GAN output result are calculated to obtain MSE_var and GAN_var respectively, and the calculation method of the first quality difference GAN_Map is: GAN_Map = (2*MSE_var*GAN_var) / (MSE_var*MSE_var+GAN_var*GAN_var).

[0118] Step S220: Obtain a second quality difference according to the second output sample image and the enhanced sample image.

[0119] In this example implementation, the second sample variance may be obtained according to the second output sample image, the enhanced sample variance may be obtained according to the enhanced sample image, and the second quality difference may be obtained according to the second sample variance and the enhanced sample variance.

[0120] The second quality difference is used to measure the difference between the output result of the second enhancement model and the enhanced sample image. For example, the second quality difference MSE_Map can be used to measure the difference between the MSE output result and the high-quality image.

[0121] The variance of the high-quality image is calculated to obtain HQ_var, and the second quality difference MSE_Map is calculated as: MSE_Map=(2*MSE_var*HQ_var) / (MSE_var*MSE_var+HQ_var*HQ_var).

[0122] Step S230: Obtain an output quality difference between the first enhancement model and the second enhancement model according to the first quality difference and the second quality difference.

[0123] The output quality difference between the first enhancement model and the second enhancement model is recorded as Map, and Map is calculated as: Map = (1-GAN_Map)*MSE_Map.

[0124] The output quality difference can be used as a means to improve the subjective quality. Its physical meaning is: if GAN_Map is small, it means that the MSE output is quite different from the GAN output, and there may be artifacts in the GAN output. At this time, if MSE_Map is large, it means that the MSE output is slightly different from the high-quality image, and the final (1–GAN_Map)*MSE_Map value is large, which can be used to constrain the subjective quality of the GAN output. On the contrary, if GAN_Map is large, it means that the GAN output is slightly different from the MSE output.

[0125] In step S140, a multi-order joint variance of the first enhancement model is obtained according to the input sample image and the first output sample image, and a quality constraint loss is obtained according to the multi-order joint variance and the output quality difference.

[0126] The multi-order joint variance of the first enhancement model can be obtained according to the input sample image and the first output sample image, and the difference between the image before and after enhancement by the first enhancement model can be measured by the multi-order joint variance of the first enhancement model. The multi-order joint variance can be calculated according to the first-order variance and the second-order variance of the input sample image and the output sample image of the first enhancement model.

[0127] In this example implementation, Figure 3As shown, obtaining the multi-order joint variance of the first enhancement model according to the input sample image and the first output sample image may specifically include the following steps:

[0128] Step S310: Calculate the first-order variance and the second-order variance of the input sample image, and calculate the first-order variance and the second-order variance of the first output sample image.

[0129] In this example implementation, Figure 4 As shown, calculating the first-order variance and second-order variance of the input sample image may include the following steps:

[0130] Step S410: Obtain the input image edge of the input sample image through the edge operator, and calculate the variance of the input image edge according to a calculation window of a preset size to obtain the first-order variance of the input sample image.

[0131] Use an edge operator, such as the Canny edge detection operator, to calculate the edge of the input sample image, and then calculate the variance of the edge using a calculation window of a preset size, such as a 3x3 window. This variance is the first-order variance.

[0132] Step S420: Calculate the second-order gradient of the input sample image by using a second-order differential operator, and obtain the variance of the second-order gradient of the input sample image according to the calculation window to obtain the second-order variance of the input sample image.

[0133] Use a second-order differential operator, such as the Laplace operator, to calculate the second-order gradient of the input sample image, and then calculate the variance of the edge with a calculation window of a set size, such as a 3x3 window. This variance is the second-order variance.

[0134] In this example implementation, Figure 5 As shown, calculating the first-order variance and the second-order variance of the first output sample image may specifically include the following steps:

[0135] Step S510: Obtain an output image edge of the first output sample image by using an edge operator, and calculate the variance of the output image edge according to a calculation window of a preset size to obtain a first-order variance of the first output sample image.

[0136] Step S520: Calculate the second-order gradient of the first output sample image by using a second-order differential operator, and obtain the variance of the second-order gradient of the first output sample image according to the calculation window to obtain the second-order variance of the first output sample image.

[0137] Method for calculating first-order variance and second-order variance of first output sample image and Figure 4 The method for calculating the first-order variance and second-order variance of the input sample image is similar and will not be repeated here.

[0138] Step S320: Obtain a multi-order joint variance of the first enhancement model according to the first-order variance and the second-order variance of the input sample image and the first-order variance and the second-order variance of the first output sample image.

[0139] According to the first-order variance and the second-order variance of the input sample image and the output sample image of the first enhancement model, the multi-order joint variance of the first enhancement model can be calculated.

[0140] In this example implementation, Figure 6 As shown, according to the first-order variance and the second-order variance of the input sample image, and the first-order variance and the second-order variance of the first output sample image, the multi-order joint variance of the first enhancement model is obtained, which may specifically include the following steps:

[0141] Step S610: Obtain a first-order sample variance ratio according to the ratio of the first-order variance of the first output sample image to the first-order variance of the input sample image.

[0142] According to the ratio of the first-order variance of the first output sample image to the first-order variance of the input sample image, a first-order sample variance ratio D1 can be obtained.

[0143] Step S620: Obtain a second-order sample variance ratio according to the ratio of the second-order variance of the first output sample image to the second-order variance of the input sample image.

[0144] According to the ratio of the second-order variance of the first output sample image to the second-order variance of the input sample image, the second-order sample variance ratio D2 can be obtained.

[0145] Step S630: Obtain the multi-order joint variance of the first enhanced model according to the first-order sample variance ratio and the second-order sample variance ratio.

[0146] The calculation method of the multi-order joint variance D is: D = (D1 x D1 + D2 x D2) / D2. The multi-order joint variance D can better measure the texture growth degree and clarity change degree of the output image of the first enhancement model.

[0147] When the multi-order joint variance D is large, the output image of the first enhancement model has a greater risk of texture artifacts and excessive sharpness. Therefore, the multi-order joint variance D of the first enhancement model can be combined with the output quality difference Map between the first enhancement model and the second enhancement model to obtain the quality constraint loss D x(1–GAN_Map)*MSE_Map.

[0148] In step S150, the model parameters of the first enhancement model are updated based on the quality constraint loss, and the image enhancement model is obtained according to the updated first enhancement model.

[0149] Finally, the quality constraint loss, i.e., D x(1–GAN_Map)*MSE_Map, is used as the final method to improve the subjective quality. It is used to update the model parameters of the first enhancement model and perform subjective quality constraint in the difference between the GAN output and the high-quality image.

[0150] like Figure 7 The figure is a flowchart of calculating the output quality difference between the MSE-guided model and the GAN-guided model in a specific implementation of the present disclosure, which is an example of the above steps in this example implementation. The specific steps of the flowchart are as follows:

[0151] The MSE output result is obtained through the MSE guided network, and the GAN output result is obtained through the GAN guided network. The difference between the MSE output result and the high-quality image is measured by MSE_Map, and the difference between the GAN output result and the MSE output result is measured by GAN_Map. Map is calculated based on MSE_Map and GAN_Map, Map = (1-GAN_Map)*MSE_Map.

[0152] By designing a subjective quality perception Map, combining the advantages of MSE output and GAN output and avoiding their disadvantages, the subjective quality can be effectively improved.

[0153] like Figure 8 The figure is a flowchart of performing subjective quality constraints on an image enhancement model in a specific implementation of the present disclosure, which is an example of the above steps in this example implementation. The specific steps of the flowchart are as follows:

[0154] Based on the first-order variance and second-order variance of the low-quality image and the first-order variance and second-order variance of the GAN output image, the multi-order joint variance D of the GAN-guided network is calculated. The multi-order joint variance D is combined with Map to obtain the quality constraint loss D x(1–GAN_Map)*MSE_Map, which is used for subjective quality constraint during the training process of the GAN-guided network.

[0155] On the other hand, this exemplary embodiment also provides an image processing method. Fig. 9 As shown, the above image processing method may include the following steps:

[0156] Step S910: Obtain the original image to be processed, and input the original image into a pre-trained image enhancement model.

[0157] Step S920: Perform image enhancement processing on the original image through the image enhancement model to obtain an enhanced image corresponding to the original image.

[0158] Among them, the image enhancement model can be Figures 1 to 6The training method of the image enhancement model in is obtained.

[0159] Performing image enhancement processing on the original image based on the image enhancement model in this example implementation can effectively improve the subjective quality, avoid the generation of image artifacts, enhance the texture details and clarity of the image, and repair low-quality signals in the image such as compression artifacts, blur, noise, etc., so that the output image effect is more natural and in line with the subjective perception of the human eye.

[0160] It should be noted that although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0161] Furthermore, the present disclosure also provides a training device for an image enhancement model. Fig.10 As shown, the training device of the image enhancement model may include a sample image acquisition module 1010, a sample image input module 1020, a quality difference determination module 1030, a constraint loss determination module 1040 and a model parameter update module 1050. Among them:

[0162] The sample image acquisition module 1010 is configured to acquire an input sample image of a first quality and an enhanced sample image of a second quality corresponding to the input sample image, wherein the second quality is higher than the first quality;

[0163] The sample image input module 1020 is configured to input the input sample image into a pre-trained first enhancement model to obtain a first output sample image, and input the input sample image into a pre-trained second enhancement model to obtain a second output sample image, wherein the first enhancement model is trained on the basis of the second enhancement model;

[0164] The quality difference determination module 1030 is configured to obtain an output quality difference between the first enhancement model and the second enhancement model according to the first output sample image, the second output sample image and the enhancement sample image;

[0165] The constraint loss determination module 1040 is configured to obtain a multi-order joint variance of the first enhancement model according to the input sample image and the first output sample image, and obtain a quality constraint loss according to the multi-order joint variance and the output quality difference;

[0166] The model parameter updating module 1050 is configured to update the model parameters of the first enhancement model based on the quality constraint loss, and obtain the image enhancement model according to the updated first enhancement model.

[0167] In some exemplary embodiments of the present disclosure, the quality difference determination module 1030 may include a first quality difference determination unit, a second quality difference determination unit, and a model output quality difference determination unit.

[0168] The first quality difference determining unit is configured to obtain a first quality difference according to the first output sample image and the second output sample image;

[0169] The second quality difference determining unit is configured to obtain a second quality difference according to the second output sample image and the enhanced sample image;

[0170] The model output quality difference determining unit is configured to obtain an output quality difference between the first enhanced model and the second enhanced model according to the first quality difference and the second quality difference.

[0171] In some exemplary embodiments of the present disclosure, the first quality difference determination unit may include a first sample variance calculation unit and a first quality difference calculation unit.

[0172] The first sample variance calculation unit is configured to obtain a first sample variance according to the first output sample image, and obtain a second sample variance according to the second output sample image;

[0173] The first quality difference calculation unit is configured to obtain a first quality difference according to the first sample variance and the second sample variance.

[0174] In some exemplary embodiments of the present disclosure, the second quality difference determination unit may include a second sample variance calculation unit and a second quality difference calculation unit.

[0175] The second sample variance calculation unit is configured to obtain a second sample variance according to the second output sample image, and obtain an enhanced sample variance according to the enhanced sample image;

[0176] The second quality difference calculation unit is configured to obtain a second quality difference according to the second sample variance and the enhanced sample variance.

[0177] In some exemplary embodiments of the present disclosure, the constraint loss determination module 1040 may include a multi-order variance calculation unit and a multi-order joint variance determination unit.

[0178] The multi-order variance calculation unit is configured to calculate the first-order variance and the second-order variance of the input sample image, and calculate the first-order variance and the second-order variance of the first output sample image;

[0179] The multi-order joint variance determination unit is configured to obtain the multi-order joint variance of the first enhancement model according to the first-order variance and the second-order variance of the input sample image and the first-order variance and the second-order variance of the first output sample image.

[0180] In some exemplary embodiments of the present disclosure, the multi-order variance calculation unit may include an input sample first-order variance calculation unit and an input sample second-order variance calculation unit.

[0181] The input sample first-order variance calculation unit is configured to obtain an input image edge of the input sample image through an edge operator, and calculate the variance of the input image edge according to a calculation window of a preset size to obtain a first-order variance of the input sample image;

[0182] The input sample second-order variance calculation unit is configured to calculate the second-order gradient of the input sample image through a second-order differential operator, and obtain the variance of the second-order gradient of the input sample image according to the calculation window to obtain the second-order variance of the input sample image.

[0183] In some exemplary embodiments of the present disclosure, the multi-order variance calculation unit may further include an output sample first-order variance calculation unit and an output sample second-order variance calculation unit.

[0184] The output sample first-order variance calculation unit is configured to obtain an output image edge of the first output sample image through an edge operator, and obtain a variance of the output image edge according to a calculation window of a preset size to obtain a first-order variance of the first output sample image;

[0185] The output sample second-order variance calculation unit is configured to calculate the second-order gradient of the first output sample image through a second-order differential operator, and obtain the variance of the second-order gradient of the first output sample image according to the calculation window to obtain the second-order variance of the first output sample image.

[0186] In some exemplary embodiments of the present disclosure, the multi-order joint variance determination unit may include a first-order sample variance ratio calculation unit, a second-order sample variance ratio calculation unit, and a multi-order joint variance calculation unit.

[0187] The first-order sample variance ratio calculation unit is configured to obtain the first-order sample variance ratio according to the ratio of the first-order variance of the first output sample image to the first-order variance of the input sample image;

[0188] The second-order sample variance ratio calculation unit is configured to obtain the second-order sample variance ratio according to the ratio of the second-order variance of the first output sample image to the second-order variance of the input sample image;

[0189] The multi-order joint variance calculation unit is configured to obtain the multi-order joint variance of the first enhanced model according to the first-order sample variance ratio and the second-order sample variance ratio.

[0190] Furthermore, the present disclosure also provides an image processing device. Fig.11As shown, the image processing device may include an original image input module 1110 and an image enhancement processing module 1120. Among them:

[0191] The original image input module 1110 can be used to obtain the original image to be processed, and input the original image into a pre-trained image enhancement model, wherein the image enhancement model is obtained by the training device of the image enhancement model as described above;

[0192] The image enhancement processing module 1120 can be used to perform image enhancement processing on the original image through an image enhancement model to obtain an enhanced image corresponding to the original image.

[0193] The specific details of each module / unit in the training device of the above-mentioned image enhancement model have been described in detail in the corresponding method embodiment part and will not be repeated here.

[0194] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0195] Fig.12 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present disclosure is shown.

[0196] It should be noted that Fig.12 The computer system 1200 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0197] like Fig.12 As shown, the computer system 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage part 1208 into a random access memory (RAM) 1203. Various programs and data required for system operation are also stored in the RAM 1203. The CPU 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0198] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1210 as needed, so that a computer program read therefrom is installed into the storage section 1208 as needed.

[0199] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 1209, and / or installed from a removable medium 1211. When the computer program is executed by a central processing unit (CPU) 1201, various functions defined in the system of the present disclosure are executed.

[0200] The exemplary embodiment of the present disclosure also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the information prompt method for reward redemption is implemented.

[0201] In one embodiment, the computer program product may be a tangible product containing a computer program, such as a computer-readable storage medium storing a computer program. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid-state drive (SSD), and the like. Exemplarily, the computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory, and the like.

[0202] In one embodiment, the computer program product may be an intangible product including a computer program. Exemplarily, the computer program product may be implemented as a virtual digital product, such as a digital file storing an executable file, an installation package, etc. of the computer program.

[0203] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0204] The computer program may be carried or transmitted via electrical, magnetic, optical, electromagnetic, infrared, or other signals. The electronic device may convert the signal carrying the computer program into a digital signal, thereby running the computer program. When the computer program is run on the electronic device, its code is used to enable the electronic device to execute (more specifically, the processor of the electronic device may execute) the method steps of various exemplary embodiments of the present disclosure, such as the information prompt method for reward redemption described above.

[0205] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0206] It should be noted that although several modules of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules described above can be concretized in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules to be concretized.

[0207] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0208] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for an image enhancement model, characterized in that: include: Acquire an input sample image of a first quality and an enhanced sample image of a second quality corresponding to the input sample image, wherein the second quality is higher than the first quality; Inputting the input sample image into a pre-trained first enhancement model to obtain a first output sample image, and inputting the input sample image into a pre-trained second enhancement model to obtain a second output sample image, wherein the first enhancement model is trained on the basis of the second enhancement model; Obtaining an output quality difference between the first enhancement model and the second enhancement model according to the first output sample image, the second output sample image, and the enhanced sample image; Obtaining a multi-order joint variance of the first enhancement model according to the input sample image and the first output sample image, and obtaining a quality constraint loss according to the multi-order joint variance and the output quality difference; The model parameters of the first enhancement model are updated based on the quality constraint loss, and an image enhancement model is obtained according to the updated first enhancement model.

2. The method for training an image enhancement model according to claim 1, characterized in that: The obtaining, according to the first output sample image, the second output sample image and the enhanced sample image, an output quality difference between the first enhanced model and the second enhanced model includes: Obtaining a first quality difference according to the first output sample image and the second output sample image; Obtaining a second quality difference according to the second output sample image and the enhanced sample image; An output quality difference between the first enhancement model and the second enhancement model is obtained according to the first quality difference and the second quality difference.

3. The training method of the image enhancement model according to claim 2, characterized in that: The obtaining a first quality difference according to the first output sample image and the second output sample image includes: Obtain a first sample variance according to the first output sample image, and obtain a second sample variance according to the second output sample image; The first quality difference is obtained according to the first sample variance and the second sample variance.

4. The method for training an image enhancement model according to claim 2, characterized in that: The obtaining a second quality difference according to the second output sample image and the enhanced sample image comprises: Obtain a second sample variance according to the second output sample image, and obtain an enhanced sample variance according to the enhanced sample image; The second quality difference is obtained according to the second sample variance and the enhanced sample variance.

5. The method for training an image enhancement model according to claim 1, characterized in that: The step of obtaining the multi-order joint variance of the first enhancement model according to the input sample image and the first output sample image includes: Calculating the first-order variance and the second-order variance of the input sample image, and calculating the first-order variance and the second-order variance of the first output sample image; A multi-order joint variance of the first enhancement model is obtained according to the first-order variance and the second-order variance of the input sample image and the first-order variance and the second-order variance of the first output sample image.

6. The method for training an image enhancement model according to claim 5, characterized in that: The calculating the first-order variance and the second-order variance of the input sample image comprises: Obtaining an input image edge of the input sample image by an edge operator, and calculating the variance of the input image edge according to a calculation window of a preset size to obtain a first-order variance of the input sample image; The second-order gradient of the input sample image is calculated by a second-order differential operator, and the variance of the second-order gradient of the input sample image is calculated according to the calculation window to obtain the second-order variance of the input sample image.

7. The method for training an image enhancement model according to claim 5, characterized in that: The calculating the first-order variance and the second-order variance of the first output sample image comprises: Obtaining an output image edge of the first output sample image by using an edge operator, and calculating a variance of the output image edge according to a calculation window of a preset size to obtain a first-order variance of the first output sample image; The second-order gradient of the first output sample image is calculated by a second-order differential operator, and the variance of the second-order gradient of the first output sample image is obtained according to the calculation window to obtain the second-order variance of the first output sample image.

8. The method for training an image enhancement model according to claim 5, characterized in that: The step of obtaining the multi-order joint variance of the first enhancement model according to the first-order variance and the second-order variance of the input sample image and the first-order variance and the second-order variance of the first output sample image comprises: Obtaining a first-order sample variance ratio according to a ratio of a first-order variance of the first output sample image to a first-order variance of the input sample image; Obtaining a second-order sample variance ratio according to a ratio of the second-order variance of the first output sample image to the second-order variance of the input sample image; The multi-order joint variance of the first enhanced model is obtained according to the first-order sample variance ratio and the second-order sample variance ratio.

9. An image processing method, characterized in that: include: Acquire an original image to be processed, and input the original image into a pre-trained image enhancement model, wherein the image enhancement model is obtained by the training method of the image enhancement model according to any one of claims 1 to 8; The original image is subjected to image enhancement processing by using the image enhancement model to obtain an enhanced image corresponding to the original image.

10. A training device for an image enhancement model, characterized in that: include: A sample image acquisition module is configured to acquire an input sample image of a first quality and an enhanced sample image of a second quality corresponding to the input sample image, wherein the second quality is higher than the first quality; A sample image input module is configured to input the input sample image into a pre-trained first enhancement model to obtain a first output sample image, and input the input sample image into a pre-trained second enhancement model to obtain a second output sample image, wherein the first enhancement model is trained on the basis of the second enhancement model; a quality difference determination module, configured to obtain an output quality difference between the first enhancement model and the second enhancement model according to the first output sample image, the second output sample image and the enhancement sample image; a constraint loss determination module, configured to obtain a multi-order joint variance of the first enhancement model according to the input sample image and the first output sample image, and obtain a quality constraint loss according to the multi-order joint variance and the output quality difference; The model parameter updating module is configured to update the model parameters of the first enhancement model based on the quality constraint loss, and obtain an image enhancement model according to the updated first enhancement model.

11. An image processing device, characterized in that: include: An original image input module, configured to obtain an original image to be processed, and input the original image into a pre-trained image enhancement model, wherein the image enhancement model is obtained by the training method of the image enhancement model according to any one of claims 1 to 8; The image enhancement processing module is configured to perform image enhancement processing on the original image through the image enhancement model to obtain an enhanced image corresponding to the original image.

12. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the training method of the image enhancement model as described in any one of claims 1 to 8, or the image processing method as described in claim 9.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the training method of the image enhancement model as described in any one of claims 1 to 8, or the image processing method as described in claim 9.

Images