Image enhancement method, device, model and training method thereof, and electronic device

Abstract

The embodiments of the present application disclose an image enhancement method, device, model and training method thereof, and electronic device, which perform multiple levels of downsampling on a first image of lower definition to obtain encoding features of multiple different sampling parameters; perform multiple levels of upsampling on the encoding features obtained by downsampling of the last level to obtain decoding features of multiple different sampling parameters; wherein the upsampling of each non-first level includes: fusing the decoding features obtained by upsampling of the previous level and the encoding features of the same sampling parameters to obtain fused features, and upsampling the fused features; the process of obtaining the fused features at at least one non-first level includes: fusing the decoding features obtained by upsampling of the previous level and the encoding features of the same sampling parameters through pre-learned fusion parameters; and obtaining a second image of higher definition based on the decoding features obtained by upsampling of the last level.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to an image enhancement method, device, model and training method thereof, and electronic equipment. Background Art

[0002] Image enhancement is a method to improve the visual effect of an image. Generally speaking, the main purpose of image enhancement is to improve the visual effect of an image and enhance its clarity.

[0003] The current image enhancement scheme uses the classic U-shaped network to enhance the low-quality original image to a high-quality image. However, the image enhancement effect of the classic U-shaped network is poor. In order to improve the image enhancement effect of the U-shaped network, a scheme proposes to add a low-resolution feature search module between the encoding module and the decoding module of the U-shaped network to extract the strong discriminative features required for enhancement, and then the decoding module decodes the strong discriminative features to obtain a high-quality image.

[0004] Although the image enhancement effect can be improved by adding a low-resolution feature search module, the scale of the low-resolution feature search module is relatively large, which significantly increases the computational complexity of the entire image enhancement network, causing the network speed to slow down and making network deployment difficult. Summary of the invention

[0005] The purpose of this application is to provide an image enhancement method, device, model and training method thereof, and electronic equipment, including the following technical solutions:

[0006] An image enhancement method, the method comprising:

[0007] Downsampling the first image at multiple levels to obtain encoding features of multiple different sampling parameters;

[0008] The coding features obtained by downsampling the last level are upsampled at multiple levels to obtain decoding features of multiple different sampling parameters; wherein the upsampling of each non-first level includes: fusing the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameters to obtain fused features, and upsampling the fused features; the process of obtaining the fused features at at least one non-first level includes: fusing the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameters through the pre-learned fusion parameters;

[0009] A second image is obtained based on the decoded features obtained by upsampling the last level; the definition of the second image is higher than that of the first image.

[0010] In a possible implementation, fusing the decoding features obtained by upsampling at the previous level and the encoding features of the same sampling parameters by using the pre-learned fusion parameters includes:

[0011] Based on the fusion parameter, the decoding feature obtained by upsampling at the previous level, and the encoding feature with the same sampling parameter, a first weight of the encoding feature having the same sampling parameter as the decoding feature obtained by upsampling at the previous level is calculated;

[0012] Based on the first weight, the decoding features obtained by upsampling at the previous level and the encoding features of the same sampling parameters are weightedly fused.

[0013] In a possible implementation manner, weighted fusion of the decoding features obtained by upsampling at the previous level and the encoding features of the same sampling parameter based on the first weight includes:

[0014] Obtaining, based on the first weight, a second weight of the decoded feature obtained by upsampling at the previous level;

[0015] Based on the first weight and the second weight, weighted calculation is performed on the decoding features obtained by upsampling of the previous layer and the encoding features of the same sampling parameters.

[0016] In a possible implementation manner, weighted fusion of the decoding features obtained by upsampling at the previous level and the encoding features of the same sampling parameter based on the first weight includes:

[0017] The decoding features obtained by upsampling the previous level and the encoding features of the same sampling parameters are summed to obtain an initial fusion feature;

[0018] Obtaining, based on the first weight, a second weight of the decoded feature obtained by upsampling at the previous level;

[0019] The initial fusion features and the decoding features obtained by upsampling at the previous level are weightedly calculated based on the first weight and the second weight.

[0020] In a possible implementation, downsampling a first image at multiple levels, upsampling a coding feature obtained by downsampling a last level at multiple levels, and obtaining a second image based on a decoding feature obtained by upsampling a last level includes:

[0021] Downsampling the first image at multiple levels by a first encoding module of an enhancement network;

[0022] The encoding features obtained by downsampling the last level are upsampled at multiple levels by the decoding module of the enhanced network, wherein the upsampling of each non-first level includes: obtaining a fusion feature obtained by fusing the decoding features obtained by upsampling of the previous level and the encoding features of the same sampling parameter by the fusion module of the enhanced network, and upsampling the fusion feature; the process of obtaining the fusion feature at at least one non-first level includes: the fusion module fuses the decoding features obtained by upsampling of the previous level and the encoding features of the same sampling parameter through the pre-learned fusion parameters;

[0023] The second image is obtained by processing the decoded features obtained by upsampling the last level through the output module of the enhancement network.

[0024] In a possible implementation, the enhanced network is trained in the following manner:

[0025] The initial network obtained by the second encoding module, the decoding module and the output module is unsupervisedly trained through the high-quality images in the first image set to obtain a pre-trained network; the first image set includes multiple low-quality images and high-quality images corresponding to each low-quality image; the second encoding module is used to perform multiple levels of downsampling on the input high-quality image to obtain encoding features of multiple different sampling parameters; based on the pre-trained network, the first encoding module and the fusion module are supervisedly trained through the first image set to obtain the trained first encoding module and fusion module; the trained first encoding module and fusion module, as well as the decoding module and the output module in the pre-trained network constitute the enhanced network.

[0026] An image enhancement model, comprising: a first encoding module, a decoding module, a fusion module and an output module;

[0027] The first encoding module is used to perform multiple-level downsampling on the first image to obtain encoding features of multiple different sampling parameters;

[0028] The decoding module is used to perform multiple-level upsampling on the coding features obtained by downsampling of the last level to obtain decoding features of multiple different sampling parameters; wherein the upsampling of each non-first level includes: obtaining a fusion feature obtained by the fusion module by fusing the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameter, and upsampling the fusion feature; the process of obtaining the fusion feature at least one non-first level includes: the fusion module fuses the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameter through the pre-learned fusion parameters;

[0029] The output module is used to obtain a second image based on the decoded features obtained by upsampling at the last level; the definition of the second image is higher than that of the first image.

[0030] In a possible implementation, when the fusion module fuses the decoding features obtained by upsampling at the previous level and the encoding features of the same sampling parameters thereof through the pre-learned fusion parameters, it is used to:

[0031] Based on the fusion parameter, the decoding feature obtained by upsampling at the previous level, and the encoding feature with the same sampling parameter, a first weight of the encoding feature having the same sampling parameter as the decoding feature obtained by upsampling at the previous level is calculated;

[0032] Based on the first weight, the decoding features obtained by upsampling at the previous level and the encoding features of the same sampling parameters are weightedly fused.

[0033] A model training method, used to train the image enhancement model described in any of the above, the model training method comprising:

[0034] The initial network obtained by the second encoding module, the decoding module and the output module is unsupervisedly trained by using high-quality images in the first image set to obtain a pre-trained network; the first image set includes a plurality of low-quality images and high-quality images corresponding to each low-quality image; the second encoding module is used to perform multiple levels of downsampling on the input high-quality image to obtain encoding features of multiple different sampling parameters;

[0035] Based on the pre-trained network, the first encoding module and the fusion module are supervisedly trained through the first image set to obtain the trained first encoding module and fusion module; the trained first encoding module and fusion module, as well as the decoding module and output module in the pre-trained network constitute the image enhancement model.

[0036] In a possible implementation, based on the pre-trained network, supervised training is performed on the first encoding module using the first image set, including:

[0037] For any low-quality image in the first image set, input the any low-quality image into the first encoding module to obtain encoding features of multiple different sampling parameters of the any low-quality image;

[0038] Inputting a high-quality image corresponding to any one of the low-quality images into a second encoding module in the pre-trained network to obtain encoding features of a plurality of different sampling parameters of the high-quality image corresponding to any one of the low-quality images;

[0039] With the goal of making the first difference in encoding features of any low-quality image and its corresponding high-quality image having the same sampling parameters smaller and smaller, the parameters of the first encoding module are updated.

[0040] An image enhancement device, comprising:

[0041] An encoding unit, configured to perform downsampling of the first image at multiple levels to obtain encoding features of multiple different sampling parameters;

[0042] A decoding unit is used to perform multiple-level upsampling on the coding features obtained by downsampling of the last level to obtain decoding features of multiple different sampling parameters; wherein the upsampling of each non-first level includes: obtaining a fusion unit to fuse the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameters to obtain a fusion feature, and upsampling the fusion feature; the process of obtaining the fusion feature at at least one non-first level includes: fusing the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameters through the pre-learned fusion parameters;

[0043] An output unit is used to obtain a second image based on the decoded features obtained by upsampling the last level; the definition of the second image is higher than the definition of the first image.

[0044] A model training device, used to train the image enhancement model as described in any one of the above items, the model training device comprising:

[0045] The first training unit is used to perform unsupervised training on the initial network obtained by the second encoding module, the decoding module and the output module through the high-quality images in the first image set to obtain a pre-trained network; the first image set includes a plurality of low-quality images and a high-quality image corresponding to each low-quality image; the second encoding module is used to perform multiple-level downsampling on the input high-quality image to obtain encoding features of multiple different sampling parameters; the structure of the second encoding module is the same as or different from the structure of the first encoding module;

[0046] The second training unit is used to perform supervised training on the first encoding module and the fusion module based on the pre-trained network through the first image set to obtain the trained first encoding module and fusion module; the trained first encoding module and fusion module, as well as the decoding module and output module in the pre-trained network constitute the enhanced network.

[0047] A readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the image enhancement method as described in any one of the above items and / or the various steps of the model training method as described in any one of the above items are implemented.

[0048] An electronic device, comprising:

[0049] Memory, used to store programs;

[0050] A processor is used to call and execute the program in the memory, and to implement the image enhancement method as described in any of the above items and / or implement the various steps of the model training method as described in any of the above items by executing the program.

[0051] It can be seen from the above scheme that the image enhancement method, device, model and training method thereof, electronic device and storage medium provided by the present application perform multiple levels of downsampling on a first image of lower definition to obtain coding features of multiple different sampling parameters; perform multiple levels of upsampling on the coding features obtained by downsampling of the last level to obtain decoding features of multiple different sampling parameters; wherein the upsampling of each non-first level includes: fusing the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameters to obtain fused features, and upsampling the fused features; the process of obtaining fused features at at least one non-first level includes: fusing the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameters through pre-learned fusion parameters; and obtaining a second image of higher definition based on the decoding features obtained by upsampling of the last level. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 A flowchart of an implementation of the image enhancement method provided in an embodiment of the present application;

[0054] Figure 2 A flowchart for implementing the fusion of the decoding features obtained by upsampling of the previous level and the encoding features of the same sampling parameters by using the pre-learned fusion parameters provided in the embodiment of the present application;

[0055] Figure 3 A flowchart for implementing weighted fusion of the decoding features obtained by upsampling at the previous level and the encoding features of the same sampling parameters based on the first weight provided in an embodiment of the present application;

[0056] Figure 4 Another implementation flow chart of weighted fusion of the decoding features obtained by upsampling at the previous level and the encoding features of the same sampling parameters based on the first weight provided in the embodiment of the present application;

[0057] Figure 5 A schematic diagram of a structure of an enhanced network provided in an embodiment of the present application;

[0058] Figure 6 Another structural diagram of an enhanced network provided in an embodiment of the present application;

[0059] Figure 7 A schematic diagram of another structure of an enhanced network provided in an embodiment of the present application;

[0060] Figure 8 A schematic diagram of the structure of an initial network provided in an embodiment of the present application;

[0061] Fig. 9 Another structural diagram of an initial network provided in an embodiment of the present application;

[0062] Fig.10 A flowchart for implementing supervised training of a first encoding module using a first image set based on a pre-trained network provided in an embodiment of the present application;

[0063] Fig.11 A flowchart for implementing supervised training of a fusion module using a first image set based on a pre-trained network provided in an embodiment of the present application;

[0064] Fig.12 A schematic diagram of the structure of an image enhancement device provided in an embodiment of the present application;

[0065] Fig.13 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0066] Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0067] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims and the above drawings are used to distinguish similar parts, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated herein. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0069] As mentioned above, in order to improve the image enhancement effect of the U-shaped network, one implementation method is to embed a low-resolution feature search module after the encoding module and before the decoding module of the U-shaped network to extract the strong discriminative features required for enhancement from the feature map of the minimum resolution output by the encoding module, and input the strong discriminative features into the decoding module for decoding processing. The low-resolution feature search module usually uses a convolution module with a large kernel, a nonlocal module, or a transformer module, and these modules often have a large amount of calculation. When deployed, they will also encounter problems such as softmax, layernorm, etc. that are difficult to quantize or slow calculation on the Neural Network Processing Unit (NPU) side.

[0070] In order to improve the processing speed of image enhancement and make the deployment more friendly, this application scheme is proposed.

[0071] The image enhancement method and device, image enhancement model, and model training method provided in the embodiments of the present application can be used in an electronic device having a processor capable of processing images, such as a CPU, GPU, etc.

[0072] The electronic device may be a terminal device or a server. The server may be a single server or a server cluster, or a cloud server, etc.

[0073] like Figure 1 As shown, a flowchart of an implementation of the image enhancement method provided in an embodiment of the present application may include:

[0074] Step S101: downsampling a first image at multiple levels to obtain encoding features of multiple different sampling parameters.

[0075] The first image is a lower quality image to be enhanced. The first image may be an RGB image, or a grayscale image, or a depth image, or an image in another format. The present application does not specifically limit the format of the first image. The first image may be an image captured by an image acquisition device, or may be an image edited by an image editor, or may be an image generated by AI, etc.

[0076] The input of the first downsampling level is the first image, and starting from the second downsampling level, the input of each downsampling level is the coded feature obtained by the previous downsampling level. That is, the input of the first downsampling level is the first image, and the input of the i-th (i=2, 3, ..., I; I is the total number of downsampling levels) downsampling level is the coded feature output by the i-1th downsampling level.

[0077] The sampling parameter of the coding feature may refer to the resolution or size of the coding feature. That is, the first image is downsampled at multiple levels to obtain coding features of multiple resolutions or multiple coding features of different sizes.

[0078] The coded features obtained at each downsampling level are a feature map of the first image, so downsampling at different levels obtains feature maps of different resolutions of the first image, or downsampling at different levels obtains feature maps of different sizes of the first image. Among the coded features obtained at two adjacent downsampling levels, the resolution or size of the coded features obtained at the latter downsampling level is smaller than the resolution or size of the coded features obtained at the former downsampling level.

[0079] Optionally, the first image may be downsampled at multiple levels by using any of the following downsampling methods: maximum pooling, convolution, mean pooling, etc. The downsampling methods used at different downsampling levels may be the same or different.

[0080] Step S102: performing upsampling at multiple levels on the coding features obtained by downsampling at the last level to obtain decoding features with multiple different sampling parameters.

[0081] The input of the first upsampling level is the encoded features output by the last downsampling level. In other words, the processing object of the first level of upsampling is the encoded features obtained by the last level of downsampling.

[0082] The upsampling of each non-first level includes: fusing the decoding features obtained by upsampling of the previous level and the encoding features of the same sampling parameters to obtain fused features, and upsampling the fused features; the process of obtaining fused features at at least one non-first level includes: fusing the decoding features obtained by upsampling of the previous level and the encoding features of the same sampling parameters through pre-learned fusion parameters.

[0083] Among the decoded features obtained at two adjacent upsampling levels, the resolution or size of the decoded features obtained at the latter upsampling level is greater than the resolution or size of the decoded features obtained at the former upsampling level.

[0084] In the present application, the number of upsampling layers is the same as the number of downsampling layers, wherein the sampling parameters of the coding features obtained by the j-th (j=1, 2, 3, ..., I-1) upsampling are the same as the sampling parameters of the coding features obtained by the Ij-th downsampling level.

[0085] The present application starts from the second upsampling level. Instead of upsampling the output of the previous upsampling level, the decoding features output by the previous upsampling level are first fused with the coding features having the same sampling parameters to obtain fused features, and the fused features are upsampled. When the coding features and decoding features having the same sampling parameters are fused, at least part of the fusion processes corresponding to the upsampling levels are obtained by fusing the coding features and decoding features having the same sampling parameters through pre-learned fusion parameters. In other words, only the fusion processes corresponding to part of the upsampling levels may use the pre-learned fusion parameters, or all the fusion processes corresponding to I-1 upsampling levels may fuse the coding features and decoding features having the same sampling parameters through pre-learned fusion parameters.

[0086] In the case where only the fusion processes corresponding to some upsampling levels use the pre-learned fusion parameters for fusion, the fusion processes corresponding to other upsampling levels may be to directly add the encoding features and the decoding features having the same sampling parameters.

[0087] Through the above-mentioned multiple levels of upsampling, the decoding features obtained at the last upsampling level have strong discriminative features required for image enhancement.

[0088] Optionally, the encoding features obtained by downsampling at the last level may be upsampled at multiple levels by any of the following upsampling methods: inverse maximum pooling, transposed convolution, inverse mean pooling, etc. The upsampling methods used at different upsampling levels may be the same or different.

[0089] Step S103: Obtain a second image based on the decoded features obtained by upsampling the last level. The definition of the second image is higher than that of the first image. The content in the second image is the same as the content in the first image.

[0090] Since the decoded features obtained by upsampling at the last level have strong discriminative power required for image enhancement, the quality of the second image is guaranteed to be higher than that of the first image.

[0091] The image enhancement method provided by the present application no longer uses a low-resolution feature search module, but after performing multiple levels of downsampling on a first image of lower quality to obtain multiple coding features with different sampling parameters, the coding features obtained by downsampling the last level are upsampled at multiple levels to obtain multiple decoding features with different sampling parameters; wherein the upsampling of each non-first level includes: fusing the decoding features obtained by upsampling of the previous level and the coding features with the same sampling parameters to obtain fused features, and upsampling the fused features; the process of obtaining the fused features at at least one non-first level includes: fusing the decoding features obtained by upsampling of the previous level and the coding features with the same sampling parameters through pre-learned fusion parameters, so that when obtaining features of sufficiently low-quality images, the generation of high-quality images is not interfered with by too many low-quality features, so that the decoding features obtained by upsampling of the last level have the strong discriminative features required for enhancement, and the process of fusing the coding features and decoding features with the same sampling parameters does not require complex calculations, so the amount of calculation is small, the processing speed of image enhancement is guaranteed, and it is easy to deploy.

[0092] In an optional embodiment, a flowchart for implementing the fusion of the decoding features obtained by upsampling at the previous level and the encoding features of the same sampling parameters by using the pre-learned fusion parameters is shown in FIG. Figure 2 As shown, it may include:

[0093] Step S201: Based on the fusion parameters, the decoding features obtained by upsampling at the previous level, and the encoding features with the same sampling parameters, a first weight of the encoding features with the same sampling parameters as the decoding features obtained by upsampling at the previous level is calculated.

[0094] The first weight can be calculated as follows:

[0095] Calculate the global feature score g: g=sigmoid(DownSample_r(e+d)* W );

[0096] Calculate the gated feature score s: s = sigmoid(UpSample_r(g)).

[0097] Among them, DownSample_r represents the downsampling operation using a window size of r, where max pooling, convolution, or average pooling can be used.

[0098] UpSample_r represents an upsampling operation with a window size of r. Here, you can use reverse maximum pooling (maxUnPooling), or transposed convolution, or reverse mean pooling (averageunpooling).

[0099] Sigmoid() is a threshold function used to map the variables in brackets to the range of [0,1].

[0100] e represents the encoding feature, and d represents the decoding feature; e and d have the same sampling parameters, that is, d represents the decoding feature obtained by upsampling at the previous level, and e is the encoding feature with the same sampling parameters as d.

[0101] W It is the pre-learned fusion parameter, which is a matrix with dimension h / r*w / r, where h is the height of the encoded feature e and w is the width of the encoded feature e; the height and width of the decoded feature d are the same as those of the encoded feature e.

[0102] The gated feature score s is the first weight of the encoded feature e.

[0103] Step S202: weighted fusion of the decoded features obtained by upsampling at the previous level and the encoded features of the same sampling parameters based on the first weight.

[0104] After obtaining the first weight s of the coding feature e, the first weight s is used to perform weighted fusion on the decoding feature d obtained by upsampling in the previous layer and the coding feature e with the same sampling parameters.

[0105] In an optional embodiment, a flowchart for implementing weighted fusion of the decoding features obtained by upsampling at the previous level and the encoding features of the same sampling parameters based on the first weight is as follows: Figure 3 As shown, it may include:

[0106] Step S301: obtaining a second weight of a decoding feature obtained by upsampling at a previous level based on the first weight.

[0107] Optionally, the second weight may be a difference between 1 and the first weight, that is, the second weight is 1-s.

[0108] Step S302: performing weighted calculation on the decoded features obtained by upsampling at the previous level and the encoded features of the same sampling parameters thereof based on the first weight and the second weight to obtain a fused feature.

[0109] In this embodiment, the fusion feature f is calculated in the following way: f = s*e+(1-s)*d.

[0110] In an optional embodiment, another implementation flowchart of weighted fusion of the decoding features obtained by upsampling at the previous level and the encoding features of the same sampling parameters based on the first weight is as follows: Figure 4 As shown, it may include:

[0111] Step S401: sum the decoded features obtained by upsampling at the previous level and the encoded features of the same sampling parameters to obtain initial fusion features.

[0112] That is to say, the initial fusion feature is: e+d.

[0113] Step S402: obtaining a second weight of the decoded feature obtained by upsampling at the previous level based on the first weight.

[0114] Optionally, the second weight may be a difference between 1 and the first weight, that is, the second weight is 1-s.

[0115] In the present application, step S401 may be executed first, and then step S402, or step S402 may be executed first, and then step S401, or both steps may be executed simultaneously. The present application does not make any specific limitation on the execution order of the two steps.

[0116] Step S403: performing weighted calculation on the initial fused features and the decoded features obtained by upsampling at the previous level based on the first weight and the second weight to obtain fused features.

[0117] In this embodiment, the fusion feature f is calculated in the following way: f=s*(e+d)+(1-s)*d.

[0118] In an optional embodiment, the process of performing multiple levels of downsampling on the first image, performing multiple levels of upsampling on the encoding features obtained by the downsampling of the last level, and obtaining the second image based on the decoding features obtained by the upsampling of the last level can be implemented by a pre-trained enhancement network (also referred to as an image enhancement model). Specifically:

[0119] The first image is downsampled at multiple levels through a first encoding module of the enhancement network.

[0120] The coding features obtained by downsampling the last level are upsampled at multiple levels through the decoding module of the enhanced network, wherein the upsampling of each non-first level includes: obtaining a fusion feature obtained by fusing the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameters by the fusion module of the enhanced network, and upsampling the fusion feature; the process of obtaining the fusion feature at at least one non-first level includes: the fusion module fuses the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameters through pre-learned fusion parameters.

[0121] The decoded features obtained by upsampling the last level are processed through the output module of the enhanced network to obtain the second image.

[0122] like Figure 5 As shown, a schematic diagram of a structure of an enhanced network provided in an embodiment of the present application may include:

[0123] An encoding module (referred to as the first encoding module for the convenience of distinction and description) 501, a decoding module 502, a fusion module 503 and an output module 504;

[0124] The first encoding module 501 is used to perform multiple-level downsampling on the first image to obtain encoding features of multiple different sampling parameters.

[0125] Figure 5 In the example shown, the first encoding module 501 performs 4 levels of downsampling on the first image. In practical applications, the first encoding module 501 may have more levels of downsampling, or may have 3 or 2 levels of downsampling, and the specific number of levels of downsampling is not specifically limited.

[0126] The decoding module 502 is used to decode the encoding features (such as Figure 5 4) performs upsampling at multiple levels to obtain decoding features of multiple different sampling parameters; wherein the upsampling of each non-first level includes: obtaining a fusion feature obtained by fusing the decoding features obtained by upsampling of the previous level and the encoding features of the same sampling parameter by the fusion module 503, and upsampling the fusion feature; the process of obtaining the fusion feature at at least one non-first level includes: the fusion module 503 fuses the decoding features obtained by upsampling of the previous level and the encoding features of the same sampling parameter through the pre-learned fusion parameters.

[0127] Figure 5 In the example shown, the decoding module 502 performs 4 levels of upsampling on the encoding features obtained by downsampling at the last level. In practical applications, the decoding module 502 may have more levels of upsampling, or may have 3 or 2 levels of upsampling. The specific number of upsampling levels is not specifically limited, as long as the number of upsampling levels is the same as the number of downsampling levels.

[0128] In the fusion module, the fusion parameters in different fusion gating modules may be the same or different, and the specific values ​​are determined according to the training of the enhanced network.

[0129] The output module 504 is used to decode the features obtained based on the upsampling of the last level (such as Figure 5 The decoding feature 4) obtains the second image; the clarity of the second image is higher than the clarity of the first image.

[0130] Figure 5 In the example shown, the fusion module fuses the coding features and decoding features with the same sampling parameters at three levels based on the fusion parameters. In other examples, only the coding features and decoding features with the same sampling parameters at two levels may be fused based on the fusion parameters, or only the coding features and decoding features with the same sampling parameters at one level may be fused based on the fusion parameters.

[0131] like Figure 6 FIG. 1 is another schematic diagram of the structure of the enhanced network provided in an embodiment of the present application. The structures of the first encoding module, decoding module and output module in this example are similar to those in FIG. Figure 5 In the example shown, the structures of the first encoding module, the decoding module and the output module are the same, and the difference lies in the structure of the fusion module. Figure 6 In the example shown, the fusion module 601 only fuses the encoding features and decoding features with the same sampling parameters at two levels based on the fusion parameters (the encoding feature 1 and the decoding feature 3 are fused only based on the fusion parameters learned by the fusion gating module 3, and the encoding feature 3 and the decoding feature 1 are fused based on the fusion parameters learned by the fusion gating module 1), and the encoding features and decoding features with the same sampling parameters at another level are not fused based on the learned fusion parameters, but are simply additively fused (i.e., the encoding feature 2 and the decoding feature 2 are directly fused by addition).

[0132] like Figure 7 FIG. 1 is another structural diagram of an enhanced network provided in an embodiment of the present application. The structures of the first encoding module, decoding module and output module in this example are similar to those in FIG. Figure 5 In the example shown, the structures of the first encoding module, the decoding module and the output module are the same, and the difference lies in the structure of the fusion module. Figure 7 In the example shown, the fusion module 701 only fuses the encoding features and decoding features with the same sampling parameters at one level based on the fusion parameters (the encoding feature 1 and the decoding feature 3 are fused only based on the fusion parameters learned by the fusion gating module 3), and the encoding features and decoding features with the same sampling parameters at the other two levels are not fused based on the learned fusion parameters, but are simply additively fused (i.e., the encoding feature 2 and the decoding feature 2 are directly fused in an additive manner, and the encoding feature 3 and the decoding feature 1 are directly fused in an additive manner).

[0133] In an optional embodiment, when the fusion module fuses the decoding features obtained by upsampling at the previous level and the encoding features of the same sampling parameters thereof through the pre-learned fusion parameters, it is used to:

[0134] Based on the fusion parameter, the decoded feature obtained by upsampling at the previous level, and the encoded feature with the same sampling parameter, the first weight of the encoded feature with the same sampling parameter as the decoded feature obtained by upsampling at the previous level is calculated. The specific implementation method can be referred to the above embodiment, which will not be repeated here.

[0135] Based on the first weight, the decoded features obtained by upsampling at the previous level and the encoded features of the same sampling parameters are weightedly fused. The specific implementation method can be found in the above embodiment, which will not be described here.

[0136] In the fusion module, each fusion gating module fuses the input encoding features and decoding features with the same sampling parameters based on the learned fusion parameters. For example, fusion gating module 1 fuses encoding feature 3 and decoding feature 1 based on the learned first fusion parameters; fusion gating module 2 fuses encoding feature 2 and decoding feature 2 based on the learned second fusion parameters; fusion gating module 3 fuses encoding feature 1 and decoding feature 3 based on the learned third fusion parameters.

[0137] Figure 5-7 In the process of each fusion module fusing the encoding features and decoding features with the same sampling parameters based on the learned fusion parameters, please refer to the aforementioned embodiment and will not be repeated here.

[0138] In an optional embodiment, the enhanced network can be trained in the following manner:

[0139] The initial network obtained by the second encoding module, the decoding module 502 and the output module 504 is unsupervisedly trained using high-quality images in the first image set to obtain a pre-trained network.

[0140] The first image set includes a plurality of low-quality images (with lower definition) and a high-quality image (with higher definition) corresponding to each low-quality image. The second encoding module is used to perform multiple levels of downsampling on the input high-quality image to obtain encoding features of multiple different sampling parameters. Among the encoding features of multiple different sampling parameters obtained by the second encoding module, there is at least one encoding feature having the same sampling parameters as at least one encoding feature of the encoding features of multiple different sampling parameters obtained by the first encoding module by performing multiple levels of downsampling on the input high-quality image.

[0141] like Figure 8As shown, it is a structural diagram of the initial network provided in an embodiment of the present application. The initial network includes: a second encoding module 801, a decoding module 802 and an output module 803; wherein, the structure of the second encoding module 801 and the first encoding module 501 can be the same or different, as long as the second encoding module 801 can perform multiple levels of downsampling processing on the input image and can obtain a coding feature with the same sampling parameters as at least one of the coding features output by the first encoding module 501. The decoding module 802 has the same structure as the decoding module 502, and the output module 803 has the same structure as the output module 504.

[0142] like Fig. 9 , which is another schematic diagram of the structure of the initial network provided in an embodiment of the present application. In this example, the second encoding module 801 has the same structure as the first encoding module 501, and both include 4 downsampling modules.

[0143] One way to implement unsupervised training of the initial network can be:

[0144] The high-quality image is input into the second encoding module 801.

[0145] The second encoding module 801 performs multiple-level downsampling on the high-quality image to obtain encoding features of multiple different sampling parameters.

[0146] The decoding module 502 performs multiple-level upsampling on the encoding features obtained by downsampling at the last level of the second encoding module 801 to obtain decoding features with multiple different sampling parameters.

[0147] The output module 504 processes the decoded features obtained by upsampling at the last level of the decoding module 502 to obtain a reconstructed high-quality image.

[0148] The parameters of the initial network are updated based on the difference between the reconstructed high-quality image and the original high-quality image (ie, the high-quality image input to the second encoding module 801).

[0149] Specifically, the parameters of the initial network can be updated with the goal of making the difference between the reconstructed high-quality image and the original high-quality image smaller and smaller (that is, compared with the high-quality image reconstructed using the initial network before the parameter update, the reconstructed image obtained by reconstructing the high-quality image using the initial network after the parameter update is closer to the original high-quality image).

[0150] The purpose of using high-quality images to perform unsupervised training on the initial network is to make full use of the features of high-quality images to obtain a better decoding module, making the enhanced image more realistic.

[0151] After obtaining the pre-trained network, based on the pre-trained network, supervised training is performed on the first encoding module (the first encoding module here may be the second encoding module in the pre-trained network or not) and the fusion module through the first image set to obtain the trained first encoding module and fusion module; the trained first encoding module and fusion module, as well as the decoding module and output module in the pre-trained network constitute an enhancement network, i.e., an image enhancement model.

[0152] Optionally, if the structure of the second encoding module is the same as that of the first encoding module, the fusion module and the second encoding module in the pre-trained network can be supervised trained based on the pre-trained network through the first image set to obtain a trained fusion module and the first encoding module (that is, the trained second encoding module is used as the first encoding module).

[0153] If the structure of the second encoding module is different from that of the first encoding module, the first encoding module and the fusion module can be supervisedly trained based on the pre-trained network through the first image set to obtain the trained first encoding module and the fusion module.

[0154] In an optional embodiment, a flowchart of implementing the supervised training of the first encoding module through the first image set based on the pre-trained network is as follows: Fig.10 As shown, it may include:

[0155] Step S1001: for any low-quality image in the first image set, input the any low-quality image into a first encoding module to obtain encoding features of multiple different sampling parameters of the any low-quality image output by the first encoding module.

[0156] The first encoding module here may be the second encoding module in the pre-trained network, or may be an encoding module different from the second encoding module in the pre-trained network.

[0157] Step S1002: inputting a high-quality image corresponding to any low-quality image into a second encoding module in a pre-trained network to obtain encoding features of multiple different sampling parameters of the high-quality image corresponding to any low-quality image output by the second encoding module.

[0158] In the case where the first encoding module is the second encoding module in the pre-trained network, the sampling parameters of the encoding features output by the second encoding module and the first encoding module in this step are the same.

[0159] In the case where the first encoding module is not the second encoding module in the pre-trained network, there are several possibilities:

[0160] The sampling parameters of the coding features output by the second coding module and the first coding module are the same.

[0161] Among the coding features output by the second coding module, some of the coding features have the same sampling parameters as at least some of the coding features output by the first coding module.

[0162] The coding features output by the second coding module have the same sampling parameters as some of the coding features in the coding features output by the first coding module.

[0163] Step S1003: updating the parameters of the first encoding module with the goal of reducing the first difference in encoding features of any low-quality image and its corresponding high-quality image having the same sampling parameters.

[0164] That is to say, compared with encoding any low-quality image using the first encoding module before parameter updating, the encoding features of multiple different sampling parameters obtained by downsampling any low-quality image at multiple levels using the first encoding module after parameter updating, and the encoding features obtained by downsampling any high-quality image corresponding to the low-quality image at multiple levels using the second encoding module, have a first difference in the encoding features with the same sampling parameters.

[0165] In this embodiment, the second encoding module in the pre-trained network is used as a teacher network (teacher network), and the first encoding module is used as a student network (student network) to perform distillation training on the first encoding module, so that the features obtained by the student network are closer to the features obtained by the teacher network.

[0166] In an optional embodiment, for a first coding feature of any sampling parameter of any low-quality image, and a second coding feature of any sampling parameter of a high-quality image corresponding to the any low-quality image (that is, the second coding feature and the first coding feature are coding features of different images with the same sampling parameter), a first difference between the first coding feature and the second coding feature is calculated as follows:

[0167] Obtain the absolute error between the first coding feature and the second coding feature (denoted as Loss1), the sum of square errors of the first classification result based on the first coding feature and the second classification result based on the second coding feature (denoted as Loss2), and the error at the maximum output of the first coding feature and the second coding feature (denoted as Loss3).

[0168] Optionally, the above three errors can be calculated as follows:

[0169] Loss1 = |f(teacher) – f(student)|1

[0170] Loss2 = |vgg(f(teacher))−vgg(f(student))|2

[0171] Loss3 = |softmax(f(teacher)×f(teacher).T)− softmax(f(student)×f(student).T)|2

[0172] Where f(teacher) represents the second encoding feature, and f(student) represents the first encoding feature. vgg represents the pre-trained vgg network for image classification, vgg(f(teacher)) represents the second classification result obtained by the vgg network based on the second encoding feature, and vgg(f(student)) represents the first classification result obtained by the vgg network based on the first encoding feature. f(teacher).T represents the transposed matrix of the second encoding feature, and f(student).T represents the transposed matrix of the first encoding feature.

[0173] The above absolute error, squared error and error at the maximum output are weighted and summed to obtain the first difference (denoted as Loss) between the first coding feature and the second coding feature.

[0174] The first difference Loss can be calculated as follows:

[0175] Loss=λ1×Loss1+λ2×Loss2+λ3×Loss3

[0176] Among them, λ1, λ2 and λ3 are the weights of the three errors, and the three weights are hyperparameters, that is, they are pre-set.

[0177] Optionally, the weight of the absolute error is negatively correlated with the downsampling level of the first coding feature. That is, the lower the downsampling level, the greater the corresponding absolute error weight; correspondingly, the larger the size (or resolution) of the first coding feature, the greater the corresponding absolute error weight.

[0178] Optionally, the weight of the sum of squared errors is positively correlated with the downsampling level of the first coding feature. That is, the higher the downsampling level, the greater the corresponding weight of the sum of squared errors; correspondingly, the smaller the size (or resolution) of the first coding feature, the greater the corresponding weight of the sum of squared errors.

[0179] In an optional embodiment, after the first coding feature is trained, a flowchart for implementing supervised training of the fusion module through the first image set based on the pre-trained network is as follows: Fig.11 As shown, it may include:

[0180] Step S1101: downsampling any low-quality image at multiple levels is performed through the trained first encoding module to obtain encoding features of multiple different sampling parameters.

[0181] The trained first encoding module refers to the first encoding module obtained through the aforementioned distillation training.

[0182] Step S1102: Perform multiple-level upsampling of the coding features obtained by downsampling the last level through the trained decoding module to obtain decoding features of multiple sampling parameters. The upsampling of each non-first level includes: obtaining a fusion feature obtained by fusing the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameters by the fusion module, and upsampling the fusion feature; the process of obtaining the fusion feature of at least one non-first level includes: the fusion module fuses the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameters through the fusion parameters.

[0183] The process of obtaining the fusion features by the fusion module can be found in the above-mentioned embodiment and will not be described again here.

[0184] The trained decoding module refers to the decoding module in the pre-trained network obtained by training the initial network as mentioned above.

[0185] Step S1103: Obtain an enhanced image corresponding to any of the above low-quality images based on the decoded features obtained by upsampling of the last level through the trained output module.

[0186] The trained output module refers to the output module in the pre-trained network obtained by training the initial network as mentioned above.

[0187] Step S1104: updating the parameters of the fusion module with the goal of reducing the second difference between the enhanced image corresponding to any of the low-quality images and the high-quality image corresponding to the low-quality image.

[0188] The parameters of the fusion module include the above fusion parameters.

[0189] The second difference between the enhanced image corresponding to any of the above-mentioned low-quality images and the high-quality image corresponding to the any of the low-quality images becomes smaller and smaller means that: compared with the image enhancement performed on any of the low-quality images based on the fusion module before the parameter update, the enhanced image obtained by performing image enhancement on any of the low-quality images based on the fusion module after the parameter update is closer to the high-quality image corresponding to the any of the low-quality images.

[0190] When training the fusion module, the parameters of the first encoding module, decoding module and output module are frozen.

[0191] Corresponding to the method embodiment, the present application also provides an image enhancement device. A structural schematic diagram of the image enhancement device provided by the present application is as follows Fig.12 As shown, it may include:

[0192] Encoding unit 1201, decoding unit 1202, fusion unit 1203 and output unit 1204;

[0193] The encoding unit 1201 is used to perform downsampling of the first image at multiple levels to obtain encoding features of multiple different sampling parameters;

[0194] The decoding unit 1202 is used to perform multiple-level upsampling on the coding features obtained by downsampling of the last level to obtain decoding features of multiple different sampling parameters; wherein the upsampling of each non-first level includes: obtaining a fusion unit 1203 to fuse the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameters to obtain a fusion feature, and upsampling the fusion feature; the process of obtaining the fusion feature at at least one non-first level includes: fusing the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameters through the pre-learned fusion parameters;

[0195] The output unit 1204 is used to obtain a second image based on the decoded features obtained by upsampling the last level; the definition of the second image is higher than the definition of the first image.

[0196] The image enhancement device provided in the embodiment of the present application performs multiple levels of downsampling on a first image of lower definition to obtain multiple coding features of different sampling parameters; performs multiple levels of upsampling on the coding features obtained by downsampling of the last level to obtain multiple decoding features of different sampling parameters; wherein the upsampling of each non-first level includes: fusing the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameters to obtain fused features, and upsampling the fused features; the process of obtaining the fused features at at least one non-first level includes: fusing the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameters through pre-learned fusion parameters; and obtaining a second image of higher definition based on the decoding features obtained by upsampling of the last level. By using the pre-learned fusion parameters to fuse the decoding features and encoding features with the same sampling parameters and then upsampling them, it is possible to obtain features of sufficiently low-quality images without having too many low-quality features interfere with the generation of high-quality images, so that the decoding features obtained by upsampling at the last level have the strong discriminative features required for enhancement. Moreover, the feature fusion process does not require complex calculations. Therefore, the image enhancement process has a small amount of calculation, which ensures the processing speed of image enhancement and is easy to deploy.

[0197] In an optional embodiment, when the fusion unit 1203 fuses the decoding features obtained by upsampling at the previous level and the encoding features of the same sampling parameters through the pre-learned fusion parameters, it is used to:

[0198] Based on the fusion parameter, the decoding feature obtained by upsampling at the previous level, and the encoding feature with the same sampling parameter, a first weight of the encoding feature having the same sampling parameter as the decoding feature obtained by upsampling at the previous level is calculated;

[0199] Based on the first weight, the decoding features obtained by upsampling at the previous level and the encoding features of the same sampling parameters are weightedly fused.

[0200] In an optional embodiment, when the fusion unit 1203 performs weighted fusion on the decoding features obtained by upsampling at the previous level and the encoding features of the same sampling parameters based on the first weight, it is used to:

[0201] Obtaining, based on the first weight, a second weight of the decoded feature obtained by upsampling at the previous level;

[0202] Based on the first weight and the second weight, weighted calculation is performed on the decoding features obtained by upsampling of the previous layer and the encoding features of the same sampling parameters.

[0203] In an optional embodiment, when the fusion unit 1203 performs weighted fusion on the decoding features obtained by upsampling at the previous level and the encoding features of the same sampling parameters based on the first weight, it is used to:

[0204] The decoding features obtained by upsampling the previous level and the encoding features of the same sampling parameters are summed to obtain an initial fusion feature;

[0205] Obtaining, based on the first weight, a second weight of the decoded feature obtained by upsampling at the previous level;

[0206] The initial fusion features and the decoding features obtained by upsampling at the previous level are weightedly calculated based on the first weight and the second weight.

[0207] In an optional embodiment, the image enhancement device performs multiple levels of downsampling on the first image through the enhancement network, performs multiple levels of upsampling on the encoding features obtained by the downsampling of the last level, and obtains the second image based on the decoding features obtained by the upsampling of the last level, specifically including:

[0208] Downsampling the first image at multiple levels by a first encoding module of an enhancement network;

[0209] The encoding features obtained by downsampling the last level are upsampled at multiple levels by the decoding module of the enhanced network, wherein the upsampling of each non-first level includes: obtaining a fusion feature obtained by fusing the decoding features obtained by upsampling of the previous level and the encoding features of the same sampling parameter by the fusion module of the enhanced network, and upsampling the fusion feature; the process of obtaining the fusion feature at at least one non-first level includes: the fusion module fuses the decoding features obtained by upsampling of the previous level and the encoding features of the same sampling parameter through the pre-learned fusion parameters;

[0210] The second image is obtained by processing the decoded features obtained by upsampling the last level through the output module of the enhancement network.

[0211] In an optional embodiment, the enhanced network is trained in the following manner:

[0212] The initial network obtained by the second encoding module, the decoding module and the output module is unsupervisedly trained by using high-quality images in the first image set to obtain a pre-trained network; the first image set includes multiple low-quality images and high-quality images corresponding to each low-quality image; the second encoding module is used to perform multiple levels of downsampling on the input high-quality image to obtain encoding features of multiple different sampling parameters;

[0213] Based on the pre-trained network, the first encoding module and the fusion module are supervisedly trained through the first image set to obtain the trained first encoding module and fusion module; the trained first encoding module and fusion module, as well as the decoding module and the output module in the pre-trained network constitute the enhanced network.

[0214] The specific implementation method can be found in the above-mentioned embodiment, which will not be described again here.

[0215] Corresponding to the method embodiment, the present application also provides a model training device for training the aforementioned image enhancement model. A structural schematic diagram of the model training device provided by the present application is shown in FIG. Fig.13 As shown, it may include: a first training unit 1301 and a second training unit 1302;

[0216] The first training unit 1301 is used to perform unsupervised training on the initial network obtained by the second encoding module, the decoding module and the output module through the high-quality images in the first image set to obtain a pre-trained network; the first image set includes multiple low-quality images and high-quality images corresponding to each low-quality image; the second encoding module is used to perform multiple levels of downsampling on the input high-quality image to obtain encoding features of multiple different sampling parameters; the structure of the second encoding module is the same as or different from the structure of the first encoding module;

[0217] The second training unit 1302 is used to perform supervised training on the first encoding module and the fusion module based on the pre-trained network through the first image set to obtain a trained first encoding module and a fusion module; the trained first encoding module and fusion module, as well as the decoding module and the output module in the pre-trained network constitute the enhanced network.

[0218] In an optional embodiment, when the second training unit 1302 performs supervised training on the first encoding module through the first image set based on the pre-trained network, it is used to:

[0219] For any low-quality image in the first image set, input the any low-quality image into the first encoding module to obtain encoding features of multiple different sampling parameters of the any low-quality image;

[0220] Inputting a high-quality image corresponding to any one of the low-quality images into a second encoding module in the pre-trained network to obtain encoding features of a plurality of different sampling parameters of the high-quality image corresponding to any one of the low-quality images;

[0221] With the goal of making the first difference in encoding features of any low-quality image and its corresponding high-quality image having the same sampling parameters smaller and smaller, the parameters of the first encoding module are updated.

[0222] The specific implementation method can be found in the above-mentioned embodiment, which will not be described again here.

[0223] Corresponding to the method embodiment, the present application also provides an electronic device, a structural schematic diagram of the electronic device is shown as follows Fig.14 As shown, it may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4.

[0224] In the embodiment of the present application, the number of the processor 1 , the communication interface 2 , the memory 3 , and the communication bus 4 is at least one, and the processor 1 , the communication interface 2 , and the memory 3 communicate with each other through the communication bus 4 .

[0225] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0226] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), etc., for example, at least one disk memory.

[0227] The memory 3 stores a program, and the processor 1 can call the program stored in the memory 3;

[0228] The program is used to: perform multiple levels of downsampling on a first image to obtain multiple coding features of different sampling parameters; perform multiple levels of upsampling on the coding features obtained by downsampling of the last level to obtain multiple decoding features of different sampling parameters; wherein the upsampling of each non-first level includes: fusing the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameters to obtain fused features, and upsampling the fused features; the process of obtaining the fused features at at least one non-first level includes: fusing the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameters through pre-learned fusion parameters; obtaining a second image based on the decoding features obtained by upsampling of the last level; the definition of the second image is higher than that of the first image;

[0229] Alternatively, the program is used to: perform unsupervised training on an initial network obtained by the second encoding module, the decoding module and the output module through high-quality images in a first image set to obtain a pre-trained network; the first image set includes multiple low-quality images, and high-quality images corresponding to each low-quality image; the second encoding module is used to perform multiple levels of downsampling on the input high-quality image to obtain encoding features of multiple different sampling parameters; based on the pre-trained network, supervised training is performed on the first encoding module and the fusion module through the first image set to obtain trained first encoding module and fusion module; the trained first encoding module and fusion module, as well as the decoding module and output module in the pre-trained network constitute the image enhancement model.

[0230] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0231] The embodiment of the present application also provides a storage medium, which can store a program suitable for execution by a processor;

[0232] The program is used to: perform multiple levels of downsampling on a first image to obtain multiple coding features of different sampling parameters; perform multiple levels of upsampling on the coding features obtained by downsampling of the last level to obtain multiple decoding features of different sampling parameters; wherein the upsampling of each non-first level includes: fusing the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameters to obtain fused features, and upsampling the fused features; the process of obtaining the fused features at at least one non-first level includes: fusing the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameters through pre-learned fusion parameters; obtaining a second image based on the decoding features obtained by upsampling of the last level; the definition of the second image is higher than that of the first image;

[0233] Alternatively, the program is used to: perform unsupervised training on an initial network obtained by the second encoding module, the decoding module and the output module through high-quality images in a first image set to obtain a pre-trained network; the first image set includes multiple low-quality images, and high-quality images corresponding to each low-quality image; the second encoding module is used to perform multiple levels of downsampling on the input high-quality image to obtain encoding features of multiple different sampling parameters; based on the pre-trained network, supervised training is performed on the first encoding module and the fusion module through the first image set to obtain trained first encoding module and fusion module; the trained first encoding module and fusion module, as well as the decoding module and output module in the pre-trained network constitute the image enhancement model.

[0234] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0235] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0236] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0237] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0238] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0239] It should be understood that in the embodiments of the present application, the various embodiments and features can be combined with each other to solve the aforementioned technical problems.

[0240] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0241] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image enhancement method, the image enhancement method is implemented based on a pre-trained image enhancement model, the method comprising: Downsampling the first image at multiple levels to obtain encoding features of multiple different sampling parameters; The coding features obtained by downsampling the last level are upsampled at multiple levels to obtain decoding features of multiple different sampling parameters; wherein the upsampling of each non-first level includes: fusing the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameters to obtain fused features, and upsampling the fused features; the process of obtaining the fused features at at least one non-first level includes: fusing the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameters through the pre-learned fusion parameters; Obtaining a second image based on the decoded features obtained by upsampling the last level; the definition of the second image is higher than the definition of the first image; The training method of the image enhancement model includes: The initial network obtained by the second encoding module, the decoding module and the output module is unsupervisedly trained by using high-quality images in the first image set to obtain a pre-trained network; the first image set includes a plurality of low-quality images and high-quality images corresponding to each low-quality image; the second encoding module is used to perform multiple levels of downsampling on the input high-quality image to obtain encoding features of multiple different sampling parameters; Based on the pre-trained network, the first encoding module and the fusion module are supervisedly trained through the first image set to obtain the trained first encoding module and the fusion module; the trained first encoding module and the fusion module, as well as the decoding module and the output module in the pre-trained network constitute the image enhancement model.

2. According to the method of claim 1, the step of fusing the decoded features obtained by upsampling at the previous level and the encoded features of the same sampling parameters by using the pre-learned fusion parameters comprises: Based on the fusion parameter, the decoding feature obtained by upsampling at the previous level, and the encoding feature with the same sampling parameter, a first weight of the encoding feature having the same sampling parameter as the decoding feature obtained by upsampling at the previous level is calculated; Based on the first weight, the decoding features obtained by upsampling at the previous level and the encoding features of the same sampling parameters are weightedly fused.

3. The method according to claim 2, wherein the weighted fusion of the decoding features obtained by upsampling at the previous level and the encoding features of the same sampling parameters based on the first weight comprises: Obtaining, based on the first weight, a second weight of the decoded feature obtained by upsampling at the previous level; Based on the first weight and the second weight, weighted calculation is performed on the decoding features obtained by upsampling of the previous layer and the encoding features of the same sampling parameters.

4. The method according to claim 2, wherein the weighted fusion of the decoding features obtained by upsampling at the previous level and the encoding features of the same sampling parameters based on the first weight comprises: The decoding features obtained by upsampling the previous level and the encoding features of the same sampling parameters are summed to obtain an initial fusion feature; Obtaining, based on the first weight, a second weight of the decoded feature obtained by upsampling at the previous level; The initial fusion features and the decoding features obtained by upsampling at the previous level are weightedly calculated based on the first weight and the second weight.

5. According to the method of claim 1, the process of downsampling the first image at multiple levels, upsampling the encoding features obtained by downsampling at the last level at multiple levels, and obtaining the second image based on the decoding features obtained by upsampling at the last level comprises: Downsampling the first image at multiple levels by a first encoding module of an enhancement network; The encoding features obtained by downsampling the last level are upsampled at multiple levels by the decoding module of the enhanced network, wherein the upsampling of each non-first level includes: obtaining a fusion feature obtained by fusing the decoding features obtained by upsampling of the previous level and the encoding features of the same sampling parameter by the fusion module of the enhanced network, and upsampling the fusion feature; the process of obtaining the fusion feature at at least one non-first level includes: the fusion module fuses the decoding features obtained by upsampling of the previous level and the encoding features of the same sampling parameter through the pre-learned fusion parameters; The second image is obtained by processing the decoded features obtained by upsampling the last level through the output module of the enhancement network.

6. According to the method of claim 5, the enhanced network is trained in the following manner: The initial network obtained by the second encoding module, the decoding module and the output module is unsupervisedly trained by using high-quality images in the first image set to obtain a pre-trained network; the first image set includes a plurality of low-quality images and high-quality images corresponding to each low-quality image; the second encoding module is used to perform multiple levels of downsampling on the input high-quality image to obtain encoding features of multiple different sampling parameters; Based on the pre-trained network, the first encoding module and the fusion module are supervisedly trained through the first image set to obtain the trained first encoding module and fusion module; the trained first encoding module and fusion module, as well as the decoding module and the output module in the pre-trained network constitute the enhanced network.

7. An image enhancement model, comprising: A first encoding module, a decoding module, a fusion module and an output module; The first encoding module is used to perform multiple-level downsampling on the first image to obtain encoding features of multiple different sampling parameters; The decoding module is used to perform multiple-level upsampling on the coding features obtained by downsampling of the last level to obtain decoding features of multiple different sampling parameters; wherein the upsampling of each non-first level includes: obtaining a fusion feature obtained by the fusion module by fusing the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameter, and upsampling the fusion feature; the process of obtaining the fusion feature at least one non-first level includes: the fusion module fuses the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameter through the pre-learned fusion parameters; The output module is used to obtain a second image based on the decoded features obtained by upsampling the last level; the definition of the second image is higher than the definition of the first image; The model training method comprises: The initial network obtained by the second encoding module, the decoding module and the output module is unsupervisedly trained by using high-quality images in the first image set to obtain a pre-trained network; the first image set includes a plurality of low-quality images and high-quality images corresponding to each low-quality image; the second encoding module is used to perform multiple levels of downsampling on the input high-quality image to obtain encoding features of multiple different sampling parameters; Based on the pre-trained network, the first encoding module and the fusion module are supervisedly trained through the first image set to obtain the trained first encoding module and the fusion module; the trained first encoding module and the fusion module, as well as the decoding module and the output module in the pre-trained network constitute the image enhancement model.

8. The image enhancement model according to claim 7, when the fusion module fuses the decoded features obtained by upsampling at the previous level and the encoded features of the same sampling parameters thereof through the pre-learned fusion parameters, is used to: Based on the fusion parameter, the decoding feature obtained by upsampling at the previous level, and the encoding feature with the same sampling parameter, a first weight of the encoding feature having the same sampling parameter as the decoding feature obtained by upsampling at the previous level is calculated; Based on the first weight, the decoding features obtained by upsampling at the previous level and the encoding features of the same sampling parameters are weightedly fused.

9. A model training method for training the image enhancement model according to any one of claims 7 to 8, the model training method comprising: The initial network obtained by the second encoding module, the decoding module and the output module is unsupervisedly trained by using high-quality images in the first image set to obtain a pre-trained network; the first image set includes a plurality of low-quality images and high-quality images corresponding to each low-quality image; the second encoding module is used to perform multiple levels of downsampling on the input high-quality image to obtain encoding features of multiple different sampling parameters; Based on the pre-trained network, the first encoding module and the fusion module are supervisedly trained through the first image set to obtain the trained first encoding module and fusion module; the trained first encoding module and fusion module, as well as the decoding module and output module in the pre-trained network constitute the image enhancement model.

10. The method according to claim 9, based on the pre-trained network, performing supervised training on the first encoding module through the first image set, comprising: For any low-quality image in the first image set, input the any low-quality image into the first encoding module to obtain encoding features of multiple different sampling parameters of the any low-quality image; Inputting a high-quality image corresponding to any one of the low-quality images into a second encoding module in the pre-trained network to obtain encoding features of a plurality of different sampling parameters of the high-quality image corresponding to any one of the low-quality images; With the goal of making the first difference in encoding features of any low-quality image and its corresponding high-quality image having the same sampling parameters smaller and smaller, the parameters of the first encoding module are updated.

11. An image enhancement device, the image enhancement device is implemented based on a pre-trained image enhancement model, the device comprising: An encoding unit, configured to perform downsampling of the first image at multiple levels to obtain encoding features of multiple different sampling parameters; A decoding unit is used to perform multiple-level upsampling on the coding features obtained by downsampling of the last level to obtain decoding features of multiple different sampling parameters; wherein the upsampling of each non-first level includes: obtaining a fusion unit to fuse the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameters to obtain a fusion feature, and upsampling the fusion feature; the process of obtaining the fusion feature of at least one non-first level includes: fusing the decoding features obtained by upsampling of the previous level and the coding features of the same sampling parameters through the pre-learned fusion parameters; An output unit, configured to obtain a second image based on the decoded features obtained by upsampling the last level; the definition of the second image is higher than the definition of the first image; The training method of the image enhancement model includes: The initial network obtained by the second encoding module, the decoding module and the output module is unsupervisedly trained by using high-quality images in the first image set to obtain a pre-trained network; the first image set includes a plurality of low-quality images and high-quality images corresponding to each low-quality image; the second encoding module is used to perform multiple levels of downsampling on the input high-quality image to obtain encoding features of multiple different sampling parameters; Based on the pre-trained network, the first encoding module and the fusion module are supervisedly trained through the first image set to obtain the trained first encoding module and the fusion module; the trained first encoding module and the fusion module, as well as the decoding module and the output module in the pre-trained network constitute the image enhancement model.

12. An electronic device comprising: Memory, used to store programs; A processor is used to call and execute the program in the memory, and to implement the image enhancement method as described in any one of claims 1 to 6 and / or implement the various steps of the model training method as described in claim 9 or 10 by executing the program.

Images