IMAGE ENHANCEMENT METHOD, ELECTRONIC DEVICE AND STORAGE MEDIUM

Abstract

The present disclosure provides an image enhancement method, an electronic device, and a storage medium. The method comprises performing a multi-stage sampling rate reduction on a first image to obtain multiple encoding features with different sampling rate parameters; performing a multi-stage sampling rate increase on an encoding feature obtained by sampling rate reduction at a final stage to obtain multiple decoding features with different sampling rate parameters; and obtaining a second image based on a decoding feature obtained by sampling rate increase at a final stage, wherein the resolution of the second image is higher than the resolution of the first image.The sampling rate increase at each non-first stage may involve merging a decoding feature obtained by sampling rate increase at a previous stage with an encoding feature of the same sampling rate parameter to obtain a merged feature, and performing a sampling rate increase on the merged feature.

Description

REFERENCE TO RELATED REGISTRATION

[0001] The present disclosure claims priority from Chinese patent application No. 202411563375.9, filed on November 4, 2024, the entire contents of which are hereby incorporated by reference. TECHNICAL AREA

[0002] The present disclosure relates generally to the field of image processing technology and in particular to an image enhancement method, an image enhancement device, an image enhancement model and a training method therefor, as well as an electronic device. TECHNICAL BACKGROUND

[0003] Image enhancement is a process for improving visual image quality. The main purpose of image enhancement is to improve visual quality and image resolution (i.e., clarity).

[0004] Current image enhancement solutions can use a classic U-shaped network to improve low-quality original images into high-quality images. However, the image enhancement performance of the classic U-shaped network can be insufficient. To improve the image enhancement performance of the U-shaped network, a low-resolution feature search module can be added between the encoding and decoding modules of the U-shaped network to extract highly distinctive features needed for enhancement. Furthermore, the decoding module can decode these highly distinctive features to produce high-quality images.

[0005] Adding the low-resolution feature search module can improve image enhancement performance. However, the low-resolution feature search module can be comparatively large, which can significantly increase the computational complexity of the entire enhancement network, potentially leading to slow network speeds and difficult network deployment. OVERVIEW OF THE INVENTION

[0006] One aspect of the present disclosure provides an image enhancement method. The image enhancement method comprises performing a multi-stage sampling rate reduction on a first image to obtain multiple encoding features with different sampling rate parameters; performing a multi-stage sampling rate increase on an encoding feature obtained by sampling rate reduction at a final stage to obtain multiple decoding features with different sampling rate parameters; and obtaining a second image based on a decoding feature obtained by sampling rate increase at a final stage, wherein the resolution of the second image is higher than the resolution of the first image.The sampling rate increase at each non-first stage comprises merging a decoding feature obtained by sampling rate increase at a previous stage with an encoding feature of the same sampling rate parameter to obtain a merged feature, and performing a sampling rate increase on the merged feature; and a process of obtaining a merged feature at at least one non-first stage comprises merging a decoding feature obtained by sampling rate increase at a previous stage with an encoding feature of the same sampling rate parameter according to a pre-learned merging parameter.

[0007] Another aspect of the present disclosure provides an electronic device. The electronic device comprises a memory configured to store a computer program; and one or more processors configured to perform an image enhancement process when the computer program is executed.The image enhancement process comprises performing a multi-stage sampling rate reduction on a first image to obtain multiple encoding features with different sampling rate parameters; performing a multi-stage sampling rate increase on an encoding feature obtained by sampling rate reduction at a final stage to obtain multiple decoding features with different sampling rate parameters; and obtaining a second image based on a decoding feature obtained by sampling rate increase at a final stage, with a resolution of the second image being higher than a resolution of the first image.The sampling rate increase at each non-first stage comprises merging a decoding feature obtained by sampling rate increase at a previous stage with an encoding feature of the same sampling rate parameter to obtain a merged feature, and performing a sampling rate increase on the merged feature; and a process of obtaining a merged feature at at least one non-first stage comprises merging a decoding feature obtained by sampling rate increase at a previous stage with an encoding feature of the same sampling rate parameter according to a pre-learned merging parameter.

[0008] Another aspect of the present disclosure provides a non-volatile, computer-readable storage medium containing a computer program which, when executed, causes one or more processors to perform an image enhancement procedure. The image enhancement procedure comprises performing a multi-stage sampling rate reduction on a first image to obtain multiple encoding features with different sampling rate parameters; performing a multi-stage sampling rate increase on an encoding feature obtained by sampling rate reduction at a final stage to obtain multiple decoding features with different sampling rate parameters; and obtaining a second image based on a decoding feature obtained by sampling rate increase at a final stage, wherein the resolution of the second image is higher than the resolution of the first image.The sampling rate increase at each non-first stage comprises merging a decoding feature obtained by sampling rate increase at a previous stage with an encoding feature of the same sampling rate parameter to obtain a merged feature, and performing a sampling rate increase on the merged feature; and a process of obtaining a merged feature at at least one non-first stage comprises merging a decoding feature obtained by sampling rate increase at a previous stage with an encoding feature of the same sampling rate parameter according to a pre-learned merging parameter.

[0009] The person skilled in the art may, in light of the description, claims and drawings of the present disclosure, conceive of other aspects of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] To more clearly illustrate the technical solutions of embodiments of the present disclosure, the accompanying drawings required in exemplary embodiments of the present disclosure are briefly described below. Obviously, the accompanying drawings described below can be exemplary embodiments of the present disclosure. A person skilled in the art can derive other accompanying drawings from the aforementioned accompanying drawings without any creative effort. Fig. Figure 1 shows an implementation flow diagram of an image enhancement method according to various embodiments of the present disclosure. Fig. Figure 2 shows an implementation flow diagram of a merging of a decoding feature obtained by upsampling at the previous stage with a coding feature of the same sampling rate parameter using a pre-learned merging parameter according to various embodiments of the present disclosure. Fig. Figure 3 shows an implementation flow diagram of a weighted merging of a decoding feature obtained by increasing the sampling rate at the previous stage with an encoding feature of the same sampling rate parameter based on a first weight according to various embodiments of the present disclosure. Fig. Figure 4 shows another implementation flow diagram of a weighted merging of a decoding feature obtained by increasing the sampling rate at the previous stage with an encoding feature of the same sampling rate parameter based on a first weight according to various embodiments of the present disclosure. Fig. Figure 5 shows a structural diagram of an improvement network according to various embodiments of the present disclosure. Fig. Figure 6 shows another structural scheme of an improvement network according to different embodiments of the present disclosure. Fig. Figure 7 shows another structural scheme of an improvement network according to different embodiments of the present disclosure. Fig. Figure 8 shows a structural scheme of an initial network according to various embodiments of the present disclosure. Fig. Figure 9 shows another structural scheme of an initial network according to different embodiments of the present disclosure. Fig. Figure 10 shows an implementation flow diagram of performing supervised training on a first coding module using a first image set based on a pre-trained network according to various embodiments of the present disclosure. Fig. Figure 11 shows an implementation flow diagram of performing supervised training on a fusion module using a first image set based on a pre-trained network according to various embodiments of the present disclosure. Fig. Figure 12 shows a structural diagram of an image enhancement device according to various embodiments of the present disclosure. Fig. Figure 13 shows a structural diagram of a model training device according to various embodiments of the present disclosure. Fig. Figure 14 illustrates a structural diagram of an electronic device according to various embodiments of the present disclosure. DETAILED DESCRIPTION

[0011] To clearly describe the objectives, technical solutions, and advantages of this disclosure, the technical solutions of this disclosure are described in detail below in combination with the accompanying drawings and embodiments. The described embodiments should not be considered a limitation of this disclosure. All other embodiments that a person skilled in the art obtains without creative effort may fall within the scope of protection of this disclosure.

[0012] As mentioned previously, to improve the image enhancement effect of the U-shaped network, one implementation method is to embed (add) a low-resolution feature search module after the encoding module and before the decoding module. This allows the highly distinctive features needed for enhancement to be extracted from the minimum-resolution feature map output by the encoding module and then fed into the decoding module for decoding. Such a low-resolution feature search module can employ a convolutional module with a large kernel, a nonlocal module, or a transformer module.Such modules can be computationally intensive; and problems can arise during their use, for example, softmax, level norm, and the like, which are difficult to quantize, or slow computation at the endpoint of a neural network processing unit (NPU).

[0013] To increase the processing speed of image enhancement and to make its use more user-friendly, the present disclosure provides an image enhancement solution.

[0014] An image enhancement method, an image enhancement device, an image enhancement model and a model trainer method provided in embodiments of the present disclosure may be set up in an electronic device; and the electronic device may include a processor capable of processing images, for example a central processing unit (CPU) or a graphics processing unit (GPU).

[0015] The electronic device can be a terminal device or a server. The server can be a single server, a server cluster, a cloud server, or the like.

[0016] It will be on Fig. 1 referenced. Fig. Figure 1 shows an implementation flowchart of an image enhancement method according to various embodiments of the present disclosure. The image enhancement method may comprise the following exemplary steps.

[0017] In S101, a sampling rate reduction (downsampling) can be performed on a first image at several stages to obtain multiple coding features with different sampling rate parameters.

[0018] The first image may be a lower-quality image that requires enhancement. The first image may be an RGB image, a grayscale image, a depth image, or an image in another format; the format of the first image is not limited in this disclosure. The first image may be an image obtained by an image capture device, an image processed by an image editor, or an AI-generated image.

[0019] The input to the first sampling rate reduction stage can be the first image. From the second sampling rate reduction stage onward, the input to each subsequent sampling rate reduction stage can be the encoding feature obtained from the previous sampling rate reduction stage. This means that the input to the first sampling rate reduction stage can be the first image, and the input to the i-th sampling rate reduction stage (i = 2, 3, ..., I; where 1 is the total number of sampling rate reduction stages) can be the encoding feature output by the (i-1)-th sampling rate reduction stage.

[0020] The sampling rate parameter of the encoding features can relate to the resolution or the size of the encoding features. That is, a reduction in the sampling rate of the first image can be implemented at several stages to obtain multiple encoding features of different resolutions or to obtain multiple encoding features of different sizes.

[0021] The encoding feature obtained at each sampling rate reduction stage can be a feature image of the first image. Therefore, sampling rate reduction at different stages can result in feature images of the first image with different resolutions or feature images of the first image with different sizes. For encoding features obtained at two adjacent sampling rate reduction stages, the resolution or size of the encoding feature obtained at the later sampling rate reduction stage can be lower than the resolution or size of the encoding feature obtained at the earlier sampling rate reduction stage.

[0022] A reduction in the sampling rate of the first image at each stage can optionally be performed using any of the following sampling rate reduction methods: max pooling, convolution, mean pooling, and the like. The sampling rate reduction methods used at different sampling rate reduction stages can be the same or different.

[0023] In S102, a sampling rate increase can be performed at several stages on the coding feature obtained at the last sampling rate reduction stage in order to obtain multiple decoding features with different sampling rate parameters.

[0024] In one embodiment, the input to the first sampling rate increase stage can be the encoding feature output by the last sampling rate decrease stage. That is, the processing object of the sampling rate increase at the first stage can be the encoding feature obtained by sampling rate decrease at the last stage.

[0025] The sampling rate increase at each non-first stage may involve merging the decoding feature obtained by sampling rate increase at the previous stage with the encoding feature of the same sampling rate parameter to obtain a merged feature, and performing a sampling rate increase on the merged feature. The process of obtaining the merged feature of at least one non-first stage may involve merging the decoding feature obtained by sampling rate increase at the previous stage with the encoding feature of the same sampling rate parameter using a pre-learned merging parameter.

[0026] For decoding features obtained on two adjacent sampling rate increase stages, the resolution or size of the decoding feature obtained on the later sampling rate increase stage may be larger than the resolution or size of the decoding feature obtained on the earlier sampling rate increase stage.

[0027] In the present disclosure, the number of sampling rate increase stages can be equal to the number of sampling rate decrease stages, wherein the sampling rate parameter of the coding feature obtained on the j-th (j = 1, 2, 3, ..., 1-1) sampling rate increase stage can be equal to the sampling rate parameter of the coding feature obtained on the (1-j)-th sampling rate decrease stage.

[0028] In the present disclosure, from the second sampling rate increase stage onwards, no sampling rate increase needs to be performed on the output of the previous sampling rate increase stage; instead, the decoding feature output by the previous sampling rate increase stage can be merged with the encoding feature of the same sampling rate parameter to obtain the merged feature, and a sampling rate increase can be performed on the merged feature. When merging the encoded feature and the decoding feature of the same sampling rate parameter, the merging process, corresponding to at least some of the sampling rate increase stages, can consist of using the pre-learned merging parameter to merge the encoded feature and the decoding feature of the same sampling rate parameter.That is, the merging process corresponding to only a subset of the sample rate increment stages can use the pre-learned merging parameter; or the merging processes corresponding to (1-1) sample rate increment stages can all use the pre-learned merging parameter to merge the encoded feature and the decoding feature of the same sample rate parameter.

[0029] In response to the fact that only some of the sample rate increase stages use the pre-learned fusion parameter, the fusion processes corresponding to other sample rate increase stages can consist of directly adding the encoded feature and the decoding feature of the same sample rate parameter.

[0030] The multi-stage sampling rate increase mentioned above allows the decoding feature obtained from the last stage of the sampling rate increase to exhibit a strong distinguishing feature required for image improvement.

[0031] Alternatively, the sample rate increase at the multiple stages can be performed on the coding feature obtained by sample rate reduction at the last stage using one of the following sample rate increase methods: inverse max pooling, transposed convolution, inverse mean pooling, and the like. Sample rate increase methods used at different sample rate increase stages can be the same or different.

[0032] In S 103, the second image can be obtained based on the decoding feature achieved by increasing the sampling rate at the last stage. The resolution of the second image can be higher than the resolution of the first image. The content in the second image can be the same as the content in the first image.

[0033] The decoding feature obtained by increasing the sampling rate at the last stage can exhibit the strong distinguishing feature required for image enhancement, which can ensure that the quality of the second image is higher than that of the first image.

[0034] The image enhancement method provided in the present disclosure no longer requires the module for searching for low-resolution features. Instead, after performing a sampling rate reduction on the first, lower-quality image at the multiple stages to obtain the multiple encoding features with different sampling rate parameters, a sampling rate increase can be performed at the multiple stages on the encoding feature obtained by sampling rate reduction at the last stage to obtain the multiple decoding features with different sampling rate parameters. The sampling rate increase at each non-first stage can include merging the decoding feature obtained by sampling rate increase at the previous stage with the encoding feature of the same sampling rate parameter to obtain a merged feature, and then performing a sampling rate increase on the merged feature.The process of obtaining the fused feature of at least one non-first stage can involve merging the decoding feature obtained by increasing the sampling rate at the previous stage with the encoding feature of the same sampling rate parameter using a pre-learned merging parameter. Therefore, if the features of sufficiently low-quality images are obtained, excessive low-quality features need not interfere with the generation of high-quality images, which can ensure that the decoding feature obtained by increasing the sampling rate at the final stage can possess the strong discriminatory power required for improvement.Furthermore, the process of merging the encoding feature and the decoding feature of the same sampling rate parameter does not necessarily require complex calculations, thus minimizing the computational effort, ensuring processing speed for image enhancement, and facilitating easy use.

[0035] According to an optional embodiment, Fig. 2 an implementation flow diagram of a merging of the decoding feature obtained by increasing the sampling rate at the previous stage with the encoding feature of the same sampling rate parameter using the pre-learned merging parameter according to various embodiments of the present disclosure, which may include the following exemplary steps.

[0036] In S201, based on the merging parameter, the decoding feature obtained by increasing the sampling rate at the previous stage, and the encoding feature of the same sampling rate parameter, the first weight of the encoding feature that has the same sampling rate parameter as the decoding feature obtained by increasing the sampling rate at the previous stage can be calculated.

[0037] The first weight can be calculated as follows: Calculate the global feature value g: g = sigmoid(Down-sample_r(e+d)*W); and calculate the gate-modulated feature value s: s = sigmoid(Up-sample_r(g)).

[0038] Down-sample_r denotes a sample rate reduction operation using a window size of r; and such an operation can be implemented using various methods such as max pooling, convolution, mean pooling, or the like.

[0039] Up-sample_r denotes a sample rate increase operation using a window size of r; and such an operation can be implemented using various methods such as inverse max pooling (max unpooling), transposed convolution, inverse mean pooling (mean unpooling), or the like.

[0040] Sigmoid() is a threshold function designed to map the variables in the parentheses to the range [0, 1].

[0041] The variable e denotes an encoding feature and d denotes a decoding feature, where e and d can have the same sampling rate parameter. That is, d denotes the decoding feature obtained by increasing the sampling rate at the previous stage, and e denotes the encoding feature with the same sampling rate parameter as d.

[0042] W denotes a pre-learned merging parameter, which is a matrix of dimension h / r*w / r, where h denotes the height of the encoding feature e and w denotes the width of the encoding feature e. The height and width of the decoding feature d can be equal to the height and width, respectively, of the encoding feature e.

[0043] The gate-modulated feature value s can be the first weight of the coding feature e.

[0044] In S202, a weighted fusion can be performed on the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter based on the first weight.

[0045] After obtaining the first weight s of the coding feature e, the first weight s can be set up to perform a weighted merging of the decoding feature d obtained by increasing the sampling rate at the previous stage and the coding feature e of the same sampling rate parameter.

[0046] According to an optional embodiment, Fig. 3 an implementation flow diagram of a weighted merging of the decoding feature obtained by increasing the sampling rate at the previous stage with the encoding feature of the same sampling rate parameter based on the first weight according to various embodiments of the present disclosure, which may include the following exemplary steps.

[0047] In S301, the second weight of the decoding feature obtained by increasing the sampling rate at the previous stage can be obtained based on the first weight.

[0048] Alternatively, the second weight can be the difference between 1 and the first weight; that is, the second weight can be 1-s.

[0049] In S302, a weighted calculation can be performed on the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter based on the first weight and the second weight to obtain the merged feature.

[0050] In one embodiment, the merged feature f can be calculated as follows: f = s*e+(1-s)*d.

[0051] According to an optional embodiment, Fig. 4. Another implementation flowchart of a weighted merging of the decoding feature obtained by increasing the sampling rate at the previous stage with the encoding feature of the same sampling rate parameter based on the first weight according to various embodiments of the present disclosure. With reference to Fig. 4. The image enhancement process can include the following exemplary steps.

[0052] In S401, the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter can be summed to obtain an initial merged feature.

[0053] That is, the initial merged feature can be e + d.

[0054] In S402, the second weight of the decoding feature obtained by increasing the sampling rate at the previous stage can be obtained based on the first weight.

[0055] Alternatively, the second weight can be the difference between 1 and the first weight; that is, the second weight can be 1-s.

[0056] In the present disclosure, exemplary step S401 can be carried out first, and then exemplary step S402 can be carried out; or exemplary step S402 can be carried out first, and then exemplary step S401 can be carried out; or both exemplary steps can be carried out simultaneously. The present disclosure need not restrict the order in which the two exemplary steps mentioned above are carried out.

[0057] In S403, a weighted calculation can be performed on the initial merged feature and the decoding feature obtained by increasing the sampling rate at the previous stage, based on the first weight and the second weight, to obtain the merged feature.

[0058] In one embodiment, the merged feature f can be calculated as follows: f = s * (e+d) + (1-s) * d.

[0059] In an optional embodiment, the above-mentioned process of performing a sampling rate reduction on the multi-stage first image, performing a sampling rate increase on the multi-stage coding feature obtained by the sampling rate reduction on the last stage, and obtaining the second image based on the decoding feature obtained by the sampling rate increase on the last stage can be implemented using a pre-trained enhancement network (also called an image enhancement model), which is described in detail below.

[0060] A reduction in the sampling rate can be performed at several stages on the first image using the first coding module of the enhancement network.

[0061] A sampling rate increase can be performed at multiple stages on the encoding feature obtained by the sampling rate decrease at the last stage, using the decoding module of the enhancement network. The sampling rate increase at each non-first stage can involve obtaining the merged feature by merging the decoding feature obtained by the sampling rate increase at the previous stage with the encoding feature of the same sampling rate parameter using the merging module of the enhancement network, and then performing a sampling rate increase on the merged feature. The process of obtaining the merged feature for at least one non-first stage can involve merging, using the merging module, the decoding feature obtained by the sampling rate increase at the previous stage with the encoding feature of the same sampling rate parameter, using the pre-learned merging parameter.

[0062] The decoding feature obtained by increasing the sampling rate at the last stage can be processed using an output module of the enhancement network to obtain the second image.

[0063] Fig. Figure 5 shows a structural diagram of an improvement network according to various embodiments of the present disclosure.

[0064] The enhancement network can include an encoding module 501 (referred to as the first encoding module for simplicity), a decoding module 502, a merging module 503, and an output module 504.

[0065] The first coding module 501 can be configured to perform a sampling rate reduction at the multiple stages on the first image in order to obtain the multiple coding features with different sampling rate parameters.

[0066] In one embodiment, which is in Fig. As shown in Figure 5, the first encoding module 501 can perform a sampling rate reduction to four levels on the first image. In practical applications, the first encoding module 501 can also perform a sampling rate reduction to more levels, or to two or three levels. The number of sampling rate reduction levels need not be limited in various embodiments of the present disclosure.

[0067] The decoding module 502 can be configured to increase the sampling rate at several stages based on the coding feature obtained by decreasing the sampling rate at the last stage (e.g., coding feature 4 in). Fig. 5) to perform the process of obtaining the multiple decoding features with different sampling rate parameters. The sampling rate increase at each non-first stage may include obtaining the merged feature by merging the decoding feature obtained by sampling rate increase at the previous stage with the encoding feature of the same sampling rate parameter using the Merging Module 503 and performing a sampling rate increase on the merged feature. The process of obtaining the merged feature for at least one non-first stage may include merging, using the Merging Module 503, the decoding feature obtained by sampling rate increase at the previous stage with the encoding feature of the same sampling rate parameter using the pre-learned Merging Parameter.

[0068] In one embodiment, which is in Fig. As shown in Figure 5, the decoding module 502 can perform a sample rate increase to four levels on the encoding feature obtained by sample rate decrease at the last level. In practical applications, the decoding module 502 may have more sample rate increase levels, or two or three sample rate increase levels. The number of sample rate increase levels need not be limited; it is only necessary to ensure that the number of sample rate increase levels equals the number of sample rate decrease levels.

[0069] In the fusion module, the fusion parameters in different fusion gate modules can be the same or different, and actual values ​​can be determined based on the training of the improvement network.

[0070] Output module 504 can be configured to output the second image based on the decoding feature obtained by increasing the sampling rate at the last stage (e.g., decoding feature 4 in Fig. 5) to generate; and the resolution of the second image can be higher than the resolution of the first image.

[0071] In one embodiment, which is in Fig. As shown in Figure 5, the merging module can be configured to merge the encoding and decoding features of the same sampling rate parameters in three stages based on the merging parameters. In other embodiments, the merging module can be configured to merge the encoding and decoding features of the same sampling rate parameters in only two stages; or to merge the encoding and decoding features of the same sampling rate parameter in a single stage.

[0072] Fig. Figure 6 shows a further structural diagram of the enhancement network according to various embodiments of the present disclosure. The structure of the first encoding module, the decoding module, and the output module in an embodiment described in Fig. Figure 6 shows the structure of the first encoding module, the decoding module, and the output module in an embodiment described in Figure 6. Fig. 5 is shown; and the only difference may be that the structure of the fusion modules may differ. In one embodiment, which is shown in Fig. As shown in Figure 6, the fusion module 601 can be configured to merge the encoding features and decoding features of the same sampling rate parameter at only two stages based on the fusion parameters (encoding feature 1 and decoding feature 3 can only be merged based on the fusion parameter learned by fusion gate module 3; and encoding feature 3 and decoding feature 1 can only be merged based on the fusion parameter learned by fusion gate module 1); and the encoding feature and decoding feature of the same sampling rate parameter at another stage do not need to be merged based on the learned fusion parameter, but can simply be merged by addition (i.e., encoding feature 2 and decoding feature 2 can be merged directly by addition).

[0073] Fig. Figure 7 shows a further structural diagram of the enhancement network according to various embodiments of the present disclosure. The structure of the first encoding module, the decoding module, and the output module in an embodiment described in Fig. Figure 7 shows the structure of the first encoding module, the decoding module, and the output module in an embodiment shown in Figure 7. Fig. 5 is shown; and the only difference may be that the structure of the fusion modules may differ. In one embodiment, which is shown in Fig. As shown in Figure 7, the merging module 701 can be configured to merge the encoding feature and decoding feature of the same sampling rate parameter at a single stage based on the merging parameter (encoding feature 1 and decoding feature 3 can be merged simply based on the merging parameter learned by the merging gate module 3); and the encoding features and decoding features of the same sampling rate parameter at two further stages need not be merged based on the learned merging parameters, but can simply be merged by addition (i.e., encoding feature 2 and decoding feature 2 can be merged directly by addition, and encoding feature 3 and decoding feature 1 can be merged directly by addition).

[0074] In an optional embodiment, when the merging module merges the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter using the pre-learned merging parameter, it can be configured to calculate, based on the merging parameter, the decoding feature obtained by increasing the sampling rate at the previous stage, and the encoding feature of the same sampling rate parameter, the first weight of the encoding feature that has the same sampling rate parameter as the decoding feature obtained by increasing the sampling rate at the previous stage. For implementation methods, reference can be made to the embodiments mentioned above, which cannot be described in detail here.

[0075] A weighted fusion can be performed on the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter based on the first weight, whereby reference can be made to the embodiments mentioned above for implementation methods, and these cannot be described in detail here.

[0076] Within the fusion module, each fusion gate module can merge input encoding features and input decoding features that have the same sampling rate parameter, based on the learned fusion parameter. For example, fusion gate module 1 can merge encoding feature 3 and decoding feature 1 based on the first learned fusion parameter; fusion gate module 2 can merge encoding feature 2 and decoding feature 2 based on the second learned fusion parameter; and fusion gate module 3 can merge encoding feature 1 and decoding feature 3 based on the third learned fusion parameter.

[0077] In Fig. Reference can be made to the embodiments mentioned above for the process in which the respective merging module merges the encoding features and decoding features of the same sampling rate parameters based on learned merging parameters, and these may not be described in detail here.

[0078] In an optional embodiment, the improvement network can be obtained by means of a training method: performing unsupervised training on the initial network obtained (generated) by the second encoding module, the decoding module 502 and the output module 504 using high-quality images from the first image set to obtain a pre-trained network.

[0079] The first image set can comprise multiple low-quality images (with lower resolution) and high-quality images (with higher resolution) corresponding to the multiple low-quality images. The second coding module can be configured to perform a sampling rate reduction at multiple stages on the input high-quality images to obtain multiple coding features with different sampling rate parameters. Among the multiple coding features with different sampling rate parameters obtained by the second coding module, at least one coding feature can have the same sampling rate parameter as at least one coding feature obtained by sampling rate reduction at multiple stages on the input high-quality images.

[0080] Fig. Figure 8 shows a structural diagram of an initial network according to various embodiments of the present disclosure. The initial network can comprise a second encoding module 801, a decoding module 802, and an output module 803. The structure of the second encoding module 801 and that of the first encoding module 801 can be the same or different; it is only necessary to ensure that the second encoding module 801 can perform a sampling rate reduction at the multiple stages of the input image and acquire the encoding feature of the same sampling rate parameter as at least one encoding feature output by the first encoding module 801. The structure of the decoding module 802 and that of the decoding module 803 can be the same; and the structure of the output module 804 and that of the output module 803 can be the same.

[0081] Fig. Figure 9 shows a further structural diagram of the initial network according to various embodiments of the present disclosure. In one embodiment, the structure of the second coding module 801 and that of the first coding module 501 can be the same; and the second coding module 801 and the first coding module 501 can each have four sampling rate reduction modules.

[0082] An implementation method for unsupervised training of the initial network is described below.

[0083] The high-quality image can be entered into the second coding module 801.

[0084] The second coding module 801 can perform a sampling rate reduction at several stages on the high-quality image in order to achieve the multiple coding features with different sampling rate parameters.

[0085] The decoding module 502 can perform a sampling rate increase at the multiple stages of the coding feature obtained by the sampling rate decrease at the last stage of the second coding module 801, in order to obtain the multiple decoding features with different sampling rate parameters.

[0086] The output module 504 can process the decoding feature obtained by increasing the sampling rate at the last stage of the decoding module 502 in order to obtain a reconstructed image of high quality.

[0087] For example, the parameters of the initial network can be updated with the aim of minimizing the difference between the reconstructed high-quality image and the original high-quality image (i.e., compared to the reconstructed image obtained by reconstructing the high-quality image using the initial network before updating the parameters, the reconstructed image obtained by reconstructing the high-quality image using the initial network after updating the parameters may be closer to the original high-quality image).

[0088] The purpose of unsupervised training of the initial network using high-quality images may be to fully exploit the properties of the high-quality images in order to obtain a desirable decoding module, so that the improved images can be more realistic.

[0089] After obtaining the pre-trained network, supervised training can be performed on the first encoding module (which may or may not be the second encoding module in the pre-trained network) and the fusion module, based on the pre-trained network and using the first image set. This results in the trained fusion module and first encoding module. The trained fusion module and first encoding module, together with the decoding module and the output module in the pre-trained network, can then form the enhancement network, i.e., the image enhancement model.

[0090] Alternatively, in response to the fact that the structure of the second coding module is the same as the structure of the first coding module, supervised training can be performed on the fusion module and the second coding module in the pretrained network using the first image set based on the pretrained network, thereby obtaining the trained first coding module and fusion module (i.e., the trained second coding module can be set up as the first coding module).

[0091] In response to the fact that the structure of the second coding module differs from the structure of the first coding module, supervised training can be performed on the first coding module and the fusion module using the first image set based on the pre-trained network, thereby obtaining the trained fusion module and first coding module.

[0092] According to an optional embodiment, Fig. 10 an implementation flow diagram of performing supervised training on the first coding module using the first image set based on the pretrained network according to various embodiments of the present disclosure, which may include the following exemplary steps.

[0093] In S1001, for a low-quality image in the first image set, a low-quality image is entered into the first coding module in order to obtain the multiple coding features with different sampling rate parameters of a low-quality image output by the first coding module.

[0094] In this case, the first coding module can be the second coding module in the pre-trained network or a coding module different from the second coding module in the pre-trained network.

[0095] In S1002, the high-quality image corresponding to a low-quality image can be entered into the second coding module in the pre-trained network to obtain the multiple coding features with different sampling rate parameters of the high-quality image corresponding to a low-quality image, as output by the second coding module.

[0096] In response to the fact that the first coding module is the second coding module in the pretrained network, the sampling rate parameters of the coding features output by the second coding module and the first coding module can be the same in such an exemplary step.

[0097] In response to the fact that the first coding module is not the second coding module in the pretrained network, the following possibilities exist.

[0098] The sampling rate parameters of the coding features output by the second coding module and the first coding module can be the same.

[0099] Some of the coding features output by the second coding module may have the same sampling rate parameters as at least some of the coding features output by the first coding module.

[0100] The coding features output by the second coding module can have the same sampling rate parameters as some of the coding features output by the first coding module.

[0101] In S1003, the parameters of the first coding module can be updated with the aim of minimizing the first difference between the coding characteristics of the low-quality image and the coding characteristics of the corresponding high-quality image for the same sampling rate parameter.

[0102] In other words, any low-quality image can be encoded using the first encoding module before updating the parameters, and the multiple encoding features with different sampling rate parameters can be obtained by reducing the sampling rate at the multiple stages on the low-quality image using the first encoding module after updating the parameters; and compared to the encoding features of the low-quality image, the encoding features of the same sampling rate parameter obtained by reducing the sampling rate at the multiple stages on the high-quality image corresponding to the low-quality image using the second encoding module can exhibit a smaller initial difference.

[0103] In one embodiment, the second coding module in the pre-trained network can be set up as a teacher network; and the first coding module can be set up as a student network to perform distillation training on the first coding module, so that the features obtained from the student network can be closer to the features obtained from the teacher network.

[0104] In an optional embodiment, for the first coding feature of a sampling rate parameter of a low-quality image and the second coding feature of a sampling rate parameter of the high-quality image corresponding to a low-quality image (i.e., the second coding feature and the first coding feature can be coding features of different images of the same sampling rate parameter), the first difference between the first coding feature and the second coding feature can be calculated below.

[0105] The following can be obtained: the absolute error between the first coding feature and the second coding feature (referred to as Loss1), the sum of the squares of the errors between the first classification result obtained based on the first coding feature and the second classification result obtained based on the second coding feature (referred to as Loss2), and the maximum output error between the first coding feature and the second coding feature (referred to as Loss3).

[0106] Alternatively, the three errors mentioned above can be calculated as follows: Loss1=|f(teacher)−f(student)|1; Loss2 = |vgg(f(teacher)) − vgg(f(student)) |2; and Loss3=|softmax(f(teacher)×f(teacher).T)−softmax(f(student)×f(student).T)|2, where f(teacher) denotes the second coding feature and f(student) denotes the first coding feature, vgg denotes the pretrained VGG network for image classification, vgg(f(teacher)) denotes the second classification result obtained from the VGG network based on the second coding feature, vgg(f(student)) denotes the first classification result obtained from the VGG network based on the first coding feature, f(teacher).T denotes the transpose of the second coding feature and f(student).T denotes the transpose of the first coding feature.

[0107] The weighted sum of the absolute error, the sum of squared errors, and the maximum output error can be configured to obtain the first difference (referred to as loss) between the first encoding feature and the second encoding feature.

[0108] The first difference loss can be calculated as follows: Loss = λ1×Loss1 + λ2×Loss2 + λ3×Loss3 where λ1, λ2, and λ3 are the weights of three errors. The weights mentioned above can be hyperparameters, meaning they can be predetermined.

[0109] Alternatively, the weight of the absolute error can be negatively correlated with the sampling rate reduction level used to obtain the first encoding feature. That is, the lower the sampling rate reduction level, the greater the corresponding weight of the absolute error. Conversely, the larger the size (or resolution) of the first encoding feature, the greater the corresponding weight of the absolute error.

[0110] Alternatively, the weight of the sum of squared errors can be positively correlated with the sampling rate reduction level of the first encoding feature. That is, the higher the sampling rate reduction level, the greater the corresponding weight of the sum of squared errors. Conversely, the smaller the size (or resolution) of the first encoding feature, the greater the corresponding weight of the sum of squared errors.

[0111] In an optional embodiment - after the first coding feature has been trained - it shows Fig. 11 an implementation flow diagram of performing supervised training on the fusion module using the first image set based on the pretrained network according to various embodiments of the present disclosure, which may include the following exemplary steps.

[0112] In S1101, using a trained first coding module, a sampling rate reduction can be performed on several stages of a low-quality image to obtain the multiple coding features with different sampling rate parameters.

[0113] The trained first coding module can refer to the first coding module obtained through the distillation training mentioned above.

[0114] In S1102, a sampling rate increase at the multiple stages of the encoding features obtained by sampling rate decrease at the last stage can be performed using the trained decoding module to obtain the multiple decoding features with sampling rate parameters. The sampling rate increase at each non-first stage can involve obtaining the merged feature from the merging module by merging the decoding feature obtained by sampling rate increase at the previous stage with the encoding feature of the same sampling rate parameter and performing a sampling rate increase on the merged feature.The process of obtaining the merged feature of at least one non-first stage may include merging, by the merging module, the decoding feature obtained by increasing the sampling rate at the previous stage with the encoding feature of the same sampling rate parameter using the merging parameter.

[0115] The process of obtaining the merged feature by the fusion module may refer to the embodiments mentioned above and may not be described in detail here.

[0116] The trained decoding module can refer to the decoding module in the pre-trained network, which was obtained by training the initial network.

[0117] In S1103, an improved image corresponding to a low-quality image can be obtained using the trained output module based on the decoding feature achieved by increasing the sampling rate at the last stage.

[0118] The trained output module can refer to the output module in the pre-trained network obtained by training the initial network.

[0119] In S1104, the parameters of the merging module can be updated with the aim of minimizing the second difference between the enhanced image corresponding to a low-quality image and the high-quality image corresponding to a low-quality image.

[0120] The parameters of the merging module can include the merging parameters mentioned above.

[0121] The second difference described above between the improved image corresponding to a low-quality image and the high-quality image corresponding to the low-quality image can be minimized, meaning that compared to the image enhancement performed on a low-quality image using the merging module before updating the parameters, the improved image obtained by enhancing a low-quality image using the merging module after updating the parameters can be closer to the high-quality image corresponding to the low-quality image.

[0122] When the fusion module is being trained, the parameters of the first encoding module, the decoding module, and the output module can be frozen and remain unchanged.

[0123] In accordance with the method implementations, the present disclosure also provides an image enhancement device. Fig. Figure 12 shows a structural diagram of an image enhancement device according to various embodiments of the present disclosure. With reference to Fig. 12. The image enhancement device can include an encoding unit 1201, a decoding unit 1202, a merging unit 1203 and an output unit 1204.

[0124] The coding unit 1201 can be configured to perform a sampling rate reduction at the multiple stages on the first image in order to obtain the multiple coding features with different sampling rate parameters.

[0125] The decoding unit 1202 can be configured to perform a sampling rate increase at the multiple stages of the encoding feature obtained by sampling rate decrease at the last stage, in order to obtain the multiple decoding features with different sampling rate parameters. The sampling rate increase at each non-first stage can include obtaining the merged feature from the merging unit 1203 by merging the decoding feature obtained by sampling rate increase at the previous stage with the encoding feature of the same sampling rate parameter and performing a sampling rate increase on the merged feature.The process of obtaining the merged feature of at least one non-first stage may include merging, by the merging module, the decoding feature obtained by increasing the sampling rate at the previous stage with the encoding feature of the same sampling rate parameter using the merging parameter.

[0126] The output unit 1204 can be configured to obtain the second image based on the decoding feature obtained by increasing the sampling rate at the last stage; and the resolution of the second image can be higher than the resolution of the first image.

[0127] The image enhancement device provided in one embodiment of the present disclosure can perform a sampling rate reduction at the multiple stages on the first image of comparatively low resolution to obtain the multiple encoding features with different sampling rate parameters; and perform a sampling rate increase at the multiple stages on the encoding feature obtained by sampling rate reduction at the last stage to obtain the multiple decoding features with different sampling rate parameters. The sampling rate increase at each non-first stage can comprise obtaining the merged feature by merging the decoding feature obtained by sampling rate increase at the previous stage with the encoding feature of the same sampling rate parameter and performing a sampling rate increase on the merged feature.The process of obtaining the merged feature of at least one non-first stage can involve merging, by the merging module, the decoding feature obtained by sampling rate increase at the previous stage with the encoding feature of the same sampling rate parameter using the pre-trained merging parameter. The image enhancement device can further obtain the second image of comparatively high resolution based on the decoding feature obtained by sampling rate increase at the last stage. The decoding features and encoding features of the same sampling rate parameters can be merged using pre-trained merging parameters, and a sampling rate increase can be performed so that the generation of high-quality images is not disturbed by excessive low-quality features when sufficiently low-quality features are obtained.Therefore, the decoding feature obtained from the final sampling rate increase stage can exhibit the strong distinguishing features required for enhancement. Furthermore, the feature merging process does not require complex calculations, which minimizes the computational complexity of the image enhancement process and ensures high processing speed and ease of use.

[0128] In an optional embodiment, the merging unit 1203, when merging the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter, can be configured, using the pre-learned merging parameter, to calculate, based on the merging parameter, the first weight of the encoding feature that has the same sampling rate parameter as the decoding feature obtained by increasing the sampling rate at the previous stage, and the encoding feature of the same sampling rate parameter; and to perform a weighted merging of the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature with the same sampling rate parameter based on the first weight.

[0129] In an optional embodiment, when performing a weighted fusion of the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter based on the first weight, the fusion unit 1203 may be configured to obtain the second weight of the decoding feature obtained by increasing the sampling rate at the previous stage based on the first weight; and to perform a weighted calculation of the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter based on the first weight and the second weight.

[0130] In an optional embodiment, when the merging unit 1203 performs a weighted merging of the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter based on the first weight, it can be configured to sum the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter; to obtain the second weight of the decoding feature obtained by increasing the sampling rate at the previous stage based on the first weight; and to perform a weighted calculation on the initial merged feature and the decoding feature obtained by increasing the sampling rate at the previous stage based on the first weight and the second weight.

[0131] In an optional embodiment, the image enhancement device can perform a sampling rate reduction on the multiple stages of the first image, a sampling rate increase on the multiple stages of the encoding feature obtained by the sampling rate reduction on the last stage, and obtain the second image based on the decoding feature obtained by the sampling rate increase on the last stage, as described in detail below.

[0132] The image enhancement device can perform a sampling rate reduction at the multiple stages of the first image using the first encoding module of the enhancement network; and a sampling rate increase at the multiple stages of the encoding feature obtained by sampling rate reduction at the last stage to obtain the multiple decoding features with different sampling rate parameters using the decoding module of the enhancement network. The sampling rate increase at each non-first stage can involve obtaining the merged feature from the merging module of the enhancement network by merging the decoding feature obtained by sampling rate increase at the previous stage with the encoding feature of the same sampling rate parameter and performing a sampling rate increase on the merged feature.The process of obtaining the merged feature of at least one non-first stage can involve merging, by the merging module, the decoding feature obtained by increasing the sampling rate at the previous stage with the encoding feature of the same sampling rate parameter, using the pre-learned merging parameter. The image enhancement device can also process the decoding feature obtained by increasing the sampling rate at the last stage through the output module of the enhancement network to obtain the second image.

[0133] In an optional embodiment, the improvement network can be obtained using a training method described below.

[0134] Unsupervised training can be performed on the initial network generated by the second encoding module, the decoding module, and the output module using high-quality images from the first image set to obtain the pre-trained network. The first image set can include several low-quality images and corresponding high-quality images for each low-quality image. The second encoding module can be configured to perform a sampling rate reduction on the multiple levels of input high-quality images to obtain the multiple encoding features with different sampling rate parameters.

[0135] Based on the pre-trained network, supervised training can be performed on the first encoding module and the fusion module using the first image set to obtain the trained first encoding module and the trained fusion module. The trained first encoding module and the trained fusion module, together with the decoding module and the output module in the pre-trained network, can then form the enhancement network.

[0136] The implementation method of the improvement network (facility or procedure) may refer to the embodiments mentioned above and may not be described in detail here.

[0137] In accordance with the method implementation forms, the present disclosure further provides a model training device for training the above-mentioned image enhancement model. Fig. Figure 13 shows a structural diagram of a model training device according to various embodiments of the present disclosure. With reference to Fig. 13. The model training facility can have the first training unit 1301 and the second training unit 1302.

[0138] The first training unit 1301 can be configured to perform unsupervised training on the initial network obtained from the second encoding module, the decoding module, and the output module, using high-quality images from the first image set, in order to obtain the pre-trained network. The first image set can include several low-quality images and corresponding high-quality images for each low-quality image. The second encoding module can be configured to perform a sampling rate reduction at several levels on the input high-quality image to obtain the multiple encoding features with different sampling rate parameters. The structure of the second encoding module can be the same or different from the structure of the first encoding module.

[0139] The second training unit 1302 can be configured to perform supervised training on the first encoding module and the fusion module based on the pre-trained network using the first image set, in order to obtain the trained first encoding module and the trained fusion module. The trained first encoding module and the trained fusion module, together with the decoding module and the output module in the pre-trained network, can form the enhancement network.

[0140] In an optional embodiment, when the second training unit 1302 performs supervised training on the first coding module using the first image set based on the pre-trained network, it can be configured to: input a low-quality image into the first coding module for a low-quality image in the first image set in order to obtain the multiple coding features with different sampling rate parameters for a low-quality image; input the high-quality image corresponding to a low-quality image into the second coding module in the pre-trained network in order to obtain the multiple coding features with different sampling rate parameters for the high-quality image corresponding to a low-quality image;and to update the parameters of the first coding module with the aim of minimizing the initial difference in coding characteristics between the low-quality image and the corresponding high-quality image with the same sampling rate parameter.

[0141] The implementation method of the model trainer device may refer to the embodiments mentioned above and may not be described in detail here.

[0142] In addition to the procedural forms, the present disclosure also provides an electronic device. Fig. Figure 14 illustrates a structural diagram of an electronic device according to various embodiments of the present disclosure. With reference to Fig. 14 The electronic device may have at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4.

[0143] In embodiments of the present disclosure, the respective number of processors 1, communication interfaces 2, memory 3 and communication buses 4 can be at least one; and the processor 1, the communication interface 2 and the memory 3 can communicate with each other via the communication bus 4.

[0144] The processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present disclosure.

[0145] Memory 3 can include high-speed RAM, non-volatile memory, or the like, for example, at least one disk drive.

[0146] Memory 3 can hold a program, and processor 1 can call the program stored in memory 3.

[0147] The program can be configured to perform a sampling rate reduction at the multiple stages of the first image to obtain the multiple encoding features with different sampling rate parameters; and it can perform a sampling rate increase at the multiple stages of the encoding feature obtained at the last sampling rate reduction stage to obtain the multiple decoding features with different sampling rate parameters. The sampling rate increase at each non-first stage can include merging the decoding feature obtained by sampling rate increase at the previous stage with the encoding feature of the same sampling rate parameter to obtain the merged feature, and then performing a sampling rate increase on the merged feature.The process of obtaining the merged feature of at least one non-first stage can involve merging the decoding feature obtained by increasing the sampling rate at the previous stage with the encoding feature of the same sampling rate parameter using a pre-learned merging parameter. The program can further be configured to obtain the second image based on the decoding feature obtained by increasing the sampling rate at the last stage. The resolution of the second image can be higher than the resolution of the first image.

[0148] Alternatively, the program can be configured to perform unsupervised training on the initial network obtained by the second encoding module, the decoding module, and the output module, using high-quality images from the first image set to obtain the pre-trained network. The first image set can include several low-quality images and corresponding high-quality images for each low-quality image. The second encoding module can be configured to perform a sampling rate reduction at several levels on the input high-quality image to obtain the multiple encoding features with different sampling rate parameters.The program can also be configured to perform supervised training on the first encoding module and the fusion module based on the pretrained network using the first image set, in order to obtain the trained first encoding module and the trained fusion module. The trained first encoding module and the trained fusion module, together with the decoding module and the output module in the pretrained network, can form the enhancement network.

[0149] Alternatively, for detailed and extended functions of the program, reference can be made to the description above.

[0150] The present disclosure further provides a storage medium. A program suitable for execution by a processor can be stored on the storage medium.

[0151] The program can be configured to perform a sampling rate reduction at the multiple stages of the first image to obtain the multiple encoding features with different sampling rate parameters; and it can perform a sampling rate increase at the multiple stages of the encoding feature obtained at the last sampling rate reduction stage to obtain the multiple decoding features with different sampling rate parameters. The sampling rate increase at each non-first stage can include merging the decoding feature obtained by sampling rate increase at the previous stage with the encoding feature of the same sampling rate parameter to obtain the merged feature, and then performing a sampling rate increase on the merged feature.The process of obtaining the merged feature of at least one non-first stage can involve merging the decoding feature obtained by increasing the sampling rate at the previous stage with the encoding feature of the same sampling rate parameter using a pre-learned merging parameter. The program can further be configured to obtain the second image based on the decoding feature obtained by increasing the sampling rate at the last stage. The resolution of the second image can be higher than the resolution of the first image.

[0152] Alternatively, the program can be configured to perform unsupervised training on the initial network obtained by the second encoding module, the decoding module, and the output module, using high-quality images from the first image set to obtain the pre-trained network. The first image set can include several low-quality images and corresponding high-quality images for each low-quality image. The second encoding module can be configured to perform a sampling rate reduction at several levels on the input high-quality image to obtain the multiple encoding features with different sampling rate parameters.The program can also be configured to perform supervised training on the first encoding module and the fusion module based on the pretrained network using the first image set, in order to obtain the trained first encoding module and the trained fusion module. The trained first encoding module and the trained fusion module, together with the decoding module and the output module in the pretrained network, can form the enhancement network.

[0153] Alternatively, for detailed and extended functions of the program, reference can be made to the description above.

[0154] Various embodiments of the present disclosure provide an electronic device. The electronic device comprises a memory configured to store a computer program; and one or more processors configured to perform an image enhancement process when the computer program is executed.The image enhancement process comprises performing a multi-stage sampling rate reduction on a first image to obtain multiple encoding features with different sampling rate parameters; performing a multi-stage sampling rate increase on an encoding feature obtained by sampling rate reduction at a final stage to obtain multiple decoding features with different sampling rate parameters; and obtaining a second image based on a decoding feature obtained by sampling rate increase at a final stage, with a resolution of the second image being higher than a resolution of the first image.The sampling rate increase at each non-first stage comprises merging a decoding feature obtained by sampling rate increase at a previous stage with an encoding feature of the same sampling rate parameter to obtain a merged feature, and performing a sampling rate increase on the merged feature; and a process of obtaining a merged feature at at least one non-first stage comprises merging a decoding feature obtained by sampling rate increase at a previous stage with an encoding feature of the same sampling rate parameter according to a pre-learned merging parameter.

[0155] Various embodiments of the present disclosure provide a non-volatile, computer-readable storage medium containing a computer program which, when executed, causes one or more processors to perform an image enhancement process. The image enhancement process comprises performing a multi-stage sampling rate reduction on a first image to obtain multiple encoding features with different sampling rate parameters; performing a multi-stage sampling rate increase on an encoding feature obtained by sampling rate reduction at a final stage to obtain multiple decoding features with different sampling rate parameters; and obtaining a second image based on a decoding feature obtained by sampling rate increase at a final stage, wherein the resolution of the second image is higher than the resolution of the first image.The sampling rate increase at each non-first stage comprises merging a decoding feature obtained by sampling rate increase at a previous stage with an encoding feature of the same sampling rate parameter to obtain a merged feature, and performing a sampling rate increase on the merged feature; and a process of obtaining a merged feature at at least one non-first stage comprises merging a decoding feature obtained by sampling rate increase at a previous stage with an encoding feature of the same sampling rate parameter according to a pre-learned merging parameter.

[0156] Compared to the prior art, the technical solutions provided by the present disclosure can achieve at least the following advantageous effects.

[0157] For the image enhancement method, image enhancement device, image enhancement model, image enhancement model training method, electronic device, and storage medium provided by the present disclosure, a sampling rate reduction can be performed on the first image with comparatively low resolution at the multiple stages to obtain the multiple encoding features with different sampling rate parameters; a sampling rate increase can be performed on the encoding feature obtained by sampling rate reduction at the last stage to obtain the multiple decoding features with different sampling rate parameters; and the second image with comparatively high resolution can be obtained based on the decoding feature obtained by sampling rate increase at the last stage.The sampling rate increase at each non-first stage can include obtaining the merged feature by merging the decoding feature obtained by sampling rate increase at the previous stage with the encoding feature of the same sampling rate parameter and performing a sampling rate increase on the merged feature. The process of obtaining the merged feature of at least one non-first stage can include merging the decoding feature obtained by sampling rate increase at the previous stage with the encoding feature of the same sampling rate parameter using the pretrained merging parameter.

[0158] The person skilled in the art understands that exemplary units and algorithm steps described in various embodiments of this disclosure can be implemented using electronic hardware or a combination of computer software and electronic hardware. Whether such functions are implemented in hardware or as software may depend on certain application and design constraints of the technical solutions. The person skilled in the art may implement the described functions in different ways for each specific application; however, such implementations should not be considered to be outside the scope of protection of this disclosure.

[0159] In some embodiments provided in the present disclosure, it is understood that the disclosed systems, devices, and methods can also be implemented in other ways. Furthermore, the couplings shown or discussed, direct couplings, or communication links can be implemented through interfaces, indirect couplings, or communication links between devices or units, and can be electrical, mechanical, or otherwise.

[0160] Units described as separate components may or may not be physically separate. Components described as display units may or may not be physical units; that is, they may be located in a single location or distributed across multiple network units. Depending on the requirements, some or all of the units may be selected to perform the tasks of the present embodiments.

[0161] Furthermore, in various embodiments of the present disclosure, each functional unit can be integrated into a single processing unit; each unit can exist physically separately; or two or more units can be integrated into a single unit.

[0162] It is understood that in embodiments of the present disclosure, dependent claims, different embodiments and features can be combined and integrated to solve the aforementioned technical problems.

[0163] In response to the fact that the described functions are implemented as functional software units and sold or used as standalone products, the functions may be stored on a computer-readable storage medium. Based on this understanding, the essence of the technical solution of the present disclosure, a part of the technical solution that may contribute to the prior art, or a part of the technical solution may be embodied in the form of a software product. The computer software product may be stored on a storage medium and may include specific instructions for a computer device (which may be a personal computer, a server, a network device, or the like) to perform all or part of the exemplary steps of the procedures described in the embodiments of the present disclosure.The storage medium mentioned above can include various media capable of storing program code, including USB drives, portable hard drives, read-only memory (ROM), random access memory (RAM), storage disks, optical disks, or the like.

[0164] The above description of disclosed embodiments may enable the person skilled in the art to use or implement the present disclosure. Various modifications of embodiments of the present disclosure may be obvious to the person skilled in the art, and general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure need not be limited to the present embodiments but should correspond to the broadest scope compatible with the principles and novel features disclosed herein. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] CH 202411563375.9

[0001]

Claims

[1] Image enhancement techniques, including: Performing a multi-stage sampling rate reduction on a first image to obtain multiple encoding features with different sampling rate parameters; Performing a multi-stage sample rate increase on an encoding feature obtained by a sample rate decrease at a final stage to obtain multiple decoding features with different sample rate parameters, wherein a sample rate increase at each non-first stage comprises merging a decoding feature obtained by a sample rate increase at a previous stage with an encoding feature of the same sample rate parameter to obtain a merged feature, and performing a sample rate increase on the merged feature; and a process of obtaining a merged feature at at least one non-first stage comprises merging a decoding feature obtained by a sample rate increase at a previous stage with an encoding feature of the same sample rate parameter according to a pre-learned merge parameter; and Obtaining a second image based on a decoding feature obtained by increasing the sampling rate at a final stage, where the resolution of the second image is higher than the resolution of the first image. [2] Method according to claim 1, wherein the merging of the decoding feature obtained by increasing the sampling rate at the previous stage with the encoding feature of the same sampling rate parameter according to the pre-learned merging parameter comprises: based on the merging parameter, the decoding feature obtained by increasing the sampling rate at the previous stage, and the encoding feature of the same sampling rate parameter, calculate a first weight of the encoding feature that has the same sampling rate parameter as the decoding feature obtained by increasing the sampling rate at the previous stage; and Performing a weighted fusion on the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter based on the first weight. [3] Method according to claim 2, wherein performing the weighted fusion on the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter based on the first weight comprises: Obtaining a second weight for the decoding feature obtained by increasing the sampling rate at the previous stage, based on the first weight; and, based on the first weight and the second weight, performing a weighted calculation on the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter. [4] Method according to claim 2, wherein performing the weighted fusion on the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter based on the first weight comprises: Summing the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter to obtain an initial merged feature; Obtaining a second weight for the decoding feature obtained by increasing the sampling rate at the previous stage, based on the first weight; and, based on the first weight and the second weight, performing a weighted calculation on the initial merged feature and the decoding feature obtained by increasing the sampling rate at the previous stage. [5] Method according to claim 1, wherein the process comprises performing the sampling rate reduction on the multiple stages on the first image, performing the sampling rate increase on the multiple stages on the encoding feature obtained by sampling rate reduction on the last stage, and obtaining the second image based on the decoding feature obtained by sampling rate increase on the last stage: Performing a sampling rate reduction at the multiple stages of the first image using a first encoding module of an enhancement network; Performing a sampling rate increase at the multiple stages of the encoding feature obtained by sampling rate decrease at the last stage using a decoding module of the enhancement network, wherein The sampling rate increase at each non-first stage comprises obtaining the merged feature by merging the decoding feature obtained by sampling rate increase at the previous stage with the encoding feature of the same sampling rate parameter using a merging module of the enhancement network and performing a sampling rate increase on the merged feature; and the process of obtaining the merged feature at at least one non-first stage comprises merging, by the merging module, the decoding feature obtained by sampling rate increase at the previous stage with the encoding feature of the same sampling rate parameter according to the pre-learned merging parameter; and Processing the decoding feature obtained by increasing the sampling rate at the last stage to obtain the second image, using an output module of the enhancement network. [6] Method according to claim 5, wherein the improvement network is obtained by means of a training method comprising: based on high-quality images in a first image set, performing unsupervised training on an initial network obtained by a second encoding module, the decoding module, and the output module to obtain a pre-trained network, wherein the first image set comprises several low-quality images and high-quality images corresponding to all low-quality images; and the second encoding module is configured to perform a multi-stage sample rate reduction on an input high-quality image to obtain multiple encoding features with different sample rate parameters; and, Based on the pre-trained network, supervised training is performed on the first encoding module and the fusion module using the first image set to obtain a trained first encoding module and a trained fusion module, wherein the trained first encoding module, the trained fusion module, the decoding module, and the output module in the pre-trained network form the improved network. [7] Method according to claim 6, wherein performing supervised training on the first coding module based on the pre-trained network using the first image set comprises: For a low-quality image in the first image set, input a low-quality image into the first coding module to obtain multiple coding features with different sampling rate parameters of a low-quality image; Inputting a high-quality image corresponding to a low-quality image into the second coding module in the pretrained network to obtain multiple coding features with different sampling rate parameters of the high-quality image corresponding to the low-quality image; and Updating a parameter of the first coding module with the aim of minimizing an initial difference between a coding feature of a low-quality image and a coding feature belonging to the high-quality image corresponding to a low-quality image and having the same sampling rate parameter as the coding feature of a low-quality image. [8] Electronic device comprising: a memory designed to store a computer program; and one or more processors configured to perform the following actions when the computer program is executed: Performing a multi-stage sampling rate reduction on a first image to obtain multiple encoding features with different sampling rate parameters; Performing a multi-stage sample rate increase on an encoding feature obtained by a sample rate decrease at a final stage to obtain multiple decoding features with different sample rate parameters, wherein a sample rate increase at each non-first stage comprises merging a decoding feature obtained by a sample rate increase at a previous stage with an encoding feature of the same sample rate parameter to obtain a merged feature, and performing a sample rate increase on the merged feature; and a process of obtaining a merged feature at at least one non-first stage comprises merging a decoding feature obtained by a sample rate increase at a previous stage with an encoding feature of the same sample rate parameter according to a pre-learned merge parameter; and Obtaining a second image based on a decoding feature obtained by increasing the sampling rate at a final stage, where the resolution of the second image is higher than the resolution of the first image. [9] Electronic device according to claim 8, wherein the one or more processors for merging the decoding feature obtained by increasing the sampling rate at the previous stage with the encoding feature of the same sampling rate parameter according to the pre-learned merging parameter are further configured to perform: based on the merging parameter, the decoding feature obtained by increasing the sampling rate at the previous stage, and the encoding feature of the same sampling rate parameter, calculate a first weight of the encoding feature that has the same sampling rate parameter as the decoding feature obtained by increasing the sampling rate at the previous stage; and Performing a weighted fusion on the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter based on the first weight. [10] Electronic device according to claim 9, wherein the one or more processors for performing the weighted fusion of the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter based on the first weight are further configured to perform: Obtaining a second weight for the decoding feature obtained by increasing the sampling rate at the previous stage, based on the first weight; and, based on the first weight and the second weight, performing a weighted calculation on the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter. [11] Electronic device according to claim 9, wherein the one or more processors for performing the weighted fusion of the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter based on the first weight are further configured to perform: Summing the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter to obtain an initial merged feature; Obtaining a second weight for the decoding feature obtained by increasing the sampling rate at the previous stage, based on the first weight; and, based on the first weight and the second weight, performing a weighted calculation on the initial merged feature and the decoding feature obtained by increasing the sampling rate at the previous stage. [12] Electronic device according to claim 8, wherein the one or more processors for the process of performing the sampling rate reduction on the multiple stages on the first image, performing the sampling rate increase on the multiple stages on the encoding feature obtained by sampling rate reduction on the last stage, and obtaining the second image based on the decoding feature obtained by sampling rate increase on the last stage are further configured to perform: Performing a sampling rate reduction at the multiple stages of the first image using a first encoding module of an enhancement network; Performing a sampling rate increase at the multiple stages of the encoding feature obtained by sampling rate decrease at the last stage using a decoding module of the enhancement network, wherein The sampling rate increase at each non-first stage comprises obtaining the merged feature by merging the decoding feature obtained by sampling rate increase at the previous stage with the encoding feature of the same sampling rate parameter using a merging module of the enhancement network and performing a sampling rate increase on the merged feature; and the process of obtaining the merged feature at at least one non-first stage comprises merging, by the merging module, the decoding feature obtained by sampling rate increase at the previous stage with the encoding feature of the same sampling rate parameter according to the pre-learned merging parameter; and Processing the decoding feature obtained by increasing the sampling rate at the last stage to obtain the second image, using an output module of the enhancement network. [13] Electronic device according to claim 12, wherein the improvement network is obtained by means of a training method comprising: based on high-quality images in a first image set, performing unsupervised training on an initial network obtained by a second encoding module, the decoding module, and the output module to obtain a pre-trained network, wherein the first image set comprises several low-quality images and high-quality images corresponding to all low-quality images; and the second encoding module is configured to perform a multi-stage sample rate reduction on an input high-quality image to obtain multiple encoding features with different sample rate parameters; and, Based on the pre-trained network, supervised training is performed on the first encoding module and the fusion module using the first image set to obtain a trained first encoding module and a trained fusion module, wherein the trained first encoding module, the trained fusion module, the decoding module, and the output module in the pre-trained network form the improved network. [14] Electronic device according to claim 13, wherein the one or more processors for performing supervised training on the first coding module based on the pre-trained network using the first image set are further configured to perform: For a low-quality image in the first image set, input a low-quality image into the first coding module to obtain multiple coding features with different sampling rate parameters of a low-quality image; Inputting a high-quality image corresponding to a low-quality image into the second coding module in the pretrained network to obtain multiple coding features with different sampling rate parameters of the high-quality image corresponding to the low-quality image; and Updating a parameter of the first coding module with the aim of minimizing an initial difference between a coding feature of a low-quality image and a coding feature belonging to the high-quality image corresponding to a low-quality image and having the same sampling rate parameter as the coding feature of a low-quality image. [15] Non-volatile, computer-readable storage medium containing a computer program which, when executed, causes one or more processors to perform: Performing a multi-stage sampling rate reduction on a first image to obtain multiple encoding features with different sampling rate parameters; Performing a multi-stage sample rate increase on an encoding feature obtained by a sample rate decrease at a final stage to obtain multiple decoding features with different sample rate parameters, wherein a sample rate increase at each non-first stage comprises merging a decoding feature obtained by a sample rate increase at a previous stage with an encoding feature of the same sample rate parameter to obtain a merged feature, and performing a sample rate increase on the merged feature; and a process of obtaining a merged feature at at least one non-first stage comprises merging a decoding feature obtained by a sample rate increase at a previous stage with an encoding feature of the same sample rate parameter according to a pre-learned merge parameter; and Obtaining a second image based on a decoding feature obtained by increasing the sampling rate at a final stage, where the resolution of the second image is higher than the resolution of the first image. [16] Storage medium according to claim 15, wherein the one or more processors for merging the decoding feature obtained by increasing the sampling rate at the previous stage with the encoding feature of the same sampling rate parameter according to the pre-learned merging parameter are further configured to perform: based on the merging parameter, the decoding feature obtained by increasing the sampling rate at the previous stage, and the encoding feature of the same sampling rate parameter, calculate a first weight of the encoding feature that has the same sampling rate parameter as the decoding feature obtained by increasing the sampling rate at the previous stage; and Performing a weighted fusion on the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter based on the first weight. [17] Storage medium according to claim 16, wherein the one or more processors for performing the weighted merging of the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter based on the first weight are further configured to perform: Obtaining a second weight for the decoding feature obtained by increasing the sampling rate at the previous stage, based on the first weight; and, based on the first weight and the second weight, performing a weighted calculation on the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter. [18] Storage medium according to claim 16, wherein the one or more processors for performing the weighted merging of the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter based on the first weight are further configured to perform: Summing the decoding feature obtained by increasing the sampling rate at the previous stage and the encoding feature of the same sampling rate parameter to obtain an initial merged feature; Obtaining a second weight for the decoding feature obtained by increasing the sampling rate at the previous stage, based on the first weight; and, based on the first weight and the second weight, performing a weighted calculation on the initial merged feature and the decoding feature obtained by increasing the sampling rate at the previous stage. [19] Storage medium according to claim 15, wherein the one or more processors for the process of performing the sampling rate reduction on the multiple stages on the first image, performing the sampling rate increase on the multiple stages on the encoding feature obtained by sampling rate reduction on the last stage, and obtaining the second image based on the decoding feature obtained by sampling rate increase on the last stage are further configured to perform: Performing a sampling rate reduction at the multiple stages of the first image using a first encoding module of an enhancement network; Performing a sampling rate increase at the multiple stages of the encoding feature obtained by sampling rate decrease at the last stage using a decoding module of the enhancement network, wherein The sampling rate increase at each non-first stage comprises obtaining the merged feature by merging the decoding feature obtained by sampling rate increase at the previous stage with the encoding feature of the same sampling rate parameter using a merging module of the enhancement network and performing a sampling rate increase on the merged feature; and the process of obtaining the merged feature at at least one non-first stage comprises merging, by the merging module, the decoding feature obtained by sampling rate increase at the previous stage with the encoding feature of the same sampling rate parameter according to the pre-learned merging parameter; and Processing the decoding feature obtained by increasing the sampling rate at the last stage to obtain the second image, using an output module of the enhancement network. [20] Storage medium according to claim 19, wherein the improvement network is obtained by means of a training method comprising: based on high-quality images in a first image set, performing unsupervised training on an initial network obtained by a second encoding module, the decoding module, and the output module to obtain a pre-trained network, wherein the first image set comprises several low-quality images and high-quality images corresponding to all low-quality images; and the second encoding module is configured to perform a multi-stage sample rate reduction on an input high-quality image to obtain multiple encoding features with different sample rate parameters; and, Based on the pre-trained network, supervised training is performed on the first encoding module and the fusion module using the first image set to obtain a trained first encoding module and a trained fusion module, wherein the trained first encoding module, the trained fusion module, the decoding module, and the output module in the pre-trained network form the improved network.

Images