An image processing method and apparatus

Abstract

Embodiments of the present application provide an image processing method and device, and relate to the technical field of image processing. The method comprises: performing feature extraction on a to-be-processed image from multiple different spatial scales respectively, to obtain a target feature and at least one to-be-fused feature; fusing the target feature and the at least one to-be-fused feature to obtain a first feature; extracting a high-frequency feature and a low-frequency feature in the target feature; processing the high-frequency feature based on a residual dense block (RDB) to obtain a second feature; fusing the low-frequency feature and the at least one to-be-fused feature to obtain a third feature; merging the first feature, the second feature and the third feature to obtain a fused feature; and processing the to-be-processed image based on the fused feature. The embodiments of the present application are used to improve the effect of image processing.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image processing method and apparatus. Background Technology

[0002] Image restoration refers to the repair and reconstruction of damaged images or the removal of unwanted objects from images.

[0003] Traditional image processing methods include those based on partial differential equations, global variational equations, and texture synthesis. However, these methods are generally inefficient, and prior information in the image is easily lost. To address the issues of lost prior information and low computational efficiency in traditional image processing methods, deep learning-based methods have been widely applied to various computer vision tasks, including image inpainting. However, due to the ineffective utilization of high-frequency information in images, the performance of current deep learning-based image inpainting network models in detail generation still needs improvement. Summary of the Invention

[0004] In view of this, the present invention provides an image processing method and apparatus for improving the effect of image processing.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] In a first aspect, embodiments of the present invention provide an image processing method, comprising:

[0007] Feature extraction is performed on the image to be processed from multiple different spatial scales to obtain target features and at least one feature to be fused.

[0008] The target feature and the at least one feature to be fused are fused to obtain a first feature;

[0009] Extract the high-frequency and low-frequency features from the target features;

[0010] The high-frequency features are processed based on the residual dense block RDB to obtain the second feature;

[0011] The low-frequency feature and the at least one feature to be fused are fused to obtain a third feature;

[0012] Merge the first feature, the second feature, and the third feature to obtain a fused feature;

[0013] The image to be processed is processed based on the fusion features.

[0014] As an optional implementation of this invention, the extraction of high-frequency and low-frequency features from the target features includes:

[0015] The target features are subjected to discrete wavelet decomposition to obtain the fourth feature;

[0016] The features of the first preset number of channels of the fourth feature are determined as the low-frequency features, and the features of the other channels of the fourth feature other than the low-frequency features are determined as the high-frequency features.

[0017] As an optional implementation of this invention, after extracting the high-frequency and low-frequency features from the target features, the method further includes:

[0018] The high-frequency features and the low-frequency features are processed separately through convolutional layers to reduce the number of channels of the high-frequency features and the low-frequency features to a preset value.

[0019] As an optional implementation of this invention, fusing the low-frequency feature and the at least one feature to be fused to obtain a third feature includes:

[0020] The at least one feature to be fused is sorted in descending order according to the spatial scale difference between the at least one feature to be fused and the low-frequency feature to obtain a first sorting result;

[0021] By fusing the first feature to be fused and the low-frequency feature, a fused feature corresponding to the first feature to be fused is obtained, wherein the first feature to be fused is the first feature to be fused in the first sorting result;

[0022] One by one, the other features to be fused in the first sorting result and the fused feature corresponding to the previous feature to be fused are merged to obtain the fused feature corresponding to the other features to be fused in the first sorting result;

[0023] The fusion feature corresponding to the last feature to be fused in the first sorting result is determined as the third feature.

[0024] As an optional implementation of this invention, the step of fusing the first feature to be fused and the low-frequency feature to obtain the fused feature corresponding to the first feature to be fused includes:

[0025] The low-frequency feature is sampled as a first sampling feature; the first sampling feature has the same spatial scale as the first feature to be fused.

[0026] Calculate the difference between the first sampled feature and the first feature to be fused to obtain the first difference feature;

[0027] The first difference feature is sampled as a second sampling feature; the second sampling feature has the same spatial scale as the low-frequency feature.

[0028] The low-frequency feature and the second sampling feature are added and fused to generate the fused feature corresponding to the first feature to be fused.

[0029] As an optional implementation of this invention, the step of sequentially fusing other features to be fused in the first sorting result with the fusion feature corresponding to the previous feature to be fused, to obtain the fusion feature corresponding to the other features to be fused in the first sorting result, includes:

[0030] The fusion feature corresponding to the (m-1)th feature to be fused in the first sorting result is sampled as the third sampled feature; the third sampled feature has the same spatial scale as the mth feature to be fused in the first sorting result, where m is an integer greater than 1;

[0031] Calculate the difference between the m-th feature to be fused and the third sampled feature to obtain the second difference feature;

[0032] The second difference feature is sampled as a fourth sampling feature; the fourth sampling feature has the same spatial scale as the fusion feature corresponding to the (m-1)th feature to be fused.

[0033] The fusion feature corresponding to the (m-1)th feature to be fused and the fourth sampling feature are added and fused to generate the fusion feature corresponding to the mth feature to be fused.

[0034] As an optional implementation of this invention, the step of fusing the target feature and the at least one feature to be fused to obtain a first feature includes:

[0035] The target features are divided into the fifth feature and the sixth feature;

[0036] The fifth feature is processed based on the residual dense block RDB to obtain the seventh feature;

[0037] The sixth feature and the at least one feature to be fused are fused to obtain the eighth feature;

[0038] The seventh feature and the eighth feature are combined to generate the first feature.

[0039] As an optional implementation of this invention, fusing the sixth feature and the at least one feature to be fused to obtain the eighth feature includes:

[0040] The at least one feature to be fused is sorted in descending order according to the spatial scale difference between the at least one feature to be fused and the sixth feature to obtain a second sorting result;

[0041] By fusing the second feature to be fused and the sixth feature, a fused feature corresponding to the second feature to be fused is obtained, wherein the second feature to be fused is the first feature to be fused in the second sorting result;

[0042] One by one, the other features to be fused in the second sorting result and the fused features corresponding to the previous feature to be fused are merged to obtain the fused features corresponding to the other features to be fused in the second sorting result;

[0043] The fusion feature corresponding to the last feature to be fused in the second sorting result is determined as the eighth feature.

[0044] As an optional implementation of this invention, the step of fusing the second feature to be fused and the sixth feature to obtain the fused feature corresponding to the second feature to be fused includes:

[0045] The sixth feature is sampled as the fifth sampled feature, and the fifth sampled feature has the same spatial scale as the second feature to be fused.

[0046] Calculate the difference between the fifth sampling feature and the first feature to be fused in the second sorting result to obtain the third difference feature;

[0047] The third difference feature is sampled to obtain a sixth sampling feature, and the sixth sampling feature has the same spatial scale as the sixth feature.

[0048] The sixth feature and the sixth sampling feature are added together and fused to generate the fused feature corresponding to the second feature to be fused.

[0049] As an optional implementation of this invention, the step of successively fusing other features to be fused in the second sorting result with the fusion feature corresponding to the previous feature to be fused, to obtain the fusion feature corresponding to the other features to be fused in the second sorting result, includes:

[0050] The fusion feature corresponding to the (n-1)th feature to be fused in the second sorting result is sampled as the seventh sampling feature; the seventh sampling feature has the same spatial scale as the nth feature to be fused in the second sorting result, where n is an integer greater than 1;

[0051] Calculate the difference between the nth feature to be fused and the seventh sampled feature to obtain the fourth difference feature;

[0052] The fourth difference feature is sampled as the eighth sampling feature, and the eighth sampling feature has the same spatial scale as the fusion feature corresponding to the (n-1)th feature to be fused.

[0053] The fusion feature corresponding to the (n-1)th feature to be fused and the eighth sampling feature are added and fused to generate the fusion feature corresponding to the nth feature to be fused.

[0054] As an optional implementation of this invention, dividing the target feature into a fifth feature and a sixth feature includes:

[0055] Based on the feature channels of the target features, the target features are divided into the fifth feature and the sixth feature.

[0056] Secondly, embodiments of the present invention provide an image processing method, including:

[0057] The image to be processed is processed by an encoding module to obtain encoded features. The encoding module includes L cascaded encoders with different spatial scales. The i-th encoder is used to extract features from the image to be processed to obtain image features on the i-th encoder, and to obtain the fusion features output by all encoders before the i-th encoder. The fusion features of the i-th encoder are obtained by the image processing method according to any one of claims 1-11, and the fusion features of the i-th encoder are output to all encoders after the i-th encoder. L and i are both positive integers, and i≤L.

[0058] The encoded features are processed by a feature restoration module consisting of at least one residual block RDB to obtain restored features;

[0059] The restored features are processed by the decoding module to obtain the processed image of the image to be processed; wherein, the decoding module includes L cascaded decoders with different spatial scales, the j-th decoder is used to fuse the image features of the encoding module on the j-th encoder and the fusion results output by all decoders before the j-th decoder to generate the fusion result of the j-th decoder, and output the fusion result of the j-th decoder to all decoders after the j-th decoder.

[0060] As an optional implementation of this invention, the step of processing the restored features through a decoding module to obtain the processed result image of the image to be processed includes:

[0061] Divide the image features on the j-th decoder into the ninth feature and the tenth feature;

[0062] The ninth feature is processed based on the residual dense block RDB to obtain the eleventh feature;

[0063] The 10th feature is fused with the fusion results of all decoder outputs before the j-th decoder to obtain the 12th feature;

[0064] The eleventh feature and the twelfth feature are combined to generate the fusion result of the j-th decoder.

[0065] Thirdly, embodiments of the present invention provide an image processing apparatus, comprising:

[0066] The feature extraction unit is used to extract features from the image to be processed from multiple different spatial scales to obtain target features and at least one feature to be fused.

[0067] The first processing unit is used to fuse the target feature and the at least one feature to be fused to obtain a first feature;

[0068] The second processing unit is used to extract high-frequency features and low-frequency features from the target features, process the high-frequency features based on residual dense block (RDB) to obtain the second feature, and fuse the low-frequency features and the at least one feature to be fused to obtain the third feature.

[0069] A fusion unit is used to merge the first feature, the second feature, and the third feature to obtain a fused feature;

[0070] The third processing unit processes the image to be processed based on the fusion features.

[0071] As an optional implementation of this invention, the second processing unit is specifically used to perform discrete wavelet decomposition on the target feature to obtain the fourth feature;

[0072] The features of the first preset number of channels of the fourth feature are determined as the low-frequency features, and the features of the other channels of the fourth feature other than the low-frequency features are determined as the high-frequency features.

[0073] As an optional implementation of the present invention, the second processing unit is further configured to process the high-frequency features and the low-frequency features respectively through convolutional layers, so as to reduce the number of channels of the high-frequency features and the low-frequency features to a preset value.

[0074] As an optional implementation of this invention, the second processing unit is specifically configured to: sort the at least one feature to be fused in descending order according to the spatial scale difference between the at least one feature to be fused and the low-frequency feature to obtain a first sorting result; fuse the first feature to be fused and the low-frequency feature to obtain a fused feature corresponding to the first feature to be fused, wherein the first feature to be fused is the first feature to be fused in the first sorting result; fuse the other features to be fused in the first sorting result one by one with the fused feature corresponding to the previous feature to be fused to obtain the fused features corresponding to the other features to be fused in the first sorting result; and determine the fused feature corresponding to the last feature to be fused in the first sorting result as the third feature.

[0075] As an optional implementation of this invention, the second processing unit is specifically used to sample the low-frequency feature as a first sampling feature; the first sampling feature has the same spatial scale as the first feature to be fused; calculate the difference between the first sampling feature and the first feature to be fused to obtain a first difference feature; sample the first difference feature as a second sampling feature; the second sampling feature has the same spatial scale as the low-frequency feature; and add and fuse the low-frequency feature and the second sampling feature to generate a fused feature corresponding to the first feature to be fused.

[0076] As an optional implementation of this invention, the second processing unit is specifically configured to sample the fusion feature corresponding to the (m-1)th feature to be fused in the first sorting result as a third sampling feature; the third sampling feature has the same spatial scale as the m-th feature to be fused in the first sorting result, where m is an integer greater than 1; calculate the difference between the m-th feature to be fused and the third sampling feature to obtain a second difference feature; sample the second difference feature as a fourth sampling feature; the fourth sampling feature has the same spatial scale as the fusion feature corresponding to the (m-1)th feature to be fused; and add and fuse the fusion feature corresponding to the (m-1)th feature to be fused and the fourth sampling feature to generate the fusion feature corresponding to the m-th feature to be fused.

[0077] As an optional implementation of this invention, the first processing unit is specifically used to divide the target feature into a fifth feature and a sixth feature; process the fifth feature based on the residual dense block RDB to obtain a seventh feature; fuse the sixth feature and the at least one feature to be fused to obtain an eighth feature; and merge the seventh feature and the eighth feature to generate the first feature.

[0078] As an optional implementation of this invention, the first processing unit is specifically configured to sort the at least one feature to be fused in descending order according to the spatial scale difference between the at least one feature to be fused and the sixth feature, and obtain a second sorting result; fuse the second feature to be fused and the sixth feature to obtain a fused feature corresponding to the second feature to be fused, wherein the second feature to be fused is the first feature to be fused in the second sorting result; fuse the other features to be fused in the second sorting result one by one with the fused feature corresponding to the previous feature to be fused, and obtain the fused features corresponding to the other features to be fused in the second sorting result; and determine the fused feature corresponding to the last feature to be fused in the second sorting result as the eighth feature.

[0079] As an optional implementation of this invention, the first processing unit is specifically configured to sample the sixth feature as a fifth sampling feature, the fifth sampling feature having the same spatial scale as the second feature to be fused; calculate the difference between the fifth sampling feature and the first feature to be fused in the second sorting result to obtain the third difference feature; sample the third difference feature as a sixth sampling feature, the sixth sampling feature having the same spatial scale as the sixth feature; and add and fuse the sixth feature and the sixth sampling feature to generate a fused feature corresponding to the second feature to be fused.

[0080] As an optional implementation of this invention, the first processing unit is specifically configured to sample the fusion feature corresponding to the (n-1)th feature to be fused in the second sorting result as a seventh sampling feature; the seventh sampling feature has the same spatial scale as the nth feature to be fused in the second sorting result, where n is an integer greater than 1; calculate the difference between the nth feature to be fused and the seventh sampling feature to obtain a fourth difference feature; sample the fourth difference feature as an eighth sampling feature, where the eighth sampling feature has the same spatial scale as the fusion feature corresponding to the (n-1)th feature to be fused; and add and fuse the fusion feature corresponding to the (n-1)th feature to be fused and the eighth sampling feature to generate the fusion feature corresponding to the nth feature to be fused.

[0081] As an optional implementation of the present invention, the first processing unit is specifically used to divide the target feature into a fifth feature and a sixth feature based on the feature channel of the target feature.

[0082] Fourthly, embodiments of the present invention provide an image processing apparatus, comprising:

[0083] A feature extraction unit is used to process the image to be processed through the encoding module to obtain encoded features; wherein, the encoding module includes L cascaded encoders with different spatial scales, the i-th encoder is used to extract features from the image to be processed to obtain image features on the i-th encoder, and to obtain the fusion features output by all encoders before the i-th encoder, and to obtain the fusion features of the i-th encoder through the image processing method according to any one of claims 1-11, and to output the fusion features of the i-th encoder to all encoders after the i-th encoder, where L and i are both positive integers, and i≤L;

[0084] The feature processing unit is used to process the encoded features through a feature restoration module composed of at least one residual block RDB to obtain restored features;

[0085] An image generation unit is used to process the restored features through a decoding module to obtain a processed image of the image to be processed; wherein, the decoding module includes L cascaded decoders with different spatial scales, the j-th decoder is used to fuse the image features of the encoding module on the j-th encoder and the fusion results output by all decoders before the j-th decoder to generate the fusion result of the j-th decoder, and output the fusion result of the j-th decoder to all decoders after the j-th decoder.

[0086] As an optional implementation of this invention, the image generation unit is specifically used to divide the image features on the j-th decoder into a ninth feature and a tenth feature; process the ninth feature based on the residual dense block RDB to obtain an eleventh feature; fuse the tenth feature with the fusion results of all decoder outputs before the j-th decoder to obtain a twelfth feature; and merge the eleventh feature and the twelfth feature to generate the fusion result of the j-th decoder.

[0087] Fifthly, embodiments of the present invention provide an electronic device, including: a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to cause the electronic device to implement any of the above-described image processing methods when the computer program is invoked.

[0088] Sixthly, embodiments of the present invention provide a computer-readable storage medium that, when executed by a computing device, causes the computing device to implement any of the above-described image processing methods.

[0089] In a seventh aspect, embodiments of the present invention provide a computer program product that, when run on a computer, enables the computer to implement any of the above-described image processing methods.

[0090] The image processing method provided in this embodiment of the invention, after extracting target features and at least one feature to be fused from an image to be processed at multiple different spatial scales, proceeds as follows: First, the target features and the at least one feature to be fused are fused to obtain a first feature. Second, high-frequency and low-frequency features are extracted from the target features, and the high-frequency features are processed based on Residual Dense Block (RDB) to obtain a second feature. The low-frequency features and the at least one feature to be fused are then fused to obtain a third feature. Finally, the first feature, the second feature, and the third feature are merged to obtain a fused feature, and the image to be processed is then processed based on the fused feature. Since feature processing based on RDB can perform feature updates and generate redundant features, and fusing low-frequency features and features to be fused can introduce effective information from features at other spatial scales, achieving multi-scale feature fusion, the image processing method provided in this embodiment of the invention can ensure the generation of new high-frequency features when achieving multi-scale feature fusion of low-frequency features. Furthermore, fusing the target features and the at least one feature to be fused can further introduce effective information from features at other spatial scales. Therefore, the image processing method provided in this embodiment of the invention can improve the image processing effect. Attached Figure Description

[0091] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0092] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0093] Figure 1 This is one of the flowcharts of the image processing method provided in the embodiments of the present invention;

[0094] Figure 2 This is one of the structural schematic diagrams of the feature fusion network provided in the embodiments of the present invention;

[0095] Figure 3 This is one of the data flow diagrams of the image processing method provided in the embodiments of the present invention;

[0096] Figure 4 This is the second schematic diagram of the data flow of the image processing method provided in the embodiments of the present invention;

[0097] Figure 5 This is the second flowchart of the image processing method provided in the embodiments of the present invention;

[0098] Figure 6 This is a second schematic diagram of the feature fusion network provided in an embodiment of the present invention;

[0099] Figure 7 A flowchart illustrating the steps of an image processing method provided in an embodiment of the present invention;

[0100] Figure 8 This is a schematic diagram of the structure of an image processing network provided in an embodiment of the present invention;

[0101] Figure 9 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of the present invention;

[0102] Figure 10 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of the present invention;

[0103] Figure 11 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0104] To better understand the above-mentioned objectives, features, and advantages of the present invention, the solutions of the present invention will be further described below. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0105] Many specific details are set forth in the following description in order to provide a full understanding of the invention, but the invention may also be practiced in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of the invention, and not all embodiments.

[0106] In the embodiments of the present invention, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner. Furthermore, in the description of the embodiments of the present invention, unless otherwise stated, "a plurality of" means two or more.

[0107] This invention provides an image processing method, referring to... Figure 1 The flowchart of the image processing method shown is as follows: Figure 2 The diagram shown illustrates the structure of the feature fusion network. This image processing method includes:

[0108] S11. Extract features from the image to be processed from multiple different spatial scales to obtain target features and at least one feature to be fused.

[0109] Specifically, in this embodiment of the invention, the target feature refers to the feature that needs to be fused and enhanced, and the feature to be fused refers to the feature used to fuse and enhance the target feature. Specifically, feature extraction can be performed on the image to be processed based on feature extraction functions or feature extraction networks at different spatial scales to obtain the target feature and the at least one feature to be fused.

[0110] S12. The target feature and the at least one feature to be fused are fused to obtain the first feature.

[0111] In this embodiment of the invention, the method of fusing the target feature and the at least one feature to be fused is not limited. The target feature and the at least one feature to be fused can be fused by any feature fusion method.

[0112] S13. Extract the (High Freq) and (Low Freq) features from the target features.

[0113] Optionally, the implementation of step S13 (extracting high-frequency and low-frequency features from the target features) may include:

[0114] The target features are subjected to discrete wavelet decomposition to obtain the fourth feature;

[0115] The features of the first preset number of channels of the fourth feature are determined as the low-frequency features, and the features of the other channels of the fourth feature other than the low-frequency features are determined as the high-frequency features.

[0116] That is, firstly, the target feature (C*H*W) is decomposed into discrete wavelet decomposition, thereby converting the target feature into low-resolution feature (4C*1 / 2H*1 / 2W). Then, the features of the 1st to Kth channels are determined as the low-frequency features, and the features of the (K+1)th to 4Cth channels are determined as the high-frequency features.

[0117] In this embodiment of the invention, a feature channel refers to the feature map contained in the feature. A feature channel is a feature map obtained by extracting features from a feature based on a certain dimension. Therefore, a feature channel is a feature map in a specific sense.

[0118] For example, if the size of the target feature is 16*H*W and the size of the fourth feature is 64*H / 2*W / 2, then the features of channels 1-16 can be determined as the low-frequency features, and the features of channels 17-48 can be determined as the high-frequency features.

[0119] As an optional implementation of the present invention, the image processing method provided by the present invention further includes:

[0120] The high-frequency features and the low-frequency features are processed separately through convolutional layers to reduce the number of channels of the high-frequency features and the low-frequency features to a preset value.

[0121] For example, the preset value can be 8. That is, the number of channels of the high-frequency feature and the low-frequency feature are compressed to 8 through two convolutional layers respectively.

[0122] Optionally, the kernel size of the convolutional layer used to process the high-frequency features and the low-frequency features is 3*3, and the stride is 2.

[0123] Reducing the number of channels for the high-frequency and low-frequency features to a preset value can reduce the amount of data processing during feature fusion, thereby improving the efficiency of feature fusion.

[0124] S14. Process the high-frequency features based on the Residual Dense Block (RDB) to obtain the second feature.

[0125] Specifically, the residual dense block in this embodiment of the invention comprises three main parts: Contiguous Memory (CM), Local Feature Fusion (LFF), and Local Residual Learning (LRL). CM is primarily used to send the output of the previous RDB to each convolutional layer of the current RDB; LFF is primarily used to fuse the output of the previous RDB with the outputs of all convolutional layers of the current RDB; and LRL is primarily used to add and fuse the output of the previous RDB with the output of the current RDB's LFF, and use the fused result as the output of the current RDB.

[0126] Since RDB can perform feature updates and generate redundant features, processing high-frequency features based on residual dense blocks can increase the diversity of high-frequency features, thereby enriching the details in the resulting image.

[0127] S15. The low-frequency feature and the at least one feature to be fused are fused to obtain a third feature.

[0128] As an optional implementation of this invention, step S15 (fusing the low-frequency feature and the at least one feature to be fused to obtain a third feature) includes the following steps a to d:

[0129] Step a: Sort the at least one feature to be fused in descending order according to the spatial scale difference between the at least one feature to be fused and the low-frequency feature to obtain the first sorting result.

[0130] The spatial scale difference between the feature to be fused and the low-frequency feature refers to the difference between the spatial scale of the feature to be fused and the spatial scale of the low-frequency feature.

[0131] That is, if the spatial scale of a certain feature to be fused differs more from the spatial scale of the low-frequency feature, the feature to be fused will be ranked higher in the first ranking result; conversely, if the spatial scale of a certain feature to be fused differs less from the spatial scale of the low-frequency feature, the feature to be fused will be ranked lower in the first ranking result.

[0132] Step b: Fuse the first feature to be fused and the low-frequency feature to obtain the fused feature corresponding to the first feature to be fused.

[0133] The first feature to be fused is the first feature to be fused in the first sorting result.

[0134] Reference Figure 3 As shown, Figure 3 In the first ranking result, the first feature to be fused (the first feature to be fused) is designated J0, and the low-frequency feature is designated j. n2 The following explains step b above. Step b can be implemented in ways including steps 1 through 4 below:

[0135] Step 1: Transfer the low-frequency feature j n2 Sampling is the first sampling feature

[0136] Wherein, the first sampling feature It has the same spatial scale as the first feature J0 to be fused.

[0137] It should be noted that the sampling in the above steps can be either upsampling or downsampling, specifically determined by the spatial scale and low-frequency feature j of the first J0 to be fused in the first sorting result. n2 The spatial scale determines this.

[0138] Step 2: Calculate the first sampling feature The first difference feature is obtained by taking the difference between the first feature J0 to be fused in the first sorting result and the first difference feature.

[0139] The process of step 2 above can be described as follows:

[0140]

[0141] Step 3: The first difference feature Sampling is the second sampling feature

[0142] Wherein, the second sampling feature With the low-frequency feature j n2 They have the same spatial scale.

[0143] Similarly, the sampling in the above steps can be either upsampling or downsampling, depending on the first difference feature. Spatial scale and low-frequency characteristics j n2 The spatial scale determines this.

[0144] Step 4: For the low-frequency feature j n2 and the second sampling feature Perform addition and fusion to generate the fused feature J0 corresponding to the first feature J0 to be fused. n .

[0145] The process of step 4 above can be described as follows:

[0146]

[0147] Step c: Merge the other features to be merged in the first sorting result one by one with the fusion feature corresponding to the previous feature to be merged, and obtain the fusion feature corresponding to the other features to be merged in the first sorting result.

[0148] Optionally, the method for fusing the m-th (positive integer greater than 1) feature to be fused in the first sorting result and the fusion feature corresponding to the previous feature to be fused (m-1th feature to be fused) in step c above includes the following steps I to VI:

[0149] Step I: Sample the fusion feature corresponding to the (m-1)th feature to be fused in the first sorting result as the third sampled feature.

[0150] The third sampling feature has the same spatial scale as the m-th feature to be fused in the first sorting result.

[0151] Step II: Calculate the difference between the m-th feature to be fused and the third sampled feature to obtain the second difference feature.

[0152] Step III: Sample the second difference feature as the fourth sampling feature.

[0153] The fourth sampling feature has the same spatial scale as the fusion feature corresponding to the (m-1)th feature to be fused.

[0154] Step VI: Add and fuse the fusion feature corresponding to the (m-1)th feature to be fused and the fourth sampling feature to generate the fusion feature corresponding to the mth feature to be fused.

[0155] The difference between obtaining the fusion result of the m-th feature to be fused in the first sorting result in steps I to VI and obtaining the fusion result of the 1-th feature to be fused in the first sorting result in steps 1 to 4 is only that: when obtaining the fusion result of the first feature to be fused, the input is the third feature and the first feature to be fused, while when obtaining the fusion result of the m-th feature to be fused, the input is the fusion feature corresponding to the (m-1)-th feature to be fused and the m-th feature to be fused. The rest of the calculation methods are the same.

[0156] For example, refer to Figure 4 As shown, Figure 4 The first sorted result includes, in order: feature to be fused J0, feature to be fused J1, feature to be fused J2, ..., feature to be fused J... t Let's take an example to illustrate step c above. Figure 3 Based on the illustrated embodiment, the fusion feature J0 corresponding to the first feature to be fused in the first sorting result is obtained. n Then, the process of obtaining the fusion features corresponding to other features to be fused in the first sorting result includes:

[0157] The fusion result J0 of the first feature J0 to be fused in the first sorting result. n The sampled feature is the same as the second feature to be fused J1 in terms of spatial scale, and the first sampled feature corresponding to the second feature to be fused is generated.

[0158] Calculate the first sampled feature corresponding to the second feature to be fused J1. The difference is used to obtain the difference feature corresponding to the second feature to be fused.

[0159] The difference feature corresponding to the second feature to be fused, J1. The sample is the fusion result J0 with the first feature to be fused J0. n For features with the same spatial scale, obtain the second sampled feature corresponding to the second feature J1 to be fused.

[0160] The fusion result J0 of the first feature to be fused J0 n The second sampled feature corresponding to the second feature to be fused, J1. Perform addition and fusion to generate the fusion result J1 of the second feature J1 to be fused. n ;

[0161] The fusion result J1 of the second feature to be fused J1 n The sampled feature is the same as the feature with the same spatial scale as the third feature to be fused (J2), and the first sampled feature corresponding to the third feature to be fused is generated.

[0162] Calculate the first sampled feature corresponding to the third feature to be fused, J2. The difference is used to obtain the difference feature corresponding to the third feature to be fused.

[0163] The difference feature corresponding to the third feature to be fused, J2. The sample is the fusion result J1 with the second feature to be fused J1. n For features with the same spatial scale, obtain the second sampled feature corresponding to the third feature to be fused, J2.

[0164] The fusion result J1 of the second feature to be fused J1 n The second sampled feature corresponding to the third feature to be fused, J2. Perform addition and fusion to generate the fusion result J2 of the third feature J2 to be fused. n ;

[0165] Based on the above method, the 4th feature J3, the 5th feature J4, ..., the tth feature J in the first sorting result are obtained one by one. t-1 and the (t+1)th feature J to be fused t The fusion result J t n .

[0166] Step d: Determine the fusion feature corresponding to the last feature to be fused in the first sorting result as the third feature.

[0167] Continuing from above Figure 4 In the illustrated embodiment, the first sorting result sequentially includes: feature J0 to be fused, feature J1 to be fused, feature J2 to be fused, ..., feature J... t Therefore, the last feature J to be fused in the first sorting result is... t The fusion result J t n This is identified as the third feature.

[0168] That is, in this embodiment of the invention, feature processing is performed in two feature processing branches. One feature processing branch performs the feature processing step S12 above, while the other feature processing branch performs the feature processing steps S13 to S15 above.

[0169] It should be noted that the embodiments of the present invention do not limit the order in which the feature processing steps of the two feature processing branches are executed. Steps S13 to S15 can be executed first, followed by step S12, or step S12 can be executed first, followed by steps S13 to S15, or they can be executed simultaneously.

[0170] S16. Merge the second feature, the third feature, and the first feature to obtain a fused feature.

[0171] Specifically, merging the second feature, the third feature, and the first feature may include: concatenating the second feature, the third feature, and the first feature in the channel dimension.

[0172] S17. Process the image to be processed based on the fusion features.

[0173] This invention provides an image processing method applicable to any image processing scenario. For example, the image processing method provided in this invention can be an image dehazing method; another example is that the image processing method provided in this invention can also be an image enhancement method; yet another example is that the image processing method provided in this invention can also be an image super-resolution method.

[0174] The image processing method provided in this embodiment of the invention, after extracting target features and at least one feature to be fused from an image to be processed at multiple different spatial scales, proceeds as follows: First, the target features and the at least one feature to be fused are fused to obtain a first feature. Second, high-frequency and low-frequency features are extracted from the target features, and the high-frequency features are processed based on Residual Dense Block (RDB) to obtain a second feature. The low-frequency features and the at least one feature to be fused are then fused to obtain a third feature. Finally, the first feature, the second feature, and the third feature are merged to obtain a fused feature, and the image to be processed is then processed based on the fused feature. Since feature processing based on RDB can perform feature updates and generate redundant features, and fusing low-frequency features and features to be fused can introduce effective information from features at other spatial scales, achieving multi-scale feature fusion, the image processing method provided in this embodiment of the invention can ensure the generation of new high-frequency features when achieving multi-scale feature fusion of low-frequency features. Furthermore, fusing the target features and the at least one feature to be fused can further introduce effective information from features at other spatial scales. Therefore, the image processing method provided in this embodiment of the invention can improve the image processing effect.

[0175] As an extension and refinement of the above embodiments, this invention provides another image processing method, referring to... Figure 5 The flowchart of the image processing method shown is as follows: Figure 6 The diagram shows the structure of the feature fusion network. This image processing method includes the following steps:

[0176] S51. Extract features from the image to be processed from multiple different spatial scales to obtain target features and at least one feature to be fused.

[0177] S52. Divide the target feature into the fifth feature and the sixth feature.

[0178] Optionally, dividing the target feature into a fifth feature and a sixth feature includes:

[0179] Based on the feature channels of the target features, the target features are divided into the fifth feature and the sixth feature.

[0180] In this embodiment of the invention, the ratio of the fifth feature to the sixth feature is not limited. A higher ratio of the fifth feature allows for the generation of more new features, and a higher ratio of the sixth feature allows for the incorporation of more effective information from features at other spatial scales. Therefore, in practical applications, the ratio of the fifth feature to the sixth feature can be determined based on the amount of effective information from features at other spatial scales that need to be introduced and the amount of new features that need to be generated. For example, the ratio of the fifth feature to the sixth feature can be 1:1.

[0181] S53. Process the fifth feature based on the residual dense block to obtain the seventh feature.

[0182] S54. The sixth feature and the at least one feature to be fused are fused to obtain the eighth feature.

[0183] As an optional implementation of this invention, step S54 (fusing the sixth feature and the at least one feature to be fused to obtain the eighth feature) includes:

[0184] The at least one feature to be fused is sorted in descending order according to the spatial scale difference between the at least one feature to be fused and the sixth feature to obtain a second sorting result;

[0185] By fusing the second feature to be fused and the sixth feature, a fused feature corresponding to the second feature to be fused is obtained, wherein the second feature to be fused is the first feature to be fused in the second sorting result;

[0186] One by one, the other features to be fused in the second sorting result and the fused features corresponding to the previous feature to be fused are merged to obtain the fused features corresponding to the other features to be fused in the second sorting result;

[0187] The fusion feature corresponding to the last feature to be fused in the second sorting result is determined as the eighth feature.

[0188] Furthermore, the process of fusing the second feature to be fused and the sixth feature to obtain the fused feature corresponding to the second feature to be fused includes:

[0189] The sixth feature is sampled as the fifth sampled feature, and the fifth sampled feature has the same spatial scale as the second feature to be fused.

[0190] Calculate the difference between the fifth sampling feature and the first feature to be fused in the second sorting result to obtain the third difference feature;

[0191] The third difference feature is sampled to obtain a sixth sampling feature, and the sixth sampling feature has the same spatial scale as the sixth feature.

[0192] The sixth feature and the sixth sampling feature are added together and fused to generate the fused feature corresponding to the second feature to be fused.

[0193] Furthermore, the step of sequentially fusing other features to be fused in the second ranking result with the fusion feature corresponding to the previous feature to be fused, to obtain the fusion feature corresponding to the other features to be fused in the second ranking result, includes:

[0194] The fusion feature corresponding to the (n-1)th feature to be fused in the second sorting result is sampled as the seventh sampling feature; the seventh sampling feature has the same spatial scale as the nth feature to be fused in the second sorting result, where n is an integer greater than 1;

[0195] Calculate the difference between the nth feature to be fused and the seventh sampled feature to obtain the fourth difference feature;

[0196] The fourth difference feature is sampled as the eighth sampling feature, and the eighth sampling feature has the same spatial scale as the fusion feature corresponding to the (n-1)th feature to be fused.

[0197] The fusion feature corresponding to the (n-1)th feature to be fused and the eighth sampling feature are added and fused to generate the fusion feature corresponding to the nth feature to be fused.

[0198] The implementation method of fusing the sixth feature and at least one feature to be fused to obtain the eighth feature is similar to... Figure 1 The implementation method of fusing low-frequency features and at least one feature to be fused to obtain a third feature in the illustrated embodiment is similar. Therefore, the implementation method of step S54 in the above embodiment can refer to the implementation method of step S14 above, and will not be repeated here.

[0199] S55. Merge the seventh feature and the eighth feature to generate the first feature.

[0200] S56. Extract the high-frequency and low-frequency features from the target features.

[0201] S57. Process the high-frequency features based on the residual dense block to obtain the second feature.

[0202] S58. The low-frequency feature and the at least one feature to be fused are fused to obtain a third feature.

[0203] S59. Merge the first feature, the second feature, and the third feature to obtain the fused feature.

[0204] It should be noted that the above embodiment is illustrated by first merging the seventh feature and the eighth feature to generate the first feature, and then merging the second feature, the third feature and the first feature to generate the target feature and the fused feature. However, in actual execution, the second feature, the third feature, the seventh feature and the eighth feature can also be synthesized and merged through the same steps to generate the fused feature.

[0205] The image processing method provided in this embodiment of the invention, after extracting target features and at least one feature to be fused from an image to be processed at multiple different spatial scales, proceeds as follows: First, the target features and the at least one feature to be fused are fused to obtain a first feature. Second, high-frequency and low-frequency features are extracted from the target features, and the high-frequency features are processed based on Residual Dense Block (RDB) to obtain a second feature. The low-frequency features and the at least one feature to be fused are then fused to obtain a third feature. Finally, the first feature, the second feature, and the third feature are merged to obtain a fused feature, and the image to be processed is then processed based on the fused feature. Since feature processing based on RDB can perform feature updates and generate redundant features, and fusing low-frequency features and features to be fused can introduce effective information from features at other spatial scales, achieving multi-scale feature fusion, the image processing method provided in this embodiment of the invention can ensure the generation of new high-frequency features when achieving multi-scale feature fusion of low-frequency features. Furthermore, fusing the target features and the at least one feature to be fused can further introduce effective information from features at other spatial scales. Therefore, the image processing method provided in this embodiment of the invention can improve the image processing effect.

[0206] It should also be noted that fusing features at multiple spatial scales generally requires upsampling / downsampling convolutions and deconvolutions, which consume significant computational resources, resulting in substantial performance overhead. The above embodiment, by dividing the target features into a fifth and a sixth feature, and only involving the sixth feature in multi-spatial-scale feature fusion, further reduces the number of features to be fused (the sixth feature has fewer features than the target feature), thereby reducing the computational load of feature fusion and improving its efficiency.

[0207] Based on the above embodiments, this invention also provides an image processing method. (Refer to...) Figure 7 As shown, the image processing method provided in this embodiment of the invention includes the following steps S71 to S73:

[0208] S71. The image to be processed is processed by the encoding module to obtain the encoded features.

[0209] The encoding module includes L cascaded encoders with different spatial scales. The m-th encoder is used to extract features from the image to be processed to obtain image features from the i-th encoder, and to obtain the fusion features output by all encoders before the i-th encoder. The fusion features of the i-th encoder are obtained by the image processing method according to any one of claims 1-11, and the fusion features of the i-th encoder are output to all encoders after the i-th encoder. L and i are both positive integers, and i≤L.

[0210] S72. The encoded features are processed by a feature restoration module consisting of at least one residual block RDB to obtain restored features.

[0211] S73. The restored features are processed by the decoding module to obtain the processed result image of the image to be processed.

[0212] The decoding module includes L cascaded decoders with different spatial scales. The j-th decoder is used to fuse the image features of the encoding module on the j-th encoder and the fusion results of all decoders before the j-th decoder to generate the fusion result of the j-th decoder, and output the fusion result of the j-th decoder to all decoders after the j-th decoder.

[0213] That is, used to perform the above Figure 7 The encoding module, feature restoration module, and decoding module in the illustrated embodiment form a U-Net.

[0214] Specifically, the U-Net is a special type of convolutional neural network. A U-Net mainly consists of an encoding module (also known as a shrinking path), a feature recovery module, and a decoding module (also known as an expanding path). The encoding module primarily captures context information from the original image, while the corresponding decoding module precisely localizes the parts of the original image that need to be segmented, thus generating the processed image. Compared to a fully convolutional neural network (FCN), the U-Net improves upon the U-Net by combining the features extracted from the encoding module with a new feature map during upsampling to accurately locate the parts to be segmented from the original image. This maximizes the preservation of important information in the features, thereby reducing the need for training samples and computational resources.

[0215] As an optional implementation of this invention, the step of processing the restored features through a decoding module to obtain the processed result image of the image to be processed includes:

[0216] Divide the image features on the j-th decoder into the ninth feature and the tenth feature;

[0217] The ninth feature is processed based on the residual dense block RDB to obtain the eleventh feature;

[0218] The 10th feature is fused with the fusion results of all decoder outputs before the j-th decoder to obtain the 12th feature;

[0219] The eleventh feature and the twelfth feature are combined to generate the fusion result of the j-th decoder.

[0220] Reference Figure 8 As shown, used to perform the above Figure 7 The network model of the embodiment shown includes: an encoding module 81, a feature restoration module 82, and a decoding module 83 forming a U-shaped network.

[0221] The encoding module 81 includes L cascaded encoders of different spatial scales, used to process the image I to obtain encoded features i. L The j-th decoder is used to fuse the image features of the encoding module on the j-th encoder with the fusion results of all decoders before the j-th decoder to generate the fusion result of the j-th decoder, and output the fusion result of the j-th decoder to all decoders after the j-th decoder.

[0222] The feature restoration module 82 includes at least one RDB for receiving the encoded feature i output by the encoding module 81. L and encoding feature i through the at least one RDB L Process the data to obtain the restored features j. L .

[0223] The decoding module 83 includes L cascaded decoders with different spatial scales. The j-th decoder is used to fuse the image features of the encoding module on the j-th encoder and the fusion results of all decoders before the j-th decoder to generate the fusion result of the j-th decoder, and output the fusion result of the j-th decoder to all decoders after the j-th decoder; and based on the fusion result j of the last decoder output... 1 , and obtain the processing result image J of the image to be processed I.

[0224] The operation of fusing the image features of the m-th encoder in the encoding module 81 with the fusion results output by all encoders before the m-th encoder (the 1st encoder to the (m-1th encoder)) using the image processing method provided in the above embodiment can be described as follows:

[0225] i m =i m1 +i m2

[0226]

[0227]

[0228] i m =i GF +i LF

[0229]

[0230]

[0231]

[0232] Among them, i m This represents the feature of encoding module 81 on the m-th encoder, i GF Indicates from i m The high-frequency features extracted are f(...), which represents the operation of processing the features based on RDB. Indicates i based on RDB GF The features obtained through processing, i LF Indicates from i mThe low-frequency features extracted from them This represents the fusion result output by the first encoder to the (m-1)th encoder. This indicates the feature fusion operation. Indicates i LF and The fusion result obtained by performing the fusion, i m1 Indicates i m The fifth feature obtained by segmentation, Indicates i based on RDB m1 The seventh feature obtained after processing, i m2 Indicates i m The sixth feature obtained by segmentation, Indicates i m2 and The fusion result obtained by fusion. The fusion result output by the m-th encoder of encoding module 81.

[0233] The operation of fusing the image features of the m-th decoder in the decoding module 83 with the fusion results output by all decoders before the m-th decoder (from the L-th decoder to the (m+1-th decoder)) using the image processing method provided in the above embodiment can be described as follows:

[0234] j m =j m1 +j m2

[0235]

[0236]

[0237]

[0238] Where, j m Let j represent the feature of decoding module 83 in the m-th decoder. m1 Indicates that for j m The ninth feature obtained from the partitioning, f(...), represents the operation of processing the feature based on RDB. Indicates that j is based on RDB m1 The eleven features obtained after processing, j m2 Indicates that for j m The tenth feature obtained from the division, where L is the total number of decoders in decoding module 83. This represents the fusion result of the outputs of the Lth decoder to the (m+1)th decoder. Indicates that for j m2 and Perform the fusion operation. Indicates that for j m2 and The fusion result obtained by fusion. This represents the fusion result output by the m-th decoder of decoding module 83.

[0239] Since the image processing method provided in this embodiment of the invention can perform feature fusion through the image processing method provided in the above embodiment, the image processing method provided in this embodiment of the invention can ensure the generation of new high-frequency features when realizing multi-scale feature fusion of low-frequency features. Therefore, the image processing method provided in this embodiment of the invention can improve the effect of image processing.

[0240] Based on the same inventive concept, as an implementation of the above method, this embodiment of the invention also provides an image processing device. This device embodiment corresponds to the aforementioned method embodiment. For ease of reading, this device embodiment will not repeat the details of the aforementioned method embodiment one by one, but it should be clear that the image processing device in this embodiment can correspondingly implement all the contents of the aforementioned method embodiment.

[0241] This invention provides an image processing apparatus. Figure 9 This is a schematic diagram of the image processing device, as shown below. Figure 9 As shown, the image processing apparatus 900 includes:

[0242] The feature extraction unit 91 is used to extract features from the image to be processed from multiple different spatial scales to obtain target features and at least one feature to be fused.

[0243] The first processing unit 92 is used to fuse the target feature and the at least one feature to be fused to obtain a first feature;

[0244] The second processing unit 93 is used to extract high-frequency features and low-frequency features from the target features, process the high-frequency features based on residual dense block RDB to obtain the second feature, and fuse the low-frequency features and the at least one feature to be fused to obtain the third feature.

[0245] Fusion unit 94 is used to merge the first feature, the second feature and the third feature to obtain a fused feature;

[0246] The third processing unit 95 processes the image to be processed based on the fusion features.

[0247] As an optional implementation of this invention, the second processing unit 93 is specifically used to perform discrete wavelet decomposition on the target feature to obtain the fourth feature;

[0248] The features of the first preset number of channels of the fourth feature are determined as the low-frequency features, and the features of the other channels of the fourth feature other than the low-frequency features are determined as the high-frequency features.

[0249] As an optional implementation of the present invention, the second processing unit 93 is further configured to process the high-frequency features and the low-frequency features respectively through convolutional layers, so as to reduce the number of channels of the high-frequency features and the low-frequency features to a preset value.

[0250] As an optional implementation of this invention, the second processing unit 93 is specifically configured to sort the at least one feature to be fused in descending order according to the spatial scale difference between the at least one feature to be fused and the low-frequency feature to obtain a first sorting result; fuse the first feature to be fused and the low-frequency feature to obtain a fused feature corresponding to the first feature to be fused, wherein the first feature to be fused is the first feature to be fused in the first sorting result; fuse the other features to be fused in the first sorting result one by one with the fused feature corresponding to the previous feature to be fused to obtain the fused features corresponding to the other features to be fused in the first sorting result; and determine the fused feature corresponding to the last feature to be fused in the first sorting result as the third feature.

[0251] As an optional implementation of this invention, the second processing unit 93 is specifically used to sample the low-frequency feature as a first sampling feature; the first sampling feature has the same spatial scale as the first feature to be fused; calculate the difference between the first sampling feature and the first feature to be fused to obtain a first difference feature; sample the first difference feature as a second sampling feature; the second sampling feature has the same spatial scale as the low-frequency feature; and add and fuse the low-frequency feature and the second sampling feature to generate a fused feature corresponding to the first feature to be fused.

[0252] As an optional implementation of this invention, the second processing unit 93 is specifically configured to sample the fusion feature corresponding to the (m-1)th feature to be fused in the first sorting result as a third sampling feature; the third sampling feature has the same spatial scale as the m-th feature to be fused in the first sorting result, where m is an integer greater than 1; calculate the difference between the m-th feature to be fused and the third sampling feature to obtain a second difference feature; sample the second difference feature as a fourth sampling feature; the fourth sampling feature has the same spatial scale as the fusion feature corresponding to the (m-1)th feature to be fused; and add and fuse the fusion feature corresponding to the (m-1)th feature to be fused and the fourth sampling feature to generate the fusion feature corresponding to the m-th feature to be fused.

[0253] As an optional implementation of this invention, the first processing unit 92 is specifically used to divide the target feature into a fifth feature and a sixth feature; process the fifth feature based on the residual dense block RDB to obtain a seventh feature; fuse the sixth feature and the at least one feature to be fused to obtain an eighth feature; and merge the seventh feature and the eighth feature to generate the first feature.

[0254] As an optional implementation of this invention, the first processing unit 92 is specifically configured to sort the at least one feature to be fused in descending order according to the spatial scale difference between the at least one feature to be fused and the sixth feature, and obtain a second sorting result; fuse the second feature to be fused and the sixth feature to obtain a fused feature corresponding to the second feature to be fused, wherein the second feature to be fused is the first feature to be fused in the second sorting result; fuse the other features to be fused in the second sorting result one by one with the fused feature corresponding to the previous feature to be fused, and obtain the fused features corresponding to the other features to be fused in the second sorting result; and determine the fused feature corresponding to the last feature to be fused in the second sorting result as the eighth feature.

[0255] As an optional implementation of this invention, the first processing unit 92 is specifically configured to sample the sixth feature as a fifth sampling feature, the fifth sampling feature having the same spatial scale as the second feature to be fused; calculate the difference between the fifth sampling feature and the first feature to be fused in the second sorting result to obtain the third difference feature; sample the third difference feature as a sixth sampling feature, the sixth sampling feature having the same spatial scale as the sixth feature; and add and fuse the sixth feature and the sixth sampling feature to generate a fused feature corresponding to the second feature to be fused.

[0256] As an optional implementation of this invention, the first processing unit 92 is specifically configured to sample the fusion feature corresponding to the (n-1)th feature to be fused in the second sorting result as a seventh sampling feature; the seventh sampling feature has the same spatial scale as the nth feature to be fused in the second sorting result, where n is an integer greater than 1; calculate the difference between the nth feature to be fused and the seventh sampling feature to obtain a fourth difference feature; sample the fourth difference feature as an eighth sampling feature, where the eighth sampling feature has the same spatial scale as the fusion feature corresponding to the (n-1)th feature to be fused; and add and fuse the fusion feature corresponding to the (n-1)th feature to be fused and the eighth sampling feature to generate the fusion feature corresponding to the nth feature to be fused.

[0257] As an optional implementation of the present invention, the first processing unit 92 is specifically used to divide the target feature into a fifth feature and a sixth feature based on the feature channel of the target feature.

[0258] The image processing apparatus provided in this embodiment can execute the image processing method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0259] Based on the same inventive concept, as an implementation of the above method, this embodiment of the invention also provides an image processing device. This device embodiment corresponds to the aforementioned method embodiment. For ease of reading, this device embodiment will not repeat the details of the aforementioned method embodiment one by one, but it should be clear that the image processing device in this embodiment can correspondingly implement all the contents of the aforementioned method embodiment.

[0260] This invention provides an image processing apparatus. Figure 10 This is a schematic diagram of the image processing device, as shown below. Figure 10 As shown, the image processing apparatus 100 includes:

[0261] The feature extraction unit 101 is used to process the image to be processed by the encoding module to obtain encoded features; wherein, the encoding module includes L cascaded encoders with different spatial scales, the i-th encoder is used to extract features from the image to be processed to obtain image features on the i-th encoder, and to obtain the fusion features output by all encoders before the i-th encoder, and to obtain the fusion features of the i-th encoder by the image processing method according to any one of claims 1-11, and to output the fusion features of the i-th encoder to all encoders after the i-th encoder, where L and i are both positive integers, and i≤L;

[0262] Feature processing unit 102 is used to process the encoded features through a feature restoration module composed of at least one residual block RDB to obtain restored features;

[0263] The image generation unit 103 is used to process the restored features through the decoding module to obtain the processed result image of the image to be processed; wherein, the decoding module includes L cascaded decoders with different spatial scales, the j-th decoder is used to fuse the image features of the encoding module on the j-th encoder and the fusion results output by all decoders before the j-th decoder to generate the fusion result of the j-th decoder, and output the fusion result of the j-th decoder to all decoders after the j-th decoder.

[0264] As an optional implementation of this invention, the image generation unit 103 is specifically used to divide the image features on the j-th decoder into a ninth feature and a tenth feature; process the ninth feature based on the residual dense block RDB to obtain an eleventh feature; fuse the tenth feature with the fusion results of all decoder outputs before the j-th decoder to obtain a twelfth feature; and merge the eleventh feature and the twelfth feature to generate the fusion result of the j-th decoder.

[0265] The image processing apparatus provided in this embodiment can execute the image processing method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0266] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 11 As shown, the electronic device provided in this embodiment includes a memory 111 and a processor 112. The memory 111 is used to store computer programs; the processor 112 is used to execute the image processing method provided in the above embodiment when the computer program is invoked.

[0267] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the computing device to implement the image processing method provided in the above embodiments.

[0268] Based on the same inventive concept, this embodiment of the invention also provides a computer program product, which, when run on a computer, enables the computing device to implement the image processing method provided in the above embodiments.

[0269] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0270] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0271] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0272] Computer-readable media include both permanent and non-permanent, removable and non-removable storage media. Storage media can store information using any method or technology; the information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media do not include transient computer-readable media, such as modulated data signals and carrier waves.

[0273] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image processing method, characterized by, include: Feature extraction is performed on the image to be processed from multiple different spatial scales to obtain target features and at least one feature to be fused. The target feature and the at least one feature to be fused are fused to obtain a first feature; Extract the high-frequency and low-frequency features from the target features; The high-frequency features are processed based on the residual dense block RDB to obtain the second feature; The low-frequency feature and the at least one feature to be fused are fused to obtain a third feature; Merge the first feature, the second feature, and the third feature to obtain a fused feature; The image to be processed is processed based on the fusion features.

2. The method according to claim 1, characterized in that, The extraction of high-frequency and low-frequency features from the target features includes: The target features are subjected to discrete wavelet decomposition to obtain the fourth feature; The features of the first preset number of channels of the fourth feature are determined as the low-frequency features, and the features of the other channels of the fourth feature other than the low-frequency features are determined as the high-frequency features.

3. The method of claim 2, wherein, After extracting the high-frequency and low-frequency features from the target features, the method further includes: The high-frequency features and the low-frequency features are processed separately through convolutional layers to reduce the number of channels of the high-frequency features and the low-frequency features to a preset value.

4. The method according to claim 1, characterized in that, The process of fusing the low-frequency feature and the at least one feature to be fused to obtain a third feature includes: The at least one feature to be fused is sorted in descending order according to the spatial scale difference between the at least one feature to be fused and the low-frequency feature to obtain a first sorting result; By fusing the first feature to be fused and the low-frequency feature, a fused feature corresponding to the first feature to be fused is obtained, wherein the first feature to be fused is the first feature to be fused in the first sorting result; One by one, the other features to be fused in the first sorting result and the fused feature corresponding to the previous feature to be fused are merged to obtain the fused feature corresponding to the other features to be fused in the first sorting result; The fusion feature corresponding to the last feature to be fused in the first sorting result is determined as the third feature.

5. The method of claim 4, wherein, The process of fusing the first feature to be fused and the low-frequency feature to obtain the fused feature corresponding to the first feature to be fused includes: The low-frequency feature is sampled as a first sampling feature; the first sampling feature has the same spatial scale as the first feature to be fused. Calculate the difference between the first sampled feature and the first feature to be fused to obtain the first difference feature; The first difference feature is sampled as a second sampling feature; the second sampling feature has the same spatial scale as the low-frequency feature. The low-frequency feature and the second sampling feature are added and fused to generate the fused feature corresponding to the first feature to be fused.

6. The method of claim 4, wherein, The step of sequentially fusing other features to be fused in the first sorting result with the fusion feature corresponding to the previous feature to be fused, to obtain the fusion feature corresponding to the other features to be fused in the first sorting result, includes: The fusion feature corresponding to the (m-1)th feature to be fused in the first sorting result is sampled as the third sampled feature; the third sampled feature has the same spatial scale as the mth feature to be fused in the first sorting result, where m is an integer greater than 1; Calculate the difference between the m-th feature to be fused and the third sampled feature to obtain the second difference feature; The second difference feature is sampled as the fourth sampling feature; the fourth sampling feature has the same spatial scale as the fusion feature corresponding to the (m-1)th feature to be fused. The fusion feature corresponding to the (m-1)th feature to be fused and the fourth sampling feature are added and fused to generate the fusion feature corresponding to the mth feature to be fused.

7. The method according to any one of claims 1 to 6, characterized in that, The step of fusing the target feature and the at least one feature to be fused to obtain the first feature includes: The target features are divided into the fifth feature and the sixth feature; The fifth feature is processed based on the residual dense block RDB to obtain the seventh feature; The sixth feature and the at least one feature to be fused are fused to obtain the eighth feature; The seventh feature and the eighth feature are combined to generate the first feature.

8. The method of claim 7, wherein, The process of fusing the sixth feature and the at least one feature to be fused to obtain the eighth feature includes: The at least one feature to be fused is sorted in descending order according to the spatial scale difference between the at least one feature to be fused and the sixth feature to obtain a second sorting result; By fusing the second feature to be fused and the sixth feature, a fused feature corresponding to the second feature to be fused is obtained, wherein the second feature to be fused is the first feature to be fused in the second sorting result; One by one, the other features to be fused in the second sorting result and the fused features corresponding to the previous feature to be fused are merged to obtain the fused features corresponding to the other features to be fused in the second sorting result; The fusion feature corresponding to the last feature to be fused in the second sorting result is determined as the eighth feature.

9. The method according to claim 8, characterized in that, The process of fusing the second feature to be fused and the sixth feature to obtain the fused feature corresponding to the second feature to be fused includes: The sixth feature is sampled as the fifth sampled feature, and the fifth sampled feature has the same spatial scale as the second feature to be fused. Calculate the difference between the fifth sampling feature and the first feature to be fused in the second sorting result to obtain the third difference feature; The third difference feature is sampled to obtain a sixth sampling feature, and the sixth sampling feature has the same spatial scale as the sixth feature. The sixth feature and the sixth sampling feature are added together and fused to generate the fused feature corresponding to the second feature to be fused.

10. The method of claim 8, wherein, The step of sequentially fusing other features to be fused in the second sorting result with the fusion feature corresponding to the previous feature to be fused, to obtain the fusion feature corresponding to the other features to be fused in the second sorting result, includes: The fusion feature corresponding to the (n-1)th feature to be fused in the second sorting result is sampled as the seventh sampling feature; the seventh sampling feature has the same spatial scale as the nth feature to be fused in the second sorting result, where n is an integer greater than 1; Calculate the difference between the nth feature to be fused and the seventh sampled feature to obtain the fourth difference feature; The fourth difference feature is sampled as the eighth sampling feature, and the eighth sampling feature has the same spatial scale as the fusion feature corresponding to the (n-1)th feature to be fused. The fusion feature corresponding to the (n-1)th feature to be fused and the eighth sampling feature are added and fused to generate the fusion feature corresponding to the nth feature to be fused.

11. The method of claim 7, wherein, The step of dividing the target features into a fifth feature and a sixth feature includes: Based on the feature channels of the target features, the target features are divided into the fifth feature and the sixth feature.

12. An image processing method, characterized by, include: The image to be processed is processed by an encoding module to obtain encoded features. The encoding module includes L cascaded encoders with different spatial scales. The i-th encoder is used to extract features from the image to be processed to obtain image features on the i-th encoder, and to obtain the fusion features output by all encoders before the i-th encoder. The fusion features of the i-th encoder are obtained by the image processing method according to any one of claims 1-11, and the fusion features of the i-th encoder are output to all encoders after the i-th encoder. L and i are both positive integers, and i≤L. The encoded features are processed by a feature restoration module consisting of at least one residual block RDB to obtain restored features; The restored features are processed by the decoding module to obtain the processed image of the image to be processed; wherein, the decoding module includes L cascaded decoders with different spatial scales, the j-th decoder is used to fuse the image features of the encoding module on the j-th encoder and the fusion results output by all decoders before the j-th decoder to generate the fusion result of the j-th decoder, and output the fusion result of the j-th decoder to all decoders after the j-th decoder.

13. The method according to claim 12, characterized in that, The step of processing the restored features through the decoding module to obtain the processed result image of the image to be processed includes: Divide the image features on the j-th decoder into the ninth feature and the tenth feature; The ninth feature is processed based on the residual dense block RDB to obtain the eleventh feature; The 10th feature is fused with the fusion results of all decoder outputs before the j-th decoder to obtain the 12th feature; The eleventh feature and the twelfth feature are combined to generate the fusion result of the j-th decoder.

14. An image processing apparatus characterized by comprising: include: The feature extraction unit is used to extract features from the image to be processed from multiple different spatial scales to obtain target features and at least one feature to be fused. The first processing unit is configured to fuse the target feature and the at least one feature to be fused to obtain a first feature; The second processing unit is used to extract high-frequency features and low-frequency features from the target features, process the high-frequency features based on residual dense block (RDB) to obtain the second feature, and fuse the low-frequency features and the at least one feature to be fused to obtain the third feature. A fusion unit is used to merge the first feature, the second feature, and the third feature to obtain a fused feature; The third processing unit processes the image to be processed based on the fusion features.

15. An image processing apparatus characterized by comprising: include: A feature extraction unit is used to process the image to be processed through the encoding module to obtain encoded features; wherein, the encoding module includes L cascaded encoders with different spatial scales, the i-th encoder is used to extract features from the image to be processed to obtain image features on the i-th encoder, and to obtain the fusion features output by all encoders before the i-th encoder, and to obtain the fusion features of the i-th encoder through the image processing method according to any one of claims 1-11, and to output the fusion features of the i-th encoder to all encoders after the i-th encoder, where L and i are both positive integers, and i≤L; The feature processing unit is used to process the encoded features through a feature restoration module composed of at least one residual block RDB to obtain restored features; An image generation unit is used to process the restored features through a decoding module to obtain a processed image of the image to be processed; wherein, the decoding module includes L cascaded decoders with different spatial scales, the j-th decoder is used to fuse the image features of the encoding module on the j-th encoder and the fusion results output by all decoders before the j-th decoder to generate the fusion result of the j-th decoder, and output the fusion result of the j-th decoder to all decoders after the j-th decoder.

16. An electronic device, characterized in that, include: A memory and a processor, the memory being used to store a computer program; the processor being used to cause the electronic device to perform the method of any one of claims 1-13 when the computer program is invoked.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a computing device, causes the computing device to perform the method according to any one of claims 1-13.

18. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1-13.

Images