A method and system for enhancing mixed degraded images

Abstract

The application discloses a method and system for enhancing a mixed degraded image, and relates to the technical field of image processing. The mixed degraded image is an image subjected to M different degradation processes. The method comprises the following steps: S1, mapping each pixel of the mixed degraded image to a high-dimensional feature space by using a shallow convolutional encoder to extract shallow features of the mixed degraded image; S2, inputting the shallow features as input features into a multi-task decoding model, and performing enhancement processing on the shallow features by the multi-task decoding model through M times of recursion to obtain deep features of the mixed degraded image; and S3, splicing and fusing the deep features and the shallow features to obtain residual features, and obtaining an enhanced image with the same size as the mixed degraded image based on the residual features.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a method and system for enhancing mixed degradation images. BACKGROUND

[0002] Real-world images are usually affected by multiple degradations simultaneously. In recent years, some pioneering works on mixed degradation have been proposed. For example, a number of lightweight convolutional neural networks are pre-trained as tools to handle different degradations, and then an agent in reinforcement learning is used to learn to select the best tool chain to handle unknown mixed degradation. The recovery method based on reinforcement learning needs to set a relatively complex reward function and needs a long time to train the model, and the training process is difficult to converge. For another example, an attention mechanism is used to realize adaptive selection of different operations on mixed degradation. However, these methods do not consider the interference between mixed degradation. For another example, the concept of disentangled feature learning is introduced to realize a feature-level divide-and-conquer strategy for mixed degradation, and by assigning different distorted feature representations to different channels, useless information is adaptively filtered out and useful content information is aggregated to reconstruct a clear image. However, separating and processing mixed degradation at the channel level can reduce the parameter amount of the model, but due to the lack of corresponding loss constraints or regularization, the model is difficult to learn different degradation processing functions on different channels, and lacks scalability. A multi-branch strategy is used to design the algorithm, and the multi-branch uses different convolution kernels, which are generally used to extract multi-scale information of the image, and the features extracted by the multi-branch have strong correlation and high similarity. However, different degradations have low similarity in the feature space, and it is difficult to use the multi-branch method to separate and process a single degradation. Existing enhancement models are mostly aimed at a single degradation factor, while in real-world scenarios, mixed degradation is usually affected by multiple degradation factors superimposed on each other, and there is currently little research on such mixed degradation enhancement. SUMMARY

[0003] In view of the problems of large parameter amount, low interpretability and low scalability of the mixed degradation enhancement method in the prior art, the present application provides a scheme for enhancing mixed degradation images.

[0004] The first aspect of the present application discloses a method for enhancing a mixed degraded image; the mixed degraded image is an image subjected to M different degradation processes; the method comprises the following steps: S1, mapping each pixel of the mixed degraded image to a high-dimensional feature space by using a shallow convolutional encoder to extract shallow features of the mixed degraded image; S2, inputting the shallow features as input features into a multi-task decoding model, and performing enhancement processing on the shallow features by the multi-task decoding model through M times of recursion to obtain deep features of the mixed degraded image; wherein the multi-task decoding model comprises a gating network and N enhancement expert units, each enhancement expert unit corresponds to an enhancement processing, and the gating network allocates weights to each enhancement expert unit; S3, splicing and fusing the deep features and the shallow features to obtain residual features, and obtaining an enhanced image with the same size as the mixed degraded image based on the residual features.

[0005] According to the method of the first aspect, in the step S1, the mixed degraded image is X, the feature size of X is 3*H*W, 3 represents the channel number of RGB, H represents the height of the mixed degraded image, W represents the width of the mixed degraded image, the convolution kernel size of the shallow convolutional encoder is 3*3, and the size of the extracted shallow features is d*H*W, d represents the channel number of the shallow features.

[0006] According to the method of the first aspect, in the step S2: after receiving the input features, the N enhancement expert units learn the shallow features to determine that M enhancement expert units among the N enhancement expert units perform enhancement processing on the mixed degraded image corresponding to the M enhancement expert units; the gating network performs a global average pooling operation on the shallow features, sends the features subjected to the global average pooling operation into a fully connected network and a Softmax function, generates weights of the N enhancement expert units through mapping, wherein the sum of the weights of the M enhancement expert units is 1, and the weights of the other N-M enhancement expert units are all 0; in the i-th recursion, the weight of the enhancement expert unit with the largest weight among the M enhancement expert units is assigned as 1, and the weights of the other M-1 enhancement expert units are assigned as 0, so that the enhancement expert unit with the largest weight performs corresponding enhancement processing on the input features in the i-th recursion; the output features subjected to the enhancement processing are taken as new input features of the multi-task decoding model to perform the i+1-th recursion, until M recursions are performed, and the deep features are obtained.

[0007] According to the method of the first aspect, in the step S2, each enhancement expert unit comprises an image-to-block sequence unit, a window self-attention model layer, a block sequence-to-image unit and a convolution layer; wherein: the image-to-block sequence unit is used to convert the image feature to be enhanced into a block sequence feature; the window self-attention model layer directly extracts the degraded feature to be enhanced corresponding to the current enhancement expert unit from the block sequence feature to be enhanced; the block sequence-to-image unit is used to convert the degraded feature to be enhanced based on the block sequence into a degraded image feature to be enhanced; the convolution layer performs enhancement processing corresponding to the current enhancement expert unit on the degraded image feature to be enhanced to obtain a single enhancement feature processed by the current enhancement expert unit; the image feature to be enhanced and the single enhancement feature are fused to obtain an output feature of the current enhancement expert unit, and the output feature is used as the new input feature to perform the next recursion.

[0008] According to the method of the first aspect, in the step S3, the residual feature is sent into two convolution layers to change the size of the residual feature from d*H*W to 3*H*W, and the kernel size of the two convolution layers is 3*3.

[0009] According to the method of the first aspect, when pre-training the multi-task decoding model, an L1 loss network is used to calculate a loss function between an original image obtained in the pre-training process and an enhanced original image, and the pre-training is completed when the loss function is lower than a loss threshold; wherein the calculation method of the loss function is:

[0010]

[0011] wherein, P ij and Y ij respectively represent the pixel value of the enhanced original image P and the original image Y at the coordinate (i, j).

[0012] The second aspect of the present application discloses a system for enhancing a mixed degraded image; the mixed degraded image is an image subjected to M different degradation processes; the system comprises: a first processing unit configured to map each pixel of the mixed degraded image to a high-dimensional feature space using a shallow convolutional encoder to extract shallow features of the mixed degraded image; a second processing unit configured to input the shallow features as input features into a multi-task decoding model, and perform M recursive enhancement processes on the shallow features by the multi-task decoding model to obtain deep features of the mixed degraded image; wherein the multi-task decoding model comprises a gating network and N enhancement expert units, each enhancement expert unit corresponds to an enhancement process, and the gating network allocates weights to each enhancement expert unit; and a third processing unit configured to splice and fuse the deep features and the shallow features to obtain residual features, and obtain an enhanced image with the same size as the mixed degraded image based on the residual features.

[0013] According to the system of the second aspect, the mixed degraded image is X, the feature size of X is 3*H*W, 3 represents the channel number of RGB, H represents the height of the mixed degraded image, and W represents the width of the mixed degraded image; the convolution kernel size of the shallow convolutional encoder is 3*3, and the size of the extracted shallow features is d*H*W, d represents the channel number of the shallow features.

[0014] According to the system of the second aspect, after receiving the input features, the N enhancement expert units learn the shallow features to determine that M enhancement expert units among the N enhancement expert units perform enhancement processes corresponding to the M enhancement expert units on the mixed degraded image; the gating network performs a global average pooling operation on the shallow features, feeds the features subjected to the global average pooling operation into a fully connected network and a Softmax function, and generates weights of the N enhancement expert units by mapping, wherein the sum of the weights of the M enhancement expert units is 1, and the weights of the other N-M enhancement expert units are all 0; the second processing unit is configured to assign a weight of 1 to an enhancement expert unit with the largest weight among the M enhancement expert units and assign weights of 0 to the other M-1 enhancement expert units in the i-th recursion, so that the enhancement expert unit with the largest weight performs a corresponding enhancement process on the input features in the i-th recursion; the output features subjected to the enhancement process are used as new input features of the multi-task decoding model to perform an i+1-th recursion, until M recursions are performed, and the deep features are obtained.

[0015] According to the system of the second aspect, each enhancement expert unit comprises an image-to-block sequence unit, a window self-attention model layer, a block sequence-to-image unit and a convolution layer; wherein: the image-to-block sequence unit is configured to convert an image feature to be enhanced into a block sequence feature; the window self-attention model layer is configured to extract a degraded feature to be enhanced corresponding to the current enhancement expert unit from the block sequence feature to be enhanced; the block sequence-to-image unit is configured to convert the block sequence-based degraded feature to be enhanced into a degraded image feature to be enhanced; the convolution layer is configured to perform enhancement processing corresponding to the current enhancement expert unit on the degraded image feature to be enhanced, to obtain a single enhancement feature processed by the current enhancement expert unit; and the image feature to be enhanced and the single enhancement feature are fused to obtain an output feature of the current enhancement expert unit, which is used as a new input feature for the next recursion.

[0016] According to the system of the second aspect, the residual feature is input into two convolution layers to change the size of the residual feature from d*H*W to 3*H*W, and the kernel size of each of the two convolution layers is 3*3.

[0017] According to the system of the second aspect, when pre-training the multi-task decoding model, an L1 loss network is used to calculate a loss function between an original image obtained in the pre-training process and an enhanced original image, and the pre-training is completed when the loss function is lower than a loss threshold; wherein the loss function is calculated in the following manner:

[0018]

[0019] wherein, P ij and Y ij respectively represent the pixel value of the enhanced original image P and the original image Y at the coordinate (i, j).

[0020] The third aspect of the present application discloses an electronic device. The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to realize the steps of the method for enhancing a mixed degraded image according to any one of the first aspect of the present application.

[0021] The fourth aspect of the present application discloses a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps of the method for enhancing a mixed degraded image according to any one of the first aspect of the present application.

[0022] In summary, the technical solution provided by the present application regards mixed degradation enhancement as a multi-task learning problem, proposes a multi-expert model to decompose multiple degradations into a single degradation, and adaptively enables each enhancement expert to learn a simple single-degradation enhancement model. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed in the specific embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0024] Figure 1 A flowchart of a method for enhancing a mixed degradation image according to an embodiment of the present application;

[0025] Figure 2 A schematic diagram of a multi-task decoding module according to an embodiment of the present application;

[0026] Figure 3 A schematic diagram of a gating network assigning weights according to an embodiment of the present application;

[0027] Figure 4 A structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0029] The present application discloses a method for enhancing a mixed degradation image, wherein the mixed degradation image is an image subjected to M different degradation processes. Figure 1 A flowchart of a method for enhancing a mixed degradation image according to an embodiment of the present application; in combination Figure 1As shown, the method comprises: step S1, mapping each pixel of the mixed degraded image to a high-dimensional feature space by using a shallow convolutional encoder to extract shallow features of the mixed degraded image; step S2, inputting the shallow features as input features into a multi-task decoding model, and performing enhancement processing on the shallow features by the multi-task decoding model through M times of recursion to obtain deep features of the mixed degraded image; wherein the multi-task decoding model comprises a gating network and N enhancement expert units, each enhancement expert unit corresponds to an enhancement processing, and the gating network allocates weights to each enhancement expert unit; step S3, splicing and fusing the deep features and the shallow features to obtain residual features, and obtaining an enhanced image with the same size as the mixed degraded image based on the residual features.

[0030] In some embodiments, in the step S1, the mixed degraded image is X, the feature size of X is 3*H*W, 3 represents the channel number of RGB, H represents the height of the mixed degraded image, W represents the width of the mixed degraded image, the convolution kernel size of the shallow convolutional encoder is 3*3, and the size of the extracted shallow features is d*H*W, d represents the channel number of the shallow features.

[0031] In some embodiments, in the step S2: after receiving the input features, the N enhancement expert units learn the shallow features to determine that M enhancement expert units among them perform enhancement processing on the mixed degraded image corresponding to the M enhancement expert units; the gating network performs a global average pooling operation on the shallow features, sends the features subjected to the global average pooling operation into a fully connected network and a Softmax function, and generates weights of the N enhancement expert units by mapping, wherein the sum of the weights of the M enhancement expert units is 1, and the weights of the other N-M enhancement expert units are all 0; in the i-th recursion, the weight of the enhancement expert unit with the largest weight among the M enhancement expert units is assigned as 1, and the weights of the other M-1 enhancement expert units are assigned as 0, so that the input features are subjected to corresponding enhancement processing by the enhancement expert unit with the largest weight in the i-th recursion; the output features subjected to the enhancement processing are taken as new input features of the multi-task decoding model to perform the i+1-th recursion, until M times of recursion are performed, and the deep features are obtained.

[0032] In some embodiments, in the step S2, each enhancement expert unit comprises an image-to-block sequence conversion unit, a window self-attention model layer, a block sequence-to-image conversion unit and a convolution layer; wherein: the image-to-block sequence conversion unit is configured to convert the image feature to be enhanced into a block sequence feature; the window self-attention model layer is configured to extract the degraded feature to be enhanced corresponding to the current enhancement expert unit directly from the block sequence feature to be enhanced; the block sequence-to-image conversion unit is configured to convert the degraded feature to be enhanced based on the block sequence into a degraded image feature to be enhanced; the convolution layer is configured to perform enhancement processing corresponding to the current enhancement expert unit on the degraded image feature to be enhanced, so as to obtain a single enhancement feature processed by the current enhancement expert unit; and the image feature to be enhanced and the single enhancement feature are fused to obtain an output feature of the current enhancement expert unit, which is used as a new input feature for the next recursion.

[0033] In some embodiments, in the step S3, the residual feature is sent into two convolution layers to change the size of the residual feature from d*H*W to 3*H*W, and the kernel size of the two convolution layers is 3*3.

[0034] In some embodiments, when pre-training the multi-task decoding model, an L1 loss network is used to calculate a loss function between an original image obtained in the pre-training process and an enhanced original image, and the pre-training is completed when the loss function is lower than a loss threshold; wherein the loss function is calculated in the following manner:

[0035]

[0036] wherein, P ij and Y ij respectively represent the pixel value of the enhanced original image P and the original image Y at the coordinate (i, j).

[0037] In other embodiments, the method comprises four parts: a shallow convolutional encoding module, a multi-task decoding module with M times of recursion, a reconstruction module and a loss function. The shallow convolutional encoding module is used to map the mixed degraded image into a high-dimensional feature map, and then the multi-task decoding module is used for multiple iterations to remove degradation, and finally the reconstruction module is used to map the restored high-dimensional feature map into a three-dimensional RGB enhanced image.

[0038] The shallow convolutional encoding module is a shallow encoder, which is used to map each pixel of the input image to a high-dimensional feature space. Using a convolution layer before the self-attention model can greatly improve the optimization stability and peak performance. Therefore, given a mixed degraded image X, the feature size of X is 3*H*W, H is the height of the image, and W is the width of the image, and a convolution kernel with 3*3 is used to extract shallow features.

[0039] The multi-task decoding module receives the high-dimensional features obtained from the shallow convolutional encoding module, and removes the degradation through recursive operation cycles to restore clean features. The mixed degradation enhancement is regarded as a multi-task learning problem. Mixed degradation data is generally generated by randomly mixing multiple degradation factors, and can be divided into multiple sub-data sets, corresponding to multiple image enhancement sub-tasks. A general single degradation enhancement method is not suitable for such complex and variable mixed degradation enhancement tasks. Therefore, the model uses a multi-expert task model to model the mixed degradation based on the multi-task learning method. Specifically, the multi-task decoding module is composed of multiple enhancement experts and a gating network, as shown in Figure 2 .

[0040] The enhancement expert is based on a window self-attention model and adaptively learns the enhancement of a single or specific degradation. The expert is composed of a window self-attention model and a traditional convolution, and uses a residual connection to fuse shallow and deep features. Unlike traditional convolution in convolutional neural networks, the self-attention model does not have an inductive bias. Traditional convolution itself has translational equivalence and locality, while the self-attention model does not. Mixed degradation enhancement belongs to the dense pixel point prediction task, and local information is crucial to the entire network learning. Therefore, the model adds a convolution after the window self-attention model layer to strengthen the learning of local features. Since the window self-attention model layer is used to process image block sequences, and the convolution is used to process feature maps, the module adds image-to-block sequence operations and block sequence-to-image operations to connect the two.

[0041] The gating network is used to generate the feature weights of the output of each enhancement expert. Taking the i-th expert as an example, the gating network process is shown in Figure 3 . (The gating network can assign weights before each enhancement expert performs enhancement processing, or after each enhancement expert performs enhancement processing.) First, the input feature (d*H*W) is subjected to a global average pooling operation to obtain a d vector, which is then sent to a fully connected network and a Softmax function to map to generate an N vector, which corresponds to the weight vector of the N enhancement experts. The i-th column vector in the weight vector is multiplied by the output feature of the i-th enhancement expert to form the output feature with weights. Finally, all the expert output features with weights are added to obtain the selected decoding features.

[0042] Each expert adaptively learns an enhancement model for a certain degradation factor, while mixed degradation images have multiple degradations superimposed. Therefore, without adding additional parameters, the patent uses a recursive method to run the multi-task decoding module. In each recursion, the multi-task decoding module focuses on the output of the relevant expert model for the degradation factors present in each feature map. Through multiple recursions, the mixed degradation is sequentially removed, and finally a clean feature map without degradation is obtained.

[0043] The reconstruction module: the features output from the decoder are spliced and fused with the shallow layer convolutional code to form a cross-connection residual feature. The residual feature is input into two convolution kernel size 3*3 convolution layers in turn, and an enhanced image with the same size as the input image is output. Through a cross-long connection, the high-frequency information in the shallow layer feature and the low-frequency information in the deep layer feature are effectively utilized and fused, so that the reconstruction module can learn more information to reconstruct the image.

[0044] Loss function: after obtaining the reconstructed image, the patent uses L1 loss for network training. Given a predicted enhanced image P (enhanced original image) and the corresponding non-degraded image label Y (original image), the calculation of L1 loss is as follows:

[0045]

[0046] Where, P ij and Y ij respectively represent the pixel values of the enhanced original image P and the original image Y at coordinates (i,j).

[0047] The second aspect of the application discloses a system for enhancing mixed degraded images; the mixed degraded image is an image subjected to M different degradation processes, and the system comprises: a first processing unit configured to map each pixel of the mixed degraded image to a high-dimensional feature space using a shallow layer convolutional encoder to extract shallow layer features of the mixed degraded image; a second processing unit configured to input the shallow layer features as input features into a multi-task decoding model, and to perform enhancement processing on the shallow layer features by M times of recursion by the multi-task decoding model to obtain deep layer features of the mixed degraded image; wherein the multi-task decoding model comprises a gating network and N enhancement expert units, each enhancement expert unit corresponding to one enhancement processing, and the gating network assigning weights to each enhancement expert unit; a third processing unit configured to splice and fuse the deep layer features and the shallow layer features to obtain a residual feature, and to obtain an enhanced image with the same size as the mixed degraded image based on the residual feature.

[0048] According to the system of the second aspect, the mixed degraded image is X, the feature size of X is 3*H*W, 3 represents the channel number of RGB, H represents the height of the mixed degraded image, and W represents the width of the mixed degraded image, the convolution kernel size of the shallow layer convolutional encoder is 3*3, and the size of the extracted shallow layer features is d*H*W, d represents the channel number of the shallow layer features.

[0049] According to the system of the second aspect, after receiving the input feature, the N enhanced expert units learn the shallow layer feature to determine that M enhanced expert units among the N enhanced expert units perform the enhanced processing corresponding to the M enhanced expert units on the mixed degraded image; the gating network performs a global average pooling operation on the shallow layer feature, and sends the feature subjected to the global average pooling operation to a fully connected network and a Softmax function to generate the weights of the N enhanced expert units through mapping, wherein the sum of the weights of the M enhanced expert units is 1, and the weights of the other N-M enhanced expert units are all 0; the second processing unit is configured to: in the i-th recursion, assign the weight of the enhanced expert unit with the maximum weight among the M enhanced expert units as 1, and assign the weights of the other M-1 enhanced expert units as 0, so that the enhanced expert unit with the maximum weight performs the corresponding enhanced processing on the input feature in the i-th recursion; the output feature subjected to the enhanced processing is taken as the new input feature of the multi-task decoding model to perform the i+1-th recursion, until M recursions are performed, and the deep layer feature is obtained.

[0050] According to the system of the second aspect, each enhanced expert unit comprises an image-to-block sequence unit, a window self-attention model layer, a block sequence-to-image unit and a convolution layer; wherein: the image-to-block sequence unit is used to convert the image feature to be enhanced into a block sequence feature; the window self-attention model layer directly extracts the degraded feature to be enhanced corresponding to the current enhanced expert unit from the block sequence feature to be enhanced; the block sequence-to-image unit is used to convert the degraded feature to be enhanced based on the block sequence into a degraded image feature to be enhanced; the convolution layer performs the enhanced processing corresponding to the current enhanced expert unit on the degraded image feature to be enhanced to obtain a single enhanced feature processed by the current enhanced expert unit; the image feature to be enhanced and the single enhanced feature are fused to obtain the output feature of the current enhanced expert unit, and the output feature is taken as the new input feature to perform the next recursion.

[0051] According to the system of the second aspect, the residual feature is sent into two convolution layers to change the size of the residual feature from d*H*W to 3*H*W, and the kernel size of the two convolution layers is both 3*3.

[0052] According to the system of the second aspect, when pre-training the multi-task decoding model, an L1 loss network is used to calculate a loss function between an original image obtained in the pre-training process and an enhanced original image, and the pre-training is completed when the loss function is lower than a loss threshold; wherein the calculation method of the loss function is:

[0053]

[0054] wherein P ij and Y ij respectively represent the pixel value of the enhanced original image P and the pixel value of the original image Y at coordinate (i,j).

[0055] A third aspect of the present application discloses an electronic device. The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps in the method for enhancing a mixed degraded image according to any one of the first aspect of the present application when executing the computer program.

[0056] Figure 4 As shown in FIG. 1, Figure 4 the electronic device comprises a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The communication interface of the electronic device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be achieved through WIFI, operator network, near field communication (NFC) or other technologies. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0057] Those skilled in the art can understand, Figure 4 the structure shown in FIG. 1 is only a structural diagram of part of the technical solution of the present application, and does not constitute a limitation on the electronic device to which the technical solution of the present application is applied. Specifically, the electronic device can comprise more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0058] A fourth aspect of the present application discloses a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in the method for enhancing a mixed degraded image according to any one of the first aspect of the present application.

[0059] To sum up, the technical scheme provided by the application takes mixed degradation enhancement as a multi-task learning problem, proposes a multi-expert model to decompose multiple degradations into a single degradation, and adaptively enables each enhancement expert to learn a simple single-degradation enhancement model. On this basis, a recursive strategy is used to train the degradation factor removal, and the performance of mixed degradation enhancement is further improved without increasing the parameter amount.

[0060] Please note that the technical features of the above embodiments can be combined in any way. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not exist, they should be considered as the scope of the description. The above-described embodiments only express several embodiments of the application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the application, a number of modifications and improvements can be made, which are within the scope of the application. Therefore, the scope of the patent of the application should be subject to the appended claims.

Claims

1. A method of enhancing a mixed degraded image, characterized by, The mixed degradation image is an image subjected to M different degradation processes, and the method comprises: Step S1, mapping each pixel of the mixed degradation image to a high-dimensional feature space by using a shallow convolutional encoder to extract shallow features of the mixed degradation image; Step S2, inputting the shallow features as input features into a multi-task decoding model, and performing enhancement processing on the shallow features by the multi-task decoding model through M times of recursion to obtain deep features of the mixed degradation image; Wherein, the multi-task decoding model comprises a gating network and N enhancement expert units, each enhancement expert unit corresponds to an enhancement processing, and the gating network allocates weights to each enhancement expert unit; Step S3, concatenating and fusing the deep features and the shallow features to obtain residual features, and obtaining an enhanced image with the same size as the mixed degradation image based on the residual features; Wherein, in the step S2: After receiving the input features, the N enhancement expert units learn the shallow features to determine that M enhancement expert units among the N enhancement expert units perform enhancement processing on the mixed degradation image corresponding to the M enhancement expert units; The gating network performs global average pooling operation on the shallow features, and sends the features subjected to the global average pooling operation to a fully connected network and a Softmax function to generate weights of the N enhancement expert units through mapping, wherein the sum of the weights of the M enhancement expert units is 1, and the weights of the other N-M enhancement expert units are all 0; In the ith recursion, the weight of the enhancement expert unit with the largest weight among the M enhancement expert units is assigned as 1, and the weights of the other M-1 enhancement expert units are assigned as 0, so that the enhancement expert unit with the largest weight performs corresponding enhancement processing on the input features in the ith recursion; The output features subjected to the enhancement processing are taken as new input features of the multi-task decoding model to perform the i+1th recursion, until M recursions are performed, and the deep features are obtained.

2. The method of claim 1, wherein, In the step S1, the mixed degradation image is X, the feature size of X is 3*H*W, 3 represents the channel number of RGB, H represents the height of the mixed degradation image, W represents the width of the mixed degradation image, the convolution kernel size of the shallow convolutional encoder is 3*3, and the size of the extracted shallow features is d*H*W, d represents the channel number of the shallow features.

3. The method of claim 2, wherein, In the step S2, each enhancement expert unit comprises an image-to-block sequence unit, a window self-attention model layer, a block sequence-to-image unit and a convolution layer; wherein: The image-to-block sequence unit is used to convert the image features to be enhanced into block sequence features; The window self-attention model layer directly extracts the degradation features to be enhanced corresponding to the current enhancement expert unit from the block sequence features to be enhanced; The block sequence-to-image unit is used to convert the block sequence-based degradation features to be enhanced into degradation image features to be enhanced; The convolutional layer performs enhancement processing corresponding to the current enhancement expert unit on the to-be-enhanced degraded image feature to obtain a single enhanced feature processed by the current enhancement expert unit; The to-be-enhanced image feature and the single enhanced feature are fused to obtain an output feature of the current enhancement expert unit, which is used as a new input feature for the next recursion.

4. The method of claim 3, wherein, In step S3, the residual feature is sent into two convolutional layers to change the size of the residual feature from d*H*W to 3*H*W, and the kernel size of each of the two convolutional layers is 3*3.

5. The method of claim 4, wherein, In the pre-training of the multi-task decoding model, an L1 loss network is used to calculate a loss function between an original image obtained in the pre-training process and an enhanced original image, and the pre-training is completed when the loss function is lower than a loss threshold; wherein the loss function is calculated in the following manner: where P ij and Y ij respectively denote the pixel values of the enhanced original image P and the original image Y at coordinates (i,j).

6. A system for enhancing a mixed degraded image, characterized in that, The mixed degraded image is an image processed by M different degradation processes, and the system comprises: A first processing unit configured to map each pixel of the mixed degraded image to a high-dimensional feature space by using a shallow convolutional encoder to extract a shallow feature of the mixed degraded image; A second processing unit configured to input the shallow feature as an input feature into a multi-task decoding model, and perform enhancement processing on the shallow feature by the multi-task decoding model through M recursions to obtain a deep feature of the mixed degraded image; The multi-task decoding model comprises a gating network and N enhancement expert units, each of which corresponds to an enhancement processing, and the gating network allocates weights to each enhancement expert unit; A third processing unit configured to splice and fuse the deep feature and the shallow feature to obtain a residual feature, and obtain an enhanced image with the same size as the mixed degraded image based on the residual feature; The second processing unit is specifically configured to: After receiving the input feature, the N enhancement expert units learn the shallow feature to determine that M enhancement expert units of the N enhancement expert units perform enhancement processing corresponding to the M enhancement expert units on the mixed degraded image; The gating network performs a global average pooling operation on the shallow feature, sends the feature processed by the global average pooling operation into a fully connected network and a Softmax function, and generates weights of the N enhancement expert units by mapping, wherein the sum of the weights of the M enhancement expert units is 1, and the weights of the other N-M enhancement expert units are all 0; In the i-th recursion, the weight of the enhancement expert unit with the largest weight among the M enhancement expert units is set to 1, and the weights of the other M-1 enhancement expert units are set to 0, so that the enhancement expert unit with the largest weight performs corresponding enhancement processing on the input feature in the i-th recursion; The output feature processed by the enhancement processing is used as a new input feature of the multi-task decoding model to perform an i+1-th recursion, until M recursions are performed, and the deep feature is obtained.

7. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to realize the steps in the method for enhancing a mixed degraded image according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the processor executes the computer program to realize the steps in the method for enhancing a mixed degraded image according to any one of claims 1-5.

Images