Image restoration method and system based on dynamic decomposition and fusion

Abstract

The present invention provides an image restoration method and system based on dynamic decomposition and fusion, which decomposes an image to be restored with mixed features according to degradation patterns, extracts degradation features of the same number as the number of types of degradation patterns; processes each degradation feature separately to obtain restoration features corresponding to each degradation pattern; dynamically fuses the restoration features to obtain a clear restoration image of the same size as the image to be restored, thereby realizing the function of high-quality restoration of complex mixed images. The present invention decomposes, processes and fuses the restoration feature map at multiple scales to obtain a restoration feature map that fuses restoration features of multiple scales, thereby improving the visual effect of restoration of mixed images. The present invention constructs a symmetrical U-shaped network based on the corresponding model formed by the method, and constructs a processing submodule corresponding to the degradation pattern, which copes well with the complex and diverse degraded images in real environments, and provides an efficient and reliable solution for the field of image restoration.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning image restoration, and specifically relates to an image restoration method and system based on dynamic decomposition and fusion. Background Art

[0002] With the rapid development of digital imaging technology and mobile Internet, in today's digital age, images are widely used in various fields as a vital information carrier, including medical, remote sensing and security. In such an application context, the requirements for image quality are increasing at an increasingly faster rate. However, images are inevitably affected by many factors during acquisition, transmission and storage: for example, channel noise can cause granularity and messy signal interference in the image; compression distortion can cause loss of image details and color deviation; data loss can cause partial image areas to be missing or erroneous information to appear. These factors work together to cause images to have mixed features such as blur, distortion and noise, which in turn leads to a serious decline in image quality and cannot meet the needs of daily life and work. Therefore, how to effectively repair and process these problematic images has become a key issue that needs to be solved urgently.

[0003] The goal of image restoration is to restore the features and details of the original image as much as possible from the degraded image. However, previous studies did not distinguish the degradation factors in detail in most cases. Different degradation factors, such as blur, rain, fog, noise, etc., were often treated in the same way. This approach obviously cannot deal with different degradation factors in a targeted manner. Different degradation modes are mixed together, which often makes the image restoration results fail to meet people's expectations for high-quality images. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an image restoration method and system based on dynamic decomposition and fusion, which are used to restore complex and mixed images with high quality.

[0005] The technical solution adopted by the present invention to solve the above technical problems is: an image restoration method based on dynamic decomposition and fusion, comprising the following steps:

[0006] S1: Obtain an image to be restored of any size with mixed features;

[0007] S2: Decompose the image to be restored according to the degradation pattern, and extract degradation features whose number is the same as the number of types of the degradation pattern;

[0008] S3: Process each degradation feature separately to obtain the restoration feature corresponding to each degradation mode;

[0009] S4: Dynamically fuse the restored features to obtain a restored feature map with the same size as the image to be restored.

[0010] According to the above scheme, in step S2, the specific steps are:

[0011] S21: extracting a mixed feature from the image to be restored through a convolution operation;

[0012] S22: According to the degradation pattern of the image to be restored, the k-means classification method is used to decompose the mixed features to obtain multiple different features containing a single degradation pattern.

[0013] Furthermore, in step S22, the specific steps are:

[0014] S221: Determine the number of types of degradation factors according to the degradation mode of the image to be restored;

[0015] S222: dividing the number of channels according to the number of types of degradation factors, and classifying the features obtained by decomposing the mixed features according to similar feature responses to decouple different types of degradation factors; the sum of the number of channels occupied by the degradation factors is the total number of channels;

[0016] S223: Perform self-attention operation on the classified features, perform mask screening on the results after the self-attention operation and the global features, and calculate the top- k The corresponding response is selected according to the input of the Scatter function, and the same number of degradation features as the number of types of degradation patterns are extracted.

[0017] Furthermore, in step S4, the specific steps are:

[0018] S41: Add and restore features at the pixel level to obtain new features fused by multiple degradation factors;

[0019] S42: Perform a global pooling operation on the new feature to obtain a global channel descriptor; S43: Obtain a corresponding intermediate feature based on the global channel descriptor and each restored feature, and calculate an output feature corresponding to each restored feature;

[0020] S44: Perform a concatenation operation on the output features in the channel dimension to obtain the final features;

[0021] S45: Use a 3×3 convolution operation to restore the channel of the feature map corresponding to the final feature to the same channel as the image to be restored, and add the restored feature map corresponding to the final feature to the image to be restored to obtain a clear restored feature map.

[0022] Furthermore, in step S43, the specific steps are:

[0023] S431: Using a 1×1 convolution operation on the global channel descriptor to obtain the relationship between different channels, and obtaining an intermediate variable for connecting different degradation factors;

[0024] S432: Perform a global pooling operation on each restored feature to obtain a corresponding query weight coefficient matrix;

[0025] S433: performing a global pooling operation on each restored feature and then performing a 1×1 convolution operation to obtain a corresponding value coefficient matrix;

[0026] S434: Perform self-attention calculation on the query weight coefficient matrix, intermediate variables and value coefficient matrix corresponding to each restored feature to obtain the corresponding intermediate feature;

[0027] S435: The intermediate feature is coefficientized through the Sigmoid function, and is element-wise multiplied with the intermediate feature to obtain the output feature corresponding to each restored feature.

[0028] According to the above scheme, multi-layer symmetrical downsampling and upsampling are performed on the restored feature map through a symmetrical U-shaped network;

[0029] The image after each downsampling is decomposed, processed and fused in sequence according to the degradation mode. On the one hand, the restored feature map is used as the input for the next downsampling. On the other hand, the restored feature map is added to the upsampled image of the same size, and then decomposed, processed and fused in sequence according to the degradation mode, and the restored feature map is used as the input for the next upsampling.

[0030] Furthermore, after adding the restored feature map obtained by fusion after the last upsampling image and the downsampling image of the same size, the image is decomposed, processed and fused in sequence according to the degradation mode, and the restored feature map is added to the image to be restored to obtain the final restored image as the restored result image.

[0031] An image restoration module based on dynamic decomposition and fusion comprises an acquisition submodule for acquiring an image to be restored of any size with mixed features; a decomposition submodule for decomposing the image to be restored according to degradation modes and extracting degradation features of the same number as the number of types of the degradation modes; a processing submodule for processing each degradation feature respectively and obtaining restoration features corresponding to each degradation mode; and a fusion submodule for dynamically fusing restoration features and obtaining a restoration feature map of the same size as the image to be restored.

[0032] An image restoration network structure is a symmetrical U-shaped network, including a plurality of corresponding downsampling layers and upsampling layers, and also including a plurality of image restoration modules, wherein the image restoration module is the image restoration module based on dynamic decomposition and fusion described in the above technical method;

[0033] An image restoration module is provided between adjacent downsampling layers and between adjacent upsampling layers, and an image restoration module is also provided between the downsampling layer and the upsampling layer connected at the bottom of the U-shaped network;

[0034] The output image of the downsampling layer is decomposed, processed and fused through the image restoration module. On the one hand, the restored feature map is used as the input of the next downsampling layer; on the other hand, the restored feature map is added to the output image of the upsampling layer of the same size, and then decomposed, processed and fused through the image restoration module, and the restored feature map is used as the input of the next upsampling layer.

[0035] Furthermore, the downsampling layer is used to halve the size of the restored feature map and double the number of channels; the upsampling layer is used to double the size of the restored feature map and halve the number of channels.

[0036] The beneficial effects of the present invention are:

[0037] 1. The present invention discloses an image restoration method and system based on dynamic decomposition and fusion, which decomposes an image to be restored with mixed features according to degradation patterns, extracts degradation features of the same number as the number of types of degradation patterns; processes each degradation feature separately to obtain restoration features corresponding to each degradation pattern; and dynamically fuses the restoration features to obtain a clear restoration image of the same size as the image to be restored, thereby realizing the function of high-quality restoration of complex mixed images.

[0038] 2. The present invention decomposes, processes and fuses the restored feature maps at multiple scales to obtain the restored feature maps at the corresponding scales; then merges the restored feature maps scale by scale to obtain a restored feature map of the same size as the image to be restored that fuses the restored features of multiple scales; thus, the visual effect of mixed image restoration is improved, and the effect and quality of image restoration are greatly improved.

[0039] 3. The present invention constructs a symmetrical U-shaped network based on the corresponding model formed by the image restoration method based on dynamic decomposition and fusion, and constructs processing sub-modules that correspond one-to-one to the degradation mode of the image to be restored. It performs comprehensive and detailed dynamic decomposition of the image at multiple scales, and copes well with the complex and diverse degraded images in the real environment, providing an efficient and reliable solution for the field of image restoration.

[0040] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 It is a flow chart of an image restoration method based on dynamic decomposition and fusion according to an embodiment of the present invention.

[0043] Figure 2 4 is a schematic diagram of an image restoration module based on dynamic decomposition and fusion according to an embodiment of the present invention.

[0044] Figure 3 Schematic diagram of the decomposition submodule of an embodiment of the present invention.

[0045] Figure 4 Schematic diagram of a screening mask submodule according to an embodiment of the present invention.

[0046] Figure 5 Schematic diagram of a fusion submodule according to an embodiment of the present invention.

[0047] Figure 6 4 is an architecture diagram of a U-type network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0049] Example 1

[0050] See also Figure 1 ,The specific steps of an image restoration method based on dynamic decomposition and fusion are as follows:

[0051] S1: Obtain an image to be restored of any size with mixed features;

[0052] S2: Decompose the image to be restored according to the degradation mode and extract k different degradation features;

[0053] S3: Process each degradation feature separately to obtain the restoration feature corresponding to each degradation mode;

[0054] S4: Dynamically fuse the restored features to obtain a restored feature map with the same size as the image to be restored.

[0055] Further, in step S2, the specific steps are:

[0056] S21: extracting a mixed feature from the image to be restored through a convolution operation;

[0057] S22: According to the degradation pattern of the image to be restored, the k-means classification method is used to decompose the mixed features to obtain multiple different features containing a single degradation pattern.

[0058] Furthermore, in the step S22, the specific steps are:

[0059] S221: Determine the number of types of degradation factors according to the degradation mode of the image to be restored;

[0060] S222: dividing the number of channels according to the number of types of degradation factors, and classifying the features obtained by decomposing the mixed features according to similar feature responses to decouple different types of degradation factors; the sum of the number of channels occupied by the degradation factors is the total number of channels;

[0061] S223: Perform self-attention operation on the classified features, perform mask screening on the results after the self-attention operation and the global features, and calculate the top- k The corresponding response is selected according to the input of the Scatter function, and the same number of degradation features as the number of types of degradation patterns are extracted.

[0062] In step S4, the specific steps are:

[0063] S41: Add and restore features at the pixel level to obtain new features fused by multiple degradation factors;

[0064] S42: Perform a global pooling operation on the new features to obtain a global channel descriptor;

[0065] S43: Obtain corresponding intermediate features according to the global channel descriptor and each restored feature, and calculate output features corresponding to each restored feature;

[0066] S44: Perform a concatenation operation on the output features in the channel dimension to obtain the final features;

[0067] S45: Use a 3×3 convolution operation to restore the channel of the feature map corresponding to the final feature to the same channel as the image to be restored, and add the restored feature map corresponding to the final feature to the image to be restored to obtain a clear restored feature map.

[0068] Furthermore, in the step S43, the specific steps are:

[0069] S431: Using a 1×1 convolution operation on the global channel descriptor to obtain the relationship between different channels, and obtaining an intermediate variable for connecting different degradation factors;

[0070] S432: Perform a global pooling operation on each restored feature to obtain a corresponding query weight coefficient matrix;

[0071] S433: performing a global pooling operation on each restored feature and then performing a 1×1 convolution operation to obtain a corresponding value coefficient matrix;

[0072] S434: Perform self-attention calculation on the query weight coefficient matrix, intermediate variables and value coefficient matrix corresponding to each restored feature to obtain the corresponding intermediate feature;

[0073] S435: The intermediate feature is coefficientized through the Sigmoid function, and is element-wise multiplied with the intermediate feature to obtain the output feature corresponding to each restored feature.

[0074] This embodiment decomposes the image to be restored with mixed features according to the degradation mode, processes each degradation feature separately, and dynamically fuses the restoration features to obtain a clear restored image of the same size as the image to be restored, thereby realizing the function of high-quality restoration of complex mixed images.

[0075] Example 2

[0076] The steps of this embodiment are the same as those of embodiment 1, except that each step is applied to a specific example. Specifically, the following steps are included:

[0077] S1: Get the size of 3× H × W The image to be restored has mixed features I ;

[0078] S2: Decompose the image to be restored according to the degradation mode and extract k different degradation characteristics; the specific steps are:

[0079] S21: extract a mixed feature from the image to be restored through a convolution operation; the specific steps are:

[0080] The image to be restored I The input feature extraction layer undergoes a 3×3 convolution operation After that, we get a mixed feature F , which is the global feature that aggregates all other channel features F for:

[0081] ;

[0082] in , F ∈ C × H × W , C ,H and W Respectively represent the number of channels, height and width of the image to be restored;

[0083] Use global draw pooling GAP The operation reduces the spatial dimension of the feature map to 1×1 while retaining the number of channels of the feature map and converting the global feature F Mapped into channel descriptors for:

[0084] , .

[0085] S22: According to the degradation mode of the image to be restored, use k The mean classification method KMeans decomposes the mixed features and obtains k different features with a single degradation mode; the specific steps are:

[0086] Determine the number of types of degradation factors according to the degradation mode of the image to be restored, and obtain the hyperparameter k ;

[0087] According to the number of types of degradation factors k Divide channels Z , and decompose the mixed features based on similar feature responses N i Classification is done to decouple different types of degradation factors:

[0088] KMeans( Z )= N 1 N 2 … N i … N k ;

[0089] in , , …, ,and , i =1, 2, …, k , the sum of the number of channels occupied by the degradation factors is the total number of channels;

[0090] Let α be a selectable learnable parameter that controls the channel descriptor Z That is, query the weight coefficient matrix and degradation factors N d The size of the result after multiplication; k Group features for self-attention operation:

[0091] ;

[0092] The result after the self-attention operation O d With global features F As the input of the mask filtering submodule; d Get the top- k The large response value is then selected according to the input of the Scatter function to extract the corresponding response. k Different degradation characteristics.

[0093] S3: Separate processing k Different degradation characteristics are obtained under the corresponding degradation mode. k Recovery Features S i ,in , , ,and .

[0094] S4: Dynamic Fusion k The restored features are obtained to obtain a restored feature map of the same size as the image to be restored; the specific steps are:

[0095] S41: Adding and restoring features at the pixel level S i , and obtain new features fused by multiple degradation factors ;

[0096] S42: Global pooling of new features GAP Operate the features of the aggregation space to obtain the global channel descriptor ;

[0097] S43: In the global channel descriptor Z ′ uses a 1×1 convolution operation Obtain the relationship between different channels and obtain the intermediate variables used to connect different degradation factors :

[0098] ;

[0099] This intermediate variable A is used to aggregate the information of all different degradation factors.

[0100] Restoration Features S i Perform global draw pooling GAP Operation to obtain the corresponding query weight coefficient matrix :

[0101] Q i =GAP(S i );

[0102] Restoration Features S i Perform global draw pooling GAP After the operation, a 1×1 convolution operation is performed Get the corresponding value coefficient matrix :

[0103] V i =Conv 1×1 (GAP( S i ));

[0104] Consider A as K in the self-attention operation, and α is used to control Q i A learnable parameter for the size of the result after multiplication with the A matrix; for recovering features S i The corresponding query weight coefficient matrix , intermediate variables Sum coefficient matrix Perform self-attention calculation and get the corresponding k Intermediate Features :

[0105] ;

[0106] set up It means element-wise multiplication, and the intermediate feature is coefficientized by Sigmoid function and combined with the intermediate feature M d Multiply by elements to get the output feature T corresponding to each restored feature i :

[0107] ;

[0108] in , , ,and ;

[0109] S44: Output feature T in channel dimension i Perform the connection operation Cat and get the final feature T:

[0110] ;

[0111] S45: Use 3×3 convolution operation The channel of the feature map corresponding to the final feature T is restored to the same channel as the image to be restored, and the feature map corresponding to the restored final feature T is added to the image to be restored I to obtain a clear restored feature map R:

[0112] .

[0113] A symmetrical U-shaped network is used to perform multi-layer symmetrical downsampling and upsampling on the restored feature map; the image after each downsampling is decomposed, processed and fused in sequence according to the degradation mode, and the restored feature map is used as the input for the next downsampling; on the other hand, the restored feature map is added to the upsampled image of the same size, and then decomposed, processed and fused in sequence according to the degradation mode, and the restored feature map is used as the input for the next upsampling. Specifically, the U-type network has N pairs of symmetrically arranged upsampling layers and downsampling layers, where N is a non-zero natural number; the image to be restored is first decomposed, processed, and fused once (i.e., steps S2 to S4, hereinafter referred to as dynamic fusion processing), and then the first downsampling (image size is halved, number of channels is doubled), the second dynamic fusion processing, ..., the mth dynamic fusion processing, the mth downsampling, the m+1th dynamic fusion processing, ..., the Nth dynamic fusion processing, the Nth downsampling, the N+1th dynamic fusion processing; then the restored feature map after the N+1th dynamic fusion processing is upsampled for the first time (image size is doubled, number of channels is halved), and then the output after the first upsampling is The image is added to the restored feature map output after the Nth dynamic fusion process, and the N+2th dynamic fusion process is performed. The restored feature map after the N+2th dynamic fusion process is upsampled for the second time, and then added to the restored feature map output after the N-1th dynamic fusion process, and the added image is subjected to the N+3th dynamic fusion process, ..., it can be seen that the output image after the nth upsampling is added to the restored feature map output after the N+1-nth dynamic fusion process, and the N+1+nth dynamic fusion process is performed, and so on. Finally, the restored feature map obtained by executing the (2N+1)th dynamic fusion process is added to the image to be restored, and the final restored image is obtained as the restored result image. Figure 4 The U-shaped network shown, where N=3, has three symmetrical downsampling layers and three upsampling layers, and performs 3 upsampling and 3 downsampling accordingly.

[0114] This embodiment decomposes, processes and fuses the restored feature maps at multiple scales to obtain the restored feature maps at the corresponding scales; then merges the restored feature maps scale by scale to obtain a restored feature map of the same size as the image to be restored and that fuses the restored features of multiple scales; thus, the visual effect of mixed image restoration is improved, and the effect and quality of image restoration are greatly improved.

[0115] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0116] The running environment of this embodiment is built based on the Ubuntu operating system in the Linux system, and is equipped with multiple powerful NVIDIA GeForce RTX 3090 graphics processors and the open source machine learning library Pytorch, which can provide efficient and stable computing support for the operation of the image restoration method.

[0117] Example 3

[0118] This embodiment is used to implement the principle of the above method embodiment to construct an image restoration module based on dynamic decomposition and fusion, including: a decomposition submodule, a processing submodule and a fusion submodule; the decomposition submodule is used to decompose the image to be restored according to the degradation mode, and extract the same number of degradation features as the number of types of degradation modes; the processing submodule is used to process each degradation feature separately to obtain the restoration feature corresponding to each degradation mode; the fusion submodule is used to dynamically fuse the restoration features to obtain a restoration feature map of the same size as the image to be restored. Figure 2 In the proposed method, multiple mixed feature decomposition submodules, processing submodules and fusion submodules are stacked together. By stacking modules in this way, the advantages of each submodule can be effectively integrated, their roles can be fully utilized, and the quality of image restoration can be improved.

[0119] During the training process of the convolutional neural network, different convolution kernel groups focus on different aspects. This characteristic makes the response values ​​obtained by the convolutional neural network different for different degradation modes. However, there are often certain similarities between the same degradation modes, and the response values ​​obtained by the convolutional neural network are roughly the same. Based on this significant characteristic, the hyperparameter k of the number of degradation factor types is determined in advance, and the complex mixed features are split according to the degradation mode, and the degradation features are decomposed into k groups. The convolutional neural network is trained to learn to deal with different degradation modes. This decomposition method can more accurately process different degradation modes.

[0120] The decomposition submodule is used to decompose the acquired image to be restored according to the degradation mode, and extract degradation features whose number is the same as the number of types of the degradation mode; Figure 3 The schematic diagram of the decomposition submodule is as follows: Figure 4 The figure is a schematic diagram of the mask screening submodule in the decomposition submodule. In order to decompose the degradation factors, the k-means classification method is used. This method can accurately classify complex degradation factors based on specific rules and algorithms. First, the hyperparameters are determined based on the common degradation factors in the mixed degraded image. k ,in kIt is the number of degradation factor types. Then the number of channels is divided according to the number of degradation factors, and the channels with similar characteristic responses are grouped together while the channels with dissimilar characteristic responses are separated, thereby achieving the decoupling of different types of degradation factors.

[0121] The processing submodule is used to process each degradation feature separately to obtain the restoration features corresponding to each degradation mode; the convolutional neural network used in the processing submodule is the ResNet network. The ResNet architecture consists of five stages. By introducing the residual connection ResNet18, information can be directly transmitted between different network layers, which makes it easier to train the network and build a deeper network structure, and performs well in the restoration task.

[0122] Figure 5 The fusion submodule is used to dynamically fuse the restored features to obtain a restored feature map of the same size as the image to be restored.

[0123] Each sub-module is mainly used to implement each step of the method embodiment, which will not be described in detail here.

[0124] Example 4

[0125] See also Figure 2 This embodiment is based on the embodiment 3, and is used to implement the principle of the above method embodiment to construct an image restoration network structure, which is a symmetrical U-shaped network, including corresponding N downsampling layers and N upsampling layers, and also includes (2N+1) image restoration modules, and the image restoration module is the image restoration module based on dynamic decomposition and fusion in embodiment 3 (not described here); wherein N is a non-zero natural number; the downsampling layer is the encoding layer (Encoder), and the upsampling layer is the decoding layer (Decoder), and in this embodiment, N=3. Image restoration modules are provided between adjacent downsampling layers and between adjacent upsampling layers, and image restoration modules are also provided between the downsampling layer and the upsampling layer connected at the bottom of the U-shaped network.

[0126] The output image of the downsampling layer is decomposed, processed, and fused through the image restoration module, and the obtained restored feature map is used as the input of the next downsampling layer on the one hand; on the other hand, the obtained restored feature map is added to the output image of the upsampling layer of the same size, and then decomposed, processed, and fused through the image restoration module, and the obtained restored feature map is used as the input of the next upsampling layer. For details, please refer to Example 2, which will not be described in detail here.

[0127] The downsampling layer is used to halve the size of the restored feature map and double the number of channels; the upsampling layer is used to double the size of the restored feature map and halve the number of channels.

[0128] This embodiment decomposes, processes and fuses the restored feature maps at multiple scales to obtain the restored feature maps at the corresponding scales; then the restored feature maps are merged scale by scale to obtain a restored feature map of the same size as the image to be restored and that fuses the restored features of multiple scales; this symmetrical U-shaped network can fully combine feature information at different levels, so that the network can better capture the details and structure of the image during the image restoration process, improve the visual effect of mixed image restoration, and greatly improve the effect and quality of image restoration.

[0129] This embodiment uses an atmospheric turbulence distortion video sequence dataset for training and testing, and the dataset includes a simulation dataset, a physical dataset, and a real-world dataset. This embodiment randomly crops the size of the input image to 224×224 during the training phase; uses the original image during the test phase; uses the back-propagation algorithm to set the batch size to 16 to train the proposed model; uses the SGD optimizer and batch normalization to train the model, and adjusts the learning rate through the StepLR schedule; considers the effectiveness of image restoration, uses the Adam optimizer to optimize it, and adjusts the learning rate through the cosine annealing schedule; sets the initial learning rate of all datasets to 0.0001.

[0130] It should be pointed out that, according to the needs of implementation, the various steps / components described in this application can be split into more steps / components, and two or more steps / components or partial operations of steps / components can be combined into new steps / components to achieve the purpose of the present invention.

[0131] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. An image restoration method based on dynamic decomposition and fusion, characterized in that: The following steps are involved: S1: Obtain an image to be restored of any size with mixed features; S2: Decompose the image to be restored according to the degradation pattern, and extract degradation features whose number is the same as the number of types of the degradation pattern; S3: Process each degradation feature separately to obtain the restoration feature corresponding to each degradation mode; S4: Dynamically fuse the restored features to obtain a restored feature map of the same size as the image to be restored; the specific steps are: S41: Add and restore features at the pixel level to obtain new features fused by multiple degradation factors; S42: Perform a global pooling operation on the new features to obtain a global channel descriptor; S43: calculating an intermediate variable for connecting different degradation factors according to the global channel descriptor; calculating a corresponding query weight coefficient matrix and a value coefficient matrix according to each restored feature; The corresponding intermediate features are calculated according to the query weight coefficient matrix, intermediate variables and value coefficient matrix corresponding to each restored feature, and the output features corresponding to each restored feature are calculated according to the intermediate features; S44: Perform a concatenation operation on the output features in the channel dimension to obtain the final features; S45: Use a 3×3 convolution operation to restore the channel of the feature map corresponding to the final feature to the same channel as the image to be restored, and add the restored feature map corresponding to the final feature to the image to be restored to obtain a clear restored feature map.

2. The image restoration method based on dynamic decomposition and fusion according to claim 1, characterized in that: In the step S2, the specific steps are: S21: extracting a mixed feature from the image to be restored through a convolution operation; S22: According to the degradation pattern of the image to be restored, the k-means classification method is used to decompose the mixed features to obtain multiple different features containing a single degradation pattern.

3. The image restoration method based on dynamic decomposition and fusion according to claim 2, characterized in that: In the step S22, the specific steps are: S221: Determine the number of types of degradation factors according to the degradation mode of the image to be restored; S222: dividing the number of channels according to the number of types of degradation factors, and classifying the features obtained by decomposing the mixed features according to similar feature responses to decouple different types of degradation factors; the sum of the number of channels occupied by the degradation factors is the total number of channels; S223: Perform self-attention operation on the classified features, perform mask screening on the results after the self-attention operation and the global features, calculate the top-k largest response values ​​of the results after the self-attention operation, and then select the corresponding response according to the input of the Scatter function to extract degradation features with the same number as the number of types of degradation patterns.

4. The image restoration method based on dynamic decomposition and fusion according to claim 1, characterized in that: In the step S43, the specific steps are: S431: Using a 1×1 convolution operation on the global channel descriptor to obtain the relationship between different channels, and obtaining an intermediate variable for connecting different degradation factors; S432: Perform a global pooling operation on each restored feature to obtain a corresponding query weight coefficient matrix; S433: performing a global pooling operation on each restored feature and then performing a 1×1 convolution operation to obtain a corresponding value coefficient matrix; S434: Perform self-attention calculation on the query weight coefficient matrix, intermediate variables and value coefficient matrix corresponding to each restored feature to obtain the corresponding intermediate feature; S435: The intermediate feature is coefficientized through the Sigmoid function, and is element-wise multiplied with the intermediate feature to obtain the output feature corresponding to each restored feature.

5. The image restoration method based on dynamic decomposition and fusion according to claim 1, characterized in that: Perform multi-layer symmetrical downsampling and upsampling on the restored feature map through a symmetrical U-shaped network; The image after each downsampling is decomposed, processed and fused in sequence according to the degradation mode. On the one hand, the restored feature map is used as the input of the next downsampling; on the other hand, the restored feature map is added to the upsampled image of the same size, and then decomposed, processed and fused in sequence according to the degradation mode, and the restored feature map is used as the input of the next upsampling.

6. The image restoration method based on dynamic decomposition and fusion according to claim 5, characterized in that: After adding the restored feature map obtained by fusion after the last upsampling and the downsampling of the same size, the image is decomposed, processed and fused in sequence according to the degradation mode. The restored feature map is added to the image to be restored to obtain the final restored image as the restored result map.

7. An image restoration module based on dynamic decomposition and fusion, characterized in that: An acquisition submodule is used to acquire an image to be restored of any size with mixed features; A decomposition submodule, used for decomposing the image to be restored according to the degradation mode, and extracting degradation features whose number is the same as the number of types of the degradation mode; A processing submodule, used for processing each degradation feature separately to obtain a restoration feature corresponding to each degradation mode; The fusion submodule is used to dynamically fuse the restored features to obtain a restored feature map of the same size as the image to be restored; specifically, it includes: adding the restored features at the pixel level to obtain a new feature fused by multiple degradation factors; performing a global flat pooling operation on the new features to obtain a global channel descriptor; calculating the intermediate variables used to connect different degradation factors according to the global channel descriptor; calculating the corresponding query weight coefficient matrix and value coefficient matrix according to each restored feature; calculating the corresponding intermediate features according to the query weight coefficient matrix, intermediate variables and value coefficient matrix corresponding to each restored feature, and calculating the output features corresponding to each restored feature according to the intermediate features; performing a connection operation on the output features in the channel dimension to obtain the final features; using a 3×3 convolution operation to restore the channels of the feature map corresponding to the final features to the same channels as the image to be restored, and adding the restored feature map corresponding to the final features to the image to be restored to obtain a clear restored feature map.

8. An image restoration network structure, characterized in that: A symmetrical U-shaped network, comprising a plurality of corresponding downsampling layers and upsampling layers, and also comprising a plurality of image restoration modules, wherein the image restoration module is the image restoration module based on dynamic decomposition and fusion as described in claim 7; An image restoration module is provided between adjacent downsampling layers and between adjacent upsampling layers, and an image restoration module is also provided between the downsampling layer and the upsampling layer connected at the bottom of the U-shaped network; The output image of the downsampling layer is decomposed, processed and fused through the image restoration module. On the one hand, the restored feature map is used as the input of the next downsampling layer; on the other hand, the restored feature map is added to the output image of the upsampling layer of the same size, and then decomposed, processed and fused through the image restoration module, and the restored feature map is used as the input of the next upsampling layer.

9. The image restoration network structure according to claim 8, characterized in that: The downsampling layer is used to halve the size of the restored feature map and double the number of channels; The upsampling layer is used to double the size of the restored feature map and halve the number of channels.

Images