Image restoration method for enhancing spatial feature optimization

Abstract

The application provides an image restoration method for enhancing spatial feature optimization, comprising: resampling a to-be-processed image into three sizes; constructing an encoder-decoder network model, inputting the images of the three sizes into corresponding encoders of the model respectively for encoding operation, fusing multi-source feature maps in the encoders, and then sequentially sending the processed results to the next encoder and the corresponding decoder after the processed results are processed by a first convolution layer, seven continuous residual convolution structures, a second convolution layer and a third convolution layer; in each decoder, fusing the multi-source feature maps, and then sending the processed results to the next decoder after the processed results are processed by a fourth convolution layer, seven continuous residual convolution structures, a fifth convolution layer, a spatial feature optimization module and a sixth convolution layer; and obtaining the output of the last decoder as the final prediction result. The method can realize image restoration of multiple types of degraded images and improve the image restoration efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image restoration, and in particular to an image restoration method for enhancing spatial feature optimization. BACKGROUND

[0002] In multimedia data processing, it is essential to restore clear images from degraded visual data, which is the cornerstone of driving downstream technology applications such as automatic driving and photographic imaging. Image degradation occurs in various forms, such as rain, fog, motion blur, etc., bringing diverse challenges. Traditional image degradation removal methods rely on manual design, which often fails to address the complexity and diversity of real-world scenarios. In recent years, deep learning technology has rapidly developed and shown remarkable effectiveness in multimedia data processing, thus driving the progress of image restoration (IR) and achieving significant results. The evolution of image restoration technology not only requires the acquisition of high-quality data, but also the innovative implementation of model design and advanced mechanisms, which is crucial for improving image restoration accuracy.

[0003] The widespread application of encoder-decoder architecture and multi-dimensional attention mechanism is attributed to their efficient and simple design. At the same time, the emergence of the Transformer model, known for its self-attention mechanism, marks a shift from traditional Convolutional Neural Network (CNN) based models to models relying on Transformer architecture. Although they have the ability to capture global features and enhance the visual quality of image restoration, Transformer also introduces more model parameters and computational requirements. For effective image restoration, key considerations include reducing model parameters and computational complexity. Although some existing image restoration methods focus on spatial domain feature learning and modeling, which can enhance image restoration results, this is at the cost of increased computation.

[0004] Therefore, how to provide an efficient method suitable for image restoration of various degradation types, achieve image restoration of various degradation types and ensure efficiency, has become a technical problem to be solved. SUMMARY

[0005] The technical problem to be solved by the present application is to provide an image restoration method for enhancing spatial feature optimization, which can be applied to the restoration of images of different degradation types, improve image restoration quality while reducing computational complexity, and realize an efficient image restoration process.

[0006] The present application is implemented as follows: an image restoration method for enhancing spatial feature optimization, the method comprising:

[0007] Step S1, resample the image to be processed into three sizes, namely the original size, 1 / 2 original size and 1 / 4 original size;

[0008] Step S2, construct a network model, which includes a first encoder, a second encoder, a third encoder, a third decoder, a second decoder and a first decoder connected in turn;

[0009] Step S3, input the images of the three sizes into the corresponding encoders of the model for encoding operation, fuse the features of the different input images in the encoders to obtain corresponding feature maps, and reduce the channel number of the feature maps to the channel number matched by the current encoder through a first convolutional layer, then sequentially pass through seven consecutive residual convolutional structures, a second convolutional layer and a third convolutional layer, and send the processing results to the next encoder and the corresponding decoder, respectively, wherein the image of the original size is input into the first encoder, the image of 1 / 2 original size is input into the second encoder, and the image of 1 / 4 original size is input into the third encoder, the input of the second encoder further includes the output of the first encoder, and the input of the third encoder further includes the output of the second encoder;

[0010] Step S4, in each decoder, fuse the different input feature maps to obtain corresponding feature maps, and reduce the channel number of the feature maps to the channel number matched by the current decoder through a fourth convolutional layer, then sequentially pass through seven consecutive residual convolutional structures, a fifth convolutional layer, a spatial feature optimization module and a sixth convolutional layer, and send the processing results to the next decoder until the output of the last decoder is obtained, the input of the third decoder includes the output of the third encoder, the input of the second decoder includes the output of the second encoder and the output of the third decoder, and the input of the first decoder includes the output of the first encoder and the output of the second decoder;

[0011] Step S5, take the output of the last decoder as the final prediction result.

[0012] Further, before each encoder, the features of the image of the corresponding size resampled are input into the current encoder after the channel number of the feature maps is raised to the corresponding channel number of the current encoder through a seventh convolutional layer.

[0013] Further, the fusion between different feature maps is performed in a joint manner.

[0014] Further, the optimization process of the spatial feature optimization module is specifically: first, the input feature map is differentiated into three different scales, including performing average down-sampling by 2 times on the first branch feature map to obtain a first feature map I / 2, performing average down-sampling by 4 times on the second branch feature map to obtain a second feature map I / 4, and not processing the third feature map of the input feature map, then splitting the I / 2 and I / 4 into windowed features respectively with a specific window size, performing non-local operation on all windowed features to establish non-local correlation within the local, recombining all windowed features, and performing global self-attention operation on the recombined features in the I / 4 in the spatial dimension, then up-sampling the first branch by 2 times and the second branch by 4 times, and then performing pixel-level addition operation on the third branch which is not processed and the first branch and the third branch which are processed, and then inputting into two learnable weighting filters to adjust and output the feature map.

[0015] Further, the processing process of the two learnable weighting filters is specifically: the feature map is respectively input into two 5x5 convolution layers, one of which is used to learn a low-pass filter, and the other is used for convolution processing and subtracting the low-pass filter from the feature after convolution processing to obtain a high-pass filter, then the low-pass filter and the high-pass filter are connected, and then output through a convolution layer.

[0016] Further, the step S5 further comprises: compressing the channel number of the feature map through an eighth convolution layer after each decoder to output a current size of the recovered image.

[0017] Further, the model is trained by using the depth supervision technology, and the output of each eighth convolution layer is used to calculate the loss.

[0018] The present application has the following advantages: the present application constructs a lightweight image restoration model based on the optimization of spatial dimension local and non-local attribute features, inputs the degraded image into the encoder to obtain multi-level feature maps, supports multi-scale degraded image input for fusion in the encoding stage, and transmits the multi-level feature maps obtained in the encoding stage into the decoder for image restoration, trains the model in a depth supervision manner, and outputs three-stage restoration result images, in each decoding module, the core spatial feature optimization module proposed in the present application is included, multi-scale feature learning and fusion, non-local attention mechanism based on local window features, and weighting filter modulation technology are combined to fully utilize the spatial domain features, effectively process different granularity of degraded elements, and realize efficient and high-quality image restoration. BRIEF DESCRIPTION OF DRAWINGS

[0019] The present application will be further described below with reference to the accompanying drawings and in conjunction with the embodiments.

[0020] Fig. 1This is a flowchart illustrating the execution of an image restoration method for enhancing spatial features according to the present invention.

[0021] Fig. 2 This is a schematic diagram of the network junction model structure of the present invention.

[0022] Fig. 3 This is a schematic diagram of the spatial feature optimization module of the present invention. Detailed Implementation

[0023] This application addresses the problems in existing image restoration methods, such as insufficient utilization of local and non-local information in the spatial region, inadequate processing speed and performance, resulting in poor restoration effects and inapplicability to various image degradation types. This invention proposes a highly efficient image restoration method for various degradation types by enhancing spatial feature optimization, significantly improving the results. The overall design concept of this invention is as follows: This invention proposes a lightweight image restoration model based on the optimization of local and non-local attribute features in the spatial dimension. The restoration quality is improved by optimizing local and non-local attribute features in the image spatial domain. The overall algorithm flow of this invention includes the following steps: 1) The degraded image is input into the encoder for encoding to obtain multi-level feature maps. During the encoding stage, multi-scale degraded image inputs are supported for fusion processing. 2) The multi-level feature maps obtained in the encoding stage are input into the decoder for image restoration. The model is trained using deep supervision, outputting restoration result maps for three stages. The output of the last decoder is used as the prediction map output. Furthermore, each decoding module incorporates the core spatial feature optimization module proposed in this invention. This spatial feature optimization module combines multi-scale feature learning and fusion, a non-local attention mechanism based on local window features, and weighted filter modulation techniques to fully utilize spatial domain features. This allows for more effective processing of degenerate elements at different granularities while reducing computational load and achieving high-quality image restoration.

[0024] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0025] Please see Figs. 1-3 As shown, the present invention provides an image restoration method with enhanced spatial feature optimization, the method comprising the following steps:

[0026] Step S1: Resample the image to be processed into three sizes: the original size, half of the original size, and one-quarter of the original size.

[0027] Step S2: Construct a network model, which includes a first encoder, a second encoder, a third encoder, a third decoder, a second decoder, and a first decoder connected in sequence;

[0028] Step S3: Input the images of the three sizes into the corresponding encoders of the model for encoding. In the encoder, the different image features are fused to obtain the corresponding feature map. The feature map is then processed by a first convolutional layer to reduce the number of channels to the number of channels matched by the current encoder. After processing by seven consecutive residual convolutional structures, a second convolutional layer, and a third convolutional layer, the processing results are sent to the next encoder and the corresponding decoder. Specifically, the original size image is input into the first encoder, half the original size image is input into the second encoder, and a quarter the original size image is input into the third encoder. The input of the second encoder also includes the output of the first encoder, and the input of the third encoder also includes the output of the second encoder.

[0029] Step S4: In each decoder, the different input feature maps are fused to obtain the corresponding feature map. This feature map is then passed through a fourth convolutional layer to reduce the number of channels to match the current decoder. It is then processed sequentially through seven consecutive residual convolutional structures, a fifth convolutional layer, a spatial feature optimization module, and a sixth convolutional layer before the processing result is sent to the next decoder. This process continues until the output of the last decoder is obtained. The input of the third decoder includes the output of the third encoder, the input of the second decoder includes the outputs of the second encoder and the third decoder, and the input of the first decoder includes the outputs of the first encoder and the second decoder. When the feature map is passed to the spatial feature optimization module, it is differentiated into different scales, and window non-local connections and global spatial self-attention are established. The structures of the encoder and decoder are as follows: Fig. 2 As shown, the decoder has an additional spatial feature optimization module compared to the encoder. The convolutional layers in the encoder are mainly used to increase the network's ability to extract features, while the convolutional layers in the decoder are mainly used to enhance the feature representation ability.

[0030] Step S5: Use the output of the last decoder (i.e. the first decoder) as the final prediction result.

[0031] Preferably, before each encoder, the resampled image features of the corresponding size are first passed through a seventh convolutional layer to boost the number of channels in the feature map to the number of channels corresponding to the current encoder before being input into the current encoder. For example, if the first encoder has 32 channels, the second encoder has 64 channels, and the third encoder has 128 channels, then when the image is input into the first encoder, it needs to be boosted from 3 channels to 32 channels through a convolutional layer before being input into the encoder for processing.

[0032] The connection operations within the decoder and encoder described above, along with the output expressions for the first convolution (i.e., the first or fourth convolutional layer) operation, are as follows:

[0033] Out = Conv 1×1(Concat(I1, I2))

[0034] Among them, Conv 1×1 This represents a convolution operation used to reduce the number of channels. Concat(I1, I2) means concatenating the feature maps I1 and I2 from two different sources.

[0035] Ideally, feature maps from different sources (i.e., feature maps from two different sources) are fused using a concatenation method. In each module, feature maps from both sources are fused via concatenation, enriching the feature information. Then, convolutional layers compress the number of channels to prepare them for processing in the next module, maintaining consistency.

[0036] Preferably, the seven consecutive residual convolutional structures in the encoder and decoder are processed in the same way. Each residual convolutional structure is processed as follows: the input is divided into two branches, one of which is not processed, and the other branch is processed by two convolutions and then added to the branch that is not processed to obtain the output of the current residual convolutional structure.

[0037] Preferably, after the seven consecutive residual convolutional structures in the encoder and decoder, the output of the seven consecutive residual convolutional structures is further divided into two branches. One branch is not processed, and the other branch is processed accordingly and then added to the unprocessed branch to become the output of the encoder or decoder.

[0038] Preferably, the optimization process of the spatial feature optimization module is as follows:

[0039] First, the input feature map is divided into three different scales, including the first branch feature map being downsampled by a factor of 2 to obtain the first feature map I / 2, the second branch feature map being downsampled by a factor of 4 to obtain the second feature map I / 4, and the third feature map being unprocessed.

[0040] The expression for the downsampling operation is as follows:

[0041] I / 2 = AveragePooling2(I)

[0042] I / 4 = AveragePooling4(I)

[0043] AveragePooling(·) represents a global average downsampling operation, where I is the input feature map;

[0044] Then, I / 2 and I / 4 are each split into fragmented features with a specific window size. After performing nonlocal operations on all window fragmented features to establish nonlocal relationships within local areas, all fragmented features are recombined. Specifically, taking I / 4 as an example, the expression is as follows:

[0045] {W i}=unfold(I / 4, window_size=7)

[0046] {W′ i} = Non_Local({W i})

[0047] I / 4 = fold({W′) i})

[0048] Where unfold(·) represents the windowing operation of the input feature map, {W i} represents the window feature set, Non_Local(·) represents a non-local operation, and fold(·) represents re-aggregating all window features to the original input feature size;

[0049] Furthermore, the features recombined from I / 4 are subjected to global self-attention in the spatial dimension, specifically expressed as I / 4 = SA(I / 4), where SA(·) represents the spatial dimension self-attention operation.

[0050] Next, the first branch is upsampled by a factor of 2, and the second branch is upsampled by a factor of 4. Then, the unprocessed third branch is added pixel-wise with the processed first and third branches. Here, the feature maps of the two branches are upsampled, their dimensions are aligned with the input feature map, and the output expression of the pixel-wise addition operation is:

[0051] Out=Ⅰ+UP×2(I / 2)+UP×4(I / 4);

[0052] Where UP(·) represents the upsampling operation;

[0053] The output feature map is then fed into two learnable weighted filters for adjustment.

[0054] The above operations enhance the model's feature learning and representation at both local and global scales. Furthermore, multi-scale feature operations help handle degenerate elements of varying granularities.

[0055] Preferably, the processing of the two learnable weighted filters involves feeding the feature maps into two 5×5 convolutional layers. One convolutional layer learns a low-pass filter, while the other performs convolution processing and subtracts the low-pass filter from the convolutional features to obtain a high-pass filter. The low-pass and high-pass filters are then concatenated and output through another convolutional layer, which halves the number of channels. This processing can be represented by the following expression:

[0056] F res=ReLU(Conv 5×5 (I))

[0057] H' = ReLU(Conv) 5×5 (I))

[0058] H = H' - F res

[0059] Out = Conv 1×1 (Concat(F res H))

[0060] Among them, F res H represents a low-pass filter, H represents a high-pass filter, ReLU(.) represents the activation function, I represents the feature image input to the 5×5 convolutional layer, and Conv... 5×5 The expression represents a convolution operation, and Concat(.) represents a concatenation operation.

[0061] Through the above operations, the model can dynamically balance the processing of detail texture and noise in the image.

[0062] Preferably, step S5 further includes: after each decoder, an eighth convolutional layer is used to compress the number of channels of the feature map (e.g., compressed to 3 channels) to output a restored map of the current size.

[0063] Preferably, the model is trained using deep supervision techniques, and the loss is calculated using the output of each eighth convolutional layer.

[0064] This invention presents a lightweight model based on convolutional neural networks that improves image restoration quality by optimizing local and non-local attribute features in the image spatial domain. This is significant for restoring degraded images and offers advantages in processing speed and performance compared to previous methods. The invention utilizes a classic encoder-decoder model. First, the degraded image is resampled to different sizes to match the feature scales of each stage of the encoder. Then, starting with the first decoder module, data is processed sequentially, with the encoder module's output simultaneously fed to the next module and the corresponding decoder module. Each decoder processes the input feature map, and its output map is fed to the next module and the side-channel convolutional layer. The decoder's side-channel convolutional layer (i.e., the eighth convolutional layer) outputs the restored image at the current size. The overall model is trained using deep supervision, which more effectively optimizes model parameters. Each encoder and decoder contains three modules mainly composed of residual networks. A carefully designed enhanced spatial feature optimization module is embedded in the residual structure of the decoder. It combines multi-scale feature learning and fusion, non-local attention mechanism based on local window features, and weighted filter modulation technology to make full use of spatial domain features, optimize local and global features, effectively handle degradation elements of different granularities, and achieve high-quality image restoration.

[0065] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An image restoration method with enhanced spatial feature optimization, characterized in that: The method includes: Step S1: Resample the image to be processed into three sizes: the original size, half of the original size, and one-quarter of the original size. Step S2: Construct a network model, which includes a first encoder, a second encoder, a third encoder, a third decoder, a second decoder, and a first decoder connected in sequence; Step S3: Input the images of the three sizes into the corresponding encoders of the model for encoding. In the encoder, the different image features are fused to obtain the corresponding feature map. The feature map channel number is reduced to the number of channels matched by the current encoder through a first convolutional layer. Then, it is processed through seven consecutive residual convolutional structures, a second convolutional layer, and a third convolutional layer. The processing results are sent to the next encoder and the corresponding decoder respectively. The original size image is input into the first encoder, half the original size image is input into the second encoder, and a quarter the original size image is input into the third encoder. The input of the second encoder also includes the output of the first encoder, and the input of the third encoder also includes the output of the second encoder. Step S4: In each decoder, the different input feature maps are fused to obtain the corresponding feature map, and then the feature map channel number is reduced to the number of channels matched by the current decoder through a fourth convolutional layer. After processing through seven consecutive residual convolutional structures, a fifth convolutional layer, a spatial feature optimization module, and a sixth convolutional layer, the processing result is sent to the next decoder until the output of the last decoder is obtained. The input of the third decoder includes the output of the third encoder, the input of the second decoder includes the output of the second encoder and the output of the third decoder, and the input of the first decoder includes the output of the first encoder and the output of the second decoder. The optimization process of the spatial feature optimization module is as follows: First, the input feature map is divided into three different scales, including downsampling the first branch feature map by a factor of 2 to obtain the first feature map. The second feature map is obtained by downsampling the second branch map by a factor of 4. And the third feature map, which is not processed from the input feature map, and then... and Each window is split into fragment features based on a specific window size. Non-local operations are performed on all window fragment features to establish local non-local relationships. Then, all fragment features are recombined, and the relationships within them are... The recombined features are subjected to global self-attention in the spatial dimension. Then, the first branch is upsampled by 2 times, the second branch is upsampled by 4 times, and the unprocessed third branch is added to the processed first and third branches at the pixel level. After that, the feature map is fed into two learnable weighted filters for adjustment and output. Step S5: Use the output of the last decoder as the final prediction result, wherein the first decoder is the last decoder.

2. The image restoration method with enhanced spatial feature optimization according to claim 1, characterized in that: Before each encoder, the image features of the corresponding size obtained by resampling are first passed through a seventh convolutional layer to increase the number of channels of the feature map to the number of channels corresponding to the current encoder before being input into the current encoder.

3. The image restoration method with enhanced spatial feature optimization according to claim 1, characterized in that: The fusion of different feature maps is performed using a connection method.

4. The image restoration method with enhanced spatial feature optimization according to claim 1, characterized in that: The specific processing steps for the two learnable weighted filters are as follows: the feature maps are fed into two 5×5 convolutional layers, one of which is used to learn the low-pass filter, and the other is used for convolution processing. The low-pass filter is subtracted from the convolutional features to obtain the high-pass filter. The low-pass filter and the high-pass filter are then concatenated and output through another convolutional layer.

5. The image restoration method with enhanced spatial feature optimization according to claim 1, characterized in that: Step S5 further includes: after each decoder, an eighth convolutional layer is used to compress the number of channels of the feature map to output a restored map of the current size.

6. The image restoration method with enhanced spatial feature optimization according to claim 5, characterized in that: The model is trained using deep supervision techniques, and the loss is calculated using the output of each eighth convolutional layer.

Images