Image restoration method for enhancing spatial feature optimization

Abstract

The invention provides an image restoration method for enhancing spatial feature optimization. The method comprises the following steps: re-sampling an image to be processed into three sizes; constructing an encoder-decoder network model, respectively inputting the images of three sizes into corresponding encoders of the model for encoding operation, and fusing the multi-source feature maps in the encoders, and then the processed signals are sent to the next encoder and the corresponding decoder after being processed by the first convolution layer, the seven continuous residual convolution structures, the second convolution layer and the third convolution layer in sequence. And in each decoder, fusing the multi-source feature maps, processing the fused multi-source feature maps through a fourth convolution layer, seven continuous residual convolution structures, a fifth convolution layer, a spatial feature optimization module and a sixth convolution layer, and sending a processing result to the next decoder until the output of the last decoder is obtained. And taking the output of the last decoder as a final prediction result. According to the method, image restoration of multiple types of degraded images can be realized, and the image restoration efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image restoration, and in particular to an image restoration method with enhanced spatial feature optimization. Background Art

[0002] In multimedia data processing, recovering clear images from degraded visual data is crucial, and it is the cornerstone for driving downstream technology applications such as autonomous driving and photographic imaging. Image degradation appears in various forms, such as rain, fog, motion blur, etc., bringing diverse challenges. Traditional image degradation removal methods rely on manual design and often fail to address the complexity and diversity of real scenes. In recent years, deep learning technology has developed rapidly and demonstrated significant efficacy in multimedia data processing, which has promoted the progress of image restoration (IR) and achieved remarkable results. The evolution of image restoration technology requires not only the acquisition of high-quality data, but also the innovative implementation of model design and advanced mechanisms, which is crucial to improving the accuracy of image restoration.

[0003] The widespread use of encoder-decoder architectures and multi-dimensional attention mechanisms is attributed to their efficient and concise designs. At the same time, the emergence of the Transformer model, known for its self-attention mechanism, marks a shift from traditional Convolutional Neural Network (CNN)-based models to models that rely on the Transformer architecture. Despite their enhanced ability to capture global features and enrich the visual quality of image restoration, Transformers also introduce more model parameters and computational requirements. For effective image restoration, key considerations include reducing model parameters and computational complexity. Although some existing image restoration methods focus on spatial domain feature learning and modeling, which can enhance image restoration effects, this comes at the cost of increased computation.

[0004] Therefore, how to provide an efficient method for restoring images of various types of degradation, and how to restore various types of degraded images while ensuring efficiency has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide an image restoration method with enhanced spatial feature optimization, which can be applied to the restoration of images of different degradation types, improve the image restoration quality while reducing the computational complexity, and realize an efficient image restoration process.

[0006] The present invention is implemented as follows: an image restoration method for enhancing spatial feature optimization, the method comprising:

[0007] Step S1, resampling the image to be processed into three sizes, namely, original size, 1 / 2 original size and 1 / 4 original size;

[0008] Step S2, constructing a network model, wherein the model includes a first encoder, a second encoder, a third encoder, a third decoder, a second decoder and a first decoder connected in sequence;

[0009] Step S3, input the images of the three sizes into the corresponding encoders of the model for encoding operations, fuse the input different image features in the encoder to obtain the corresponding feature map, and reduce the number of channels of the feature map to the number of channels matched by the current encoder through a first convolution layer, and then sequentially pass through seven consecutive residual convolution structures, a second convolution layer, and a third convolution layer to send the processing results to the next encoder and the corresponding decoder, respectively, wherein the image of the original size is input to the first encoder, the image of 1 / 2 of the original size is input to the second encoder, and the image of 1 / 4 of the original size is input to the third encoder, the input of the second encoder also includes the output of the first encoder, and the input of the third encoder also includes the output of the second encoder;

[0010] Step S4, in each decoder, different input feature maps are fused to obtain corresponding feature maps, and the number of feature map channels is reduced to the number of channels matched by the current decoder through a fourth convolution layer, and then the processing results are sent to the next decoder after being processed by seven consecutive residual convolution structures, a fifth convolution layer, a spatial feature optimization module and a sixth convolution layer, until the output of the last decoder is obtained, the input of the third decoder includes the output of the third encoder, the input of the second decoder includes the output of the second encoder and the output of the third decoder, and the input of the first decoder includes the output of the first encoder and the output of the second decoder;

[0011] Step S5: Use the output of the last decoder as the final prediction result.

[0012] Furthermore, before each encoder, the resampled image features of the corresponding size are first passed through a seventh convolutional layer to increase the number of feature map channels to the number of channels corresponding to the current encoder and then input into the current encoder.

[0013] Furthermore, the fusion between different feature maps is performed by connection.

[0014] Furthermore, the optimization process of the spatial feature optimization module is specifically as follows: first, the input feature map is differentiated into three different scales, including downsampling the first branch feature map by an average of 2 times to obtain the first feature map I / 2, downsampling the second branch map by an average of 4 times to obtain the second feature map I / 4, and the third feature map without processing the input feature map, and then I / 2 and I / 4 are split into slice features with specific window sizes respectively, and non-local operations are performed within all window slice features to establish non-local associations within the local area, and then all slice features are reorganized, and the I / 4 reorganized features are subjected to global self-attention operations in the spatial dimension, and then the first branch is upsampled by 2 times, and the second branch is upsampled by 4 times, and then the unprocessed third branch is subjected to pixel-level addition operations with the processed first branch and third branch, and then sent to two learnable weighted filters for adjustment and output feature maps.

[0015] Furthermore, the processing process of the two learnable weighted filters is specifically as follows: the feature maps are respectively passed into two 5×5 convolutional layers, wherein one convolutional layer is used to learn the low-pass filter, and the other convolutional layer is used for convolution processing and the low-pass filter is subtracted from the convolutional processed features to obtain a high-pass filter, and then the low-pass filter and the high-pass filter are connected and output through a convolutional layer.

[0016] Furthermore, the step S5 also includes: after each decoder, an eighth convolutional layer is used to compress the number of channels of the feature map to output a restored map of the current size.

[0017] Furthermore, the model is trained using deep supervision technology, using the output of each eighth convolutional layer to calculate the loss.

[0018] The present invention has the following advantages: the present invention constructs a lightweight image restoration model based on the optimization of local and non-local attribute features in the spatial dimension, passes the degraded image into the encoder for encoding, obtains a multi-level feature map, supports multi-scale degraded image input for fusion in the encoding stage, and passes the multi-level feature map obtained in the encoding stage into the decoder for image restoration, and trains the model in a deep supervision manner to output three-stage restoration result maps. In each decoding module, the core spatial feature optimization module proposed by the present invention is included, which combines multi-scale feature learning and fusion, a non-local attention mechanism based on local window features, and a weighted filter modulation technology to fully utilize spatial domain features, effectively process degraded elements of different granularities, and achieve efficient and high-quality image restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0020] Figure 1The present invention is a flowchart of an image restoration method for enhancing spatial feature optimization.

[0021] Figure 2 It is a schematic diagram of the network model structure of the present invention.

[0022] Figure 3 This is a schematic diagram of the structure of the spatial feature optimization module of the present invention. DETAILED DESCRIPTION

[0023] The embodiments of the present application are aimed at the problems in existing image restoration processing methods, such as insufficient utilization of local and non-local type information in spatial regions, inability to meet the requirements in processing speed and performance, resulting in poor restoration effect, and inability to apply to various types of image degradation. The present invention proposes an image restoration method that enhances spatial feature optimization and efficiently restores various types of degradation, and significantly improves the effect. The overall design idea of ​​the present invention is as follows: The present invention proposes a lightweight image restoration model based on optimization of local and non-local attribute features in spatial dimensions, and improves the restoration quality by optimizing local and non-local attribute features in the image space domain. The overall algorithm flow of the present invention includes the following steps: 1) The degraded image is passed into the encoder for encoding to obtain a multi-level feature map. In the encoding stage, multi-scale degraded image input is supported for fusion processing. 2) The multi-level feature map obtained in the encoding stage is passed into the decoder for image restoration, and the model is trained in a deep supervision manner to output the restoration result map of the three stages, and the output of the last decoder is used as the prediction map output. In addition, in each decoding module, the core spatial feature optimization module proposed in the present invention is embedded. The spatial feature optimization module of the present invention combines multi-scale feature learning and fusion, non-local attention mechanism based on local window features and weighted filter modulation technology to make full use of spatial domain features, which can more effectively process degraded elements of different granularities, while reducing the amount of calculation and achieving high-quality image restoration.

[0024] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0025] See also Figures 1 to 3 As shown, an image restoration method for enhancing spatial feature optimization of the present invention comprises the following steps:

[0026] Step S1, resampling the image to be processed into three sizes, namely, original size, 1 / 2 original size and 1 / 4 original size;

[0027] Step S2, constructing a network model, wherein the model includes a first encoder, a second encoder, a third encoder, a third decoder, a second decoder and a first decoder connected in sequence;

[0028] Step S3, input the images of the three sizes into the corresponding encoders of the model for encoding operations, fuse the input different image features in the encoder to obtain the corresponding feature map, and reduce the number of channels of the feature map to the number of channels matched by the current encoder through a first convolution layer, and then sequentially pass through seven consecutive residual convolution structures, a second convolution layer, and a third convolution layer to send the processing results to the next encoder and the corresponding decoder, respectively, wherein the image of the original size is input to the first encoder, the image of 1 / 2 of the original size is input to the second encoder, and the image of 1 / 4 of the original size is input to the third encoder, the input of the second encoder also includes the output of the first encoder, and the input of the third encoder also includes the output of the second encoder;

[0029] Step S4: In each decoder, different input feature maps are fused to obtain corresponding feature maps, and the number of feature map channels is reduced to the number of channels matched by the current decoder through a fourth convolution layer, and then the processing results are sent to the next decoder after being processed by seven consecutive residual convolution structures, the fifth convolution layer, the spatial feature optimization module and the sixth convolution layer, until the output of the last decoder is obtained. The input of the third decoder includes the output of the third encoder, the input of the second decoder includes the output of the second encoder and the output of the third decoder, and the input of the first decoder includes the output of the first encoder and the output of the second decoder. When the feature map is passed into the spatial feature optimization module, the input feature map is differentiated into different scales, and window non-local connections and global spatial self-attention are established. The structure of the encoder and decoder is as follows: Figure 2 As shown in the figure, the decoder has an additional spatial feature optimization module compared to the encoder. The convolutional layer in the encoder is mainly used to increase the network's ability to extract features, while the convolutional layer in the decoder is mainly used to enhance feature expression capabilities.

[0030] Step S5: Use the output of the last decoder (ie, the first decoder) as the final prediction result.

[0031] Preferably, before each encoder, the resampled image features of the corresponding size are first passed through a seventh convolutional layer to increase the number of channels of the feature map to the number of channels corresponding to the current encoder, and then input into the current encoder. For example, the number of channels of the first encoder is 32, the number of channels of the second encoder is 64, and the number of channels of the third encoder is 128; when the image is input into the first encoder, it is necessary to pass through a convolutional layer to increase the number of channels of the image with 3 to 32 before inputting into the encoder for processing;

[0032] The output expression of the connection operation in the above decoder and encoder and the first convolution (i.e. the first convolution layer or the fourth convolution layer) operation is as follows:

[0033] Out=Conv 1×1(Concat(I1, I2))

[0034] Among them, Conv 1×1 Represents a convolution operation, which is used to reduce the number of channels. Concat(I1, I2) represents the concatenation of the feature maps I1 and I2 input from two sources.

[0035] Preferably, the fusion between different feature maps (i.e., feature maps from two sources) is fused in a concatenated manner. In each module, the feature maps from two sources are fused in a concatenated manner to enrich the feature information, and then the number of channels is compressed through the convolution layer to prepare for transmission to the next module for processing, maintaining consistency.

[0036] Preferably, the seven consecutive residual convolution structures in the encoder and the decoder are processed in the same manner, and each residual convolution structure is processed as follows: the input is differentiated into two branches, one of which is not processed, and the other branch is added to the unprocessed branch after two convolution processes to obtain the output of the current residual convolution structure;

[0037] Preferably, after the seven consecutive residual convolution structures in the encoder and decoder, the outputs of the seven consecutive residual convolution structures are further differentiated into two branches, one of which is not processed, and the other branch continues to perform corresponding processing and is added to the unprocessed branch as the output of the encoder or decoder.

[0038] Preferably, the optimization process of the spatial feature optimization module is specifically as follows:

[0039] First, the input feature map is differentiated into three different scales, including downsampling the first branch feature map by an average of 2 times to obtain the first feature map I / 2, downsampling the second branch map by an average of 4 times to obtain the second feature map I / 4, and the third feature map without processing the input feature map;

[0040] The expression of the downsampling operation is as follows:

[0041] I / 2 = AveragePooling2(I)

[0042] I / 4 = AveragePooling4(I)

[0043] AveragePooling(·) represents the global average downsampling operation, where I is the input feature map;

[0044] Then, I / 2 and I / 4 are split into slice features with specific window sizes, and non-local operations are performed on all window slice features to establish non-local associations within the local area, and then all slice features are reorganized. Specifically, I / 4 is taken as an example here, and the expression is as follows:

[0045] {W i}=unfold(I / 4, window_size=7)

[0046] {W′ i}=Non_Local({W i})

[0047] I / 4 = fold({W′ i})

[0048] Among them, unfold(·) represents the operation of windowing the feature map of the input, {W i} represents the window feature set, Non_Local(·) represents the non-local operation, and fold(·) represents reaggregating all window features into the feature size of the original input;

[0049] In addition, a global self-attention operation is performed on the I / 4 reorganized features in the spatial dimension. The specific expression is I / 4=SA(I / 4), where SA(·) represents the self-attention operation in the spatial dimension.

[0050] Then the first branch is upsampled by 2 times, the second branch is upsampled by 4 times, and the unprocessed third branch is added to the processed first and third branches at the pixel level. Here, the two branch feature maps are upsampled, and the size is aligned with the input feature map. The output expression of the pixel-level addition operation is:

[0051] Out=Ⅰ+UP×2(I / 2)+UP×4(I / 4);

[0052] Where UP(·) represents upsampling operation;

[0053] It is then sent to two learnable weighted filters for adjustment and output feature maps.

[0054] Through the above operations, the model's feature learning and expression at both local and global scales can be enhanced. In addition, multi-scale feature operations help to process degraded elements of various granularities.

[0055] Preferably, the processing process of the two learnable weighted filters is as follows: the feature maps are respectively passed to two 5×5 convolutional layers, one of which is used to learn the low-pass filter, and the other is used for convolution processing and the low-pass filter is subtracted from the convolution processed features to obtain a high-pass filter, and then the low-pass filter and the high-pass filter are connected and then output through a convolutional layer, and the number of channels is halved through the convolutional layer. The processing process can be expressed by the following expression:

[0056] F res=ReLU(Conv 5×5 (I))

[0057] H'=ReLU(Conv 5×5 (I))

[0058] H=H'-F res

[0059] Out=Conv 1×1 (Concat(F res , H))

[0060] Among them, F res represents a low-pass filter, H represents a high-pass filter, ReLU(.) represents an activation function, I represents the feature image input to the 5×5 convolutional layer, Conv 5×5 represents a convolution operation, and Concat(.) represents a concatenation operation.

[0061] Through the above operations, the model's processing of detailed textures and noise in the image can be dynamically balanced.

[0062] Preferably, the step S5 further comprises: after each decoder, compressing the number of channels of the feature map (such as compressing to 3 channels) through an eighth convolutional layer to output a restored map of the current size.

[0063] Preferably, the model is trained using deep supervision technology, using the output of each eighth convolutional layer to calculate the loss.

[0064] A lightweight model based on a convolutional neural network of the present invention improves the restoration quality by optimizing the local and non-local attribute features in the image space domain, which is of great significance for restoring degraded images. Compared with previous methods, this method has advantages in processing speed and performance. The present invention uses the classic encoder-decoder mode to build a model. The degraded image is first resampled to different sizes to match the feature scales of each stage of the encoder. Then, starting from the first decoder module, the data is processed in sequence, and the output of the encoder module is sent to the next module and the corresponding module of the decoder at the same time. Each decoder processes the input feature map, and its output map is sent to the next module for processing and the side convolution layer respectively. The side convolution layer (i.e., the eighth convolution layer) of the decoder is used to output the restoration map of the current size. The overall model is trained in a deep supervised manner, which can more effectively optimize the model parameters. Each encoder and decoder contains three modules mainly composed of residual networks, and a carefully designed enhanced spatial feature optimization module is embedded in the residual structure of the decoder, which combines multi-scale feature learning and fusion, non-local attention mechanism based on local window features, and weighted filter modulation technology to make full use of spatial domain features, optimize local and global features, effectively process degraded elements of different granularities, and achieve high-quality image restoration.

[0065] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. An image restoration method with enhanced spatial feature optimization, characterized in that: The method comprises: Step S1, resampling the image to be processed into three sizes, namely, original size, 1 / 2 original size and 1 / 4 original size; Step S2, constructing a network model, wherein the model includes a first encoder, a second encoder, a third encoder, a third decoder, a second decoder and a first decoder connected in sequence; Step S3, input the images of the three sizes into the corresponding encoders of the model for encoding operations, fuse the input different image features in the encoder to obtain the corresponding feature map, and reduce the number of channels of the feature map to the number of channels matched by the current encoder through a first convolution layer, and then sequentially pass through seven consecutive residual convolution structures, a second convolution layer, and a third convolution layer to send the processing results to the next encoder and the corresponding decoder, respectively, wherein the image of the original size is input to the first encoder, the image of 1 / 2 of the original size is input to the second encoder, and the image of 1 / 4 of the original size is input to the third encoder, the input of the second encoder also includes the output of the first encoder, and the input of the third encoder also includes the output of the second encoder; Step S4, in each decoder, different input feature maps are fused to obtain corresponding feature maps, and the number of feature map channels is reduced to the number of channels matched by the current decoder through a fourth convolution layer, and then the processing results are sent to the next decoder after being processed by seven consecutive residual convolution structures, a fifth convolution layer, a spatial feature optimization module and a sixth convolution layer, until the output of the last decoder is obtained, the input of the third decoder includes the output of the third encoder, the input of the second decoder includes the output of the second encoder and the output of the third decoder, and the input of the first decoder includes the output of the first encoder and the output of the second decoder; Step S5: Use the output of the last decoder as the final prediction result.

2. The image restoration method according to claim 1, characterized in that: Before each encoder, the resampled image features of the corresponding size are first passed through a seventh convolutional layer to increase the number of feature map channels to the number of channels corresponding to the current encoder and then input into the current encoder.

3. The image restoration method with enhanced spatial feature optimization according to claim 1, characterized in that: The fusion between different feature maps is performed by connection.

4. The image restoration method with enhanced spatial feature optimization according to claim 1, characterized in that: The optimization process of the spatial feature optimization module is as follows: first, the input feature map is differentiated into three different scales, including downsampling the first branch feature map by 2 times on average to obtain the first feature map I / 2 , the second branch graph is downsampled by 4 times on average to obtain the second feature graph I / 4 And the third feature map without processing the input feature map, and then I / 2 and I / 4 Split into slice features with a specific window size, perform non-local operations on all window slice features to establish non-local associations within the local area, reorganize all slice features, and / 4 The reorganized features are subjected to global self-attention operation in the spatial dimension, and then the first branch is upsampled by 2 times, the second branch is upsampled by 4 times, and the unprocessed third branch is added to the processed first and third branches at the pixel level. The unprocessed third branch is then sent to two learnable weighted filters for adjustment and the output feature map is output.

5. The image restoration method with enhanced spatial feature optimization according to claim 4, characterized in that: The processing process of the two learnable weighted filters is specifically as follows: the feature maps are respectively passed into two 5×5 convolutional layers, one of which is used to learn the low-pass filter, and the other is used for convolution processing and the high-pass filter is obtained by subtracting the low-pass filter from the convolution processed features, and then the low-pass filter and the high-pass filter are connected and output through a convolution layer.

6. The image restoration method with enhanced spatial feature optimization according to claim 1, characterized in that: The step S5 also includes: after each decoder, an eighth convolutional layer is used to compress the number of channels of the feature map to output a restored map of the current size.

7. The image restoration method with enhanced spatial feature optimization according to claim 6, characterized in that: The model is trained using deep supervision techniques, using the output of every eighth convolutional layer to calculate the loss.

Images