Image restoration method, system and device based on prior features and linear scanning

Abstract

The invention belongs to the technical field of image restoration, and particularly relates to an image restoration method, system and device based on priori features and linear scanning, image restoration is guided by utilizing the priori features, and the method comprises the following steps: injecting the priori features into an image to be restored through linear processing for feature extraction; performing linear processing on the prior features and injecting the prior features into the first feature map for linear scanning; and injecting the prior features into the second scanning feature map through linear processing for generating a high-quality image. Besides, in the linear scanning process, six scanning modes are adopted to carry out omnibearing scanning, so that the perception capability of easily missing areas (such as edges and textures) is enhanced, and two-dimensional global information is effectively obtained at low-complexity calculation cost.

Description

Technical Field

[0001] The present invention belongs to the technical field of image restoration, and specifically relates to an image restoration method, system, and device based on prior features and linear scanning. Background Art

[0002] Deep learning-based methods (such as convolutional neural network CNN and Transformer) have made significant progress in the field of image restoration, capable of capturing deep features of images and restoring details. However, these methods still have deficiencies in dealing with long-range dependencies, real-time performance, and multi-task adaptability. Mamba combines the state space model (SSM) with advanced deep learning techniques and uses selective state representation to dynamically adjust according to input data. Although the Mamba model performs well in capturing long-range information, it may lose local structural details when dealing with edge information of two-dimensional images, resulting in blurred local semantic information. Summary of the Invention

[0003] The present invention provides an image restoration method, system, and device based on prior features and linear scanning.

[0004] The technical solution of the present invention is as follows: The present invention provides an image restoration method based on prior features and linear scanning, including: S1: Obtain the image to be restored, compress the image to be restored into the latent feature space to obtain a conditional vector; use the conditional vector for the initial noise randomly sampled from the standard Gaussian distribution to perform iterative denoising to generate the predicted prior features; S2: Inject the predicted prior features into the image to be restored through linear processing, and after obtaining the feature map to be restored and performing a pooling operation, obtain the first tensor; After removing the width dimension of the first tensor, exchange the channel dimension and the height dimension to obtain the second tensor, and after performing a convolution operation, exchange the channel dimension and the height dimension to obtain the first branch feature map; After the first tensor undergoes a fully connected process to obtain the second branch feature map, fuse it with the first branch feature map to obtain the first feature map; S3: Inject the predicted prior features into the first feature map through linear processing to obtain the feature map to be scanned; after the feature map to be scanned undergoes deep convolution processing, perform two-way scanning in the vertical and horizontal directions from top left to bottom right and from bottom left to top right, and then obtain the first scanned feature map after Mamba processing; after multiplying the first scanned feature map by the feature map to be scanned after deep convolution processing, perform front-back direction scanning to obtain the second scanned feature map; S4: Inject the predicted prior features into the second scanned feature map, and then aggregate through point convolution and depthwise separable convolution to generate the restored image.

[0005] In the above S2, the predicted prior features are linearly processed and injected into the image to be restored, obtaining the feature map to be restored, which is realized by the formula: , realized; In the formula, is the feature map to be restored, is the image to be restored, is the predicted prior feature, ⊙ represents element-wise multiplication, represents layer normalization, 、 represents linear processing.

[0006] In the above S2, the first feature map is obtained, which is realized by the formula: , realized; In the formula, is the first feature map, is the feature map to be restored, is the first branch feature map, is the second branch feature map, represents the sigmoid function, which is used to compress the input data into the interval (0,1) to generate the attention weight; represents a 1×1 convolution; ⊙ represents element-wise multiplication, represents the fusion operation.

[0007] In the above S3, the second scanned feature map is obtained, which is realized by the formula: , realized; In the formula, is the second scanned feature map, is the feature map after performing bidirectional scans of top-left to bottom-right and bottom-left to top-right in the vertical and horizontal directions, is the feature map obtained by convolution processing on the feature map to be scanned, represents a 1×1 convolution operation, represents the pooling operation, ⊙ represents element-wise multiplication, represents the Mamba basic block operation.

[0008] In the above S4, aggregation is performed through point convolution and depthwise separable convolution to generate the restored image, which is realized by the formula: , realized; In the formula, is the restored image, is the feature map after the predicted prior features are linearly processed and injected into the second scanned feature map, is the second scanned feature map, 、 is point convolution, 、 is depthwise separable convolution as the activation function

[0009] After the aggregation of the said S4 through point convolution and depthwise separable convolution, it further includes: Taking the feature map after aggregation through point convolution and depthwise separable convolution as the image to be restored, and executing S2 until a preset condition is reached

[0010] In the said S1, the image to be restored is compressed into the latent feature space to obtain a conditional vector. After the features of the image to be restored are compressed into a global vector of a fixed size through global average pooling, two fully connected processes with ReLU activation functions are performed to obtain the conditional vector

[0011] The present invention also provides an image restoration system based on prior features and linear scanning, including: Prior feature generation module: used to obtain the image to be restored, compress the image to be restored into the latent feature space to obtain a conditional vector; use the conditional vector for the initial noise randomly sampled from the standard Gaussian distribution to perform iterative denoising to generate the predicted prior features Feature fusion module: After the predicted prior features are linearly processed and injected into the image to be restored to obtain the feature map to be restored, after a pooling operation, the first tensor is obtained After removing the width dimension of the first tensor, the channel dimension and the height dimension are exchanged to obtain the second tensor. After a convolution operation, the channel dimension and the height dimension are exchanged to obtain the first branch feature map After the first tensor is processed by a fully connected layer to obtain the second branch feature map, it is fused with the first branch feature map to obtain the first feature map Linear scanning module: After the predicted prior features are linearly processed and injected into the first feature map to obtain the feature map to be scanned; after the feature map to be scanned is subjected to deep convolution processing, a two-way scan of upper left - lower right and lower left - upper right is performed longitudinally and horizontally, and after Mamba processing, the first scanned feature map is obtained; after the first scanned feature map is multiplied by the feature map to be scanned after deep convolution processing, a front - back direction scan is performed to obtain the second scanned feature map Restoration module: After the predicted prior features are linearly processed and injected into the second scanned feature map, aggregation is performed through point convolution and depthwise separable convolution to generate the restored image

[0012] The present invention also provides an image restoration device based on prior features and linear scanning, including a processor and a memory. Among them, when the processor executes the computer program saved in the memory, the image restoration method based on prior features and linear scanning as described above is implemented

[0013] Beneficial effects The present invention utilizes prior features to guide image restoration, including: injecting the prior features into the image to be restored through linear processing for feature extraction; injecting the prior features into the first feature map through linear processing for linear scanning; injecting the prior features into the second scanned feature map through linear processing for generating high-quality images. Additionally, during the linear scanning process, six scanning methods are adopted for omnidirectional scanning to enhance the perception ability of easily missing regions (such as edges and textures), and two-dimensional global information can be effectively obtained at a low computational cost. Description of the Drawings

[0014] Figure 1 It is a schematic diagram of the processing effect of adopting the image restoration method of the present invention. Detailed Embodiments

[0015] The following embodiments are intended to illustrate the present invention rather than further limit the present invention.

[0016] The present invention provides an image restoration method based on prior features and linear scanning, including: S1: Obtain the image to be restored, compress the image to be restored into the latent feature space to obtain a conditional vector; use the conditional vector for the initial noise randomly sampled from the standard Gaussian distribution, and perform iterative denoising to generate predicted prior features.

[0017] Preferably, compressing the image to be restored into the latent feature space to obtain a conditional vector means that after the image to be restored compresses the features into a global vector of a fixed size through global average pooling, it undergoes two fully connected processes with ReLU activation functions to obtain the conditional vector.

[0018] In actual operation, in order to make the predicted prior features more accurate, before processing the image to be restored, actual prior features are also generated using the low-quality image and the real high-quality image, and a loss function is used for optimization to make the predicted prior features closer to the actual real prior features. Specifically as follows: First, the low-quality image and the real high-quality image undergo a Pixel-Unshuffle operation to rearrange the spatial information to a higher channel dimension, and the rearranged features are subjected to a concatenation operation. Deep hierarchical features are captured through a 3×3 convolutional layer and multiple residual blocks. The hierarchical features gradually increase the feature dimension through a series of convolutional layers, and the features are compressed into a global vector of a fixed size through global average pooling. Through two fully connected layers with ReLU activation functions, actual prior features are generated.

[0019] Then, using the forward diffusion process of the diffusion model, In the subsequent iteration steps, Gaussian noise is gradually added to the actual prior features until they are converted into pure noise.

[0020] Next, to generate more realistic prior features, the low-quality image is compressed into the latent feature space to obtain a conditional vector. During the denoising process, the noise at each step is predicted under the guidance of the conditional vector, and after denoising, the predicted prior features are generated.

[0021] Finally, to make the predicted prior features closer to the actual true prior features, the loss function is used to optimize the model.

[0022] Using the optimized diffusion model, after processing the image to be restored, the predicted prior features can be generated.

[0023] S2: The predicted prior features are linearly processed and injected into the image to be restored. After obtaining the feature map to be restored and performing a pooling operation, the first tensor is obtained; After removing the width dimension of the first tensor, the channel dimension and the height dimension are exchanged to obtain the second tensor. After a convolution operation, the channel dimension and the height dimension are exchanged to obtain the first branch feature map; After the first tensor undergoes a fully connected process to obtain the second branch feature map, it is fused with the first branch feature map to obtain the first feature map.

[0024] Since the image to be restored is a low-quality image, before subsequent processing, it is first preliminarily processed through a 3×3 convolution operation to extract the initial feature map. Since convolution has a local receptive field, this method can enhance the feature expression ability and convert the low-quality image into features suitable for subsequent processing.

[0025] Then, the predicted prior features are linearly processed and injected into the initial feature map or the image to be restored to obtain the feature map to be restored, which is achieved by the formula: , where is the feature map to be restored, is the image to be restored, is the predicted prior feature, ⊙ represents element-wise multiplication, represents layer normalization, , represent linear processing.

[0026] Among them, the feature map to be restored has the shape of [B, C, H, W], where B is the processing batch size, C is the number of channels, H is the height of the feature map, and W is the width of the feature map.

[0027] Then, the feature map to be restored After the pooling operation, a tensor with the shape of [B, C, 1, 1] is obtained, denoted as the first tensor. This step is to compress the spatial information and extract the global features of each channel.

[0028] Next, operations on two branches are performed respectively: One is to remove the dimension with a size of 1 at the last dimension of the first tensor, obtaining a tensor with the shape of [B, C, 1]. Then, the last two dimensions, i.e., the channel dimension and the height dimension, are swapped to obtain a tensor with the shape of [B, 1, C], denoted as the second tensor. Through one-dimensional convolution on the second tensor, the shape of the obtained tensor remains [B, 1, C], and then the dimension is swapped back to [B, C, 1] to obtain the feature map of the first branch .

[0029] One is that the first tensor undergoes a fully connected process to obtain the feature map of the second branch . After the data undergoes the pooling operation, although the global features of each channel are retained, the information between channels is relatively independent. The fully connected process can perform a linear combination of these channel features, enabling the features of different channels to be fused with each other, learning the correlation and dependence between channels, thereby capturing more comprehensive and rich global semantic information and enhancing the feature expression ability.

[0030] Finally, according to the generated weights, dual-branch fusion is performed to obtain the first feature map, enhancing the important feature information in the image to be restored and improving the perception ability of key features.

[0031] Preferably, the first feature map is obtained , which is realized by the formula: . In the formula, is the first feature map, is the feature map to be restored, is the feature map of the first branch, is the feature map of the second branch, represents the sigmoid function, which is used to compress the input data into the interval (0, 1) to generate attention weights; represents a 1×1 convolution; ⊙ represents element-wise multiplication, represents the fusion operation on the results of the dual-branch processing.

[0032] In the present invention, first, global information is obtained through the guidance of prior features in branch one, local information is obtained through branch two, and finally, better fusion of global and local feature information is achieved by using the obtained weights.

[0033] S3: The predicted prior features are linearly processed and injected into the first feature map to obtain the feature map to be scanned. After deep convolutional processing of the feature map to be scanned, a two-way scan of top-left to bottom-right and bottom-left to top-right is performed longitudinally and horizontally, and then processed by Mamba to obtain the first scanned feature map. After multiplying the first scanned feature map by the feature map to be scanned after deep convolutional processing, a front-back direction scan is performed to obtain the second scanned feature map.

[0034] First, according to the formula: , the predicted prior features are linearly processed and injected into the first feature map to obtain the feature map to be scanned .

[0035] Then, the feature map to be scanned after convolutional processing, the obtained is subjected to linear scan processing. The linear scan processing includes six matrix scan methods, where "H forward scan", "W forward scan", and "C forward scan" respectively represent scanning the feature channels from top-left to bottom-right, from bottom-left to top-right, and from front to back on the two-dimensional image plane, and the other three are methods with the opposite scan direction. Specifically as follows: After deep convolutional processing, it serves as the input stream , while serves as the input stream .

[0036] The input stream , performs "H, W two-way scan" longitudinally and horizontally to capture the planar two-dimensional information of the features. Then, after being processed by Mamba, the first scanned feature map is obtained. The Mamba processing includes operations of stacking and reshaping of the Mamba module composed of linear projection, convolutional layer, SiLU activation function, and state space model.

[0037] After multiplying the first scanned feature map by , a pooling operation is performed, which significantly reduces the computational complexity while maintaining almost unchanged performance, and then "C front-back scan" is used to obtain the feature information of the plane.

[0038] Finally, convolution is used to integrate the feature channel information to obtain with the same dimension size as .

[0039] Furthermore, the second scanned feature map is obtained by the formula: , realized; In the formula, is the second scanned feature map, which is the feature map after performing bidirectional scans in the vertical and horizontal directions from top left to bottom right and from bottom left to top right, is the feature map obtained by subjecting the feature map to be scanned to convolution processing, represents a 1×1 convolution operation, represents a pooling operation, ⊙ represents element-wise multiplication, represents a Mamba basic block operation.

[0040] In the process S3, a series of scan operations are performed on the input image to extract multi-scale features. Then, these multi-scale features are passed to Mamba to optimize feature extraction. Based on Mamba's ability to capture long-range information, this linear scan mechanism can effectively obtain two-dimensional global information at a low computational cost.

[0041] S4: The predicted prior features are linearly processed and injected into the second scanned feature map and then aggregated through point convolution and depthwise separable convolution to generate the restored image.

[0042] Regarding the linear processing and injection of the predicted prior features into the second scanned feature map, it is similar to the injection operations in S2 and S3.

[0043] Preferably, the aggregation through point convolution and depthwise separable convolution to generate the restored image is achieved by the formula: , wherein, is the restored image, is the feature map after the predicted prior features are linearly processed and injected into the second scanned feature map , is the second scanned feature map, , is point convolution, , is depthwise separable convolution, is an activation function that uses a gating mechanism to control the output result of this branch.

[0044] In the image restoration stage, convolution is used to aggregate information from different channels, and depthwise separable convolution is used to aggregate information of spatially neighboring pixels. In addition, a gating mechanism is introduced to restore information encoding. By dynamically adjusting the information flow through parameter learning, it suppresses the interference of irrelevant feature information flows, balances the modeling requirements of local and global information, and restores the expressive and generalization capabilities.

[0045] The present invention utilizes prior features to guide image restoration, including: injecting the prior features into the image to be restored through linear processing for feature extraction; injecting the prior features into the first feature map through linear processing for linear scanning; injecting the prior features into the second scanned feature map through linear processing for generating a high-quality image. Additionally, during the linear scanning process, six scanning methods are adopted for omnidirectional scanning to enhance the perception ability of areas prone to missing (such as edges and textures), and two-dimensional global information can be effectively obtained at a low computational cost.

[0046] As Figure 1 shown, the method of the present application has good restoration effects on images blurred by rain and fog, low light, noise, etc.

[0047] The present invention also provides an image restoration system based on prior features and linear scanning, including: Prior feature generation module: used to obtain the image to be restored, compress the image to be restored into the latent feature space to obtain a conditional vector; the initial noise randomly sampled from the standard Gaussian distribution uses the conditional vector for iterative denoising to generate the predicted prior features; Feature fusion module: the predicted prior features are injected into the image to be restored through linear processing, and after obtaining the feature map to be restored and performing a pooling operation, the first tensor is obtained; After removing the width dimension of the first tensor, the channel dimension and the height dimension are exchanged to obtain the second tensor, and after performing a convolution operation, the channel dimension and the height dimension are exchanged to obtain the first branch feature map; After the first tensor undergoes a fully connected process to obtain the second branch feature map, it is fused with the first branch feature map to obtain the first feature map; Linear scanning module: the predicted prior features are injected into the first feature map through linear processing to obtain the feature map to be scanned; the feature map to be scanned after deep convolution processing is scanned bidirectionally in the upper left - lower right and lower left - upper right directions longitudinally and horizontally, and after being processed by Mamba, the first scanned feature map is obtained; the first scanned feature map is multiplied by the feature map to be scanned after deep convolution processing, and then scanned in the front - back direction to obtain the second scanned feature map; Restoration module: the predicted prior features are injected into the second scanned feature map through linear processing, and then aggregated through point convolution and depthwise separable convolution to generate the restored image.

[0048] The present invention also provides an image restoration device based on prior features and linear scanning, including a processor and a memory. Among them, when the processor executes the computer program stored in the memory, it implements the image restoration method based on prior features and linear scanning.

Claims

1. An image restoration method based on prior features and linear scanning, characterized in that, Including: S1: Obtain the image to be restored, compress the image to be restored into the latent feature space to obtain a conditional vector; use the conditional vector for the initial noise randomly sampled from the standard Gaussian distribution to perform iterative denoising to generate the predicted prior feature; S2: Inject the predicted prior feature into the image to be restored through linear processing, and after obtaining the feature map to be restored, perform a pooling operation to obtain the first tensor; After removing the width dimension of the first tensor, swap the channel dimension and the height dimension to obtain the second tensor, and after performing a convolution operation, swap the channel dimension and the height dimension to obtain the first branch feature map; After the first tensor undergoes a fully connected process to obtain the second branch feature map, fuse it with the first branch feature map to obtain the first feature map; S3: Inject the predicted prior feature into the first feature map through linear processing to obtain the feature map to be scanned; for the feature map to be scanned after deep convolution processing, perform two-way scans in the upper left - lower right and lower left - upper right directions longitudinally and horizontally, and then obtain the first scanned feature map after Mamba processing; multiply the first scanned feature map by the feature map to be scanned after deep convolution processing and then perform a front - back direction scan to obtain the second scanned feature map; S4: Inject the predicted prior feature into the second scanned feature map through linear processing, and then aggregate it through point convolution and depth - separable convolution to generate the restored image.

2. The image restoration method based on prior features and linear scanning according to claim 1, wherein In the step S2, the predicted prior features are linearly processed and injected into the image to be restored, and the feature map to be restored is obtained by the formula: which is implemented; Wherein, is the feature map to be restored, is the image to be restored, is the predicted prior feature, ⊙ represents element-wise multiplication, represents layer normalization, and represent linear processing.

3. The image restoration method based on prior features and linear scanning according to claim 1, characterized in that The S2 obtains a first feature map, which is implemented by the formula: , implemented; Wherein, is the first feature map, is the feature map to be restored, is the first branch feature map, is the second branch feature map, represents the sigmoid function, which is used to compress the input data into the interval (0, 1) to generate the attention weight; represents a 1×1 convolution; ⊙ represents element-wise multiplication, represents the fusion operation.

4. The image restoration method based on prior features and linear scanning according to claim 1, wherein The S3 obtains a second scanned feature map, which is implemented by the formula: , and is achieved. In the formula, is the second scanned feature map, is the feature map after performing two-way scans of upper-left to lower-right and lower-left to upper-right in the vertical and horizontal directions, is the feature map obtained by performing convolution processing on the feature map to be scanned, represents a 1×1 convolution operation, represents a pooling operation, ⊙ represents element-wise multiplication, represents a Mamba basic block operation.

5. The image restoration method based on prior features and linear scanning according to claim 1, characterized in that The above-mentioned S4 is aggregated through point convolution and depthwise separable convolution to generate a restored image, which is realized by the formula: , achieved; Wherein, is the restored image, is the feature map after the predicted prior feature is linearly processed and injected into the second scanned feature map, is the second scanned feature map, and are point convolutions, and are depthwise separable convolutions, is the activation function.

6. The image restoration method based on prior features and linear scanning according to claim 1, wherein After aggregating through point convolution and depth - separable convolution in S4, it further includes: The feature map after aggregating through point convolution and depth - separable convolution is used as the image to be restored, and execute S2 until a preset condition is reached.

7. The image restoration method based on prior features and linear scanning according to claim 1, characterized in that In S1, when compressing the image to be restored into the latent feature space to obtain a conditional vector, the feature of the image to be restored is compressed into a global vector of a fixed size through global average pooling, and then after two fully connected processes with ReLU activation functions, the conditional vector is obtained.

8. An image restoration system based on prior features and linear scanning, characterized in that, Including: Prior feature generation module: used to obtain the image to be restored, compress the image to be restored into the latent feature space to obtain a conditional vector; use the conditional vector for the initial noise randomly sampled from the standard Gaussian distribution to perform iterative denoising to generate the predicted prior feature; Feature fusion module: Inject the predicted prior feature into the image to be restored through linear processing, and after obtaining the feature map to be restored, perform a pooling operation to obtain the first tensor; After removing the width dimension of the first tensor, swap the channel dimension and the height dimension to obtain the second tensor, and after performing a convolution operation, swap the channel dimension and the height dimension to obtain the first branch feature map; After the first tensor undergoes a fully connected process to obtain the second branch feature map, fuse it with the first branch feature map to obtain the first feature map; Linear scanning module: Inject the predicted prior feature into the first feature map through linear processing to obtain the feature map to be scanned; for the feature map to be scanned after deep convolution processing, perform two-way scans in the upper left - lower right and lower left - upper right directions longitudinally and horizontally, and then obtain the first scanned feature map after Mamba processing; multiply the first scanned feature map by the feature map to be scanned after deep convolution processing and then perform a front - back direction scan to obtain the second scanned feature map; Recovery module: After the predicted prior features are linearly processed and injected into the second scanned feature map, they are aggregated through point convolution and depthwise separable convolution to generate a restored image.

9. An image restoration device based on prior features and linear scanning, characterized in that, It includes a processor and a memory. Among them, when the processor executes the computer program stored in the memory, it implements the image restoration method based on prior features and linear scanning according to any one of claims 1-7.

Images