Image Restoration Method, System, and Device Based on Prior Features and Linear Scanning

Abstract

The present invention belongs to the technical field of image restoration, and specifically relates to an image restoration method, system, and device based on prior features and linear scanning, which use prior features to guide image restoration, including: injecting the prior features into the image to be restored through linear processing for feature extraction; injecting the prior features into the first feature map through linear processing for linear scanning; injecting the prior features into the second scanned feature map through linear processing for generating a high-quality image. Additionally, during the linear scanning process, six scanning methods are adopted for omnidirectional scanning to enhance the perception ability of areas prone to missing (such as edges and textures), and two-dimensional global information can be effectively obtained at a low computational cost.

Description

Technical Field

[0001] The present invention belongs to the technical field of image restoration, and specifically relates to an image restoration method, system, and device based on prior features and linear scanning. Background Art

[0002] Deep learning-based methods (such as convolutional neural network CNN and Transformer) have made significant progress in the field of image restoration, capable of capturing deep features of images and restoring details. However, these methods still have deficiencies in dealing with long-range dependencies, real-time performance, and multi-task adaptability. Mamba combines the state space model (SSM) with advanced deep learning techniques and uses selective state representation to dynamically adjust according to input data. Although the Mamba model performs well in capturing long-range information, it may lose local structural details when dealing with edge information of two-dimensional images, resulting in blurred local semantic information. Summary of the Invention

[0003] The present invention provides an image restoration method, system, and device based on prior features and linear scanning.

[0004] The technical solution of the present invention is as follows:

[0005] The present invention provides an image restoration method based on prior features and linear scanning, including:

[0006] S1: Obtain the image to be restored, compress the image to be restored into the latent feature space to obtain a conditional vector; use the conditional vector for the initial noise randomly sampled from the standard Gaussian distribution for iterative denoising to generate the predicted prior features;

[0007] S2: Inject the predicted prior features into the image to be restored through linear processing, obtain the feature map to be restored, and then obtain the first tensor through a pooling operation;

[0008] After removing the width dimension of the first tensor, exchange the channel dimension and the height dimension to obtain the second tensor, and after performing a convolution operation, exchange the channel dimension and the height dimension to obtain the first branch feature map;

[0009] After the first tensor undergoes a fully connected process to obtain the second branch feature map, fuse it with the first branch feature map to obtain the first feature map;

[0010] S3: Inject the predicted prior features into the first feature map through linear processing to obtain the feature map to be scanned; after deep convolution processing of the feature map to be scanned, perform two-way scans in the upper left - lower right and lower left - upper right directions longitudinally and horizontally, and then obtain the first scanned feature map through Mamba processing; multiply the first scanned feature map by the feature map to be scanned after deep convolution processing, and then perform a front - back direction scan to obtain the second scanned feature map;

[0011] S4: After the predicted prior features are linearly processed and injected into the second scanned feature map, they are aggregated through point convolution and depthwise separable convolution to generate a restored image.

[0012] In the above S2, the predicted prior features are linearly processed and injected into the image to be restored, and a feature map to be restored is obtained, which is realized by the formula: , realized;

[0013] In the formula, is the feature map to be restored, is the image to be restored, is the predicted prior feature, ⊙ represents element-wise multiplication, represents layer normalization, , represents linear processing.

[0014] In the above S2, the first feature map is obtained, which is realized by the formula: , realized;

[0015] In the formula, is the first feature map, is the feature map to be restored, is the first branch feature map, is the second branch feature map, represents the sigmoid function, which is used to compress the input data into the interval (0,1) to generate attention weights; represents a 1×1 convolution; ⊙ represents element-wise multiplication, represents a fusion operation.

[0016] In the above S3, the second scanned feature map is obtained, which is realized by the formula: , realized;

[0017] In the formula, is the second scanned feature map, is the feature map after performing bidirectional scans of upper left - lower right and lower left - upper right in the vertical and horizontal directions, is the feature map obtained after the feature map to be scanned is convolutionally processed, represents a 1×1 convolution operation, represents a pooling operation, ⊙ represents element-wise multiplication, represents the Mamba basic block operation.

[0018] In the above S4, aggregation is performed through point convolution and depthwise separable convolution to generate a restored image, which is realized by the formula: , realized;

[0019] In the formula, To restore the image, is the feature map after the predicted prior feature is linearly processed and injected into the second scanned feature map, is the second scanned feature map, 、 is point convolution, 、 is depthwise separable convolution, is the activation function.

[0020] After the aggregation of the S4 through point convolution and depthwise separable convolution, it further includes:

[0021] The feature map after aggregation through point convolution and depthwise separable convolution is used as the image to be restored, and S2 is executed until a preset condition is reached.

[0022] In the S1, the image to be restored is compressed into the latent feature space to obtain a conditional vector. After the features of the image to be restored are compressed into a fixed-size global vector through global average pooling, two fully connected processes with ReLU activation functions are performed to obtain the conditional vector.

[0023] The present invention also provides an image restoration system based on prior features and linear scanning, including:

[0024] Prior feature generation module: used to obtain the image to be restored, compress the image to be restored into the latent feature space to obtain a conditional vector; use the conditional vector for the initial noise randomly sampled from the standard Gaussian distribution to perform iterative denoising to generate the predicted prior feature;

[0025] Feature fusion module: After the predicted prior feature is linearly processed and injected into the image to be restored, the to-be-restored feature map is obtained, and after pooling operation, the first tensor is obtained;

[0026] After removing the width dimension of the first tensor, the channel dimension and the height dimension are exchanged to obtain the second tensor. After convolution operation, the channel dimension and the height dimension are exchanged to obtain the first branch feature map;

[0027] After the first tensor is fully connected to obtain the second branch feature map, it is fused with the first branch feature map to obtain the first feature map;

[0028] Linear scanning module: The predicted prior feature is linearly processed and injected into the first feature map to obtain the to-be-scanned feature map; the to-be-scanned feature map after deep convolution processing is scanned bidirectionally in the upper left - lower right and lower left - upper right directions longitudinally and horizontally, and after Mamba processing, the first scanned feature map is obtained; the first scanned feature map is multiplied by the to-be-scanned feature map after deep convolution processing, and then scanned in the front - back direction to obtain the second scanned feature map;

[0029] Recovery module: After the predicted prior features are linearly processed and injected into the second scanned feature map, they are aggregated through pointwise convolution and depthwise separable convolution to generate a restored image.

[0030] The present invention also provides an image restoration device based on prior features and linear scanning, including a processor and a memory. When the processor executes the computer program stored in the memory, the image restoration method based on prior features and linear scanning is implemented.

[0031] Beneficial effects

[0032] The present invention uses prior features to guide image restoration, including: injecting the prior features into the image to be restored through linear processing for feature extraction; injecting the prior features into the first feature map through linear processing for linear scanning; injecting the prior features into the second scanned feature map through linear processing for generating a high-quality image. In addition, during the linear scanning process, six scanning methods are adopted for omnidirectional scanning to enhance the perception ability of easily missing areas (such as edges and textures), and two-dimensional global information can be effectively obtained at a low computational cost. Description of the drawings

[0033] Figure 1 It is a schematic diagram of the processing effect of adopting the image restoration method of the present invention. Specific embodiments

[0034] The following embodiments are intended to illustrate the present invention rather than further limit the present invention.

[0035] The present invention provides an image restoration method based on prior features and linear scanning, including:

[0036] S1: Obtain the image to be restored, compress the image to be restored into the latent feature space to obtain a conditional vector; use the conditional vector for the initial noise randomly sampled from the standard Gaussian distribution for iterative denoising to generate predicted prior features.

[0037] Preferably, compressing the image to be restored into the latent feature space to obtain a conditional vector means that after the image to be restored compresses the features into a global vector of a fixed size through global average pooling, it is processed by two fully connected layers with ReLU activation functions to obtain the conditional vector.

[0038] In actual operation, in order to make the predicted prior features more accurate, before processing the image to be restored, actual prior features are also generated using the low-quality image and the real high-quality image, and the loss function is used for optimization so that the predicted prior features are closer to the actual real prior features. Specifically as follows:

[0039] First, the low-quality image and the real high-quality image undergo the Pixel-Unshuffle operation to rearrange the spatial information into a higher channel dimension. These rearranged features are concatenated, and a deep hierarchical feature is captured through a 3×3 convolutional layer and multiple residual blocks. The hierarchical feature gradually increases the feature dimension through a series of convolutional layers and is compressed into a fixed-size global vector through global average pooling. The actual prior feature is generated through two fully connected layers with ReLU activation functions.

[0040] Then, using the forward diffusion process of the diffusion model, Gaussian noise is gradually added to the actual prior feature in iterative steps until it is converted into pure noise.

[0041] Next, to generate a more realistic prior feature, the low-quality image is compressed into the latent feature space to obtain a conditional vector. During the denoising process, the noise at each step is predicted under the guidance of the conditional vector, and after denoising, the predicted prior feature is generated.

[0042] Finally, to make the predicted prior feature closer to the actual real prior feature, the loss function is used to optimize the model.

[0043] Using the optimized diffusion model, after processing the image to be restored, the predicted prior feature can be generated.

[0044] S2: The predicted prior feature is linearly processed and injected into the image to be restored. After obtaining the feature map to be restored and performing the pooling operation, the first tensor is obtained;

[0045] After removing the width dimension of the first tensor, the channel dimension and the height dimension are exchanged to obtain the second tensor. After convolutional operation, the channel dimension and the height dimension are exchanged to obtain the first branch feature map;

[0046] After the first tensor undergoes fully connected processing to obtain the second branch feature map, it is fused with the first branch feature map to obtain the first feature map.

[0047] Since the image to be restored is a low-quality image, before subsequent processing, it is first preliminarily processed through a 3×3 convolutional operation to extract the initial feature map. Since convolution has a local receptive field, this method can improve the feature expression ability and convert the low-quality image into features suitable for subsequent processing.

[0048] Then, the predicted prior feature is linearly processed and injected into the initial feature map or the image to be restored to obtain the feature map to be restored, which is realized by the formula: , realized;

[0049] In the formula, is the feature map to be restored, is the image to be restored, is the predicted prior feature, and ⊙ represents element-wise multiplication, represents layer normalization, and represent linear processing.

[0050] Among them, the feature map to be restored has a shape of [B, C, H, W], where B is the processing batch size, C is the number of channels, H is the height of the feature map, and W is the width of the feature map.

[0051] Then, after the feature map to be restored undergoes a pooling operation, a tensor with a shape of [B, C, 1, 1] is obtained, denoted as the first tensor. This step is to compress spatial information and extract the global features of each channel.

[0052] Next, operations on two branches are performed respectively:

[0053] One is to remove the dimension with a size of 1 at the last dimension of the first tensor to obtain a tensor with a shape of [B, C, 1], then swap the last two dimensions, that is, the channel dimension and the height dimension, to obtain a tensor with a shape of [B, 1, C], denoted as the second tensor. Through one-dimensional convolution on the second tensor, the shape of the obtained tensor is still [B, 1, C], and then the dimension is swapped back to [B, C, 1] to obtain the first branch feature map .

[0054] One is that the first tensor undergoes a fully connected process to obtain the second branch feature map . After the data undergoes the pooling operation, although the global features of each channel are retained, the information between channels is relatively independent. The fully connected process can perform a linear combination of these channel features, enabling the features of different channels to be fused with each other, learning the correlation and dependence between channels, thereby capturing more comprehensive and richer global semantic information and enhancing the feature expression ability.

[0055] Finally, according to the generated weights, double-branch fusion is performed to obtain the first feature map, enhancing the important feature information in the image to be restored and improving the perception ability of key features.

[0056] Preferably, the first feature map is obtained by the formula: , implemented;

[0057] In the formula, is the first feature map, is the feature map to be restored, is the feature map of the first branch, is the feature map of the second branch, represents the sigmoid function, which is used to compress the input data into the interval (0, 1) to generate attention weights; represents a 1×1 convolution; ⊙ represents element-wise multiplication, represents performing a fusion operation on the results of the dual-branch processing.

[0058] In the present invention, first, global information is obtained through the guidance of the prior features by the first branch, local information is obtained through the second branch, and finally, the obtained weights are used to better fuse the global and local feature information.

[0059] S3: The predicted prior features are linearly processed and injected into the first feature map to obtain the feature map to be scanned; the feature map to be scanned after deep convolutional processing is subjected to bidirectional scanning in the vertical and horizontal directions, i.e., from top-left to bottom-right and from bottom-left to top-right, and then processed by Mamba to obtain the first scanned feature map; after the first scanned feature map is multiplied by the feature map to be scanned after deep convolutional processing, scanning is performed in the front-back direction to obtain the second scanned feature map.

[0060] First, according to the formula: , the predicted prior features are linearly processed and injected into the first feature map to obtain the feature map to be scanned .

[0061] Then, the feature map to be scanned after passing through convolutional processing, the obtained is subjected to linear scanning processing. The linear scanning processing includes six matrix scanning methods, where "H forward scan", "W forward scan", and "C forward scan" respectively represent scanning the feature channels from top-left to bottom-right, from bottom-left to top-right, and from front to back on the two-dimensional image plane, and the other three are methods with the opposite scanning directions. Specifically as follows:

[0062] After deep convolutional processing, it serves as the input stream , while serves as the input stream .

[0063] The input stream , performs "H, W bidirectional scanning" in the vertical and horizontal directions to capture the two-dimensional planar information of the features. Then, after being processed by Mamba, the first scanned feature map is obtained. The Mamba processing includes operations of stacking and reshaping of the Mamba module composed of linear projection, convolutional layer, SiLU activation function, and state space model.

[0064] After the first scanned feature map is multiplied and then a pooling operation is performed, the computational complexity is significantly reduced while maintaining almost unchanged performance. Then, "C forward and backward scanning" is used to obtain the feature information of the plane.

[0065] Finally, convolution is used to integrate the feature channel information to obtain with the same dimensional size as .

[0066] Furthermore, the second scanned feature map is obtained by the formula: , which is realized;

[0067] In the formula, is the second scanned feature map, is the feature map after performing bidirectional scanning of top - left to bottom - right and bottom - left to top - right longitudinally and horizontally, is the feature map obtained after the feature map to be scanned is processed by convolution, represents a 1×1 convolution operation, represents a pooling operation, ⊙ represents element - wise multiplication, represents the Mamba basic block operation.

[0068] In the S3 process, a series of scanning operations are performed on the input image to extract multi - scale features. Then, these multi - scale features are passed to Mamba to optimize feature extraction. Based on Mamba's capture of long - range information, this linear scanning mechanism can effectively obtain two - dimensional global information at a low - complexity computational cost.

[0069] S4: After the predicted prior features are linearly processed and injected into the second scanned feature map , aggregation is performed through point convolution and depth - separable convolution to generate the restored image.

[0070] Regarding the linear processing and injection of the predicted prior features into the second scanned feature map, it is similar to the injection operations in S2 and S3.

[0071] Preferably, aggregation is performed through point convolution and depth - separable convolution to generate the restored image, which is realized by the formula: , which is realized;

[0072] In the formula, is the restored image, is the feature map after the predicted prior features are linearly processed and injected into the second scanned feature map , is the second scanned feature map, , is point convolution, , is depthwise separable convolution, is the activation function, which uses a gating mechanism to control the output result of this branch.

[0073] In the image restoration stage, convolution is used to aggregate information of different channels, and depthwise separable convolution is used to aggregate information of spatial neighborhood pixels. In addition, a gating mechanism is introduced to restore information encoding. By dynamically adjusting the information flow through parameter learning, the interference of incoherent feature information flow is suppressed, the modeling requirements of local and global information are balanced, and the expressive ability and generalization ability are restored.

[0074] The present invention uses prior features to guide image restoration, including: injecting the prior features into the image to be restored through linear processing for feature extraction; injecting the prior features into the first feature map through linear processing for linear scanning; injecting the prior features into the second scanned feature map through linear processing for generating a high-quality image. In addition, during the linear scanning process, six scanning methods are adopted for omnidirectional scanning to enhance the perception ability of easily missing regions (such as edges and textures), and two-dimensional global information is effectively obtained at a low computational cost.

[0075] As Figure 1 shown, the method of the present application has good restoration effects on processing blurred images such as rain and fog, low light, and noise.

[0076] The present invention also provides an image restoration system based on prior features and linear scanning, including:

[0077] Prior feature generation module: used to obtain the image to be restored, compress the image to be restored into the latent feature space to obtain a conditional vector; the initial noise randomly sampled from the standard Gaussian distribution uses the conditional vector for iterative denoising to generate the predicted prior features;

[0078] Feature fusion module: after the predicted prior features are injected into the image to be restored through linear processing to obtain the to-be-restored feature map, and then through a pooling operation, a first tensor is obtained;

[0079] After removing the width dimension of the first tensor, the channel dimension and the height dimension are exchanged to obtain a second tensor. After a convolution operation, the channel dimension and the height dimension are exchanged to obtain a first branch feature map;

[0080] After the first tensor undergoes a fully connected process to obtain a second branch feature map, it is fused with the first branch feature map to obtain a first feature map;

[0081] Linear scanning module: The predicted prior features are linearly processed and injected into the first feature map to obtain the feature map to be scanned; the feature map to be scanned after deep convolutional processing is subjected to two-way scanning in the upper-left to lower-right and lower-left to upper-right directions longitudinally and horizontally, and then processed by Mamba to obtain the first scanned feature map; after the first scanned feature map is multiplied by the feature map to be scanned after deep convolutional processing, a front-back direction scan is performed to obtain the second scanned feature map.

[0082] Recovery module: After the predicted prior features are linearly processed and injected into the second scanned feature map, aggregation is performed through point convolution and depthwise separable convolution to generate a recovered image.

[0083] The present invention also provides an image recovery device based on prior features and linear scanning, including a processor and a memory. Among them, when the processor executes the computer program stored in the memory, the image recovery method based on prior features and linear scanning as described above is implemented.

Claims

1. An image restoration method based on prior features and linear scanning, characterized in that, Including: S1: Obtain the image to be restored, compress the image to be restored into the latent feature space to obtain a conditional vector; use the conditional vector for the initial noise randomly sampled from the standard Gaussian distribution to perform iterative denoising to generate the predicted prior feature; S2: Inject the predicted prior feature into the image to be restored through linear processing, obtain the feature map to be restored, and then obtain the first tensor through pooling operation; After removing the width dimension of the first tensor, swap the channel dimension and the height dimension to obtain the second tensor, and after performing a convolution operation, swap the channel dimension and the height dimension to obtain the first branch feature map; After the first tensor undergoes fully connected processing to obtain the second branch feature map, fuse it with the first branch feature map to obtain the first feature map; S3: Inject the predicted prior feature into the first feature map through linear processing to obtain the feature map to be scanned; for the feature map to be scanned after deep convolution processing, perform two-way scans in the vertical and horizontal directions of top-left to bottom-right and bottom-left to top-right, and then obtain the first scanned feature map through Mamba processing; multiply the first scanned feature map by the feature map to be scanned after deep convolution processing and then perform a scan in the front-back direction to obtain the second scanned feature map; S4: Inject the predicted prior feature into the second scanned feature map through linear processing, and then aggregate it through point convolution and depthwise separable convolution to generate the restored image.

2. The image restoration method based on prior features and linear scanning according to claim 1, wherein In S2, the predicted prior features are linearly processed and injected into the image to be restored, obtaining the feature map to be restored, which is implemented by the formula: , to achieve; Wherein, is the feature map to be restored, is the image to be restored, is the predicted prior feature, ⊙ represents element-wise multiplication, represents layer normalization, 、 represent linear processing.

3. The image restoration method based on prior features and linear scanning according to claim 1, characterized in that The S2 obtains a first feature map, which is implemented by the formula: , where the implementation is as follows; In the formula, is the first feature map, is the feature map to be restored, is the first branch feature map, is the second branch feature map, represents the sigmoid function, which is used to compress the input data into the interval (0, 1) to generate attention weights; represents a 1×1 convolution; ⊙ represents element-wise multiplication, represents a fusion operation.

4. The image restoration method based on prior features and linear scanning according to claim 1, characterized in that The step S3 obtains a second scanned feature map, which is implemented by the formula: , where ";" is used to separate statements In the formula, is the second scanned feature map, is the feature map after performing two-way scanning of upper left - lower right and lower left - upper right in the vertical and horizontal directions, is the feature map obtained by performing convolution processing on the feature map to be scanned, represents a 1×1 convolution operation, represents a pooling operation, ⊙ represents element-wise multiplication, represents the Mamba basic block operation.

5. The image restoration method based on prior features and linear scanning according to claim 1, wherein The S4 aggregates through point convolution and depthwise separable convolution to generate a restored image, which is implemented by the formula: , achieved; Wherein, is the restored image, is the feature map after the predicted prior feature is linearly processed and injected into the second scanned feature map, is the second scanned feature map, 、 are point convolutions, 、 are depthwise separable convolutions, is the activation function.

6. The image restoration method based on prior features and linear scanning according to claim 1, wherein After the aggregation in S4 through point convolution and depthwise separable convolution, it further includes: The feature map after aggregation through point convolution and depthwise separable convolution is used as the image to be restored, and S2 is executed until a preset condition is reached.

7. The image restoration method based on prior features and linear scanning according to claim 1, wherein In S1, when compressing the image to be restored into the latent feature space to obtain a conditional vector, the feature of the image to be restored is compressed into a global vector of a fixed size through global average pooling, and then the conditional vector is obtained through two fully connected processes with ReLU activation functions.

8. An image restoration system based on prior features and linear scanning, characterized in that, Including: Prior feature generation module: used to obtain the image to be restored, compress the image to be restored into the latent feature space to obtain a conditional vector; use the conditional vector for the initial noise randomly sampled from the standard Gaussian distribution to perform iterative denoising to generate the predicted prior feature; Feature fusion module: Inject the predicted prior feature into the image to be restored through linear processing, obtain the feature map to be restored, and then obtain the first tensor through pooling operation; After removing the width dimension of the first tensor, swap the channel dimension and the height dimension to obtain the second tensor, and after performing a convolution operation, swap the channel dimension and the height dimension to obtain the first branch feature map; After the first tensor undergoes fully connected processing to obtain the second branch feature map, fuse it with the first branch feature map to obtain the first feature map; Linear scanning module: Inject the predicted prior feature into the first feature map through linear processing to obtain the feature map to be scanned; for the feature map to be scanned after deep convolution processing, perform two-way scans in the vertical and horizontal directions of top-left to bottom-right and bottom-left to top-right, and then obtain the first scanned feature map through Mamba processing; multiply the first scanned feature map by the feature map to be scanned after deep convolution processing and then perform a scan in the front-back direction to obtain the second scanned feature map; Recovery module: After the predicted prior features are linearly processed and injected into the second scanned feature map, they are aggregated through point convolution and depthwise separable convolution to generate a restored image.

9. An image restoration device based on prior features and linear scanning, characterized in that, It includes a processor and a memory. Among them, when the processor executes the computer program stored in the memory, it implements the image recovery method based on prior features and linear scanning as described in any one of claims 1-7.

Images