A multi-weather degraded image restoration method based on a state space model

Abstract

The application discloses a multi-weather degraded image recovery method based on a state space model. The method comprises the following steps: acquiring multi-scene clear images and synthesizing images under rain, snow and fog weather conditions by using an image synthesis technology to construct a standard multi-weather dataset; dividing the dataset after preprocessing; constructing a multi-weather degraded image recovery model based on a state space model, and training and optimizing the model; acquiring images degraded due to weather influence and preprocessing the images; inputting the preprocessed images into the pre-trained multi-weather degraded image recovery model for recovery, and outputting the processed clear images. The application is based on a state space model, introduces an adaptive channel attention mechanism, enhances the network's ability to capture local features of images, and can effectively process images of various scales through overlapping block embedding and multi-scale processing strategies, thereby significantly improving the performance of image recovery. The method of the application can process image recovery tasks of various weather degradations including but not limited to rain, snow and fog, and has a wide application prospect.

Description

Technical Field

[0001] The present invention belongs to the technical field of image restoration, and in particular relates to a multi-weather degradation image restoration method based on a state space model. Background Art

[0002] With the rapid development of computer vision technology, it has been widely applied in various fields. However, outdoor vision systems are often affected by various weather conditions, including rain, snow, and fog. These conditions can severely impact image quality, thus affecting the reliability of models used in advanced vision tasks and misleading machine systems, such as autonomous driving. Therefore, efficient image restoration through algorithms is a key issue in the field of computer vision.

[0003] In recent years, with the introduction of deep learning models such as convolutional neural networks and Transformers, the performance of image restoration technology has significantly improved. To make outdoor vision systems more resilient to adverse weather conditions, numerous restoration methods have been proposed, such as image restoration targeting specific weather degradation. These methods perform well in specific weather conditions, but outdoor vision systems often face a wide range of weather conditions. These methods require different model parameters or prediction training weights to handle various weather conditions, which incurs additional storage and computational overhead.

[0004] Recently, methods proposed in the literature have used a single network parameter to remove multiple weather degradations. However, these methods have limitations in practical application and deployment. Existing restoration networks are mostly based on CNNs and transformers. CNN-based methods are computationally efficient and can be deployed in resource-constrained environments, but they have a limited receptive field. Transformer-based restoration methods have a larger receptive field and outperform CNNs, but at a higher computational cost. Existing methods have proposed some effective attention mechanisms to alleviate computational issues, but this comes at the expense of global attention, and the balance between global receptive field and computational efficiency remains difficult. Summary of the Invention

[0005] In response to the defects in the existing technology, the present invention proposes a multi-weather degraded image restoration method based on a state-space model. This method is a lightweight and efficient end-to-end model for restoring images degraded in various weather conditions, including rain, snow, and fog. It integrates the local feature extraction of convolution and the ability of the state-space model to capture long-range dependencies. It can maintain high computational efficiency while having a global receptive field, overcomes the limitations of the state-space model in the field of image restoration, and has high performance when processing degraded images of different sizes and various weather conditions.

[0006] The technical solution adopted is: a multi-weather degraded image restoration method based on a state space model, including the following steps:

[0007] Step S1: Acquire multiple weather clear images, process them into degraded images of corresponding weather, and establish a multi-weather standard dataset, wherein the dataset stores {degraded image, clear image} pairs; pre-process the dataset and divide it into proportions;

[0008] Step S2: constructing a multi-weather degraded image restoration network based on a state-space model, wherein the restoration network includes: a shallow feature extraction stage, a deep feature extraction stage, and a high-quality image reconstruction stage;

[0009] In the shallow feature extraction stage, the convolution layer is used to extract the shallow features F of the degraded image I. s The deep feature extraction stage includes an upsampling layer, a downsampling layer, n residual visual state space groups, and a convolution layer at the end; the shallow feature F s Processing to obtain deep features F d The high-quality image reconstruction stage is based on shallow features F s and deep features F d After fusion, it is the input, and after processing, a clear image I is obtained. c ;

[0010] Step S3: Based on the multi-weather standard dataset, iteratively train the multi-weather degraded image restoration network to obtain a trained ideal restoration model;

[0011] Step S4: collecting the weather-degraded image to be restored, pre-processing it, and inputting it into the trained ideal restoration model for multi-weather degraded images for restoration, so as to obtain a clear image of the corresponding weather.

[0012] The acquiring of multi-weather clear images comprises: collecting a series of high-quality images of outdoor scenes, wherein the images cover a variety of outdoor scenes and lighting conditions;

[0013] The image processing is to use an image synthesis method to adjust parameters to control the authenticity and degradation degree of different weather effects, and add virtual weather effects to the collected images.

[0014] The parameters include the size and density of raindrops, the shape and falling speed of snowflakes, and the concentration and coverage of fog, which are used to realistically render various weather phenomena in the image.

[0015] The preprocessing of the dataset includes: randomly grouping all images, randomly cropping, flipping, and manipulating the images in each group, performing data amplification to obtain the preprocessed dataset; and dividing it into training set, validation set, and test set in proportion.

[0016] The shallow feature extraction stage converts the channel dimension of the input image from 3 to a new embedding space 48 through a 3*3 convolution kernel, and keeps the spatial dimension of the image unchanged; then, the output feature map is flattened in the spatial dimension; its output is the shallow feature F s， , the formula is:

[0017] F s =Reshape(Conv2D(I))

[0018] The deep feature extraction stage includes the following steps:

[0019] Step a1: transform the shallow feature F s After the first residual visual state space group RVSSG, the feature F1 is obtained;

[0020] Step a2: Through downsampling, the F1 channel dimension is doubled and the height and width are reduced to 1 / 2 of the original;

[0021] Step a3: Pass the downsampled features obtained in step a2 through the second residual visual state space group RVSSG to obtain feature F2;

[0022] Step a4: Repeat steps a2 and a3 to perform multi-level downsampling and feature extraction. After each downsampling, the channel dimension is doubled and the height and width are halved. At the same time, deep features are extracted through the residual visual state space group RVSSG until the preset depth level is reached, and features F3 and F4 are obtained in sequence.

[0023] Step a5: After reaching the deepest level, perform upsampling to extract feature G i , while collecting the feature map F of the same resolution in the downsampling branch i , feature fusion is performed through the residual visual state space group RVSSG, gradually reconstructing the height and width of the image while reducing the channel dimension until it is reconstructed to the same spatial resolution as the input image, and the final deep feature representation F is obtained d ; The formula is:

[0024] F1=R1(F s )

[0025] F i =R i (D i-1 (F i-1 )) (i=2,3,4)

[0026] F j =R j (U j-4 (F j-1)) (j=5,6,7)

[0027] F d =Conv2D(F j )

[0028] Among them, R i Represents the first residual visual state space group, F i Denotes the obtained i-th feature map, D i-1 represents the i-1th downsampling layer, U j-4 represents the j-4th upsampling layer, and Conv2D represents the convolution operation.

[0029] The specific operations of the high-quality image reconstruction stage are:

[0030] The shallow feature F s and deep features F d After addition, it is used as the input of the high-quality image reconstruction stage, and the feature is reconstructed by the residual visual state space group RVSSG. The reconstructed features are mapped to the target output space through the convolution layer to obtain the final clear image restoration result I c , its formula is expressed as:

[0031] I c =Conv2D(R f (F s +F d ))

[0032] Among them, R f It is represented as the last residual visual state space group, and Conv2D represents the two-dimensional convolution operation.

[0033] The residual visual state space group RVSSG consists of several residual visual state space blocks RVSSB connected in series and a convolutional layer to refine the features extracted from RVSSB. The specific operation of RVSSB is:

[0034] First input feature F d i Processing is performed through a normalization layer LayerNorm to reduce internal covariate shift, stabilize the training process, and speed up convergence;

[0035] Then, the visual state space module (VSSM) is used to capture long-term spatial dependencies to enhance the correlation between features.

[0036] Then, the information from the skip connection is controlled by using a learnable scaling factor s1;

[0037] Then, the second layer normalization and local enhancement feedforward neural network LEFF is used to enhance local features and maintain the dimension of feature space while reducing the number of parameters and calculations;

[0038] Finally, another adjustable scaling factor s2 is used in the residual connection to obtain the final output of RVSSB; its formula is expressed as:

[0039]

[0040] Among them, F d i Represents the deep features extracted by the i-th RVSSM, s1, s2 represent adjustable scale factors, Z i Represents the intermediate features after processing by the i-th VSSM.

[0041] VSSM uses state-space equations to capture long-range dependencies and adaptively weights channel attention to achieve importance-weighted feature extraction, extracting important features and suppressing irrelevant information, thereby reducing channel redundancy caused by an excessive number of hidden states. The specific operations of VSSM are:

[0042] The input feature X passes through two parallel branches. In the first branch, the feature channel is expanded to λC through a linear layer, where λ is a predefined channel expansion factor, followed by a depthwise convolution DWConv, a SiLU activation function, a two-dimensional adaptive channel state space module 2D-ACSSM, and a normalization layer LayerNorm. In the second branch, the feature channel is also expanded to λC through a linear layer and a SiLU activation function. Then, the features of the two branches are aggregated using the Hadamard product operation. Finally, the channel number is projected back to C to generate an output X with the same shape as the input. VSSM :

[0043] X1=LayerNorm(2DACSSM(SILU(DWConv(Linear(x))))))

[0044] X2=SILU(Linear(x))

[0045] X VSSM =X1⊙X2

[0046] Among them, DWConv represents depthwise convolution, ⊙ represents Hadamard product, X1 represents the output result of the first branch, and X2 represents the output result of the second branch.

[0047] 2D-ACSSM includes:

[0048] a. Flatten the two-dimensional image features into one-dimensional sequence image features;

[0049] b. Each input image feature is scanned along different directions by four different selective scanning branches SSMBLock to capture the long-range dependencies of each sequence according to the discrete state space equation;

[0050] c. Introducing an adaptive channel attention module ACAtn in each branch to adaptively weight the features of each channel in each branch and determine the channel weight by aggregating and linearly transforming the global features;

[0051] d. Use the summation method to combine all sequences and perform a reshaping operation to restore the 2D image structure.

[0052] The present invention has the following beneficial effects and advantages:

[0053] 1. This paper introduces the state-space model into the field of image restoration and proposes a method that effectively solves the multi-weather degradation problem with a single model. It has a global receptive field and high computational efficiency, and can effectively replace traditional methods based on convolutional neural networks and Transformers to achieve more efficient restoration effects.

[0054] 2. The two-dimensional adaptive channel state-space module proposed in the present invention processes each input image feature through four selective scanning branches in different directions, solving the problem that the state-space model will destroy the natural local two-dimensional dependency when processing two-dimensional image data. By introducing an adaptive channel attention module on each selective scanning branch, the channels and labels in each branch feature are adaptively weighted to achieve importance-weighted feature extraction, highlighting important features and suppressing irrelevant information, thereby reducing the channel redundancy caused by the excessive number of hidden states in the state-space model.

[0055] 3. The present invention incorporates a local enhanced feedforward neural network to enhance local features, maintain the feature space dimension, and reduce the number of parameters and computational complexity. It can better understand the local structure and texture information of the image, especially when processing images in complex scenes and diverse weather conditions, and exhibits excellent performance.

[0056] 4. By introducing a multi-scale processing strategy and residual connections, the present invention enables the model to capture multi-scale features from low-level to high-level, capture low-level detail features in the image, and effectively identify and understand high-level semantic information, thereby achieving a comprehensive grasp of the multi-scale features of the image, enhancing the model's adaptability to objects of different sizes and images degraded by various weather conditions, and improving the model's robustness and generalization ability in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 Flowchart of the multi-weather degraded image restoration method based on state space model;

[0058] Figure 2 Network architecture diagram for the multi-weather degraded image restoration model;

[0059] Figure 3 Schematic diagram of the network structure of the residual visual state space block RVSSB;

[0060] Figure 4 Schematic diagram of the network structure of the visual state space module VSSM;

[0061] Figure 5 Schematic diagram of the network structure of the two-dimensional adaptive channel selection scanning module 2D-ACSSM;

[0062] Figure 6 Schematic diagram of the network structure of the adaptive channel state space module ACAtn;

[0063] Figure 7 Visual comparison of this method with other methods on three datasets. DETAILED DESCRIPTION

[0064] To make the above-mentioned objects, features, and advantages of the present invention more readily apparent, the specific implementation methods of the present invention are described in detail below with reference to the accompanying drawings. The following description sets forth many specific details to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the invention. Therefore, the present invention is not limited to the specific implementation methods disclosed below.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art of the art to which the present invention pertains. The terms used in the specification of the invention herein are for the purpose of describing specific embodiments only and are not intended to limit the present invention.

[0066] like Figure 1 As shown in FIG, a multi-weather degraded image restoration method based on a state space model is described. The detailed steps are as follows:

[0067] Step S1: Acquire images under various weather conditions and corresponding clear images, establish a standard data set, preprocess the data set, and divide the processed data set into proportions;

[0068] By collecting a series of high-quality photos of outdoor scenes, ensuring that the collected images cover a variety of outdoor scenes and lighting conditions, the team used image synthesis techniques to add virtual weather effects, including but not limited to simulated rain, snow, and fog, to realistically render various weather phenomena in the images. The realism and degree of degradation of different weather effects were controlled by adjusting parameters, including raindrop size and density, snowflake shape and falling speed, and fog concentration and coverage. The synthesized images were matched one-to-one with the original images to construct a final multi-scene dataset under various weather conditions. The dataset was preprocessed by randomly grouping all images, randomly cropping, flipping, and other operations on the images in each group, and then amplifying them to obtain the preprocessed dataset. The preprocessed dataset was then divided into training, validation, and test sets in an 8:1:1 ratio. The training set was used for iterative model training, the validation set was used to verify model accuracy, and the test set was used for testing.

[0069] Step S2: Construct a restoration network based on the state space model, which includes three stages: shallow feature extraction, deep feature extraction, and high-quality image reconstruction. In the shallow feature extraction stage, the convolution layer is used to extract the shallow features F of the noisy image I. s .

[0070] Step S3: shallow features F obtained in the shallow feature extraction stage s , as the input of the deep feature extraction stage, after the second stage, the deep feature F d The deep feature extraction stage includes: an upsampling layer, a downsampling layer, n residual visual state space groups, and a convolutional layer at the end.

[0071] Step S4: Extract the shallow features F s and the deep features F in the deep feature extraction stage d After fusion, it is used as the input of the high-quality image reconstruction stage, and a clear image I is obtained after processing in this stage. c .

[0072] Step S5: Construct a multi-weather degradation image restoration model based on the network, such as Figure 2 The network model architecture is shown in FIG. The industrial defect detection network model is trained and optimized to obtain a trained model;

[0073] Step S6: obtaining an image to be restored that has been degraded due to weather influence, pre-processing the image, and inputting the image into the trained multi-weather degraded image restoration model for restoration to obtain a clear image.

[0074] like Figure 2As shown, the shallow feature extraction stage of the present invention specifically includes the following operations: the channel dimension of the input image is converted from 3 to a new embedding space 48 through a 3*3 convolution kernel, and the spatial dimension of the image is kept unchanged. Then, the output feature map is flattened in the spatial dimension so that the subsequent network layer can process these features more effectively. The output is the shallow feature F s， Its formula is expressed as:

[0075] F s =Reshape(Conv2D(I))

[0076] Where I is the input degraded image, Conv2D represents the two-dimensional convolution operation

[0077] like Figure 2 As shown, in step S3, the deep feature extraction stage includes the following steps:

[0078] Step S31: The shallow feature F described in step S2 s After the first residual visual state space group RVSSG, the feature F1 is obtained.

[0079] Step S32: Through downsampling operation, the F1 channel dimension is increased to twice the original one, and the height and width are reduced to 1 / 2 of the original one.

[0080] Step S33: The downsampled features obtained in step S32 are passed through the second residual visual state space group RVSSG to obtain feature F2.

[0081] Step S34: Repeat steps S32 and S33 to perform multi-level downsampling and feature extraction. After each downsampling, the channel dimension is doubled and the height and width are halved. At the same time, deep features are extracted through the residual visual state space group RVSSG until the preset depth level is reached, and features F3 and F4 are obtained in turn.

[0082] Step S35: After reaching the deepest level, upsampling is started, and feature fusion is performed through the residual visual state space group RVSSG, gradually reconstructing the height and width of the image while reducing the channel dimension until the spatial resolution is the same as the input image, and the final deep feature representation F is obtained. d The formula is:

[0083] F1=R1(F s )

[0084] F i =R i (D i-1 (F i-1 )) (i=2,3,4)

[0085] Fj =R j (U j-4 (F j-1 )) (j=5,6,7)

[0086] F d =Conv2D(F j )

[0087] Among them, R i Represents the first residual visual state space group, F i Denotes the obtained i-th feature map, D i-1 represents the i-1th downsampling layer, U j-4 represents the j-4th upsampling layer, and Conv2D represents the convolution operation.

[0088] In step S4, the specific operations of the high-quality image reconstruction stage are:

[0089] The shallow feature F in step S2 s and the deep features F in step S3 d After addition, it is used as the input of the high-quality image reconstruction stage, and the feature is reconstructed by the residual visual state space group RVSSG. The reconstructed features are mapped to the target output space through the convolution layer to obtain the final clear image restoration result I c , its formula is expressed as:

[0090] I c =Conv2D(R f (F s +F d ))

[0091] Among them, R f It is represented as the last residual visual state space group, and Conv2D represents the two-dimensional convolution operation.

[0092] like Figure 2 As shown in FIG, the residual visual state space group RVSSG contains several residual visual state space blocks RVSSB, and an additional convolutional layer is added at the end to refine the features extracted from RVSSB.

[0093] like Figure 3 As shown, the specific process of RVSSB is:

[0094] First, the input feature F d iA normalization layer, LayerNorm, is used to reduce the internal covariate shift, which helps stabilize the training process and accelerate the convergence speed. Then, the visual state space module, VSSM, is used to capture the spatial long-term dependencies to enhance the relevance between features. Then, a learnable scaling factor s1 is used to control the information from the skip connection.

[0095] Then, the second layer normalization, LayerNorm, and the local enhanced feedforward neural network, LEFF, are used to enhance the local features, maintain the feature space dimension, and reduce the parameter quantity and computational complexity. Finally, another adjustable scaling factor s2 is used in the residual connection to obtain the final output of the RVSSB. The formula is represented as:

[0096]

[0097] where F d i represents the deep features extracted by the i-th RVSSM, s1 and s2 represent the adjustable scaling factors, Z i represents the intermediate features processed by the i-th VSSM.

[0098] The LEFF has the following specific stages:

[0099] Stage 1: The input feature X is mapped to a high-dimensional space by a linear layer linear1, which helps the model capture more complex feature representations.

[0100] Stage 2: The features after linear transformation are activated by the GELU activation function, which increases the non-linear processing ability and enhances the expression ability of the model.

[0101] Stage 3: The features after the activation function are transformed in two-dimensional space, and then processed by the depth separable convolution dwconv, which maintains the feature channel number and enhances the spatial feature extraction ability.

[0102] Stage 4: The features processed by the depth convolution are mapped back to the original dimension by the linear layer linear2 and the activation function, and then rearranged.

[0103] The VSSM can capture long-range dependencies using the state space equation, adaptively weight using the adaptive channel attention, and extract important features and suppress irrelevant information, thereby reducing channel redundancy caused by excessive hidden state quantities. As shown in Figure 4 The specific process of VSSM is as follows:

[0104] The input feature X passes through two parallel branches. In the first branch, the feature channel is expanded to λC through a linear layer, where λ is a predefined channel expansion factor, followed by a depthwise convolution DWConv, a SiLU activation function, a two-dimensional adaptive channel state space module 2D-ACSSM, and a LayerNorm. In the second branch, the feature channel is also expanded to λC through a linear layer and a SiLU activation function. The features of the two branches are then aggregated using the Hadamard product. Finally, the channel number is projected back to C to generate an output X with the same shape as the input. VSSM The process formula is as follows:

[0105] X1=LayerNorm(2DACSSM(SILU(DWConv(Linear(x))))))

[0106] X2=SILU(Linear(x))

[0107] X VSSM =X1⊙X2

[0108] Among them, DWConv represents depthwise convolution, ⊙ represents Hadamard product, X1 represents the output result of the first branch, and X2 represents the output result of the second branch.

[0109] like Figure 5 As shown, the 2D-ACSSM includes the following specific processes:

[0110] The two-dimensional image features are flattened into a one-dimensional sequence. Each input image feature is processed by four different selective scanning branches that scan in different directions. Then, the long-range dependencies of each sequence are captured according to the discrete state space equation. To enable the model to focus on important information and suppress irrelevant information, the present invention introduces an adaptive channel attention module ACAtn in each branch. This module adaptively weights the features of each channel in each branch and determines the channel weights by aggregating and linearly transforming the global features. Finally, all sequences are merged using the summation method, followed by a reshaping operation to restore the two-dimensional structure.

[0111] Among them, the method of the present invention adopts a selective scanning mechanism S6, and the specific process of the adaptive channel attention module ACAtn is as follows: Figure 6 As shown:

[0112] First, the input feature x is normalized by LayerNorm to reduce internal covariate shift and enhance the stability of the model. Then, average pooling AvgPool is performed on the spatial dimension to obtain the global feature x global, then reduce the dimension through a fully connected layer FC to obtain the global feature representation after dimension reduction, and process the global feature through the activation function GELU and then through another fully connected layer FC to obtain the channel attention coefficient C attn ; C attn It is processed by the gating function Sigmoid and converted into a probability value between 0 and 1, indicating the importance of each channel Channel attn; finally, the original input feature X is multiplied by the channel attention coefficient to achieve weighting of different channels, enhance the features of important channels and suppress unimportant channels. The weighted features are the output of this module.

[0113] During the training process, the loss function and training parameters are defined, the model is iteratively trained and optimized, and the pre-trained model is saved. The parameter settings of the model during the training process are as follows:

[0114] A progressive training strategy was adopted to train degraded images under various weather conditions. The model was iteratively trained for 200 epochs using the AdamW optimizer (β1 = 0.9, β2 = 0.999, weight decay 1e-4) and L1 loss. The initial learning rate was 3e-4 and gradually decreased to 1e-6 using cosine annealing. For progressive learning, the present invention started training with a patch size of 128×128 and a batch size of 8. The patch size and batch size pairs were updated to [(1602, 8), (1922, 4), (2562, 2), (3202, 1), (3842, 1)]. The L1 loss function formula is as follows:

[0115]

[0116] Among them, ||-||1 represents the L1 criterion. HQ Represents the high-quality image after network restoration, I LQ Represented as a low-quality image of the input.

[0117] Table 1 compares this method with other methods. Using PSNR and SSIM metrics, the performance of different methods on the Test1 (dehazing and rain removal), Snow100K-L (snow removal), RainDrop (raindrop removal), and Rain1400 (rain streak removal) test datasets is effectively evaluated. This effectively assesses the performance of different methods on tasks such as dehazing, rain removal, snow removal, and raindrop removal. Compared to methods for restoring images degraded in multiple weather conditions and those for restoring images degraded in specific weather conditions, this method achieves significant performance improvements over existing methods while requiring fewer parameters. While maintaining high performance, it also offers lower computational cost and faster processing speed.

[0118] Table 1

[0119]

[0120] As Figure 7 shown, the effect of the present method compared with other methods in visual pairs; where the red box corresponds to the enlarged patch. Column 1 is the original input image, columns 2 and 3 are the results recovered by Trans Weather and Utility IR methods respectively, column 4 is the result recovered by the ACMamba method of the present application, and column 5 is the label image. The image recovered by the present method is almost consistent with the real background image in visual effect, without the problems such as artifacts, blur and unremoved noise commonly seen in other methods. This shows that the present method has obvious advantages in the quality of image recovery.

[0121] Specifically, it is crucial to have a large receptive field for image restoration tasks, especially for rain and snow removal. Activating more pixels usually produces better recovery results. Because of the larger receptive field, high-level patterns and architectures of the image can be extracted, information can be captured from a wider area, more pixels can be referenced, and the reconstruction of the target pixel can be facilitated. The present method is based on a state space model, which can model very long dependencies when equipped with specially designed structured reparameterization, so the recovery network model constructed has a large receptive field and can activate more pixels, thereby improving the quality of image recovery. In addition, the local enhancement feedforward neural network integrated in the model not only inherits the advantages of convolutional neural networks in local feature extraction, but also combines the ability of state space models in capturing long-range dependencies. This hybrid design enables the model to extract global features while effectively encoding local detail information of the image, further enhancing the accuracy and clarity of the recovered image.

[0122] In summary, by combining advanced network architecture and training strategy, the present method not only achieves excellent performance in multiple challenging weather removal tasks, but also has significant advantages in computational efficiency, providing an effective solution for the field of multi-weather degraded image restoration.

[0123] The above is the preferred embodiment of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, several improvements and refinements can be made, which should be considered within the scope of protection of the present application.

Claims

1. A multi-weather degraded image restoration method based on a state space model, characterized in that: The steps include: Step S1: Acquire multiple weather clear images, process them into degraded images of corresponding weather, and establish a multi-weather standard dataset, wherein the dataset stores {degraded image, clear image} pairs; Preprocess the data set and divide it into proportions; Step S2: Construct a multi-weather degraded image restoration network based on a state-space model. The restoration network includes: a shallow feature extraction stage, a deep feature extraction stage, and a high-quality image reconstruction stage. In the shallow feature extraction stage, a convolutional layer is used to extract shallow features F of the degraded image I. s The deep feature extraction stage includes an upsampling layer, a downsampling layer, n residual visual state space groups, and a convolution layer at the end; the shallow feature F s Processing to obtain deep features F d The high-quality image reconstruction stage is based on shallow features F s and deep features F d After fusion, it is the input, and after processing, a clear image I is obtained. c The specific operations of the high-quality image reconstruction stage are: The shallow feature F s and deep features F d After addition, it is used as the input of the high-quality image reconstruction stage, and the feature is reconstructed by the residual visual state space group RVSSG. The reconstructed features are mapped to the target output space through the convolution layer to obtain the final clear image restoration result I c , its formula is expressed as: I c =Conv2D(R f (F s +F d )) Among them, R f Represented as the last residual visual state space group, Conv2D represents a two-dimensional convolution operation; Step S3: Based on the multi-weather standard dataset, iteratively train the multi-weather degraded image restoration network to obtain a trained ideal restoration model; Step S4: collecting the weather-degraded image to be restored, pre-processing it, and inputting it into the trained ideal restoration model for multi-weather degraded images for restoration, so as to obtain a clear image of the corresponding weather.

2. The multi-weather degraded image restoration method based on the state space model according to claim 1 is characterized in that: The acquiring of multi-weather clear images comprises: collecting a series of high-quality images of outdoor scenes, wherein the images cover a variety of outdoor scenes and lighting conditions; The image processing is to use an image synthesis method to adjust parameters to control the authenticity and degradation degree of different weather effects, and add virtual weather effects to the collected images.

3. The multi-weather degraded image restoration method based on the state space model according to claim 2, characterized in that: The parameters include the size and density of raindrops, the shape and falling speed of snowflakes, and the concentration and coverage of fog, which are used to realistically render various weather phenomena in the image.

4. The method for restoring images degraded by multiple weather conditions based on a state space model according to claim 1, characterized in that: The preprocessing of the dataset includes: randomly grouping all images, randomly cropping, flipping, and manipulating the images in each group, performing data amplification to obtain the preprocessed dataset; and dividing it into training set, validation set, and test set in proportion.

5. The method for restoring a multi-weather-degraded image based on a state space model according to claim 1, characterized in that: The shallow feature extraction stage converts the channel dimension of the input image from 3 to a new embedding space 48 through a 3*3 convolution kernel, and keeps the spatial dimension of the image unchanged; then, the output feature map is flattened in the spatial dimension; its output is the shallow feature F s , the formula is: F s =Reshape(Conv2D(I))。 6. The method for restoring a multi-weather degraded image based on a state space model according to claim 1, characterized in that: The deep feature extraction stage includes the following steps: Step a1: transform the shallow feature F s After the first residual visual state space group RVSSG, the feature F1 is obtained; Step a2: Through downsampling, the F1 channel dimension is doubled and the height and width are reduced to 1 / 2 of the original; Step a3: Pass the downsampled features obtained in step a2 through the second residual visual state space group RVSSG to obtain feature F2; Step a4: Repeat steps a2 and a3 to perform multi-level downsampling and feature extraction. After each downsampling, the channel dimension is doubled and the height and width are halved. At the same time, deep features are extracted through the residual visual state space group RVSSG until the preset depth level is reached, and features F3 and F4 are obtained in sequence. Step a5: After reaching the deepest level, perform upsampling to extract feature G i , while collecting the feature map F of the same resolution in the downsampling branch i , feature fusion is performed through the residual visual state space group RVSSG, gradually reconstructing the height and width of the image while reducing the channel dimension until it is reconstructed to the same spatial resolution as the input image, and the final deep feature representation F is obtained d ; The formula is: F1=R1(F s ) F i =R i (D i-1 (F i-1 ))(i=2,3,4) F j =R j (U j-4 (F j-1 ))(j=5,6,7) F d =Conv2D(F j ) Among them, R i Represents the first residual visual state space group, F i Denotes the obtained i-th feature map, D i-1 represents the i-1th downsampling layer, U j-4 represents the j-4th upsampling layer, and Conv2D represents the convolution operation.

7. The method for restoring a multi-weather-degraded image based on a state space model according to claim 1 or 6, characterized in that: The residual visual state space group RVSSG consists of several residual visual state space blocks RVSSB connected in series and a convolutional layer to refine the features extracted from RVSSB. The specific operation of RVSSB is: First input feature F d i Processing is performed through a normalization layer LayerNorm to reduce internal covariate shift, stabilize the training process, and speed up convergence; Then, the visual state space module (VSSM) is used to capture long-term spatial dependencies to enhance the correlation between features. Then, the information from the skip connection is controlled by using a learnable scaling factor s1. Next, the second layer is normalized and the local enhanced feedforward neural network LEFF is used to reduce the number of parameters and computations while enhancing local features and maintaining the dimension of the feature space. Finally, another adjustable scaling factor s2 is used in the residual connection to obtain the final output of RVSSB. Its formula is expressed as: Among them, F d i Represents the deep features extracted by the i-th RVSSM, s1, s2 represent adjustable scale factors, Z i Represents the intermediate features after processing by the i-th VSSM.

8. The method for restoring a multi-weather-degraded image based on a state space model according to claim 7, characterized in that: VSSM uses state-space equations to capture long-range dependencies and adaptively weights channel attention to achieve importance-weighted feature extraction, extracting important features and suppressing irrelevant information, thereby reducing channel redundancy caused by an excessive number of hidden states. The specific operations of VSSM are: The input feature X passes through two parallel branches. In the first branch, the feature channel is expanded to λC through a linear layer, where λ is a predefined channel expansion factor, followed by a depthwise convolution DWConv, a SiLU activation function, a two-dimensional adaptive channel state space module 2D-ACSSM, and a normalization layer LayerNorm. In the second branch, the feature channel is also expanded to λC through a linear layer and a SiLU activation function. Then, the features of the two branches are aggregated using the Hadamard product operation. Finally, the channel numbers are projected back to C, generating an output X of the same shape as the input VSSM : X1=LayerNorm(2DACSSM(SILU(DWConv(Linear(x)))))) X2=SILU(Linear(x)) X VSSM =X1⊙X2 Among them, DWConv represents depthwise convolution, ⊙ represents Hadamard product, X1 represents the output result of the first branch, and X2 represents the output result of the second branch.

9. The method for restoring a multi-weather degraded image based on a state space model according to claim 8, characterized in that: 2D-ACSSM includes: a. flattening the two-dimensional image features into image features of a one-dimensional sequence; b. scanning and processing each input image feature along different directions through four different selective scanning branches SSMBLock to capture the long-range dependency of each sequence according to the discrete state space equation; c. introducing an adaptive channel attention module ACAtn in each branch to adaptively weight the features of each channel in each branch, and determining the channel weight by aggregating and linearly transforming the global features; d. using the summation method to merge all sequences for reshaping operations to restore the two-dimensional image structure.

Images