Bad weather image restoration method based on multi-modal state space model

Abstract

The invention discloses a bad weather image restoration method based on a multi-modal state space model, and belongs to the technical field of computer vision. In order to solve the problem that an existing unified bad weather image restoration method has limitations in the aspects of global receptive field and computational efficiency, the unified restoration of various bad weather images is realized by designing a parallel multi-mode encoder and an adapter to generate comprehensive prompts containing degeneration semantics. Through parallel connection of a state space module and a local context sensing module of a double-attention mechanism, simultaneous capture of global long-range dependence and local detailed features is realized. Experiments show that the method is high in generalization ability, high in processing speed and low in calculation complexity, and can show good adaptive capacity on a plurality of bad weather image removal tasks.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a bad weather image restoration method based on a multimodal state space model. Background Art

[0002] Due to poor conditions and camera limitations, images captured in bad weather conditions are often subject to degradation (e.g., rain, raindrops, snow, and haze). These degradations not only affect image clarity but also pose challenges to many downstream computer vision tasks, such as object detection, semantic segmentation, autonomous driving, and object tracking. Therefore, restoring degraded images is crucial for improving image quality and enhancing the robustness of computer vision tasks. However, this problem is particularly complex due to its ill-posed nature. Therefore, how to restore clear image data from the large amount of degraded data in bad weather conditions and improve autonomous perception and adaptation to the environment has become a research hotspot in fields such as robotics, autonomous driving, and computer vision.

[0003] Early research in bad weather image restoration focused primarily on addressing single degradation problems (such as rain removal, fog removal, snow removal, or raindrop removal). Techniques such as convolutional neural networks (CNNs), generative adversarial networks (GANs), visual transformers (ViTs), and diffusion models have been widely used in these tasks with significant success. However, a limitation of these approaches lies in their specificity—each model is typically only effective at handling one type of degradation. Given that real-world outdoor scenarios (such as autonomous driving and outdoor surveillance) often face diverse weather conditions, building a universal model that can handle all weather conditions is more practical than maintaining multiple specialized models. To this end, unified approaches designed to handle multiple types of degradation tasks have emerged in recent years. Some strategies involve task-specific fine-tuning of the base model. While these unified architectures can handle a wide range of image degradations, the network modules corresponding to each task still require independent training.

[0004] The above limitations make these methods computationally inefficient and complex in actual deployment. To overcome these obstacles, a unified image restoration paradigm was subsequently developed, the core of which is to use a model with shared pre-trained weights to deal with multiple degradations. Although the unified framework based on CNN, Visual Transformer (ViT) and diffusion model has performed well, they all face key bottlenecks: CNN has deficiencies in model fitting ability and global semantic modeling on large datasets; diffusion model introduces a heavy computational burden due to the dense denoising steps; visual transformers have difficulty in efficiently processing large-scale images due to their quadratic computational complexity with respect to the input scale. Therefore, designing more suitable network structures and exploring efficient learning paradigms are key ways to further improve the adaptability and generalization capabilities of the unified method.

[0005] Furthermore, degradation semantic information, such as degradation type and degree, is equally important for achieving unified bad weather removal. Compared to single-modal image information, multimodal information can more comprehensively and accurately describe degradation semantics. At the same time, generating comprehensive degradation semantic information using large multimodal models is also a challenge. Summary of the Invention

[0006] To address the limitations of existing unified bad weather image restoration methods in terms of global receptive field and computational efficiency, the present invention provides a bad weather image restoration method based on a multimodal state space model. The method mainly consists of three steps: first, adaptively perceive multimodal information from input data; second, fuse the degradation semantics of different modalities and transmit them to the restoration network; finally, establish a locally enhanced Mamba UNet to restore the image under the guidance of multimodal information.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A bad weather image restoration method based on a multimodal state space model comprises the following steps:

[0009] Step 1: Design a bad weather image restoration network model based on a multimodal state-space model. Input the image text description and image into the network model and use the parallel multimodal feature encoder in the network model to represent the text and image features. The parallel multimodal feature encoder includes a text encoder using the CLIP text encoder and an image encoder using the local enhancement Mamba block.

[0010] The specific operations of step 1 are:

[0011] Pre-train text encoders and image encoders. Use the pre-trained CLIP text encoder to extract degenerate semantics from text descriptions, and use the image encoder to capture local detail features and global long-range dependencies. The feature encoding process of multimodal data is defined as follows:

[0012]

[0013] in, and Represents text encoder and image encoder respectively, represents the extracted text modal features, represents the extracted image modality features, Represents text data, Represents image data.

[0014] Step 2: The text / image adaptive perception module in the network model adaptively extracts the different degraded semantics of the two modalities through a parallel adapter;

[0015] The specific operations of step 2 are:

[0016] The text and image adapters for degradation types adaptively perceive the characteristics of bad weather types, while the text and image adapters for degradation semantics adaptively extract other semantics (such as degradation level and contextual semantics). The adaptively perceived bimodal information is fused through a cross-attention mechanism.

[0017] In the encoding feature and Apply global average pooling on the spatial dimension of Generate global semantic features, and then these global semantic features pass through the double layer and Operation generates prompt weight , these weights are used to adjust the prompt component and transform the contextual prompt by connecting a 3×3 convolutional layer; the processing flow of the text / image adaptive perception module can be summarized as follows:

[0018]

[0019] in, represents a hint element consisting of learnable parameters, represents the text / image adaptive perception module, Represents the learned prompt features.

[0020] Step 3: The degradation semantics of different modalities are interactively fused using a multimodal information fusion strategy through a cross-attention mechanism and transmitted to the restoration network to guide the removal of bad weather conditions;

[0021] The structure of the restoration network in step 3 includes: an input convolutional layer, an encoder, a decoder, and an output convolutional layer; a skip connection is provided between the encoder and the decoder; each stage of the encoder and the decoder is configured with multiple locally enhanced Mamba blocks (LEMBs), and the number of blocks gradually increases from the top stage to the bottom stage to maintain computational efficiency;

[0022] The encoder includes: a local enhancement Mamba block and a downsampling block;

[0023] The decoder includes: a local enhanced Mamba block and an upsampling block;

[0024] The locally enhanced Mamba block includes: a locally enhanced state space module, a multi-scale gated feedforward network;

[0025] The locally enhanced state space module includes: a long-range dependency perception branch and a local context perception branch;

[0026] The long-range dependency perception branch consists of a selective state space module and a dual attention module;

[0027] The local context perception branch includes feature expansion transformation, local context modeling, feature compression transformation, and feature fusion;

[0028] The multi-scale gated feedforward network includes: a normalization layer and a dual-scale gating structure; each gating structure is composed of the element-by-element product of two parallel branches.

[0029] The specific operations of step 3 are:

[0030] Step 3: The degradation semantics of different modalities are interactively fused using a multimodal information fusion strategy through a cross-attention mechanism and transmitted to the restoration network to guide the removal of bad weather conditions;

[0031] Step 3.1: Extract low-level feature embeddings from the image input from step 1 through convolution operations. These feature embeddings are processed through a three-level hierarchical encoder to gradually reduce the spatial resolution while expanding the channel capacity to generate low-resolution latent representations.

[0032] Step 3.2: The feature embedding processed by the encoder and the learned prompt features are input into the decoder of the corresponding layer to gradually decode the features of the clear image;

[0033] Step 3.3: The refinement network consisting of a local enhancement Mamba block and a 3×3 convolutional layer optimizes the features and generates a clear image.

[0034] The structure of the local enhancement Mamba block follows the process of "normalization → local enhancement state space model (LESSM) → normalization → multi-scale gated convolutional feedforward network (MGFFN)" with residual connections. The operation process of the local enhancement Mamba block is expressed as follows:

[0035]

[0036] in, represents the local augmented state space model, represents a multi-scale gated convolutional feedforward network, represents normalization, represents the features processed by the local enhanced state space model, Represents features processed by a multi-scale feedforward network;

[0037] The locally enhanced state-space module not only benefits from the selective state-space module (SSM) to capture long-range dependencies, but also contains a local context-aware branch to perceive local detail features.

[0038] Specifically, the locally enhanced state-space module consists of two main branches: the long-range dependency-aware branch and the local context-aware branch (LCB).

[0039] The long-range dependency perception branch consists of a selective state space module and a dual attention module;

[0040] The operation of the locally enhanced state-space module is:

[0041] The selective state space module in the locally enhanced state space module first performs four scans in the vertical and horizontal directions to capture the two-dimensional information of the features. The two-dimensional information of the features is then used by the selective state space module to extract long-range dependencies. At the same time, the spatial attention and channel attention mechanisms are introduced to enable the selective state space module to adaptively recalibrate the feature response at the pixel and channel levels. The above process can be expressed as:

[0042]

[0043] in, represents the input features, and denote channel attention and channel attention respectively;

[0044] The Selective State Space Module implements bidirectional scanning of the image in both horizontal and vertical directions. This operation enables each pixel in the image to effectively integrate information from all other pixels in different directions, which is conducive to establishing a global receptive field in two-dimensional space.

[0045] The operation of the selective state-space module is:

[0046] The input image blocks are expanded into sequences along different traversal directions, and then each sequence is processed in parallel using an independent selective state space module. Finally, the output features are formed by reshaping and merging the sequences. The selective state space module uses a selective scanning mechanism to configure the parameters as a function of the input. The parameter generation process of the selective state space module is as follows:

[0047]

[0048] in, , , Represents the state dimension, represents the time scale parameter, represents the learnable parameters, and Is to input A linear function projected to dimension N, Project the input to dimension , It is the Softplus function;

[0049] The local context-aware branch exploits the advantages of convolutional neural networks in perceiving local detail features to make up for the limitations of the selective state-space module in local feature extraction.

[0050] The operations of the local context-aware branch are as follows: 1) Feature expansion transformation: To perceive rich detail information, the input features are first subjected to channel expansion transformation through a 1×1 convolution layer; 2) Local context modeling: From the transformed features, new features are first generated through a 3×3 convolution operation with a batch normalization layer. At the same time, a 3×3 convolution layer and a 1×1 convolution layer are used with a softmax operation to generate pixel-by-pixel attention from the transformed features; then the two branch features are element-wise multiplied to obtain the context feature descriptor; 3) Feature compression transformation: To capture inter-channel dependencies, the context features are transformed using a 1×1 standard convolution; 4) Feature fusion: The context features are aggregated with the input features through a pixel-by-pixel addition operation. The process is expressed as:

[0051]

[0052] in, represents the input features, represents the extracted local features, is the feature expansion transformation, is local context modeling, is the feature compression transformation;

[0053] Finally, the global long-range dependency features and local features are integrated in the channel dimension. The integrated features pass through a 1×1 convolution layer and perform a residual connection with the input features to obtain the output. The process is expressed as:

[0054]

[0055] [ ] indicates the concat operation, Represents a 1×1 convolution operation.

[0056] After the local enhanced state-space model is built, the features undergo layer normalization to mitigate mode collapse. The normalized features are then fed into an efficient multi-scale gated feedforward network. The multi-scale gated convolutional feedforward network employs a dual-scale gating structure, where each gating structure consists of the element-wise product of two parallel branches.

[0057] The operation of the multi-scale gated feedforward network is as follows: first, the feature channel dimension is transformed through a 1×1 convolutional layer. One branch uses a 3×3 depthwise separable convolutional layer to further extract comprehensive semantic features for effective restoration; the other two branches use 3×3 and 5×5 depthwise separable convolutional layers with GELU activation functions to generate gating weights for multiple receptive fields. Finally, the multi-scale features are concatenated and the channel dimension is compressed through a 1×1 convolutional layer to match the input channel size. The process is expressed as:

[0058]

[0059] in, Indicates passing Features processed by depthwise separable convolution branches, represents the GELU activation function, represents depthwise separable convolution, Represents the features after processing by the multi-scale feedforward network, and [ ] represents the concat operation.

[0060] Step 4: Test the effectiveness of the method on real bad weather degraded images.

[0061] Compared with the prior art, the present invention has the following advantages:

[0062] This method proposes a bad weather image restoration method based on a multimodal state-space model for unified bad weather processing. By fine-tuning a pre-trained large model, comprehensive multimodal information containing degradation semantics is generated to guide bad weather removal. A multimodal adaptive perception module is designed, which independently encodes information from different modalities. The encoded features are then fed into a multi-adapter to learn different degradation semantic cues, such as degradation type or level. A locally enhanced state-space module is designed. By parallelizing the state-space module with a dual attention mechanism and a local context perception module, this module can simultaneously capture global long-range dependencies and local detailed features. A novel multiscale gated feedforward network is designed to achieve efficient propagation of multiscale features. A multimodal information fusion strategy is designed to integrate cues from different modalities through a cross-attention mechanism and then feed them into the restoration network to guide bad weather removal. Experimental results demonstrate that this method achieves excellent final image restoration results under unified bad weather degradation. Compared with other methods, this method exhibits superior generalization and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is the overall network framework diagram of the present invention;

[0064] Figure 2 Graphs showing the restoration results of the present invention's elimination tasks under different bad weather conditions. DETAILED DESCRIPTION

[0065] To gain a deeper understanding of the present invention, we will provide a comprehensive and detailed description thereof. However, the present invention has various implementations and is not limited to the specific examples listed herein. These examples are presented to enhance a comprehensive understanding of the present disclosure.

[0066] A bad weather image restoration method based on a multimodal state space model comprises the following steps:

[0067] Step 1: Design a bad weather image restoration network model based on a multimodal state-space model. Input the image text description and image into the network model and use the parallel multimodal feature encoder in the network model to represent the text and image features. The parallel multimodal feature encoder includes a text encoder using the CLIP text encoder and an image encoder using the local enhancement Mamba block.

[0068] The specific operations of step 1 are:

[0069] Pre-train text encoders and image encoders. Use the pre-trained CLIP text encoder to extract degenerate semantics from text descriptions, and use the image encoder to capture local detail features and global long-range dependencies. The feature encoding process of multimodal data is defined as follows:

[0070]

[0071] in, and Represents text encoder and image encoder respectively, represents the extracted text modal features, represents the extracted image modality features, Represents text data, Represents image data.

[0072] Step 2: The text / image adaptive perception module in the network model adaptively extracts the different degraded semantics of the two modalities through a parallel adapter;

[0073] The specific operations of step 2 are:

[0074] The text and image adapters for degradation types adaptively perceive the characteristics of bad weather types, while the text and image adapters for degradation semantics adaptively extract other semantics (such as degradation level and contextual semantics). The adaptively perceived bimodal information is fused through a cross-attention mechanism.

[0075] In the encoding feature and Apply global average pooling on the spatial dimension of Generate global semantic features, and then these global semantic features pass through the double layer and Operation generates prompt weight , these weights are used to adjust the prompt component and transform the contextual prompt by connecting a 3×3 convolutional layer; the processing flow of the text / image adaptive perception module can be summarized as follows:

[0076]

[0077] in, represents a hint element consisting of learnable parameters, represents the text / image adaptive perception module, Represents the learned prompt features.

[0078] Step 3: The degradation semantics of different modalities are interactively fused using a multimodal information fusion strategy through a cross-attention mechanism and transmitted to the restoration network to guide the removal of bad weather conditions;

[0079] The structure of the restoration network in step 3 includes: an input convolutional layer, an encoder, a decoder, and an output convolutional layer; a skip connection is provided between the encoder and the decoder; each stage of the encoder and the decoder is configured with multiple locally enhanced Mamba blocks (LEMBs), and the number of blocks gradually increases from the top stage to the bottom stage to maintain computational efficiency;

[0080] The encoder includes: a local enhancement Mamba block and a downsampling block;

[0081] The decoder includes: a local enhanced Mamba block and an upsampling block;

[0082] The locally enhanced Mamba block includes: a locally enhanced state space module, a multi-scale gated feedforward network;

[0083] The locally enhanced state space module includes: a long-range dependency perception branch and a local context perception branch;

[0084] The long-range dependency perception branch consists of a visual state space module and a dual attention module;

[0085] The local context perception branch includes feature expansion transformation, local context modeling, feature compression transformation, and feature fusion;

[0086] The multi-scale gated feedforward network includes: a normalization layer and a dual-scale gating structure; each gating structure is composed of the element-by-element product of two parallel branches.

[0087] The specific operations of step 3 are:

[0088] Step 3.1: Extract low-level feature embeddings from the input image in step 1 through convolution operations. These feature embeddings are processed through a three-level hierarchical encoder to gradually reduce the spatial resolution while expanding the channel capacity to generate low-resolution latent representations.

[0089] Step 3.2: The feature embedding processed by the encoder and the learned prompt features are input into the decoder of the corresponding layer to gradually decode the features of the clear image;

[0090] Step 3.3: The refinement network consisting of a local enhancement Mamba block and a 3×3 convolutional layer optimizes the features and generates a clear image.

[0091] The structure of the local enhancement Mamba block follows the process of "normalization → local enhancement state space model (LESSM) → normalization → multi-scale gated convolutional feedforward network (MGFFN)" with residual connections. The operation process of the local enhancement Mamba block is expressed as follows:

[0092]

[0093] in, represents the local augmented state space model, represents a multi-scale gated convolutional feedforward network, represents normalization, represents the features processed by the local enhanced state space model, Represents features processed by a multi-scale feedforward network;

[0094] The locally enhanced state-space module not only benefits from the selective state-space module (SSM) to capture long-range dependencies, but also contains a local context-aware branch to perceive local detail features.

[0095] Specifically, the locally enhanced state-space module consists of two main branches: the long-range dependency-aware branch and the local context-aware branch (LCB).

[0096] The long-range dependency perception branch is mainly composed of a visual state space module and a dual attention module.

[0097] The operation of the locally enhanced state-space module is:

[0098] The selective state space module in the locally enhanced state space module first performs four scans in the vertical and horizontal directions to capture the two-dimensional information of the features. The two-dimensional information of the features is then used by the selective state space module to extract long-range dependencies. At the same time, the spatial attention and channel attention mechanisms are introduced to enable the selective state space module to adaptively recalibrate the feature response at the pixel and channel levels. The above process can be expressed as:

[0099]

[0100] in, represents the input features, and denote channel attention and channel attention respectively;

[0101] The Selective State Space Module implements bidirectional scanning of the image in both horizontal and vertical directions. This operation enables each pixel in the image to effectively integrate information from all other pixels in different directions, which is conducive to establishing a global receptive field in two-dimensional space.

[0102] The operation of the selective state-space module is:

[0103] The input image blocks are expanded into sequences along different traversal directions, and then each sequence is processed in parallel using an independent selective state space module. Finally, the output features are formed by reshaping and merging the sequences. The selective state space module uses a selective scanning mechanism to configure the parameters as a function of the input. The parameter generation process of the selective state space module is as follows:

[0104]

[0105] in, , , Represents the state dimension, represents the time scale parameter, represents the learnable parameters, and Is to input A linear function projected to dimension N, Project the input to dimension , It is the Softplus function;

[0106] The local context-aware branch exploits the advantages of convolutional neural networks in perceiving local detail features to make up for the limitations of the selective state-space module in local feature extraction.

[0107] The operations of the local context-aware branch are as follows: 1) Feature expansion transformation: To perceive rich detail information, the input features are first subjected to channel expansion transformation through a 1×1 convolution layer; 2) Local context modeling: From the transformed features, new features are first generated through a 3×3 convolution operation with a batch normalization layer. At the same time, a 3×3 convolution layer and a 1×1 convolution layer are used with a softmax operation to generate pixel-by-pixel attention from the transformed features; then the two branch features are element-wise multiplied to obtain the context feature descriptor; 3) Feature compression transformation: To capture inter-channel dependencies, the context features are transformed using a 1×1 standard convolution; 4) Feature fusion: The context features are aggregated with the input features through a pixel-by-pixel addition operation. The process is expressed as:

[0108]

[0109] in, represents the input features, represents the extracted local features, is the feature expansion transformation, is local context modeling, is the feature compression transformation;

[0110] Finally, the global long-range dependency features and local features are integrated in the channel dimension. The integrated features pass through a 1×1 convolution layer and perform a residual connection with the input features to obtain the output. The process is expressed as:

[0111]

[0112] [ ] indicates the concat operation, Represents a 1×1 convolution operation.

[0113] After the local enhanced state-space model is built, the features undergo layer normalization to mitigate mode collapse. The normalized features are then fed into an efficient multi-scale gated feedforward network. The multi-scale gated convolutional feedforward network employs a dual-scale gating structure, where each gating structure consists of the element-wise product of two parallel branches.

[0114] The operation of the multi-scale gated feedforward network is as follows: first, the feature channel dimension is transformed through a 1×1 convolutional layer. One branch uses a 3×3 depthwise separable convolutional layer to further extract comprehensive semantic features for effective restoration; the other two branches use 3×3 and 5×5 depthwise separable convolutional layers with GELU activation functions to generate gating weights for multiple receptive fields. Finally, the multi-scale features are concatenated and the channel dimension is compressed through a 1×1 convolutional layer to match the input channel size. The process is expressed as:

[0115]

[0116] in, Indicates passing Features processed by depthwise separable convolution branches, represents the GELU activation function, represents depthwise separable convolution, Represents the features after processing by the multi-scale feedforward network, and [ ] represents the concat operation.

[0117] Step 4: Test the effectiveness of the method on real bad weather degraded images.

[0118] Figure 2 Table 1 is the visualization result of the present invention method in different bad weather elimination tasks. Table 1 is the quantitative result of the present invention method in different bad weather elimination tasks. Figure 2 From the information in Table 1, it can be concluded that the method of the present invention performs stably in various bad weather removal tasks and can generate clear images with good visual effects.

[0119] Table 1 Quantitative results of the elimination task of the method of the present invention in different bad weather conditions.

[0120] index rain raindrop Snow fog PSNR 35.207 29.901 30.223 31.321 SSMI 0.963 0.953 0.942 0.971

[0121] Any matters not described in detail in this specification are prior art known to those skilled in the art. Although the above description of the present invention is based on specific embodiments to facilitate understanding of the present invention by those skilled in the art, it should be understood that the present invention is not limited to the scope of the specific embodiments. As long as various modifications are within the spirit and scope of the present invention as defined and determined by the appended claims, such modifications will be obvious to those skilled in the art, and all inventions and creations utilizing the concepts of the present invention are protected.

Claims

1. A bad weather image restoration method based on a multimodal state space model, characterized in that: The method comprises the following steps: Step 1: Design a bad weather image restoration network model based on a multimodal state-space model. Input the image text description and image into the network model and use the parallel multimodal feature encoder in the network model to represent the text and image features. The parallel multimodal feature encoder includes a text encoder using the CLIP text encoder and an image encoder using the local enhancement Mamba block. Step 2: The text / image adaptive perception module in the network model adaptively extracts the different degraded semantics of the two modalities through a parallel adapter; Step 3: The degradation semantics of different modalities are interactively fused using a multimodal information fusion strategy through a cross-attention mechanism and transmitted to the restoration network to guide the removal of bad weather conditions; Step 4: Test the effectiveness of the method on real bad weather degraded images.

2. The bad weather image restoration method based on the multimodal state space model according to claim 1 is characterized in that: The specific operations of step 1 are: Pre-train text encoders and image encoders. Use the pre-trained CLIP text encoder to extract degenerate semantics from text descriptions, and use the image encoder to capture local detail features and global long-range dependencies. The feature encoding process of multimodal data is defined as follows: ， in, and Represents text encoder and image encoder respectively, represents the extracted text modal features, represents the extracted image modality features, Represents text data, Represents image data.

3. The bad weather image restoration method based on the multimodal state space model according to claim 2 is characterized in that: The specific operations of step 2 are: In the encoding feature and Apply global average pooling on the spatial dimension of Generate global semantic features, and then these global semantic features pass through the double layer and Operation generates prompt weight , these weights are used to adjust the prompt component and transform the contextual prompt by connecting a 3×3 convolutional layer; the processing flow of the text / image adaptive perception module can be summarized as follows: ， in, represents a hint element consisting of learnable parameters, represents the text / image adaptive perception module, Represents the learned prompt features.

4. The bad weather image restoration method based on the multimodal state space model according to claim 3 is characterized in that: The structure of the restoration network in step 3 includes: an input convolutional layer, an encoder, a decoder, and an output convolutional layer; a skip connection is provided between the encoder and the decoder; each stage of the encoder and the decoder is configured with multiple local enhancement Mamba blocks, and the number of blocks gradually increases from the top stage to the bottom stage to maintain computational efficiency; The encoder includes: a local enhancement Mamba block and a downsampling block; The decoder includes: a local enhanced Mamba block and an upsampling block; The locally enhanced Mamba block includes: a locally enhanced state space module, a multi-scale gated feedforward network; The locally enhanced state space module includes: a long-range dependency perception branch and a local context perception branch; The long-range dependency perception branch consists of a selective state space module and a dual attention module; The local context perception branch includes feature expansion transformation, local context modeling, feature compression transformation, and feature fusion; The multi-scale gated feedforward network includes: a normalization layer and a dual-scale gating structure; each gating structure is composed of the element-by-element product of two parallel branches.

5. The bad weather image restoration method based on the multimodal state space model according to claim 4 is characterized in that: The specific operations of step 3 are: Step 3.1: Extract low-level feature embeddings from the input image in step 1 through convolution operations. These feature embeddings are processed through a three-level hierarchical encoder to gradually reduce the spatial resolution while expanding the channel capacity to generate low-resolution latent representations. Step 3.2: The feature embedding processed by the encoder and the learned prompt features are input into the decoder of the corresponding layer to gradually decode the features of the clear image; Step 3.3: The refinement network consisting of the local enhancement Mamba block and the 3×3 convolutional layer optimizes the features and generates a clear image. The operation process of the local enhancement Mamba block is expressed as follows: ， in, represents the local augmented state space model, represents a multi-scale gated convolutional feedforward network, represents normalization, represents the features processed by the local enhanced state space model, Represents features processed by a multi-scale feedforward network; The operation of the locally enhanced state-space module is: The selective state space module in the locally enhanced state space module first performs four scans in the vertical and horizontal directions to capture the two-dimensional information of the features. The two-dimensional information of the features is then used by the selective state space module to extract long-range dependencies. At the same time, the spatial attention and channel attention mechanisms are introduced to enable the selective state space module to adaptively recalibrate the feature response at the pixel and channel levels. The above process can be expressed as: ， in, represents the input features, and denote channel attention and channel attention respectively; The operation of the selective state-space module is: The input image blocks are expanded into sequences along different traversal directions, and then each sequence is processed in parallel using an independent selective state space module. Finally, the output features are formed by reshaping and merging the sequences. The selective state space module uses a selective scanning mechanism to configure the parameters as a function of the input. The parameter generation process of the selective state space module is as follows: ， in, , , Represents the state dimension, represents the time scale parameter, represents the learnable parameters, and Is to input A linear function projected to dimension N, Project the input to dimension , It is the Softplus function; The operations of the local context-aware branch are as follows: 1) Feature expansion transformation: To perceive rich detail information, the input features are first subjected to channel expansion transformation through a 1×1 convolution layer; 2) Local context modeling: From the transformed features, new features are first generated through a 3×3 convolution operation with a batch normalization layer. At the same time, a 3×3 convolution layer and a 1×1 convolution layer are used with a softmax operation to generate pixel-by-pixel attention from the transformed features; then the two branch features are element-wise multiplied to obtain the context feature descriptor; 3) Feature compression transformation: To capture inter-channel dependencies, the context features are transformed using a 1×1 standard convolution; 4) Feature fusion: The context features are aggregated with the input features through a pixel-by-pixel addition operation. The process is expressed as: ， in, represents the input features, represents the extracted local features, is the feature expansion transformation, is local context modeling, is the feature compression transformation; Finally, the global long-range dependency features and local features are integrated in the channel dimension. The integrated features pass through a 1×1 convolution layer and perform a residual connection with the input features to obtain the output. The process is expressed as: ， [ ] indicates the concat operation, Represents a 1×1 convolution operation; The operation of the multi-scale gated feedforward network is as follows: first, the feature channel dimension is transformed through a 1×1 convolutional layer. One branch uses a 3×3 depthwise separable convolutional layer to further extract comprehensive semantic features for effective restoration; the other two branches use 3×3 and 5×5 depthwise separable convolutional layers with GELU activation functions to generate gating weights for multiple receptive fields. Finally, the multi-scale features are concatenated and the channel dimension is compressed through a 1×1 convolutional layer to match the input channel size. The process is expressed as: ， in, Indicates passing Features processed by depthwise separable convolution branches, represents the GELU activation function, represents depthwise separable convolution, Represents the features after processing by the multi-scale feedforward network, and [ ] represents the concat operation.

Images