Video rain removal method and system based on spatiotemporal difference state space model
By using a spatiotemporal difference state-space model-based approach, and leveraging consistency-guided differential operations and reparameterized three-plane feedforward networks, the problems of rain artifact preservation and background detail smoothing in existing technologies are solved, achieving high-quality video de-raining effects.
Patent Information
- Application Number
- CN202610297664.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-12
- Publication Date
- 2026-07-03
AI Technical Summary
Existing state-space models tend to mistakenly preserve rain artifacts as salient features in video deraining tasks, resulting in incomplete deraining. At the same time, they over-smooth background details when suppressing rain artifacts, affecting the background fidelity of video restoration.
A method based on a spatiotemporal difference state space model is adopted. Rain pattern features are removed by consistency-guided difference operations, and background texture details are captured by a reparameterized three-plane feedforward network. Spatiotemporal texture enhancement is performed by combining orthogonal convolution branches, and a rain-free video frame sequence is output.
It achieves the goal of preserving background texture to the greatest extent while ensuring thorough rain removal, thereby improving the fidelity of video restoration and significantly enhancing the model's generalization ability and practical application value in complex dynamic scenes.
Smart Images

Figure CN122335612A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a video deraining method and system based on a spatiotemporal difference state space model, belonging to the field of deep learning technology. Background Technology
[0002] Video deraining aims to recover clear background content from video sequences degraded by rain, and is a key preprocessing step in outdoor vision systems such as autonomous driving and intelligent surveillance. With the development of deep learning, video deraining methods have evolved from convolutional neural networks (CNNs) to the Transformer architecture. CNNs excel at extracting local features, but have limitations in capturing long-range spatiotemporal dependencies; while Transformers possess global modeling capabilities, their computational complexity increases quadratically with sequence length, making them unsuitable for real-time processing.
[0003] In recent years, State Space Models (SSMs), especially the Mamba architecture, have become the mainstream technology in the field of video restoration due to their linear complexity and powerful long sequence modeling capabilities. However, directly applying existing State Space Models to video deraining tasks faces two core technical challenges: (1) Attention noise: When filtering key information, Mamba's selective scanning mechanism is prone to mistaking high-frequency rain artifacts as salient features and retaining them, resulting in incomplete deraining. (2) Texture erosion: Although Mamba is good at capturing global context, it often lacks sufficient discriminative power when processing local high-frequency details. Since rain patterns and background textures (such as leaves, texture details, and vehicle edges) are highly coupled in the spatial dimension and have similar features, existing Mamba models often have difficulty accurately decoupling the two when suppressing rain pattern interference. While removing rain patterns, they over-smooth the background details, resulting in texture erosion, which seriously affects the background fidelity of video restoration. Summary of the Invention
[0004] To address the challenge of preserving background texture to the greatest extent possible while ensuring accurate rain removal, this invention proposes a video rain removal method and system based on a spatiotemporal difference state space model.
[0005] The technical solution adopted by this invention to solve the above problems is: a video deraining method based on a spatiotemporal difference state-space model, comprising:
[0006] Step 1: Perform spatiotemporal coding on the given rainy day video and extract its shallow spatiotemporal features; Step 2: Based on shallow spatiotemporal features, rain pattern features are removed through consistency-guided differential operations to obtain differential features; Step 3: Perform spatiotemporal texture enhancement and reconstruction on the differential features, and output a rainless video frame sequence.
[0007] Furthermore, step 1 specifically includes: For the input rainy day video ,in, It is the video frame rate. and These are the height and width, respectively. A 3D convolutional layer is used to extract shallow spatiotemporal features from rainy weather videos. ; Shallow spatiotemporal features The extraction formula is: (1); In formula (1), This represents a 3D convolution operation. These are the extracted shallow spatiotemporal features. It is the feature channel dimension.
[0008] Furthermore, in step 2, a multi-plane scanning mechanism is used to construct volumetric anchor features and planar reference features for the shallow spatiotemporal features. Geometric consistency is calculated based on the volumetric anchor features and planar reference features, and dynamic rain streak features in the shallow spatiotemporal features are removed through a gated difference mechanism to obtain the difference features. The multi-plane scanning mechanism includes a full scan of height, width, and time. Height-time plane scan Width-Time Plane Scan Full scan via height-width-time Volume anchor point features are obtained by combining state-space model processing. Scanning via height-time plane Width-Time Plane Scan The corresponding planar reference features are obtained by processing them using the state-space model respectively. and ; Calculate the correlation between planar reference features and volumetric anchor point features to generate a motion consistency map. and Based on motion consistency graph and For planar reference features respectively and Weighting is performed, and the volume anchor point features are... Subtracting the weighted planar reference features and To obtain the difference features .
[0009] Furthermore, step 3 specifically includes: Based on the reparameterized three-plane feedforward network for differential features Spatiotemporal texture enhancement is performed by capturing background texture details through parallel convolutional branches in three orthogonal planes: height-width, height-time, and width-time, to obtain enhanced features. Furthermore, during the inference phase, the multi-branch parameters are reparameterized into single-stream convolutions to obtain enhanced features for the inference phase. Then, the enhanced features are processed by the decoder. Reconstruction was performed to obtain a sequence of rainless video frames.
[0010] Furthermore, the reparameterized three-plane feedforward network processes and sums the difference features through parallel convolutional branches in three orthogonal planes: height-width, height-time, and width-time, to obtain the enhanced features. ; Enhanced features The calculation formula is: (2); In formula (2), Indicates the convolution operation; It is a spatial plane convolution kernel used to capture intra-frame texture. It is a height-time plane convolution kernel used to capture the continuity of vertical motion. It is a width-temporal plane convolutional kernel used to capture the dynamics of horizontal scenes; It is an enhanced feature.
[0011] Furthermore, the convolutional kernels corresponding to the three orthogonal planes are dimensionally aligned using a zero-padding operation and then fused into an equivalent 3D convolutional kernel. Fill the anisotropic convolution kernel to The volume shape is used to achieve dimension alignment; equivalent 3D convolution kernels are directly used during inference. For difference features Perform a single convolution computation to obtain enhanced features for the inference stage. ; Equivalent 3D convolution kernel The expression is: (3); Enhanced features of the reasoning stage The expression is: (4); In formulas (3) and (4), This indicates a zero-fill operation.
[0012] Furthermore, a balance coefficient is introduced. Based on pixel-level loss function and perceptual loss function Construct a comprehensive loss function The parameters of the reparameterized three-plane feedforward network in the reconstruction process are optimized by minimizing the comprehensive loss function, where the pixel-level loss function is... Perceptual loss function used to constrain pixel-level reconstruction accuracy Used to constrain semantic feature consistency; Comprehensive loss function The expression is: (5).
[0013] The beneficial effects of this invention are: 1. This invention is based on deep learning methods. It achieves precise decoupling of features by utilizing the motion differences between rain patterns and the background on the spatiotemporal projection plane. It also combines orthogonal three-plane convolution to capture anisotropic spatiotemporal details, effectively solving the problems of attention noise and texture erosion in the state space model. This results in high-quality video reconstruction that balances thorough rain removal with background fidelity.
[0014] 2. Compared with existing methods, this invention significantly breaks the performance constraint between rain removal cleanliness and texture preservation, greatly improves rain removal accuracy while maintaining extremely low inference latency, and significantly improves the model's generalization ability and practical application value in complex dynamic scenarios. Attached Figure Description
[0015] Figure 1 This is a flowchart illustrating a video deraining method based on a spatiotemporal difference state space model. Figure 2-4 Input a sequence of video frames for rainy days to be processed in a video deraining application; Figure 5-7 They are respectively for Figure 2-4 Processed rainless video frames; Figure 8 This is a structural block diagram of a video deraining system based on a spatiotemporal difference state space model. Detailed Implementation
[0016] like Figure 1 As shown, the steps of the video deraining method based on the spatiotemporal difference state space model described in this embodiment include: S1: Extract shallow spatiotemporal features from the input video; like Figure 2-4 As shown, given a rainy day video as input ,in, It is the video frame rate. and These are the height and width, respectively; shallow spatiotemporal features are extracted from the video using 3D convolutional layers, represented as: (1); In formula (1), This represents a 3D convolution operation. These are the extracted shallow spatiotemporal features. It is the feature channel dimension.
[0017] S2: Use multi-plane scanning to construct anchor points and reference features, generate a gated map by calculating geometric consistency, and use a difference mechanism to remove rain ripple information from the features; The consistency-guided differential gating module first generates volumetric anchor point features and planar reference features through multi-plane scanning, specifically including: full-volume scanning of height, width, and time. Volume anchor point features are obtained by combining state-space model processing. Scanning via height-time plane Width-Time Plane Scan The corresponding planar reference features are obtained by processing them using the state-space model respectively. and : (2); (3); (4); In formula (2)-(4),, This indicates a full scan with height-width-time. Indicates height-time plane scan. Indicates width-time plane scan; This represents a bidirectional state-space model processing unit; It is a volume anchor point feature that contains spatiotemporal information. and These are the height-time plane reference features and width-time plane reference features of the motion pattern, respectively.
[0018] Then, a motion consistency map is generated by calculating the correlation between planar reference features and volume anchor point features. and Based on motion consistency graph and For planar reference features respectively and Weighting is performed, and the volume anchor point features are... Subtracting the weighted planar reference features and To obtain the difference features : (5); (6); (7); In formulas (5)-(7), It represents the Hadamah accumulation. It is the Sigmoid activation function. This represents a multilayer perceptron; and These are motion consistency maps on the height-time plane and the width-time plane, respectively, used to indicate the spatial location of rain streaks; It is an output feature that retains background features after gating differential processing.
[0019] S3: During training, orthogonal plane convolution is used to capture local spatiotemporal details, and during inference, reparameterized single-stream convolution is used to enhance the background texture of the video and output rain-free video.
[0020] The reparameterized triplane feedforward network includes parallel spatial, height-time, and width-time branches. During the training phase, anisotropic convolutional kernels are used to process and sum the differential features of the input, as shown below: (8); In formula (2), Indicates the convolution operation; It is a spatial plane convolution kernel used to capture intra-frame texture. It is a height-time plane convolution kernel used to capture the continuity of vertical motion. It is a width-temporal plane convolutional kernel used to capture the dynamics of horizontal scenes; It is an enhanced feature.
[0021] Then, during the inference phase, the convolutional kernels of the three branches are reparameterized and fused into an equivalent 3D convolutional kernel. , represented as: (9); In formula (9), Indicates zero-filling operation; Fill the anisotropic convolution kernel to The volume shape is used to achieve dimension alignment; equivalent 3D convolution kernels are directly used during inference. For difference features Perform a single convolution computation to obtain enhanced features for the inference stage. : (10); Finally, the decoder reconstructs a clean, rain-free video frame sequence from the enhanced features. By minimizing Parameters of the loss L optimization model: (11).
[0022] In formula (11), The Charbonnier loss function is used to constrain pixel-level reconstruction accuracy. Used to constrain semantic feature consistency It is the balance coefficient.
[0023] In addition, such as Figure 8 As shown, this embodiment also proposes a video deraining system based on a spatiotemporal difference state-space model, including: The 3D convolutional layer module is used to perform spatiotemporal encoding on the input rainy weather video and extract shallow spatiotemporal features of the rainy weather video; The consistency-guided differential gating module calculates geometric consistency based on shallow spatiotemporal characteristics and removes dynamic rain streaks through a gating differential mechanism, outputting the differential features after rain removal. A reparameterized triplane feedforward network is used to enhance the spatiotemporal texture of differential features. Parallel convolutional branches capture background texture details to obtain enhanced features. Finally, a decoder reconstructs the enhanced features, outputting a sequence of rain-free video frames, such as... Figure 5-7 As shown.
[0024] To verify the effectiveness of the video deraining method proposed in this invention, a comparison was made with existing video deraining algorithms on three video deraining datasets, as shown below: Data sets and metrics The model was trained and tested on three video deraining datasets: VRDS (72 training videos and 30 test videos), RainVID&SS (208 training videos and 88 test videos), and RainSynAll100 (900 training and test videos). During training, the input rainy video frames were cropped to... The size of the blocks. Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) metrics are used as evaluation indicators during testing.
[0025] Results Analysis We compared our results with three publicly available video deraining datasets: VRDS, RainVID&SS, and RainSynAll100, and with several existing video deraining algorithms, including VDMamba, RainMamba, ViMP-Net, BasicVSR++, DRSformer, and ESINet. Detailed results are shown in Table 1.
[0026] As shown in Table 1, compared with the current state-of-the-art video deraining methods, the method proposed in this application achieves varying degrees of improvement in metrics (PSNR and SSIM) across multiple test datasets. Particularly on the RainSynAll100 dataset, compared to the current state-of-the-art method RainMamba, PSNR is improved by 2.80 dB and SSIM by 0.0127. Results on other datasets also outperform the current state-of-the-art video deraining methods.
[0027] Table 1 compares objective results. In the video deraining task, the method proposed in this application achieves the best results on all publicly available test datasets compared to existing methods.
[0028] Table 1
[0029] This invention proposes an efficient and robust video deraining method for applications such as outdoor video surveillance and autonomous driving. Specifically, it involves a Spatio-Temporal Differential Mamba for Video Deraining framework (STDVD), aiming to break the trade-off between thorough deraining and background fidelity in existing deraining techniques.
[0030] First, a consistency-guided differential gating module is innovatively proposed. It uses a multi-plane scanning mechanism to capture the motion differences between rain patterns and the background in the spatiotemporal manifold. Through gating differential operations, it achieves accurate decoupling of rain pattern artifacts and improves the adaptability and robustness of features to dynamic scenes.
[0031] Secondly, a reparameterized three-plane feedforward network was developed. It captures anisotropic details through parallel convolutional branches in three orthogonal planes: spatial, height-time, and width-time. Combined with structural reparameterization technology, it achieves efficient single-stream computation during the inference stage, thereby outputting high-quality rain-removed videos with clear textures and complete structures, and completely solving the texture erosion problem.
[0032] Experimental results show that STDVD achieves better performance compared to existing methods.
[0033] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent substitutions, and improvements made to the above embodiments without departing from the scope of the present invention, based on the technical essence of the present invention and within the spirit and principles of the present invention, shall still fall within the protection scope of the present invention.
Claims
1. A video deraining method based on a spatiotemporal difference state-space model, characterized in that, include: Step 1: Perform spatiotemporal coding on the given rainy day video and extract its shallow spatiotemporal features; Step 2: Based on the shallow spatiotemporal features, the rain pattern features are removed through consistency-guided differential operations to obtain differential features; Step 3: Perform spatiotemporal texture enhancement and reconstruction on the differential features to output a rainless video frame sequence.
2. The video deraining method based on a spatiotemporal difference state-space model according to claim 1, characterized in that, Step 1 specifically includes: For the input rainy day video ,in, It is the video frame rate. and These are the height and width, respectively. A 3D convolutional layer is used to extract shallow spatiotemporal features from rainy weather videos. ; Shallow spatiotemporal features The extraction formula is: (1); In formula (1), This represents a 3D convolution operation. These are the extracted shallow spatiotemporal features. It is the feature channel dimension.
3. The video deraining method based on a spatiotemporal difference state-space model according to claim 1, characterized in that, In step 2, a multi-plane scanning mechanism is used to construct volumetric anchor features and planar reference features for the shallow spatiotemporal features. Geometric consistency is calculated based on the volumetric anchor features and planar reference features, and dynamic rain streak features in the shallow spatiotemporal features are removed through a gated differential mechanism to obtain differential features. The multi-plane scanning mechanism includes a full scan of height, width, and time. Height-time plane scan Width-Time Plane Scan Full scan via height-width-time Volume anchor point features are obtained by combining state-space model processing. Scanning via height-time plane Width-Time Plane Scan The corresponding planar reference features are obtained by processing them using the state-space model respectively. and ; Calculate the correlation between planar reference features and volumetric anchor point features to generate a motion consistency map. and Based on motion consistency graph and For planar reference features respectively and Weighting is performed, and the volume anchor point features are... Subtracting the weighted planar reference features and To obtain the difference features .
4. The video deraining method based on a spatiotemporal difference state-space model according to claim 1, characterized in that, Step 3 specifically includes: Based on the reparameterized three-plane feedforward network for differential features Spatiotemporal texture enhancement is performed by capturing background texture details through parallel convolutional branches in three orthogonal planes: height-width, height-time, and width-time, to obtain enhanced features. Furthermore, during the inference phase, the multi-branch parameters are reparameterized into single-stream convolutions to obtain enhanced features for the inference phase. Then, the enhanced features are processed by the decoder. Reconstruction was performed to obtain a sequence of rainless video frames.
5. A video deraining method based on a spatiotemporal difference state-space model according to claim 4, characterized in that, The reparameterized three-plane feedforward network processes and sums the difference features through parallel convolutional branches in three orthogonal planes: height-width, height-time, and width-time, to obtain the enhanced features. ; Enhanced features The calculation formula is: (2); In formula (2), Indicates the convolution operation; It is a spatial plane convolution kernel used to capture intra-frame texture. It is a height-time plane convolution kernel used to capture the continuity of vertical motion. It is a width-temporal plane convolutional kernel used to capture the dynamics of horizontal scenes; It is an enhanced feature.
6. The video deraining method based on a spatiotemporal difference state-space model according to claim 4, characterized in that, The convolutional kernels corresponding to the three orthogonal planes are aligned in dimension through a zero-padding operation and then fused into an equivalent 3D convolutional kernel. Fill the anisotropic convolution kernel to The volume shape is used to achieve dimension alignment; equivalent 3D convolution kernels are directly used during inference. For difference features Perform a single convolution computation to obtain enhanced features for the inference stage. ; Equivalent 3D convolution kernel The expression is: (3); Enhanced features of the reasoning stage The expression is: (4); In formulas (3) and (4), This indicates a zero-fill operation.
7. A video deraining method based on a spatiotemporal difference state-space model according to claim 4, characterized in that, In the decoding process, a balance coefficient is introduced. Based on pixel-level loss function and perceptual loss function Construct a comprehensive loss function The parameters of the reparameterized three-plane feedforward network in the reconstruction process are optimized by minimizing the comprehensive loss function, where the pixel-level loss function is... Perceptual loss function used to constrain pixel-level reconstruction accuracy Used to constrain semantic feature consistency; Comprehensive loss function The expression is: (5)。 8. A video deraining system based on a spatiotemporal difference state-space model, applied to the video deraining method based on a spatiotemporal difference state-space model as described in any one of claims 1-7, characterized in that, include: The 3D convolutional layer module is used to perform spatiotemporal encoding on the input rainy weather video and extract shallow spatiotemporal features of the rainy weather video; The consistency-guided differential gating module calculates geometric consistency based on shallow spatiotemporal characteristics and removes dynamic rain streaks through a gating differential mechanism, outputting the differential features after rain removal. A reparameterized three-plane feedforward network is used to enhance the spatiotemporal texture of the differential features. It captures background texture details through parallel convolutional branches to obtain enhanced features. Finally, the enhanced features are reconstructed by the decoder to output a rainless video frame sequence.
