Video rain removal architecture and method based on hierarchical feature cascade purification and frame difference long and short term memory neural network
Through the combination of the hierarchical feature cascade purification module, the time difference perception module and the video quality improvement module, the problem of insufficient time information during video rain removal in the prior art is solved, and efficient rain line removal and video quality improvement are achieved.
Patent Information
- Application Number
- CN202510505388.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-22
- Publication Date
- 2025-08-08
AI Technical Summary
The existing video rain removal technology lacks time information and lacks generalization capabilities when processing continuous frames, resulting in poor rain removal effect.
The combination of hierarchical feature cascade purification module, time difference perception module and video quality improvement module is adopted to capture the spatial and temporal characteristics of rain lines to improve video quality through multi-scale feature extraction, frame difference analysis and depth feature recovery.
It effectively captures the high-frequency changes of rain lines, significantly improves rain removal effect and video quality, and is suitable as a pre-task for object detection. When processing NTUrain data sets, the average image processing time per frame is 0.3 seconds, and the rain removal rate reaches 0.978.
Smart Images
Figure CN120451008A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a video deraining architecture and method based on hierarchical feature cascade purification and frame difference long short-term memory neural network. Background Art
[0002] CNNs have made breakthrough progress in the field of computer vision and have become one of the core technologies in this field. Its main applications include:
[0003] Image Classification: This is one of the most mature applications of CNNs. From the early application of LeNet-5 in handwritten digit recognition (MNIST) to AlexNet's victory in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC), CNNs have become the model of choice for image classification tasks. Today, CNN-based image classification models are widely used in various scenarios, such as security monitoring, medical image analysis, and product recognition.
[0004] When shooting videos on rainy days, raindrops form on the camera lens, blurring the video and reducing brightness, seriously affecting the viewing experience and subsequent processing. Therefore, video deraining technology has emerged to eliminate raindrops and the effect of rain in videos, restoring clear video images.
[0005] Video rain removal can be broadly categorized into three categories: time-domain-based methods, frequency-domain-based methods, and methods based on prior knowledge. Tripathi et al. developed a correlation model that captures the dynamics of rain and a physics-based motion blur model that accounts for rain photometry. Barnum et al. combined physical and statistical models to derive a physical model of raindrops and snowflakes to determine the general shape and brightness of individual streaks. This streak model was combined with the statistical properties of rain and snow to determine how they affect the spatiotemporal frequency of image sequences. Santhaseelan et al. proposed a rain detection and removal algorithm based on phase congruency features. This algorithm detects rain streaks in a specific frame using phase congruency features. Pixels identified as rain streaks in the frame are then replaced with information from their spatial and temporal neighbors unaffected by rain. When combined with phase correlation, this method can remove moderately dense rain from videos even with complex camera motion. However, this method can produce some false detections of rain streaks when the camera is in motion. Based on this, Santhaseelan et al. proposed a new video rain removal framework consisting of two parts: rain streak detection and scene reconstruction. The rain track detection part uses phase congruence features to detect rain tracks and reduces false detections through chromaticity constraints. The scene reconstruction part adopts a new reconstruction technique that uses the intensity, spatial neighborhood, and temporal neighborhood information of rain track pixels to estimate the actual intensity of rain track pixels by minimizing the inter-frame registration error.
[0006] With the advancement of deep learning technology, Liu et al. constructed a joint periodic rain removal and reconstruction network (J4R-Net), which seamlessly integrates rain degradation classification, rain removal based on spatial texture appearance, and background detail reconstruction based on temporal coherence. Rain degradation classification provides a binary map that reveals whether a location is degraded by linear additive streaks or occlusions. Chen et al. proposed a SPAC-CNN algorithm based on superpixel alignment and CNN detail compensation. This algorithm segments the scene into depth-consistent units, addressing content shift caused by scene and camera motion. It also performs content alignment at the superpixel level to more accurately locate rain traces and restore occluded background content. Yang et al. improved the rain model and applied recurrent neural networks and generative adversarial networks to effectively improve the performance of rain removal for multi-frame video sequences, making video rain removal more accurate and efficient. Yan et al. proposed a video-based rain removal method with feature-level alignment. The proposed method avoids the use of optical flow by using deformable convolutional layers in the encoder. They then proposed a minority loss that combines depth, transmission map, and camera pose to handle rain accumulation. The use of depth and camera pose enables the method to handle ambiguity issues of depth and water droplets, thus improving the results.
[0007] Although existing methods have achieved good results in video deraining, they lack temporal information for extracting consecutive frames and have insufficient generalization capabilities. Summary of the Invention
[0008] The purpose of the present invention is to provide a video deraining architecture and method based on hierarchical feature cascade purification and frame difference long short-term memory neural network to solve the above technical problems.
[0009] To solve the above technical problems, the specific technical solutions of the video deraining architecture and method based on hierarchical feature cascade purification and frame difference long short-term memory neural network of the present invention are as follows:
[0010] A video deraining architecture based on hierarchical feature cascade purification and frame difference long short-term memory neural network includes a hierarchical feature cascade purification module, a temporal difference perception module and a video quality improvement module. The hierarchical feature cascade purification module adopts an encoder-decoder structure and captures the spatial characteristics of rain lines through multi-scale feature extraction and cascade fusion. This module converts the input rainy day video sequence into a feature map and integrates feature information of different scales through multi-level feature fusion technology to generate preliminary deraining results. A bidirectional processing mechanism is designed to simultaneously consider forward and backward temporal information. The temporal difference perception module uses a temporal displacement mechanism to capture temporal difference information in the video sequence and realizes the interaction of inter-frame information by performing partial feature displacement in the channel dimension. The video quality improvement module receives the outputs of the hierarchical feature cascade purification module and the temporal difference perception module through a residual dense block structure and short-circuit connection. Through deep feature extraction and refined processing, it restores the detailed texture destroyed by rain lines while maintaining the temporal consistency of the scene.
[0011] A video deraining method based on hierarchical feature cascade purification and frame difference long short-term memory neural network includes the following steps:
[0012] Step 1: Data processing and loading;
[0013] Step 2: Data enhancement and transformation;
[0014] Step 3: Data loader configuration;
[0015] Step 4: Enter the hierarchical feature cascade purification module;
[0016] Step 5: Enter the time difference perception module;
[0017] Step 6: Enter the video quality improvement module.
[0018] Furthermore, the step 1 includes the following steps:
[0019] First, load continuous video frames from the file path and use the Image.open of the PIL library to read the file to form a frame sequence list;
[0020] A sliding window sequence is generated according to the window_size parameter. Each window contains continuous time segments to ensure the temporal correlation of the input model. The window sliding step is 1 to generate time series data. During the test phase, mirror frames are added at the beginning and end through padding to maintain temporal continuity. The frame sequence is stacked into a tensor and the dimension order is adjusted to (C, T, H, W) through permute(1,0,2,3) to explicitly separate the channel, time and space dimensions.
[0021] Furthermore, the step 2 includes the following steps:
[0022] Random cropping: The cropping area is randomly selected during the training phase, and all frames in the same window share the same cropping position to maintain spatiotemporal consistency;
[0023] Flip and Rotation: Apply horizontal / vertical flip and random rotation to enhance data diversity;
[0024] Standardization and tensor transformation: Normalization is achieved through externally passed transform parameters.
[0025] Furthermore, the step 3 includes the following steps:
[0026] The dataset is encapsulated using `DataLoader`, which supports batch loading, multithreading, and memory page locking. During the training phase, shuffle is enabled to disrupt the data order to prevent model overfitting. Video timing information is explicitly modeled through time window partitioning. A shared parameter enhancement strategy is used to ensure the consistency of spatiotemporal transformations within the same window. The final output is a tensor of dimension (C, T, H, W), which adapts to the input requirements of video processing models.
[0027] Furthermore, step 4 includes the following steps:
[0028] After the data is loaded, it first enters the hierarchical feature cascade purification module, which is responsible for extracting multi-scale features from the rainy video sequence and performing preliminary rain line removal. This module combines multi-feature fusion technology to effectively capture the spatial feature distribution of rain lines. The input format is a four-dimensional tensor. First, the pre-trained resnet18 is used to extract spatial features, and after each extraction, the downsampling operation is performed. The resolution is changed to 1 / 2 of the previous one each time until the lowest 1 / 16, and four different feature maps are obtained. Among them, block iRepresents the feature map of the i-th layer. As i increases, the spatial resolution of the feature map gradually decreases, while the semantic information gradually becomes richer. After that, the feature map with reduced resolution is resampled to the initial resolution. Each upsampling operation adopts bilinear interpolation upsampling with a magnification of 2.
[0029] After multi-scale splicing of block1-block4, it passes through an enhancement module. Through multi-layer processing, the enhancement module can selectively enhance the features related to raindrops while suppressing irrelevant or noise features. It integrates and refines features at different levels to provide high-quality feature representation for subsequent modules; finally, the upsampled tensors are fused.
[0030] Form the output and input into the next module.
[0031] Furthermore, the step 5 includes the following steps:
[0032] The temporal difference perception module is implemented based on a convolutional long short-term memory network and a frame difference analysis mechanism. It identifies the temporal characteristics of rain lines by analyzing the differences between adjacent frames. The frame difference analysis mechanism is designed to capture the high-frequency temporal changes of rain lines:
[0033] F t Represents the feature map of the current frame, F t -F t-1 Represents the differential features between the current frame and the previous frame, F t -F t-2 Represents the differential features between the current frame and the previous two frames. These three features are concatenated and replaced with the short-term memory features in the initial version of CNN-LSTM to update the cell state.
[0034] Furthermore, the step 6 includes the following steps:
[0035] The video quality enhancement module further improves the visual quality of the video through the TSM and residual dense block RRDB structure. This module receives the output of the hierarchical feature cascade purification module and the temporal difference perception module, and restores the detailed texture destroyed by rain lines through deep feature extraction and refined processing while maintaining the temporal consistency of the scene.
[0036] The video deraining architecture and method based on hierarchical feature cascade purification and frame difference long short-term memory neural network of the present invention have the following advantages:
[0037] 1. The first proposed Frame Difference Long Short-Term Memory (LSTM) network uses the frame difference information between the previous two frames and the current frame, replacing the short-term information in traditional LSTM networks. This facilitates capturing the high-frequency variations in raindrops between adjacent frames. This frame difference information helps identify areas temporarily obscured by raindrops, maintaining sensitivity to fast-moving raindrops even in slowly changing scenes.
[0038] 2. A video quality enhancement module (VQEM) is introduced, which further improves the visual quality of derained videos and reduces artifacts and blur through meticulous feature refinement and enhancement.
[0039] 3. It is suitable as a prerequisite for object detection. When processing videos with a resolution of 481*321 on the NTUrain dataset, the average processing time per frame is approximately 0.3 seconds, and the rain removal rate reaches 0.978. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 It is the overall framework diagram of the model of the present invention;
[0041] Figure 2 This is a framework diagram of the hierarchical feature cascade purification module of the present invention;
[0042] Figure 3 This is a framework diagram of the time difference perception module of the present invention;
[0043] Figure 4 This is a framework diagram of the video quality improvement module of the present invention. DETAILED DESCRIPTION
[0044] In order to better understand the purpose, structure and function of the present invention, the video deraining architecture and method based on hierarchical feature cascade purification and frame difference long short-term memory neural network of the present invention are further described in detail below with reference to the accompanying drawings.
[0045] like Figure 1 As shown in the figure, the video deraining architecture based on hierarchical feature cascade purification and frame difference long short-term memory neural network of the present invention includes a hierarchical feature cascade purification module (Hierarchical Feature Cascade Purification, HFCP), a temporal discrepancy aware model (TDAM) and a video quality enhancement module (Video Quality Elevation Model, VQEM).
[0046] First, the hierarchical feature cascade purification module employs an encoder-decoder architecture to effectively capture the spatial characteristics of rain lines through multi-scale feature extraction and cascade fusion. This module converts the input rainy video sequence into a feature map and integrates feature information at different scales through multi-level feature fusion (MFF) technology to generate preliminary rain removal results. Notably, we designed a bidirectional processing mechanism that simultaneously considers forward and backward temporal information, significantly improving our ability to understand rain line patterns across consecutive frames.
[0047] Secondly, the Temporal Shift Mechanism (TSM) module efficiently captures temporal differences in video sequences using a temporal shift mechanism. Unlike traditional 3D convolutional or recurrent neural network approaches, the TSM module shifts some features along the channel dimension, enabling inter-frame information interaction while maintaining computational efficiency. This design enables the network to identify the temporal consistency of rain lines and effectively distinguish between static scene content and dynamic rain line patterns.
[0048] Finally, the video quality enhancement module further improves the visual quality of the video through a residual dense block (RRDB) structure and short-circuit connections. This module receives the outputs of the hierarchical feature cascade purification module and the temporal difference perception module. Through deep feature extraction and refinement, it restores the texture details destroyed by rain lines while maintaining the temporal consistency of the scene. Our design specifically focuses on improving the quality of the center frame, ensuring the clarity and detail preservation of key frames.
[0049] The video deraining method based on hierarchical feature cascade purification and frame difference long short-term memory neural network of the present invention comprises the following steps:
[0050] Step 1: Data processing and loading
[0051] First, load continuous video frames from the file path and use the Image.open of the PIL library to read the file to form a frame sequence list.
[0052] A sliding window sequence is generated based on the window_size parameter (default is 5). (The sliding window is the number of video frames fed into the neural network at once. Loading the entire video at once would consume a large amount of GPU memory.) Each window contains a continuous time segment, ensuring temporal correlation of the input model. The window sliding step is 1, generating time series data. During the test phase, mirrored frames are added at the beginning and end through padding to maintain temporal continuity. The frame sequence is stacked into a tensor, and the dimension order is adjusted to (C, T, H, W) using permute(1,0,2,3), explicitly separating the channel, time, and space dimensions.
[0053] Step 2: Data augmentation and transformation
[0054] In order to improve sample diversity, data augmentation operations are performed.
[0055] Random cropping: The cropping area is randomly selected during the training phase, and all frames in the same window share the same cropping position to maintain spatiotemporal consistency.
[0056] Flip and Rotation: Apply horizontal / vertical flip (probability 0.5) and random rotation (0°, 90°, 180°, 270°) to enhance data diversity.
[0057] Standardization and tensor transformation: Normalization is achieved through externally passed transform parameters.
[0058] Step 3: Data loader configuration
[0059] The dataset is encapsulated using DataLoader, supporting batch loading (batch_size), multithreading (num_workers), and memory page locking (pin_memory). Shuffle is enabled during training to disrupt the data order and prevent model overfitting. The core of this implementation is to explicitly model video timing information through time windowing. A shared parameter enhancement strategy ensures the consistency of spatiotemporal transformations within the same window. The final output is a tensor of (C, T, H, W) dimensions, which meets the input requirements of video processing models.
[0060] Step 4: Enter the hierarchical feature cascade purification module
[0061] After the data is loaded, it first enters the hierarchical feature cascade purification module (such as Figure 2 As shown in Figure 2, it is mainly responsible for extracting multi-scale features from rainy day video sequences and performing preliminary rain line removal. This module combines multi-feature fusion technology to effectively capture the spatial feature distribution of rain lines. The input format is a four-dimensional tensor. First, the pre-trained resnet18 is used to extract spatial features. After each extraction, the downsampling operation is performed, and the resolution is reduced to 1 / 2 each time until the lowest 1 / 16, resulting in four different feature maps. Among them, block i represents the feature map of layer i. As i increases, the spatial resolution of the feature map decreases while the semantic information becomes richer. This multi-scale feature extraction strategy enables the network to simultaneously capture local details and global structural information of rain lines. The feature map with reduced resolution is then upsampled back to the initial resolution. Each upsampling operation uses bilinear interpolation with a magnification factor of 2.
[0062] After multi-scale splicing of blocks 1-4, they pass through an enhancement module. Through multi-layer processing, this module can selectively enhance features related to raindrops while suppressing irrelevant or noisy features. It integrates and refines features at different levels, providing high-quality feature representation for subsequent modules, thereby improving the overall rain removal effect.
[0063] Finally, the upsampled tensors are fused to form the output input to the next module.
[0064] Step 5: Enter the time difference perception module
[0065] The Temporal Difference Awareness Module (TDAM) is the second key component of this model (e.g. Figure 3 The Temporal Difference Perception Module (shown in Figure 2) is specifically designed to capture the temporal differences between adjacent frames in a video sequence, which is crucial for distinguishing dynamic rain lines from static backgrounds. The Temporal Difference Perception Module, based on a Convolutional Long Short-Term Memory (CNN-LSTM) network and a frame difference analysis mechanism, provides an effective approach for modeling temporal dependencies in videos.
[0066] The core idea of the temporal difference perception module is to identify the temporal characteristics of rain lines by analyzing the differences between adjacent frames. Compared with the static background, rain lines usually show high-frequency temporal changes. We designed a frame difference analysis mechanism to capture this change: t Represents the feature map of the current frame, F t -F t-1 Represents the differential features between the current frame and the previous frame, F t -F t-2 Represents the differential features between the current frame and the previous two frames. These three features are concatenated and used to update the cell state instead of the short-term memory features in the initial version of CNN-LSTM.
[0067] Step 6: Enter the video quality improvement module
[0068] The first two modules have been able to achieve most of the rain removal effect, but through experiments, we know that some fine rain streaks are still not processed enough. Finally, the video quality improvement module combines TSM with RRDB structure (such as Figure 4 ), further improving the visual quality of the video. TSM is a lightweight temporal information modeling method, commonly used in video-related tasks. It performs "temporal shifting" on feature channels, allowing the network to capture dynamic information in the temporal dimension while extracting spatial features. RRDB is an advanced feature extraction module that was first used in super-resolution networks. It improves the feature expression ability and network stability through multiple residual connections and dense connections. This module receives the output of the hierarchical feature cascade purification module and the temporal difference perception module, and through deep feature extraction and refined processing, it restores the detailed texture destroyed by the rain line while maintaining the temporal consistency of the scene. Our design pays special attention to improving the quality of the center frame to ensure the clarity and detail retention of the key frames.
[0069] This module splices the initial input image data and the TDAM output to ensure that the video quality improvement module can refer to the actual content of the original frame when processing the video, avoiding excessive rain removal that may cause image distortion or loss of details. It obtains the final rain-free image and converts the continuous frame image into a video output.
[0070] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.
Claims
1. A video deraining architecture based on hierarchical feature cascade purification and frame difference long short-term memory neural network, characterized by: It includes a hierarchical feature cascade purification module, a temporal difference perception module and a video quality improvement module. The hierarchical feature cascade purification module adopts an encoder-decoder structure and captures the spatial characteristics of rain lines through multi-scale feature extraction and cascade fusion. The module converts the input rainy day video sequence into a feature map and integrates feature information of different scales through multi-level feature fusion technology to generate preliminary deraining results. A bidirectional processing mechanism is designed, and forward and backward time series information is considered at the same time. The temporal difference perception module uses a temporal displacement mechanism to capture the temporal difference information in the video sequence and realizes the interaction of inter-frame information by performing partial feature displacement in the channel dimension. The video quality improvement module receives the outputs of the hierarchical feature cascade purification module and the temporal difference perception module through a residual dense block structure and short-circuit connection. Through deep feature extraction and refined processing, the module restores the detailed texture destroyed by rain lines while maintaining the temporal consistency of the scene.
2. A video deraining method based on hierarchical feature cascade purification and frame difference long short-term memory neural network, characterized by: The steps include: Step 1: Data processing and loading; Step 2: Data enhancement and transformation; Step 3: Data loader configuration; Step 4: Enter the hierarchical feature cascade purification module; Step 5: Enter the time difference perception module; Step 6: Enter the video quality improvement module.
3. The video deraining method based on hierarchical feature cascade purification and frame difference long short-term memory neural network according to claim 2 is characterized in that: The step 1 comprises the following steps: First, load continuous video frames from the file path and use the Image.open of the PIL library to read the file to form a frame sequence list; A sliding window sequence is generated according to the window_size parameter. Each window contains continuous time segments to ensure the temporal correlation of the input model. The window sliding step is 1 to generate time series data. During the test phase, mirror frames are added at the beginning and end through padding to maintain temporal continuity. The frame sequence is stacked into a tensor and the dimension order is adjusted to (C, T, H, W) through permute(1, 0, 2, 3) to explicitly separate the channel, time and space dimensions.
4. The video deraining method based on hierarchical feature cascade purification and frame difference long short-term memory neural network according to claim 2 is characterized in that: The step 2 comprises the following steps: Random cropping: The cropping area is randomly selected during the training phase, and all frames in the same window share the same cropping position to maintain spatiotemporal consistency; Flip and Rotation: Apply horizontal / vertical flip and random rotation to enhance data diversity; Standardization and tensor transformation: Normalization is achieved through externally passed transform parameters.
5. The video deraining method based on hierarchical feature cascade purification and frame difference long short-term memory neural network according to claim 2 is characterized in that: The step 3 comprises the following steps: The dataset is encapsulated using `DataLoader`, which supports batch loading, multithreading, and memory page locking. During the training phase, shuffle is enabled to disrupt the data order to prevent model overfitting. Video timing information is explicitly modeled through time window partitioning. A shared parameter enhancement strategy is used to ensure the consistency of spatiotemporal transformations within the same window. The final output is a tensor of dimension (C, T, H, W), which adapts to the input requirements of video processing models.
6. The video deraining method based on hierarchical feature cascade purification and frame difference long short-term memory neural network according to claim 2 is characterized in that: The step 4 comprises the following steps: After the data is loaded, it first enters the hierarchical feature cascade purification module, which is responsible for extracting multi-scale features from the rainy video sequence and performing preliminary rain line removal. This module combines multi-feature fusion technology to effectively capture the spatial feature distribution of rain lines. The input format is a four-dimensional tensor. First, the pre-trained resnet18 is used to extract spatial features, and after each extraction, the downsampling operation is performed. The resolution is changed to 1 / 2 of the previous one each time until the lowest 1 / 16, and four different feature maps are obtained. Among them, block i Represents the feature map of the i-th layer. As i increases, the spatial resolution of the feature map gradually decreases, while the semantic information gradually becomes richer. After that, the feature map with reduced resolution is resampled to the initial resolution. Each upsampling operation adopts bilinear interpolation upsampling with a magnification of 2. After multi-scale splicing of blocks 1-4, the model passes through an enhancement module. Through multi-layer processing, the enhancement module can selectively enhance features related to raindrops while suppressing irrelevant or noisy features. It integrates and refines features at different levels to provide high-quality feature representation for subsequent modules. Finally, the upsampled tensors are fused to form the output, which is input to the next module.
7. The video deraining method based on hierarchical feature cascade purification and frame difference long short-term memory neural network according to claim 2 is characterized in that: The step 5 comprises the following steps: The temporal difference perception module is implemented based on the convolutional long short-term memory network and the frame difference analysis mechanism. It identifies the temporal characteristics of rain lines by analyzing the differences between adjacent frames. The frame difference analysis mechanism is designed to capture the high-frequency temporal changes of rain lines: t Represents the feature map of the current frame, F t -F t-1 Represents the differential features between the current frame and the previous frame, F t -F t-2 Represents the differential features between the current frame and the previous two frames. These three features are concatenated and replaced with the short-term memory features in the initial version of CNN-LSTM to update the cell state.
8. The video deraining method based on hierarchical feature cascade purification and frame difference long short-term memory neural network according to claim 2 is characterized in that: The step 6 comprises the following steps: The video quality enhancement module further improves the visual quality of the video by combining TSM with RRDB structure. This module receives the output of the hierarchical feature cascade purification module and the temporal difference perception module. Through deep feature extraction and refined processing, it restores the detailed texture destroyed by rain lines while maintaining the temporal consistency of the scene.