Progressive image rain stripe elimination system, method and equipment
Through the progressive image rain pattern elimination system, the multi-layer attention mechanism and recursive rain pattern elimination module are used, combined with Swin Transformer and Anchored Transformer, the performance impact of computer vision technology under rainfall conditions is solved, achieving high-quality Rain pattern removal and image recovery enhance the global and local feature modeling capabilities of natural images.
Patent Information
- Application Number
- CN202410991663.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The performance of existing computer vision technology is affected under rainfall conditions, making it difficult to efficiently remove rain patterns in images. The existing rain removal technology has shortcomings in maintaining image quality and detail authenticity, especially when processing natural images, it is difficult to take into account the overall situation and Modeling of local features.
Adopt a progressive image rain pattern elimination system, including image input and processing module, feature extraction module, progressive rain pattern elimination module and image fusion module, through a multi-layer attention mechanism and recursive rain pattern elimination module, combined with Swin Transformer and Anchored Transformer, performs multi-stage progressive and jump-through connections, uses a dual-channel multi-dimensional attention unit and SENet module to optimize the similarity between the image generated by the model and the target image.
High-quality rain pattern removal is achieved, which enhances attention to the global context and local detail features of the image, reduces computational complexity, and improves the applicability and effectiveness of the model in various real-world scenarios.
Smart Images

Figure CN118968079A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a system, method and device for removing rain streaks from a progressive image. Background Art
[0002] Computer vision technology has shown rapid development in many fields, especially in facial recognition, object detection for self-driving cars, and outdoor security monitoring applications. However, its performance is significantly affected by complex environmental challenges, such as rainfall conditions. The visual interference caused by raindrops not only obscures key background information, but also causes data loss, seriously affecting the accuracy of intelligent monitoring systems. In this context, there is an urgent need for a solution that can effectively eliminate traces of rain streaks and raindrops in the image while maintaining the computational efficiency and architectural simplicity of the model, and accurately restore the original scene information.
[0003] Although the existing Transformer model is widely praised for its excellent performance, the high computational overhead associated with its core global self-attention mechanism has become a major obstacle to its implementation in resource-constrained devices and environments. On the other hand, although existing image deraining technology has achieved certain results, there is still a lot of room for improvement in ensuring the high quality of restored images and maintaining the authenticity of details. In addition, the inherent complexity of natural images introduces additional challenges: it is necessary to consider global characteristics such as macroscopic color layout, wide texture distribution, and large object contours, and to pay attention to local details such as edge sharpness, subtle color transitions, and fine texture structures, which are indispensable elements for the success of image deraining tasks. Therefore, the algorithm design must not only have the ability to recognize cross-scale features, but also subtly integrate and construct complex interaction models between these multi-level features, so as to promote image restoration to achieve higher fidelity and naturalness. Summary of the invention
[0004] In order to solve the above-mentioned technical problems, the present invention provides a system, method and device for removing rain streaks from a progressive image.
[0005] The technical solution of the present invention is achieved in this way:
[0006] A progressive image rain streak elimination system comprises an image input and processing module, a feature extraction module, a progressive rain streak elimination module and an image fusion module;
[0007] The image input and processing module receives the original image input containing rain streaks and performs necessary preprocessing;
[0008] The feature extraction module extracts basic features from the original image;
[0009] The progressive rain streak removal module is composed of several recursive rain streak removal submodules, which extract and process key information through K-stage progressive and jump connections to achieve stage-by-stage rain streak removal;
[0010] The image fusion module combines the results of shallow and deep feature extraction, generates a high-quality image after rain removal by feature summation, and achieves the final rain removal effect;
[0011] Furthermore, the dual-channel multi-dimensional attention unit includes a three-dimensional attention sub-unit, a parallel transformer and a multi-layer perceptron.
[0012] A method for removing rain streaks from a progressive image comprises the following steps:
[0013] S1. Input an image containing rain streaks, the size of which is H (image height) × W (image width) × 3 (3 channels of RGB image), and preprocess the image;
[0014] S2, shallow feature extraction, output initial feature map x_first;
[0015] S3, through the progressive rain streak removal module, the output x_first of the shallow feature extraction layer is used as input, and key information is further extracted from the shallow feature map and the key information is processed through multi-dimensional attention to perform progressive rain streak removal to generate a new feature map X K ;
[0016] S4, using dual-channel multi-dimensional attention unit to X K Perform deep learning and feature fusion;
[0017] S5, add the feature map processed by the dual-channel multi-dimensional attention unit and x_first obtained by shallow feature extraction element by element to perform feature fusion;
[0018] S6. Further process the fused feature map to generate a feature map that matches the size of the original image, and add the processed feature map to the original rain streak image element by element to generate a high-quality image after rain removal.
[0019] A progressive image rain streak removal device, comprising a computer for carrying a progressive image rain streak removal system, the computer being connected to a camera device for real-time capturing images to be processed, the computer being built with a storage device for storing an operating system, an application, a trained model and an image data set, and a central processor for coordinating and managing the work of other hardware components;
[0020] The computer is also equipped with an operating system and a deep learning framework for providing tools and libraries for building and training deep learning models.
[0021] Compared with the prior art, the present invention has the following beneficial effects:
[0022] 1. The present invention adopts a multi-layer attention mechanism, such as a three-dimensional attention module and a dual-channel multi-dimensional attention unit, to enhance the attention and learning of the global context and local detail features of the image;
[0023] 2. By designing a progressive framework consisting of multiple recursive rain streak removal modules, the key information is gradually extracted and processed through multi-stage progressive and skip connections to achieve high-quality rain streak removal;
[0024] 3. The recursive rain streak elimination module uses a recursive structure connection to achieve weight sharing between attention modules, reducing the dependence on large-scale parameter self-attention modules;
[0025] 4. The present invention combines Swin Transformer and Anchored Transformer to enhance the learning efficiency of global and local features of natural images and feature maps through parallel processing;
[0026] 5. Negative structural similarity that considers brightness, contrast, and structural information is used as the loss function to optimize the similarity between the image generated by the model and the target image;
[0027] 6. Multiple datasets with different rainfall intensities and complexities were used for training and testing to ensure the applicability and effectiveness of the model in handling various real-world scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 This is a system framework diagram of a progressive image rain streak elimination system in Example 1;
[0029] Figure 2 This is a flow chart of a progressive image rain streak elimination method according to Embodiment 2;
[0030] Figure 3 This is a working principle diagram of the progressive image rain streak elimination method of the present invention;
[0031] Figure 4 Schematic diagram of the structure of the recursive rain streak elimination unit in the present invention;
[0032] Figure 5 Schematic diagram of the structure of the dual-channel multi-dimensional attention unit in the present invention. DETAILED DESCRIPTION
[0033] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0034] Example 1
[0035] like Figure 1 As shown, a progressive image rain streak elimination system includes an image input and processing module, a feature extraction module, a progressive rain streak elimination module and an image fusion module;
[0036] The image input and processing module receives the original image input containing rain streaks and performs necessary preprocessing;
[0037] The feature extraction module extracts basic features from the original image;
[0038] The progressive rain streak removal module is composed of several recursive rain streak removal submodules, which extract and process key information through K-stage progressive and jump connections to achieve stage-by-stage rain streak removal;
[0039] The image fusion module combines the results of shallow and deep feature extraction, generates a high-quality image after rain removal by feature summation, and achieves the final rain removal effect;
[0040] Furthermore, the recursive rain streak elimination submodule is composed of a plurality of dual-channel multi-dimensional attention units; the end of the recursive rain streak elimination submodule is connected to a SENet module for reducing the Loss value during training and selectively weakening the weights of invalid or less contributing feature maps;
[0041] Adding the SENet module at the end of the recursive rain streak elimination unit not only helps to reduce the loss value during training, but also selectively weakens the weights of invalid or less contributing feature maps, thereby strengthening the weights of key valid feature maps. The SENet module converts X ′ As the input feature map, the SENet module adopts a fully connected layer structure that first reduces the number of channels and then restores them, which effectively reduces the computational burden. The ReLU activation function is embedded between the two fully connected layers, and the Sigmoid function is applied after the fully connected layer for normalized activation to obtain the learning weights. Finally, the feature map is element-wise multiplied with the learning weights to achieve weighted adjustment of the features of each channel. Finally, the input X r The feature map jumps across layers and performs matrix addition with the output of SENet to obtain the output Xk;
[0042] X k =SE[σ(C k (X r ))]+X r =SE[T4(T3(T2(T1(X r ))))]+X r
[0043] Among them, the feature map X r The feature map X is the input of the module k-1 It is obtained by convolution by adding channels to the original rain map x_first.
[0044] Further, the dual-channel multi-dimensional attention unit includes a three-dimensional attention sub-unit, a parallel transformer and a multi-layer perceptron;
[0045] The dual-channel multi-dimensional attention unit structure is as follows Figure 5 As shown in the figure, the following takes the T=1th dual-channel multi-dimensional attention unit as an example. The feature map is used as the input of this module, and the global context information and local detail features contained in the feature map are learned along the three-dimensional attention sub-unit and the parallel transformer module respectively;
[0046] The three-dimensional attention subunit adopts a unique architecture of two-stage (3×3) convolutional layers in series and a spatially invariant multi-head attention module (SimAm). The two-stage (3×3) convolutional layers in series expand the network depth while maintaining a relatively large receptive field. SimAM uses a three-dimensional attention mechanism (3DAttention Map, 3DAM) to focus on the key information of the size of the feature map after convolution (two dimensions of length and width) and the network depth (one dimension), and selects key areas in space and important network channels in depth to obtain a weighted attention map. More nonlinear transformations are introduced through SimAM to highlight the key weight distribution in space and network channels, which greatly improves the expressiveness of the neural network and the accuracy of key information screening, and realizes deep mining of local features in the feature map. The weighted attention map is then multiplied element by element with the original input feature map, cleverly realizing the fusion in the feature space and compression in the channel dimension, thereby extracting a more refined and discriminative feature representation, providing strong support for subsequent processing steps.
[0047] The final output of the 3D attention subunit is:
[0048] X out =φ(W 3×3 (W 3×3(X″′)))
[0049] Where X″′ is the input feature map of the dual-channel multi-dimensional attention unit, X out is the final output feature map, W 3×3 is the feature fusion 3×3 convolution. φ(·) is the SimAM attention mechanism calculation, which is calculated as:
[0050]
[0051] Where X is the SimAM input feature map, φ(·) is the SimAM attention mechanism calculation, is the Sigmoid activation function, ν(·) is the feature map calculated in each channel and μ is the average value of the feature map calculated in each channel of the input feature map X, and h are the width and height of the input feature map X, λ is a hyperparameter, and ⊙ is the matrix dot multiplication operation.
[0052] The parallel transformer module combines two advanced Transformer variants, the Swin Transformer driven by a moving window attention mechanism and the Anchored Transformer with a stripe attenuation mechanism, aiming to synergistically enhance the learning efficiency of global and local features of natural images and feature maps. Specifically, in this architecture, the input feature map X″′ with 32 channels is first evenly split into two 16-channel feature maps along the channel dimension, and processed in parallel in the attention modules of two parallel independent Transformer variants, and then re-stitched through the channel dimension to integrate feature information. The Swin Transformer module adopts a moving window self-attention mechanism, which cleverly reduces the computational complexity while moving a small window to scan government images to ensure that local features in the image can be accurately captured, such as the shape, size, and texture details of raindrops. The Anchored Transformer innovatively applies a stripe attenuation strategy to alternately inject the horizontal and vertical stripe structures of natural images into the Transformer network, and effectively controls the computational cost of global self-attention while maintaining the ability to model the global range through multi-head attention technology;
[0053] The multi-dimensional attention weight feature map extracted by the three-dimensional attention sub-unit and the parallel transformer module is integrated, and the feature map X″′ is matrix added and superimposed, and further processed by the subsequent multi-layer perceptron (MLP) for nonlinear mapping, so as to achieve deep understanding of image features and high-level abstract expression.
[0054] Example 2
[0055] This embodiment aims to provide a method for removing rain streaks from a progressive image. Figure 2 As shown, the following steps are included:
[0056] S1. Input an image containing rain streaks with a size of H (image height) × W (image width) × 3 (3 channels of RGB image) and preprocess the image; ensure that the input image size and data type remain consistent, and input them into the shallow feature extraction layer and the final matrix addition part respectively;
[0057] S2, shallow feature extraction, output initial feature map x_first;
[0058] The shallow feature extraction layer mainly converts the input RGB image into a feature map processed by the neural network, and propagates the image with 3 input channels forward into a 16-channel feature map x_first through a 3×3 convolution kernel;
[0059] With a 3×3 convolutional layer (such as Figure 5 As shown in FIG. 1 , the feature map of 16 channels output by the recursive rain streak elimination unit is used as the output of the progressive rain streak elimination framework, and a new feature map x′ of 16 channels is generated by convolution mapping;
[0060] In this embodiment, the effectiveness of the proposed progressive lightweight single image deraining model integrating the multi-layer attention mechanism and the Transformer architecture is verified through comparative experiments. In order to comprehensively evaluate the performance of the model, it is tested on Rain100L, Rain100H and other related datasets, and the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used as key indicators to measure the deraining effect, specifically:
[0061] 1. The initial training phase mainly focuses on the Rain100L dataset, which contains 1,800 images for training and 100 images for testing. All background images are selected from the BSD200 dataset to ensure diversity and representativeness of practical applications;
[0062] Among them, Rain800, Rain100L and Rain100H are synthetic datasets that integrate real rainfall. They can be used to simulate rainfall conditions of different degrees and types to test the performance of the model in dealing with complex rain streak effects:
[0063] Rain100L: This dataset contains 1,800 training images and 100 test images. The background images are selected from the BSD200 dataset. Each image consists of light rain streaks in one direction.
[0064] Rain100H: Similar to Rain100L, this dataset consists of 1800 images and 100 test images. The background is also taken from BSD200. Each image can be composed of rainstorm streaks in multiple directions, which improves the offline performance.
[0065] Rain200L and Rain200H contain 1800 synthetic rain images for training and 200 synthetic rain images for testing;
[0066] By using the above series of datasets with different rainfall intensities and complexities for training and testing, we can more comprehensively evaluate the applicability and effectiveness of the deraining method in various real-world scenarios;
[0067] 2. In terms of optimization strategy, the AdamW optimizer was selected to achieve a stable and fast learning process. The batch size (batch_size) was set to 5, which means that the data information of 5 images will be considered when updating the model parameters each time. The initial learning rate was set to 0.001. During the entire training cycle, the model went through 100 epochs to fully converge and obtain the optimal weight parameters.
[0068] 3. By comparing the performance of the LDR model under the above settings with the current most advanced deraining methods on different data sets, we can comprehensively evaluate the advantages of the LDR model from the aspects of quantitative indicators (such as PSNR and SSIM) and computing resource consumption (such as the number of required parameters and computing cost), and further prove that it effectively reduces the model complexity while maintaining high performance, and is suitable for device deployment environments with limited memory and computing power;
[0069] During the training process of the entire deep learning network, in order to improve the training speed and effectively reduce the demand for video memory resources, we converted the original training data set into the efficient H5 (Hierarchical Data Formatversion 5) format. The H5 format is favored because it supports efficient storage and access of large-scale data sets. It can significantly improve data loading efficiency. Compared with other common data formats, H5 can compress data to save storage space and has good I / O performance;
[0070] In addition, H5 format data sets can be easily read and processed in parallel, which is particularly important for multi-threaded or distributed training environments because it can make full use of GPU resources to accelerate data preprocessing, model training, and inference processes. By optimizing the data format and loading process, it ensures that complex deep learning models can be trained as efficiently as possible under limited video memory conditions.
[0071] The similarity of an image is composed of three parts: luminance, contrast, and structure. Assume that the output high-quality image after rain removal is x and its corresponding clear image y. Therefore, this model uses negative structural similarity as the loss function, which comprehensively considers the brightness, contrast, and structural information of the image to optimize the similarity between the image generated by the model and the target image.
[0072] brightness:
[0073]
[0074] Contrast ratio:
[0075]
[0076] structure:
[0077]
[0078] where μ x and μ y are the means of x and y, σ x and σ y are the variances of x and y, σ xy is the covariance between x and y; multiplying the above three and taking the negative value, the error can be expressed as:
[0079] SSIM(x,y)=[L(x,y)·C(x,y)·S(x,y)]
[0080] Right now
[0081]
[0082] S3, through the progressive rain streak removal module, the output x_first of the shallow feature extraction layer is used as input, and key information is further extracted from the shallow feature map and the key information is processed through multi-dimensional attention to perform progressive rain streak removal to generate a new feature map X K ;
[0083] S4, using dual-channel multi-dimensional attention unit to X K Perform deep learning and feature fusion;
[0084] Preferably, the progressive rain streak elimination module (such as Figure 3 As shown in FIG. 1 , the method is composed of K=6 recursive rain streak removal units, which are connected in a multi-stage progressive and skip manner, and use the output of the shallow feature extraction layer as input. The method further extracts key information from the shallow feature map and focuses on and processes the key information through multi-dimensional attention to perform progressive rain streak removal.
[0085] Among them, K = 6 recursive rain streak elimination units are the core of the whole model. Each unit is as follows Figure 4 As shown, the image X output by the previous recursive rain streak elimination unit is k-1 The concatenation of the original rain image x_first is used as the input of the next recursive rain streak removal unit. At the end of the last recursive rain streak removal unit, a 3×3 convolutional layer is used as the output of the deep feature extraction layer.
[0086] Define X K is the feature map generated by the K-th recursive rain streak elimination unit, x_first is the initial input feature map, and the feature map output generated by the K-th recursive rain streak elimination unit is:
[0087]
[0088] In the formula, C K is the 3×3 convolutional layer in the Kth recursive rain streak removal unit, and σ is the attention mechanism calculation of the Kth recursive rain streak removal unit;
[0089] Taking the K=2nd recursive rain streak elimination unit as an example, its structure is as follows: Figure 4 As shown in the figure, a stage-by-stage recursive calculation strategy is adopted to connect four dual-channel multi-dimensional attention units in series. The entire process ends with the squeeze-and-excitation (SENet) module. The original rain image x_first is concatenated with the image X1 output by the K=1th recursive rain streak elimination unit in the channel direction to obtain a 32-channel feature map. The 3×3 convolution layer is responsible for compressing the 32 input feature maps into 16 new feature maps, while keeping the 16 network channels and the spatial size (width and height) of the feature map unchanged. This process aims to achieve the layer-by-layer accumulation and fusion of subsequent feature information. The four dual-channel multi-dimensional attention units focus on key feature information in multiple dimensions while effectively controlling the amount of model parameters, and obtain a weighted attention map.
[0090] S5, add the feature map processed by the dual-channel multi-dimensional attention unit and x_first obtained by shallow feature extraction element by element to perform feature fusion;
[0091] S6. Further process the fused feature map to generate a feature map that matches the size of the original image, and add the processed feature map to the original rain streak image element by element to generate a high-quality image after rain removal.
[0092] The image fusion output layer mainly performs element-by-element addition operation on the output x′ of the deep feature extraction layer and the output x_first of the shallow feature extraction layer, and keeps the feature map as 16 channels. In the image fusion output layer, a 3×3 convolution layer is used to convert the 16-channel feature map into a 3-channel RGB image O, and an element-by-element addition operation is performed with the original image B containing rain streak information to remove the rain streak information from the original rain image to achieve the image deraining effect I, which can be expressed by the following formula:
[0093] I=B+O
[0094] I is the image after removing rain streak information, B is the original input image containing rain streak information, and O is the feature map after the image deraining network.
[0095] In summary, in order to solve the modeling problem of deep learning that it is difficult to take into account both global and local features when processing natural images, in-depth analysis and design were carried out from the two levels of modularization and overall architecture. Dual-channel multi-dimensional attention units were adopted in the module to achieve the learning of global and local features in a small range. On the whole, a progressive approach was adopted to improve the learning of global features. After connecting 4 dual-channel multi-dimensional attention units in series, the SE module was added to enhance the learning of local features.
[0096] This embodiment provides a progressive image rain streak removal method, which takes an image containing rain streak information as input, processes the image, and outputs a high-quality image with rain streak information removed. Specifically, the method includes three parts: (1) shallow feature extraction. A simple (3×3) convolutional layer is used to extract features and map the input RGB image to a 16-channel feature map; (2) progressive rain streak removal module: Based on the shallow features, through K-stage progressive and jump connections, the internal rain streak removal module adopts a recursive structure connection, takes the output of the shallow feature extraction layer as input, further extracts key information from the shallow feature map, and uses multi-dimensional attention to focus on and process the key information to perform progressive rain streak removal; (3) image fusion output layer: the image fusion output layer is based on the previously obtained shallow features and ultra-deep features, and performs feature summation with the original rainy day image to generate a high-quality image after rain removal, removes the rain streak information from the original rain image, and finally achieves the elimination of rain streaks in stages, improving the image rain removal effect.
[0097] Example 3
[0098] This embodiment aims to provide a progressive image rain streak removal device, comprising a computer for carrying a progressive image rain streak removal system, the computer being connected to a camera device for real-time capturing images to be processed, the computer being built with a storage device for storing an operating system, an application, a trained model and an image data set, and a central processor for coordinating and managing the work of other hardware components;
[0099] The computer is also equipped with an operating system and a deep learning framework for providing tools and libraries for building and training deep learning models.
Claims
1. A progressive image rain streak removal system, characterized by: It includes image input and processing module, feature extraction module, progressive rain streak elimination module and image fusion module; The image input and processing module receives the original image input containing rain streaks and performs necessary preprocessing; The feature extraction module extracts basic features from the original image; The progressive rain streak removal module is composed of several recursive rain streak removal submodules, which extract and process key information through K-stage progressive and jump connections to achieve stage-by-stage rain streak removal; The image fusion module combines the results of shallow and deep feature extraction, generates a high-quality image after deraining by feature summation, and achieves the final deraining effect.
2. The progressive image rain streak removal system according to claim 1, characterized in that: The recursive rain streak elimination submodule is composed of a plurality of dual-channel multi-dimensional attention units; the end of the recursive rain streak elimination submodule is connected to a SENet module for reducing the Loss value during training and selectively weakening the weights of invalid or less contributing feature maps.
3. The progressive image rain streak removal system according to claim 2, characterized in that: The dual-channel multi-dimensional attention unit includes a three-dimensional attention sub-unit, a parallel transformer and a multi-layer perceptron.
4. The progressive image rain streak removal system according to claim 2, characterized in that: The SENet module is added at the end of the recursive rain streak elimination unit. The SENet module takes X′ as the input feature map. A fully connected layer structure is adopted inside the SENet module to first reduce the number of channels and then restore them. A ReLU activation function is embedded between the two fully connected layers, and a Sigmoid function is applied after the fully connected layer for normalized activation. The input X r The feature map jumps across layers and performs matrix addition with the output of SENet to obtain the output Xk; X k =SE[σ(C k (X r ))]+X r =SE[T4(T3(T2(T1(X r ))))]+X r Among them, the feature map X r The feature map X is the input of the module k-1 It is obtained by convolution by adding channels to the original rain map x_first.
5. The progressive image rain streak removal system according to claim 3, characterized in that: The three-dimensional attention subunit adopts a unique architecture of two-stage (3×3) convolutional layers in series and a spatially invariant multi-head attention module. The two-stage (3×3) convolutional layers in series expand the network depth while maintaining a relatively large receptive field. SimAM uses a three-dimensional attention mechanism to focus on the size of the convolutional feature map and the network depth. It selects key areas in space and important network channels in depth to obtain the weighted attention map 2. The final output of the three-dimensional attention subunit is: X out =φ(w 3×3 (w 3×3 (X″′))) Where X″′ is the input feature map of the dual-channel multi-dimensional attention unit, X out is the final output feature map, W 3×3 is the feature fusion 3×3 convolution, and φ(·) is the SimAM attention mechanism calculation, which is calculated as: Where X is the SimAM input feature map, φ(·) is the SimAM attention mechanism calculation, is the Sigmoid activation function, v(·) is the feature map calculated in each channel and μ is the average value of the feature map calculated in each channel of the input feature map X. and h are the width and height of the input feature map X, λ is a hyperparameter, and ⊙ is the matrix dot multiplication operation; The parallel transformer module combines two Transformer variants: SwinTransformer driven by a moving window attention mechanism and Anchored Transformer with a stripe attenuation mechanism.
6. A progressive image rain streak removal method, characterized by: The following steps are involved: S1. Input an image containing rain streaks, the size of which is H (image height) × W (image width) × 3 (3 channels of RGB image), and preprocess the image; S2, shallow feature extraction, output initial feature map x_first; S3, through the progressive rain streak removal module, the output x_first of the shallow feature extraction layer is used as input, and key information is further extracted from the shallow feature map and the key information is processed through multi-dimensional attention to perform progressive rain streak removal to generate a new feature map X K ; S4, using dual-channel multi-dimensional attention unit to X K Perform deep learning and feature fusion; S5, add the feature map processed by the dual-channel multi-dimensional attention unit and x_first obtained by shallow feature extraction element by element to perform feature fusion; S6. Further process the fused feature map to generate a feature map that matches the size of the original image, and add the processed feature map to the original rain streak image element by element to generate a high-quality image after rain removal.
7. The method for removing rain streaks from a progressive image according to claim 6, characterized in that: The S2 shallow feature extraction layer mainly converts the input RGB image into a feature map processed by the neural network, and propagates the image with 3 input channels forward into a 16-channel feature map x_first through a 3×3 convolution kernel; A 3×3 convolutional layer is used as the output of the progressive rain streak removal framework, and the 16-channel feature map output by the recursive rain streak removal unit is convoluted to generate a new 16-channel feature map x′; The image fusion output layer mainly performs element-by-element addition operation on the output x of the deep feature extraction layer and the output x_first of the shallow feature extraction layer, and keeps the feature map as 16 channels. In the image fusion output layer, a 3×3 convolution layer is used to convert the 16-channel feature map into a 3-channel RGB image 0, and an element-by-element addition operation is performed with the original image B containing rain streak information to remove the rain streak information from the original rain map to achieve the image deraining effect I, which can be expressed by the following formula: I=B+0 I is the image after removing rain streak information, B is the original input image containing rain streak information, and 0 is the feature map after the image deraining network.
8. The method for removing rain streaks from a progressive image according to claim 6, characterized in that: The progressive rain streak removal module is composed of K=6 recursive rain streak removal units, which are connected in a multi-stage progressive and skip manner, take the output of the shallow feature extraction layer as input, further extract key information from the shallow feature map and focus on and process the key information through multi-dimensional attention, and perform progressive rain streak removal; Among them, K = 6 recursive rain streak removal units are the core of the whole model. The image X output by the previous recursive rain streak removal unit is k-1 The concatenation of the original rain image x_first is used as the input of the next recursive rain streak removal unit. At the end of the last recursive rain streak removal unit, a 3×3 convolutional layer is used as the output of the deep feature extraction layer. Define X K is the feature map generated by the K-th recursive rain streak elimination unit, x_first is the initial input feature map, and the feature map output generated by the K-th recursive rain streak elimination unit is: In the formula, C K is the 3×3 convolutional layer in the Kth recursive rain streak removal unit, and σ is the attention mechanism calculation of the Kth recursive rain streak removal unit.
9. The method for removing rain streaks from a progressive image according to claim 8, characterized in that: The recursive rain streak elimination unit adopts a stage-by-stage internal recursive calculation strategy, and connects four dual-channel multi-dimensional attention units in series. The entire process ends with a squeeze-and-excitation module. The original rain image x_first is spliced with the image X1 output by the K=1th recursive rain streak elimination unit in the channel direction to obtain a 32-channel feature map. The 3×3 convolutional layer is responsible for compressing the input 32 feature maps into 16 new feature maps. While maintaining 16 network channels and keeping the spatial size of the feature map unchanged, it achieves focus on key feature information in multiple dimensions and obtains a weighted attention map.
10. A progressive image rain streak removal device, characterized in that: The computer includes a computer for carrying a progressive image rain streak removal system, the computer is connected to a camera device for real-time capturing images to be processed, and the computer is built with a storage device for storing an operating system, an application, a trained model and an image data set, and a central processor for coordinating and managing the work of other hardware components; The computer is also equipped with an operating system and a deep learning framework for providing tools and libraries for building and training deep learning models.
Citation Information
Patent Citations
Dual-channel single image fine rain removal method
CN110728640A
Single-image rain removal method and system based on deep convolutional neural network, and storage medium
CN113450288A
Gesture recognition method based on lightweight convolutional neural network
CN113792635A
Point cloud target detection method, device and equipment based on multi-scale feature fusion
CN116152622A
Cited By
Multi-scale Transform image rain removal method based on residual channel prior guidance
CN119624811A
