Image rain removal method and device based on enhanced prior and mixed attention

Abstract

The embodiment of the specification provides a rain removal method and device for an image based on enhanced priori and mixed attention, wherein the method comprises the following steps: performing color channel analysis on a feature map of an input image, performing first filtering on the feature map by using a Gaussian filtering kernel, performing second filtering on the combined feature map by using a median filtering kernel, performing edge extraction on the feature map processed by the second filtering by using a Sobel edge extraction operator, performing weighted average fusion on the feature map processed by the second filtering and the edge extraction, inputting the feature map to calculate channel attention and spatial attention, splicing the channel attention and the spatial attention by processing through a multi-head attention module, performing channel rearrangement on the processed input feature map, and obtaining a final output.

Description

Technical Field

[0001] This document relates to the field of computer technology, and in particular to an image deraining method and apparatus based on enhanced prior and hybrid attention. Background Technology

[0002] In today's intelligent era, applications combined with artificial intelligence are developing rapidly, with image processing technology being particularly prominent. Whether it's image denoising, image recognition, or image style transfer, all are taking root and flourishing on the backbone of computer vision. However, images obtained in reality are often unsuitable for image processing. Image defects, blurriness, and noise can all affect image quality. In many real-world scenarios, weather conditions remain a significant challenge. When capturing or observing images, raindrops can be a common source of interference, leading to blurriness, loss of detail, and even inaccurate target identification. Therefore, the research and development of image deraining technology is of great significance for improving image quality and enhancing the accuracy of image analysis and recognition.

[0003] In the field of image deraining, researchers have been exploring various algorithms and methods to improve image quality. However, with the continuous development of technology, traditional image restoration methods, while ensuring image quality, often come at the cost of increased system complexity. Therefore, it is necessary to propose a new method to balance the relationship between image restoration quality and system complexity.

[0004] Neural network-based methods offer novel approaches to image deraining and other image restoration tasks. By constructing multiple or multi-layered neural networks and employing encoder-decoder hierarchical structures, contextual information and spatial details are progressively recovered from images, resulting in potentially high-quality images. Furthermore, some networks utilize adaptive feature fusion mechanisms and selective multi-head attention mechanisms to achieve information exchange and preserve key information. Through this tightly interconnected hierarchical structure, this method demonstrates significant performance improvements across various image restoration tasks.

[0005] Research on image deraining technology is not only significant for improving the user experience of image browsing under different weather conditions, but also supports the performance enhancement of visual perception systems such as drones and robots, thereby improving the accuracy of image analysis and object recognition. Meanwhile, the introduction of hybrid hierarchical network structures has brought new ideas to the field of image restoration, providing a useful reference for solving the problem of balancing quality and complexity in image restoration. Future research can further explore and refine this method to address ever-changing challenges and promote the development and application of image restoration technology.

[0006] Existing research both domestically and internationally mainly focuses on prior features, feature extraction modules, training paths, and the VisionTransformers (ViT) architecture.

[0007] 1. Based on prior features.

[0008] This research primarily focuses on improving the prior features provided during each iteration of training for the target rain map, aiming to further enhance the training effect in each round. In the rain map, rain, as a background layer, influences the main content of the image. Physically, the difference between the channel values ​​of the rain map and the background map is more significant, and prior features can effectively guide the neural network to learn this difference. Several studies have addressed this issue. For example, Yu Li et al. used a Gaussian Mixture Model (GMM) to represent the image background as a mixture of multiple Gaussian distributions and used special encoding between neural network layers to estimate and optimize the GMM parameters. On the other hand, Yu Luo et al. employed discriminative sparse coding, learning the sparse representation of the data by minimizing reconstruction error and maximizing inter-class differences, thereby minimizing the reconstruction error of the background and maximizing the inter-class differences. Furthermore, Shuhang Gu et al. proposed joint convolutional analysis and sparse representation coding, combining coding with convolutional networks, achieving good results for rain maps.

[0009] 2. Feature-based capture modules.

[0010] By designing different feature extraction modules, image deraining networks can effectively extract features from rain images. These feature extraction modules mainly employ image deraining models represented by Convolutional Neural Networks (CNNs). For example, research has shown that rain streaks are learned by establishing a mapping relationship between rain images and rain-free images. Complex neural networks are used to establish a correspondence between rain streaks and actual images, and finally, the deraining effect is achieved by subtracting rain streak features. This type of method has created many innovative deraining networks. Among them, the most representative is the Deep Convolutional Neural Network (DCNN), which achieves an end-to-end mapping from rain streaks to clear images. Recurrent compression-excited residual networks introduce residual networks to handle the deraining task, progressively optimizing the residual network and using different masking methods as priors to provide information such as rain streak density. In addition, there are networks such as dilated residual dense models and rain convolutional dictionary models.

[0011] 3. Based on different training paths.

[0012] This research significantly improves the performance of most computer vision tasks, including image inpainting, by refining the underlying model architecture, designing different training paths, and fusing features at different training stages. During neural network training, different feedback and fusion mechanisms transform the originally linear feedback neural network into a more powerful form. For example, by learning residuals between stages, dense connections between channels, and concatenating layers, parallel paths within a multi-stage framework can be achieved. Gating mechanisms, which fuse training features from different paths and those before and after a single path, further enhance the performance of the final training results.

[0013] 4. Based on the Vision Transformers (ViT) architecture.

[0014] Vision Transformers (ViT), when pre-trained with sufficient data, can achieve better results than convolutional neural networks. Its main idea is to decompose an image into several feature vectors of unequal size but equal dimensions. Similar model designs include hybrid multilayer perceptron models, generative multilayer perceptron models, gated feedforward networks, and Fourier neural networks. These models simplify the global operations for each feature block. Furthermore, ViT also has specific network schemes for processing high-quality images.

[0015] Existing network architectures such as Figure 1 As shown. The network mainly adopts a multi-level parallel structure, connected in series with multiple sub-modules. Residual neural network channels are connected in parallel with the next level through channel transformation, while multiple residual neural networks at the same level are connected in series. The structure of each residual network channel is as follows. Figure 2 As shown.

[0016] Each residual neural network channel consists of three key modules: a prior module, a feature extraction module, and a feature fusion module. This network architecture presents several problems: First, the quality of the prior feature images is often low. In previous studies, prior features were typically generated by deep convolutional neural networks or by analyzing the color contrast characteristics of rain and no-rain images. However, the prior images generated by these methods are easily affected by noise and the image's own structure, resulting in poor prior quality and ineffective feature extraction, which is detrimental to subsequent feature processing and fusion. Specifically, there are the following three shortcomings:

[0017] 1. Noise impact:

[0018] Methods for generating prior feature images are susceptible to noise interference. For example, prior images generated using deep convolutional neural networks, while capable of extracting some important features, also introduce noise accumulated during network training.

[0019] 2. Insufficient color contrast:

[0020] The quality of a prior image generated by analyzing the color contrast characteristics of rain and no-rain images depends on the accuracy of the color information and the degree of contrast. In complex scenes, the color contrast between rain and no-rain images may be insignificant, resulting in a low-quality prior image.

[0021] 3. Limitations of prior feature generation methods:

[0022] Many traditional methods, which use simple statistical analysis or experience-based rules to generate prior feature images, fail to fully capture the details and complex structures of the images.

[0023] Secondly, feature extraction modules often employ channel attention mechanisms to extract object features obscured by rain streaks. However, a single channel attention mechanism often neglects the global receptive field, meaning it has limited ability to capture the overall contextual information of the image. This limitation negatively impacts image deraining, as effective deraining requires a comprehensive understanding of both rain streaks and object features within the image. Specifically, there are three main shortcomings:

[0024] 1. Strong locality:

[0025] Channel attention mechanisms primarily focus on weight allocation for different feature channels, processing each channel independently. This approach lacks integration of global image information, causing the network to rely solely on local information when processing rain streaks, potentially ignoring the overall structure and pattern of the rain streaks. Consequently, rain streaks in the image cannot be completely and effectively removed, and residual rain streaks may still be clearly visible in the de-rained image.

[0026] 2. Insufficient understanding of context:

[0027] Channel attention mechanisms have limited ability to capture overall contextual information of an image, and can only perform feature enhancement within a single channel. Lacking an understanding of the global image context, the network struggles to recognize the global relationship between rain streaks and object features. The de-raining effect is unstable, potentially leading to incomplete object features in the image, or misidentifying object features as rain streaks and removing them.

[0028] 3. Insufficient retention of details:

[0029] Single-channel attention mechanisms may enhance certain feature channels while suppressing detail in other channels. Insufficient detail preservation can lead to the neglect or loss of important image details. De-raining images may appear blurry with poor detail, impacting visual quality and accuracy. Summary of the Invention

[0030] The purpose of this invention is to provide an image deraining method and apparatus based on enhanced prior and hybrid attention, aiming to solve the above-mentioned problems in the prior art.

[0031] This invention provides an image deraining method based on enhanced prior and hybrid attention, comprising:

[0032] Color channel analysis is performed on the feature map of the input image to calculate the color difference of each channel and obtain the contrast information of the image in the color dimension. Based on the magnitude of the contrast information, the variation range of each channel is determined, and the significant regions in the input image are identified. A Gaussian filter kernel is used to perform the first filtering on the feature map. The first filtered feature map is weighted and merged with the original feature map to obtain the merged feature map. A median filter kernel is used to perform the second filtering on the merged feature map. The Sobel edge extraction operator is used to extract edges from the feature map after the second filtering to obtain the bright edges and gradient information in the input image.

[0033] Based on the salient regions, bright edges, and gradient information in the input image, the feature map after secondary filtering and edge extraction is weighted and averaged to generate the final prior image, i.e., the input feature map.

[0034] A portion of the input feature map is processed by a global average pooling layer to extract the information required for channel attention. The input feature map after global average pooling is then processed by a fully connected layer, and pointwise convolution and depthwise separable convolution are used to enhance the expressive power of the channel features. Channel attention is calculated through an activation function. Another portion of the input feature map is processed by group normalization, and spatial attention is calculated through a sliding window attention module based on a series of linear transformations and a sigmoid activation function.

[0035] The multi-head attention module concatenates channel attention and spatial attention, and then rearranges the channels of the processed input feature map to obtain the final output.

[0036] This invention provides an image deraining device based on enhanced prior and hybrid attention, comprising:

[0037] The filtering and extraction module is used to perform color channel analysis on the feature map of the input image, calculate the color difference of each channel, obtain the contrast information of the image in the color dimension, determine the variation range of each channel based on the magnitude of the contrast information, and identify the salient regions in the input image. A Gaussian filter kernel is used to perform the first filtering on the feature map. The first-filtered feature map is then weighted and merged with the original feature map to obtain a merged feature map. A median filter kernel is used to perform a second filtering on the merged feature map. Finally, the Sobel edge extraction operator is used to extract edges from the feature map after the second filtering process to obtain bright edges and gradient information in the input image.

[0038] The weighted average fusion module is used to perform weighted average fusion on the feature map after secondary filtering and edge extraction based on the salient regions, bright edges and gradient information in the input image, to generate the final prior image, i.e., the input feature map.

[0039] The attention calculation module is used to process a portion of the input feature map through a global average pooling layer to extract the information required for channel attention. It then processes the input feature map after global average pooling through a fully connected layer and enhances the expressive power of the channel features through pointwise convolution and depthwise separable convolution. Channel attention is calculated using an activation function. The module also performs group normalization on another portion of the input feature map and calculates spatial attention using a sliding window attention module based on a series of linear transformations and a sigmoid activation function.

[0040] The concatenation and rearrangement module is used to concatenate channel attention and spatial attention through the multi-head attention module, and rearrange the channels of the processed input feature map to obtain the final output.

[0041] This invention also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the above-described image deraining method based on enhanced prior and hybrid attention.

[0042] This invention also provides a computer-readable storage medium storing an information transmission implementation program, which, when executed by a processor, implements the steps of the above-described image deraining method based on enhanced priors and hybrid attention.

[0043] By employing embodiments of the present invention, optimizations are made from two aspects: prior information and attention mechanism, which significantly improves the image deraining effect, especially in terms of peak signal-to-noise ratio (PSNR). Attached Figure Description

[0044] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of the network architecture for image deraining in existing technologies;

[0046] Figure 2 This is a schematic diagram of each residual network channel structure in the prior art;

[0047] Figure 3 This is a flowchart of an image deraining method based on enhanced prior and hybrid attention according to an embodiment of the present invention;

[0048] Figure 4 This is a schematic diagram illustrating the process of acquiring a priori images according to an embodiment of the present invention;

[0049] Figure 5 This is a schematic diagram of the SCMA hybrid attention enhancement mechanism according to an embodiment of the present invention;

[0050] Figure 6 This is a schematic diagram comparing the extreme value processing of the synthetic rain map channel and the hybrid filtering enhancement processing in an embodiment of the present invention;

[0051] Figure 7 This is a schematic diagram comparing the synthesized rain map, the baseline model, the optimized model, and the original image according to an embodiment of the present invention;

[0052] Figure 8 This is a schematic diagram comparing the image deraining effects of an embodiment of the present invention;

[0053] Figure 9 This is a schematic diagram comparing the details of rain removal in an embodiment of the present invention;

[0054] Figure 10 This is a schematic diagram of an image deraining device based on enhanced prior and hybrid attention according to an embodiment of the present invention;

[0055] Figure 11 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0056] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0057] Method Implementation Examples

[0058] According to embodiments of the present invention, an image deraining method based on enhanced prior and hybrid attention is provided. Figure 3 This is a flowchart of an image deraining method based on enhanced prior and hybrid attention according to an embodiment of the present invention, as follows: Figure 3 As shown, the image deraining method based on enhanced prior and hybrid attention according to an embodiment of the present invention specifically includes:

[0059] Step S301: Perform color channel analysis on the feature map of the input image, calculate the color difference of each channel, obtain the contrast information of the image in the color dimension, determine the variation range of each channel based on the magnitude of the contrast information, and identify the significant regions in the input image; use a Gaussian filter kernel to perform a first filtering on the feature map, weight the first filtered feature map and the original feature map and merge them to obtain a merged feature map, use a median filter kernel to perform a second filtering on the merged feature map; use the Sobel edge extraction operator to extract edges from the feature map after the second filtering to obtain bright edges and gradient information in the input image; wherein, the size of the Gaussian filter kernel is 5×5, the standard deviation of the first filtering is 0.5, and the size of the median filter kernel is 3×3.

[0060] Step S302: Based on the salient regions, bright edges and gradient information in the input image, the feature map after secondary filtering and edge extraction is weighted and averaged to generate the final prior image, i.e., the input feature map.

[0061] Step S303: A portion of the input feature map is processed by a global average pooling layer to extract the information required for channel attention. The input feature map after global average pooling is then processed by a fully connected layer, and pointwise convolution and depthwise separable convolution are used to enhance the expressive power of the channel features. Channel attention is calculated through an activation function. Another portion of the input feature map is processed by group normalization, and spatial attention is calculated through a sliding window attention module based on a series of linear transformations and a sigmoid activation function.

[0062] Specifically, the spatial attention calculation through the sliding window attention module includes: compressing the length and width dimensions of the input feature map into one dimension, and performing segmentation and connection along the same channel at this scale. That is, performing fully connected processing in different spatial regions within the same channel to finally obtain spatial attention.

[0063] Step S304: The channel attention and spatial attention are concatenated by the multi-head attention module, and the processed input feature map is rearranged by channel to obtain the final output.

[0064] In step S304, the processed input feature map is rearranged through channels to obtain the final output, specifically including:

[0065] The input feature map [B,C,H,W] is divided into several groups based on the number of channels. The input feature map is then rearranged, and a transpose operation is performed between the second and third dimensions to exchange the group order with the channel order, resulting in a tensor of shape [B,C / groups,groups,H,W]. This tensor is then flattened, and the channels within each group are connected together to obtain another tensor of shape [B,C,H,W]. This is the final output, where B represents the number of feature maps in each batch of images, C represents the number of channels in the feature image, H represents the height of the feature image, and W represents the width of the feature image.

[0066] The technical solutions of the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0067] Step 1: Color Channel Residual Analysis

[0068] 1. Residual Mechanism Analysis:

[0069] The color channels of the feature map are analyzed to calculate the color differences between each channel, thereby obtaining the contrast information of the image in the color dimension. The range of variation for each channel is determined based on the magnitude of the color differences; a larger difference value indicates a larger range of variation in that channel, thus identifying salient regions in the image.

[0070] Step 2: Primary Gaussian Filtering

[0071] 2. Gaussian Filtering Application: The image is first filtered using a 5×5 Gaussian filter kernel, with a standard deviation set to 0.5 to control the filter strength. This filtering process smooths the image, reduces noise interference, and preserves important features of the original image.

[0072] 3. Weighted processing of filtered images: The filtered image is weighted and merged with the original image to preserve the original object features while enhancing the texture features of the image.

[0073] Step 3: Secondary median filtering

[0074] 4. Median Filtering Application: A 3×3 median filter is used to perform a second filtering on the image, further refining the removal of noise remaining after Gaussian blur, especially fine line noise. Median filtering ensures detail preservation while enhancing image clarity.

[0075] Filter Selection: Noise removal is crucial in enhancing prior images. This invention employs various filters to denoise prior images, with Gaussian filtering and median filtering being two primary methods. The following is a detailed comparative analysis of these two filters to guide the scientific and effective denoising process.

[0076] Gaussian filtering:

[0077] Gaussian filtering reduces noise and blurs the image by weighting the neighborhood around each pixel in the rain map.

[0078] Advantages: Gaussian filtering can better preserve the overall structure and details of an image, and it is relatively fast in calculation.

[0079] Disadvantages: Although Gaussian filtering may slightly blur some image details, it can effectively smooth out noise generated by channel extremum processing of feature maps.

[0080] Median filtering:

[0081] Principle: Median filtering replaces the pixel values ​​of the surrounding neighborhood pixels with the median value of the pixel, thereby achieving image denoising.

[0082] Advantages: Median filtering can effectively remove salt-and-pepper noise and other types of noise from images without over-filtering image details.

[0083] Weighted combination effect: When applying mean filtering after Gaussian filtering with a large convolution kernel, it can effectively improve the areas where the weighted averaging effect is not good in Gaussian filtering and improve the feature granularity.

[0084] Step 4: Edge Feature Extraction

[0085] 5. Edge Extraction Operation: The Sobel edge extraction operator is used to extract edges from the filtered image to extract bright edges and gradient information. Edge extraction enhances the detail of the image, making it clearer.

[0086] Specifically, the advantages of Sobel edge extraction are: The Sobel operator utilizes a larger convolutional kernel, resulting in a wider receptive field, which better captures image details and features, producing more accurate and stable edge extraction results. For noisy images, the Sobel operator better preserves edge width, improving the distinction between features and noise during subsequent filtering and reducing the risk of feature loss. In contrast, the Sobel operator exhibits more complex and comprehensive characteristics in edge detection. It uses two convolutional kernels, one along the horizontal direction (x-direction) and the other along the vertical direction (y-direction), to calculate the edge gradient of the image. This design allows the Sobel operator to have a larger receptive field, covering a wider image region and better capturing image details and features. When using the Sobel operator, different channel combinations can be selected according to actual needs, making the operator more flexible in adapting to different image characteristics. The Sobel operator is more powerful than the Robert operator and can better handle various image variations and challenges. Due to its larger receptive field, the Sobel operator can capture a wider edge region, and the main edges of objects in the generated feature map are more prominent. Even when facing noisy images, the Sobel operator can effectively detect and enhance subtle edges. For images with a large number of raindrops, the Sobel operator can distinguish noise from the main features while maintaining good performance.

[0087] Step 5: Weighted average fusion.

[0088] 6. Feature Map Fusion: The feature maps, after multiple filtering and edge extraction processes, are weighted and averaged to generate the final prior image. The fusion process comprehensively utilizes the information obtained from each processing step to improve the quality and detail representation of the prior image.

[0089] The above processing steps can be referred to Figure 4 Understanding. After processing through the above steps, the prior image significantly reduces noise and improves image quality while basically preserving texture. The image deraining algorithm of this invention combines multiple techniques such as Gaussian filtering, median filtering, and Sobel edge extraction to achieve the goals of noise removal, preservation of image details, and enhancement of image features.

[0090] Step Six: SCMA-based attention enhancement mechanisms, such as... Figure 5 As shown, specifically:

[0091] 1. Global Average Pooling: The input feature map x is processed through a global average pooling layer to extract the information needed for channel attention from the input feature map.

[0092] 2. Channel Attention Calculation: The pooled feature map is processed by a fully connected layer, and channel attention is calculated using an activation function. The weights of the channel attention are calculated using the weight parameter cweight and the bias cbias, and then applied to the input feature map.

[0093] 3. Pointwise convolution and depthwise separable convolution: The input feature map is processed by pointwise convolution and depthwise separable convolution to enhance the expressive power of channel features and obtain the output of channel attention.

[0094] 4. Spatial Attention Calculation: Another part of the input feature map is processed separately to obtain the spatial attention input. This part of the feature map is normalized and then spatial attention is calculated through a series of linear transformations and the Sigmoid activation function. Spatial attention is calculated using a sliding window attention module. Specifically, the image's width and height dimensions are compressed into one dimension, and segmentation and connection are performed along the same channel at this scale. This is equivalent to performing fully connected processing on different spatial regions within the same channel, ultimately yielding the spatial attention output.

[0095] 5. Concatenation and Multi-head Attention: The channel attention output and spatial attention output are concatenated. After processing by the multi-head attention module, the feature relationships between global and local features are captured.

[0096] 6. Channel Rearrangement: The processed feature map is rearranged by channel to better preserve the relationship between spatial and channel dimensions, resulting in the final output. The detailed steps of channel rearrangement are as follows:

[0097] Channel rearrangement involves dividing the input feature map [B,C,H,W] into several groups based on the number of channels (CC), and then readjusting the feature map. A transpose operation is performed between the second and third dimensions, exchanging the group order with the channel order, resulting in a tensor of shape [B,C / groups,groups,H,W]. This tensor is then flattened, and the channels within each group are concatenated, resulting in another tensor of shape [B,C,H,W], ensuring sufficient feature extraction. These detailed steps aim to ensure effective extraction and utilization of channel and spatial information from the input feature map, enhancing the overall performance of the model.

[0098] In one example, this embodiment of the invention is deployed on a Tesla T4 server, using one GPU and a 4-core CPU for model pre-training. To balance training efficiency and server capacity, the model size is adjusted to 2, which is the superposition of two rain removal main networks. The training run consists of 250 epochs, representing the number of times the dataset will be traversed during training. Each training epoch is divided into 8 batches, with each image patch being a 64×64 square image, containing 32 feature maps at a time. The number of residual blocks is set to 3, and the loss functions used are mean squared error (MSE) and structural similarity index (SSIM). The initial learning rate is set to 0.0005, and the results are expressed using mean squared error.

[0099] The technical solutions described in the embodiments of the present invention will be verified below.

[0100] The Rain200H dataset consists of 200 test rain images and 1800 training rain images. The synthetic rain patterns in these images were added from the rain patterns in the Rain100H dataset. The Rain100H dataset was proposed by W. Yang et al. in 2017

[46] and contains 100 pairs of images, each pair consisting of one image with rain and one image without rain. The synthetic rain patterns cover rain patterns in 5 different directions.

[0101] Validation of the advantages of hybrid filtering to enhance prior modules.

[0102] The prior information after hybrid filtering and prior algorithm processing is shown below. Figure 6 As shown, compared to the prior of the original channel extremum processing algorithm, the black noise left by rain streaks is significantly removed, while the object contours in the image are basically preserved. Therefore, this method based on weighted hybrid filtering can effectively repair damaged images and improve image quality.

[0103] Validating the advantages of combining prior knowledge with hybrid channel fusion attention.

[0104] Since channel extremum processing and SE attention mechanism are relatively stable and classic in image deraining algorithms, the prior module of the baseline model is set to channel extremum processing and the attention mechanism module is set to SE attention mechanism, which is used to improve the model for comparison.

[0105] The improved model was tested by selecting an improved prior module, hybrid filtering to enhance the prior, and comparing it with the improved attention mechanism and SCMA. The results showed that... Figure 7The results are shown in the image below. Comparing the results, after deraining, the improved model shows clearer details in the wrinkles of the black bear's fur compared to the baseline model, indicating improved image quality. Although some detail differences are minor, they can still be quantified by the Peak Signal-to-Noise Ratio (PSNR), as shown in Table 1. Through the adopted prior processing method and improved attention mechanism, the model achieves a higher PSNR score, indicating that it can better recover the details and structure of the image, making it clearer, more realistic, and more detailed. This demonstrates that our improved module has achieved certain results in improving the quality of derained images.

[0106] Table 1 Comparison between optimization modules

[0107]

[0108]

[0109] In addition, for the sake of experimental rigor, the different effects of different attention mechanisms under the prior condition of channel extremum processing were also investigated. That is, the baseline model with only the attention mechanism changed is shown in Table 2. The results show that the SCMA attention mechanism achieves a better score than other attention mechanisms under the same conditions.

[0110] Table 2 Comparison of different attention mechanisms

[0111]

[0112] The image above is obtained after deraining using the improved method. Observation reveals that the result not only achieves deraining but also maintains very clear texture details. Figure 8 Furthermore, it can be observed that the unprocessed image has coarser details compared to the image after model processing. The image after model processing has finer details and richer textures, as shown in [the image description]. Figure 9 This demonstrates the effectiveness of the improved image deraining network in this embodiment of the invention.

[0113] In summary, this invention proposes an image enhancement method that combines different filter characteristics and contour extraction operators, specifically for the prior processing of rain removal networks. This algorithm effectively removes rain streaks from prior images, significantly improving the quality of the prior image and thus enhancing the overall performance of the rain removal network.

[0114] This invention proposes a novel attention mechanism, SCMA, which integrates channel attention and spatial attention. Compared with existing attention mechanisms (such as CBAM, Axial, ECA, SE, etc.), it achieves a higher peak signal-to-noise ratio (PSNR) in image deraining tasks, demonstrating better deraining effects and improved image clarity.

[0115] Device Example 1

[0116] According to embodiments of the present invention, an image deraining device based on enhanced prior and hybrid attention is provided. Figure 10 This is a schematic diagram of an image deraining device based on enhanced prior and hybrid attention according to an embodiment of the present invention, as shown below. Figure 10 As shown, the image deraining device based on enhanced prior and hybrid attention according to an embodiment of the present invention specifically includes:

[0117] The filtering and extraction module 100 is used to perform color channel analysis on the feature map of the input image, calculate the color difference of each channel, obtain the contrast information of the image in the color dimension, determine the variation range of each channel based on the magnitude of the contrast information, and identify the significant regions in the input image; a Gaussian filter kernel is used to perform a first filtering on the feature map, the first filtered feature map is weighted and merged with the original feature map to obtain a merged feature map, and a median filter kernel is used to perform a second filtering on the merged feature map; the Sobel edge extraction operator is used to extract edges from the feature map after the second filtering to obtain bright edges and gradient information in the input image; the size of the Gaussian filter kernel is 5×5, the standard deviation of the first filtering is 0.5, and the size of the median filter kernel is 3×3.

[0118] The weighted average fusion module 102 is used to perform weighted average fusion on the feature map after secondary filtering and edge extraction based on the salient regions, bright edges and gradient information in the input image, to generate the final prior image, i.e., the input feature map.

[0119] The attention calculation module 104 is used to process a portion of the input feature map through a global average pooling layer to extract the information required for channel attention, process the input feature map after global average pooling through a fully connected layer, and enhance the expressive power of channel features through pointwise convolution and depthwise separable convolution, and calculate channel attention through an activation function; and to process another portion of the input feature map through group normalization, and calculate spatial attention through a sliding window attention module based on a series of linear transformations and a sigmoid activation function; the attention calculation module 104 is specifically used for:

[0120] The length and width dimensions of the input feature map are compressed into one dimension, and segmentation and connection are performed along the same channel at this scale. That is, full connection processing is performed in different spatial regions within the same channel, ultimately resulting in spatial attention.

[0121] The concatenation and rearrangement module 106 is used to concatenate channel attention and spatial attention through multi-head attention module processing, and to rearrange the channels of the processed input feature map to obtain the final output. Specifically, the concatenation and rearrangement module 106 is used for:

[0122] The input feature map [B,C,H,W] is divided into several groups based on the number of channels. The input feature map is then rearranged, and a transpose operation is performed between the second and third dimensions to exchange the group order with the channel order, resulting in a tensor of shape [B,C / groups,groups,H,W]. This tensor is then flattened, and the channels within each group are connected together to obtain another tensor of shape [B,C,H,W]. This is the final output, where B represents the number of feature maps in each batch of images, C represents the number of channels in the feature image, H represents the height of the feature image, and W represents the width of the feature image.

[0123] The embodiments of the present invention are device embodiments corresponding to the above method embodiments. The specific operation of each module can be understood with reference to the description of the method embodiments, and will not be repeated here.

[0124] Device Example 2

[0125] This invention provides an electronic device, such as... Figure 11 As shown, it includes: a memory 110, a processor 112, and a computer program stored in the memory 110 and executable on the processor 112, wherein the computer program, when executed by the processor 112, performs the steps as described in the method embodiment.

[0126] Device Example 3

[0127] This invention provides a computer-readable storage medium storing an information transmission implementation program, which, when executed by a processor 112, implements the steps described in the method embodiment.

[0128] The computer-readable storage media described in this embodiment include, but are not limited to, ROM, RAM, disk, or optical disk.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image deraining method based on enhanced prior and hybrid attention, characterized in that, include: Color channel analysis is performed on the feature map of the input image to calculate the color difference of each channel, obtain the contrast information of the image in the color dimension, determine the range of change of each channel based on the magnitude of the contrast information, and identify the significant regions in the input image. The feature map is first filtered using a Gaussian filter kernel. The first filtered feature map is then weighted and merged with the original feature map to obtain a merged feature map. The merged feature map is then filtered a second time using a median filter kernel. The Sobel edge extraction operator is then used to extract edges from the feature map after the second filtering process to obtain bright edges and gradient information in the input image. Based on the salient regions, bright edges, and gradient information in the input image, the feature map after secondary filtering and edge extraction is weighted and averaged to generate the final prior image, i.e., the input feature map. A portion of the input feature map is processed by a global average pooling layer to extract the information required for channel attention. The input feature map after global average pooling is then processed by a fully connected layer, and pointwise convolution and depthwise separable convolution are used to enhance the expressive power of the channel features. Channel attention is calculated through an activation function. Another portion of the input feature map is processed by group normalization, and spatial attention is calculated through a sliding window attention module based on a series of linear transformations and a sigmoid activation function. The multi-head attention module concatenates channel attention and spatial attention, and then rearranges the channels of the processed input feature map to obtain the final output.

2. The method according to claim 1, characterized in that, The Gaussian filter kernel has a size of 5×5, the standard deviation of the first filter is 0.5, and the median filter kernel has a size of 3×3.

3. The method according to claim 1, characterized in that, Calculating spatial attention using the sliding window attention module specifically includes: The length and width dimensions of the input feature map are compressed into one dimension, and segmentation and connection are performed along the same channel at this scale. That is, full connection processing is performed in different spatial regions within the same channel, ultimately resulting in spatial attention.

4. The method according to claim 1, characterized in that, The processed input feature map is then rearranged through channels to obtain the final output, which includes: The input feature map [B,C,H,W] is divided into several groups based on the number of channels. The input feature map is then rearranged, and a transpose operation is performed between the second and third dimensions to exchange the group order with the channel order, resulting in a tensor of shape [B,C / groups,groups,H,W]. This tensor is then flattened, and the channels within each group are connected together to obtain another tensor of shape [B,C,H,W]. This is the final output, where B represents the number of feature maps in each batch of images, C represents the number of channels in the feature image, H represents the height of the feature image, and W represents the width of the feature image.

5. An image deraining device based on enhanced prior and hybrid attention, characterized in that, include: The filtering and extraction module is used to perform color channel analysis on the feature map of the input image, calculate the color difference of each channel, obtain the contrast information of the image in the color dimension, determine the variation range of each channel based on the magnitude of the contrast information, and identify the significant regions in the input image. The feature map is first filtered using a Gaussian filter kernel. The first filtered feature map is then weighted and merged with the original feature map to obtain a merged feature map. The merged feature map is then filtered a second time using a median filter kernel. The Sobel edge extraction operator is then used to extract edges from the feature map after the second filtering process to obtain bright edges and gradient information in the input image. The weighted average fusion module is used to perform weighted average fusion on the feature map after secondary filtering and edge extraction based on the salient regions, bright edges and gradient information in the input image, to generate the final prior image, i.e., the input feature map. The attention calculation module is used to process a portion of the input feature map through a global average pooling layer to extract the information required for channel attention. It then processes the input feature map after global average pooling through a fully connected layer and enhances the expressive power of the channel features through pointwise convolution and depthwise separable convolution. Channel attention is calculated using an activation function. The module also performs group normalization on another portion of the input feature map and calculates spatial attention using a sliding window attention module based on a series of linear transformations and a sigmoid activation function. The concatenation and rearrangement module is used to concatenate channel attention and spatial attention through the multi-head attention module, and rearrange the channels of the processed input feature map to obtain the final output.

6. The apparatus according to claim 5, characterized in that, The Gaussian filter kernel has a size of 5×5, the standard deviation of the first filter is 0.5, and the median filter kernel has a size of 3×3.

7. The apparatus according to claim 5, characterized in that, The attention calculation module is specifically used for: The length and width dimensions of the input feature map are compressed into one dimension, and segmentation and connection are performed along the same channel at this scale. That is, full connection processing is performed in different spatial regions within the same channel, ultimately resulting in spatial attention.

8. The apparatus according to claim 5, characterized in that, The splicing and rearrangement module is specifically used for: The input feature map [B,C,H,W] is divided into several groups based on the number of channels. The input feature map is then rearranged, and a transpose operation is performed between the second and third dimensions to exchange the group order with the channel order, resulting in a tensor of shape [B,C / groups,groups,H,W]. This tensor is then flattened, and the channels within each group are connected together to obtain another tensor of shape [B,C,H,W]. This is the final output, where B represents the number of feature maps in each batch of images, C represents the number of channels in the feature image, H represents the height of the feature image, and W represents the width of the feature image.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the image deraining method based on enhanced priors and hybrid attention as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an implementation program for information transmission, which, when executed by a processor, implements the steps of the image deraining method based on enhanced priors and hybrid attention as described in any one of claims 1 to 4.

Images