Image rain removal method, device, equipment and storage medium

Abstract

The present application relates to the field of image processing technology, and discloses an image rain removal method, device, equipment and storage medium for improving the accuracy and efficiency of image rain removal. The image rain removal method comprises: inputting the original rainy image into the data preprocessor of the pre-trained target image rain removal model for preprocessing to obtain a candidate rainy image; inputting the candidate rainy image into the feature extractor, extracting features through the feature extraction layers of multiple stages connected in sequence to obtain a multi-scale feature map; inputting the multi-scale feature map into the decoder, segmenting it through the first feature pyramid network propagated from bottom to top in the cascade feature pyramid fusion module, processing the initial segmentation feature map of each layer through the pyramid pooling module and the multi-scale large selection kernel attention module, and segmenting it through the second feature pyramid network propagated from the top layer to the bottom to obtain the target segmentation feature map of each layer; predicting based on the target fusion feature map to obtain the target rain-free image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image rain removal method, device, equipment and storage medium. Background Art

[0002] In rainy weather, high-density rain not only reduces visibility, but also causes light scattering and reflection, resulting in problems such as loss of details and reduced contrast in the acquired images, seriously affecting subsequent target recognition and detection tasks.

[0003] In image rain removal related technologies, due to the complexity and variability of the shape and scale of rain streaks, it is difficult to accurately separate rain streaks and background textures when processing complex rainy images. When fusion of features is performed, a simple feature superposition or splicing method is often used, which lacks an effective integration mechanism and results in poor rain removal effect. Summary of the invention

[0004] The present application provides an image rain removal method, device, equipment and storage medium for improving the accuracy and efficiency of image rain removal.

[0005] The first aspect of the present application provides an image rain removal method, comprising: inputting an original rainy image into a data preprocessor of a pre-trained target image rain removal model for preprocessing to obtain a candidate rainy image;

[0006] The candidate rainy image is input into a feature extractor, and feature extraction is performed through a plurality of stages of feature extraction layers connected in sequence to obtain a multi-scale feature map, and the feature extraction layer of each stage is connected to a cascade feature pyramid fusion module of a corresponding channel dimension in a decoder;

[0007] The multi-scale feature map is input into the decoder, and is segmented step by step through the first feature pyramid network propagated from bottom to top in the cascade feature pyramid fusion module, the initial segmentation feature map of each layer is processed by the pyramid pooling module and the multi-scale large selection kernel attention module, and is segmented step by step through the second feature pyramid network propagated from the top layer to bottom, so as to obtain the target segmentation feature map of each layer;

[0008] The target segmentation feature maps of each layer are fused, and prediction is performed based on the target fusion feature map to obtain the target rain-free map.

[0009] A second aspect of the present application provides an image rain removal device, comprising: a preprocessing unit, configured to input an original rainy image into a data preprocessor of a pre-trained target image rain removal model for preprocessing, to obtain a candidate rainy image;

[0010] An extraction unit, used for inputting the candidate rainy image into a feature extractor, performing feature extraction through a plurality of stages of feature extraction layers connected in sequence, and obtaining a multi-scale feature map, wherein the feature extraction layer of each stage is connected to a cascade feature pyramid fusion module of a corresponding channel dimension in a decoder;

[0011] A decoding unit, used for inputting the multi-scale feature map into a decoder, performing step-by-step segmentation through a first feature pyramid network propagating from bottom to top in a cascade feature pyramid fusion module, processing the initial segmentation feature map of each layer through the pyramid pooling module and the multi-scale large selection kernel attention module, and performing step-by-step segmentation through a second feature pyramid network propagating from the top layer to bottom, so as to obtain a target segmentation feature map of each layer;

[0012] The target segmentation feature maps of each layer are fused, and prediction is performed based on the target fusion feature map to obtain the target rain-free map.

[0013] The third aspect of the present application provides an image rain removal device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the image rain removal device executes the above-mentioned image rain removal method.

[0014] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned image rain removal method.

[0015] In the technical solution provided by the present application, by introducing multiple stages of feature extraction layers, the feature information of the rain image can be fully extracted from the multi-scale level. By optimizing the segmentation results of the auxiliary edge segmentation branch, the rainy images with similar backscattering characteristics can be avoided to cause errors in the rain removal process, thereby providing more accurate guidance for subsequent deep feature extraction, effectively improving the ability to separate rain streak edges and textures in rain removal tasks, and reducing interference. The decoder adopts a cascaded feature pyramid fusion module, which is directly connected to the multi-scale feature map in the feature extractor. It integrates features at different levels through multi-path connections, and can achieve efficient fusion and optimized decoding of features at different scales. It can not only accurately identify rain streaks, but also reconstruct the details of the image obscured by rain streaks, thereby generating a high-quality target rain-free image. The target rain removal model of the present application significantly improves the effect of the image rain removal task, taking into account both accuracy and efficiency, especially when dealing with complex rain image scenes. It has strong robustness and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of an embodiment of the image rain removal method in the present application;

[0017] Figure 2 A schematic diagram of a network architecture of the target rain removal model in this application;

[0018] Figure 3 A schematic diagram of another embodiment of the image rain removal method in the present application;

[0019] Figure 4 Another schematic diagram of the network architecture of the target rain removal model in this application;

[0020] Figure 5 A schematic diagram of a network architecture of a feature extractor in this application;

[0021] Figure 6 A schematic diagram of a network architecture of a cascade feature pyramid fusion module in this application;

[0022] Figure 7 A schematic diagram of a network architecture for selecting a core module in this application;

[0023] Figure 8 A schematic diagram of an embodiment of an image rain removal device in the present application;

[0024] Fig. 9 is a schematic diagram of another embodiment of the image rain removal device in the present application;

[0025] Fig.10 A schematic diagram of an embodiment of an image rain removal device in the present application. DETAILED DESCRIPTION

[0026] The present application provides an image rain removal method, device, equipment and storage medium for improving the accuracy and efficiency of image rain removal.

[0027] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] For ease of understanding, the specific process of the embodiment of the present application is described below. Figure 1 , an embodiment of the image rain removal method in the present application includes:

[0029] 101. The original rainy image is input into a data preprocessor of a pre-trained target image rain removal model for preprocessing to obtain a candidate rainy image.

[0030] It can be understood that the execution subject of the present application can be an image rain removal device, or a terminal or a server, which is not specifically limited here.

[0031] Reference Figure 2 A network architecture diagram of the target image deraining model of the present embodiment is provided. The target image deraining model of the present embodiment is an efficient transformer deraining network (ETDNet) based on an encoder and decoder (Encoder-Decoder) architecture, including a data preprocessor 201, a feature extractor 202 (i.e., an encoder) and a decoder 203, wherein the feature extractor 202 includes feature extraction layers connected in sequence at multiple stages, and the decoder 203 includes a cascaded feature pyramid fusion module, such as Figure 2 The target segmentation layer corresponds to the feature extraction layer in each stage, and the channel dimensions of the interconnected feature extraction layer and target segmentation layer correspond to each other.

[0032] Optionally, the target image rain removal model also includes an auxiliary segmentation branch 204, and the feature extractor 202 includes a middle-level feature extraction layer (i.e., the feature extraction layer of the Nth stage), whose outputs are respectively connected to the auxiliary segmentation branch 204 and the corresponding middle-level target segmentation layer, and the feature graphs output by each feature extraction layer have different scales, wherein the middle-level feature extraction layer is connected to the auxiliary edge segmentation branch, and the feature extraction layers of each stage are connected to the cascaded feature pyramid fusion module of the corresponding channel dimension in the decoder.

[0033] Among them, the auxiliary segmentation branch 204 converts the middle-level feature map and uses it to calculate the auxiliary loss to optimize the segmentation result and improve the segmentation accuracy. The decoder 203 integrates features at different levels through multi-path connections through the cascaded feature pyramid fusion module to obtain the target fused feature map for prediction and output the target rain-free map.

[0034] In this embodiment, the data preprocessor preprocesses the original rainy image to obtain a candidate rainy image that meets the input requirements of the target image rain removal model. The above preprocessing includes format conversion, data enhancement, etc. Specifically, the original rainy image is converted into a model input format, and after data enhancement, the image is input into a feature extractor for subsequent processing.

[0035] 102. The candidate rainy image is input into a feature extractor, and feature extraction is performed through multiple stages of feature extraction layers connected in sequence to obtain multiple scale feature maps.

[0036] The candidate rainy image is input into the feature extractor, and multiple levels of feature maps are extracted from left to right through multiple stages of feature extraction layers to convert the rainy image into multiple scale feature maps.

[0037] It is understandable that the feature maps output at each stage correspond to different scales. The background of rainy images is complex and changeable, and the scale variation range is large. Using only one layer of features extracted by the feature extractor will lead to poor segmentation effect. Features at different levels contain information with different focuses. The underlying features mainly reflect high-fidelity detail feature information such as light and dark, edges, and brightness, while high-level features contain richer semantic information.

[0038] 103. The multi-scale feature map is input into the decoder, and is segmented step by step through the first feature pyramid network propagated from bottom to top in the cascade feature pyramid fusion module. The initial segmentation feature map of each layer is processed through the pyramid pooling module and the multi-scale large selection kernel attention module, and is segmented step by step through the second feature pyramid network propagated from the top layer to bottom, so as to obtain the target segmentation feature map of each layer.

[0039] It is understandable that it is difficult to accurately determine the representative category of pixels if only the underlying features are analyzed. It is necessary to consider all hierarchical features at the same time. Therefore, this paper designs a cascaded feature pyramid fusion module (CPF) based on the UperNet network and the feature pyramid network (FPN) to better capture contextual information and retain the details of small targets.

[0040] In this implementation, the cascaded feature pyramid fusion module includes a first feature pyramid network that propagates from bottom to top, a second feature pyramid network that propagates from top to bottom, a multi-scale large selection kernel attention module (Multi-scale LargeSelection Kernel Attention Module, MLSK) and a pyramid pooling module. The top layer of the first feature pyramid network is connected to the top layer of the second feature pyramid network through the pyramid pooling module and the multi-scale large selection kernel attention module, and the remaining layers of the first feature pyramid network are connected to the remaining layers of the second feature pyramid network through the multi-scale large selection kernel attention module according to the corresponding channel dimension.

[0041] Among them, the feature pyramid network FPN involves constructing a series of images or features of different scales. It simply upsamples the feature map through bilinear interpolation instead of time-consuming deconvolution, and fuses the top-down path or the bottom-up path through a 1×1 convolutional layer, and then sums the elements. There is no complex refinement module, achieving a simple and efficient segmentation effect.

[0042] Specifically, the multi-scale feature map output by the feature extractor is segmented through the first feature pyramid network according to the corresponding channel dimension, and the initial segmentation feature map of each layer is propagated layer by layer from the bottom layer to the upper layer for segmentation until the top layer, so as to obtain the initial segmentation feature map of each layer;

[0043] The initial segmentation feature maps of each layer are processed by the pyramid pooling module and the multi-scale large selection kernel attention module to obtain the initial segmentation feature maps corresponding to each layer; the initial segmentation feature maps corresponding to each layer are segmented by the second feature pyramid network according to the corresponding channel dimension, and the candidate segmentation feature maps output by each layer are propagated layer by layer from the top layer to the next layer for segmentation until the bottom layer, to obtain the target segmentation feature maps of each layer; the target segmentation feature maps of each layer are fused, and the target fused feature map is predicted to obtain the target rain-free map.

[0044] The CPF module is combined with pyramid pooling and multi-scale large-selection kernel attention mechanism for fusion, and features at different levels are integrated through multi-path connection to obtain a target fusion feature map, so as to better capture contextual information and retain the detail information of small targets. The target fusion feature map is used for prediction to obtain a target rain-free map.

[0045] Among them, pyramid pooling can be performed through the pyramid pooling module (PPM), which is used to aggregate contextual information of different regions to better perform multi-scale context aggregation and global information collection, and assist in image scene parsing.

[0046] Among them, the multi-scale large selection kernel attention mechanism is implemented through the Multi-scale Large Selection Kernel Attention Module (MLSK), which is used to judge the value of the channel, adjust the spatial receptive field of deep targets, and capture channel information and spatial information more comprehensively.

[0047] 104. The target segmentation feature maps of each layer are fused, and prediction is performed based on the target fusion feature map to obtain a target rain-free map.

[0048] The target segmentation feature maps of each layer output by the second feature pyramid network are fused to obtain a target fusion feature map; and prediction is performed based on the target fusion feature map to obtain a target rain-free map.

[0049] In this embodiment, the original rainy image is preprocessed by data preprocessing to improve the accuracy of subsequent processing. By introducing multiple stages of feature extraction layers through the feature extractor, the feature information of the rain image can be fully extracted from multi-scale levels. The decoder adopts a cascaded feature pyramid fusion module combined with pyramid pooling and a multi-scale large selection kernel attention mechanism to integrate features at different levels from multi-path connections. It can achieve efficient fusion and optimized decoding of features between different scales, and can not only accurately identify rain streaks, but also reconstruct the details of the image obscured by rain streaks, thereby generating a high-quality target rain-free image. The target rain removal model of the present application significantly improves the effect of image rain removal tasks, taking into account both accuracy and efficiency, especially when processing complex rain image scenes. It has obvious advantages in maintaining the integrity of image details and edge structures, has strong robustness and adaptability, and provides a more reliable image foundation for subsequent target recognition and detection tasks.

[0050] In practical applications, traditional rain removal models generally use a single feature extraction path in the feature extraction stage, lacking effective use of mid-level features. This makes it difficult for the model to accurately distinguish between rain and background texture features when dealing with complex rain type features, affecting the rain removal effect. In addition, the lack of a dedicated edge information preservation mechanism can easily cause the loss of image edge structure while removing rain, affecting the image's detail performance. Please refer to Figure 3 , the second embodiment of the image rain removal method in the present application includes:

[0051] 301. Input the training sample set into a preset initial image rain removal model, and pass through a data preprocessor and a feature extractor in sequence to obtain a multi-scale feature map.

[0052] The model of this embodiment is trained on Ubuntu 18.04 system, GPU1060, and Pytorch1.6. Adam is used as the optimizer to optimize the network, and the initial learning rate is set to 0.002. The above parameters can be adjusted according to actual needs.

[0053] Optionally, the data preprocessor can preprocess the original rainy image through RainMix data enhancement to increase the diversity of training data, so as to better simulate the real-world rainfall conditions.

[0054] The above-mentioned training sample set can be a synthetic data set, such as Rain100L and Rain100H, wherein the rain streaks in Rain100L are lighter and are suitable for testing the rain removal algorithm under light rain conditions, and the rain streaks in Rain100H are heavier and are suitable for testing the rain removal algorithm under heavy rain conditions; the training sample set can also be an SPA data set, which is obtained based on a video deraining method and is not a real photo, and can also be other rain removal data sets, which are not specifically limited in this embodiment.

[0055] 302. Input the multi-scale feature map into the auxiliary edge segmentation branch and the decoder, and update the model parameters by providing a supervision signal through the auxiliary edge segmentation branch to obtain a target image rain removal model.

[0056] It is understandable that the semantics of pixels inside each object are consistent, and semantic inconsistency only occurs along the boundaries of adjacent objects. Rain images may have similar backscattering characteristics, which easily leads to errors in the image rain removal process. To this end, this embodiment uses an auxiliary edge segmentation branch (AES) at the neck position of the network to optimize the segmentation results.

[0057] Specifically, the middle-level feature map extracted by the middle-level feature extraction layer is used to determine the auxiliary loss value corresponding to the first prediction result through the auxiliary edge segmentation branch; the initial feature map output by each layer of the feature extractor is predicted by the decoder, and the main loss value corresponding to the second prediction result is determined through the mixed loss function; back propagation optimization is performed according to the auxiliary loss value and the main loss value to obtain the target image rain removal model.

[0058] Among them, AES can be composed of a fully convolutional neural network (FCN) and a hybrid loss function.

[0059] The input of FCN is the middle-level features obtained after the feature extractor processes the image, and then combined with the designed auxiliary hybrid loss function to guide the feature extraction of the deep feature extraction layer, thereby improving the performance of the overall model.

[0060] The calculation process can be expressed as:

[0061]

[0062] in, represents the result obtained after the auxiliary edge segmentation branch, Represents the middle-level features, represents a 3×3 convolution, represents another 3×3 convolution.

[0063] To facilitate understanding, an example is provided. Figure 4The network architecture diagram of the target rain removal model is shown in the figure, in which the feature extraction layer of the third stage of the feature extractor is connected to the FCN of the auxiliary edge segmentation branch, and the auxiliary loss is calculated through the auxiliary loss function (Auxiliary Loss). That is, the third-scale feature map output by the feature extraction layer of the third stage is input into the auxiliary edge segmentation branch, and a preliminary prediction is performed through the fully convolutional neural network to obtain a preliminary prediction result; the auxiliary loss value is calculated based on the prediction result and the preset auxiliary mixed loss function, and optimized according to the auxiliary loss value, and the optimized third-scale feature map is input into the deep feature extraction layer for feature extraction, that is:

[0064]

[0065] in, represents the result obtained after the auxiliary edge segmentation branch, represents the middle-level features, i.e. Figure 4 , 5 The third layer features in represents a 3×3 convolution, represents another 3×3 convolution.

[0066] In this embodiment, the auxiliary loss value and the main loss value of the cascade feature pyramid fusion module are weighted and combined according to preset weights to obtain a total loss value, and the network parameters are updated through back propagation based on the total loss value; the middle-layer feature map is reprocessed using the updated network parameters to obtain an optimized edge feature map; the edge feature map is feature-fused with the middle-layer feature map, and the fused middle-layer feature map is input into the deep feature extraction layer for feature extraction. Among them, the auxiliary loss value and the main loss value can be calculated by the same hybrid loss function.

[0067] The above hybrid loss function can be CLloss, including the cross entropy loss function and the Lovasz-Softmax loss function. The CLloss loss function expression is

[0068]

[0069] in, represents the cross entropy loss function, represents the Lovasz-Softmax loss function, Represents the weight used to balance the cross entropy loss and Lovasz-Softmax loss function.

[0070] Among them, the cross entropy loss function is used to measure the difference between the predicted probability and the true label in the classification task, thereby promoting the model to learn the correct pixel classification. Its expression is:

[0071]

[0072] in, Indicates the number of pixels, Represents pixels The truth class vector of Represents pixels The model output.

[0073] Among them, the Lovasz-Softmax loss function pays more attention to the accuracy of the boundary area based on the Jaccard index, which can improve the details of the rain removal results.

[0074]

[0075] in, represents the number of categories, Is a class The pixel error vector is Represents the smoothed extended discrete loss of the Jaccard indicator.

[0076] Below The derivation process is explained as follows:

[0077] Among them, the Jaccard index is defined as follows:

[0078]

[0079] Among them, c represents the category, A vector representing labels, Represents the vector of predicted labels, and the Jaccard index loss is:

[0080]

[0081] The mispredicted pixel set of class c can be expressed as: , thus, the Jaccard index can be expressed as:

[0082]

[0083] Since the above loss function is not differentiable, in order to optimize the Jaccard index of the training model, a smooth extension of discrete loss is performed based on the submodular analysis of the set function. The smooth extension is called the Lovasz extension, which is a set function expression:

[0084]

[0085] Where p is the number of pixels in the image, m is the vector of c-type pixel errors, in, represents the order of the components of m, then .

[0086] 303. Input the original rainy image into a data preprocessor of a pre-trained target image rain removal model for preprocessing to obtain a candidate rainy image.

[0087] 304. The candidate rainy image is processed through a convolution layer and a normalization layer to obtain an initial feature map.

[0088] Specifically, the input candidate rain image is first convolved through a convolution layer, and then normalized through a normalization layer (Layers Normalization, LN) to obtain an initial feature map.

[0089] For example, a candidate rainy image (512×512) first passes through a 4×4 convolutional layer and then a normalization layer. After these two layers of processing, the resolution of the image becomes one quarter of the original (128×128).

[0090] 305. Input the initial feature map into the feature extraction layer of the first stage for feature extraction to obtain a first scale feature map.

[0091] Specifically, the initial feature map is input into the first ConvNeXt module in the feature extraction layer of the first stage for feature extraction, and is element-by-element summed with the initial feature map to obtain a candidate feature map; the candidate feature map output by the previous ConvNeXt module is sequentially input into the next ConvNeXt module for processing until the last ConvNeXt module outputs the first scale feature map.

[0092] The feature extractor of this embodiment is an encoder based on ConvNeXt as the network skeleton. The feature extractor includes feature extraction layers of multiple stages. Each feature extraction layer can include a corresponding number of ConvNeXt modules. The number of ConvNeXt modules in each feature extraction layer can be the same or different, and there is no specific limitation. This embodiment is explained by taking the example that the middle feature extraction layer includes a first preset number of ConvNeXt modules, and the feature extraction layers of the remaining stages include a second preset number of ConvNeXt modules, and the first preset number is greater than the second preset number.

[0093] Reference Figure 5The ConvNeXt module of this embodiment includes a depthwise two-dimensional convolution layer (Depthwiseconv2d), a normalization layer (LN), a GELU activation function, a Layer Scale scaling layer and a Drop Path regularization layer. The first ConvNeXt module of the feature extraction layer of the first stage is taken as an example for explanation: the initial feature map is subjected to a depthwise convolution operation through the depthwise two-dimensional convolution layer of the first ConvNeXt module to obtain a first feature map; the first feature map is input into the LN normalization layer for normalization processing to obtain a second feature map; the second feature map is input into the two-dimensional convolution for convolution operation to obtain a third feature map; the third feature map is activated through the GELU activation function to obtain a fourth feature map; the fourth feature map is input into the two-dimensional convolution for convolution operation to obtain a fifth feature map; the fifth feature map is input into the Layer Scale scaling layer for processing to obtain a sixth feature map; the sixth feature map is input into the Drop Path regularization layer for processing, and element-by-element addition is performed with the initial feature map to obtain a first scale feature map.

[0094] In this embodiment, the ConvNeXt module can improve the stage ratio and adjust the number of stacking times of the residual network ResNet. Referring to ResNet, a more radical depth-separable convolution is adopted to increase the depth of the network at each stage; an inverted bottleneck is adopted to increase the convolution kernel size, the depth-separable convolution module is moved up, and the size of the convolution kernel is changed from 3×3 to 7×7; the ReLU activation function is replaced by GELU, fewer activation functions are used, and the GELU activation function is added only to the fully connected layer after the DW convolution, fewer normalization layers are used, and LN is connected only after the DW convolution, and batch normalization (BachNormalization, BN) is replaced by LN to improve the network effect.

[0095] 306. The scale feature map outputted by the previous stage is sequentially inputted into the feature extraction layer of the next stage for feature extraction, until a deep feature map outputted by the feature extraction layer of the last stage is obtained.

[0096] Specifically, the shallow feature map outputted by the previous stage is inputted into the first ConvNeXt module in the feature extraction layer (such as the middle feature extraction layer) of the next stage for feature extraction, and is element-by-element summed with the shallow feature map to obtain a candidate feature map; the candidate feature map outputted by the previous ConvNeXt module is inputted into the next ConvNeXt module in turn for processing, until the last ConvNeXt module outputs the middle feature map; this cycle is repeated until the feature extraction layer of the last stage outputs the deep feature map.

[0097] Reference Figure 5A network structure diagram of the feature extractor of this embodiment is provided. The feature extractor includes feature extraction layers of four stages. The feature extraction layer of the first stage includes three ConvNeXt modules, the feature extraction layer of the second stage includes three ConvNeXt modules (the second preset number is 3), the feature extraction layer of the third stage includes nine ConvNeXt modules (the first preset number is 9), and the feature extraction layer of the fourth stage includes three ConvNeXt modules. The ConvNeXt feature extractor extracts four multi-scale feature information of different sizes, which are respectively recorded as ( , , , ), these features then enter the decoder for feature fusion and other operations. This embodiment enhances the ability to extract middle-level features by setting the first preset number of ConvNeXt modules to be greater than the second preset number, and by configuring more ConvNeXt modules in the middle-level feature extraction layer, thereby solving the problem that a single feature extraction path is generally used in the feature extraction stage, and the middle-level features are not effectively utilized. This makes it difficult for the model to accurately distinguish between rain and background texture features when processing complex rain type features, affecting the rain removal effect.

[0098] Furthermore, the auxiliary segmentation branch is connected to the middle-level feature extraction layer, and the middle-level feature map is segmented to guide the extraction of deep-level feature maps. Due to the enhanced ability to extract middle-level features, the auxiliary edge segmentation branch (AES) optimizes the middle-level features and achieves accurate preservation of image edge information through the constraints of a fully convolutional neural network and a hybrid loss function.

[0099] Optionally, a downsampling module is included between the feature extraction layers of each stage to adjust the image resolution processed by each layer through downsampling processing to achieve feature extraction of different scales. Figure 5 Another network architecture diagram of the target image rain removal model shown in the figure includes three downsampling modules (Downsample), where the module dimension (Block dim) of the feature extraction layer in the first stage is 96, the Block dim of the feature extraction layer in the second stage is 192, the Block dim of the feature extraction layer in the third stage is 384, and the Block dim of the feature extraction layer in the fourth stage is 768, realizing the conversion from low-level features (coarser features) to high-level features (finer features). As the feature dimension increases, the network gradually transitions from low-level features to high-level features, thereby being able to capture more complex patterns and details.

[0100] 307. The multi-scale feature map is input into the decoder, and is segmented step by step through the first feature pyramid network propagating from bottom to top in the cascade feature pyramid fusion module. The initial segmentation feature map of each layer is processed through the pyramid pooling module and the multi-scale large selection kernel attention module, and is segmented step by step through the second feature pyramid network propagating from the top layer to bottom, so as to obtain the target segmentation feature map of each layer.

[0101] Reference Figure 6 The cascaded feature pyramid fusion module shown in Figure 1, the Multi-scale Large Selection Kernel Attention Module (MLSK) is a combined designed module consisting of an Efficient Channel Attention (ECA) module and a Large Selective Kernel (LSK) module.

[0102] Optionally, the multi-scale large selection kernel attention module includes a large selection kernel module and a plurality of channel attention modules, the top layer of the first feature pyramid network is connected to the top layer of the second feature pyramid network through the pyramid pooling module and the large selection kernel module, and the remaining layers of the first feature pyramid network are connected to the remaining layers of the second feature pyramid network through the channel attention module according to the corresponding channel dimension;

[0103] The initial segmentation feature map of each layer is processed by a pyramid pooling module and a multi-scale large selection kernel attention module, and is gradually segmented by a second feature pyramid network that propagates from the top layer to the bottom layer to obtain a target segmentation feature map of each layer, including: inputting the initial segmentation feature map output by the top layer of the first feature pyramid network into the pyramid pooling module and the large selection kernel module for processing, and inputting each initial segmentation feature map of the remaining layers into the corresponding channel attention module for processing to obtain a candidate segmentation feature map corresponding to each layer; summing each initial segmentation feature map and the corresponding candidate segmentation feature map element by element, and inputting the second feature pyramid network for segmentation according to the corresponding channel dimension, and the second feature pyramid network propagates the feature map output by each layer from the top layer to the next layer for processing to obtain a target segmentation feature map of each layer.

[0104] The initial segmentation feature map outputted from the top layer of the first feature pyramid network is inputted into the pyramid pooling module and the large selection kernel module for processing, including: concatenating different kernel features with a deep receptive field range from the top layer of the first feature pyramid network, and extracting spatial relationships using the pyramid pooling module based on the channel average and maximum pooling method to obtain various spatial attention maps; for each spatial attention map, using a sigmoid activation function to weight the features in the decomposed large kernel sequence according to the corresponding spatial selection mask, and fusing them through a convolutional layer to obtain an attention feature map; performing element-by-element dot product between the attention feature map and the initial segmentation feature map outputted from the top layer of the first feature pyramid network to obtain a candidate segmentation feature map corresponding to the top layer.

[0105] Taking the top layer processing in the multi-scale large selection kernel attention module as an example, the top layer of the first feature pyramid network is based on the initial segmentation feature map output by the second-to-last layer (such as Figure 4 From left to right, the third FPN output feature map) and the deep feature map output by the feature extraction layer in the last stage of the feature extractor (such as Figure 4 The feature map output by the fourth stage) is segmented, and the initial segmentation feature map output by the top layer of the first feature pyramid network is input into the pyramid pooling module and the large selection kernel module for processing, and is summed element by element with the feature map input by the pyramid pooling module (that is, the initial feature map output by the top layer of FPN) to obtain the candidate segmentation feature map of the top layer.

[0106] The complexity and variability of rainy images make the structure and semantic information of the images unclear. In order to enhance the representation ability of multi-shape targets and effectively deal with the scale changes and shape differences of targets in rainy images, the positioning of key channels during FPN fusion needs to be considered after using the CPF module.

[0107] For this purpose, the MLSK module is designed, referring to Figure 6 The MLSK module consists of three groups of channel attention modules (ECA) and one group of large selection kernel modules (LSK), which can judge the value of the channel, adjust the spatial receptive field of deep objects, and capture the channel information and spatial information more comprehensively.

[0108] The ECA channel attention mechanism pays more attention to the important features at the lower level and enhances the representation ability of shallow features to the target. Three ECA modules are used in feature pyramid modules of different scales to enhance the importance of useful channels in the first three layers of pyramid fusion modules and achieve fast and accurate positioning of useful channels.

[0109] After the multi-scale features pass through the CPF module, the obtained 4 feature maps of different sizes are recorded as ( , , , ), the calculation process can be expressed as:

[0110]

[0111] in, and Represents the feature map after ECA and LSK, It means a series of operations are performed in the order of 1×1 convolution, batch normalization and ReLU activation function. Indicates that bilinear interpolation is used for upsampling operations, the purpose of which is to adjust the size of high-level features so that they can be added to the target feature layer.

[0112] These feature maps are then concatenated through ECA and LSK in the channel dimension, and finally a composite feature map integrating multiple scales is output, thereby achieving the purpose of taking into account both global semantic information and local detail information.

[0113] When the CPF module performs bottom-up downsampling, the semantic features of small-sized objects will be lost quickly with layer-by-layer downsampling, making the semantic information of the top-level features unclear. To solve this problem, the LSK large selection kernel module is designed at the top level of the feature pyramid. It dynamically adjusts the receptive field of the feature through a spatial selection mechanism together with PPM. LSK weights the features processed by a large number of deep kernel sequences and then spatially merges them.

[0114] The weights of these kernels are dynamically determined according to the input, so that the model can adaptively use different large convolution kernels and adjust the spatial receptive field of deep targets as needed, increase the receptive field, and more effectively process the features of deep background objects. The structure of the LSK module is as follows: Figure 7 shown.

[0115] Introducing the channel attention mechanism in convolution has the potential to improve network performance. Different weight factors are assigned to different channels, focusing on more valuable channels and improving the quality of features generated by the network. This module has fewer parameters but can bring significant performance improvements. It determines the coverage of local cross-channel interactions by adaptively selecting the size of the one-dimensional convolution kernel. In order to aggregate the top-level context information and further improve the network's ability to segment the most relevant spatial background area at the top level of the pyramid, LSK is fused after the PPM at the top level of the pyramid to enhance the network's ability to extract high-level feature semantic information at a deep level. The specific calculation steps are as follows:

[0116] First, the different kernel features with deep receptive field range at the top layer of the feature pyramid are concatenated.

[0117]

[0118] in, Indicates the spatial relationship features efficiently extracted using channel-based average and maximum pooling methods. represents the kernel features of the first layer depth, represents the kernel feature of the i-th layer depth, Indicates layer depth.

[0119] Then, channel-based averaging is used and max pooling Method to efficiently extract spatial relationships :

[0120]

[0121] in, represents the average spatial characteristic expression, Represents the maximum spatial feature expression.

[0122] In order to achieve information interaction between different spatial expressions, the spatially aggregated features are connected in series and the convolutional layer is used to Convert the pooled features (2 channels) into N spatial attention maps:

[0123]

[0124] For each spatial attention map , use the sigmoid activation function to obtain a single spatial selection mask for each large kernel after decomposition:

[0125]

[0126] in, The sigmoid function is used to weight the features in the decomposed large kernel sequence according to their corresponding spatial selection masks, and then passes through the convolution layer. After fusion, we can get the attention feature S:

[0127]

[0128] The final output of the LSK module is:

[0129]

[0130] in, For input features Element-wise dot product with S.

[0131] 308. The target segmentation feature maps of each layer are fused, and prediction is performed based on the target fusion feature map to obtain a target rain-free map.

[0132] Step 308 can be performed with reference to step 104 and will not be described in detail here.

[0133] In this embodiment, an auxiliary edge segmentation branch is designed to optimize the middle-layer features. Through the constraints of the fully convolutional neural network and the hybrid loss function, the image edge information is accurately preserved, thereby providing more accurate guidance for subsequent deep feature extraction, effectively improving the ability to separate rain streak edges and textures in rain removal tasks, and reducing interference. The problem that the rain removal model lacks a special edge information preservation mechanism, which easily causes the loss of image edge structure while removing rain and affects the detail performance of the image is solved; by introducing multiple stages of feature extraction layers, the feature information of the rain image can be fully extracted from multi-scale levels, and a hierarchical feature extraction strategy is adopted. More ConvNeXt modules are configured in the middle-level feature extraction layer to enhance the ability to extract middle-level features. The problem that the traditional feature extraction stage generally adopts a single feature extraction path and lacks effective use of middle-level features makes it difficult for the model to accurately distinguish rain and background texture features when dealing with complex rain type features, affecting the rain removal effect; a cascade feature pyramid fusion module is proposed, which combines pyramid pooling and multi-scale large selection kernel attention mechanism, integrates features of different levels through multi-path connection, and can achieve efficient fusion and optimized decoding of features at different scales. It can not only accurately identify rain streaks, but also reconstruct the details blocked by rain streaks in the image, thereby generating a high-quality target rain-free image, solving the problem that traditional models often use simple feature superposition or splicing methods when performing feature fusion, lack an effective integration mechanism for multi-scale features, and perform poorly when dealing with raindrops of different scales. The target rain removal model of this application significantly improves the effect of image rain removal tasks, taking into account both accuracy and efficiency. In particular, it has obvious advantages in maintaining the integrity of image details and edge structures when processing complex rainy image scenes. It has strong robustness and adaptability, and provides a more reliable image foundation for subsequent target recognition and detection tasks.

[0134] The above describes the image rain removal method in this application. The following describes the image rain removal device in this application. Figure 8 , an embodiment of the image rain removal device in the present application includes:

[0135] A preprocessing unit 801 is used to input the original rainy image into a data preprocessor of a pre-trained target image rain removal model for preprocessing to obtain a candidate rainy image;

[0136] An extraction unit 802 is used to input the candidate rainy image into a feature extractor, perform feature extraction through multiple stages of feature extraction layers connected in sequence, and obtain a multi-scale feature map. The feature extraction layer of each stage is connected to a cascade feature pyramid fusion module of a corresponding channel dimension in the decoder;

[0137] The decoding unit 803 is used to input the multi-scale feature map into the decoder, and perform step-by-step segmentation through the first feature pyramid network propagated from the bottom to the top in the cascade feature pyramid fusion module, process the initial segmentation feature map of each layer through the pyramid pooling module and the multi-scale large selection kernel attention module, and perform step-by-step segmentation through the second feature pyramid network propagated from the top layer to the bottom, so as to obtain the target segmentation feature map of each layer;

[0138] The target segmentation feature maps of each layer are fused, and prediction is performed based on the target fusion feature map to obtain the target rain-free map.

[0139] In this embodiment, the original rainy image is preprocessed by data preprocessing to improve the accuracy of subsequent processing. By introducing multiple stages of feature extraction layers through the feature extractor, the feature information of the rain image can be fully extracted from multi-scale levels. The decoder adopts a cascaded feature pyramid fusion module combined with pyramid pooling and a multi-scale large selection kernel attention mechanism to integrate features at different levels from multi-path connections. It can achieve efficient fusion and optimized decoding of features between different scales, and can not only accurately identify rain streaks, but also reconstruct the details of the image obscured by rain streaks, thereby generating a high-quality target rain-free image. The target rain removal model of the present application significantly improves the effect of image rain removal tasks, taking into account both accuracy and efficiency, especially when processing complex rain image scenes. It has obvious advantages in maintaining the integrity of image details and edge structures, has strong robustness and adaptability, and provides a more reliable image foundation for subsequent target recognition and detection tasks.

[0140] See also Fig. 9 Another embodiment of the image rain removal device in the present application includes:

[0141] A preprocessing unit 801 is used to input the original rainy image into a data preprocessor of a pre-trained target image rain removal model for preprocessing to obtain a candidate rainy image;

[0142] An extraction unit 802 is used to input the candidate rainy image into a feature extractor, perform feature extraction through multiple stages of feature extraction layers connected in sequence, and obtain a multi-scale feature map. The feature extraction layer of each stage is connected to a cascade feature pyramid fusion module of a corresponding channel dimension in the decoder;

[0143] The decoding unit 803 is used to input the multi-scale feature map into the decoder, and perform step-by-step segmentation through the first feature pyramid network propagated from the bottom to the top in the cascade feature pyramid fusion module, process the initial segmentation feature map of each layer through the pyramid pooling module and the multi-scale large selection kernel attention module, and perform step-by-step segmentation through the second feature pyramid network propagated from the top layer to the bottom, so as to obtain the target segmentation feature map of each layer;

[0144] The target segmentation feature maps of each layer are fused, and prediction is performed based on the target fusion feature map to obtain the target rain-free map.

[0145] Optionally, the image rain removal model further includes an auxiliary edge segmentation branch, and the middle feature extraction layer of the feature extractor is connected to the auxiliary edge segmentation branch;

[0146] The image rain removal device also includes: an auxiliary updating unit 804, which is used to input the training sample set into a preset initial image rain removal model, and sequentially pass through a data preprocessor and a feature extractor to obtain a multi-scale feature map; input the multi-scale feature map into an auxiliary edge segmentation branch and a decoder, and provide a supervision signal through the auxiliary edge segmentation branch to update the model parameters to obtain a target image rain removal model.

[0147] Optionally, the auxiliary updating unit 804 is specifically used to determine the auxiliary loss value corresponding to the first prediction result through the auxiliary edge segmentation branch of the middle-level feature map extracted by the middle-level feature extraction layer;

[0148] The initial feature maps output by each layer of the feature extractor are predicted by the decoder, and the backbone loss value corresponding to the second prediction result is determined by the hybrid loss function;

[0149] Back propagation optimization is performed based on the auxiliary loss value and the backbone loss value to obtain the target image rain removal model.

[0150] Optionally, the extraction unit 802 includes:

[0151] The processing subunit 8021 is used to process the candidate rainy image through a convolution layer and a normalization layer to obtain an initial feature map;

[0152] The extraction subunit 8022 is used to input the initial feature map into the feature extraction layer of the first stage for feature extraction to obtain a first scale feature map; and sequentially input the scale feature map output by the previous stage into the feature extraction layer of the next stage for feature extraction until a deep feature map output by the feature extraction layer of the last stage is obtained.

[0153] Optionally, the extraction subunit 8022 is specifically used to: input the initial feature map into the first ConvNeXt module in the feature extraction layer of the first stage for feature extraction, and sum it element by element with the initial feature map to obtain a candidate feature map; input the candidate feature map output by the previous ConvNeXt module into the next ConvNeXt module in sequence for processing until the last ConvNeXt module outputs the first scale feature map.

[0154] Optionally, the cascade feature pyramid fusion module includes a first feature pyramid network propagated from bottom to top, a second feature pyramid network propagated from top to bottom, a multi-scale large selection kernel attention module and a pyramid pooling module, the top layer of the first feature pyramid network is connected to the top layer of the second feature pyramid network through the pyramid pooling module and the multi-scale large selection kernel attention module, the remaining layers of the first feature pyramid network are connected to the remaining layers of the second feature pyramid network through the multi-scale large selection kernel attention module according to the corresponding channel dimension, and the decoding unit 803 includes:

[0155] The first segmentation subunit 8031 ​​is used to segment the multi-scale feature map output by the feature extractor through the first feature pyramid network according to the corresponding channel dimension, and propagate the initial segmentation feature map of each layer from the bottom layer to the previous layer layer by layer to segment until the top layer, so as to obtain the initial segmentation feature map of each layer;

[0156] The adjustment subunit 8032 is used to input the initial segmentation feature map output by the top layer of the first feature pyramid network into the pyramid pooling module and the large selection kernel module for processing, and input each initial segmentation feature map of the remaining layers into the corresponding channel attention module for processing to obtain a candidate segmentation feature map corresponding to each layer, and perform element-by-element summation of each initial segmentation feature map and the corresponding candidate segmentation feature map to obtain a candidate segmentation feature map corresponding to each layer;

[0157] The second segmentation subunit 8033 is used to segment the candidate segmentation feature maps corresponding to each layer through the second feature pyramid network according to the corresponding channel dimension, and propagate the feature maps output by each layer from the top layer to the next layer for segmentation until the bottom layer, so as to obtain the target segmentation feature maps of each layer;

[0158] The fusion prediction subunit 8034 is used to fuse the target segmentation feature maps of each layer, predict the target fusion feature map, and obtain the target rain-free map.

[0159] Optionally, the multi-scale large selection kernel attention module includes a large selection kernel module and multiple channel attention modules. The top layer of the first feature pyramid network is connected to the top layer of the second feature pyramid network through the pyramid pooling module and the large selection kernel module. The remaining layers of the first feature pyramid network are connected to the remaining layers of the second feature pyramid network through the channel attention module according to the corresponding channel dimension. The fusion prediction subunit 8034 is used to concatenate different kernel features with deep receptive domain range in the top layer of the first feature pyramid network, and use the pyramid pooling module to extract spatial relationships based on the channel average and maximum pooling method to obtain each spatial attention map. For each spatial attention map, the sigmoid activation function is used to weight the features in the decomposed large kernel sequence according to the corresponding spatial selection mask, and the features are fused through the convolution layer to obtain the attention feature map. The attention feature map is dot-producted with the initial segmentation feature map output by the top layer of the first feature pyramid network to obtain the candidate segmentation feature map corresponding to the top layer.

[0160] In this embodiment, an auxiliary edge segmentation branch is designed to optimize the middle-layer features. Through the constraints of the fully convolutional neural network and the hybrid loss function, the image edge information is accurately preserved, thereby providing more accurate guidance for subsequent deep feature extraction, effectively improving the ability to separate rain streak edges and textures in rain removal tasks, and reducing interference. The problem that the rain removal model lacks a special edge information preservation mechanism, which easily causes the loss of image edge structure while removing rain and affects the detail performance of the image is solved; by introducing multiple stages of feature extraction layers, the feature information of the rain image can be fully extracted from multi-scale levels, and a hierarchical feature extraction strategy is adopted. More ConvNeXt modules are configured in the middle-level feature extraction layer to enhance the ability to extract middle-level features. The problem that the traditional feature extraction stage generally adopts a single feature extraction path and lacks effective use of middle-level features makes it difficult for the model to accurately distinguish rain and background texture features when dealing with complex rain type features, affecting the rain removal effect; a cascade feature pyramid fusion module is proposed, which combines pyramid pooling and multi-scale large selection kernel attention mechanism, integrates features of different levels through multi-path connection, and can achieve efficient fusion and optimized decoding of features at different scales. It can not only accurately identify rain streaks, but also reconstruct the details blocked by rain streaks in the image, thereby generating a high-quality target rain-free image, solving the problem that traditional models often use simple feature superposition or splicing methods when performing feature fusion, lack an effective integration mechanism for multi-scale features, and perform poorly when dealing with raindrops of different scales. The target rain removal model of this application significantly improves the effect of image rain removal tasks, taking into account both accuracy and efficiency. In particular, it has obvious advantages in maintaining the integrity of image details and edge structures when processing complex rainy image scenes. It has strong robustness and adaptability, and provides a more reliable image foundation for subsequent target recognition and detection tasks.

[0161] above Figure 8 and Fig. 9 The image rain removal device in the present application is described in detail from the perspective of modular functional entities, and the image rain removal device in the present application is described in detail from the perspective of hardware processing.

[0162] See also Fig.10 As shown, the image rain removal device includes a processor 1000 and a memory 1001 , wherein the memory 1001 stores machine executable instructions that can be executed by the processor 1000 , and the processor 1000 executes the machine executable instructions to implement the above-mentioned image rain removal method.

[0163] Further, Fig.10 The image rain removal device shown also includes a bus 1002 and a communication interface 1003 , and the processor 1000 , the communication interface 1003 and the memory 1001 are connected via the bus 1002 .

[0164] Among them, the memory 1001 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), for example, at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 1003 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 1002 can be an ISA bus, a PCI bus or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0165] The processor 1000 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1000. The above processor 1000 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1001, and the processor 1000 reads the information in the memory 1001 and completes the method steps of the above-mentioned embodiment in combination with its hardware.

[0166] The present application also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the image rain removal method.

[0167] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0168] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., and other media that can store program codes.

[0169] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image rain removal method, characterized in that: The image rain removal method comprises: The original rainy image is input into the data preprocessor of the pre-trained target image rain removal model for preprocessing to obtain a candidate rainy image; The candidate rainy image is input into a feature extractor, and feature extraction is performed through a plurality of stages of feature extraction layers connected in sequence to obtain a multi-scale feature map, and the feature extraction layer of each stage is connected to a cascade feature pyramid fusion module of a corresponding channel dimension in a decoder; The multi-scale feature map is input into the decoder, and is segmented step by step through the first feature pyramid network propagated from bottom to top in the cascade feature pyramid fusion module, the initial segmentation feature map of each layer is processed through the pyramid pooling module and the multi-scale large selection kernel attention module, and is segmented step by step through the second feature pyramid network propagated from the top layer to bottom, so as to obtain the target segmentation feature map of each layer; The target segmentation feature maps of each layer are fused, and prediction is performed based on the target fusion feature map to obtain the target rain-free map.

2. The image rain removal method according to claim 1, characterized in that: The candidate rainy image is input into a feature extractor, and feature extraction is performed through feature extraction layers of multiple stages connected in sequence to obtain a multi-scale feature map, including: Processing the candidate rainy image through a convolution layer and a normalization layer to obtain an initial feature map; Inputting the initial feature map into the feature extraction layer of the first stage to perform feature extraction to obtain a first scale feature map; The scale feature map outputted by the previous stage is sequentially inputted into the feature extraction layer of the next stage for feature extraction, until the deep feature map outputted by the feature extraction layer of the last stage is obtained.

3. The image rain removal method according to claim 2, characterized in that: Each of the feature extraction layers includes a preset number of ConvNeXt modules; The inputting the initial feature map into the feature extraction layer of the first stage to extract features to obtain a first scale feature map comprises: Input the initial feature map into the first ConvNeXt module in the feature extraction layer of the first stage for feature extraction, and perform element-by-element summation with the initial feature map to obtain a candidate feature map; The candidate feature map output by the previous ConvNeXt module is input into the next ConvNeXt module for processing in sequence until the last ConvNeXt module outputs the first-scale feature map.

4. The image rain removal method according to claim 1, characterized in that: The multi-scale large selection core attention module includes a large selection core module and a plurality of channel attention modules, the top layer of the first feature pyramid network is connected to the top layer of the second feature pyramid network through the pyramid pooling module and the large selection core module, and the remaining layers of the first feature pyramid network are connected to the remaining layers of the second feature pyramid network through the channel attention module according to the corresponding channel dimension; The initial segmentation feature map of each layer is processed by the pyramid pooling module and the multi-scale large selection kernel attention module, and is segmented step by step through the second feature pyramid network propagated from the top layer to the bottom, so as to obtain the target segmentation feature map of each layer, including: Inputting the initial segmentation feature map outputted from the top layer of the first feature pyramid network into the pyramid pooling module and the large selection kernel module for processing, and inputting each of the initial segmentation feature maps of the remaining layers of the first feature pyramid network into the corresponding channel attention module for processing, to obtain candidate segmentation feature maps corresponding to each layer; Each of the initial segmentation feature maps is summed element by element with the corresponding candidate segmentation feature map, and is input into a second feature pyramid network for segmentation according to the corresponding channel dimension. The second feature pyramid network propagates the feature maps output by each layer from the top layer to the next layer for processing, so as to obtain the target segmentation feature maps of each layer.

5. The image rain removal method according to claim 4, characterized in that: The step of inputting the initial segmentation feature map outputted from the top layer of the first feature pyramid network into the pyramid pooling module and the large selection kernel module for processing comprises: The top layer output of the first feature pyramid network with different kernel features in the deep receptive field range is concatenated, and the pyramid pooling module is used to extract spatial relationships based on the channel average and maximum pooling method to obtain each spatial attention map; For each of the spatial attention maps, a sigmoid activation function is used to weight the features in the decomposed large core sequence according to the corresponding spatial selection mask, and the features are fused through a convolutional layer to obtain an attention feature map; Perform element-by-element dot product of the attention feature map and the initial segmentation feature map output by the top layer of the first feature pyramid network to obtain a candidate segmentation feature map corresponding to the top layer.

6. The image rain removal method according to any one of claims 1 to 5, characterized in that: The image rain removal model also includes an auxiliary edge segmentation branch, and the middle feature extraction layer of the feature extractor is connected to the auxiliary edge segmentation branch; Before the original rainy image is input into the data preprocessor of the pre-trained target image rain removal model for preprocessing to obtain the candidate rainy image, the method further includes: The training sample set is input into the preset initial image rain removal model, and passes through the data preprocessor and feature extractor in sequence to obtain a multi-scale feature map; The multi-scale feature map is input into the auxiliary edge segmentation branch and the decoder, and the auxiliary edge segmentation branch provides a supervision signal to update the model parameters to obtain the target image rain removal model.

7. The image rain removal method according to claim 6, characterized in that: The multi-scale feature map is input into the auxiliary edge segmentation branch and the decoder, and the auxiliary edge segmentation branch provides a supervision signal to update the model parameters to obtain the target image rain removal model, including: Determine the auxiliary loss value corresponding to the first prediction result by using the middle-level feature map extracted by the middle-level feature extraction layer through an auxiliary edge segmentation branch; The initial feature maps output by each layer of the feature extractor are predicted by a decoder, and a backbone loss value corresponding to a second prediction result is determined by a hybrid loss function; Back propagation optimization is performed according to the auxiliary loss value and the trunk loss value to obtain a target image rain removal model.

8. An image rain removal device, characterized in that: The image rain removal device comprises: A preprocessing unit, used for inputting the original rainy image into a data preprocessor of a pre-trained target image rain removal model for preprocessing to obtain a candidate rainy image; An extraction unit, used for inputting the candidate rainy image into a feature extractor, performing feature extraction through a plurality of stages of feature extraction layers connected in sequence, and obtaining a multi-scale feature map, wherein the feature extraction layer of each stage is connected to a cascade feature pyramid fusion module of a corresponding channel dimension in a decoder; A decoding unit, used for inputting the multi-scale feature map into a decoder, performing step-by-step segmentation through a first feature pyramid network propagating from bottom to top in a cascade feature pyramid fusion module, processing the initial segmentation feature map of each layer through a pyramid pooling module and a multi-scale large selection kernel attention module, and performing step-by-step segmentation through a second feature pyramid network propagating from the top layer to bottom, so as to obtain a target segmentation feature map of each layer; The target segmentation feature maps of each layer are fused, and prediction is performed based on the target fusion feature map to obtain the target rain-free map.

9. An image rain removal device, characterized in that: The image rain removal device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory to enable the image rain removal device to perform the image rain removal method according to any one of claims 1 to 7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instruction is read and executed, the image rain removal method according to any one of claims 1 to 7 is executed.

Images