Image rain removal method and system based on expert model and multi-stage rain stripe extraction

Abstract

The invention discloses an image rain removal method and system based on an expert model and multi-stage rain stripe extraction, and the method comprises the steps: carrying out the preprocessing of an input rain image, obtaining a separation image, inputting the separation image into a trained rain removal model, and carrying out the processing, inputting the separated image into a mixed rain stripe prior module for processing to obtain a mixed prior feature; inputting the mixed prior features into a multi-stage rain feature extraction module for step-by-step feature extraction to obtain rain stripe layer features; and based on the rain image and the rain stripe layer features, image details are recovered by using a detail recovery enhancement module to generate a final rain-removed image. The method comprises two parts of rain removal model training and image rain removal using the rain removal model, can utilize adaptive prior to combine with a recursive extraction strategy to realize accurate separation of rain stripe features and fine recovery of background details, can effectively process diversified rain stripe modes and complex backgrounds, and improves the precision and robustness of image rain removal.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more specifically, to an image rain removal method and system based on an expert model and multi-stage rain streak extraction. Background Art

[0002] Rain is a common natural phenomenon, but the impact of raindrops and rain streaks on image quality cannot be ignored. They not only cause blurring and distortion of visual information, but also interfere with the computer vision system's accurate interpretation of image content. Especially in high-risk scenarios that rely on image perception, such as autonomous driving and drone navigation, the presence of rain streaks may make the environmental data collected by the camera incomplete or unclear, thereby hindering the system's accurate judgment of road conditions, obstacles or traffic signs, and increasing the risk of traffic accidents. To meet this challenge, researchers are committed to developing efficient image deraining algorithms with the help of deep learning and image processing technology to restore the clarity and details of the image, providing technical support for the reliable operation of various intelligent vision systems.

[0003] In order to solve this problem, researchers have relied on the development of modern computing science to conduct in-depth research and innovation on image deraining technology. They have combined traditional image processing methods with deep learning technology to explore efficient algorithm models to separate rain streak features and reconstruct clear background images. These studies not only provide technical support for improving image quality, but also provide more reliable solutions for visual perception systems in fields such as unmanned driving and video surveillance.

[0004] In the existing research on image deraining based on deep learning technology, some researchers proposed a low-rank decomposition model, which captures the repetitive features of rain streaks to model them, thereby effectively separating the rain layer from the background layer. Other researchers have used a multi-stage method for rain removal. However, the existing deep learning-based image deraining methods generally only use the original input rain-containing images and lack the corresponding prior knowledge guidance, resulting in the inability to remove rain streaks or loss of background details. Summary of the invention

[0005] The purpose of the present invention is to provide an image deraining method and system based on an expert model and multi-stage rain streak extraction, so as to solve the defects of the prior art such as insufficient processing capability for complex rain streak patterns, poor model adaptability, and difficulty in simultaneously taking into account the deraining effect and background detail restoration.

[0006] A first aspect of the present invention provides an image deraining method based on an expert model and multi-stage rain streak extraction, comprising the following steps:

[0007] The input rainy image is preprocessed to obtain a separated image, and the separated image is input into the trained rain removal model for processing, wherein:

[0008] Inputting the separated image into a mixed rain streak prior module for processing to obtain mixed prior features;

[0009] Inputting the mixed prior features into a multi-stage rain feature extraction module to perform step-by-step feature extraction to obtain rain ripple layer features;

[0010] Based on the rainy image and the rain pattern layer features, a detail recovery and enhancement module is used to recover image details to generate a final de-rained image.

[0011] In this solution, the method further includes constructing and training the rain removal model, specifically including:

[0012] Obtain multiple groups of unpaired rainy and rainless images, and perform preprocessing to obtain training images;

[0013] The training image is input into a mixed rain streak prior module for training to obtain mixed prior features, wherein the mixed rain streak prior module includes a mixed prior expert module, which adaptively fuses multiple rain streak priors by using a dynamic gating mechanism through the decision of an expert network, specifically including a dynamic color prior, a gradient prior and a Gaussian blur prior;

[0014] Inputting the hybrid prior features into a multi-stage rain feature extraction module to extract and refine rain streak features to obtain rain streak layer features, wherein the multi-stage rain feature extraction module includes a multi-channel attention block, an active focus module, and a scale-aware convolutional attention module, and obtains the rain streak layer features by gradually extracting and refining rain streak features and separating rain streaks and background images layer by layer;

[0015] Based on the rainy image and the rain pattern layer features, a detail recovery enhancement module is used for training to restore image details, wherein the detail recovery enhancement module integrates a multi-expansion channel attention block, a scale-aware convolution attention module, a batch normalization and a PReLU activation module, and retains the original depth and clarity of the background image through multi-scale and multi-channel processing;

[0016] The learning rate and number of training rounds during training are differentiated based on the type of rain pattern. The learning rate for sparse rain patterns is 4×10 -4 The number of training rounds for sparse rain patterns is 150, and the learning rate for dense and complex rain patterns is 1×10 -4 The number of training rounds for dense and complex rain patterns is 100, and the composite loss function used includes a structural similarity loss function, an edge loss function, and a preset loss function, wherein the preset loss function includes the Charbonnier loss function and its improved version loss function.

[0017] In this solution, the training image is input into the mixed rain streak prior module for training to obtain mixed prior features, specifically including:

[0018] Give n prior expert networks: {Prior0,...,Prior i ,...,Prior n-1}.;

[0019] Output of the hybrid prior expert module Where P(x) i Represents the weight of the i-th expert network, generated by the gating network; Prior i (x) represents the output of the i-th expert network, and the calculation formula of the weight P(x) is P(x) = Softmax(Conv(FFN(x))). Among them, dynamic color prior, gradient prior and Gaussian blur prior are used as the corresponding weights P(x) respectively. i .

[0020] In this solution, the hybrid prior features are input into the multi-stage rain feature extraction module to extract rain streak features and refine the training to obtain rain streak layer features, specifically including:

[0021] The shallow features of rain streaks are initially extracted by using a multi-channel attention block, and the features related to rain streaks in the mixed prior features are processed by a multi-channel attention mechanism, and irrelevant background noise is suppressed;

[0022] The active focus module is used to obtain the key area for rain streak removal based on the pixel-level attention mechanism.

[0023] The multi-scale convolution of the scale-aware convolutional attention module is used to capture multi-scale features, and the attention mechanism of the scale-aware convolutional attention module is used to assign different weights to each channel.

[0024] In this solution, training is performed using a detail recovery enhancement module based on the rainy image and the rain streak layer features to restore image details, specifically including:

[0025] The detail recovery and enhancement module includes an initial convolutional layer, four recovery blocks and a final convolutional layer, wherein:

[0026] Extracting basic features based on the initial convolutional layer;

[0027] Each recovery block integrates a multi-expansion channel attention block, a scale-aware convolutional attention module, a batch normalization and a PReLU activation module, and gradually recovers the global structure and detail features based on the recovery block;

[0028] Based on the last convolutional layer, feature refinement and background refinement are performed on the data processed by the recovery block.

[0029] In this scheme, the structural similarity loss function is as follows:

[0030] L SSIM =-SSIM(B out ,B gt );

[0031] The Charbonnier loss function is as follows:

[0032]

[0033] The edge loss function is as follows:

[0034]

[0035] The improved loss function corresponding to Charbonnier loss is as follows:

[0036]

[0037] Among them, B out is the network output, B out =OR out +B rec , R out is the rain streak output predicted by the RSE branch, B rec is the detail image restored by the BDR branch, O is the original rainy training image, ò=1×10 -3 ;

[0038] The composite loss function is as follows:

[0039] L=L SSIM +λ1·L char +λ2·L edge +λ3·L char' ;

[0040] Among them, parameter λ1=5, parameter λ2=0.05, and parameter λ3=1.

[0041] A second aspect of the present invention further provides an image deraining system based on an expert model and multi-stage rain streak extraction, comprising a memory and a processor, wherein the memory comprises an image deraining method program based on an expert model and multi-stage rain streak extraction, and when the image deraining method program based on an expert model and multi-stage rain streak extraction is executed by the processor, the following steps are implemented:

[0042] The input rainy image is preprocessed to obtain a separated image, and the separated image is input into the trained rain removal model for processing, wherein:

[0043] Inputting the separated image into a mixed rain streak prior module for processing to obtain mixed prior features;

[0044] Inputting the mixed prior features into a multi-stage rain feature extraction module to perform step-by-step feature extraction to obtain rain ripple layer features;

[0045] Based on the rainy image and the rain pattern layer features, a detail recovery and enhancement module is used to recover image details to generate a final de-rained image.

[0046] In this solution, the method further includes constructing and training the rain removal model, specifically including:

[0047] Obtain multiple groups of unpaired rainy and rainless images, and perform preprocessing to obtain training images;

[0048] The training image is input into a mixed rain streak prior module for training to obtain mixed prior features, wherein the mixed rain streak prior module includes a mixed prior expert module, which adaptively fuses multiple rain streak priors by using a dynamic gating mechanism through the decision of an expert network, specifically including a dynamic color prior, a gradient prior and a Gaussian blur prior;

[0049] Inputting the hybrid prior features into a multi-stage rain feature extraction module to extract and refine rain streak features to obtain rain streak layer features, wherein the multi-stage rain feature extraction module includes a multi-channel attention block, an active focus module, and a scale-aware convolutional attention module, and obtains the rain streak layer features by gradually extracting and refining rain streak features and separating rain streaks and background images layer by layer;

[0050] Based on the rainy image and the rain pattern layer features, a detail recovery enhancement module is used for training to restore image details, wherein the detail recovery enhancement module integrates a multi-expansion channel attention block, a scale-aware convolution attention module, a batch normalization and a PReLU activation module, and retains the original depth and clarity of the background image through multi-scale and multi-channel processing;

[0051] The learning rate and number of training rounds during training are differentiated based on the type of rain pattern. The learning rate for sparse rain patterns is 4×10 -4 The number of training rounds for sparse rain patterns is 150, and the learning rate for dense and complex rain patterns is 1×10 -4 The number of training rounds for dense and complex rain patterns is 100, and the composite loss function used includes a structural similarity loss function, an edge loss function, and a preset loss function, wherein the preset loss function includes the Charbonnier loss function and its improved version loss function.

[0052] In this solution, the training image is input into the mixed rain streak prior module for training to obtain mixed prior features, specifically including:

[0053] Give n prior expert networks: {Prior0,...,Prior i ,...,Prior n-1}.;

[0054] Output of the hybrid prior expert module Where P(x) i Represents the weight of the i-th expert network, generated by the gating network; Prior i (x) represents the output of the i-th expert network, and the calculation formula of the weight P(x) is P(x) = Softmax(Conv(FFN(x))). Among them, dynamic color prior, gradient prior and Gaussian blur prior are used as the corresponding weights P(x) respectively. i .

[0055] In this solution, the hybrid prior features are input into the multi-stage rain feature extraction module to extract rain streak features and refine the training to obtain rain streak layer features, specifically including:

[0056] The shallow features of rain streaks are initially extracted by using a multi-channel attention block, and the features related to rain streaks in the mixed prior features are processed by a multi-channel attention mechanism, and irrelevant background noise is suppressed;

[0057] The active focus module is used to obtain the key area for rain streak removal based on the pixel-level attention mechanism.

[0058] The multi-scale convolution of the scale-aware convolutional attention module is used to capture multi-scale features, and the attention mechanism of the scale-aware convolutional attention module is used to assign different weights to each channel.

[0059] In this solution, training is performed using a detail recovery enhancement module based on the rainy image and the rain streak layer features to restore image details, specifically including:

[0060] The detail recovery and enhancement module includes an initial convolutional layer, four recovery blocks and a final convolutional layer, wherein:

[0061] Extracting basic features based on the initial convolutional layer;

[0062] Each recovery block integrates a multi-expansion channel attention block, a scale-aware convolutional attention module, a batch normalization and a PReLU activation module, and gradually recovers the global structure and detail features based on the recovery block;

[0063] Based on the last convolutional layer, feature refinement and background refinement are performed on the data processed by the recovery block.

[0064] In this scheme, the structural similarity loss function is as follows:

[0065] L SSIM =-SSIM(B out ,B gt );

[0066] The Charbonnier loss function is as follows:

[0067]

[0068] The edge loss function is as follows:

[0069]

[0070] The improved loss function corresponding to Charbonnier loss is as follows:

[0071]

[0072] Among them, B out is the network output, B out =OR out +B rec , R out is the rain streak output predicted by the RSE branch, B rec is the detail image restored by the BDR branch, O is the original rainy training image, ò=1×10 -3 ;

[0073] The composite loss function is as follows:

[0074] L=L SSIM +λ1·L char +λ2·L edge +λ3·L char' ;

[0075] Among them, parameter λ1=5, parameter λ2=0.05, and parameter λ3=1.

[0076] A third aspect of the present invention provides a computer-readable storage medium, which includes a machine program for an image deraining method based on an expert model and multi-stage rain ripple extraction. When the image deraining method program based on an expert model and multi-stage rain ripple extraction is executed by a processor, the steps of the image deraining method based on an expert model and multi-stage rain ripple extraction as described in any one of the above items are implemented.

[0077] The present invention discloses an image deraining method and system based on an expert model and multi-stage rain streak extraction, which includes two parts: training a rain deraining model and applying the rain deraining model to derain images. The method and system can utilize adaptive priors combined with a recursive extraction strategy to achieve accurate separation of rain streak features and fine restoration of background details, and can effectively handle diverse rain streak patterns and complex backgrounds, thereby improving the accuracy and robustness of image deraining. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 A schematic diagram of the steps of an image deraining method based on an expert model and multi-stage rain streak extraction according to the present invention is shown;

[0079] Figure 2 A schematic diagram of a process of an image deraining method based on an expert model and multi-stage rain streak extraction according to the present invention is shown;

[0080] Figure 3 A schematic diagram of a process of an image deraining method based on an expert model and multi-stage rain streak extraction according to the present invention is shown;

[0081] Figure 4 A schematic diagram of an active focusing module of an image deraining method based on an expert model and multi-stage rain streak extraction according to the present invention is shown;

[0082] Figure 5 A schematic diagram of a scale-aware convolutional attention module of an image deraining method based on an expert model and multi-stage rain streak extraction according to the present invention is shown;

[0083] Figure 6 A schematic diagram of a detail recovery module of an image deraining method based on an expert model and multi-stage rain streak extraction according to the present invention is shown;

[0084] Figure 7 A block diagram of an image rain removal system based on an expert model and multi-stage rain streak extraction according to the present invention is shown. DETAILED DESCRIPTION

[0085] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0086] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0087] Figure 1A flowchart of an image deraining method based on an expert model and multi-stage rain streak extraction is shown in the present application.

[0088] like Figure 1 As shown, the present application discloses an image deraining method based on an expert model and multi-stage rain streak extraction, comprising the following steps:

[0089] S102, preprocessing the input rainy image to obtain a separated image, and inputting the separated image into a trained rain removal model for processing;

[0090] S104, inputting the separated image into a mixed rain pattern prior module for processing to obtain a mixed prior feature;

[0091] S106, inputting the mixed priori features into a multi-stage rain feature extraction module to perform step-by-step feature extraction to obtain rain ripple layer features;

[0092] S108, restoring image details using a detail recovery and enhancement module based on the rainy image and the rain pattern layer features to generate a final de-rained image.

[0093] It should be noted that, in this embodiment, the content of applying the trained rain removal model to perform rain removal on images is specifically described, wherein, by dynamically fusing multiple rain streak priors and gradually recursively extracting rain streak features, combined with a detail recovery mechanism, efficient removal of complex rain streaks and accurate recovery of the background image are achieved. This method is efficient and robust and is suitable for a variety of complex rainy day scenes.

[0094] Specifically, Figure 2 As shown, it is a rain removal flow chart, the input rainy image is preprocessed to obtain a separated image, specifically, the background information and rain streak features in the image are preliminarily separated, and the separated image is input into the trained rain removal model for processing. When the trained rain removal model is processing, the separated image is input into the mixed rain streak prior module for processing to obtain mixed prior features, wherein a variety of rain streak priors are adaptively fused through a dynamic gating mechanism, including dynamic color prior, gradient prior and Gaussian blur prior, and the weight of each prior can be dynamically adjusted according to the local rain streak density and background complexity. The dynamic color prior captures the brightness and color change characteristics of the high-density rain area; the gradient prior extracts the high-frequency linear features of the sparse rain area; the Gaussian blur prior is used to process the blurred rain streak features in the complex background, thereby suppressing interference while maintaining the background information. Finally, the integrated mixed prior features are output as the input of the subsequent multi-stage rain feature extraction module.

[0095] Furthermore, the hybrid prior features are input into a multi-stage rain feature extraction module for step-by-step feature extraction to obtain rain streak layer features. Specifically, rain streak features are gradually extracted through a multi-stage recursive structure. First, the shallow rain streak features are extracted using a multi-channel attention block (Multi-Dilated Channel Attention Block, MDCAB), and then the visually significant rain streak areas are prioritized through the active focus module (Active Focus Module, AFM) to improve the accuracy of feature extraction. In addition, the scale-aware convolutional attention module (Scale Aware Convolutional Attention Module, SACAM) captures rain streak features of different scales through multi-scale convolution operations, and combines the channel attention mechanism to enhance the feature expression capability. Finally, after step-by-step feature extraction in the recursive stage, the rain streak layer features are obtained and output.

[0096] Furthermore, based on the rainy image and the rain streak layer features, a detail recovery and enhancement module is used to recover image details to generate a final derained image. Specifically, by receiving the rain streak layer features and the original input rainy image, detail information in the background image is further recovered, and background texture details are gradually restored through a series of recovery blocks (RB). Residual connection is used to reduce over-smoothing effect to ensure that the restored background features are both clear and realistic. After multi-stage processing, the final derained image is obtained.

[0097] It should be noted that in the present invention, the network includes three branches, namely, the Mixture Rain of Prior Experts (MRP) branch, the Multi-stage Rain Feature Extraction (RFE) branch, and the Detail Recovery Enhancement (DRE) branch. Among them, the MRP branch is used to mix multiple rain streak priors. The MPE module integrates multiple rain streak priors through a dynamic gating mechanism. This mechanism can adaptively adjust the contribution of each prior according to the input image features. The dynamic gating mechanism is crucial for the effective combination of these priors because it determines the weight distribution of dynamic color priors, gradient priors, and Gaussian blur priors at different spatial positions of the image. Specifically, the gated network receives input image features and learns to generate a set of spatially varying weights, which correspond to the weights of dynamic color priors, gradient priors, and Gaussian blur priors, respectively. In this way, the model can flexibly adjust the influence of each prior to adapt to the changes in the local content of the image and the density of rain streaks in different areas. The output of each prior and its corresponding weight are weighted by element-by-element multiplication, and then the weighted results are added to generate the final prior-enhanced feature representation. This feature representation combines the advantages of each prior and achieves better feature expression capabilities in a coordinated manner. Through the method of the present invention, the model can selectively focus on the area affected by rain streaks while suppressing background noise, thereby significantly improving the deraining accuracy.

[0098] According to an embodiment of the present invention, the method further includes constructing and training the rain removal model, specifically including:

[0099] Obtain multiple groups of unpaired rainy and rainless images, and perform preprocessing to obtain training images;

[0100] The training image is input into a mixed rain streak prior module for training to obtain mixed prior features, wherein the mixed rain streak prior module includes a mixed prior expert module, which adaptively fuses multiple rain streak priors by using a dynamic gating mechanism through the decision of an expert network, specifically including a dynamic color prior, a gradient prior and a Gaussian blur prior;

[0101] Inputting the hybrid prior features into a multi-stage rain feature extraction module to extract and refine rain streak features to obtain rain streak layer features, wherein the multi-stage rain feature extraction module includes a multi-channel attention block, an active focus module, and a scale-aware convolutional attention module, and obtains the rain streak layer features by gradually extracting and refining rain streak features and separating rain streaks and background images layer by layer;

[0102] Based on the rainy image and the rain pattern layer features, a detail recovery enhancement module is used for training to restore image details, wherein the detail recovery enhancement module integrates a multi-expansion channel attention block, a scale-aware convolution attention module, a batch normalization and a PReLU activation module, and retains the original depth and clarity of the background image through multi-scale and multi-channel processing;

[0103] The learning rate and number of training rounds during training are differentiated based on the type of rain pattern. The learning rate for sparse rain patterns is 4×10 -4 The number of training rounds for sparse rain patterns is 150, and the learning rate for dense and complex rain patterns is 1×10 -4 The number of training rounds for dense and complex rain patterns is 100, and the composite loss function used includes a structural similarity loss function, an edge loss function, and a preset loss function, wherein the preset loss function includes the Charbonnier loss function and its improved version loss function.

[0104] It should be noted that, in this embodiment, Figure 3 As shown, it is displayed as a flow chart. First, multiple groups of unpaired rainy and rainless images are obtained and preprocessed to obtain training images. In addition to collecting data by taking images, ready-made synthetic data sets such as Rain100L and Rain100H data sets can also be used instead. However, during preprocessing, the images of the two domains must be shuffled because the unsupervised learning method is used in this embodiment. Unpaired images are required for model training during training, and then the images are preprocessed. The preprocessing process is as follows: first, the images collected in the data set are resized, and the image size is uniformly converted to 256*256*3, converted to a tensor, and finally the image data is converted to a standard normal distribution to make the model easier to converge.

[0105] Further, according to an embodiment of the present invention, the step of inputting the training image into a mixed rain streak prior module for training to obtain mixed prior features specifically includes:

[0106] Give n prior expert networks: {Prior0,...,Prior i ,...,Prior n-1}.;

[0107] Output of the hybrid prior expert module Where P(x) i Represents the weight of the i-th expert network, generated by the gating network; Prior i(x) represents the output of the i-th expert network, and the calculation formula of the weight P(x) is P(x) = Softmax(Conv(FFN(x))). Among them, dynamic color prior, gradient prior and Gaussian blur prior are used as the corresponding weights P(x) respectively. i .

[0108] It should be noted that, in this embodiment, for a given input rain map x, the output of the MPE module is determined by the weighted sum of the outputs of the expert network, where the weight is determined by the output of the gating network. Given n prior expert networks: {Prior0,...,Prior i ,...,Prior n-1}. The output y of the MPE module can be expressed as: Where P(x) i Represents the weight of the i-th expert network, generated by the gating network; Prior i (x) represents the output of the i-th expert network, and the calculation formula of the weight P(x) is: P(x) = Softmax(Conv(FFN(x))).

[0109] Furthermore, the above-mentioned dynamic color prior is an effective prior method designed for rain streak features, which is used to capture the brightness and color change characteristics of high-density rain areas. By analyzing the impact of rain streaks on image brightness and color, the input image is converted into a grayscale image to enhance the brightness change, and a trainable threshold is used to segment the rain streak salient area. In addition, the prior strengthens the dense rain streak area through the maximum pooling operation to ensure that the rain streak features can be accurately extracted while reducing the interference of background information. The dynamic color prior has strong adaptability, and its parameters can be dynamically adjusted according to the density and distribution of rain streaks to adapt to different rainy day scenes. In the present invention, the dynamic color prior, the gradient prior and the Gaussian blur prior are respectively used as the expert function P(x) i , by dynamically adjusting the weights, accurate capture and optimization processing of different rain pattern characteristics can be achieved.

[0110] Furthermore, the dynamic gating mechanism in this embodiment enables the MPE module to balance the contribution of each prior, optimize the accuracy of rain streak detection and the effect of background separation, and the proposed framework shows strong robustness in a variety of rainy day scenes, including dense rainy days to sparse rain scenes with complex backgrounds.

[0111] Further, according to an embodiment of the present invention, the mixed prior features are input into a multi-stage rain feature extraction module to extract rain streak features and perform refinement training to obtain rain streak layer features, specifically including:

[0112] The shallow features of rain streaks are initially extracted by using a multi-channel attention block, and the features related to rain streaks in the mixed prior features are processed by a multi-channel attention mechanism, and irrelevant background noise is suppressed;

[0113] The active focus module is used to obtain the key area for rain streak removal based on the pixel-level attention mechanism.

[0114] The multi-scale convolution of the scale-aware convolutional attention module is used to capture multi-scale features, and the attention mechanism of the scale-aware convolutional attention module is used to assign different weights to each channel.

[0115] It should be noted that, in this embodiment, the multi-stage rain feature extraction module (RFE) adopts a multi-stage recursive design, which gradually extracts and refines rain streak features and separates rain streak and background information layer by layer. The input of the RFE module includes fusion features from the original image and the prior module (MRP branch), and its output is the refined rain streak features and the preliminarily restored background image. Accordingly, the RFE branch includes the following parts:

[0116] (1) Multi-channel attention block (MDCAB): It is used to preliminarily extract the shallow features of rain streaks, enhance the rain streak-related features through the multi-channel attention mechanism, and suppress irrelevant background noise.

[0117] (2) Active Focus Module (AFM): This is the core innovation part of the RFE module, such as Figure 4 As shown in the figure, based on the pixel-level attention mechanism, the architecture of AFM (active focus module) is designed to calculate the interdependencies between features and assign appropriate weights to them, so that the network can focus on the most critical areas for rain streak removal. The operation framework of AFM first inputs the feature x t With the hidden state h of the previous stage t-1 To splice: y t =[x t ,h t-1 ], where t represents the current tth stage. Next, the pixel attention operation is used to calculate the pixel attention map M t :M t =PA(y t ). Subsequently, the input of each gating unit can be calculated by the following formula: Among them, the symbol Represents the element-by-element multiplication operation between two matrices. The calculation method of each gate unit in AFM is the same as that of LSTM, as follows: t ,h t ]=LSTM(G t ). In this formula, c t and ht Represent the unit state and hidden state of the tth stage respectively, and the output h t It will be passed as input to the module of the next stage. Through the AFM module, the model can more effectively focus on areas with significant rain streak visual effects, thereby improving the accuracy and effect of rain streak removal.

[0118] Furthermore, the Scale Aware Convolutional Attention Module (SACAM) is used to extract multi-scale features, which is crucial for effectively processing rain streaks, considering that rain streaks have different scale characteristics. Figure 5 As shown in the figure, the proposed scale-aware convolutional attention module (SACAM) contains two main components: multi-scale convolution and channel attention. The multi-scale convolution part consists of two convolution layers with different kernel sizes. The outputs of each layer are integrated through a splicing operation to fuse features from different receptive fields. By utilizing different convolution kernel sizes, SACAM is able to capture multi-scale features and enhance its ability to handle diverse rain streak patterns. In addition, the inter-layer data flow is fully connected, ensuring the full fusion of features from different scales. After the multi-scale convolution, the channel attention mechanism highlights the most relevant features by assigning different weights to each channel, thereby further optimizing the final output, which can refine the extracted features and improve the processing effect of multi-scale rain streaks.

[0119] According to an embodiment of the present invention, training is performed using a detail recovery enhancement module based on the rainy image and the rain streak layer features to restore image details, specifically including:

[0120] The detail recovery and enhancement module includes an initial convolutional layer, four recovery blocks and a final convolutional layer, wherein:

[0121] Extracting basic features based on the initial convolutional layer;

[0122] Each recovery block integrates a multi-expansion channel attention block, a scale-aware convolutional attention module, a batch normalization and a PReLU activation module, and gradually recovers the global structure and detail features based on the recovery block;

[0123] Based on the last convolutional layer, feature refinement and background refinement are performed on the data processed by the recovery block.

[0124] It should be noted that, in this embodiment, the detail recovery enhancement branch aims to restore the fine-grained background details blocked by rain streaks, wherein the traditional rain removal method may lead to excessive smoothing of background features, while the DRE branch focuses on preserving the fine texture and structure in the rain-removed image.

[0125] Specifically, the DRE branch gradually enhances background features through multi-scale and multi-channel processing. First, it uses the initial convolutional layer to extract basic features, and then gradually recovers global structure and detail features through four recovery blocks (RBs), such as Figure 6 As shown in the figure, each restoration block integrates a multi-dilated channel attention block (MDCAB), a scale-aware convolutional attention module (SACAM), and a batch normalization (Batch Normalization) and PReLU activation module (BN+PReLU). After the RBs sequence, the input features are added to the output of the restoration block using a residual connection to retain key information and stabilize the learning process. The final convolution layer further refines the features and completes the background refinement. This minimizes the over-smoothing effect while retaining delicate textures, achieving a balanced deraining process. The DRE branch not only effectively removes rain streaks, but also retains the original depth and clarity of the background, ensuring high-quality presentation of the derained image.

[0126] Further, according to an embodiment of the present invention, the structural similarity loss function is as follows:

[0127] L SSIM =-SSIM(B out ,B gt );

[0128] The Charbonnier loss function is as follows:

[0129]

[0130] The edge loss function is as follows:

[0131]

[0132] The improved loss function corresponding to Charbonnier loss is as follows:

[0133]

[0134] Among them, B out is the network output, B out =OR out +B rec , R out is the rain streak output predicted by the RSE branch, B rec is the detail image restored by the BDR branch, O is the original rainy training image, ò=1×10 -3 ;

[0135] The composite loss function is as follows:

[0136] L=L SSIM+λ1·L char +λ2·L edge +λ3·L char' ;

[0137] Among them, parameter λ1=5, parameter λ2=0.05, and parameter λ3=1.

[0138] It should be noted that in this embodiment, the deraining model uses the Pytorch framework to build the network, and all experiments are completed on the NVIDIA RTX 3090Ti GPU. In terms of data sets, Rain100L, Rain100H and Rain800, which are commonly used in image deraining tasks, are used. For the training process, the learning rate and the number of training rounds are set according to different data sets. Specifically, for the Rain100L data set with sparse rain streaks, this paper sets the learning rate to 4×10 -4 , the number of training rounds is set to 150; for the Rain100H dataset with dense and complex rain patterns, this paper sets the learning rate to 1×10 -4 , the number of training rounds is set to 100, and the Adam optimizer is used. Based on the cosine annealing strategy, all the above learning rates will eventually steadily decrease to 1×10 -6 In the hyperparameter setting of the loss function, this paper sets λ1=5, λ2=0.05, λ3=1. By inputting multiple sets of paired rainy and rainless images into the fused model for training, the image reconstruction results after each rain removal are obtained. After the model training is completed, the rainy image is input, and the rain-removed image can be obtained after being processed by the trained rain removal model.

[0139] Figure 7 A block diagram of an image rain removal system based on an expert model and multi-stage rain streak extraction according to the present invention is shown.

[0140] like Figure 7 As shown, the present invention discloses an image deraining system based on an expert model and multi-stage rain streak extraction, comprising a memory and a processor, wherein the memory comprises an image deraining method program based on an expert model and multi-stage rain streak extraction, and when the image deraining method program based on an expert model and multi-stage rain streak extraction is executed by the processor, the following steps are implemented:

[0141] The input rainy image is preprocessed to obtain a separated image, and the separated image is input into the trained rain removal model for processing, wherein:

[0142] Inputting the separated image into a mixed rain streak prior module for processing to obtain mixed prior features;

[0143] Inputting the mixed prior features into a multi-stage rain feature extraction module to perform step-by-step feature extraction to obtain rain ripple layer features;

[0144] Based on the rainy image and the rain pattern layer features, a detail recovery and enhancement module is used to recover image details to generate a final de-rained image.

[0145] It should be noted that since the specific implementation method of this embodiment corresponds to the aforementioned method embodiment, the same details will not be repeated here. Those skilled in the art should also understand that the division corresponding to the various execution steps in the embodiment is only a division of logical functions. In actual implementation, it can be fully or partially integrated into one or more physical entities, and these execution steps can all be implemented in the form of software calling through processing elements, or all in the form of hardware, and some execution steps can be implemented in the form of processing elements calling software, and some execution steps can be implemented in the form of hardware.

[0146] A third aspect of the present invention provides a computer-readable storage medium, which includes a program for an image deraining method based on an expert model and multi-stage rain streak extraction. When the program for an image deraining method based on an expert model and multi-stage rain streak extraction is executed by a processor, the steps of the image deraining method based on an expert model and multi-stage rain streak extraction as described in any one of the above items are implemented.

[0147] The present invention discloses an image deraining method and system based on an expert model and multi-stage rain streak extraction, which includes two parts: training a rain deraining model and applying the rain deraining model to derain images. The method and system can utilize adaptive priors combined with a recursive extraction strategy to achieve accurate separation of rain streak features and fine restoration of background details, and can effectively handle diverse rain streak patterns and complex backgrounds, thereby improving the accuracy and robustness of image deraining.

[0148] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0149] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0150] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0151] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, and other media that can store program codes.

[0152] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

Claims

1. An image deraining method based on an expert model and multi-stage rain streak extraction, characterized in that: The following steps are involved: The input rainy image is preprocessed to obtain a separated image, and the separated image is input into the trained rain removal model for processing, wherein: Inputting the separated image into a mixed rain streak prior module for processing to obtain mixed prior features; Inputting the mixed prior features into a multi-stage rain feature extraction module to perform step-by-step feature extraction to obtain rain ripple layer features; Based on the rainy image and the rain pattern layer features, a detail recovery and enhancement module is used to recover image details to generate a final de-rained image.

2. The image deraining method based on expert model and multi-stage rain streak extraction according to claim 1, characterized in that: The method further includes constructing and training the rain removal model, specifically including: Obtain multiple groups of unpaired rainy and rainless images, and perform preprocessing to obtain training images; The training image is input into a mixed rain streak prior module for training to obtain mixed prior features, wherein the mixed rain streak prior module includes a mixed prior expert module, which adaptively fuses multiple rain streak priors by using a dynamic gating mechanism through the decision of an expert network, specifically including a dynamic color prior, a gradient prior and a Gaussian blur prior; Inputting the hybrid prior features into a multi-stage rain feature extraction module to extract and refine rain streak features to obtain rain streak layer features, wherein the multi-stage rain feature extraction module includes a multi-channel attention block, an active focus module, and a scale-aware convolutional attention module, and obtains the rain streak layer features by gradually extracting and refining rain streak features and separating rain streaks and background images layer by layer; Based on the rainy image and the rain pattern layer features, a detail recovery enhancement module is used for training to restore image details, wherein the detail recovery enhancement module integrates a multi-expansion channel attention block, a scale-aware convolution attention module, a batch normalization and a PReLU activation module, and retains the original depth and clarity of the background image through multi-scale and multi-channel processing; The learning rate and number of training rounds during training are differentiated based on the type of rain pattern. The learning rate for sparse rain patterns is 4×10 -4 The number of training rounds for sparse rain patterns is 150, and the learning rate for dense and complex rain patterns is 1×10 -4 The number of training rounds for dense and complex rain patterns is 100, and the composite loss function used includes a structural similarity loss function, an edge loss function, and a preset loss function, wherein the preset loss function includes the Charbonnier loss function and its improved version loss function.

3. The image deraining method based on expert model and multi-stage rain streak extraction according to claim 2, characterized in that: The step of inputting the training image into a mixed rain streak prior module for training to obtain mixed prior features specifically includes: Give n prior expert networks: {Prior0,...,Prior i ,...,Prior n-1 }.; Output of the hybrid prior expert module Where P(x) i Represents the weight of the i-th expert network, generated by the gating network; Prior i (x) represents the output of the i-th expert network, and the calculation formula of the weight P(x) is P(x) = Softmax(Conv(FFN(x))). Among them, dynamic color prior, gradient prior and Gaussian blur prior are used as the corresponding weights P(x) respectively. i .

4. The image deraining method based on expert model and multi-stage rain streak extraction according to claim 3, characterized in that: The mixed prior features are input into the multi-stage rain feature extraction module to extract rain streak features and refine the training to obtain rain streak layer features, specifically including: The shallow features of rain streaks are initially extracted by using a multi-channel attention block, and the features related to rain streaks in the mixed prior features are processed by a multi-channel attention mechanism, and irrelevant background noise is suppressed; The active focus module is used to obtain the key area for rain streak removal based on the pixel-level attention mechanism. The multi-scale convolution of the scale-aware convolutional attention module is used to capture multi-scale features, and the attention mechanism of the scale-aware convolutional attention module is used to assign different weights to each channel.

5. The image deraining method based on expert model and multi-stage rain streak extraction according to claim 4, characterized in that: Based on the rainy image and the rain pattern layer features, a detail recovery enhancement module is used for training to recover image details, specifically including: The detail recovery and enhancement module includes an initial convolutional layer, four recovery blocks and a final convolutional layer, wherein: Extracting basic features based on the initial convolutional layer; Each recovery block integrates a multi-expansion channel attention block, a scale-aware convolutional attention module, a batch normalization and a PReLU activation module, and gradually recovers the global structure and detail features based on the recovery block; Based on the last convolutional layer, feature refinement and background refinement are performed on the data processed by the recovery block.

6. The image deraining method based on expert model and multi-stage rain streak extraction according to claim 5, characterized in that: The structural similarity loss function is as follows: L SSIM =-SSIM(B out ,B gt ); The Charbonnier loss function is as follows: The edge loss function is as follows: The improved loss function corresponding to Charbonnier loss is as follows: Among them, B out is the network output, B out =OR out +B rec , R out is the rain streak output predicted by the RSE branch, B rec is the detail image restored by the BDR branch, O is the original rainy training image, ò=1×10 -3 ; The composite loss function is as follows: L=L SSIM +λ1·L char +λ2·L edge +λ3·L char' ; Among them, parameter λ1=5, parameter λ2=0.05, and parameter λ3=1.

7. An image deraining system based on expert model and multi-stage rain streak extraction, characterized in that: The method comprises a memory and a processor, wherein the memory comprises an image deraining method program based on an expert model and multi-stage rain streak extraction, and the image deraining method program based on an expert model and multi-stage rain streak extraction is executed by the processor to implement the following steps: The input rainy image is preprocessed to obtain a separated image, and the separated image is input into the trained rain removal model for processing, wherein: Inputting the separated image into a mixed rain streak prior module for processing to obtain mixed prior features; Inputting the mixed prior features into a multi-stage rain feature extraction module to perform step-by-step feature extraction to obtain rain ripple layer features; Based on the rainy image and the rain pattern layer features, a detail recovery and enhancement module is used to recover image details to generate a final de-rained image.

8. The image deraining system based on expert model and multi-stage rain streak extraction according to claim 7, characterized in that: The method further includes constructing and training the rain removal model, specifically including: Obtain multiple groups of unpaired rainy and rainless images, and perform preprocessing to obtain training images; The training image is input into a mixed rain streak prior module for training to obtain mixed prior features, wherein the mixed rain streak prior module includes a mixed prior expert module, which adaptively fuses multiple rain streak priors by using a dynamic gating mechanism through the decision of an expert network, specifically including a dynamic color prior, a gradient prior and a Gaussian blur prior; Inputting the hybrid prior features into a multi-stage rain feature extraction module to extract and refine rain streak features to obtain rain streak layer features, wherein the multi-stage rain feature extraction module includes a multi-channel attention block, an active focus module, and a scale-aware convolutional attention module, and obtains the rain streak layer features by gradually extracting and refining rain streak features and separating rain streaks and background images layer by layer; Based on the rainy image and the rain pattern layer features, a detail recovery enhancement module is used for training to restore image details, wherein the detail recovery enhancement module integrates a multi-expansion channel attention block, a scale-aware convolution attention module, a batch normalization and a PReLU activation module, and retains the original depth and clarity of the background image through multi-scale and multi-channel processing; The learning rate and number of training rounds during training are differentiated based on the type of rain pattern. The learning rate for sparse rain patterns is 4×10 -4 The number of training rounds for sparse rain patterns is 150, and the learning rate for dense and complex rain patterns is 1×10 -4 The number of training rounds for dense and complex rain patterns is 100, and the composite loss function used includes a structural similarity loss function, an edge loss function, and a preset loss function, wherein the preset loss function includes the Charbonnier loss function and its improved version loss function.

9. The image deraining system based on expert model and multi-stage rain streak extraction according to claim 8, characterized in that: The step of inputting the training image into a mixed rain streak prior module for training to obtain mixed prior features specifically includes: Give n prior expert networks: {Prior0,...,Prior i ,...,Prior n-1 }.; Output of the hybrid prior expert module Where P(x) i Represents the weight of the i-th expert network, generated by the gating network; Prior i (x) represents the output of the i-th expert network, and the calculation formula of the weight P(x) is P(x) = Softmax(Conv(FFN(x))). Among them, dynamic color prior, gradient prior and Gaussian blur prior are used as the corresponding weights P(x) respectively. i .

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a program for an image deraining method based on an expert model and multi-stage rain ripple extraction. When the program for an image deraining method based on an expert model and multi-stage rain ripple extraction is executed by a processor, the steps of the image deraining method based on an expert model and multi-stage rain ripple extraction as described in any one of claims 1 to 6 are implemented.

Images