Image rain removal method and system based on expert model and multi-stage rain streak extraction

Abstract

The application discloses an image rain removal method and system based on an expert model and multi-stage rain streak extraction. The method comprises the following steps: performing preprocessing on an input rainy image to obtain a separated image, inputting the separated image into a trained rain removal model for processing, wherein the separated image is input into a mixed rain streak prior module for processing to obtain mixed prior features; the mixed prior features are input into a multi-stage rain feature extraction module for step-by-step feature extraction to obtain rain streak layer features; and details of the image are recovered by using a detail recovery enhancement module based on the rainy image and the rain streak layer features to generate a final rain removal image. The application comprises two parts of training of a rain removal model and image rain removal by using the rain removal model, can realize accurate separation of rain streak features and fine recovery of background details by using an adaptive prior combined with a recursive extraction strategy, can effectively process diversified rain streak patterns and complex backgrounds, and improves the accuracy and robustness of image rain removal.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, more particularly, to an image rain removal method and system based on an expert model and multi-stage rain streak extraction. BACKGROUND

[0002] Rain is a common natural phenomenon, but the influence of raindrops and rain streaks on image quality cannot be ignored. They not only cause visual information to be blurred and distorted, but also interfere with the accurate interpretation of image content by computer vision systems. Especially in high-risk scenarios such as autonomous driving and unmanned aerial vehicle navigation that rely on image perception, the presence of rain streaks may make the environmental data collected by the camera incomplete or unclear, thereby hindering the system's accurate judgment of road conditions, obstacles, or traffic signs, and thus increasing the risk of traffic accidents. To address this challenge, researchers have used deep learning and image processing techniques to develop efficient image rain removal algorithms to restore image clarity and detail, providing technical support for the reliable operation of various intelligent vision systems.

[0003] To solve this problem, researchers have relied on the development of modern computing science and conducted in-depth research and innovation on image rain removal technology. By combining traditional image processing methods with deep learning techniques, they have explored efficient algorithm models to separate rain streak features and reconstruct clear background images. These studies not only provide technical support for improving image quality, but also provide more reliable solutions for vision perception systems in the fields of autonomous driving and video surveillance.

[0004] In existing research on image rain removal based on deep learning technology, some researchers have proposed a low-rank decomposition model that captures the repetitive characteristics of rain streaks to model them, thereby effectively separating the rain layer and the background layer. Other researchers have used a multi-stage approach for rain removal. However, existing deep learning-based image rain removal methods generally only use the original input rain-containing image, lack corresponding prior knowledge guidance, and result in incomplete rain streak removal or loss of background details. SUMMARY

[0005] The purpose of the present application is to provide an image rain removal method and system based on an expert model and multi-stage rain streak extraction, which solves the defects of insufficient processing capability for complex rain streak patterns, poor model adaptability, and difficulty in simultaneously considering rain removal effect and background detail restoration in the prior art.

[0006] The present application provides an image rain removal method based on an expert model and multi-stage rain streak extraction in a first aspect, comprising the following steps:

[0007] The input rain-containing image is preprocessed to obtain a separated image, and the separated image is input into a trained rain removal model for processing, wherein,

[0008] inputting the separated image into a mixed rain streak prior module for processing to obtain a mixed prior feature;

[0009] inputting the mixed prior feature into a multi-stage rain feature extraction module for step-by-step feature extraction to obtain a rain streak layer feature;

[0010] based on the rainy image and the rain streak layer feature, using a detail recovery enhancement module to recover image details to generate a final rain-removed image.

[0011] In this scheme, the method further comprises constructing and training the rain-removed model, specifically comprising:

[0012] obtaining a plurality of groups of unpaired rainy images and rain-free images taken and pre-processing to obtain training images;

[0013] inputting the training image into a mixed rain streak prior module for training to obtain a mixed prior feature, wherein the mixed rain streak prior module includes a mixed prior expert module, which uses a dynamic gating mechanism to adaptively fuse multiple rain streak priors through the decision of the expert network, specifically including dynamic color prior, gradient prior and Gaussian blur prior;

[0014] inputting the mixed prior feature into a multi-stage rain feature extraction module for rain streak feature extraction and refinement training to obtain a rain streak layer feature, wherein the multi-stage rain feature extraction module includes a multi-channel attention block, an active focusing module and a scale-aware convolution attention module, which extracts and refines rain streak features step by step and separates rain streaks and background images layer by layer to obtain the rain streak layer feature;

[0015] based on the rainy image and the rain streak layer feature, using a detail recovery enhancement module for training to recover image details, wherein the detail recovery enhancement module integrates a multi-channel attention block, a scale-aware convolution attention module, a batch normalization and a PReLU activation module, which uses multi-scale and multi-channel processing to preserve the original depth and clarity of the background image;

[0016] During the training process, the learning rate and the number of training rounds are distinguished based on the rain streak type, wherein the learning rate of sparse rain streaks is , the number of training rounds of sparse rain streaks is 150, the learning rate of dense and complex rain streaks is , the number of training rounds of dense and complex rain streaks is 100, and the composite loss function used includes a structural similarity loss function, an edge loss function and a preset loss function, wherein the preset loss function includes a Charbonnier loss function and an improved version of the loss function.

[0017] In the scheme, the training image is input into the mixed rain streak prior module for training to obtain a mixed prior feature, specifically including:

[0018] To a prior expert network: ;

[0019] The output of the mixed prior expert module Wherein, indicates the weight of the expert network, generated by the gating network; indicates the output of the expert network, and the weight The calculation formula is Wherein, the dynamic color prior, gradient prior and Gaussian blur prior are respectively corresponding weights .

[0020] In the scheme, the mixed prior feature is input into the multi-stage rain feature extraction module for rain streak feature extraction and refinement training to obtain a rain streak layer feature, specifically including:

[0021] Preliminary extraction of shallow features of rain streaks is performed using a multi-channel attention block, which processes rain streak-related features in the mixed prior feature through a multi-channel attention mechanism and suppresses irrelevant background noise;

[0022] A rain streak removal key area is obtained by processing based on a pixel-level attention mechanism using an active focusing module;

[0023] Multi-scale convolution of the scale perception convolution attention module is used to capture multi-scale features, and the attention mechanism of the scale perception convolution attention module is used to assign different weights to each channel.

[0024] In the scheme, the image details are recovered based on the rainy image and the rain streak layer feature using a detail recovery enhancement module for training, specifically including:

[0025] The detail recovery enhancement module includes an initial convolution layer, four recovery blocks and a final convolution layer, wherein,

[0026] The basic features are extracted based on the initial convolution layer;

[0027] Each recovery block integrates a multi-channel attention block, a scale perception convolution attention module, a batch normalization and a PReLU activation module, and gradually recovers global structure and detail features based on the recovery block;

[0028] Based on the last convolution layer, the data processed by the recovery block is refined and the background is refined.

[0029] In the scheme, the structural similarity loss function is as follows:

[0030] ;

[0031] The Charbonnier loss function is as follows:

[0032] ;

[0033] The edge loss function is as follows:

[0034] ;

[0035] The improved version of the Charbonnier loss function corresponding to the Charbonnier loss is as follows:

[0036] ;

[0037] wherein, is the network output, , is the rain streak output predicted by the RSE branch, is the detail image recovered by the BDR branch, is the original rain-containing training image, ;

[0038] The composite loss function is as follows:

[0039] ;

[0040] wherein, the parameter , the parameter , and the parameter .

[0041] The second aspect of the present application also provides an image rain removal system based on an expert model and multi-stage rain streak extraction, comprising a memory and a processor, wherein the memory comprises an image rain removal method program based on an expert model and multi-stage rain streak extraction, and the image rain removal method program based on an expert model and multi-stage rain streak extraction is implemented when the processor is executed to realize the following steps:

[0042] An input rain-containing image is preprocessed to obtain a separated image, and the separated image is input into a trained rain removal model for processing, wherein,

[0043] The separated image is input into a mixed rain streak prior module for processing to obtain mixed prior features;

[0044] The mixed prior features are input into a multi-stage rain feature extraction module for step-by-step feature extraction to obtain rain streak layer features;

[0045] restore image details based on the rainy image and the rain layer features using a detail restoration enhancement module to generate a final rain-removed image.

[0046] In this scheme, the method further comprises constructing and training the rain-removed model, specifically comprising:

[0047] A plurality of groups of unpaired rainy images and rain-free images are obtained, preprocessed, and training images are obtained;

[0048] The training images are input into a hybrid rain streak prior module for training to obtain hybrid prior features, wherein the hybrid rain streak prior module includes a hybrid prior expert module, which adaptively fuses multiple rain streak priors using a dynamic gating mechanism through the decision of an expert network, specifically including a dynamic color prior, a gradient prior, and a Gaussian blur prior;

[0049] The hybrid prior features are input into a multi-stage rain feature extraction module for rain streak feature extraction and refinement training to obtain rain layer features, wherein the multi-stage rain feature extraction module includes a multi-channel attention block, an active focusing module, and a scale-aware convolution attention module, which gradually extracts and refines rain streak features and separates rain streaks and background images layer by layer to obtain the rain layer features;

[0050] The hybrid prior features are input into a multi-stage rain feature extraction module for rain streak feature extraction and refinement training to obtain rain layer features, wherein the multi-channel attention block, the scale-aware convolution attention module, the batch normalization, and the PReLU activation module are integrated into the detail restoration enhancement module, which processes through multiple scales and multiple channels to preserve the original depth and clarity of the background image;

[0051] The learning rate and the number of training rounds are distinguished based on the rain streak type during the training process, wherein the learning rate of sparse rain streaks is , the number of training rounds of sparse rain streaks is 150, the learning rate of dense complex rain streaks is , the number of training rounds of dense complex rain streaks is 100, and the composite loss function used includes a structural similarity loss function, an edge loss function, and a preset loss function, wherein the preset loss function includes a Charbonnier loss function and an improved version of the loss function.

[0052] In this scheme, the training images are input into a hybrid rain streak prior module for training to obtain hybrid prior features, specifically comprising:

[0053]

[0054] The output of the hybrid prior expert module is wherein, represents the​​​ weights of the expert networks, generated by the gating network; representing the output of the i-th expert network, the weights The calculation formula of the weights is wherein the dynamic color prior, the gradient prior and the Gaussian blur prior are respectively taken as the corresponding weights .

[0055] In the scheme, the mixed prior feature is input into a multi-stage rain feature extraction module for rain streak feature extraction and refinement training to obtain a rain streak layer feature, specifically including:

[0056] The multi-channel attention block is used to preliminarily extract the shallow features of the rain streaks, the multi-channel attention mechanism is used to process the rain streak related features in the mixed prior feature, and the irrelevant background noise is suppressed;

[0057] The active focusing module is used to process the rain streak removal key area based on the pixel-level attention mechanism;

[0058] The multi-scale convolution of the scale perception convolution attention module is used to capture multi-scale features, and the attention mechanism of the scale perception convolution attention module is used to assign different weights to each channel.

[0059] In the scheme, the detail recovery enhancement module is used to train based on the rainy image and the rain streak layer feature to recover the image details, specifically including:

[0060] The detail recovery enhancement module includes an initial convolution layer, four recovery blocks and a final convolution layer, wherein,

[0061] The initial convolution layer is used to extract basic features;

[0062] Each recovery block integrates a multi-channel attention block, a scale perception convolution attention module, a batch normalization and a PReLU activation module, and the global structure and the detail features are gradually recovered based on the recovery block;

[0063] The final convolution layer is used to refine the features and refine the background based on the data processed by the recovery block.

[0064] In the scheme, the structural similarity loss function is as follows:

[0065] ;

[0066] The Charbonnier loss function is as follows:

[0067] ;

[0068] The edge loss function is as follows: ​

[0069] ;

[0070] The improved loss function corresponding to the Charbonnier loss is as follows:

[0071] ;

[0072] wherein, is the network output, , is the rain streak output predicted by the RSE branch, is the detail image recovered by the BDR branch, is the original rain-containing training image, ;

[0073] The composite loss function is as follows:

[0074] ;

[0075] wherein, the parameter , the parameter , and the parameter .

[0076] The third aspect of the present application provides a computer readable storage medium, wherein the computer readable storage medium comprises a program of an image rain removal method based on an expert model and multi-stage rain streak extraction of a machine, and the program of the image rain removal method based on the expert model and the multi-stage rain streak extraction is executed by a processor to realize the steps of the image rain removal method based on the expert model and the multi-stage rain streak extraction according to any one of the above.

[0077] The image rain removal method and system based on the expert model and the multi-stage rain streak extraction disclosed by the present application include the training of the rain removal model and the application of the rain removal model to the image rain removal, can realize the accurate separation of rain streak features and the fine recovery of background details by using the adaptive prior combined with the recursive extraction strategy, can effectively process diversified rain streak patterns and complex backgrounds, and improves the accuracy and robustness of image rain removal. BRIEF DESCRIPTION OF DRAWINGS

[0078] Figure 1 FIG. 1 shows a step schematic diagram of the image rain removal method based on the expert model and the multi-stage rain streak extraction of the present application;

[0079] Figure 2 FIG. 2 shows a flow schematic diagram of the image rain removal method based on the expert model and the multi-stage rain streak extraction of the present application;

[0080] Figure 3 FIG. 3 shows a flow schematic diagram of the image rain removal method based on the expert model and the multi-stage rain streak extraction of the present application;

[0081] Figure 4 A schematic diagram of the active focusing module of an image deraining method based on expert models and multi-stage rain pattern extraction according to the present invention is shown;

[0082] Figure 5 This diagram illustrates a scale-aware convolutional attention module of an image deraining method based on an expert model and multi-stage rain pattern extraction according to the present invention.

[0083] Figure 6 This diagram illustrates a detail restoration module of an image deraining method based on an expert model and multi-stage rain pattern extraction according to the present invention.

[0084] Figure 7 The diagram shows a block diagram of an image deraining system based on an expert model and multi-stage rain pattern extraction according to the present invention. Detailed Implementation

[0085] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0086] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0087] Figure 1 The flowchart of an image deraining method based on an expert model and multi-stage rain pattern extraction according to this application is shown.

[0088] like Figure 1 As shown, this application discloses an image deraining method based on expert models and multi-stage rain ripple extraction, including the following steps:

[0089] S102, preprocess the input rainy image to obtain a separated image, and input the separated image into the trained rain removal model for processing;

[0090] S104, The separated image is input into the hybrid rain pattern prior module for processing to obtain hybrid prior features;

[0091] S106, The hybrid prior features are input into the multi-stage rain feature extraction module to perform stepwise feature extraction to obtain rain ripple layer features;

[0092] S108, based on the rainy image and the rain streak layer features, the detail restoration enhancement module is used to restore image details to generate the final de-raining image.

[0093] It should be noted that, in this embodiment, the application of a trained rain removal model for image rain removal is specifically described. By dynamically fusing multiple rain pattern priors and progressively recursively extracting rain pattern features, and combining a detail restoration mechanism, the efficient removal of complex rain patterns and accurate restoration of the background image are achieved. This method is efficient and robust, and is suitable for various complex rainy weather scenarios.

[0094] Specifically, such as Figure 2 The diagram shows the rain removal process. The input rainy image is preprocessed to obtain a separated image. Specifically, background information and rain streak features are initially separated from the image. This separated image is then input into a trained rain removal model for further processing. During processing, the separated image is input into a hybrid rain streak prior module to obtain hybrid prior features. Multiple rain streak priors, including dynamic color prior, gradient prior, and Gaussian blur prior, are adaptively fused using a dynamic gating mechanism. This allows for dynamic adjustment of the weight of each prior based on local rain streak density and background complexity. The dynamic color prior captures the brightness and color variation features of high-density rain areas; the gradient prior extracts high-frequency linear features from sparse rain areas; and the Gaussian blur prior is used to process blurred rain streak features in complex backgrounds, thus suppressing interference while preserving background information. Finally, the integrated hybrid prior features are output as input to the subsequent multi-stage rain feature extraction module.

[0095] Furthermore, the hybrid prior features are input into a multi-stage rain feature extraction module for stepwise feature extraction to obtain rain ripple layer features. Specifically, rain ripple features are extracted stepwise through a multi-stage recursive structure. First, a multi-dilated channel attention block (MDCAB) is used to extract shallow rain ripple features. Then, an active focus module (AFM) prioritizes visually significant rain ripple regions to improve the accuracy of feature extraction. In addition, a scale-aware convolutional attention module (SACAM) captures rain ripple features at different scales through multi-scale convolution operations and combines channel attention mechanisms to enhance feature representation capabilities. Finally, after stepwise feature extraction in the recursive stage, rain ripple layer features are obtained and output.

[0096] Further, based on the rainy image and the rain streak feature, an image detail recovery enhancement module is used to recover image details to generate a final rain-removed image. Specifically, by receiving the rain streak feature and the original input rainy image, the detail information in the background image is further recovered, the background texture details are gradually recovered through a series of recovery blocks (RB), residual connection is used to reduce the over-smoothing effect, and the recovered background features are ensured to be clear and real. After a multi-stage processing, a final rain-removed image is obtained.

[0097] It should be noted that in the present application, the network includes three branches in total, namely a mixture rain of prior experts (MRP) branch, a multi-stage rain feature extraction (RFE) branch, and a detail recovery enhancement (DRE) branch. In the MRP branch, the mixture of prior experts (MPE) module is used to mix multiple rain streak priors. The MPE module integrates multiple rain streak priors through a dynamic gating mechanism. The mechanism can adaptively adjust the contribution of each prior according to the input image features. The dynamic gating mechanism is crucial for effectively combining these priors, as it determines the weight distribution of the dynamic color prior, gradient prior, and Gaussian blur prior at different spatial positions of the image. Specifically, the gating network receives the input image features and learns to generate a set of spatially varying weights, which correspond to the weights of the dynamic color prior, gradient prior, and Gaussian blur prior, respectively. In this way, the model can flexibly adjust the influence of each prior to adapt to the changes in image local content and rain streak density in different regions. The outputs of each prior and its corresponding weight are weighted through element-wise multiplication, and then the weighted results are added to generate the final prior enhanced feature representation. This feature representation combines the advantages of each prior to achieve better feature expression capability in a coordinated manner. Through the method of the present application, the model can selectively focus on the areas affected by rain streaks while suppressing background noise, thereby significantly improving the rain removal accuracy.

[0098] According to an embodiment of the present application, the method further comprises constructing and training the rain removal model, specifically comprising:

[0099] A plurality of groups of unpaired rainy images and rain-free images are acquired and preprocessed to obtain training images;

[0100] input the training image into a hybrid rain streak prior module for training to obtain a hybrid prior feature, wherein the hybrid rain streak prior module comprises a hybrid prior expert module, and a dynamic gating mechanism is used to adaptively fuse multiple rain streak priors through decision of an expert network, specifically including a dynamic color prior, a gradient prior and a Gaussian blur prior;

[0101] input the hybrid prior feature into a multi-stage rain feature extraction module for rain streak feature extraction and refinement training to obtain a rain streak layer feature, wherein the multi-stage rain feature extraction module comprises a multi-channel attention block, an active focusing module and a scale-aware convolution attention module, and the rain streak layer feature is obtained by gradually extracting and refining the rain streak feature and separating the rain streak and the background image layer by layer;

[0102] based on the rainy image and the rain streak layer feature, a detail recovery enhancement module is used for training to recover image details, wherein the detail recovery enhancement module integrates a multi-channel attention block, a scale-aware convolution attention module, a batch normalization and a PReLU activation module, and multi-scale and multi-channel processing is used to retain the original depth and clarity of the background image;

[0103] during the training process, the learning rate and the training number of rounds are distinguished based on the rain streak type, wherein the learning rate of sparse rain streaks is , the training number of rounds of sparse rain streaks is 150, the learning rate of dense complex rain streaks is , the training number of rounds of dense complex rain streaks is 100, and a composite loss function is used, which includes a structural similarity loss function, an edge loss function and a preset loss function, wherein the preset loss function includes a Charbonnier loss function and an improved version of the loss function.

[0104] It should be noted that, in the present embodiment, as shown in Figure 3 , a flow chart is displayed, first, a plurality of groups of unpaired rainy images and rain-free images are acquired and preprocessed to obtain training images, wherein in addition to shooting images to collect data, ready-made synthetic data sets such as Rain100L and Rain100H data sets can be used instead, but when preprocessing, the images of the two domains are shuffled, because the unsupervised learning method is used in the present embodiment, unpaired images are used for model training during training, and then the images are preprocessed, wherein the preprocessing procedure is as follows: first, the images in the collected data set are resized, the image size is uniformly converted to 256*256*3, converted to a tensor, and finally the image data is converted to a standard normal distribution, so that the model is more easily converged.

[0105] Further, according to the embodiment of the present application, the training image is input into the hybrid rain streak prior module to obtain the hybrid prior feature, specifically including:

[0106] give a point prior expert network: ;

[0107] output of the mixed prior expert module wherein, weight of the i-th expert network, generated by the gating network; output of the i-th expert network, with weight The calculation formula of the weight is wherein, the dynamic color prior, the gradient prior and the Gaussian blur prior are respectively taken as the corresponding weight . It should be noted that in the embodiment, for a given input with a rain map The output of the MPE module is determined by the weighted sum of the outputs of the expert networks, wherein the weights are determined by the output of the gating network, given

[0108] prior expert network: The output of the MPE module can be represented as: wherein, weight of the i-th expert network, generated by the gating network; output of the i-th expert network, with weight The calculation formula of the weight is Further, the above dynamic color prior is an effective prior method designed for rain streak features, which is used to capture the brightness and color change characteristics of high-density rain areas. By analyzing the influence of rain streaks on image brightness and color, the input image is converted into a gray-scale image to enhance the brightness change, and a trainable threshold is used to segment the rain streak significant area. In addition, the prior strengthens the dense rain streak area through the max-pooling operation, ensures that the rain streak features are accurately extracted, and reduces the interference of background information. The dynamic color prior has strong adaptability, and its parameters can be dynamically adjusted according to the rain streak density and distribution, so as to adapt to different rainy scenes. In the present application, the dynamic color prior, the gradient prior and the Gaussian blur prior are respectively taken as the expert function By dynamically adjusting the weight, the accurate capture and optimization processing of different rain streak features are realized.

[0109] Further, the above dynamic color prior is an effective prior method designed for rain streak features, which is used to capture the brightness and color change characteristics of high-density rain areas. By analyzing the influence of rain streaks on image brightness and color, the input image is converted into a gray-scale image to enhance the brightness change, and a trainable threshold is used to segment the rain streak significant area. In addition, the prior strengthens the dense rain streak area through the max-pooling operation, ensures that the rain streak features are accurately extracted, and reduces the interference of background information. The dynamic color prior has strong adaptability, and its parameters can be dynamically adjusted according to the rain streak density and distribution, so as to adapt to different rainy scenes. In the present application, the dynamic color prior, the gradient prior and the Gaussian blur prior are respectively taken as the expert function By dynamically adjusting the weight, the accurate capture and optimization processing of different rain streak features are realized.

[0110] ​​Further, the dynamic gating mechanism in the embodiment enables the MPE module to balance the contributions of each prior, optimize the accuracy of rain streak detection and the effect of background separation, and the proposed framework exhibits strong robustness in various rainy scenarios, including from dense rainy days to sparse rainy scenarios with complex backgrounds.

[0111] Further, according to the embodiment of the present application, the mixed prior feature is input into a multi-stage rain feature extraction module for rain streak feature extraction and refinement training to obtain rain streak layer features, specifically including:

[0112] The multi-channel attention block is used to preliminarily extract the shallow features of the rain streaks, the multi-channel attention mechanism is used to process the rain streak-related features in the mixed prior feature, and irrelevant background noise is suppressed;

[0113] The active focusing module is used to process the rain streak removal key area based on the pixel-level attention mechanism;

[0114] The multi-scale convolution of the scale perception convolution attention module captures multi-scale features, and the attention mechanism of the scale perception convolution attention module assigns different weights to each channel.

[0115] It should be noted that in the embodiment, the multi-stage rain feature extraction module (RFE) adopts a multi-stage recursive design, which gradually extracts and refines the rain streak features and separates the rain streaks and background information layer by layer. The input of the RFE module includes the fusion features from the original image and the prior module (MRP branch), and the output is the refined rain streak features and the preliminary restored background image. Accordingly, the RFE branch includes the following parts:

[0116] (1) Multi-channel attention block (MDCAB): used to preliminarily extract the shallow features of the rain streaks, and the multi-channel attention mechanism is used to enhance the rain streak-related features and suppress irrelevant background noise.

[0117] (2) Active focusing module (AFM): this is the core innovative part of the RFE module, as shown in Figure 4 , based on the pixel-level attention mechanism, the architecture of AFM (active focusing module) aims to calculate the mutual dependence between features and assign appropriate weights to them, so that the network can focus on the most critical areas for rain streak removal. The operation framework of AFM first concatenates the input features with the hidden state of the previous stage: wherein, represents the current stage. Then, the pixel attention operation is applied to calculate the pixel attention map : Subsequently, the input to each gating unit can be calculated using the following formula: Here, the symbol ⊗ represents an element-wise multiplication operation between two matrices. The gating units in AFM are calculated in the same way as those in LSTM, as follows: In this formula, and They represent the first The phase's unit state and hidden state, output The rain streak is then passed as input to the next stage module. Through the AFM module, the model can more effectively focus on areas with significant rain streak visual effects, thereby improving the accuracy and effectiveness of rain streak removal.

[0118] Furthermore, the Scale-Aware Convolutional Attention Module (SACAM), considering the different scale characteristics of rain streaks, extracting multi-scale features is crucial for effectively processing rain streaks, such as... Figure 5 As shown, the proposed Scale-Aware Convolutional Attention Module (SACAM) comprises two main components: multi-scale convolution and channel attention. The multi-scale convolution consists of two convolutional layers with different kernel sizes, and the outputs of each layer are concatenated to fuse features from different receptive fields. By utilizing different kernel sizes, SACAM can capture multi-scale features, enhancing its ability to process diverse rain pattern patterns. Furthermore, the fully connected data flow between layers ensures the full fusion of features from different scales. Following multi-scale convolution, the channel attention mechanism further optimizes the final output by assigning different weights to each channel to highlight the most relevant features, refining the extracted features and improving the processing performance for multi-scale rain patterns.

[0119] According to an embodiment of the present invention, a detail restoration enhancement module is trained based on the rainy image and the rain streak layer features to restore image details, specifically including:

[0120] The detail recovery enhancement module includes an initial convolutional layer, four recovery blocks, and a final convolutional layer, wherein...

[0121] Based on the initial convolutional layer, basic features are extracted;

[0122] Each recovery block integrates a multi-channel attention block, a scale-aware convolutional attention module, a batch normalization and PReLU activation module, and gradually recovers the global structure and detailed features based on the recovery block;

[0123] Based on the final convolutional layer, feature refinement and background refinement are performed on the data after the recovery block is processed.

[0124] It should be noted that in the embodiment, the detail recovery enhancement branch aims to recover the fine-grained background details blocked by rain streaks, wherein the traditional rain removal method can cause over-smoothing of the background features, and the DRE branch focuses on preserving fine textures and structures in the rain-removed image.

[0125] Specifically, the DRE branch gradually enhances the background features through multi-scale and multi-channel processing. First, it uses an initial convolutional layer to extract basic features, and then gradually recovers global structure and detail features through four recovery blocks (RBs), as shown in Figure 6 Each recovery block integrates a multi-channel attention block (MDCAB), a scale-aware convolutional attention module (SACAM), and a batch normalization (BN) and PReLU activation module (BN+PReLU). After the RBs sequence, the input features are added to the output of the recovery block using a residual connection to preserve key information and stabilize the learning process. The last convolutional layer further refines the features to complete the background refinement process. Thus, the over-smoothing effect can be minimized while preserving delicate textures, achieving a balanced rain removal process. The DRE branch not only effectively removes rain streaks but also preserves the original depth and clarity of the background, ensuring high-quality presentation of the rain-removed image.

[0126] Further, according to an embodiment of the present application, the structural similarity loss function is as follows:

[0127] ;

[0128] The Charbonnier loss function is as follows:

[0129] ;

[0130] The edge loss function is as follows:

[0131] ;

[0132] The improved version of the Charbonnier loss function corresponding to the Charbonnier loss is as follows:

[0133] ;

[0134] wherein, is the network output, , is the rain streak output predicted by the RSE branch, is the detail image recovered by the BDR branch, is the original rain-containing training image, ;

[0135] The composite loss function is as follows:

[0136]

[0137] Wherein, the parameter , the parameter , the parameter .

[0138] It should be noted that in the present embodiment, the rain removal model uses the Pytorch framework to build the network, and all experiments are completed on a NVIDIA RTX 3090Ti GPU, and in terms of data sets, Rain100L, Rain100H and Rain800 commonly used in image rain removal tasks are used, and for the training process, the learning rate and the number of training rounds are set according to different data sets, specifically, for the Rain100L data set containing sparse rain streaks, the learning rate is set to , and the number of training rounds is set to 150; for the Rain100H data set containing dense and complex rain streaks, the learning rate is set to , the number of training rounds is set to 100, and the Adam optimizer is used, based on the cosine annealing strategy, all the above learning rates will eventually decrease steadily to , in the hyperparameter setting of the loss function, the present application sets , , By inputting a plurality of paired rainy and non-rainy images into the fused model for training, the reconstruction results of each rain-removed image are obtained, and after the model training is completed, the rainy image is input into the trained rain removal model for processing to obtain the rain-removed image.

[0139] Figure 7 A block diagram of an image rain removal system based on an expert model and multi-stage rain streak extraction is shown.

[0140] As shown in Figure 7 , the present application discloses an image rain removal system based on an expert model and multi-stage rain streak extraction, comprising a memory and a processor, the memory comprising an image rain removal method based on an expert model and multi-stage rain streak extraction program, the image rain removal method based on an expert model and multi-stage rain streak extraction program is executed by the processor to realize the following steps:

[0141] The input rainy image is preprocessed to obtain a separated image, and the separated image is input into the trained rain removal model for processing, wherein

[0142] The separated image is input into the mixed rain streak prior module for processing to obtain mixed prior features;

[0143] ​The mixed prior feature is input into a multi-stage rain feature extraction module for step-by-step feature extraction to obtain rain layer features.

[0144] Based on the rainy image and the rain layer features, an image detail recovery enhancement module is used to recover image details to generate a final rain-removed image.

[0145] It should be noted that, since the specific implementation mode of the embodiment corresponds to the foregoing method embodiment, the same details will not be repeated here, and those skilled in the art should understand that the division of each execution step in the embodiment is only a logical functional division, and in actual implementation, all or part of the execution steps can be integrated onto one or more physical entities, and the execution steps can all be implemented in the form of software calling by a processing element, or all can be implemented in the form of hardware, or part of the execution steps can be implemented in the form of software calling by a processing element, and part of the execution steps can be implemented in the form of hardware.

[0146] The third aspect of the application provides a computer-readable storage medium, wherein the computer-readable storage medium comprises a rain-removed image method program based on an expert model and multi-stage rain feature extraction, and the rain-removed image method program based on the expert model and the multi-stage rain feature extraction is executed by a processor to implement the steps of the rain-removed image method based on the expert model and the multi-stage rain feature extraction according to any one of the foregoing aspects.

[0147] The rain-removed image method and system based on the expert model and the multi-stage rain feature extraction disclosed in the application include training of a rain-removed model and image rain removal using the rain-removed model, can realize accurate separation of rain feature and fine recovery of background details by using adaptive prior combined with a recursive extraction strategy, can effectively process diversified rain patterns and complex backgrounds, and improves the accuracy and robustness of image rain removal.

[0148] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only illustrative, for example, the division of the units is only a logical functional division, and in actual implementation, there can be another division mode, for example, multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling or communication connection between the displayed or discussed components can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0149] The units described as separate components above can or can not be physically separate, and the components shown as units can or can not be physical units; they can be located in one place or distributed on multiple network units; and part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0150] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0151] Those of ordinary skill in the art can understand that all or part of the steps of the above method embodiments can be completed by relevant hardware instructed by programs, and the foregoing programs can be stored in a computer readable storage medium, and when the programs are executed, steps including the above method embodiments are executed; and the foregoing storage medium includes mobile storage devices, read-only memories (ROMs), random access memories (RAMs), magnetic discs or optical discs, and various media that can store program codes.

[0152] Alternatively, the integrated units of the present application, if implemented in the form of software functional modules and sold or used as independent products, can also be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of software products, and the computer software products are stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the embodiments of the present application. The foregoing storage medium includes mobile storage devices, ROMs, RAMs, magnetic discs or optical discs, and various media that can store program codes.

Claims

1. An image deraining method based on expert model and multi-stage rain streak extraction, characterized in that, The method comprises the following steps: The input rain image is preprocessed to obtain a separated image, and the separated image is input into a trained rain removal model for processing, wherein The separated image is input into a mixed rain streak prior module for processing to obtain mixed prior features; The mixed prior features are input into a multi-stage rain feature extraction module for step-by-step feature extraction to obtain rain streak layer features; An image detail recovery enhancement module is used to recover image details based on the rain image and the rain streak layer features to generate a final rain-removed image; The method further comprises constructing and training the rain removal model, specifically comprising: Obtain a plurality of groups of unpaired rain images and non-rain images taken, and pre-process the rain images and the non-rain images to obtain training images; The training images are input into a mixed rain streak prior module for training to obtain mixed prior features, wherein the mixed rain streak prior module includes a mixed prior expert module, which uses a dynamic gating mechanism to adaptively fuse multiple rain streak priors through the decision of the expert network, specifically including dynamic color prior, gradient prior and Gaussian blur prior; The mixed prior features are input into a multi-stage rain feature extraction module for rain streak feature extraction and refinement training to obtain rain streak layer features, wherein the multi-stage rain feature extraction module includes a multi-channel attention block, an active focusing module and a scale-aware convolution attention module, which extracts and refines rain streak features step by step, and separates rain streaks and background images layer by layer to obtain the rain streak layer features; An image detail recovery enhancement module is used to recover image details based on the rain image and the rain streak layer features, wherein the detail recovery enhancement module integrates a multi-channel attention block, a scale-aware convolution attention module, a batch normalization and a PReLU activation module, which uses multi-scale and multi-channel processing to retain the original depth and clarity of the background image. The active focusing module first concatenates the input features with the hidden state of the previous stage : where denotes the current stage; then, the pixel attention operation is applied to compute the pixel attention map : Subsequently, the input of each gating unit is computed by the following equation: where the symbol denotes the element-wise multiplication operation between two matrices; each gating unit in AFM is calculated in the same way as LSTM, as follows: In this equation, and denote the cell state and hidden state of the stage, respectively, and the output will be passed as input to the module of the next stage; The scale-aware convolution attention module includes two components: multi-scale convolution and channel attention. The multi-scale convolution part is composed of two convolution layers with different kernel sizes. The outputs of each layer are integrated together through a concatenation operation to fuse features from different receptive fields.

2. The image deraining method based on expert model and multi-stage rain streak extraction according to claim 1, characterized in that, The learning rate and the training round number in the training process are distinguished based on the rain streak type, wherein the learning rate of the sparse rain streak is , the training round number of the sparse rain streak is 150, the learning rate of the dense complex rain streak is , the training round number of the dense complex rain streak is 100, and the composite loss function used includes a structural similarity loss function, an edge loss function and a preset loss function, wherein the preset loss function includes a Charbonnier loss function and an improved version thereof.

3. The image deraining method based on expert model and multi-stage rain streak extraction according to claim 2, characterized in that, The training images are input into a mixed rain streak prior module for training to obtain mixed prior features, specifically comprising: To give ; output of the mixed prior expert module wherein, denotes the weight of the th expert network, generated by the gating network; denotes the output of the th expert network, with weight The calculation formula of wherein, the dynamic color prior, the gradient prior and the Gaussian blur prior are respectively taken as the corresponding weight .

4. The image deraining method based on expert model and multi-stage rain streak extraction according to claim 3, characterized in that, The mixed prior features are input into a multi-stage rain feature extraction module for rain streak feature extraction and refinement training to obtain rain streak layer features, specifically comprising: A multi-channel attention block is used to preliminarily extract shallow features of rain streaks, and a multi-channel attention mechanism is used to process rain streak-related features in the mixed prior features and suppress irrelevant background noise; An active focusing module is used to process rain streak removal key areas based on a pixel-level attention mechanism; A multi-scale convolution of a scale-aware convolution attention module is used to capture multi-scale features, and an attention mechanism of the scale-aware convolution attention module is used to assign different weights to each channel.

5. The image deraining method based on expert model and multi-stage rain streak extraction according to claim 4, characterized in that, An image detail recovery enhancement module is used to recover image details based on the rain image and the rain streak layer features, specifically comprising: The detail recovery enhancement module includes an initial convolution layer, four recovery blocks and a final convolution layer, wherein Basic features are extracted based on the initial convolution layer; Each recovery block integrates a multi-channel attention block, a scale-aware convolution attention module, a batch normalization and a PReLU activation module, and based on the recovery block, global structure and detail features are gradually recovered; Based on the last convolution layer, the data processed by the recovery block is subjected to feature refinement and background refinement.

6. The image deraining method based on expert model and multi-stage rain streak extraction according to claim 5, characterized in that, The structural similarity loss function is as follows: ； The Charbonnier loss function is as follows: ； The edge loss function is as follows: ； The improved version of the Charbonnier loss function is as follows: ； wherein, is the network output, , is the rain streak output for RFE branch prediction, is the detail image for DRE branch recovery, is the original rain-containing training image, ; The composite loss function is as follows: ； wherein the parameters , the parameters , the parameters .

7. An image deraining system based on expert model and multi-stage rain streak extraction, characterized in that, The image rain removal method based on an expert model and multi-stage rain streak extraction comprises the following steps: The input rainy image is preprocessed to obtain a separated image, and the separated image is input into a trained rain removal model for processing, wherein, The separated image is input into a hybrid rain streak prior module for processing to obtain hybrid prior features; The hybrid prior features are input into a multi-stage rain feature extraction module for step-by-step feature extraction to obtain rain streak layer features; Based on the rainy image and the rain streak layer features, a detail recovery enhancement module is used to recover image details to generate a final rain-removed image; The method further comprises constructing and training the rain removal model, specifically comprising: A plurality of groups of unpaired rainy images and rain-free images are obtained by shooting and preprocessed to obtain training images; The training images are input into a hybrid rain streak prior module for training to obtain hybrid prior features, wherein the hybrid rain streak prior module comprises a hybrid prior expert module, which uses a dynamic gating mechanism to adaptively fuse multiple rain streak priors through the decision of an expert network, specifically including a dynamic color prior, a gradient prior and a Gaussian blur prior; The hybrid prior features are input into a multi-stage rain feature extraction module for rain streak feature extraction and refinement training to obtain rain streak layer features, wherein the multi-stage rain feature extraction module comprises a multi-channel attention block, an active focusing module and a scale-aware convolution attention module, which gradually extracts and refines rain streak features and separates rain streaks and background images layer by layer to obtain the rain streak layer features; Based on the rainy image and the rain streak layer features, a detail recovery enhancement module is used for training to recover image details, wherein the detail recovery enhancement module integrates a multi-channel attention block, a scale-aware convolution attention module, a batch normalization and a PReLU activation module, and through multi-scale and multi-channel processing, the original depth and clarity of the background image are preserved. The active focusing module first concatenates the input features with the hidden state of the previous stage : where, denotes the current stage; then, the pixel attention operation is applied to compute the pixel attention map : Subsequently, the input of each gating unit is computed by the following equation: where the symbol denotes the element-wise multiplication operation between two matrices; each gating unit in AFM is calculated in the same way as LSTM, which is as follows: In this equation, and denote the cell state and hidden state of the th stage, respectively, and the output will be passed as input to the module of the next stage; The scale-aware convolution attention module comprises two components: multi-scale convolution and channel attention. The multi-scale convolution part is composed of two convolution layers with different kernel sizes, and the outputs of each layer are integrated together through a concatenation operation to fuse features from different receptive fields.

8. The image deraining system based on expert model and multi-stage rain streak extraction of claim 7, wherein, The learning rate and the training round number in the training process are distinguished based on the rain streak type, wherein the learning rate of the sparse rain streak is , the training round number of the sparse rain streak is 150, the learning rate of the dense complex rain streak is , the training round number of the dense complex rain streak is 100, and the composite loss function used includes a structural similarity loss function, an edge loss function and a preset loss function, wherein the preset loss function includes a Charbonnier loss function and an improved version thereof.

9. The image deraining system based on expert model and multi-stage rain streak extraction of claim 8, wherein, The training of the training images into the hybrid rain streak prior module to obtain hybrid prior features specifically comprises: To a priori expert network: ; output of the mixture prior expert module wherein, denotes the weight of the th expert network, generated by the gating network; denotes the output of the th expert network, with weight The calculation formula of wherein, the dynamic color prior, the gradient prior and the Gaussian blur prior are respectively taken as the corresponding weight .

10. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises an image rain removal method program based on an expert model and multi-stage rain streak extraction, and the image rain removal method program based on the expert model and the multi-stage rain streak extraction is executed by the processor to realize the steps of the image rain removal method based on the expert model and the multi-stage rain streak extraction in any one of claims 1 to 6.

Images