Image deraining method and system based on multi-dimensional aggregation network for rain line feature reconstruction

Abstract

The present invention discloses an image deraining method and system based on a multidimensional aggregation network for rain line feature reconstruction. The method comprises: first, obtaining an image dataset from a database, performing image preprocessing, and applying a rain line effect to the obtained image; second, constructing a multidimensional aggregation network based on rain line feature reconstruction and extraction, including modules such as a deep-domain feature reconstruction module and a wide-domain feature reconstruction module. The network is fed with rain-affected images as input, multidimensional aggregation features are obtained, and after passing through the image reconstruction module, a derained image is output; third, the network is trained to obtain a trained multidimensional aggregation network; and finally, the rain-affected image to be processed is fed into the trained multidimensional aggregation network based on rain line feature reconstruction and extraction, outputting a derained image. The proposed multidimensional aggregation network is more efficient, and its feature module can learn the complex and variable morphology of rain lines to extract information at different scales in the image, thereby maximizing the preservation of image details and achieving better visual effects.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, image processing, and machine learning, and in particular to an image deraining method, system, device, and storage medium based on a multidimensional aggregation network for rain line feature reconstruction. Background Art

[0002] In recent years, with the rapid development of smartphones, it has become increasingly convenient to use a mobile phone for taking photos anytime, anywhere. However, images taken on rainy days often contain varying degrees of rain lines, which can obscure and blur the background, resulting in unclear images. Furthermore, the influence of rain lines often reduces the accuracy of some advanced computer vision algorithms, such as image semantic segmentation and object detection. Therefore, to obtain clearer images, rainy images require processing to eliminate the influence of rain lines. Traditional image processing methods were initially used to address the problem of image de-raining. These methods typically employ optimization models combined with prior knowledge of rainy images to remove rain lines by separating them from the background. However, the direction and shape of rain lines in real rainy images are complex and varied, and most images differ from the prior information. Therefore, these methods often struggle to fully represent rain lines, resulting in incomplete rain line removal.

[0003] With the continuous development of computing and deep learning in recent years, deep learning methods have been effectively applied in computer vision and image processing. This has led to the emergence of deep learning-based image deraining methods. These methods primarily input rainy images to be processed, train neural networks, and directly output the derained images end-to-end. Their performance has achieved considerable improvement over traditional methods. However, due to limitations in AI-related technologies, the output derained images still exhibit residual rain lines, texture loss, and artifacts.

[0004] One existing technique is a generative adversarial network (GAN)-based image deraining method. This method utilizes a GAN architecture and employs a combination of conditional adversarial loss, content loss, and perceptual loss to train the network model. The trained generator then directly generates derained images. However, its shortcomings include: first, the GAN-based approach makes training difficult; second, it fails to consider multi-scale information in rainy images, which can easily result in residual rain lines and loss of texture details.

[0005] The second existing technology is a neural network-based image deraining method. This method uses a deep residual network to extract features of rainy images and output the deraining results. Its shortcomings are: first, convolutional neural networks can only extract local information from the image. This method does not consider information across the entire spatial region of the image, and cannot effectively distinguish between rainy and rain-free areas. This can easily lead to residual rain lines in rainy areas and over-smoothing of rain-free areas. Second, features from each channel are indiscriminately aggregated, which can easily lead to the loss of important features during the transmission process in the deep network. Summary of the Invention

[0006] This invention aims to overcome the shortcomings of existing methods by proposing an image deraining method based on a multidimensional aggregation network for rain line feature reconstruction. The main issues addressed by this invention are: first, model selection and training issues, as well as the inability to distinguish between rainy and rain-free areas due to a lack of consideration of information across the entire spatial region of the image; second, the failure to consider multi-scale information in rainy images, resulting in residual rain lines and loss of texture detail; and third, feature extraction and information loss during transmission in each channel.

[0007] In order to solve the above problems, the present invention proposes an image deraining method based on a multi-dimensional aggregation network for rain line feature reconstruction, the method comprising:

[0008] Obtain image datasets from the database, perform image preprocessing, add rain line effects to the acquired images, and construct training and test sets;

[0009] Construct a deep-area feature reconstruction module and a wide-area feature reconstruction module, input the features to be reconstructed, perform channel reconstruction and spatial reconstruction respectively, and output deep-area reconstructed features and wide-area reconstructed features respectively;

[0010] Constructing a rain line feature reconstruction and extraction module, which is composed of the deep-area feature reconstruction module and the wide-area feature reconstruction module, splicing the deep-area reconstructed features and the wide-area reconstructed features along the channel dimension, and performing operations such as convolution to obtain intermediate features;

[0011] Constructing a multidimensional feature aggregation module to perform multidimensional aggregation on the input intermediate features and output the aggregated multidimensional aggregated features;

[0012] A multidimensional aggregation network based on rain line feature reconstruction and extraction is constructed, which mainly consists of a convolution block, the rain line feature reconstruction and extraction module, the multidimensional feature aggregation module, and an image reconstruction module. The input is a rainy image, the multidimensional aggregation features are obtained, and the image reconstruction module outputs a rain-free image;

[0013] Using the training set, using multiple losses to train the multidimensional aggregation network based on rain line feature reconstruction and extraction, and using the test set to perform testing and verification, to obtain a trained multidimensional aggregation network based on rain line feature reconstruction and extraction;

[0014] The rainy image to be processed is input into the trained multi-dimensional aggregation network based on rain line feature reconstruction and extraction, and a rain-free image is output.

[0015] Preferably, the method of obtaining an image dataset from a database, performing image preprocessing, adding a rain line effect to the obtained image, and constructing a training set and a test set is as follows:

[0016] Randomly obtain 1000 images from the SUN database as an image dataset and perform image preprocessing;

[0017] Use Photoshop to create dynamic blur filters, setting the blur angles to 70, 80, 90, 100, and 110 degrees, and the distances to 30, 40, and 50 pixels, creating a total of 15 filters that mimic different rain line effects.

[0018] Photoshop was used to apply a filter to each image in the dataset, resulting in 15 images with rain streaks, totaling 15,000 rain-no-rain image pairs. From these, 12,000 pairs were randomly sampled, and approximately 2 million 64×64 rain-no-rain image patches were cropped to form the training set.

[0019] The remaining 3000 image pairs are used as the test set.

[0020] Preferably, the deep-area feature reconstruction module and the wide-area feature reconstruction module are constructed, and the features to be reconstructed are input for channel reconstruction and spatial reconstruction respectively, and the deep-area reconstructed features and the wide-area reconstructed features are output respectively, specifically:

[0021] Construct a deep domain feature reconstruction module (DFRM). The module uses a squeeze-and-extraction network structure. Its input is the H×W×C dimensional feature to be reconstructed. Each channel of the feature is assigned a corresponding weight, and the output is the deep domain reconstructed feature.

[0022] The features of each channel of the feature to be reconstructed are average pooled to obtain a 1×1×C-dimensional feature. Then, it passes through a fully connected layer, a ReLU layer, a fully connected layer, and a sigmoid layer in sequence to output a 1×1×C-dimensional deep domain weight. This weight is multiplied by the feature to be reconstructed to obtain the deep domain reconstructed feature.

[0023] Construct a wide-area feature reconstruction module (WFRM), whose input is the H×W×C-dimensional features to be reconstructed and output is the wide-area reconstructed features;

[0024] Calculate the average value of the channel dimension for the feature to be reconstructed to obtain a feature of H×W×1 dimension, and then perform a 1×1 convolution on the feature to be reconstructed to obtain another feature of H×W×1 dimension;

[0025] The two obtained H×W×1-dimensional features are spliced ​​along the channel dimension to obtain H×W×2-dimensional features, and then pass through 1×1 convolution, ReLU layer, 1×1 convolution and sigmoid layer in sequence to output H×W×1-dimensional wide-area weights. Finally, this weight is multiplied by the feature to be reconstructed to obtain the wide-area reconstructed feature.

[0026] Preferably, the rain line feature reconstruction and extraction module is composed of the deep-area feature reconstruction module and the wide-area feature reconstruction module. The deep-area reconstructed features and the wide-area reconstructed features are spliced ​​along the channel dimension and subjected to convolution and other operations to obtain intermediate features, specifically:

[0027] Constructing a rain line feature reconstruction and extraction module (RFREM), which is mainly composed of the deep-area feature reconstruction module and the wide-area feature reconstruction module, and adopts a residual structure;

[0028] The rain line feature reconstruction and extraction module takes as input features of H×W×C dimensions and outputs intermediate features of H×W×C dimensions. The dilation rate of the module's dilated convolution is r. Using different dilation rates can obtain feature information at different scales of the image.

[0029] The input H×W×C dimensional features are reconstructed by the deep-area feature reconstruction module and the wide-area feature reconstruction module respectively to obtain deep-area reconstructed features and wide-area reconstructed features;

[0030] The deep domain reconstruction features and the wide domain reconstruction features are spliced ​​along the channel dimension to obtain H×W×(2×C) dimensional features, and then sequentially pass through the dilated convolution with a convolution kernel size of 3×3 and a dilation rate of r, the ReLU layer, the batch normalization layer (BatchNormalization) and the dilated convolution with a convolution kernel size of 3×3 and a dilation rate of r to output H×W×C dimensional features. Finally, they are added to the input H×W×C dimensional features to output the intermediate features.

[0031] Preferably, the multi-dimensional feature aggregation module is constructed to perform multi-dimensional aggregation on the input intermediate features and output the aggregated multi-dimensional aggregated features, specifically:

[0032] Construct a Multi Feature Aggregation Module (MFAM), whose input is n different H×W×C dimensional intermediate features and output is a multi-dimensional aggregated feature of H×W×(n×C) dimensions after multi-dimensional aggregation operation;

[0033] For each input H×W×C dimensional intermediate feature, it first undergoes a 1×1 convolution to obtain a H×W×(C / n) dimensional feature, and then undergoes an average pooling operation to obtain a 1×1×(C / n) dimensional feature;

[0034] These n 1×1×(C / n)-dimensional features are concatenated along the channel to obtain a 1×1×C-dimensional feature, which is then passed through a fully connected layer, a ReLU layer, a fully connected layer, and a sigmoid layer to obtain a 1×1×n-dimensional feature weight, which is then expanded to 1×1×(n×C) dimensions.

[0035] The n H×W×C dimensional features initially input are concatenated along the channel to obtain H×W×(n×C) dimensional features, which are multiplied by the feature weights obtained previously, and then pass through 1×1 convolution and ReLU layers in sequence to output H×W×(n×C) dimensional multidimensional aggregated features.

[0036] Preferably, the multidimensional aggregation network based on rain line feature reconstruction and extraction is mainly composed of a convolution block, the rain line feature reconstruction and extraction module, the multidimensional feature aggregation module and an image reconstruction module, and its input is a rain image and its output is a rain-free image, specifically:

[0037] The input H×W×3 dimensional rain image is processed through a feature extraction layer that includes a 3×3 convolution layer, a ReLU layer, a batch normalization layer, and a 3×3 convolution layer to obtain the initial features of H×W×24 dimensions;

[0038] The initial features are passed through three U-Net structured encoding and decoding networks to obtain three H×W×24-dimensional intermediate features. The encoding and decoding network consists of 5 encoder layers, 1 intermediate layer, and 5 decoder layers. There are skip connections between the encoders and decoders of the corresponding layers.

[0039] Its encoder, decoder and intermediate layer are all composed of the rain line feature reconstruction and extraction module. The expansion rates r in the rain line feature reconstruction and extraction module used by these three encoding and decoding networks are 1, 2, and 3 respectively to obtain feature information of different scales.

[0040] These three intermediate features obtained by different encoding and decoding networks are aggregated through the multidimensional feature aggregation module to obtain H×W×72-dimensional multidimensional aggregated features;

[0041] The obtained multi-dimensional aggregated features are passed through two image reconstruction modules consisting of 3×3 convolution, ReLU layer, and batch normalization layer for image reconstruction, and then a 3×3 convolution is performed to output the H×W×3 dimensional derained image.

[0042] Preferably, the multidimensional aggregation network based on rain line feature reconstruction and extraction is trained using the training set using multiple losses, and the test set is used for testing and verification to obtain a trained multidimensional aggregation network based on rain line feature reconstruction and extraction, specifically:

[0043] The multi-dimensional aggregation network based on rain line feature reconstruction and extraction is trained using multiple losses, where the multiple losses used in the training are the sum of pixel-level MSE loss, perceptual loss, and multi-stage gradient loss (MSG Loss);

[0044] The Adam algorithm was used to train the network using the training set, with 40 images per batch, until the loss function converged. The learning rate was initially set to 0.0001 during training, and decayed to 0.00001 at the 100,000th round and 0.000001 at the 150,000th round.

[0045] The multiple loss functions are expressed as follows:

[0046] L=L MSE +L per +L MSG

[0047] Among them L MSE represents pixel-level MSE loss, L per Represents the perceptual loss, L MSG Represents the multi-layer gradient loss of the image;

[0048] The pixel-level MSE loss function in multiple losses is expressed as follows:

[0049]

[0050] Where M is the total number of training data, Y i and Y gt,i The de-rained image and the true rain-free image are output by the multi-dimensional aggregation network based on rain line feature reconstruction and extraction, respectively. This loss directly measures the difference between the pixels of the two images;

[0051] The perceptual loss function in multiple losses is expressed as follows:

[0052]

[0053] Where M is the total number of training data, Yi and Y gt,i are the rain-free image and the real rain-free image output by the multi-dimensional aggregation network based on rain line feature reconstruction and extraction, respectively. i ) and F(Y gt,i ) is the result of the derained image and the real rain-free image being respectively obtained by the pre-trained VGG16 network. This loss measures the difference in the visual features of the two images;

[0054] The multi-layer gradient loss (MSG Loss) function in multiple losses is expressed as follows:

[0055]

[0056] in:

[0057]

[0058] To find the gradient operator, Y i The result obtained by downsampling k times, each downsampling with a window of 2 and a step size of 2 average pooling operation, will get the original image The downsampled image of the size is N=2 in training. This loss measures the difference in gradients between the two images, thereby measuring the difference in details and textures.

[0059] The test set is used for testing and verification to obtain a trained multi-dimensional aggregation network based on rain line feature reconstruction and extraction.

[0060] Accordingly, the present invention also provides an image deraining system based on a multi-dimensional aggregation network for rain line feature reconstruction, comprising:

[0061] The dataset construction unit obtains the image dataset from the database, performs image preprocessing, adds rain line effects to the obtained images, and constructs training and test sets;

[0062] Module construction and feature extraction unit, build rain line feature reconstruction and extraction module, composed of deep domain feature reconstruction module and wide domain feature reconstruction module, extract intermediate features, build multi-dimensional feature aggregation module, extract multi-dimensional aggregation features;

[0063] The network training and rain removal unit constructs a multi-dimensional aggregation network based on rain line feature reconstruction and extraction. Its input is a rainy image, and its output is a rain-removed image. The network is trained using a training set and multiple losses, and tested and verified using a test set to obtain a trained network. The rainy image to be processed is input into the trained network, and the rain-removed image is output.

[0064] Correspondingly, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the steps of the above-mentioned image deraining method based on a multi-dimensional aggregation network for rain line feature reconstruction.

[0065] Accordingly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned image deraining method based on a multi-dimensional aggregation network for rain line feature reconstruction.

[0066] The implementation of the present invention has the following beneficial effects:

[0067] First, the present invention proposes an image deraining method based on a multi-dimensional aggregation network (RREMAN) for rain line feature reconstruction and extraction. In actual use, the image only needs to be input into the network, and the derained image can be obtained through a single forward propagation of the network, which is more efficient than traditional image processing methods. Second, the network adds a rain line feature reconstruction and extraction module (RFREM), which can take into account the deep domain information and wide domain information of the features, and use it to reconstruct and re-extract the extracted features, making it easier to retain and highlight the important information about rain lines, which is conducive to the network focusing on removing rain lines in rainy areas, thereby maximally retaining the background texture details in rain-free areas, and the resulting derained image is clearer. Third, the network adopts a multi-dimensional feature aggregation module (MFAM), which uses dilated convolution with different dilation rates to extract information of different scales in the image, and then aggregates it to obtain the overall feature, so that the network can better learn the complex and changeable shapes of rain lines, thereby obtaining a derained image with clearer rain line removal. Fourth, the network adopts a multiple loss (Multiple Loss) including MSE loss, perceptual loss and multi-layer gradient loss (MSG Loss). Loss) for training, which can make the output derained image have better visual effects and retain more image details. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is an overall flow chart of an image deraining method based on a multi-dimensional aggregation network for rain line feature reconstruction according to an embodiment of the present invention;

[0069] Figure 2 1 is a network structure diagram of a rain line feature reconstruction and extraction module (RFREM) according to an embodiment of the present invention;

[0070] Figure 3 This is a structural diagram of a multi-dimensional feature aggregation module according to an embodiment of the present invention;

[0071] Figure 4 This is a multi-dimensional aggregation network structure diagram based on rain line feature reconstruction and extraction according to an embodiment of the present invention;

[0072] Figure 5 4 is a structural diagram of an image deraining system based on a multi-dimensional aggregation network for rain line feature reconstruction according to an embodiment of the present invention. DETAILED DESCRIPTION

[0073] The following will clearly and completely describe the technical inventions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0074] Figure 1 This is an overall flow chart of the image deraining method based on a multi-dimensional aggregation network for rain line feature reconstruction according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0075] S1, obtains an image dataset from the database, performs image preprocessing, adds rain line effects to the obtained images, and constructs training and test sets;

[0076] S2, build a deep-area feature reconstruction module and a wide-area feature reconstruction module, input the features to be reconstructed, perform channel reconstruction and spatial reconstruction respectively, and output deep-area reconstructed features and wide-area reconstructed features respectively;

[0077] S3, constructing a rain line feature reconstruction and extraction module, which is composed of the deep-area feature reconstruction module and the wide-area feature reconstruction module, splicing the deep-area reconstructed features and the wide-area reconstructed features along the channel dimension, and performing operations such as convolution to obtain intermediate features;

[0078] S4, constructing a multi-dimensional feature aggregation module, performing multi-dimensional aggregation on the input intermediate features, and outputting the aggregated multi-dimensional aggregated features;

[0079] S5, constructing a multidimensional aggregation network based on rain line feature reconstruction and extraction, which mainly consists of a convolution block, the rain line feature reconstruction and extraction module, the multidimensional feature aggregation module, and an image reconstruction module. The input is a rainy image, the multidimensional aggregation features are obtained, and the image reconstruction module outputs a rain-free image;

[0080] S6, using the training set to train the multidimensional aggregation network based on rain line feature reconstruction and extraction using multiple losses, and using the test set to perform testing and verification to obtain a trained multidimensional aggregation network based on rain line feature reconstruction and extraction;

[0081] S7, inputting the rainy image to be processed into the trained multi-dimensional aggregation network based on rain line feature reconstruction and extraction, and outputting a rain-free image.

[0082] Step S1 is as follows:

[0083] S1-1: Get an image dataset from a database. Here we select the SUN database and randomly get 1,000 images from it.

[0084] S1-2: Use Photoshop software to create a dynamic blur filter, setting the blur angle to 70 degrees, 80 degrees, 90 degrees, 100 degrees, and 110 degrees; the distance to 30 pixels, 40 pixels, and 50 pixels; a total of 15 filters that imitate different rain line effects are created.

[0085] S1-3: Use Photoshop to apply the filters in S21 to each image in S1-1. Each image corresponds to 15 images with rain effects, totaling 15,000 pairs of rain-no-rain image pairs. From these, 12,000 pairs of images are randomly selected, and approximately 2 million pairs of 64×64 rain-no-rain image patches are cropped as the training set; the remaining 3,000 pairs of images are used as the test set.

[0086] Step S2 is as follows:

[0087] S2-1: Construct a deep domain feature reconstruction module (DFRM), whose input is the H×W×C-dimensional features to be reconstructed, and the output is the H×W×C-dimensional features after reconstructing the information at the channel depth. This module uses a squeeze-and-extraction network structure, which can assign corresponding weights to each channel of the input features, thereby highlighting important features and ignoring irrelevant features, so that useful information can be retained during network transmission. First, the features of each channel of the input features to be reconstructed are average pooled to obtain 1×1×C-dimensional features, and then pass through the fully connected layer, ReLU layer, fully connected layer and sigmoid layer in sequence to output 1×1×C-dimensional deep domain weights. Finally, this weight is multiplied by the features to be reconstructed to obtain the deep domain reconstructed features.

[0088] S2-2: Construct a wide-field feature reconstruction module (WFRM). Its input is an H×W×C-dimensional feature to be reconstructed, and its output is an H×W×C-dimensional feature reconstructed from the spatial width. This module assigns a weight to each pixel position of the input feature, thereby highlighting important spatial features and ignoring irrelevant features, thereby preserving useful information during network transmission. First, the input feature to be reconstructed is averaged along the channel dimension to obtain an H×W×1-dimensional feature. Furthermore, this input feature to be reconstructed is subjected to a 1×1 convolution to obtain another H×W×1-dimensional feature. Next, these two H×W×1-dimensional features are concatenated along the channel dimension to obtain an H×W×2-dimensional feature. This feature is then passed through a 1×1 convolution, a ReLU layer, a 1×1 convolution, and a sigmoid layer, outputting an H×W×1-dimensional wide-field weight. Finally, this weight is multiplied by the feature to be reconstructed to obtain the wide-field reconstructed feature.

[0089] Step S3 is as follows:

[0090] S3-1: If Figure 2 As shown in the figure, a rain feature reconstruction and extraction module (RFREM) is constructed. This module is mainly composed of a deep-area feature reconstruction module (DFRM) and a wide-area feature reconstruction module (WFRM). It can perform channel and spatial reconstruction on the input features, and then extract new features through further convolution and output them.

[0091] S3-2: This module primarily consists of the deep-domain feature reconstruction module in S2-1 and the wide-domain feature reconstruction module in S2-2. It employs a residual structure, taking as input H×W×C-dimensional features and outputting H×W×C-dimensional features. The module's dilated convolution has a dilation rate of r. Using different dilation rates can capture feature information at different image scales. First, the input H×W×C-dimensional features are reconstructed by the deep-domain feature reconstruction module and the wide-domain feature reconstruction module, respectively, to produce deep-domain and wide-domain reconstructed features. Second, the deep-domain and wide-domain reconstructed features are concatenated along the channel dimension to produce H×W×(2×C)-dimensional features. These features are then passed through a dilated convolution with a 3×3 kernel size and dilation rate r, a ReLU layer, a batch normalization layer, and a dilated convolution with a 3×3 kernel size and dilation rate r, to produce H×W×C-dimensional features. Finally, these features are added to the input H×W×C-dimensional features to produce the final result.

[0092] Step S4 is as follows:

[0093] S4-1: If Figure 3 As shown in the figure, a Multi Feature Aggregation Module (MFAM) is constructed. This module can perform multi-dimensional aggregation on the input features and output new aggregated features. Its input is n different H×W×C dimensional features, and the output is a new H×W×(n×C) dimensional feature after the multi-dimensional aggregation operation.

[0094] S4-2: First, for each input H×W×C dimensional feature, it first undergoes 1×1 convolution to obtain a H×W×(C / n) dimensional feature, and then undergoes an average pooling operation to obtain a 1×1×(C / n) dimensional feature. Secondly, these n 1×1×(C / n) dimensional features are spliced ​​along the channel to obtain a 1×1×C dimensional feature, and then pass through the fully connected layer, ReLU layer, fully connected layer and sigmoid layer in sequence to obtain a 1×1×n dimensional feature weight, and then the weight is expanded to 1×1×(n×C) dimension. Finally, the n H×W×C dimensional features input at the beginning are spliced ​​along the channel to obtain a H×W×(n×C) dimensional feature, which is multiplied by the feature weight obtained previously, and passed through the 1×1 convolution and ReLU layers in sequence to output an H×W×(n×C) dimensional aggregated feature. The structure of the multidimensional feature aggregation module is as follows: Figure 3 As shown, where n=3.

[0095] Step S5 is as follows:

[0096] S5-1: If Figure 4 As shown in the figure, a multi-dimensional aggregation network (RREMAN) based on rain line feature reconstruction and extraction is constructed. Its input is a rainy image and its output is a rain-free image. The input H×W×3 rainy image first passes through a feature extraction layer consisting of a 3×3 convolution, a ReLU layer, a batch normalization layer, and a 3×3 convolution to obtain initial features of H×W×24 dimensions. Then, the initial features are passed through three U-Net-structured encoder-decoder networks to obtain three H×W×24-dimensional intermediate features. The encoder-decoder network consists of 5 encoder layers, 1 intermediate layer, and 5 decoder layers, with skip connections between the encoder and decoder layers. The encoder, decoder, and intermediate layers are all composed of the feature reconstruction and extraction module proposed in S2. The dilation rates used in the feature reconstruction and extraction modules of these three encoder-decoder networks are 1, 2, and 3, respectively, to obtain feature information at different scales.

[0097] S5-2: Finally, these three features obtained by different encoding and decoding networks are aggregated through the multi-dimensional aggregation module in S4 to obtain H×W×72-dimensional aggregated features. The image is then reconstructed in sequence through two image reconstruction modules containing 3×3 convolution, ReLU layer, and batch normalization layer. Finally, a 3×3 convolution is performed to output the H×W×3-dimensional derained image.

[0098] Step S6 is as follows:

[0099] S6-1: Use multiple losses to train the network proposed in S5. The multiple losses used in training are the sum of pixel-level MSE loss, perceptual loss, and multi-stage gradient loss (MSGLoss). Using the training set of 2 million pairs of 64×64 rain-no rain image patches proposed in S1-3, the network is trained using the Adam algorithm, with 40 images input per batch, and training until the loss function converges. During training, the learning rate is initially set to 0.0001, decays to 0.00001 at the 100,000th round, and decays to 0.000001 at the 150,000th round. The multiple loss function is expressed as follows:

[0100] L=L MSE +L per +L MSG

[0101] Among them L MSE represents pixel-level MSE loss, L per Represents the perceptual loss, L MSG Represents the multi-layer gradient loss of the image.

[0102] S6-2: The pixel-level MSE loss function in multiple losses is expressed as follows:

[0103]

[0104] Where M is the total number of training data, Y i and Y gt,i The loss directly measures the difference between the pixels of the two images.

[0105] S6-3: The perceptual loss function in multiple losses is expressed as follows:

[0106]

[0107] Where M is the total number of training data, Y i and Y gt,i are the de-rained image and the real rain-free image output by the network respectively. i) and F(Y gt,i ) is the result of passing the pre-trained VGG16 network through the derained image and the real rain-free image respectively. This loss measures the difference in visual features between the two images.

[0108] S6-4: The multi-layer gradient loss (MSG Loss) function in multiple losses is expressed as follows:

[0109]

[0110] in

[0111]

[0112] To find the gradient operator, Y i The result is obtained after k times of downsampling. Each downsampling is performed with an average pooling operation of 2 windows and a step size of 2, and the original image is obtained. The loss measures the difference in gradients between two images, thereby measuring the difference in details and textures.

[0113] S6-5: Use the test set of S1-3 for testing and verification to obtain a trained multi-dimensional aggregation network based on rain line feature reconstruction and extraction.

[0114] Step S7 is as follows:

[0115] S7-1: Input the rainy image to be processed into the trained multi-dimensional aggregation network based on rain line feature reconstruction and extraction, and output the rain-free image.

[0116] Accordingly, the present invention also provides an image deraining system based on a multi-dimensional aggregation network for rain line feature reconstruction, such as Figure 5 Shown, including:

[0117] The data set construction unit 1 obtains an image data set from a database, performs image preprocessing, adds rain line effects to the obtained images, and constructs a training set and a test set.

[0118] Specifically, 1,000 images were randomly obtained from the SUN database as an image dataset, and image preprocessing was performed. Photoshop software was used to construct a dynamic blur filter, and the blur angles were set to 70 degrees, 80 degrees, 90 degrees, 100 degrees, and 110 degrees, and the distances were 30 pixels, 40 pixels, and 50 pixels, respectively. A total of 15 filters imitating different rain line effects were constructed. Photoshop software was used to add filters to each image in the image dataset, and 15 images with rain line effects were obtained, totaling 15,000 pairs of rain-no rain image pairs. 12,000 pairs of images were randomly selected from them, and about 2 million pairs of 64×64 rain-no rain image blocks were cropped out as a training set, and the remaining 3,000 pairs of images were used as a test set.

[0119] Module construction and feature extraction unit 2 constructs a rain line feature reconstruction and extraction module, which consists of a deep-area feature reconstruction module and a wide-area feature reconstruction module, extracts intermediate features, constructs a multidimensional feature aggregation module, and extracts multidimensional aggregation features.

[0120] Specifically, a deep-domain feature reconstruction module and a wide-area feature reconstruction module are constructed to output deep-domain reconstructed features and wide-area reconstructed features respectively. A rain line feature reconstruction and extraction module is constructed, which is composed of the deep-domain feature reconstruction module and the wide-area feature reconstruction module. The deep-domain reconstructed features and the wide-area reconstructed features are spliced ​​along the channel dimension, and operations such as convolution are performed to obtain intermediate features. A multidimensional feature aggregation module is constructed to perform multidimensional aggregation on the input intermediate features and output the aggregated multidimensional aggregated features.

[0121] Network training and rain removal unit 3 constructs a multidimensional aggregation network based on rain line feature reconstruction and extraction. Its input is a rainy image and its output is a rain-removed image. The network is trained using a training set and multiple losses. It is tested and verified using a test set to obtain a trained network. The rainy image to be processed is input into the trained network, and the rain-removed image is output.

[0122] Specifically, a multidimensional aggregation network based on rain line feature reconstruction and extraction is constructed, which is mainly composed of a convolution block, the rain line feature reconstruction and extraction module, the multidimensional feature aggregation module and an image reconstruction module. Its input is a rainy image, and the multidimensional aggregation feature is obtained. After passing through the image reconstruction module, the output is a derained image. The multidimensional aggregation network based on rain line feature reconstruction and extraction is trained using the training set using multiple losses, and is tested and verified using the test set to obtain a trained network. The rainy image to be processed is input into the trained network, and a derained image is output.

[0123] Therefore, this paper proposes a novel neural network-based image deraining method. This method utilizes a multidimensional aggregation network based on rain line feature reconstruction and extraction, fully leveraging the multidimensional information of the image. The rain line feature reconstruction and extraction module leverages the overall spatial information of the rain line image and the importance information between channel features, enabling the network to better learn important information about rain line features. This method enables the network to focus on removing rain lines in rainy areas while preserving background texture details as much as possible, resulting in clearer derained images.

[0124] Accordingly, the present invention further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned image deraining method based on a multidimensional aggregation network for rain line feature reconstruction are implemented. Furthermore, the present invention further provides a computer-readable storage medium storing the computer program, and when the processor executes the computer program, the steps of the above-mentioned ship identification method are implemented.

[0125] The above describes in detail the image deraining method, system, device, and storage medium based on a multi-dimensional aggregation network for rain line feature reconstruction provided by the embodiments of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only intended to help understand the method and core concept of the present invention. At the same time, for those skilled in the art, according to the concept of the present invention, there may be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. Image deraining method based on multi-dimensional aggregation network for rain line feature reconstruction, characterized by: The method comprises: Obtain image datasets from the database, perform image preprocessing, add rain line effects to the acquired images, and construct training and test sets; Construct a deep-area feature reconstruction module and a wide-area feature reconstruction module, input the features to be reconstructed, perform channel reconstruction and spatial reconstruction respectively, and output deep-area reconstructed features and wide-area reconstructed features respectively; Constructing a rain line feature reconstruction and extraction module, which is composed of the deep-area feature reconstruction module and the wide-area feature reconstruction module, splicing the deep-area reconstructed features and the wide-area reconstructed features along the channel dimension and performing a convolution operation to obtain intermediate features; Constructing a multidimensional feature aggregation module to perform multidimensional aggregation on the input intermediate features and output the aggregated multidimensional aggregated features; A multidimensional aggregation network based on rain line feature reconstruction and extraction is constructed, which consists of a convolution block, the rain line feature reconstruction and extraction module, the multidimensional feature aggregation module, and an image reconstruction module. The input is a rainy image, the multidimensional aggregation features are obtained, and the image reconstruction module outputs a rain-free image; Using the training set, using multiple losses to train the multidimensional aggregation network based on rain line feature reconstruction and extraction, and using the test set to perform testing and verification, to obtain a trained multidimensional aggregation network based on rain line feature reconstruction and extraction; Inputting the rainy image to be processed into the trained multi-dimensional aggregation network based on rain line feature reconstruction and extraction, and outputting a rain-free image; The rain line feature reconstruction and extraction module is composed of the deep-area feature reconstruction module and the wide-area feature reconstruction module. The deep-area reconstructed features and the wide-area reconstructed features are spliced ​​along the channel dimension and convolution operations are performed to obtain intermediate features, specifically: Constructing a rain line feature reconstruction and extraction module, which is composed of the deep-area feature reconstruction module and the wide-area feature reconstruction module and adopts a residual structure; The rain line feature reconstruction and extraction module inputs H×W×C dimensional features and outputs H×W×C dimensional intermediate features. The dilation rate of the module's dilated convolution is r, and different dilation rates are used to obtain feature information at different scales of the image. The input H×W×C dimensional features are reconstructed by the deep-area feature reconstruction module and the wide-area feature reconstruction module respectively to obtain deep-area reconstructed features and wide-area reconstructed features; The deep domain reconstruction features and the wide domain reconstruction features are spliced ​​along the channel dimension to obtain H×W×(2×C)-dimensional features, which are then passed through a dilated convolution with a kernel size of 3×3 and a dilation rate of r, a ReLU layer, a batch normalization layer, and a dilated convolution with a kernel size of 3×3 and a dilation rate of r to output H×W×C-dimensional features. Finally, they are added to the input H×W×C-dimensional features to output the intermediate features. The multi-dimensional feature aggregation module is constructed to perform multi-dimensional aggregation on the input intermediate features and output the aggregated multi-dimensional aggregated features, specifically: Construct a multidimensional feature aggregation module, whose input is n different H×W×C dimensional intermediate features, and the output is the multidimensional aggregated features of H×W×(n×C) dimensions after the multidimensional aggregation operation; For each input H×W×C dimensional intermediate feature, it first undergoes a 1×1 convolution to obtain a H×W×(C / n) dimensional feature, and then undergoes an average pooling operation to obtain a 1×1×(C / n) dimensional feature; These n 1×1×(C / n)-dimensional features are concatenated along the channel to obtain a 1×1×C-dimensional feature, which is then passed through a fully connected layer, a ReLU layer, a fully connected layer, and a sigmoid layer to obtain a 1×1×n-dimensional feature weight, which is then expanded to 1×1×(n×C) dimensions. The n H×W×C dimensional features initially input are concatenated along the channel to obtain H×W×(n×C) dimensional features, which are multiplied by the feature weights obtained previously, and then passed through 1×1 convolution and ReLU layers in sequence to output H×W×(n×C) dimensional multidimensional aggregated features; The multidimensional aggregation network based on rain line feature reconstruction and extraction is trained using the training set and multiple losses, and the test set is used for testing and verification to obtain a trained multidimensional aggregation network based on rain line feature reconstruction and extraction, specifically: The multidimensional aggregation network based on rain line feature reconstruction and extraction is trained using multiple losses, where the multiple losses used in the training are the sum of pixel-level MSE loss, perceptual loss, and multi-layer gradient loss; The Adam algorithm was used to train the network using the training set, with 40 images per batch, until the loss function converged. The learning rate was initially set to 0.0001 during training, and decayed to 0.00001 at the 100,000th round and 0.000001 at the 150,000th round. The multiple loss functions are expressed as follows: L=L MSE +L per +L MSG Among them L MSE represents pixel-level MSE loss, L per Represents the perceptual loss, L MSG Represents the multi-layer gradient loss of the image; The pixel-level MSE loss function in multiple losses is expressed as follows: Where M is the total number of training data, Y i and Y gt,i The de-rained image and the true rain-free image are output by the multi-dimensional aggregation network based on rain line feature reconstruction and extraction, respectively. This loss directly measures the difference between the pixels of the two images; The perceptual loss function in multiple losses is expressed as follows: Where M is the total number of training data, Y i and Y gt,i are the rain-free image and the real rain-free image output by the multi-dimensional aggregation network based on rain line feature reconstruction and extraction, respectively. i ) and F(Y gt,i ) is the result of the derained image and the real rain-free image being respectively obtained by the pre-trained VGG16 network. This loss measures the difference in the visual features of the two images; The multi-layer gradient loss function in multiple losses is expressed as follows: in: To find the gradient operator, Y i The result obtained after k times of downsampling is For the real rain-free image Y gt,i The gradient is calculated by downsampling k times, and each downsampling is performed with an average pooling operation of 2 windows and a step size of 2, which will give the original image. The downsampled image of the size is N=2 in training. This loss measures the difference in gradients between the two images, thereby measuring the difference in details and textures. The test set is used for testing and verification to obtain a trained multi-dimensional aggregation network based on rain line feature reconstruction and extraction.

2. The image deraining method based on a multi-dimensional aggregation network for rain line feature reconstruction according to claim 1, characterized in that: The image dataset is obtained from the database, image preprocessing is performed, rain line effects are added to the obtained images, and a training set and a test set are constructed, specifically: Randomly obtain 1000 images from the SUN database as an image dataset and perform image preprocessing; Use Photoshop to create dynamic blur filters, setting the blur angles to 70, 80, 90, 100, and 110 degrees, and the distances to 30, 40, and 50 pixels, creating a total of 15 filters that mimic different rain line effects. Photoshop was used to apply a filter to each image in the dataset, resulting in 15 images with rain streaks, totaling 15,000 rain-no-rain image pairs. From these, 12,000 pairs were randomly sampled, and 2 million pairs of 64×64 rain-no-rain image patches were cropped to form the training set. The remaining 3000 image pairs are used as the test set.

3. The image deraining method based on a multi-dimensional aggregation network for rain line feature reconstruction according to claim 1, characterized in that: The deep-area feature reconstruction module and the wide-area feature reconstruction module are constructed, and the features to be reconstructed are input for channel reconstruction and spatial reconstruction, respectively, and the deep-area reconstructed features and the wide-area reconstructed features are output respectively. Specifically: Construct a deep domain feature reconstruction module. The module uses a squeeze-excited network structure. Its input is the H×W×C-dimensional feature to be reconstructed. Each channel of the feature is assigned a corresponding weight, and the output is the deep domain reconstructed feature. The features of each channel of the feature to be reconstructed are average pooled to obtain a 1×1×C-dimensional feature. Then, it passes through the fully connected layer, ReLU layer, fully connected layer, and sigmoid layer in sequence to output a 1×1×C-dimensional deep domain weight. This weight is multiplied by the feature to be reconstructed to obtain the deep domain reconstructed feature. Construct a wide-area feature reconstruction module, whose input is the feature to be reconstructed of H×W×C dimensions, and output is the wide-area reconstructed feature; Calculate the average value of the channel dimension for the feature to be reconstructed to obtain a feature of H×W×1 dimension, and then perform a 1×1 convolution on the feature to be reconstructed to obtain another feature of H×W×1 dimension; The two obtained H×W×1-dimensional features are spliced ​​along the channel dimension to obtain H×W×2-dimensional features, and then pass through 1×1 convolution, ReLU layer, 1×1 convolution and sigmoid layer in sequence to output H×W×1-dimensional wide-area weights. Finally, this weight is multiplied by the feature to be reconstructed to obtain the wide-area reconstructed feature.

4. The image deraining method based on a multi-dimensional aggregation network for rain line feature reconstruction according to claim 1, characterized in that: The multidimensional aggregation network based on rain line feature reconstruction and extraction is constructed, which consists of a convolution block, the rain line feature reconstruction and extraction module, the multidimensional feature aggregation module and an image reconstruction module. Its input is a rain image and its output is a rain-free image. Specifically: The input H×W×3 dimensional rain image is processed through a feature extraction layer that includes a 3×3 convolution layer, a ReLU layer, a batch normalization layer, and a 3×3 convolution layer to obtain the initial features of H×W×24 dimensions; The initial features are passed through three U-Net structured encoding and decoding networks to obtain three H×W×24-dimensional intermediate features. The encoding and decoding network consists of 5 encoder layers, 1 intermediate layer, and 5 decoder layers. There are skip connections between the encoders and decoders of the corresponding layers. Its encoder, decoder and intermediate layer are all composed of the rain line feature reconstruction and extraction module. The expansion rates r in the rain line feature reconstruction and extraction module used by these three encoding and decoding networks are 1, 2, and 3 respectively to obtain feature information of different scales. These three intermediate features obtained by different encoding and decoding networks are aggregated through the multidimensional feature aggregation module to obtain H×W×72-dimensional multidimensional aggregated features; The obtained multi-dimensional aggregated features are passed through two image reconstruction modules consisting of 3×3 convolution, ReLU layer, and batch normalization layer for image reconstruction, and then a 3×3 convolution is performed to output the H×W×3 dimensional derained image.

5. Image deraining system based on multi-dimensional aggregation network for rain line feature reconstruction, characterized by: The system comprises: The dataset construction unit obtains the image dataset from the database, performs image preprocessing, adds rain line effects to the obtained images, and constructs training and test sets; A module construction and feature extraction unit is provided, which constructs a deep-domain feature reconstruction module and a wide-domain feature reconstruction module, inputs the features to be reconstructed, performs channel reconstruction and spatial reconstruction respectively, and outputs deep-domain reconstructed features and wide-domain reconstructed features respectively; constructs a rain line feature reconstruction and extraction module, which is composed of the deep-domain feature reconstruction module and the wide-domain feature reconstruction module, splices the deep-domain reconstructed features and the wide-domain reconstructed features along the channel dimension, and performs a convolution operation to obtain intermediate features; constructs a multidimensional feature aggregation module, performs multidimensional aggregation on the input intermediate features, and outputs the aggregated multidimensional aggregated features; A network training and rain removal unit constructs a multidimensional aggregation network based on rain line feature reconstruction and extraction, which is composed of a convolution block, the rain line feature reconstruction and extraction module, the multidimensional feature aggregation module, and an image reconstruction module. The network input is a rainy image, the multidimensional aggregation features are obtained, and after passing through the image reconstruction module, the rain-removed image is output. The multidimensional aggregation network based on rain line feature reconstruction and extraction is trained using the training set using multiple losses, and is tested and verified using the test set to obtain a trained multidimensional aggregation network based on rain line feature reconstruction and extraction. The rainy image to be processed is input into the trained multidimensional aggregation network based on rain line feature reconstruction and extraction, and the rain-removed image is output. Among them, in the module construction and feature extraction unit, a rain line feature reconstruction and extraction module is constructed, specifically: Constructing a rain line feature reconstruction and extraction module, which is composed of the deep-area feature reconstruction module and the wide-area feature reconstruction module and adopts a residual structure; The rain line feature reconstruction and extraction module inputs H×W×C dimensional features and outputs H×W×C dimensional intermediate features. The dilation rate of the module's dilated convolution is r, and different dilation rates are used to obtain feature information at different scales of the image. The input H×W×C dimensional features are reconstructed by the deep-area feature reconstruction module and the wide-area feature reconstruction module respectively to obtain deep-area reconstructed features and wide-area reconstructed features; The deep domain reconstruction features and the wide domain reconstruction features are spliced ​​along the channel dimension to obtain H×W×(2×C)-dimensional features, which are then passed through a dilated convolution with a kernel size of 3×3 and a dilation rate of r, a ReLU layer, a batch normalization layer, and a dilated convolution with a kernel size of 3×3 and a dilation rate of r to output H×W×C-dimensional features. Finally, they are added to the input H×W×C-dimensional features to output the intermediate features. Among them, in the module construction and feature extraction unit, a multi-dimensional feature aggregation module is constructed, specifically: Construct a multidimensional feature aggregation module, whose input is n different H×W×C dimensional intermediate features, and the output is the multidimensional aggregated features of H×W×(n×C) dimensions after the multidimensional aggregation operation; For each input H×W×C dimensional intermediate feature, it first undergoes a 1×1 convolution to obtain a H×W×(C / n) dimensional feature, and then undergoes an average pooling operation to obtain a 1×1×(C / n) dimensional feature; These n 1×1×(C / n)-dimensional features are concatenated along the channel to obtain a 1×1×C-dimensional feature, which is then passed through a fully connected layer, a ReLU layer, a fully connected layer, and a sigmoid layer to obtain a 1×1×n-dimensional feature weight, which is then expanded to 1×1×(n×C) dimensions. The n H×W×C dimensional features initially input are concatenated along the channel to obtain H×W×(n×C) dimensional features, which are multiplied by the feature weights obtained previously, and then passed through 1×1 convolution and ReLU layers in sequence to output H×W×(n×C) dimensional multidimensional aggregated features; In the network training and rain removal unit, the multidimensional aggregation network based on rain line feature reconstruction and extraction is trained using the training set and multiple losses, and the test set is used for testing and verification to obtain a trained multidimensional aggregation network based on rain line feature reconstruction and extraction, specifically: The multidimensional aggregation network based on rain line feature reconstruction and extraction is trained using multiple losses, where the multiple losses used in the training are the sum of pixel-level MSE loss, perceptual loss, and multi-layer gradient loss; The Adam algorithm was used to train the network using the training set, with 40 images per batch, until the loss function converged. The learning rate was initially set to 0.0001 during training, and decayed to 0.00001 at the 100,000th round and 0.000001 at the 150,000th round. The multiple loss functions are expressed as follows: L=L MSE +L per +L MSG Among them L MSE represents pixel-level MSE loss, L per Represents the perceptual loss, L MSG Represents the multi-layer gradient loss of the image; The pixel-level MSE loss function in multiple losses is expressed as follows: Where M is the total number of training data, Y i and Y gt,i The de-rained image and the true rain-free image are output by the multi-dimensional aggregation network based on rain line feature reconstruction and extraction, respectively. This loss directly measures the difference between the pixels of the two images; The perceptual loss function in multiple losses is expressed as follows: Where M is the total number of training data, Y i and Y gt,i are the rain-free image and the real rain-free image output by the multi-dimensional aggregation network based on rain line feature reconstruction and extraction, respectively. i ) and F(Y gt,i ) is the result of the derained image and the real rain-free image being respectively obtained by the pre-trained VGG16 network. This loss measures the difference in the visual features of the two images; The multi-layer gradient loss function in multiple losses is expressed as follows: in: To find the gradient operator, Y i k Y i The result obtained after k times of downsampling is For the real rain-free image Y gt,i The gradient is calculated by downsampling k times, and each downsampling is performed with an average pooling operation of 2 windows and a step size of 2, which will give the original image. The downsampled image of the size is N=2 in training. This loss measures the difference in gradients between the two images, thereby measuring the difference in details and textures. The test set is used for testing and verification to obtain a trained multi-dimensional aggregation network based on rain line feature reconstruction and extraction.

6. The image deraining system based on a multi-dimensional aggregation network for rain line feature reconstruction according to claim 5, characterized in that: The data set construction unit randomly obtains 1,000 images from the SUN database as an image data set, performs image preprocessing, and uses Photoshop software to construct a dynamic blur filter, setting the blur angles to 70 degrees, 80 degrees, 90 degrees, 100 degrees, and 110 degrees, and the distances to 30 pixels, 40 pixels, and 50 pixels, respectively. A total of 15 filters that imitate different rain line effects are constructed, and Photoshop software is used to add filters to each image in the image data set to obtain 15 images with rain line effects, forming a total of 15,000 pairs of rain-no rain image pairs. 12,000 pairs of images are randomly selected from them, and 2 million pairs of 64×64 rain-no rain image blocks are cropped as training sets, and the remaining 3,000 pairs of images are used as test sets.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images