A rainy image fault-tolerant device and rain removal method based on improved convolutional long short-term memory network

Abstract

The present application relates to a rainy day image fault-tolerant model based on an improved convolutional long short-term memory network, comprising: a feature extraction module and a regional feature enhancement module; the feature extraction module is used to extract rain streak features in rainy day images and efficiently learn key areas of interest, outputting a feature map; the regional feature enhancement module captures and removes raindrop information of different scales based on the feature map, and outputs a rain-removed image; the present application can capture rain streaks of different scales, while removing basic rain lines, protecting important image details, textures, and background information, thereby obtaining a rain-free image with higher color saturation and quality, making the restored image cleaner and having better visual effects.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a rainy day image fault-tolerant model based on an improved convolutional long short-term memory network. Background Art

[0002] Rain significantly reduces visibility and impairs the application of computer vision. Image deraining technology primarily addresses the problem of rain interference encountered when capturing images, aiming to improve image quality and readability. Generally speaking, image deraining technologies fall into two categories: physics-based methods and data-based methods. Physics-based methods generally assume that raindrop landing points and sizes conform to a certain probability distribution and restore clear, rain-free images by modeling aspects such as sensor noise and scene degradation. This approach typically requires sophisticated model construction and calculations, resulting in high computational complexity. Data-based methods, on the other hand, directly utilize a large number of rainy and non-rainy images for training, establishing a mapping from rain-involved images to rain-free images, thereby achieving image deraining. Compared to physics-based methods, this approach offers improved speed and scalability.

[0003] At present, image deraining technology still has the following shortcomings and limitations: (1) The deraining effect is not satisfactory, which may lead to some problems such as loss of detailed information or artifacts; (2) The effect is poor in extreme cases such as large raindrops and heavy rain; (3) A large amount of training data is still required for optimization, which takes a lot of time and computing resources. Summary of the Invention

[0004] In order to overcome at least one deficiency in the prior art, the present application provides a rainy day image fault-tolerant model based on an improved convolutional long short-term memory network.

[0005] First, a rainy day image fault-tolerant model based on an improved convolutional long short-term memory network is provided, comprising: a feature extraction module and a regional feature enhancement module; the feature extraction module is used to extract rain streak features from rainy day images and efficiently learn key areas of interest, outputting a feature map; the regional feature enhancement module is used to capture and remove raindrop information of different scales based on the feature map, and output an image after rain removal.

[0006] In one embodiment, the feature extraction module includes multiple feature extraction units connected in sequence, each feature extraction unit includes a densely connected network and a convolutional long short-term memory network; the output of the previous feature extraction unit serves as the input of the densely connected network in the next feature extraction unit, and the output of the convolutional long short-term memory network in the previous feature extraction unit serves as the input of the convolutional long short-term memory network in the next feature extraction unit; the output of the last feature extraction unit is the feature map.

[0007] In one embodiment, a densely connected network includes multiple Dense Layer layers. The input passes through the nonlinear composite function in each DenseLayer layer and outputs feature maps of the same specifications. The feature maps are spliced in the densely connected network according to the set splicing rules; the output of the densely connected network serves as the input of the convolutional long short-term memory network.

[0008] In one embodiment, the dense connection network includes a first Dense Layer layer, a second Dense Layer layer, a first splicing layer, a third Dense Layer layer, a fourth Dense Layer layer, a second splicing layer, a third splicing layer, and a fifth Dense Layer layer connected in sequence. The rainy day image is input to the first Dense Layer layer, the output of the first Dense Layer layer is input to the second Dense Layer layer, the output of the second Dense Layer layer and the output of the first Dense Layer layer are spliced in the first splicing layer, the output of the first splicing layer is input to the third Dense Layer layer, the output of the third Dense Layer layer is input to the fourth Dense Layer layer, the output of the fourth Dense Layer layer and the output of the third Dense Layer layer are spliced in the second splicing layer, the output of the second splicing layer and the output of the first splicing layer are spliced in the third splicing layer, the output of the third splicing layer is input to the fifth Dense Layer layer, the output of the fifth Dense Layer layer is the output of the dense connection network, and the output of the fifth Dense Layer layer is input to the convolutional long short-term memory network.

[0009] In one embodiment, the feature extraction unit further includes a first convolutional layer, the output of the convolutional long short-term memory network is input to the first convolutional layer, the output of the first convolutional layer is element-wise multiplied with the rainy day image, and the result obtained is the output of the feature extraction unit.

[0010] In one embodiment, the regional feature enhancement module includes an average pooling layer, a downsampling layer, a second convolutional layer, an upsampling layer, a fourth splicing layer and a third convolutional layer; the input of the average pooling layer is the feature map, the output of the average pooling layer passes through the downsampling layer to obtain downsampling results of different scales, the downsampling results of different scales pass through the second convolutional layer respectively, and the convolution results of different scales are input into the upsampling layer respectively to obtain upsampling results, the upsampling results and the feature map are spliced in the fourth splicing layer to obtain a splicing result; the splicing result is input into the third convolutional layer, and the output of the third convolutional layer is the image after rain removal.

[0011] In one embodiment, the Dense Layer includes a convolutional layer, a RELU function, and a pooling layer.

[0012] In a second aspect, a method for removing rain from a rainy image is provided, comprising:

[0013] Get rainy day images;

[0014] The rainy image is input into the rainy image fault-tolerant model based on the improved convolutional long short-term memory network to obtain the image after rain removal;

[0015] The rainy day image fault-tolerant model based on the improved convolutional long short-term memory network is based on the above-mentioned rainy day image fault-tolerant model based on the improved convolutional long short-term memory network.

[0016] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the above-mentioned rain removal method for rainy images is implemented.

[0017] In a fourth aspect, a computer program product is provided, comprising a computer program / instruction, which, when executed by a processor, implements the above-mentioned method for removing rain from rainy images.

[0018] Compared with the existing technology, the present application has the following beneficial effects: the present application can capture rain streaks of different scales, and while removing basic rain lines, it protects important detail textures and background information of the image, thereby obtaining a rain-free image with higher color saturation and quality, making the restored image cleaner and having better visual effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The present application may be better understood by referring to the following description in conjunction with the accompanying drawings, which together with the following detailed description are incorporated into and form a part of this specification. In the drawings:

[0020] Figure 1 The following is a structural block diagram of a rainy day image fault-tolerant model based on an improved convolutional long short-term memory network according to an embodiment of the present application;

[0021] Figure 2 The following figure shows a comparison of the deraining results of the model proposed in this application and the existing deraining methods, where (a) is a rainy image, (b) is the deraining result of the DSC algorithm, (c) is the deraining result of the DNN algorithm, (d) is the deraining result of the GMM algorithm, (e) is the deraining result of the GCANet algorithm, (f) is the deraining result of the DID-MDN algorithm, (g) is the deraining result of the JORDER algorithm, and (h) is the deraining result of the model proposed in this application;

[0022] Figure 3 A comparison chart of the rain removal effects of the model proposed in this application and existing rain removal methods is shown. DETAILED DESCRIPTION

[0023] Exemplary embodiments of the present application are described below with reference to the accompanying drawings. For the sake of clarity and conciseness, not all features of actual embodiments are described in this specification. However, it should be understood that in the process of developing any such actual embodiment, many implementation-specific decisions may be made to achieve the developer's specific goals, and these decisions may vary from one implementation to another.

[0024] It is also necessary to explain here that, in order to avoid obscuring the present application due to unnecessary details, the accompanying drawings only show the device structure closely related to the solution according to the present application, while other details that are not closely related to the present application are omitted.

[0025] It should be understood that the present application is not limited to the described embodiments due to the following description with reference to the accompanying drawings. In this document, where feasible, the embodiments may be combined with each other, features between different embodiments may be replaced or borrowed, and one or more features may be omitted in one embodiment.

[0026] This application provides a rainy day image fault-tolerant model based on an improved convolutional long short-term memory network. To address the problems of unrealistic colors, blurred backgrounds, and incomplete details in existing rainy day image processing methods, the model uses a densely connected network to extract and reuse complex rain streak features in the image, and then passes the extracted features into a convolutional long short-term memory network (Conv-LSTM). The Conv-LSTM finds the parts that need to be focused on from the feature map through layer-by-layer iteration and efficiently learns. Finally, it uses a regional feature enhancement module to capture and remove raindrop information of different scales, ensuring that the restored image is clean and the visual effect is good, providing high-quality assurance for subsequent applications in the field of computer vision.

[0027] The embodiment of the present application provides a rainy day image fault-tolerant model based on an improved convolutional long short-term memory network. Figure 1 The structural block diagram of the rainy day image fault tolerance model based on the improved convolutional long short-term memory network provided in accordance with the embodiment of the present application is shown in FIG. Figure 1 The model includes: a feature extraction module and a regional feature enhancement module; the feature extraction module is used to extract the rain streak features of rainy images and efficiently learn the key areas of interest, outputting feature maps; the regional feature enhancement module is used to capture and remove raindrop information of different scales based on the feature maps, and output the image after rain removal.

[0028] In one embodiment, the feature extraction module includes multiple feature extraction units connected in sequence, each feature extraction unit includes a densely connected network and a convolutional long short-term memory network Conv-LSTM; the output of the previous feature extraction unit serves as the input of the densely connected network in the next feature extraction unit, and the output of the convolutional long short-term memory network in the previous feature extraction unit serves as the input of the convolutional long short-term memory network in the next feature extraction unit; the output of the last feature extraction unit is the feature map.

[0029] In one embodiment, the densely connected network includes multiple Dense Layer layers, and the input passes through the nonlinear composite function in each DenseLayer layer. Then the feature maps of the same size are output, and the dense connection network splices the feature maps according to the set splicing rules; the output of the dense connection network serves as the input of the convolutional long short-term memory network. Here, the nonlinear composite function It consists of a convolutional layer, a RELU function, and a pooling layer.

[0030] In one embodiment, the dense connection network includes a first Dense Layer layer, a second Dense Layer layer, a first splicing layer, a third Dense Layer layer, a fourth Dense Layer layer, a second splicing layer, a third splicing layer, and a fifth Dense Layer layer connected in sequence. The rainy day image is input to the first Dense Layer layer, the output of the first Dense Layer layer is input to the second Dense Layer layer, the output of the second Dense Layer layer and the output of the first Dense Layer layer are spliced in the first splicing layer, the output of the first splicing layer is input to the third Dense Layer layer, the output of the third Dense Layer layer is input to the fourth Dense Layer layer, the output of the fourth Dense Layer layer and the output of the third Dense Layer layer are spliced in the second splicing layer, the output of the second splicing layer and the output of the first splicing layer are spliced in the third splicing layer, the output of the third splicing layer is input to the fifth Dense Layer layer, the output of the fifth Dense Layer layer is the output of the dense connection network, and the output of the fifth Dense Layer layer is input to the convolutional long short-term memory network.

[0031] In this embodiment, the Dense Layer passes low-level features to higher levels to learn rain streak information from the original image. The Dense Layer consists of a convolutional layer, a RELU function, and a pooling layer, with a convolution kernel size of 3. In the densely connected network, the feature maps generated by each Dense Layer are concatenated to achieve feature reuse and generate dense image features. The Convolutional Long Short-Term Memory (Conv-LSTM) network identifies areas of focus from the feature map. Based on the dense image features input by the densely connected network, it uses a forget gate to learn the missing rain streak information. Then, in the input gate, a tanh layer is used to add new details at this moment. The background layer information is updated by adjusting the current input vector. Finally, the output gate discards irrelevant information and outputs the updated background layer features. Simultaneously, the Convolutional Long Short-Term Memory (Conv-LSTM) network uses the feature information from the rain removal process of the previous feature extraction unit as guidance to transmit to the next feature extraction unit, locating the target area in the image. Based on the acquired target area features, the network focuses on the areas that need to be deleted or restored.

[0032] In one embodiment, the feature extraction unit further includes a first convolutional layer, the output of the convolutional long short-term memory network is input to the first convolutional layer, the output of the first convolutional layer is element-wise multiplied with the rainy day image, and the result obtained is the output of the feature extraction unit.

[0033] In one embodiment, the regional feature enhancement module includes an average pooling layer, a downsampling layer, a second convolutional layer, an upsampling layer, a fourth splicing layer, and a third convolutional layer. The input of the average pooling layer is the feature map. The output of the average pooling layer passes through the downsampling layer to obtain downsampling results of different scales. The downsampling results of different scales pass through the second convolutional layer to obtain convolution results of different scales. The convolution results of different scales are respectively input into the upsampling layer to obtain upsampling results. The upsampling results and the feature map are spliced in the fourth splicing layer to obtain a spliced result. The spliced result is input into the third convolutional layer, and the output of the third convolutional layer is the derained image. Here, the downsampling scales include 1 / 32, 1 / 16, and 1 / 8.

[0034] In this embodiment, the features extracted from the convolutional long short-term memory (Conv-LSTM) network are of great help in recovering image structural information, but they still lack a description of specific details. Since the size and length of rain streaks in an image vary, the area of information covered also varies. Therefore, a regional feature enhancement module is built based on the pyramid pooling block. Starting from the global context information in target recognition, different spatial information is extracted and features at different scales are described in detail to ensure that the final result contains texture features of different scales, thereby restoring the true scene details in the rain-free image. Here, the output of the average pooling layer is downsampled at different scales. Different feature maps have different receptive fields and can capture features of different scales in the input image. In order to reconstruct images at different scales, a second convolutional layer with a kernel size of 1 is used for dimensionality reduction, and the feature map is upsampled to the original image size and spliced together with the front-end feature map. Finally, a 3*3 third convolutional layer is used to perform a convolution operation on the spliced feature maps to further extract features and obtain the final output.

[0035] Furthermore, the model of the present application is a trained model, and the dataset used in the training process is the Synthetic Rain Datasets.

[0036] The present application also provides a method for removing rain from rainy images, including:

[0037] Acquire a rainy day image; input the rainy day image into a rainy day image fault-tolerant model based on an improved convolutional long short-term memory network to obtain a rain-free image; here, the rainy day image fault-tolerant model based on the improved convolutional long short-term memory network is the model in the previous embodiment.

[0038] An embodiment of the present application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the aforementioned rainy image deraining method is implemented.

[0039] An embodiment of the present application provides a computer program product, including a computer program / instruction. When the computer program / instruction is executed by a processor, the above-mentioned rainy image deraining method is implemented.

[0040] In order to further verify the effectiveness of the model proposed in this application, the model proposed in this application is verified with the existing deraining method on the REAL dataset. Figure 2The comparison diagram of the deraining results of the model proposed in this application and the existing deraining methods is shown, where (a) is a rainy day image, (b) is the deraining result diagram of the DSC algorithm, (c) is the deraining result diagram of the DNN algorithm, (d) is the deraining result diagram of the GMM algorithm, (e) is the deraining result diagram of the GCANet algorithm, (f) is the deraining result diagram of the DID-MDN algorithm, (g) is the deraining result diagram of the JORDER algorithm, and (h) is the deraining result diagram of the model of this application. Figure 2 Both the DSC algorithm and the DNN algorithm leave some white rain streaks, and the rain removal effect is poor. The GMM and GCANet algorithms can remove most rain lines but blur the background and erase some details in the image. The DID-MDN algorithm almost removes all visible rain marks but has poor clarity, and the local texture of the restored image is unclear. The image restored by the JORDER algorithm is clearer and the background contour of the original image is restored, but the color of some areas of the image is darker, and the restoration of scene details is still not ideal. Compared with the above algorithms, the model proposed in this application not only removes complex rain lines of different scales but also retains the scene contour information in the image. The resulting image color is more saturated and the quality is higher.

[0041] The information entropy index is used to quantitatively compare the rain removal effects of different algorithms on real rainy scene images. Figure 3 The comparison chart of the rain removal effect between the model proposed in this application and the existing rain removal method is shown. Figure 3 Compared with other algorithms, the restoration results of the model of this application have a greater advantage in information entropy, and the corresponding values are generally higher than those of other comparison algorithms, which proves that the model of this application has stable rain removal performance, the restored image is clean, and the background edge is complete.

[0042] The above descriptions are merely examples of various embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any modifications or substitutions that can be readily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included within the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A rainy day image fault tolerance device based on an improved convolutional long short-term memory network, characterized in that: include: Feature extraction module and regional feature enhancement module; The feature extraction module is used to extract rain streak features from rainy images and efficiently learn the key areas of interest, outputting a feature map. The regional feature enhancement module is used to capture and remove raindrop information of different scales based on the feature map, and output an image after rain removal. The feature extraction module includes a plurality of sequentially connected feature extraction units, each of which includes a densely connected network and a convolutional long short-term memory network; the output of the previous feature extraction unit serves as the input of the densely connected network in the next feature extraction unit, and the output of the convolutional long short-term memory network in the previous feature extraction unit serves as the input of the convolutional long short-term memory network in the next feature extraction unit; the output of the last feature extraction unit is the feature map; The dense connection network includes a first Dense Layer layer, a second Dense Layer layer, a first splicing layer, a third Dense Layer layer, a fourth Dense Layer layer, a second splicing layer, a third splicing layer, and a fifth DenseLayer layer connected in sequence. The rainy day image is input into the first Dense Layer layer, the output of the first Dense Layer layer is input into the second Dense Layer layer, the output of the second Dense Layer layer and the output of the first Dense Layer layer are spliced in the first splicing layer, the output of the first splicing layer is input into the third Dense Layer layer, the output of the third Dense Layer layer is input into the fourth Dense Layer layer, the output of the fourth Dense Layer layer and the output of the third Dense Layer layer are spliced in the second splicing layer, the output of the second splicing layer and the output of the first splicing layer are spliced in the third splicing layer, the output of the third splicing layer is input into the fifth Dense Layer layer, the output of the fifth Dense Layer layer is the output of the dense connection network, and the output of the fifth Dense Layer layer is input into the convolutional long short-term memory network.

2. The device according to claim 1, wherein The densely connected network includes multiple Dense Layer layers. The input passes through the nonlinear composite function in each Dense Layer layer and outputs feature maps of the same specifications. The feature maps are spliced in the densely connected network according to the set splicing rules; the output of the densely connected network serves as the input of the convolutional long short-term memory network.

3. The device according to claim 1, wherein The feature extraction unit also includes a first convolutional layer, the output of the convolutional long short-term memory network is input into the first convolutional layer, the output of the first convolutional layer and the rainy day image are element-wise multiplied, and the result obtained is the output of the feature extraction unit.

4. The device according to claim 1, wherein The regional feature enhancement module includes an average pooling layer, a downsampling layer, a second convolutional layer, an upsampling layer, a fourth splicing layer and a third convolutional layer; the input of the average pooling layer is the feature map, the output of the average pooling layer passes through the downsampling layer to obtain downsampling results of different scales, the downsampling results of different scales pass through the second convolutional layer respectively to obtain convolution results of different scales, the convolution results of different scales are respectively input into the upsampling layer to obtain upsampling results, the upsampling results and the feature map are spliced in the fourth splicing layer to obtain a splicing result; the splicing result is input into the third convolutional layer, and the output of the third convolutional layer is the image after rain removal.

5. The device according to claim 1 or 2, characterized in that The Dense Layer layer includes a convolution layer, a RELU function and a pooling layer.

6. A rainy image deraining method, characterized in that: include: Get rainy day images; Inputting the rainy day image into a rainy day image fault-tolerant device based on an improved convolutional long short-term memory network to obtain a rain-free image; The rainy day image fault-tolerant device based on the improved convolutional long short-term memory network is the rainy day image fault-tolerant device based on the improved convolutional long short-term memory network according to any one of claims 1-5.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for removing rain from rainy images according to claim 6 is implemented.

8. A computer program product, characterized in that The method comprises a computer program / instruction, and when the computer program / instruction is executed by a processor, the method for removing rain from rainy images according to claim 6 is implemented.

Images