Image raindrop removal method, device, equipment and readable storage medium

Abstract

The present invention provides an image raindrop removal method, device, equipment and readable storage medium. The image raindrop removal method comprises: obtaining an image with raindrop occlusion collected by a camera, and inputting the image with raindrop occlusion into an encoder of a raindrop removal model; the encoder obtains a feature map based on the image with raindrop occlusion, and inputs the feature map into a decoder of the raindrop removal model; the decoder of the raindrop removal model obtains a target image after raindrop removal based on the feature map. By embedding the raindrop removal model of the present invention into a vehicle-mounted chip with limited computing power, high accuracy and high real-time performance of the image rain removal result can be achieved under the premise of low computing power consumption, so that the driver or the automatic driving control unit can use the visual information after rain removal to make more accurate driving behavior judgments and driving behavior operations, reduce the safety hazards of driving or parking in intelligent driving, and improve the safety factor and intelligence level of driving.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to an image raindrop removal method, device, equipment and readable storage medium. Background Art

[0002] With the popularization and deepening of intelligent assisted driving, more and more intelligent assisted driving technologies are being accepted and used by people, such as reversing images, 360-degree surround images, adaptive cruise control, etc. While these technologies bring convenience to the audience, their safety and potential risks are also receiving more and more attention. Among them, the rearview camera, as one of the most critical hardware for intelligent assisted driving, often causes the car's reversing image to be blurred or blocked due to raindrops in rainy environments or splashes of dirty muddy water on the ground, increasing safety hazards during driving or parking.

[0003] To address this issue, the solutions currently provided by mainstream intelligent driving assistance software suppliers rely on enhancing the robustness of algorithms, such as the calculation accuracy of obstacle detection algorithms and lane line detection algorithms in such scenarios, or using special mechanical structures to reduce the amount of raindrops on the camera image. However, software-based enhancement methods such as removing raindrops and dirt from reversing images have rarely been proposed or studied.

[0004] In addition, most scholars have proposed some technical solutions for image enhancement algorithm models for the problem of image deraining. Although these solutions have achieved certain results in the field of image deraining, there are also some common problems. First, most algorithm models use camera images after correction of distortion as data sets for training, and the raindrops in the data set samples are mostly artificially synthesized raindrops. These images have a large data distribution difference compared to the data collected by the real car rearview camera, resulting in limitations and low accuracy in the application of the trained model on the vehicle rearview wide-angle camera; second, most models use a relatively complex structure, which affects the real-time operation of the model, resulting in limited application on vehicle-mounted chips with limited computing power. Summary of the invention

[0005] The main purpose of the present invention is to provide an image raindrop removal method, device, equipment and readable storage medium, aiming to solve the technical problem that the image acquired by the existing intelligent assisted driving function related to the image is blocked by raindrops due to weather conditions, thereby causing safety hazards during driving.

[0006] In a first aspect, the present invention provides a method for removing raindrops from an image, the method comprising the following steps:

[0007] Acquire an image blocked by raindrops captured by a camera, and input the image blocked by raindrops into an encoder of a raindrop model;

[0008] The encoder obtains a feature map based on the image occluded by raindrops, and inputs the feature map into a decoder of a raindrop removal model;

[0009] The decoder of the raindrop removal model obtains the target image after the raindrops are removed based on the feature map.

[0010] Optionally, the step of the encoder obtaining a feature map based on the image occluded by raindrops, and inputting the feature map into a decoder of a raindrop removal model comprises:

[0011] The feature integration layer in the encoder resizes the image occluded by raindrops to obtain a first feature map;

[0012] The N encoding modules in the encoder receive the feature map, perform size reduction, weighted calculation, raindrop feature extraction and normalization on the feature map to obtain a second feature map;

[0013] Performing weighted calculation, raindrop feature extraction and normalization processing on the feature map to obtain a third feature map;

[0014] Input the third feature map into the decoder of the raindrop removal model;

[0015] Reducing the size of the third feature map to obtain a fourth feature map;

[0016] Performing residual processing on the second feature map and the fourth feature map to obtain a fifth feature map;

[0017] The feature map received by the first encoding module in the encoder is the first feature map, and the feature maps received by the second to N encoding modules are the fifth feature maps obtained by the first to N-1 encoding modules respectively;

[0018] The deep self-attention transformation network layer in the encoder receives the fifth feature map obtained by the Nth encoding module, performs weighted calculation and raindrop feature extraction on the fifth feature map obtained by the Nth encoding module, and obtains a sixth feature map;

[0019] Normalizing the sixth feature map to obtain a seventh feature map;

[0020] The seventh feature map is input into the decoder of the raindrop removal model.

[0021] Optionally, the decoder of the raindrop removal model obtains the target image after the raindrops are removed based on the feature map, comprising:

[0022] The N decoding modules in the decoder enlarge the size of the first branch feature map and extract raindrop features to obtain an eighth feature map, and perform residual processing on the eighth feature map and the second branch feature map to obtain a ninth feature map;

[0023] The first branch feature map received by the first decoding module in the decoder is the seventh feature map, and the second branch feature map received by the first decoding module in the decoder is the third feature map obtained by the Nth encoding module;

[0024] The first branch feature graphs received by the 2nd to Nth decoding modules in the decoder are the ninth feature graphs obtained by the 1st to N-1th decoding modules, respectively, and the second branch feature graphs received by the 2nd to Nth decoding modules in the decoder are the third feature graphs obtained by the N-1th to 1st encoding modules, respectively;

[0025] The deconvolutional neural network layer in the decoder receives the ninth feature map obtained by the Nth decoder;

[0026] The ninth feature map obtained by the Nth decoder is enlarged and raindrop features are extracted to obtain a target image after raindrops are removed.

[0027] Optionally, before the step of inputting the image occluded by raindrops into the trained raindrop removal model, the step includes:

[0028] Obtain an image to be processed in a training data set and a target raindrop-free image corresponding to the image to be processed;

[0029] Inputting the image to be processed into a raindrop removal model to be trained to obtain a first image after raindrops are removed;

[0030] Calculating a peak signal-to-noise ratio loss based on the first image and the target raindrop-free image;

[0031] Inputting the first image and the target raindrop-free image into a pre-trained VGG network for processing, obtaining VGG feature maps corresponding to the first image and the target raindrop-free image, respectively, and calculating the perceptual loss based on the VGG feature maps;

[0032] Based on the peak signal-to-noise ratio loss and the perceptual loss, a joint loss is obtained;

[0033] detecting whether the joint loss converges;

[0034] If convergence has not occurred, adjusting the parameters of the raindrop removal model to be trained based on the joint loss, using the raindrop removal model to be trained after the parameter adjustment as a new raindrop removal model to be trained, and returning to execute the step of obtaining the image to be processed in the training data set and the target raindrop-free image corresponding to the image to be processed;

[0035] If converged, the latest raindrop removal model to be trained is used as the trained raindrop removal model.

[0036] Optionally, the step of calculating the peak signal-to-noise ratio loss based on the first image and the target raindrop-free image includes:

[0037] The first image and the target raindrop-free image are input into a first formula to calculate the peak signal-to-noise ratio loss, wherein the first formula is:

[0038]

[0039] in, is the peak signal-to-noise ratio loss, is the first image, G is the target raindrop-free image, is a pixel on the first image, G(i,j) is a pixel on the target raindrop-free image, W is the width of the image, and H is the height of the image.

[0040] Optionally, the step of inputting the first image and the target raindrop-free image into a pre-trained VGG network for processing to obtain VGG feature maps corresponding to the first image and the target raindrop-free image, respectively, and calculating the perceptual loss based on the VGG feature maps includes:

[0041] Inputting the first image and the target raindrop-free image into a pre-trained VGG network for processing to obtain VGG feature maps corresponding to the first image and the target raindrop-free image respectively;

[0042] The VGG feature maps corresponding to the first image and the target raindrop-free image are respectively input into the second formula to calculate the perceptual loss, wherein the second formula is:

[0043]

[0044] in, is the perceived loss, is the first image, G is the target raindrop-free image, is the VGG feature map corresponding to the first image, and VGG(G) is the VGG feature map corresponding to the target raindrop-free image.

[0045] In a second aspect, the present invention further provides an image raindrop removal device, the image raindrop removal device comprising:

[0046] An input module, used for acquiring an image occluded by raindrops collected by a camera, and inputting the image occluded by raindrops into an encoder for removing the raindrop model;

[0047] An encoder of the raindrop removal model is used to obtain a feature map based on the image occluded by raindrops, and input the feature map into a decoder of the raindrop removal model;

[0048] The decoder of the raindrop removal model is used to obtain a target image after raindrops are removed based on the feature map.

[0049] Optionally, the encoder of the raindrop removal model includes:

[0050] Feature integration layer, specifically used for:

[0051] Resizing the image occluded by raindrops to obtain a first feature map;

[0052] N encoding modules, specifically used for:

[0053] receiving a feature map, performing size reduction, weighted calculation, raindrop feature extraction, and normalization processing on the feature map to obtain a second feature map;

[0054] Performing weighted calculation, raindrop feature extraction and normalization processing on the feature map to obtain a third feature map;

[0055] Input the third feature map into the decoder of the raindrop removal model;

[0056] Reducing the size of the third feature map to obtain a fourth feature map;

[0057] Performing residual processing on the second feature map and the fourth feature map to obtain a fifth feature map;

[0058] The feature map received by the first encoding module in the encoder is the first feature map, and the feature maps received by the second to N encoding modules are the fifth feature maps obtained by the first to N-1 encoding modules respectively;

[0059] Deep Self-Attention Transformer Network layer, specifically used for:

[0060] receiving a fifth feature map obtained by the Nth encoding module, performing weighted calculation and raindrop feature extraction on the fifth feature map obtained by the Nth encoding module to obtain a sixth feature map;

[0061] Normalizing the sixth feature map to obtain a seventh feature map;

[0062] The seventh feature map is input into the decoder of the raindrop removal model.

[0063] Optionally, the decoder of the raindrop removal model includes:

[0064] N decoding modules, used for enlarging the size of the first branch feature map and extracting raindrop features to obtain an eighth feature map, and performing residual processing on the eighth feature map and the second branch feature map to obtain a ninth feature map;

[0065] The first branch feature map received by the first decoding module in the decoder is the seventh feature map, and the second branch feature map received by the first decoding module in the decoder is the third feature map obtained by the Nth encoding module;

[0066] The first branch feature graphs received by the 2nd to Nth decoding modules in the decoder are the ninth feature graphs obtained by the 1st to N-1th decoding modules, respectively, and the second branch feature graphs received by the 2nd to Nth decoding modules in the decoder are the third feature graphs obtained by the N-1th to 1st encoding modules, respectively;

[0067] A deconvolutional neural network layer, for receiving the ninth feature map obtained by the Nth decoder;

[0068] The ninth feature map obtained by the Nth decoder is enlarged and raindrop features are extracted to obtain a target image after raindrops are removed.

[0069] Optionally, the image raindrop removal device further includes a training module, which is used to:

[0070] Obtain an image to be processed in a training data set and a target raindrop-free image corresponding to the image to be processed;

[0071] Inputting the image to be processed into a raindrop removal model to be trained to obtain a first image after raindrops are removed;

[0072] Calculating a peak signal-to-noise ratio loss based on the first image and the target raindrop-free image;

[0073] Inputting the first image and the target raindrop-free image into a pre-trained VGG network for processing, obtaining VGG feature maps corresponding to the first image and the target raindrop-free image, respectively, and calculating the perceptual loss based on the VGG feature maps;

[0074] Based on the peak signal-to-noise ratio loss and the perceptual loss, a joint loss is obtained;

[0075] detecting whether the joint loss converges;

[0076] If convergence has not occurred, adjusting the parameters of the raindrop removal model to be trained based on the joint loss, using the raindrop removal model to be trained after the parameter adjustment as a new raindrop removal model to be trained, and returning to execute the step of obtaining the image to be processed in the training data set and the target raindrop-free image corresponding to the image to be processed;

[0077] If converged, the latest raindrop removal model to be trained is used as the trained raindrop removal model.

[0078] Optionally, the training module is specifically used for:

[0079] The first image and the target raindrop-free image are input into a first formula to calculate the peak signal-to-noise ratio loss, wherein the first formula is:

[0080]

[0081] in, is the peak signal-to-noise ratio loss, is the first image, G is the target raindrop-free image, is a pixel on the first image, G(i,j) is a pixel on the target raindrop-free image, W is the width of the image, and H is the height of the image.

[0082] Optionally, the training module is specifically used for:

[0083] Inputting the first image and the target raindrop-free image into a pre-trained VGG network for processing to obtain VGG feature maps corresponding to the first image and the target raindrop-free image respectively;

[0084] The VGG feature maps corresponding to the first image and the target raindrop-free image are respectively input into the second formula to calculate the perceptual loss, wherein the second formula is:

[0085]

[0086] in, is the perceived loss, is the first image, G is the target raindrop-free image, is the VGG feature map corresponding to the first image, and VGG(G) is the VGG feature map corresponding to the target raindrop-free image.

[0087] In a third aspect, the present invention also provides an image raindrop removal device, which includes a processor, a memory, and an image raindrop removal program stored in the memory and executable by the processor, wherein when the image raindrop removal program is executed by the processor, the steps of the image raindrop removal method described above are implemented.

[0088] In a fourth aspect, the present invention further provides a readable storage medium, on which is stored an image raindrop removal program, wherein when the image raindrop removal program is executed by a processor, the steps of the image raindrop removal method as described above are implemented.

[0089] The present invention provides an image raindrop removal method, device, equipment and readable storage medium. The image raindrop removal method comprises: obtaining an image with raindrop occlusion collected by a camera, and inputting the image with raindrop occlusion into an encoder of a raindrop removal model; the encoder obtains a feature map based on the image with raindrop occlusion, and inputs the feature map into a decoder of the raindrop removal model; the decoder of the raindrop removal model obtains a target image after raindrop removal based on the feature map. By embedding the raindrop removal model of the present invention into a vehicle-mounted chip with limited computing power, high accuracy and high real-time performance of the image rain removal result can be achieved under the premise of low computing power consumption, so that the driver or the automatic driving control unit can use the visual information after rain removal to make more accurate driving behavior judgments and driving behavior operations, reduce the safety hazards of driving or parking in intelligent driving, and improve the safety factor and intelligence level of driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 Schematic diagram of the hardware structure of the image raindrop removal device involved in the embodiment of the present invention;

[0091] Figure 2 A schematic diagram of a process flow of an embodiment of a method for removing raindrops from an image according to the present invention;

[0092] Figure 3 It is a structural schematic diagram of a feature integration layer of an embodiment of an image raindrop removal method of the present invention;

[0093] Figure 4 This is a schematic diagram of the structure of a deep self-attention network in an embodiment of the image raindrop removal method of the present invention;

[0094] Figure 5 It is a structural schematic diagram of a raindrop removal model according to an embodiment of a method for removing raindrops from an image of the present invention;

[0095] Figure 6 FIG. 4 is a schematic diagram of functional modules of an image raindrop removal device according to an embodiment of the present invention.

[0096] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0097] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0098] In a first aspect, an embodiment of the present invention provides an image raindrop removal device.

[0099] Reference Figure 1 , Figure 1Schematic diagram of the hardware structure of the image raindrop removal device involved in the embodiment of the present invention. In the embodiment of the present invention, the image raindrop removal device may include a processor 1001 (e.g., a central processing unit, CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components; the user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard); the network interface 1004 may optionally include a standard wired interface, a wireless interface (such as wireless fidelity WIreless-FIdelity, WI-FI interface); the memory 1005 may be a high-speed random access memory (random access memory, RAM), or a stable memory (non-volatile memory), such as a disk storage, and the memory 1005 may optionally be a storage device independent of the aforementioned processor 1001. Those skilled in the art will understand that Figure 1 The hardware structure shown in the figure does not constitute a limitation of the present invention, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0100] Continue to refer to Figure 1 , Figure 1 The memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an image raindrop removal program. The processor 1001 may call the image raindrop removal program stored in the memory 1005 and execute the image raindrop removal method provided by the embodiment of the present invention.

[0101] In a second aspect, an embodiment of the present invention provides a method for removing raindrops from an image.

[0102] Reference Figure 2 , Figure 2 FIG. 4 is a flow chart of an embodiment of a method for removing raindrops from an image according to the present invention.

[0103] In an embodiment of the image raindrop removal method of the present invention, the image raindrop removal method includes:

[0104] Step S10, obtaining an image blocked by raindrops collected by a camera, and inputting the image blocked by raindrops into an encoder for removing the raindrop model;

[0105] Step S20, the encoder obtains a feature map based on the image occluded by raindrops, and inputs the feature map into a decoder of a raindrop removal model;

[0106] Step S30: The decoder of the raindrop removal model obtains the target image after the raindrops are removed based on the feature map.

[0107] In this embodiment, considering that the camera is one of the hardware of the existing intelligent driving assistance function, the image it collects may appear blurred or blocked due to weather conditions, resulting in safety hazards during the driving or parking process of intelligent driving. Most of the existing solutions are to enhance the robustness (computational accuracy) of the algorithm (such as obstacle detection algorithm, lane line detection algorithm) in such situations, or use special mechanical structures to reduce the image of raindrops facing the camera, but there is no software method for removing raindrops from images.

[0108] Therefore, this embodiment provides a software method for removing raindrops from an image in response to the situation where the image captured by the camera is blocked by raindrops when the vehicle is driving on a rainy road. In the above method, firstly, the image blocked by raindrops captured by the camera is obtained, and then the image blocked by raindrops is input into the encoder of the raindrop removal model. The encoder of the raindrop removal model obtains a number of feature maps based on the input image blocked by raindrops, and then inputs the several feature maps into the decoder of the raindrop removal model. When the decoder of the raindrop removal model receives the several feature maps input by the encoder, it can obtain the target image after the raindrops are removed based on the several feature maps input by the encoder.

[0109] After obtaining the target image after raindrop removal based on the above method, the lane lines, obstacles, etc. in the corresponding rainy driving environment can be clearly displayed, which is conducive to the driver or the automatic driving control unit to make more accurate driving behavior judgments and driving behavior operations based on the corresponding target image, thereby reducing the safety hazards of driving or parking processes in intelligent driving. In addition, the above-mentioned raindrop removal model extracts raindrop features based on a deep self-attention transformation network, and the encoding-decoding structure adopted is relatively simple, which can ensure the high accuracy and high real-time performance of the image raindrop removal function under the actual vehicle computing power. After testing, the raindrop removal model of the embodiment can reach a running speed of 34.5ms on an edge device with a computing power similar to that of an on-board chip, and can be widely used to run on on-board chips with limited computing power, making up for the lack of similar solutions in mainstream cars and software.

[0110] Further, in one embodiment, the step S20 includes:

[0111] The feature integration layer in the encoder resizes the image occluded by raindrops to obtain a first feature map;

[0112] The N encoding modules in the encoder receive the feature map, perform size reduction, weighted calculation, raindrop feature extraction and normalization on the feature map to obtain a second feature map;

[0113] Performing weighted calculation, raindrop feature extraction and normalization processing on the feature map to obtain a third feature map;

[0114] Input the third feature map into the decoder of the raindrop removal model;

[0115] Reducing the size of the third feature map to obtain a fourth feature map;

[0116] Performing residual processing on the second feature map and the fourth feature map to obtain a fifth feature map;

[0117] The feature map received by the first encoding module in the encoder is the first feature map, and the feature maps received by the second to N encoding modules are the fifth feature maps obtained by the first to N-1 encoding modules respectively;

[0118] The deep self-attention transformation network layer in the encoder receives the fifth feature map obtained by the Nth encoding module, performs weighted calculation and raindrop feature extraction on the fifth feature map obtained by the Nth encoding module, and obtains a sixth feature map;

[0119] Normalizing the sixth feature map to obtain a seventh feature map;

[0120] The seventh feature map is input into the decoder of the raindrop removal model.

[0121] In this embodiment, the encoder of the raindrop removal model includes a feature integration layer, N encoding modules, a deep self-attention network layer, and an LN layer (i.e., LayerNorm, layer normalization). Among them, N is based on the actual model size requirement, that is, it can be determined based on the amount of image data that needs to be processed for training. When the amount of image data is not large, in order to prevent the image data from being over-fitted, the model size requirement is not large, and the value of N is not large; it can also be determined according to different model size requirements. In order to achieve a faster model inference speed while ensuring that the raindrop removal accuracy meets the established requirements, the model size can be appropriately reduced. Specifically, refer to Figure 3 In the left half of the image, taking the encoding module as 3 as an example, the encoder of the raindrop removal model obtains a plurality of feature maps based on the input image with raindrop occlusion, and then inputs the plurality of feature maps into the decoder of the raindrop removal model, including the following steps:

[0122] First, the image occluded by raindrops is resized using the feature integration layer in the encoder to obtain the first feature map enc1.

[0123] The first encoding module in the encoder receives the first feature map enc1, and through the first branch on the left, performs size reduction, weighted calculation, raindrop feature extraction and normalization on the feature map to obtain the second feature map enc2-1. Through the second branch on the right, weighted calculation, raindrop feature extraction and normalization are performed on the feature map to obtain the third feature map enc3-1 (the third feature map enc3-1 needs to be input into the decoder of the raindrop removal model), and the third feature map enc3-1 is reduced in size to obtain the fourth feature map enc4-1. The second feature map enc2-1 and the fourth feature map enc4-1 obtained by the two branches are subjected to residual processing to obtain the fifth feature map enc5-1 (the fifth feature map enc5-1 needs to be input into the next encoding module).

[0124] The second encoding module in the encoder receives the fifth feature map enc5-1 output by the first encoding module, and through the first branch on the left, performs size reduction, weighted calculation, raindrop feature extraction and normalization on the feature map to obtain the second feature map enc2-2. Through the second branch on the right, weighted calculation, raindrop feature extraction and normalization are performed on the feature map to obtain the third feature map enc3-2 (the third feature map enc3-2 needs to be input into the decoder of the raindrop removal model), and the third feature map enc3-2 is reduced in size to obtain the fourth feature map enc4-2. The second feature map enc2-2 and the fourth feature map enc4-2 obtained by the two branches are subjected to residual processing to obtain the fifth feature map enc5-2 (the fifth feature map enc5-2 needs to be input into the next encoding module).

[0125] The third encoding module in the encoder receives the fifth feature map enc5-2 output by the second encoding module, and through the first branch on the left, performs size reduction, weighted calculation, raindrop feature extraction and normalization on the feature map to obtain the second feature map enc2-3. Through the second branch on the right, the feature map is weighted calculated, raindrop feature extracted and normalized to obtain the third feature map enc3-3 (the third feature map enc3-3 needs to be input into the decoder of the raindrop removal model), and the third feature map enc3-3 is reduced in size to obtain the fourth feature map enc4-3. The second feature map enc2-3 and the fourth feature map enc4-3 obtained by the two branches are subjected to residual processing to obtain the fifth feature map enc5-3 (the fifth feature map enc5-3 needs to be input into the deep self-attention transformation network layer).

[0126] The deep self-attention transformation network layer in the encoder receives the fifth feature map enc5-3 obtained by the third encoding module, performs weighted calculation and raindrop feature extraction on the fifth feature map enc5-3 obtained by the third encoding module, and obtains the sixth feature map enc6. After the sixth feature map enc6 is normalized by the LN layer, the seventh feature map enc7 can be obtained (the seventh feature map enc7 needs to be input into the decoder of the raindrop removal model).

[0127] The structure of the feature integration layer is as follows: Figure 4 , mainly composed of a convolutional neural network layer (reducing the image size), a flattening layer (reducing the dimension of the feature map to prepare for the extraction of raindrop features that need to be focused on) and a LN layer (normalizing the feature map). Assume that the image size input to the feature integration layer is (B, W, H, C), where B is the training batch size (the total number of images input to the raindrop removal model), W is the image width, H is the image height, and C is the number of image channels. After passing through the convolutional neural network, the output feature map is obtained, and its size is set to (B, W2, H2, C2), where W2 is smaller than W and H2 is smaller than H. Then, after this feature map is processed by the flattening layer and the LN layer, the final output of the output size (B, N, C2) will be obtained, where N is the product of W2 and H2. That is, the feature integration layer can reduce the length and width of the image to save computing resources, thereby reducing the time for the model to process the image and ensuring the high real-time performance of the model calculation.

[0128] In addition, the above-mentioned deep self-attention transformation network layer contains several deep self-attention transformation networks. The structure of the deep self-attention transformation network refers to Figure 5 . Assume that the input to the deep self-attention transformer network is a feature map in, and after a layer of image normalization processing of the LN layer, the feature map m1 is obtained. Then, the feature map m1 is processed by three parallel branches to obtain the outputs Q, K, and V. Among them, the first branch and the second branch have the same structure, consisting of a convolutional neural network layer (extracting input features), a LN layer (normalizing the feature map) and a linear layer (performing feature integration processing in the channel direction of the feature map). The outputs of these two branches are defined as K and V respectively. The third branch consists of a single linear layer (directly performing feature integration processing on the channel direction of the input feature map), and the output obtained is called Q. The generated Q, K, and V are calculated by the following formula to obtain the feature map atten.

[0129] F(Q,K,V)=(Q×K T ) T ×V

[0130] After the feature map atten is normalized by a linear layer, it is subjected to a residual processing (i.e., matrix addition operation) with the original input feature map in to obtain the feature map res1. Subsequently, the feature map res1 is processed in sequence by a LN layer normalization processing, a linear layer feature integration in the image channel direction, a deep separation convolutional neural network feature integration processing in the image length and width direction, and a linear layer normalization processing to obtain the feature map res2. The feature map res2 is again subjected to a residual processing (i.e., matrix addition operation) with the original input feature map in to obtain the final output of the deep self-attention transform network. Based on the structural characteristics of the deep self-attention transform network, the above-mentioned raindrop removal model using the deep self-attention network is applied to the process of image raindrop removal, so that the final output image obtained by the processing can have a better overall processing effect, a smoother image, and a faster speed than the final output image obtained by other models that do not use the above-mentioned deep self-attention transform network, that is, the high accuracy and high real-time performance of the model calculation can be guaranteed.

[0131] Further, in one embodiment, the step S30 includes:

[0132] The N decoding modules in the decoder enlarge the size of the first branch feature map and extract raindrop features to obtain an eighth feature map, and perform residual processing on the eighth feature map and the second branch feature map to obtain a ninth feature map;

[0133] The first branch feature map received by the first decoding module in the decoder is the seventh feature map, and the second branch feature map received by the first decoding module in the decoder is the third feature map obtained by the Nth encoding module;

[0134] The first branch feature graphs received by the 2nd to Nth decoding modules in the decoder are the ninth feature graphs obtained by the 1st to N-1th decoding modules, respectively, and the second branch feature graphs received by the 2nd to Nth decoding modules in the decoder are the third feature graphs obtained by the N-1th to 1st encoding modules, respectively;

[0135] The deconvolutional neural network layer in the decoder receives the ninth feature map obtained by the Nth decoder;

[0136] The ninth feature map obtained by the Nth decoder is enlarged and raindrop features are extracted to obtain a target image after raindrops are removed.

[0137] In this embodiment, the decoder of the raindrop removal model includes N decoding modules and a deconvolution neural network layer. Where N is determined based on the actual model size requirements. The deconvolution neural network layer includes several deconvolution neural networks. Specifically, refer to Figure 3In the right half of the figure, taking the decoding modules as 3 as an example, the decoder of the raindrop removal model obtains the target image after the raindrops are removed based on the feature map, including the following steps:

[0138] The first branch feature map received by the first decoding module in the decoder is the seventh feature map enc7 output by the encoder, and the second branch feature map is the third feature map enc3-3 obtained by the third encoding module in the encoder. The decoding process of the first decoding module is: the first branch feature map enc7 is enlarged and the raindrop feature is extracted to obtain the eighth feature map enc8-1. The eighth feature map enc8-1 and the second branch feature map enc3-3 are then subjected to residual processing to obtain the ninth feature map enc9-1 (the ninth feature map enc9-1 needs to be input to the next decoding module).

[0139] The first branch feature map received by the second decoding module in the decoder is the ninth feature map enc9-1 output by the previous decoding module, and the second branch feature map is the third feature map enc3-2 obtained by the second encoding module in the encoder. The decoding process of the second decoding module is: the first branch feature map enc9-1 is enlarged and the raindrop feature is extracted to obtain the eighth feature map enc8-2. The eighth feature map enc8-2 and the second branch feature map enc3-2 are then subjected to residual processing to obtain the ninth feature map enc9-2 (the ninth feature map enc9-2 needs to be input to the next decoding module).

[0140] The first branch feature map received by the third decoding module in the decoder is the ninth feature map enc9-2 output by the previous decoding module, and the second branch feature map is the third feature map enc3-1 obtained by the second encoding module in the encoder. The decoding process of the second decoding module is: the first branch feature map enc9-2 is enlarged and the raindrop feature is extracted to obtain the eighth feature map enc8-3. The eighth feature map enc8-3 and the second branch feature map enc3-1 are then subjected to residual processing to obtain the ninth feature map enc9-3 (the ninth feature map enc9-3 needs to be input to the inverse convolutional neural network layer).

[0141] The inverse convolutional neural network layer in the decoder receives the ninth feature map enc9-3 obtained by the third decoder, and further enlarges the size and extracts raindrop features of the ninth feature map enc9-3 obtained by the third decoder to obtain the target image after raindrops are removed.

[0142] Among them, the decoding process in each decoding module will accept the input of the feature maps of the two branches for hierarchical residual processing. Compared with the non-hierarchical residual, the resulting image effect is better and the accuracy is higher.

[0143] Furthermore, in one embodiment, the step before step S10 includes:

[0144] Obtain an image to be processed in a training data set and a target raindrop-free image corresponding to the image to be processed;

[0145] Inputting the image to be processed into a raindrop removal model to be trained to obtain a first image after raindrops are removed;

[0146] Calculating a peak signal-to-noise ratio loss based on the first image and the target raindrop-free image;

[0147] Inputting the first image and the target raindrop-free image into a pre-trained VGG network for processing, obtaining VGG feature maps corresponding to the first image and the target raindrop-free image, respectively, and calculating the perceptual loss based on the VGG feature maps;

[0148] Based on the peak signal-to-noise ratio loss and the perceptual loss, a joint loss is obtained;

[0149] detecting whether the joint loss converges;

[0150] If convergence has not occurred, adjusting the parameters of the raindrop removal model to be trained based on the joint loss, using the raindrop removal model to be trained after the parameter adjustment as a new raindrop removal model to be trained, and returning to execute the step of obtaining the image to be processed in the training data set and the target raindrop-free image corresponding to the image to be processed;

[0151] If converged, the latest raindrop removal model to be trained is used as the trained raindrop removal model.

[0152] In this embodiment, before the above-mentioned raindrop removal model is applied to the vehicle-mounted computing platform, the raindrop removal model needs to be trained, and at this time, a training data set needs to be obtained first. Among them, the raindrop image training data sets for model training, verification and testing in this embodiment are all collected during the actual vehicle test process, and all images have been screened, cleaned, and desensitized. For example, for reversing images, the image data of the training data set are all collected from the actual vehicle-mounted rear-view wide-angle camera, and its viewing angle range is 195°. The data set consists of 4437 groups of image data. The image data collection scenes include outdoor daytime on rainy days, outdoor daytime on sunny days, outdoor evening, and underground parking lots; the image data raindrop occlusion scenes are divided into small-area raindrop occlusion, large-area raindrop occlusion, and no raindrop occlusion.

[0153] Among them, collecting images of small raindrops and large raindrops can make it closer to the simulation process of raining, which increases the universality of the raindrop removal model. The large raindrop and small raindrop image data collected by the on-board camera of a real car in a real scene are randomly scattered to form a data set and then input into the raindrop removal model for training, so that the image raindrop removal function of this embodiment scheme can be highly practical. In this embodiment scheme, for the training process, the raindrop removal model is trained based on input data with an image length and width of (512, 512); for the model application process, the raindrop removal model can be inferred with images of any output size.

[0154] Specifically, before obtaining an image with raindrop occlusion captured by a camera and inputting the image with raindrop occlusion into the encoder of the raindrop removal model: it is necessary to first obtain the image to be processed in the training data set and the target raindrop-free image (i.e., the corresponding true value image) corresponding to the image to be processed. Then, the above-mentioned image to be processed is input into the raindrop removal model to be trained to obtain the first image after the raindrops are removed.

[0155] Based on the first image and the target raindrop-free image, the peak signal-to-noise ratio loss is calculated. The first image and the target raindrop-free image are input into the pre-trained VGG network for feature map calculation to obtain the VGG feature maps corresponding to the first image and the target raindrop-free image, respectively, and the perceptual loss is calculated based on the VGG feature map. Among them, the VGG network can be pre-trained through the imagenet classification task, and the VGG feature map is a feature map obtained by extracting the feature map output by a certain layer of the VGG network when the first image and the target raindrop-free image are processed layer by layer by the pre-trained VGG network. Based on the peak signal-to-noise ratio loss and the perceptual loss, a joint loss is obtained. Among them, the joint loss function expression used in model training is as follows:

[0156]

[0157] in, represents the predicted image obtained after the input image is processed by the raindrop removal model to be trained; G represents the true value image (i.e., the target raindrop-free image corresponding to the image to be processed); L PSNR and L percep Refers to the peak signal-to-noise ratio loss function and the perceptual loss function.

[0158] After determining the joint loss, it is detected whether the joint loss converges to determine whether the training is completed. If it does not converge, the parameters of the raindrop removal model to be trained are adjusted based on the joint loss, the raindrop removal model to be trained with the adjusted parameters is used as the new raindrop removal model to be trained, and the step of obtaining the image to be processed in the training data set and the target raindrop-free image corresponding to the image to be processed is returned to execute; if it converges, the training is completed, and the latest raindrop removal model to be trained is used as the trained raindrop removal model.

[0159] All real vehicle image data in this embodiment are divided into a training set, a calibration set, and a test set in a ratio of 7:1:2. After the raindrop removal model is debugged and trained, the trained raindrop removal model is tested on the test set, and the output image signal-to-noise ratio and structural similarity results are shown in Table 1.

[0160] Table 1

[0161]

[0162] Among them, in all acquisition scenarios, the larger the peak value of the peak signal-to-noise ratio of the image, the smaller the loss and the better the model performance; the greater the structural similarity, that is, the closer to the maximum value 1 of the structural similarity index, the better the image effect and the better the model performance. Therefore, referring to Table 1, it can be seen that the scheme has a better effect of removing raindrop occlusion for images with large and small raindrop occlusions.

[0163] Furthermore, in one embodiment, the step of calculating the peak signal-to-noise ratio loss based on the first image and the target raindrop-free image includes:

[0164] The first image and the target raindrop-free image are input into a first formula to calculate the peak signal-to-noise ratio loss, wherein the first formula is:

[0165]

[0166] in, is the peak signal-to-noise ratio loss, is the first image, G is the target raindrop-free image, is a pixel on the first image, G(i,j) is a pixel on the target raindrop-free image, W is the width of the image, and H is the height of the image.

[0167] In this embodiment, the step of calculating the peak signal-to-noise ratio loss based on the first image and the target raindrop-free image includes:

[0168] The first image and the target raindrop-free image are input into the first formula to calculate the peak signal-to-noise ratio loss, where the peak signal-to-noise ratio loss function L PSNRIt is directly generated by using the calculation formula of the peak signal-to-noise ratio. Specifically, the first formula is:

[0169]

[0170] in, is the peak signal-to-noise ratio loss, is the first image, G is the target raindrop-free image, is a pixel on the first image, G(i,j) is a pixel on the target raindrop-free image, W is the width of the image, and H is the height of the image.

[0171] Furthermore, in one embodiment, the step of inputting the first image and the target raindrop-free image into a pre-trained VGG network for processing, obtaining VGG feature maps corresponding to the first image and the target raindrop-free image respectively, and calculating the perceptual loss based on the VGG feature maps includes:

[0172] Inputting the first image and the target raindrop-free image into a pre-trained VGG network for processing to obtain VGG feature maps corresponding to the first image and the target raindrop-free image respectively;

[0173] The VGG feature maps corresponding to the first image and the target raindrop-free image are respectively input into the second formula to calculate the perceptual loss, wherein the second formula is:

[0174]

[0175] in, is the perceived loss, is the first image, G is the target raindrop-free image, is the VGG feature map corresponding to the first image, and VGG(G) is the VGG feature map corresponding to the target raindrop-free image.

[0176] In this embodiment, the step of inputting the first image and the target raindrop-free image into a pre-trained VGG network for processing to obtain corresponding processing results, and calculating the perceptual loss based on the processing results includes:

[0177] Inputting the first image and the target raindrop-free image into a pre-trained VGG network for processing to obtain corresponding processing results;

[0178] The processing result is input into the second formula to calculate the perceptual loss, wherein the second formula is:

[0179]

[0180] in, is the perceived loss, is the first image, G is the target raindrop-free image, is the VGG feature map corresponding to the first image, and VGG(G) is the VGG feature map corresponding to the target raindrop-free image.

[0181] In this embodiment, a method for removing raindrops from an image is provided, including: obtaining an image occluded by raindrops collected by a camera, and inputting the image occluded by raindrops into an encoder of a raindrop removal model; the encoder obtains a feature map based on the image occluded by raindrops, and inputs the feature map into a decoder of the raindrop removal model; the decoder of the raindrop removal model obtains a target image after raindrops are removed based on the feature map. By embedding the raindrop removal model of the present invention into a vehicle-mounted chip with limited computing power, high accuracy and high real-time performance of the image rain removal results can be achieved under the premise of low computing power consumption, thereby facilitating the driver or the automatic driving control unit to use the visual information after rain removal to make more accurate driving behavior judgments and driving behavior operations, reduce safety hazards in driving or parking processes in intelligent driving, and improve the safety factor and intelligence level of driving.

[0182] In a third aspect, an embodiment of the present invention further provides an image raindrop removal device.

[0183] Reference Figure 6 , a schematic diagram of functional modules of an embodiment of an image raindrop removal device.

[0184] In this embodiment, the image raindrop removal device includes:

[0185] An input module 10 is used to obtain an image with raindrop occlusion collected by a camera, and input the image with raindrop occlusion into an encoder of a raindrop removal model;

[0186] The encoder 20 of the raindrop removal model is used to obtain a feature map based on the image occluded by raindrops, and input the feature map into the decoder of the raindrop removal model;

[0187] The decoder 30 of the raindrop removal model is used to obtain a target image after raindrops are removed based on the feature map.

[0188] Further, in one embodiment, the encoder 20 of the raindrop removal model includes:

[0189] Feature integration layer, specifically used for:

[0190] Resizing the image occluded by raindrops to obtain a first feature map;

[0191] N encoding modules, specifically used for:

[0192] receiving a feature map, performing size reduction, weighted calculation, raindrop feature extraction, and normalization processing on the feature map to obtain a second feature map;

[0193] Performing weighted calculation, raindrop feature extraction and normalization processing on the feature map to obtain a third feature map;

[0194] Input the third feature map into the decoder of the raindrop removal model;

[0195] Reducing the size of the third feature map to obtain a fourth feature map;

[0196] Performing residual processing on the second feature map and the fourth feature map to obtain a fifth feature map;

[0197] The feature map received by the first encoding module in the encoder is the first feature map, and the feature maps received by the second to N encoding modules are the fifth feature maps obtained by the first to N-1 encoding modules respectively;

[0198] Deep Self-Attention Transformer Network layer, specifically used for:

[0199] receiving a fifth feature map obtained by the Nth encoding module, performing weighted calculation and raindrop feature extraction on the fifth feature map obtained by the Nth encoding module to obtain a sixth feature map;

[0200] Normalizing the sixth feature map to obtain a seventh feature map;

[0201] The seventh feature map is input into the decoder of the raindrop removal model.

[0202] Further, in one embodiment, the decoder 30 of the raindrop removal model includes:

[0203] N decoding modules, used for enlarging the size of the first branch feature map and extracting raindrop features to obtain an eighth feature map, and performing residual processing on the eighth feature map and the second branch feature map to obtain a ninth feature map;

[0204] The first branch feature map received by the first decoding module in the decoder is the seventh feature map, and the second branch feature map received by the first decoding module in the decoder is the third feature map obtained by the Nth encoding module;

[0205] The first branch feature graphs received by the 2nd to Nth decoding modules in the decoder are the ninth feature graphs obtained by the 1st to N-1th decoding modules, respectively, and the second branch feature graphs received by the 2nd to Nth decoding modules in the decoder are the third feature graphs obtained by the N-1th to 1st encoding modules, respectively;

[0206] A deconvolutional neural network layer, for receiving the ninth feature map obtained by the Nth decoder;

[0207] The ninth feature map obtained by the Nth decoder is enlarged and raindrop features are extracted to obtain a target image after raindrops are removed.

[0208] Furthermore, in one embodiment, the image raindrop removal device further includes a training module, which is used to:

[0209] Obtain an image to be processed in a training data set and a target raindrop-free image corresponding to the image to be processed;

[0210] Inputting the image to be processed into a raindrop removal model to be trained to obtain a first image after raindrops are removed;

[0211] Calculating a peak signal-to-noise ratio loss based on the first image and the target raindrop-free image;

[0212] Inputting the first image and the target raindrop-free image into a pre-trained VGG network for processing, obtaining VGG feature maps corresponding to the first image and the target raindrop-free image, respectively, and calculating the perceptual loss based on the VGG feature maps;

[0213] Based on the peak signal-to-noise ratio loss and the perceptual loss, a joint loss is obtained;

[0214] detecting whether the joint loss converges;

[0215] If convergence has not occurred, adjusting the parameters of the raindrop removal model to be trained based on the joint loss, using the raindrop removal model to be trained after the parameter adjustment as a new raindrop removal model to be trained, and returning to execute the step of obtaining the image to be processed in the training data set and the target raindrop-free image corresponding to the image to be processed;

[0216] If converged, the latest raindrop removal model to be trained is used as the trained raindrop removal model.

[0217] Furthermore, in one embodiment, the training module is specifically used for:

[0218] The first image and the target raindrop-free image are input into a first formula to calculate the peak signal-to-noise ratio loss, wherein the first formula is:

[0219]

[0220] in, is the peak signal-to-noise ratio loss, is the first image, G is the target raindrop-free image, is a pixel on the first image, G(i,j) is a pixel on the target raindrop-free image, W is the width of the image, and H is the height of the image.

[0221] Furthermore, in one embodiment, the training module is specifically used for:

[0222] Inputting the first image and the target raindrop-free image into a pre-trained VGG network for processing to obtain VGG feature maps corresponding to the first image and the target raindrop-free image respectively;

[0223] The VGG feature maps corresponding to the first image and the target raindrop-free image are respectively input into the second formula to calculate the perceptual loss, wherein the second formula is:

[0224]

[0225] in, is the perceived loss, is the first image, G is the target raindrop-free image, is the VGG feature map corresponding to the first image, and VGG(G) is the VGG feature map corresponding to the target raindrop-free image.

[0226] Among them, the functional implementation of each module in the above-mentioned image raindrop removal device corresponds to each step in the above-mentioned image raindrop removal method embodiment, and its functions and implementation processes are no longer repeated here.

[0227] In a fourth aspect, an embodiment of the present invention further provides a readable storage medium.

[0228] The image raindrop removal program is stored on the readable storage medium of the present invention, wherein when the image raindrop removal program is executed by a processor, the steps of the image raindrop removal method as described above are implemented.

[0229] The method implemented when the image raindrop removal program is executed can refer to the various embodiments of the image raindrop removal method of the present invention, and will not be described in detail here.

[0230] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0231] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0232] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present invention.

[0233] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for removing raindrops from an image, characterized in that: The image raindrop removal method comprises: Acquire an image blocked by raindrops captured by a camera, and input the image blocked by raindrops into an encoder of a raindrop model; The encoder obtains a feature map based on the image occluded by raindrops, and inputs the feature map into a decoder of a raindrop removal model; The decoder of the raindrop removal model obtains the target image after the raindrops are removed based on the feature map; The encoder obtains a feature map based on the image blocked by raindrops, and the step of inputting the feature map into a decoder of a raindrop removal model comprises: The feature integration layer in the encoder resizes the image occluded by raindrops to obtain a first feature map; The N encoding modules in the encoder receive the feature map, perform size reduction, weighted calculation, raindrop feature extraction and normalization on the feature map to obtain a second feature map; Performing weighted calculation, raindrop feature extraction and normalization processing on the feature map to obtain a third feature map; Input the third feature map into the decoder of the raindrop removal model; Reducing the size of the third feature map to obtain a fourth feature map; Performing residual processing on the second feature map and the fourth feature map to obtain a fifth feature map; The feature map received by the first encoding module in the encoder is the first feature map, and the feature maps received by the second to N encoding modules are the fifth feature maps obtained by the first to N-1 encoding modules respectively; The deep self-attention transformation network layer in the encoder receives the fifth feature map obtained by the Nth encoding module, performs weighted calculation and raindrop feature extraction on the fifth feature map obtained by the Nth encoding module, and obtains a sixth feature map; Normalizing the sixth feature map to obtain a seventh feature map; The seventh feature map is input into the decoder of the raindrop removal model.

2. The image raindrop removal method according to claim 1, characterized in that: The decoder of the raindrop removal model obtains the target image after the raindrops are removed based on the feature map, comprising: The N decoding modules in the decoder enlarge the size of the first branch feature map and extract raindrop features to obtain an eighth feature map, and perform residual processing on the eighth feature map and the second branch feature map to obtain a ninth feature map; The first branch feature map received by the first decoding module in the decoder is the seventh feature map, and the second branch feature map received by the first decoding module in the decoder is the third feature map obtained by the Nth encoding module; The first branch feature graphs received by the 2nd to Nth decoding modules in the decoder are the ninth feature graphs obtained by the 1st to N-1th decoding modules, respectively, and the second branch feature graphs received by the 2nd to Nth decoding modules in the decoder are the third feature graphs obtained by the N-1th to 1st encoding modules, respectively; The deconvolutional neural network layer in the decoder receives the ninth feature map obtained by the Nth decoder; The ninth feature map obtained by the Nth decoder is enlarged and raindrop features are extracted to obtain a target image after raindrops are removed.

3. The image raindrop removal method according to claim 1, characterized in that: Before the step of inputting the image blocked by raindrops into the trained raindrop removal model, the method includes: Obtain an image to be processed in a training data set and a target raindrop-free image corresponding to the image to be processed; Inputting the image to be processed into a raindrop removal model to be trained to obtain a first image after raindrops are removed; Calculating a peak signal-to-noise ratio loss based on the first image and the target raindrop-free image; Inputting the first image and the target raindrop-free image into a pre-trained VGG network for processing, obtaining VGG feature maps corresponding to the first image and the target raindrop-free image, respectively, and calculating the perceptual loss based on the VGG feature maps; Based on the peak signal-to-noise ratio loss and the perceptual loss, a joint loss is obtained; detecting whether the joint loss converges; If convergence has not occurred, adjusting the parameters of the raindrop removal model to be trained based on the joint loss, using the raindrop removal model to be trained after the parameter adjustment as a new raindrop removal model to be trained, and returning to execute the step of obtaining the image to be processed in the training data set and the target raindrop-free image corresponding to the image to be processed; If converged, the latest raindrop removal model to be trained is used as the trained raindrop removal model.

4. The image raindrop removal method according to claim 3, characterized in that: The step of calculating the peak signal-to-noise ratio loss based on the first image and the target raindrop-free image comprises: The first image and the target raindrop-free image are input into a first formula to calculate the peak signal-to-noise ratio loss, wherein the first formula is: in, is the peak signal-to-noise ratio loss, is the first image, G is the target raindrop-free image, is a pixel on the first image, G(i,j) is a pixel on the target raindrop-free image, W is the width of the image, and H is the height of the image.

5. The image raindrop removal method according to claim 3, characterized in that: The step of inputting the first image and the target raindrop-free image into a pre-trained VGG network for processing to obtain VGG feature maps corresponding to the first image and the target raindrop-free image, respectively, and calculating the perceptual loss based on the VGG feature maps includes: Inputting the first image and the target raindrop-free image into a pre-trained VGG network for processing to obtain VGG feature maps corresponding to the first image and the target raindrop-free image respectively; The VGG feature maps corresponding to the first image and the target raindrop-free image are respectively input into the second formula to calculate the perceptual loss, wherein the second formula is: in, is the perceived loss, is the first image, G is the target raindrop-free image, is the VGG feature map corresponding to the first image, and VGG(G) is the VGG feature map corresponding to the target raindrop-free image.

6. An image raindrop removal device for implementing the image raindrop removal method according to claim 1, characterized in that: The image raindrop removal device comprises: An input module, used for acquiring an image occluded by raindrops collected by a camera, and inputting the image occluded by raindrops into an encoder for removing the raindrop model; An encoder of the raindrop removal model is used to obtain a feature map based on the image occluded by raindrops, and input the feature map into a decoder of the raindrop removal model; The decoder of the raindrop removal model is used to obtain a target image after raindrops are removed based on the feature map.

7. The image raindrop removal device according to claim 6, characterized in that: The encoder of the raindrop removal model includes: Feature integration layer, specifically used for: Resizing the image occluded by raindrops to obtain a first feature map; N encoding modules, specifically used for: receiving a feature map, performing size reduction, weighted calculation, raindrop feature extraction, and normalization processing on the feature map to obtain a second feature map; Performing weighted calculation, raindrop feature extraction and normalization processing on the feature map to obtain a third feature map; Input the third feature map into the decoder of the raindrop removal model; Reducing the size of the third feature map to obtain a fourth feature map; Performing residual processing on the second feature map and the fourth feature map to obtain a fifth feature map; The feature map received by the first encoding module in the encoder is the first feature map, and the feature maps received by the second to N encoding modules are the fifth feature maps obtained by the first to N-1 encoding modules respectively; Deep Self-Attention Transformer Network layer, specifically used for: receiving a fifth feature map obtained by the Nth encoding module, performing weighted calculation and raindrop feature extraction on the fifth feature map obtained by the Nth encoding module to obtain a sixth feature map; Normalizing the sixth feature map to obtain a seventh feature map; The seventh feature map is input into the decoder of the raindrop removal model.

8. An image raindrop removal device, characterized in that: The image raindrop removal device includes a processor, a memory, and an image raindrop removal program stored in the memory and executable by the processor, wherein when the image raindrop removal program is executed by the processor, the steps of the image raindrop removal method according to any one of claims 1 to 5 are implemented.

9. A readable storage medium, characterized in that: The readable storage medium stores an image raindrop removal program, wherein when the image raindrop removal program is executed by a processor, the steps of the image raindrop removal method according to any one of claims 1 to 5 are implemented.

Images