Single-image raindrop removal method based on physical model, computer readable storage medium and program product

Abstract

The invention belongs to the technical field of image data processing, and particularly relates to a single-image raindrop removal method based on a physical model, a computer readable storage medium and a program product. The method comprises the following steps: firstly, deducing a corresponding dedegradation model according to a physical model of a degraded image caused by raindrops; secondly, setting a parameter estimation method based on a convolutional neural network (CNN) to estimate parameters required by the image restoration model; then, substituting the parameters into a degeneration model to obtain a preliminary raindrop-removed image; and finally, training the preliminary raindrop removal image by introducing a real clean-raindrop image pair to optimize the result, thereby solving the problem that the raindrops on the camera lens can greatly reduce the performance of the visual system.

Description

Technical Field

[0001] The present invention belongs to the technical field of image data processing, and more specifically, relates to a single image raindrop removal method based on a physical model, a computer-readable storage medium and a program product. Background Art

[0002] Raindrops attached to camera lenses can cause severe degradation of visual images. Unlike non-raindrop areas, raindrop areas are formed by reflected light from the wider environment. Due to the blurriness of raindrop areas, much of the image data is obscured, which is not conducive to visual systems that use images as raw data. For example, tasks based on visual sensors such as object detection, feature matching, size measurement, positioning and mapping cannot be performed reliably due to the presence of attached raindrops. Therefore, it is crucial to study algorithms to remove attached raindrops from images.

[0003] Early studies obtained the statistical characteristics of attached raindrops through traditional methods and used them as a basis for raindrop detection and removal. Kshitiz Garg et al. first comprehensively analyzed the visual impact of rain on imaging systems and developed a correlation model that captures the dynamics of rain and a physics-based motion blur model. Based on these models, they proposed an effective algorithm for detecting and removing rain from videos. Based on the local spatiotemporal derivatives of raindrops, Shaodi You et al. proposed a method for automatically detecting and removing attached raindrops in videos. For local image degradation caused by raindrops, they used temporal intensity derivatives to solve the mixing function of the detected partially occluded areas and used video completion technology to restore the fully occluded areas. David Eigen et al. designed a neural network with 2 hidden layers to learn the mapping between degraded image patches and clean images, implicitly capturing the characteristic appearance of dirt and water droplets in natural images.

[0004] In recent years, more and more research has been devoted to removing raindrops from images through deep learning methods. RuiQian et al. used adversarial training to apply attention to the generative network, injecting visual attention into the generative network and the discriminative network. Through this operation, the generative network will pay more attention to the raindrop area and surrounding structures, while the discriminative network will be able to evaluate the local consistency of the restored area. Yuhui Quan et al. developed a dual attention mechanism based on convolutional neural network (CNN) to remove raindrops on images. The mechanism simultaneously guides CNN to use shape-driven attention and channel recalibration. RuijieQuan et al. used neural structure search (NAS) to build a degraded image restoration network. The overall approach is to remove raindrops and rain streaks in a complementary manner, that is, remove raindrops before removing rain streaks, and vice versa.

[0005] Although these studies have removed raindrops to a certain extent, the problem of removing raindrops attached to images is still a challenge. The current image raindrop removal results cannot restore the blurred areas of the image well, and still leave degraded residual areas. They cannot integrate physical models and deep learning for deep optimization, cannot adapt to raindrop areas of different scales, and have poor generalization capabilities in real scenes and synthetic scenes.

[0006] Chinese patent document CN112767280B discloses a single image raindrop removal method based on a cyclic iteration mechanism, including: preprocessing the original attached raindrop degraded image and the training image pair of the clean image to obtain an image block data set composed of the original attached raindrop degraded image and the training image pair of the clean image; using the motivation of continuous iterative rain removal, designing a convolutional neural network for single image raindrop removal; designing a target loss function loss for optimizing the network, using the image block data set as training data, and using the back propagation method to calculate the gradient of each parameter in the designed image raindrop removal convolutional neural network according to the designed target loss function loss, and using the stochastic gradient descent method to update the parameters, and finally learning the optimal parameters of the model; inputting the image to be tested, and using the trained model to predict and generate a clean image after raindrop removal.

[0007] Chinese patent document CN113658074A discloses a single image raindrop removal method based on a LAB color space multi-scale fusion network, comprising the following steps: step A: generating a training image pair according to the degraded state and the clean state of the original attached raindrops in the same scene, performing color space transformation on the training image pair, and obtaining a training image pair corresponding to the LAB color space; step B: preprocessing the training image pair to obtain an image block data set in the LAB color space; step C: based on the characteristics of the image pair in the LAB color space, using a multi-scale learning strategy to design a convolutional neural network for single image raindrop removal; step D: designing a target loss function loss, using the image block data set as training data, using the back propagation method to calculate the gradient of each parameter in the convolutional neural network, and using the stochastic gradient descent method to update the parameters; step E: using the image raindrop removal network to generate an image after raindrop removal, and restoring it to the RGB color space.

[0008] The above patent ignores the intrinsic mechanism of image degradation caused by attached water droplets, and does not introduce the physical model of image degradation caused by attached water droplets into the image restoration process. Instead, it treats the image with attached water droplets as a conventional image restoration task, and cannot deeply explore the intrinsic characteristics of image degradation caused by attached water droplets. Therefore, it has limitations in both methodology and test results.

[0009] In view of this, the present invention proposes a single image raindrop removal method based on physical model. Summary of the invention

[0010] The present invention aims to overcome the defects of the above-mentioned prior art and provide a single image raindrop removal method based on a physical model;

[0011] The present invention also discloses a computer-readable storage medium for implementing the above method;

[0012] The invention also discloses a program product for realizing the method.

[0013] The detailed technical scheme of the present invention is as follows:

[0014] A single image raindrop removal method based on a physical model, the method comprising the following steps:

[0015] S1. Physical model of degraded images caused by raindrops, and derivation of corresponding image restoration model;

[0016] S2. Design a parameter estimation network based on a convolutional neural network (CNN) to estimate the parameters of the raindrop mask M and raindrop layer B required by the image restoration model;

[0017] S3, substituting the parameters into the image restoration model to obtain a preliminary raindrop removal image;

[0018] S4, introduce synthetic clean-degraded image pairs to perform end-to-end fine-tuning on the image restoration model;

[0019] S5. Input the restored image output by the fine-tuned image restoration model into the optimization module, and output the final raindrop removal image.

[0020] Preferably, according to the present invention, the detailed derivation process of step S1 is as follows:

[0021] The physical model of the degraded image caused by raindrops is formulated as follows:

[0022] I=J(1-M)+BM (1)

[0023] In formula (1), I represents the degraded image caused by raindrops; J represents the clean image without raindrops; M is the raindrop mask, when the value is 1, it represents the raindrop area, otherwise it is the clean area; B is the raindrop layer;

[0024] According to the physical model of the degraded image, the image restoration model is derived as follows:

[0025]

[0026] In formula (2), J represents a clean image without raindrops; I represents a degraded image caused by raindrops; M is a raindrop mask; and B is a raindrop layer.

[0027] Preferably, according to the present invention, the design of a parameter estimation network based on a convolutional neural network CNN is divided into the following steps:

[0028] S21. Data preprocessing:

[0029] First, the raindrop mask M and the raindrop layer B are analyzed in the frequency domain. The raindrop mask M represents the position, size, and shape information of the raindrops, which is equivalent to the noise part of the image and has high-frequency characteristics. The raindrop layer B is a complex fuzzy result obtained by mixing the clean image J with the background information and the surrounding environment, which has both high-frequency and low-frequency characteristics.

[0030] Then, in order to enhance the data source for parameter estimation, the features of the input image are preprocessed according to the frequency domain analysis results: the original RGB data of the input image is divided into high-frequency and low-frequency parts by using a frequency domain filtering method based on guided filtering;

[0031] S22. Set the raindrop mask M estimation network:

[0032] A mask estimation network with three parallel channels is set up, and the three parallel channels correspond to estimating raindrop masks of large, medium and small scales, namely, M L , M M , M S , input the high-frequency part after guided filtering, and finally synthesize it into a complete raindrop mask M;

[0033] In the raindrop mask M estimation network, each channel includes a convolution layer, a densely connected residual block and a deconvolution layer. Different convolution kernel sizes are used when estimating raindrop masks of different scales. The larger the scale of the raindrop mask, the larger the corresponding convolution kernel size. It is preferred to set the corresponding M L , M M , M S The convolution kernel sizes are 7, 5, and 3 respectively;

[0034] S23. Set the raindrop layer B estimation network:

[0035] At the front end of the raindrop layer B estimation network, the high-frequency part, the low-frequency part and the original RGB data are connected in series in the channel direction to form a fusion feature. The convolution layer and the pooling layer of the raindrop layer B estimation network form an autoencoder to learn and infer the raindrop layer B from the fusion feature. In order to comprehensively consider the role of the local image domain and the surrounding environment, a pyramid pooling module is set at the back end of the raindrop layer B estimation network to fuse the local features and the global features, which can help obtain a more accurate B.

[0036] S24, combining the raindrop mask M estimation network and the raindrop layer B estimation network to obtain a complete parameter estimation network of the raindrop degradation model.

[0037] Preferably, according to the present invention, the detailed process of step S4 is as follows:

[0038] First, based on the clear clean image, the corresponding degraded image is generated by adding simulated atmospheric scattering effects, thus constructing a clean-degraded image pair; then, the clean-degraded image pair is used to perform supervised learning on the image restoration model, so that the image restoration model can learn the effective mapping relationship from raindrop input to clear raindrop-free output, thereby improving the performance of the model in the actual raindrop removal task. After fine-tuning on the synthetic degraded image, the model reaches the best state on the synthetic raindrop image;

[0039] Preferably, according to the present invention, the optimization module is an encoder-decoder based on CNN, the encoder and the decoder both include a 4-layer network, the restored image output after fine-tuning of the network is input, and the final raindrop removal image is output.

[0040] A computer-readable storage medium stores computer instructions, which can implement the steps of the above method when executed by a processor.

[0041] A computer program product loaded with the above method comprises computer instructions, which can implement the steps of the above method when executed by a processor.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] (1) The present invention fully explores the intrinsic mechanism of attached water droplet image degradation, utilizes the attached water droplet image degradation physical model to realize image degradation restoration, can integrate physical model and deep learning for deep optimization, can restore the blurred area of ​​the image in a targeted manner, and will not leave degraded residual areas. A parameter estimation method based on convolutional neural network is proposed to obtain the key parameters of the raindrop degradation model.

[0044] (2) The present invention uses synthetic images for fine-tuning and introduces real clean-degraded image pairs to optimize the raindrop removal algorithm based on the physical model, thereby achieving the refinement of degraded image restoration, having the ability to preserve details, being able to adapt to raindrop areas of different scales, and having good generalization ability in real scenes and synthetic scenes, and achieving the best restoration effect in the restoration of attached water droplets. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of a single image raindrop removal method based on a physical model according to the present invention;

[0046] Figure 2 is a diagram of a data preprocessing process in an embodiment of the present invention;

[0047] Figure 3is a raindrop mask M estimation network diagram in an embodiment of the present invention;

[0048] Figure 4 is a raindrop layer B estimation network diagram in an embodiment of the present invention;

[0049] Figure 5 is a complete parameter estimation network diagram in an embodiment of the present invention;

[0050] Figure 6 is a diagram of a preliminary restored image process synthesized according to a physical model in an embodiment of the present invention;

[0051] Figure 7 is a structural diagram of an optimization module in an embodiment of the present invention;

[0052] Figure 8 is a flow chart of synthetic data production in an embodiment of the present invention;

[0053] Fig. 9 is a performance comparison diagram of the raindrop removal algorithm in the embodiment of the present invention. DETAILED DESCRIPTION

[0054] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0055] Embodiment 1,

[0056] like Figure 1 As shown, a single image raindrop removal method based on a physical model comprises the following steps:

[0057] S1. Physical model of image degradation caused by raindrops, and derivation of corresponding de-degradation model;

[0058] S2. Design a parameter estimation network based on a convolutional neural network (CNN) to estimate the parameters of the raindrop mask M and raindrop layer B required by the image restoration model;

[0059] S3, substituting the parameters into the image restoration model to obtain a preliminary raindrop removal image;

[0060] S4, introduce synthetic clean-degraded image pairs to perform end-to-end fine-tuning on the image restoration model;

[0061] S5. Input the restored image output by the fine-tuned image restoration model into the optimization module, and output the final raindrop removal image.

[0062] The detailed derivation process of step S1 is as follows:

[0063] The physical model formula of image degradation caused by raindrops is as follows:

[0064] I=J(1-M)+BM (1)

[0065] In formula (1), I represents the image degraded by raindrops; J represents the clean image without raindrops; M is the mask, which represents the raindrop area when the value is 1, otherwise it is the clean area; B is the raindrop layer, which represents the complex mixture of the original background and the light reflected by the environment. It can be seen that the degraded image caused by raindrops is the result of the interaction between the clean image and the raindrop layer.

[0066] The degradation model provides us with the following useful information. First, synthetic degraded data can be created based on this model, thus avoiding the complex operation of establishing a real raindrop image dataset. Second, the prior parameters set according to the model can provide process guidance for raindrop removal training and provide conditions for subsequent restoration algorithms based on physical models. Finally, this model can help derive the mathematical formula for degraded image restoration, that is, establish an image restoration model, thus laying a theoretical foundation for raindrop removal based on physical models.

[0067] According to the degradation physical model, we can derive the image restoration model as follows:

[0068]

[0069] In formula (2), J represents a clean image without raindrops; I represents a degraded image caused by raindrops; M is a raindrop mask; B is a raindrop layer. The model shows that the clean image J can be synthesized by estimating parameters M and B.

[0070] In step S2, designing a network to estimate the parameters required for the image restoration model is divided into the following steps:

[0071] From formula (2), we can know that to derive the clean image J from the raindrop image I based on the physical model, there are two necessary parameters to be estimated, namely the raindrop mask M and the raindrop layer B. Therefore, this section proposes a parameter estimation method based on convolutional neural network (CNN).

[0072] First, we analyze M and B in the frequency domain. The raindrop mask M represents the position, size, shape and other information of the raindrops. M is equivalent to the noise part of the image, and considering that many studies have obtained the raindrop position by edge detection, M has high-frequency characteristics. The raindrop layer B is a complex blurred result obtained by mixing the clean image J with the background information and the surrounding environment, so it has both high-frequency and low-frequency characteristics.

[0073] In order to enhance the data source for parameter estimation, we preprocess the features of the input image according to the results of frequency domain analysis. The preprocessing process is as follows: Figure 2As shown. We use a frequency domain filtering method based on guided filtering to separate the original data into high frequency and low frequency parts. The filter inputs an RGB image of H*W*3 and outputs a high frequency and low frequency part of H*W*15. The filtered data will be conducive to targeted estimation based on the frequency domain properties of the parameters.

[0074] Because raindrops have different sizes, the raindrop mask M also corresponds to areas of different sizes. In order to accurately detect raindrops of different sizes and shapes, it is necessary to design a mask inference network that can infer raindrop masks of different sizes. To this end, we designed a mask estimation network with three parallel channels. The three parallel channels correspond to estimating raindrop masks of large, medium, and small scales (i.e., ML, MM, MS), input the high-frequency part after guided filtering, and finally synthesize it into a complete raindrop mask M.

[0075] The estimation network of raindrop mask M is as follows Figure 3 As shown in Figure 2, Conv represents convolution; DeConv represents deconvolution; k represents the convolution kernel size; and s represents the stride. We have labeled the network parameters of the main convolution operation of each residual block. For clarity, we use different colors to represent dense connections.

[0076] In the M estimation network, each channel consists of a convolutional layer, a densely connected residual block, and a deconvolutional layer. Different convolution kernel sizes are used to estimate raindrop masks of different scales. The larger the scale of the raindrop mask, the larger the corresponding convolution kernel size. Therefore, the corresponding ML, MM, and MS convolution kernel sizes are 7, 5, and 3, respectively.

[0077] Since the formation of B is a complex result of the interaction between the local image and the surrounding environment, it is more difficult to estimate B from the degraded image. The present invention designs a B estimation network, such as Figure 4 As shown in the figure, the blue arrows indicate long jump connections. In the early stage of the network, the high-frequency part, the low-frequency part, and the original RGB data are connected in series in the channel direction to form a fusion feature. The main component of the network is the autoencoder composed of convolutional layers and pooling layers, which undertakes the main task of learning and inferring B from the fusion features. In order to comprehensively consider the role of the local image domain and the surrounding environment, we added a pyramid pooling module to the back end of the network. The pyramid pooling module has the function of fusing local features and global features, which can help obtain a more accurate B.

[0078] Combining the M estimation network and the B estimation network shown above, we can get the complete physical parameter estimation network of the raindrop degradation model. Figure 5 shown.

[0079] It is worth noting that the true values ​​of B and M used in the above network training process are all artificially synthesized. The synthesis process is shown in detail in the experimental section.

[0080] The step S4 is specifically as follows:

[0081] After estimating the parameters of the raindrop degradation model, we obtain all the necessary conditions for synthesizing a clean raindrop removal image, as shown in Equation (2). Therefore, we can obtain preliminary restoration results through the raindrop removal degradation model. The process is as follows: Figure 6 As shown in the figure, after the data preprocessing process, the image feature map is decomposed into high-frequency and low-frequency parts; according to the frequency domain characteristics of M and B, the M estimation network and the B estimation network are set up respectively to obtain the parameters M and B; according to the physical model of the attached water droplet image degradation analyzed in the previous article, the inverse operation is performed to obtain the physical model of degraded image restoration, that is, the restored image J is obtained based on the parameters B, M and image I.

[0082] However, at this time, the network only has synthetic degraded images, mask truth and raindrop layer truth, but no corresponding clean images. Therefore, the constraints are not enough to obtain the target raindrop removal image. Therefore, we introduced synthetic clean-degraded image pairs to fine-tune the network end-to-end. After fine-tuning on synthetic degraded images, the model reached the best state on synthetic raindrop images. Specifically, first, according to the clear clean images, the corresponding degraded images were generated by adding simulated atmospheric scattering effects, thereby constructing clean-degraded image pairs; then, the clean-degraded image pairs were used to perform supervised learning on the image restoration model, so that the image restoration model can learn the effective mapping relationship from raindrop input to clear raindrop-free output, thereby improving the performance of the model in the actual raindrop removal task.

[0083] Finally, considering the domain difference between synthetic raindrop images and real raindrop images, this paper designs an optimization module to introduce real clean-degraded data pairs into the raindrop removal process. The optimization module is essentially a CNN-based encoder-decoder. Figure 7 As shown in the figure, the encoder and decoder are both composed of a 4-layer network. Its input is the restored image output after fine-tuning the network, and its output is the final raindrop removal image. Through the optimization module, the raindrop removal image is optimized using a real clean-degraded image pair, achieving domain adaptation in actual scenes.

[0084] Embodiment 2,

[0085] experiment

[0086] Comprehensive experiments were conducted to validate the effectiveness of the proposed raindrop removal algorithm. Our algorithm has several components and the training process is not a one-time process, so we describe the training process in this paper. We use synthetic data to guide the estimation of parameters in the raindrop degradation physical model, which makes it necessary to show the details of the synthetic data. The most critical experimental part is the test of the raindrop removal effect of the complete algorithm, which we compare with other state-of-the-art algorithms in a qualitative and quantitative way. In addition, we also conduct ablation studies to measure the contribution of the proposed components.

[0087] 1 Training details

[0088] The complete training process is divided into three stages. The first stage is the training of the physical parameter estimation network, which aims to independently obtain the raindrop mask M and the raindrop layer B. The second stage is the fine-tuning of the physical parameters after the introduction of synthetic images. The third stage is the optimization of the initial restoration results after the introduction of real clean-degraded image pairs, which makes up for the domain difference between synthetic data and real data.

[0089] In the training and testing process of this paper, both synthetic and real-world datasets are used. In the raindrop model physical parameter estimation network, the true value is necessary because the raindrop mask M and raindrop layer B need to be supervised. This is mainly because the images in the real world cannot clearly give the raindrop mask and raindrop layer of the degraded image. In order to better refine the restoration results, synthetic degraded images and clean-degraded image pairs of real scenes have to be introduced into the network training process.

[0090] The production process of synthetic data is as follows Figure 8 As shown in Figure 1. First, the clean image J from KITTI is blurred to obtain the raindrop layer B. Then the raindrop position, size and shape in the raindrop mask M are randomly set. Finally, the synthesized degraded image is obtained through the physical model of image degradation caused by raindrops shown in formula (1). The clean-degraded image pairs of real scenes used are from the public dataset of Rui Qian et al.

[0091] 2 Raindrop Removal Test

[0092] In this section, comprehensive experiments are conducted on the proposed raindrop removal algorithm. Not only the effectiveness of the overall algorithm is verified, but also the contribution of each component is evaluated through ablation experiments. Since the usage scenario of the algorithm in this paper is a real raindrop degradation environment, the algorithm is only tested in real scenarios.

[0093] exist Fig. 9In the figure, the effectiveness of the algorithms is compared visually. It can be seen that the proposed method has the best recovery effect compared with the method of RuiQian and Yuhui Quan. The proposed method not only removes most of the raindrops on the image, but also retains the details.

[0094] PSNR and SSIM are used as evaluation indicators to quantitatively measure the performance of the algorithm, as shown in Table 1. The smaller the PSNR and SSIM, the closer the restored result is to the Ground Truth, that is, the better the algorithm performance. It can be seen that the raindrop removal algorithm based on the physical model in this paper has the smallest PSNR and SSIM, that is, it has the best performance.

[0095] Table 1 PSNR and SSIM of different algorithms

[0096] method PSNR SSIM Rui Qian's[4] 30.55 0.9023 Yuhui Quan's[5] 30.86 0.9263 Ours 31.22 0.9461

[0097] In addition, we conduct ablation experiments to verify the effectiveness of the designed components. We use the symbol S to represent the synthetic raindrop removal image based only on the independent parameter estimation results, the symbol F to represent the introduction of synthetic degraded images for fine-tuning, and the symbol O to represent the introduction of real clean-degraded image pairs for optimization. There are three experimental settings for the ablation study, as follows: S, S+F, and S+F+O. Similarly, we use PSNR and SSIM to measure the performance of different methods, as shown in Table 2.

[0098] Table 2 Ablation experiment results

[0099] method PSNR SSIM S 39.97 0.8975 S+F 30.59 0.9287 S+F+O 31.22 0.9461

[0100] As can be seen from Table 2, the effect of S+F+O is the best, followed by S+F, and the raindrop removal effect of S is the roughest. This reflects that with the addition of fine-tuning and optimization steps, the raindrop removal ability of the algorithm is improved. In particular, the introduction of the optimization module solves the domain difference between synthetic data and real data.

Claims

1. A single image raindrop removal method based on a physical model, characterized in that: The method comprises the following steps: S1. Physical model of degraded images caused by raindrops, and derivation of corresponding image restoration model; S2. Design a parameter estimation network based on a convolutional neural network (CNN) to estimate the parameters of the raindrop mask M and raindrop layer B required by the image restoration model; S3, substituting the parameters into the image restoration model to obtain a preliminary raindrop removal image; S4, introduce synthetic clean-degraded image pairs to perform end-to-end fine-tuning on the image restoration model; S5. Input the restored image output by the fine-tuned image restoration model into the optimization module, and output the final raindrop removal image.

2. The single image raindrop removal method based on physical model according to claim 1, characterized in that: The detailed derivation process of step S1 is as follows: The physical model of the degraded image caused by raindrops is formulated as follows: I=J(1-M)+BM (1) In formula (1), I represents the degraded image caused by raindrops; J represents the clean image without raindrops; M is the raindrop mask, when the value is 1, it represents the raindrop area, otherwise it is the clean area; B is the raindrop layer; According to the physical model of the degraded image, the image restoration model is derived as follows: In formula (2), J represents a clean image without raindrops; I represents a degraded image caused by raindrops; M is a raindrop mask; and B is a raindrop layer.

3. The single image raindrop removal method based on physical model according to claim 1, characterized in that: The design of the parameter estimation network based on the convolutional neural network CNN is divided into the following steps: S21. Data preprocessing: First, the raindrop mask M and the raindrop layer B are analyzed in the frequency domain. The raindrop mask M represents the position, size, and shape information of the raindrops, which is equivalent to the noise part of the image and has high-frequency characteristics. The raindrop layer B is a complex fuzzy result obtained by mixing the clean image J with the background information and the surrounding environment, which has both high-frequency and low-frequency characteristics. Then, the features of the input image are preprocessed according to the frequency domain analysis results: the original RGB data of the input image is divided into high-frequency part and low-frequency part by using the frequency domain filtering method based on guided filtering; S22. Set the raindrop mask M estimation network: A mask estimation network with three parallel channels is set up, and the three parallel channels correspond to estimating raindrop masks of large, medium and small scales, namely, M L , M M , M S , input the high-frequency part after guided filtering, and finally synthesize it into a complete raindrop mask M; In the raindrop mask M estimation network, each channel includes a convolution layer, a densely connected residual block and a deconvolution layer. Different convolution kernel sizes are used when estimating raindrop masks of different scales. The larger the scale of the raindrop mask, the larger the corresponding convolution kernel size. It is preferred to set the corresponding M L , M M , M S The convolution kernel sizes are 7, 5, and 3 respectively; S23. Set the raindrop layer B estimation network: At the front end of the raindrop layer B estimation network, the high-frequency part, the low-frequency part and the original RGB data are connected in series in the channel direction to form a fusion feature. The convolution layer and the pooling layer of the raindrop layer B estimation network form an autoencoder to learn and infer the raindrop layer B from the fusion feature; at the back end of the raindrop layer B estimation network, a pyramid pooling module is set to fuse the local features and the global features. S24, combining the raindrop mask M estimation network and the raindrop layer B estimation network to obtain a complete parameter estimation network of the raindrop degradation model.

4. The single image raindrop removal method based on physical model according to claim 1, characterized in that: The detailed process of step S4 is as follows: Firstly, based on the clear clean image, the corresponding degraded image is generated by adding simulated atmospheric scattering effect, thereby constructing a clean-degraded image pair; then, the clean-degraded image pair is used to perform supervised learning on the image restoration model, so that the image restoration model can learn the effective mapping relationship from raindrop input to clear raindrop-free output, thereby improving the performance of the model in the actual raindrop removal task.

5. The single image raindrop removal method based on physical model according to claim 1, characterized in that: The optimization module is an encoder-decoder based on CNN, and both the encoder and the decoder include a 4-layer network, input the restored image output after fine-tuning the network, and output the final raindrop removal image.

6. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, they are used to implement the steps of the method as described in any one of claims 1 to 5.

7. A computer program product comprising computer instructions, characterized in that When the computer instruction is executed by a processor, it is used to implement the steps of the method according to any one of claims 1 to 5.

Images