Dual-stage image rain removal method based on rain physical model and deep learning

Abstract

The invention relates to a two-stage image rain removal method based on a rain physical model and deep learning, and relates to the technical field of image rain removal. The method comprises the following steps: acquiring an original rain image; predicting according to the original rain image to obtain a corresponding atmospheric light image A, a transmissivity image T and a rain fringe image S; generating a plurality of rain-removed images Ji according to a rain physical model; performing weighted summation on the plurality of rain-removed images Ji by using a grouping weighting mechanism to obtain a preliminary rain-removed image J; and a PromptFormer is used to optimize the quality of the preliminary rain removal image J. According to the method, in the first stage, the rain removal reasoning logic is optimized in combination with a physical model of rain, in the second stage, a four-stage U-shaped network is introduced to optimize the output of the first stage, and in addition, the robustness of the whole algorithm is further enhanced in combination with a grouping weighting mechanism.

Description

Technical Field

[0001] The present invention relates to the technical field of image deraining, and in particular to a two-stage image deraining method based on a rain physics model and deep learning. Background Art

[0002] As a common natural phenomenon, rain always poses challenges to some outdoor vision algorithms that assume clear weather. Environmental perception in autonomous driving systems, object detection in monitoring systems, and drone aerial image processing are all affected by problems such as false detection and missed detection due to the presence of rain. Rain significantly affects the stability and reliability of outdoor vision systems. Therefore, rain removal algorithms have received increasing attention in academia and industry in recent years.

[0003] In the real world, rain streaks may be intertwined with the superposition effect of rain traces, and the visual information of some background scenes may be attenuated by rain, especially in the distance. Traditional methods usually rely too much on hand-crafted prior knowledge or physical models that assume specific characteristics of rain streaks. This reliance limits their adaptability in complex real-world environments, where the shape, size, and distribution of rain streaks vary greatly. Summary of the invention

[0004] Based on this, it is necessary to provide a two-stage image deraining method based on rain physics model and deep learning to address the above problems.

[0005] The present invention provides a two-stage image deraining method based on a rain physics model and deep learning, characterized in that the method comprises:

[0006] Collect original rainy images;

[0007] The corresponding atmospheric light map A, transmittance map T and rain streak map S are predicted based on the original rainy image;

[0008] Generate multiple rain-free images Ji according to the rain physics model;

[0009] Apply the group weighted mechanism to perform weighted summation on multiple derained images Ji to obtain a preliminary derained image J;

[0010] PromptFormer is applied to optimize the quality of the preliminary derained image J.

[0011] In one embodiment, the step of predicting the corresponding atmospheric light map A, transmittance map T and rain streak map S according to the original rainy image includes:

[0012] Apply AtmosNet to predict the atmospheric light map A from the input image;

[0013] Apply TransmissionNet to predict the transmittance map T from the input image;

[0014] StreaksNet is applied to predict the rain streak map S from the input image.

[0015] In one embodiment, the calculation formula of the rain physics model is: Where I represents the observed rainy image, T is the transmittance map, J is the clear rain-free image, and S i is the rain line map, and A is the global atmospheric light map.

[0016] In one embodiment, the step of generating a plurality of rain-removed images Ji according to a rain physics model includes:

[0017] The variant formula is obtained based on the rain physics model Where I represents the observed rainy image, T is the transmittance map, J is the clear rain-free image, and S i is the rain line map, and A is the global atmospheric light map;

[0018] Multiple derained images Ji are obtained by calculating the variant formula.

[0019] In one embodiment, the step of applying a group weighting mechanism to weighted sum these images to obtain a preliminary derained image J includes:

[0020] Apply WeightNet to adaptively assign weights to each set of parameters;

[0021] According to the weights assigned to each group of parameters, all derained images Ji are weighted summed to obtain the preliminary derained image J.

[0022] In one embodiment, all the derained images Ji are weighted and summed according to the weights assigned to each group of parameters, and the calculation formula for obtaining the preliminary derained image J is: Among them, n is the number of groups of derained images, i is the group number of the derained image, and w is the weight corresponding to the derained image.

[0023] In one embodiment, the PromptFormer is a four-stage U-shaped model, the backbone networks of the first two stages are composed of CNN (Deformable Convolutional Neural Network) blocks, and the backbone networks of the last two stages are composed of Transformer blocks.

[0024] In one embodiment, the process formula for optimizing the quality of the preliminary derained image J by using PromptFormer is: final = PromptFormer(J+X), where J finalis the predicted final clear image, J is the preliminary de-rained image, and X is the original rainy image.

[0025] The present invention also provides a two-stage image deraining system based on a rain physics model and deep learning, the system comprising:

[0026] A collection module, used to collect original rainy images;

[0027] A prediction module is used to predict the corresponding atmospheric light map A, transmittance map T and rain streak map S according to the original rainy image;

[0028] A calculation module is used to generate multiple rain-free images Ji according to the rain physics model;

[0029] The fusion module is used to apply the group weighted mechanism to perform weighted summation on multiple derained images Ji to obtain a preliminary derained image J;

[0030] The optimization module is used to apply PromptFormer to optimize the quality of the preliminary derained image J.

[0031] The above two-stage image deraining method based on rain physics model and deep learning, in the first stage, combines the physical model of rain to optimize the deraining reasoning logic, and in the second stage, introduces a four-level U-shaped network to optimize the output of the first stage. In addition, combined with the group weighting mechanism, the robustness of the entire algorithm is further enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1 A flowchart of a two-stage image deraining method based on a rain physics model and deep learning proposed by the present invention;

[0034] Figure 2 This is the architecture diagram of RainPhysicTransNet in the two-stage image deraining method based on rain physics model and deep learning proposed in the present invention;

[0035] Figure 3 for Figure 2 The architecture diagram of PromptFormer in ;

[0036] Figure 4A schematic diagram of the basic modules used by the first-stage sub-network of RainPhysicTransNet in the two-stage image deraining method based on rain physics model and deep learning proposed in the present invention;

[0037] Figure 5 This is a quantitative comparison result diagram of the two-stage image deraining method based on rain physics model and deep learning proposed in this invention and the previous methods on four deraining datasets (Rain100L, Rain100H, Test1200, Test100);

[0038] Figure 6 Schematic diagram of the performance of RainPhysicTransNet and previous methods on the Test1200 dataset;

[0039] Figure 7 This is a qualitative comparison result of the two-stage image deraining method based on rain physics model and deep learning proposed in the present invention and other methods on a synthetic dataset;

[0040] Figure 8 This is a qualitative comparison chart of the two-stage image deraining method based on rain physics model and deep learning proposed in this invention and six other deraining algorithms on real data;

[0041] Fig. 9 This is a table of ablation experiment data of each module in the two-stage image deraining method based on rain physics model and deep learning proposed in the present invention;

[0042] Fig.10 This is a schematic diagram of the structure of a two-stage image deraining system based on rain physics model and deep learning proposed in the present invention;

[0043] Fig.11 The figure is a diagram showing the internal structure of a computer device according to an embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] AbstractSince deep learning revived image restoration technology, research on rain removal algorithms that combine physical principles or prior knowledge has become increasingly rare. Relying solely on training neural networks (even with large-scale datasets), the generalization ability of algorithms in unfamiliar scenes is still limited. Such methods usually make inferences based only on learned data, but lack an understanding of the physical phenomenon of "rain". Incorporating rain-related physical principles (such as raindrop shape, falling trajectory, and light scattering) into the algorithm can enhance the model's understanding ability and thus improve robustness under different rainfall intensities, angles, and types.

[0046] There are some studies that try to incorporate rain-related constraints and prior knowledge into deep learning. There is currently a two-stage deraining algorithm. In the first stage, the algorithm performs frequency decomposition on the original input image, and then inputs the decomposed image into three branch sub-networks to predict the physical parameters in the rain physics model formula, and then substitutes these parameters into the variant formula of the rain physics model formula to derive a clear image J. In the second stage, the author uses a conditional generative adversarial network (CGAN) as an optimization model, which takes the output of the first stage as input to further improve the quality of the derained image.

[0047] Inspired by this study, we can summarize the following two points: First, the introduction of the physical model allows the deraining algorithm to not only rely on data-driven methods, but also utilize physical principles. Second, the physical model uses the physical characteristics related to rain as prior knowledge to constrain the model's predictions during training, prevent the neural network from generating unreasonable solutions, accelerate its convergence, improve deraining performance, and enhance the network's robustness and generalization capabilities.

[0048] However, in the real world, rain streaks may be intertwined with rain trace superposition effects, and the visual information of some background scenes may be attenuated by rain, especially in the distance. Therefore, it is difficult for physical models to fully represent these complex characteristics. Instead of designing a more complex physical model, it is simpler and more efficient to introduce an additional optimization network in the second stage to further optimize the quality of the output image in the first stage.

[0049] In view of the above problems, the following Figure 1-Figure 11 The present invention describes a two-stage image deraining method based on rain physics model and deep learning.

[0050] like Figure 1 As shown, in one embodiment, a two-stage image deraining method based on a rain physics model and deep learning includes:

[0051] Step S100: collecting original rainy images.

[0052] Step S200, predicting the corresponding atmospheric light map A, transmittance map T and rain streak map S according to the original rainy image.

[0053] Step S300, generating multiple rain-removed images Ji according to the rain physics model.

[0054] Step S400 , applying a group weighted mechanism to perform weighted summation on multiple derained images Ji to obtain a preliminary derained image J.

[0055] Specifically, the group weighting mechanism extracts multiple groups of physical parameters by utilizing multiple CNN (convolutional neural network) sub-networks to generate multiple derained images and enhance the image deraining effect.

[0056] Step S500: Apply PromptFormer to optimize the quality of the preliminary derained image J.

[0057] Specifically, Figure 2 and Figure 3 As shown in the figure, according to the two-stage neural network RainPhysicTransNet (rain physics transmission network) we proposed, in the process of image deraining, in the first stage, the corresponding multiple sets of atmospheric light maps A, transmittance maps T and rain streak maps S are predicted based on the original rainy images obtained, and multiple derained images Ji are generated strictly following the rain physics model. Then, these images are weighted summed to obtain the preliminary derained image J. In the second stage, a U-shaped network composed of CNN and Transformer components is used to optimize the quality of the output image of the first stage.

[0058] It should be noted that if Figure 4 As shown in the figure, the basic modules used by the first-stage sub-network are composed of LayerNorm (layer normalization technology), DeformConv (deformable convolution), SimpleGate (gating mechanism) and MSConv (multi-scale convolution operation). Among them, DeformConv is used to enhance the feature representation ability of the network because it allows the convolution kernel to adaptively adjust its shape and position. MSConv, a simplified version of SFConv (selective feature convolution), is designed to enrich the contextual information of features because it combines a multi-scale feature enhancement mechanism and a channel attention mechanism. SimpleGate is used as an activation function to help the network selectively focus on more useful features.

[0059] The beneficial effect of this embodiment is that, based on the two-stage framework design, the group weighting mechanism is integrated into the design of the first stage, by generating multiple groups of physical parameters (each group corresponds to a derained image), and then weighting and summing all derived derained images to reduce the uncertainty of a single prediction result, thereby enhancing the stability of the final output. And the deformable convolutional neural network and multi-scale channel attention mechanism are integrated into the design of all branch sub-networks to improve their ability to extract multi-scale features. At the same time, the step of frequency decomposition of the original input image is deleted to allow the neural network to process information more freely and extract features.

[0060] In one embodiment, the step of predicting the corresponding atmospheric light map A, transmittance map T and rain streak map S according to the original rainy image includes:

[0061] Step S210, applying AtmosNet (atmospheric network) to predict the atmospheric light map A from the input image.

[0062] Step S220, applying TransmissionNet to predict the transmittance map T from the input image.

[0063] Step S230, applying StreaksNet (stripe network) to predict the rain streak map S from the input image.

[0064] Specifically, the process of predicting the atmospheric light map A, the transmittance map T and the rain streak map S from the input image can be expressed as the formula A=AtmosNet(X), T=TransmissionNet(X), S=StreaksNet(X), where X represents the input image.

[0065] It should be noted that AtmosNet, TransmissionNet, and StreaksNet are all small U-shaped networks composed of residual blocks.

[0066] In one embodiment, the calculation formula of the rain physics model is: Where I represents the observed rainy image, T is the transmittance map, J is the clear rain-free image, and S i is the rain line map, and A is the global atmospheric light map.

[0067] In this embodiment, the steps of generating a plurality of rain-removed images Ji according to the rain physics model include:

[0068] Step S310, obtaining a variant formula according to the rain physics model Where I represents the observed rainy image, T is the transmittance map, J is the clear rain-free image, and S i is the rain line map, and A is the global atmospheric light map.

[0069] Step S320: obtain multiple rain-removed images Ji by using a variant formula.

[0070] In one embodiment, the step of applying a group weighting mechanism to perform weighted summation on these images to obtain a preliminary derained image J includes:

[0071] Step S410, applying WeightNet to adaptively assign weights to each group of parameters.

[0072] Specifically, the process of WeightNet adaptively extracting the weight vector from the original input image can be expressed as follows: 1 ,w 2 ,w 3 )-Weightnet(X), where X represents the input image and w 1 、w 2 、w 3 Represent the weights assigned to the three groups of physical parameters. WeightNet adaptively assigns weights to each group of parameters so that the features of each group of derained images can be reasonably combined when performing weighted summation. This mechanism improves the robustness and visual quality of the final derained image and effectively copes with the complexity of different rain patterns.

[0073] Step S420 , performing weighted summation on all derained images Ji according to the weights assigned to each group of parameters, to obtain a preliminary derained image J.

[0074] Specifically, the sub-networks AtmosNet, TransmissionNet, StreaksNet, and WeightNet all contain basic modules consisting of LayerNorm, DeformConv, SimpleGate, and MSConv. In its composition, DeformConv is used to enhance the feature representation ability of the network because it allows the convolution kernel to adaptively adjust its shape and position. MSConv, a simplified version of SFConv, is designed to enrich the contextual information of features because it combines a multi-scale feature enhancement mechanism and a channel attention mechanism. SimpleGate, as an activation function, helps the network selectively focus on more useful features.

[0075] In this embodiment, all the derained images Ji are weighted and summed according to the weights assigned to each group of parameters, and the calculation formula for the preliminary derained image J is obtained as follows: Among them, n is the number of groups of derained images, i is the group number of the derained image, and w is the weight corresponding to the derained image.

[0076] In one embodiment, the PromptFormer is a four-stage U-shaped model, the backbone networks of the first two stages are composed of CNN (Deformable Convolutional Neural Network) blocks, and the backbone networks of the last two stages are composed of Transformer (converter) blocks.

[0077] In this embodiment, the process formula for optimizing the quality of the preliminary derained image J by using PromptFormer is: final = PromptFormer(J+X), where J final is the predicted final clear image, J is the preliminary de-rained image, and X is the original rainy image.

[0078] It should be noted that in this study, the only loss function used in the training process is the loss function based on the peak signal-to-noise ratio (PSNR). PSNR is a widely used metric in image quality assessment to measure the quality of the restored image. The higher the PSNR value, the better the image restoration quality, because it indicates that the deviation between the predicted image and the target image is smaller. The core idea of ​​this loss function is to first calculate the mean square error (MSE) between the predicted image and the target image, and then convert it into a PSNR value for optimization. The calculation formula of MSE in this process is: Among them, n is the number of pixels of the image, i is the pixel sequence number of the image, pred is the predicted clear image, and target is the true clear image.

[0079] And to evaluate the performance of our algorithm, we compare it with the baseline method and the best method on synthetic rain datasets and real rain datasets.

[0080] First, we will introduce the experimental settings, and then show the qualitative and quantitative comparison results on multiple datasets. In addition, we will verify the rationality of our method through ablation experiments. In the table, the best quality score of the evaluation method is marked in bold, and the second best quality score is underlined.

[0081] Next, we will introduce the details about the dataset, evaluation metrics, training process, and hyperparameter settings.

[0082] The training dataset we use is Rain13K, which is a synthetic rain dataset containing 13,712 pairs of clear images and rainy images. The synthetic datasets used for verification and testing include Rain100L, Rain100H, Test1200, and Test100. The real rain dataset we choose is LHP-rain for qualitative comparative experiments.

[0083] The metrics we use to evaluate network performance are PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index). The higher the PSNR and SSIM values, the better the performance of the algorithm.

[0084] We divide the training process of RainPhysicTransNet into two steps. In the first step, we independently train the first part of RainPhysicTransNet, including AtmosNet, TransmissionNet, StreaksNet and WeightNet. The training loss of the first step comes from the MSELoss (mean square error loss) between the derained image generated by the first part and the real image. In the second step, we load the model weights obtained in the first step and start training the entire model. The purpose of this design is to constrain the first part of RainPhysicTransNet in advance through the physical model of rain, so that it can first obtain sufficient prior knowledge related to the physical properties of rain, and prepare for the subsequent training of the entire model to impose more reasonable physical constraints.

[0085] In the first stage based on the physical model, the number of groups N is set to 3. In the second stage PromptFormer, the number of basic modules in each stage is set to [1,1,6,8], where the first two values ​​are the number of CNN blocks and the last two values ​​are the number of Transformer blocks. We use the Adam optimizer with a batch size of 8 and train on 8 NVIDIA3090 GPUs for a total of 400 rounds each time. The learning rate is initialized to 2e-4 and adjusted by the CosineAnnealingLR scheduler until it decays to 4e-8. The loss function used to train RainPhysicTransNet is the PSNR-based loss function PSNRLoss (peak signal-to-noise ratio loss).

[0086] We compared RainPhysicTransNet with eight other methods (including the best methods and baseline methods) on four test datasets. The quantitative comparison results are shown in Figure 5 As shown in the table, our method is comparable to or even better than these methods in performance. For example, on the Rain100H dataset, RainPhysicTransNet surpasses Restormer and MHNet by 0.49 dB and 0.87 dB in PSNR respectively. In particular, RainPhysicTransNet achieves the best (SOTA) performance in SSIM on the Test1200 dataset (see Figure 6). On the other hand, the poor performance of our algorithm on the Rain100L and Test100 datasets may be due to the significant pattern differences between these datasets and the training datasets, which indicates that our algorithm may be more proficient in image reconstruction in moderate to heavy rainfall scenes. In addition, although DRSformer has the best overall performance, its computational complexity is quite high, about 1.34 times that of our algorithm. By further analyzing Figure 5 From the results shown in , we find that our model generally outperforms PSNR in terms of SSIM. This suggests that during the rain removal process, our model may focus more on maintaining the integrity of the visual structure of the image and make some compromises in recovering details, resulting in a slight decrease in PSNR.

[0087] like Figure 7 As shown in the figure, the first column is the input rainy image, the second column is the real image, and the other columns are the images after rain removal by different methods. Five rainy images are randomly selected from the synthetic dataset. From the comparison of the rain removal effects in the figure, it can be seen that our method performs very well. It can be clearly seen from the PSNR indicator shown below each image that our method can usually achieve PSNR values ​​comparable to or even higher than other methods. For example, in the second row of images, our method surpasses all other methods with a PSNR of 35.09 and performs best. In the third row of images, our method achieves a PSNR of 32.61, ranking second, second only to DRSformer, whose PSNR is 33.36.

[0088] like Figure 8 As shown in the figure, to further verify the generalization ability of our RainPhysicTransNet, we randomly selected four rainy images from a real-world dataset called LHP-rain and applied our method to each image together with six other deraining algorithms. The red rectangular boxes in the images highlight specific local areas and provide enlarged details to better demonstrate the deraining effect of each deraining network. Although our method fails to completely eliminate rain streaks in real-world rainy images, it still performs visually comparable to other methods. Overall, our method is able to effectively remove rain streaks while preserving image details. For example, in the first row of images, the railings of the building and the vegetation in the background are more clearly restored, and the details are kept finer.

[0089] like Fig. 9As shown in Figure 2, in the ablation experiment, we evaluated the impact of different components in the RainPhysicTransNet model. Specifically, we evaluated the first stage based on the rain physics model, the second stage based on PromptFormer, and the role of our proposed group weighting mechanism. The evaluation was performed on the synthetic dataset Test1200, and the results are shown in Figure 2. Fig. 9 Phase1 refers to a network that only contains the first phase based on the rain physics model, without the second phase for optimization. Phase2 refers to a network that only contains the second phase, without the first phase. Phase1+Phase2 refers to the complete RainPhysicTransNet network. Fig. 9 As can be seen in Figure 2, our complete model outperforms each individual stage or ablation variant. Specifically, combining the first stage based on the rain physics model and the second stage based entirely on neural networks, as well as the group weighting mechanism (GWM), achieves significant improvements in both PSNR and SSIM metrics. This shows that each component of the model contributes to the overall performance, especially the GWM plays a vital role in improving the structural integrity and visual quality in the deraining task.

[0090] It should be noted that in Fig. 9 In the table, “-” means “not supported”, “×” means “not activated”, and “√” means “activated”.

[0091] The RainPhysicTransNet proposed in the embodiment of the present invention effectively utilizes the physical properties of rain to enhance the deraining performance, and introduces a group weighting mechanism (GWM) to capture complex rain streak patterns based on the physical properties of rain. And through extensive experimental verification, the two-stage image deraining method based on rain physics model and deep learning provided by the present invention has competitive and even better performance in qualitative and quantitative indicators compared with existing deraining methods. It should be pointed out in particular that the model we proposed has achieved significant improvements in the structural similarity index (SSIM), achieved the best (SOTA) performance on the Test1200 dataset, and also showed excellent performance on other datasets. The results show that the introduction of the physical properties of rain and the innovative GWM not only improves the visual quality of the derained image, but also better preserves the details and structure of the image.

[0092] like Fig.10 As shown, in one embodiment, the present invention also provides a two-stage image deraining system based on rain physics model and deep learning, the system comprising:

[0093] The acquisition module 10 is used to acquire original rainy images.

[0094] The prediction module 20 is used to predict the corresponding atmospheric light map A, transmittance map T and rain streak map S according to the original rainy image.

[0095] The calculation module 30 is used to generate a plurality of rain-removed images Ji according to the rain physics model.

[0096] The fusion module 40 is used to perform weighted summation on multiple derained images Ji using a group weighting mechanism to obtain a preliminary derained image J.

[0097] The optimization module 50 is used to optimize the quality of the preliminary derained image J by applying PromptFormer.

[0098] Fig.11 The physical structure diagram of an electronic device is shown as an example. The electronic device may be a smart terminal, and its internal structure diagram may be as follows: Fig.11 As shown. The electronic device includes a processor, a memory and a network interface connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a two-stage image deraining method based on a rain physics model and deep learning is implemented, and the method includes:

[0099] Collect original rainy images.

[0100] According to the original rainy image, the corresponding atmospheric light map A, transmittance map T and rain streak map S are predicted.

[0101] Generate multiple derained images Ji according to the rain physics model.

[0102] The group weighted mechanism is applied to perform weighted summation on multiple derained images Ji to obtain a preliminary derained image J.

[0103] PromptFormer is applied to optimize the quality of the preliminary derained image J.

[0104] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present invention, and does not constitute a limitation on the electronic device to which the scheme of the present invention is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0105] On the other hand, the present invention further provides a computer storage medium storing a computer program, which, when executed by a processor, implements a two-stage image deraining method based on a rain physics model and deep learning, the method comprising:

[0106] Collect original rainy images.

[0107] According to the original rainy image, the corresponding atmospheric light map A, transmittance map T and rain streak map S are predicted.

[0108] Generate multiple derained images Ji according to the rain physics model.

[0109] The group weighted mechanism is applied to perform weighted summation on multiple derained images Ji to obtain a preliminary derained image J.

[0110] PromptFormer is applied to optimize the quality of the preliminary derained image J.

[0111] In another aspect, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, a two-stage image deraining method based on a rain physics model and deep learning is implemented, the method comprising:

[0112] Collect original rainy images.

[0113] According to the original rainy image, the corresponding atmospheric light map A, transmittance map T and rain streak map S are predicted.

[0114] Generate multiple derained images Ji according to the rain physics model.

[0115] The group weighted mechanism is applied to perform weighted summation on multiple derained images Ji to obtain a preliminary derained image J.

[0116] PromptFormer is applied to optimize the quality of the preliminary derained image J.

[0117] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.

[0118] By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0119] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0120] The above-described embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A two-stage image deraining method based on rain physics model and deep learning, characterized in that: The method comprises: Collect original rainy images; The corresponding atmospheric light map A, transmittance map T and rain streak map S are predicted based on the original rainy image; Generate multiple rain-free images Ji according to the rain physics model; Apply the group weighted mechanism to perform weighted summation on multiple derained images Ji to obtain a preliminary derained image J; PromptFormer is applied to optimize the quality of the preliminary derained image J.

2. The two-stage image deraining method based on rain physics model and deep learning according to claim 1 is characterized in that: The step of predicting the corresponding atmospheric light map A, transmittance map T and rain streak map S according to the original rainy image includes: Apply AtmosNet to predict the atmospheric light map A from the input image; Apply TransmissionNet to predict the transmittance map T from the input image; StreaksNet is applied to predict the rain streak map S from the input image.

3. The two-stage image deraining method based on rain physics model and deep learning according to claim 1, characterized in that: The calculation formula of the rain physics model is: Where I represents the observed rainy image, T is the transmittance map, J is the clear rain-free image, and S i is the rain line map, and A is the global atmospheric light map.

4. The two-stage image deraining method based on rain physics model and deep learning according to claim 3 is characterized in that: The steps of generating multiple derained images Ji according to the rain physics model include: The variant formula is obtained based on the rain physics model Where I represents the observed rainy image, T is the transmittance map, J is the clear rain-free image, and S i is the rain line map, and A is the global atmospheric light map; Multiple derained images Ji are obtained by calculating the variant formula.

5. The two-stage image deraining method based on rain physics model and deep learning according to claim 1, characterized in that: The steps of applying a group weighting mechanism to weighted sum these images to obtain a preliminary derained image J include: Apply WeightNet to adaptively assign weights to each set of parameters; According to the weights assigned to each group of parameters, all derained images Ji are weighted summed to obtain the preliminary derained image J.

6. The two-stage image deraining method based on rain physics model and deep learning according to claim 5, characterized in that: According to the weights assigned to each group of parameters, all the derained images Ji are weighted summed to obtain the preliminary calculation formula of the derained image J: Among them, n is the number of groups of derained images, i is the group number of the derained image, and w is the weight corresponding to the derained image.

7. The two-stage image deraining method based on rain physics model and deep learning according to claim 1, characterized in that: The PromptFormer is a four-stage U-shaped model, where the backbone networks of the first two stages are composed of CNN blocks and the backbone networks of the last two stages are composed of Transformer blocks.

8. The two-stage image deraining method based on rain physics model and deep learning according to claim 7, characterized in that: The process formula for optimizing the quality of the preliminary derained image J by using PromptFormer is: final =PromptFormer(J+X), where Jfinal is the predicted final clear image, J is the preliminary de-rained image, and X is the original rainy image.

9. A two-stage image deraining system based on a rain physics model and deep learning, applied to a two-stage image deraining method based on a rain physics model and deep learning as described in any one of claims 1 to 8, characterized in that: The system comprises: A collection module, used to collect original rainy images; A prediction module is used to predict the corresponding atmospheric light map A, transmittance map T and rain streak map S according to the original rainy image; A calculation module is used to generate multiple rain-free images Ji according to the rain physics model; The fusion module is used to apply the group weighted mechanism to perform weighted summation on multiple derained images Ji to obtain a preliminary derained image J; The optimization module is used to apply PromptFormer to optimize the quality of the preliminary derained image J.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images