A two-stage image rain removal method based on a rain physics model and deep learning

Abstract

The present application relates to a kind of two-stage image rain removal methods based on rain physics model and deep learning, it is related to image rain removal technical field.The method includes: collecting original rain image;According to original rain image, corresponding atmospheric light graph A, transmissivity graph T and rain stripe graph S are predicted;Multiple rain removal images are generated following rain physics model;The weighted summation of multiple rain removal images is obtained by applying grouping weighting mechanism, and preliminary rain removal image J is obtained;The quality of preliminary rain removal image J is optimized by applying PromptFormer.The method, in the first stage, combines the rain physics model to optimize rain removal inference logic, in the second stage, a four-stage U-shaped network is introduced to optimize the output of the first stage, in addition, the robustness of the entire algorithm is further enhanced by combining grouping weighting mechanism.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image rain removal, and particularly relates to a two-stage image rain removal method based on a rain physical model and deep learning. BACKGROUND

[0002] As a common natural phenomenon, rain always brings challenges to certain outdoor vision algorithms that assume clear weather. Environmental perception in autonomous driving systems, target detection in surveillance systems, and unmanned aerial vehicle aerial image processing are all affected by false positives and false negatives due to the presence of rain. Rain significantly affects the stability and reliability of outdoor vision systems. Therefore, rain removal algorithms have received increasing attention in both academia and industry in recent years.

[0003] In the real world, rain streaks can be intertwined with rain trace superposition effects, and the visual information of certain background scenes may be attenuated by rain, especially in long-range views. Traditional methods usually rely too much on hand-designed prior knowledge or physical models that assume specific characteristics of rain streaks. This dependence limits their adaptability in complex real-world environments, where the shape, size, and distribution of rain streaks vary greatly. SUMMARY

[0004] Therefore, it is necessary to provide a two-stage image rain removal method based on a rain physical model and deep learning to solve the above problems.

[0005] The two-stage image rain removal method based on a rain physical model and deep learning provided by the present application comprises the following steps:

[0006] Collecting an original rainy image;

[0007] Predicting a corresponding atmospheric light map A, transmittance map T, and rain streak map S from the original rainy image;

[0008] Generating multiple rain-removed images according to the rain physical model ;

[0009] Performing weighted summation on the multiple rain-removed images using a grouping and weighting mechanism to obtain a preliminary rain-removed image J;

[0010] Optimizing the quality of the preliminary rain-removed image J using PromptFormer.

[0011] In one embodiment, the step of predicting a corresponding atmospheric light map A, transmittance map T, and rain streak map S from the original rainy image comprises:

[0012] Predicting the atmospheric light map A from the input image using AtmosNet.​

[0013] Apply TransmissionNet to predict the transmittance map T from the input image;

[0014] StreaksNet is applied to predict the rain streak map S from the input image.

[0015] In one embodiment, the calculation formula of the rain physics model is: ,in, represents the observed rain image, is the transmittance graph, For clear rain-free images, For the rain line diagram, is the global atmospheric light map, and n is the number of groups.

[0016] In one embodiment, multiple rain-removed images are generated according to the rain physics model. The steps include:

[0017] The variant formula is obtained based on the rain physics model ,in, represents the observed rain image, is the transmittance graph, For clear rain-free images, For the rain line diagram, is the global atmospheric light map;

[0018] Multiple derained images are obtained by calculating the variant formula .

[0019] In one embodiment, the step of applying a group weighting mechanism to perform weighted summation on the images to obtain a preliminary derained image J includes:

[0020] Apply WeightNet to adaptively assign weights to each set of parameters;

[0021] According to each set of parameters, weights are assigned to all derained images. Perform weighted summation to obtain the preliminary derained image J.

[0022] In one embodiment, all rain removal images are assigned weights according to each set of parameters. Perform weighted summation to obtain the preliminary calculation formula of the rain-removed image J: ,in, is the number of groups, is the group number of the derained image, is the weight corresponding to the derained image.

[0023] In one embodiment, the PromptFormer is a four-stage U-shaped model, where the backbone networks of the first two stages are composed of CNN (Deformable Convolutional Neural Network) blocks, and the backbone networks of the last two stages are composed of Transformer blocks.

[0024] In one embodiment, the process formula for optimizing the quality of the preliminary derained image J by using PromptFormer is: ,in, is the predicted final clear image, is the preliminary derained image, This is the original rainy image.

[0025] The present invention also provides a two-stage image deraining system based on a rain physics model and deep learning, the system comprising:

[0026] Acquisition module, used to collect original rainy images;

[0027] The prediction module is used to predict the corresponding atmospheric light map A, transmittance map T and rain streak map S based on the original rain image;

[0028] The calculation module is used to generate multiple rain-free images according to the rain physics model. ;

[0029] Fusion module, used to apply group weighting mechanism to multiple derained images Perform weighted summation to obtain the preliminary derained image J; apply WeightNet to adaptively assign weights to each set of parameters; and perform weighted summation on all derained images according to the weights assigned to each set of parameters. Perform weighted summation to obtain the preliminary derained image J; the calculation formula of the preliminary derained image J is: ,in, is the number of groups, is the group number of the derained image, is the weight corresponding to the derained image;

[0030] The optimization module is used to apply PromptFormer to optimize the quality of the preliminary derained image J. The PromptFormer is a four-stage U-shaped model. The backbone network of the first two stages is composed of CNN blocks, and the backbone network of the last two stages is composed of Transformer blocks. The process formula for applying PromptFormer to optimize the quality of the preliminary derained image J is: ,in, is the predicted final clear image, is the preliminary derained image, This is the original rainy image.

[0031] The two-stage image rain removal method based on the rain physical model and deep learning optimizes the rain removal inference logic in combination with the rain physical model in the first stage, introduces a four-stage U-shaped network to optimize the output of the first stage in the second stage, and further enhances the robustness of the entire algorithm in combination with the grouping weighting mechanism. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0033] Figure 1 The flow chart of the two-stage image rain removal method based on the rain physical model and deep learning proposed by the present application;

[0034] Figure 2 The architecture diagram of RainPhysicTransNet in the two-stage image rain removal method based on the rain physical model and deep learning proposed by the present application;

[0035] Figure 3 The architecture diagram of PromptFormer in the two-stage image rain removal method based on the rain physical model and deep learning proposed by the present application; Figure 2

[0036] Figure 4 The basic module schematic diagram used by the first stage subnetwork of RainPhysicTransNet in the two-stage image rain removal method based on the rain physical model and deep learning proposed by the present application;

[0037] Figure 5 The quantitative comparison result diagram of the two-stage image rain removal method based on the rain physical model and deep learning proposed by the present application and previous methods on four rain removal data sets (Rain100L, Rain100H, Test1200, Test100);

[0038] Figure 6 The performance schematic diagram of RainPhysicTransNet and previous methods on the data set Test1200;

[0039] Figure 7 The qualitative comparison result diagram of the two-stage image rain removal method based on the rain physical model and deep learning proposed by the present application and other methods on a synthetic data set;

[0040] Figure 8 ​A qualitative comparison chart of the two-stage image rain removal method based on the rain physical model and deep learning proposed in the present application with other six rain removal algorithms on real data;

[0041] Figure 9 Ablation experiment data table of each module in the two-stage image rain removal method based on the rain physical model and deep learning proposed in the present application;

[0042] Figure 10 A structure diagram of the two-stage image rain removal system based on the rain physical model and deep learning proposed in the present application;

[0043] Figure 11 An internal structure diagram of a computer device of an embodiment. DETAILED DESCRIPTION

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0045] Abstract Since deep learning has revived the image inpainting technique, the research on rain removal algorithms combined with physical principles or prior knowledge has become less and less. The generalization ability of an algorithm in an unfamiliar scene is still limited only by the training of a neural network (even if a large-scale dataset is used). Such methods are usually based only on learned data for inference, but lack an understanding of the physical phenomenon of “rain”. By incorporating the physical principles related to rain (such as raindrop shape, falling trajectory, and light scattering, etc.) into the algorithm, the understanding ability of the model can be enhanced, thereby improving the robustness under different rainfall intensities, angles, and types.

[0046] Some studies have attempted to incorporate constraints and prior knowledge related to rain into deep learning, and there is currently a two-stage rain removal algorithm. In the first stage, the algorithm performs frequency decomposition on the original input image, and then inputs the decomposed image into three branch sub-networks to predict the physical parameters in the formula of the rain physical model. Then, these parameters are substituted into the variant formula of the rain physical model formula to derive the clear image J. In the second stage, the authors use a conditional generative adversarial network (CGAN) as an optimization model, which takes the output of the first stage as input to further improve the quality of the rain-removed image.

[0047] Inspired by this research, we can conclude the following two points: First, the introduction of a physical model allows rain removal algorithms to rely not only on data-driven approaches but also on physical principles. Second, by incorporating rain-related physical characteristics as prior knowledge, the physical model can constrain the model's predictions during training, preventing the neural network from generating unreasonable solutions, accelerating its convergence, improving rain removal performance, and enhancing the network's robustness and generalization capabilities.

[0048] However, in the real world, rain streaks can be intertwined with the superposition of rain tracks, and visual information of certain background scenes can be attenuated by rain, especially in distant scenes. Therefore, physical models struggle to fully represent these complex characteristics. Rather than designing a more complex physical model, it's simpler and more efficient to introduce an additional optimization network in the second stage to further improve the quality of the output image from the first stage.

[0049] In response to the above problems, the following Figures 1-11 The present invention describes a two-stage image deraining method based on rain physics model and deep learning.

[0050] like Figure 1 As shown, in one embodiment, a two-stage image deraining method based on a rain physics model and deep learning includes:

[0051] Step S100: collecting original rainy images.

[0052] Step S200 , predicting and obtaining the corresponding atmospheric light map A, transmittance map T and rain streak map S based on the original rainy image.

[0053] Step S300: Generate multiple rain-removed images according to the rain physics model .

[0054] Step S400: Applying a group weighting mechanism to multiple rain removal images Perform weighted summation to obtain the preliminary derained image J.

[0055] Specifically, the group weighting mechanism uses multiple CNN (convolutional neural network) sub-networks to extract multiple groups of physical parameters, generate multiple derained images, and enhance the image deraining effect.

[0056] Step S500 : Apply PromptFormer to optimize the quality of the preliminary derained image J.

[0057] Specifically, such as Figure 2 and Figure 3As shown, according to the two-stage neural network RainPhysicTransNet proposed by us in the process of image deraining, in the first stage, according to the obtained original rainy image, a corresponding multiple sets of atmospheric light graph A, transmittance graph T and rain streak graph S are predicted, and multiple deraining images are generated in strict accordance with the rain physical model Then, the images are weighted and summed to obtain a preliminary deraining image J. In the second stage, a U-shaped network composed of CNN and Transformer components is used to optimize the quality of the output image of the first stage.

[0058] It should be noted that, as Figure 4 As shown, the basic module used by the first stage subnetwork is composed of LayerNorm (layer normalization technology), DeformConv (deformable convolution), SimpleGate (gate mechanism) and MSConv (multi-scale convolution operation). Among them, DeformConv is used to enhance the feature representation ability of the network, because it allows the convolution kernel to adaptively adjust its shape and position. MSConv is a simplified version of SFConv (selective feature convolution), which aims to enrich the context information of the features, because it combines a multi-scale feature enhancement mechanism and a channel attention mechanism. SimpleGate acts as an activation function, helping the network to selectively focus on more useful features.

[0059] The beneficial effects of the present embodiment are that, based on the two-stage framework design, the grouping weighting mechanism is integrated into the design of the first stage, by generating multiple sets of physical parameters (each set corresponds to one deraining image), and then weighting and combining all the derived deraining images to reduce the uncertainty of a single prediction result, thereby enhancing the stability of the final output. And deformable convolutional neural network and multi-scale channel attention mechanism are integrated into the design of all branch subnetworks to improve their ability to extract multi-scale features. At the same time, the step of frequency decomposition of the original input image is deleted to allow the neural network to process information more freely and extract features.

[0060] In one embodiment, the step of predicting the corresponding atmospheric light graph A, transmittance graph T and rain streak graph S from the original rainy image includes:

[0061] Step S210, applying AtmosNet (atmospheric network) to predict the atmospheric light graph A from the input image.

[0062] Step S220, applying TransmissionNet (transmission network) to predict the transmittance graph T from the input image.

[0063] Step S230, StreaksNet is applied to predict rain streaks map S from the input image.

[0064] Specifically, the process of predicting atmospheric light map A, transmittance map T and rain streaks map S from the input image can be represented by the formula wherein, represents the input image.

[0065] It should be noted that AtmosNet, TransmissionNet and StreaksNet are all small U-shaped networks composed of residual blocks.

[0066] In one embodiment, the calculation formula of the rain physical model is wherein, represents the observed rainy image, is the transmittance map, is the clear rain-free image, is the rain line map, is the global atmospheric light map, and n is the number of groups.

[0067] In this embodiment, the steps of generating multiple rain-removed images according to the rain physical model include;

[0068] Step S310, according to the rain physical model, the variant formula is obtained wherein, represents the observed rainy image, is the transmittance map, is the clear rain-free image, is the rain line map, is the global atmospheric light map.

[0069] Step S320, multiple rain-removed images are calculated by the variant formula.

[0070] In one embodiment, the steps of applying a grouping weighting mechanism to weight sum these images to obtain a preliminary rain-removed image J include;

[0071] Step S410, WeightNet is applied to adaptively assign weights to each group of parameters.

[0072] Specifically, the process of adaptively extracting the weight vector from the original input image by WeightNet can be represented by the formula wherein, represents the input image, respectively, represent the weights assigned to the three groups of physical parameters. The weights are adaptively assigned to each group of parameters by the WeightNet, so that the features of each group of rain-removed images can be reasonably combined when weighted summation is performed later. This mechanism improves the robustness and visual quality of the final rain-removed image, effectively dealing with the complexity of different rain patterns.

[0073] Step S420, weighted summation of all rain-removed images according to the weight assigned to each group of parameters to obtain a preliminary rain-removed image J.

[0074] Specifically, the subnetworks of AtmosNet, TransmissionNet, StreaksNet and WeightNet all contain basic modules composed of LayerNorm, DeformConv, SimpleGate and MSConv. In its composition, DeformConv is used to enhance the feature representation ability of the network, because it allows the convolution kernel to adaptively adjust its shape and position. MSConv, a simplified version of SFConv, aims to enrich the contextual information of features, as it combines a multi-scale feature enhancement mechanism and a channel attention mechanism. SimpleGate, as an activation function, helps the network selectively focus on more useful features.

[0075] In this embodiment, the weighted summation of all rain-removed images according to the weight assigned to each group of parameters to obtain a preliminary rain-removed image J. , wherein is the number of groups, is the group number of the rain-removed image, is the weight corresponding to the rain-removed image.

[0076] In one embodiment, the PromptFormer is a four-stage U-shaped model, the backbone network of the first two stages is composed of CNN (Convolutional Neural Network) blocks, and the backbone network of the last two stages is composed of Transformer blocks.

[0077] In this embodiment, the process formula for optimizing the quality of the preliminary rain-removed image J by applying PromptFormer is , wherein is the predicted final clear image, is the preliminary rain-removed image, is the original rainy image.

[0078] It is worth noting that in this study, the only loss function used in the training process is the peak signal-to-noise ratio (PSNR) based loss function. PSNR is a widely used metric in image quality assessment, used to measure the quality of the recovered image. The higher the PSNR value, the better the image recovery quality, as it indicates a smaller deviation between the predicted image and the target image. The core idea of this loss function is to first calculate the mean square error (MSE) between the predicted image and the target image, and then convert it to the PSNR value for optimization. The calculation formula of MSE in this process is where, is the number of pixels of the image, is the pixel number of the image, is the predicted clear image, is the true value clear image.

[0079] And to evaluate the performance of our algorithm, we compare it with the baseline method and the optimal method on the synthetic rain dataset and the real rain dataset.

[0080] First, we will introduce the experimental setup, followed by the qualitative and quantitative comparison results on multiple datasets. In addition, we will also verify the rationality of our method through ablation experiments. In the table, the best quality score of the evaluation method is highlighted in bold, and the second best quality score is underlined.

[0081] Next, we will introduce the details of the dataset, evaluation indicators, training process, and hyperparameter settings.

[0082] The training dataset we adopted is Rain13K, which is a synthetic rain dataset containing 13,712 pairs of clear images and rain scene images. The synthetic datasets used for verification and testing include Rain100L, Rain100H, Test1200, and Test100. The real rain dataset we selected is LHP-rain, which is used for qualitative comparison experiments.

[0083] The indicators we used to evaluate the performance of the network are PSNR (peak signal-to-noise ratio) and SSIM (structural similarity index). The higher the PSNR and SSIM values, the better the performance of the algorithm.

[0084] We divide the training process of RainPhysicTransNet into two steps. In the first step, we train the first part of RainPhysicTransNet independently, including AtmosNet, TransmissionNet, StreaksNet, and WeightNet. The training loss of the first step comes from the MSELoss (Mean Squared Error Loss) between the rain-removed image generated by the first part and the ground-truth image. In the second step, we load the model weights obtained in the first step and start training the entire model. The purpose of this design is to constrain the first part of RainPhysicTransNet through the physical model of rain in advance, so that it can obtain sufficient prior knowledge related to the physical properties of rain in advance, and prepare for the subsequent training of the entire model to impose more reasonable physical constraints.

[0085] Based on the first stage of the physical model, the number of groups N is set to 3. In the PromptFormer (Prompt Generator) of the second stage, the number of basic modules in each stage is set to [1, 1, 6, 8], where the first two values are the number of CNN blocks, and the last two values are the number of Transformer blocks. We use the Adam optimizer with a batch size of 8 and train on 8 NVIDIA 3090 GPUs for a total of 400 rounds each time. The learning rate is initialized to 2e-4 and adjusted by the CosineAnnealingLR (Cosine Annealing Learning Rate) scheduler until it decays to 4e-8. The loss function used to train RainPhysicTransNet is the PSNR-based loss function PSNRLoss (Peak Signal-to-Noise Ratio Loss).

[0086] We compare RainPhysicTransNet with eight other methods (including the optimal method and the benchmark method) on four test datasets. The quantitative comparison results are shown in Table Figure 5 As can be seen from the data in the table, our method is comparable or even superior in performance to these methods. For example, on the Rain100H dataset, RainPhysicTransNet outperforms Restormer and MHNet by 0.49 dB and 0.87 dB in PSNR, respectively. In particular, RainPhysicTransNet achieves the best (SOTA) performance in SSIM on the Test1200 dataset (see Figure 6). On the other hand, our algorithm's poorer performance on Rain100L and Test100 datasets can be attributed to significant pattern differences between these datasets and the training datasets, suggesting that our algorithm may be more proficient in handling image reconstruction in heavy rain scenarios. Furthermore, although DRSformer is the best overall, its computational complexity is quite high, approximately 1.34 times that of our algorithm. Through further analysis Figure 5 of the results shown in the

[0087] As shown in Figure 7 , the first column is the input rainy image, the second column is the true image, and the other columns are the images after rain removal by different methods. The five rainy images are randomly selected from the synthetic dataset. From the rain removal effect comparison in the figure, it can be seen that our method performs very well. From the PSNR indicators displayed below each image, it can be seen that our method can usually achieve comparable or even higher PSNR values than other methods. For example, in the second row of images, our method outperforms all other methods with a PSNR of 35.09, showing the best performance. In the third row of images, our method achieves a PSNR of 32.61, ranking second, only slightly lower than DRSformer, which has a PSNR of 33.36.

[0088] As shown in Figure 8 , to further verify the generalization ability of our RainPhysicTransNet, we randomly selected four rainy images from a real-world dataset called LHP-rain and applied our method to each image along with six other rain removal algorithms. The red rectangular boxes in the images highlight specific local areas and provide enlarged details to better demonstrate the rain removal effect of each rain removal network. Although our method does not completely eliminate rain streaks in real-world rainy images, it still performs visually comparable to other methods. Overall, our method can effectively remove rain streaks while preserving image details. For example, in the first row of images, the railing of the building and the vegetation in the background are restored more clearly, with finer details preserved.

[0089] As shown in Figure 9As shown in the ablation experiment, we evaluate the impact of different components in the RainPhysicTransNet model. Specifically, we evaluate the first stage based on the rain physics model, the second stage based on PromptFormer, and the Group Weighting Mechanism we proposed, respectively. The evaluation is carried out on the synthetic dataset Test1200, and the results are shown in Figure 9 Phase1 refers to a network that only contains the first stage based on the rain physics model and does not contain the second stage for optimization. Phase2 refers to a network that only contains the second stage and does not contain the first stage. Phase1 + Phase2 refers to the complete RainPhysicTransNet network. As can be seen from Figure 9 , our complete model outperforms each individual stage or ablation variant in terms of performance. Specifically, the combination of the first stage based on the rain physics model and the second stage based entirely on neural networks, as well as the Group Weighting Mechanism (GWM), has achieved significant improvement in both PSNR and SSIM indicators. This indicates that each component of the model contributes to improving the overall performance, and the GWM plays a crucial role in improving the structural integrity and visual quality in the rain removal task.

[0090] It should be noted that in Figure 9 “-” means “not supported”, “×” means “not activated”, and “✓” means “activated”.

[0091] The RainPhysicTransNet proposed by the embodiments of the present application effectively utilizes the physical properties of rain to enhance the rain removal performance, and introduces the Group Weighting Mechanism (GWM) to capture complex rain streak patterns based on the physical properties of rain. Through extensive experimental verification, the two-stage image rain removal method based on the rain physics model and deep learning provided by the present application has competitive and even better performance in qualitative and quantitative indicators compared with existing rain removal methods. It is particularly important to note that the model we proposed has achieved significant improvement in the Structural Similarity Index (SSIM) and achieved the best (SOTA) performance on the Test1200 dataset, and also showed excellent performance on other datasets. The results show that the introduction of the physical properties of rain and the innovative GWM not only improves the visual quality of the rain-removed image, but also better preserves the details and structure of the image.

[0092] As shown in Figure 10 , in one embodiment, the present application also provides a two-stage image rain removal system based on the rain physics model and deep learning, which comprises:

[0093] The acquisition module 10 is used to acquire original rainy images.

[0094] The prediction module 20 is used to predict the corresponding atmospheric light map A, transmittance map T and rain streak map S according to the original rain image.

[0095] The calculation module 30 is used to generate multiple rain-removed images according to the rain physics model. .

[0096] Fusion module 40 is used to apply a group weighting mechanism to multiple rain-removed images Perform weighted summation to obtain the preliminary derained image J; apply WeightNet to adaptively assign weights to each set of parameters; and perform weighted summation on all derained images according to the weights assigned to each set of parameters. Perform weighted summation to obtain the preliminary derained image J; the calculation formula of the preliminary derained image J is: ,in, is the number of groups, is the group number of the derained image, is the weight corresponding to the derained image.

[0097] The optimization module 50 is used to optimize the quality of the preliminary derained image J by using PromptFormer. The PromptFormer is a four-stage U-shaped model. The backbone networks of the first two stages are composed of CNN blocks, and the backbone networks of the last two stages are composed of Transformer blocks. The process formula for optimizing the quality of the preliminary derained image J by using PromptFormer is: ,in, is the predicted final clear image, is the preliminary derained image, This is the original rainy image.

[0098] Figure 11 The following is a schematic diagram of the physical structure of an electronic device. The electronic device may be a smart terminal, and its internal structure diagram may be as follows: Figure 11 As shown. The electronic device includes a processor, a memory, and a network interface connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a two-stage image deraining method based on a rain physics model and deep learning is implemented, the method comprising:

[0099] Collect original rainy images.

[0100] According to the original rainy image, a corresponding atmospheric light map A, transmittance map T and rain streak map S are predicted.

[0101] A plurality of rain-removed images are generated following a rain physics model .

[0102] The plurality of rain-removed images are weighted and summed using a grouping weighting mechanism to obtain a preliminary rain-removed image J.

[0103] The quality of the preliminary rain-removed image J is optimized using a PromptFormer.

[0104] Those skilled in the art can understand that Figure 11 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0105] In another aspect, the present application also provides a computer storage medium storing a computer program, the computer program being executed by a processor to implement a two-stage image rain-removal method based on a rain physics model and deep learning, the method comprising:

[0106] An original rainy image is collected.

[0107] According to the original rainy image, a corresponding atmospheric light map A, transmittance map T and rain streak map S are predicted.

[0108] A plurality of rain-removed images are generated following a rain physics model .

[0109] The plurality of rain-removed images are weighted and summed using a grouping weighting mechanism to obtain a preliminary rain-removed image J.

[0110] The quality of the preliminary rain-removed image J is optimized using a PromptFormer.

[0111] In yet another aspect, a computer program product or computer program is provided, the computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. The processor of the electronic device reads the computer instructions from the computer-readable storage medium, and the processor implements a two-stage image rain-removal method based on a rain physics model and deep learning when executing the computer instructions, the method comprising:

[0112] An original rainy image is collected.

[0113] ​​According to the original rainy image prediction, a corresponding atmospheric light map A, transmittance map T and rain streak map S are obtained.

[0114] According to the rain physical model, a plurality of rain-removed images are generated .

[0115] The plurality of rain-removed images are weighted and summed by using a grouping weighting mechanism to obtain a preliminary rain-removed image J.

[0116] The quality of the preliminary rain-removed image J is optimized by using a PromptFormer.

[0117] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. Any reference to memory, storage, database or other medium used in each embodiment of the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory.

[0118] As an illustration but not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM) and the like.

[0119] Each technical feature of the above embodiments can be combined arbitrarily. In order to make the description simple, not all possible combinations of each technical feature in the above embodiments are described, however, as long as the combinations of these technical features do not exist, they should be considered as the scope of the present disclosure.

[0120] ​The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the present application. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, several modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A two-stage image deraining method based on rain physics model and deep learning, characterized by: The method comprises: Collect original rainy images; The corresponding atmospheric light map A, transmittance map T and rain streak map S are predicted based on the original rain image; Generate multiple rain-free images according to the rain physics model ; Apply WeightNet to adaptively assign weights to each set of parameters; According to each set of parameters, weights are assigned to all derained images. Perform weighted summation to obtain the preliminary derained image J; the calculation formula of the preliminary derained image J is: ,in, is the number of groups, is the group number of the derained image, is the weight corresponding to the derained image; Applying PromptFormer to optimize the quality of the preliminary derained image J; the PromptFormer is a four-stage U-shaped model, where the backbone networks of the first two stages are composed of CNN blocks, and the backbone networks of the last two stages are composed of Transformer blocks; The process formula for optimizing the quality of the preliminary derained image J using PromptFormer is: ,in, is the predicted final clear image, is the preliminary derained image, This is the original rainy image.

2. The two-stage image deraining method based on rain physics model and deep learning according to claim 1 is characterized in that: The steps of predicting the corresponding atmospheric light map A, transmittance map T, and rain streak map S according to the original rain image include: Apply AtmosNet to predict the atmospheric light map A from the input image; Apply TransmissionNet to predict the transmittance map T from the input image; StreaksNet is applied to predict the rain streak map S from the input image.

3. The two-stage image deraining method based on rain physics model and deep learning according to claim 1, characterized in that: The calculation formula of the rain physics model is: ,in, represents the observed rain image, is the transmittance graph, For clear rain-free images, For the rain line diagram, is the global atmospheric light map, and n is the number of groups.

4. The two-stage image deraining method based on rain physics model and deep learning according to claim 3 is characterized in that: Generate multiple rain-free images according to the rain physics model The steps include: The variant formula is obtained based on the rain physics model ,in, represents the observed rain image, is the transmittance graph, For clear rain-free images, For the rain line diagram, is the global atmospheric light map; Multiple derained images are obtained by calculating the variant formula .

5. A two-stage image deraining system based on a rain physics model and deep learning, used to implement the two-stage image deraining method based on a rain physics model and deep learning according to any one of claims 1 to 4, characterized in that: The system comprises: Acquisition module, used to collect original rainy images; The prediction module is used to predict the corresponding atmospheric light map A, transmittance map T and rain streak map S based on the original rain image; The calculation module is used to generate multiple rain-free images according to the rain physics model. ; Fusion module, used to apply group weighting mechanism to multiple derained images Perform weighted summation to obtain the preliminary derained image J; The optimization module is used to apply PromptFormer to optimize the quality of the preliminary derained image J.

6. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

Images