Image rain removal method and system based on improved diffusion model

By improving the image rain removal method of the diffusion model, using frequency dependence strategy and zero-value domain decomposition technology, the problems of rainwater residue and detail loss are solved, the quality and consistency of image rain removal are improved, and it is suitable for real-time processing on mobile devices.

CN119006326APending Publication Date: 2024-11-22SHENYANG AEROSPACE UNIVERSITY
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202411105784.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing image rain removal method has problems such as rainwater residue and incomplete local details, resulting in low quality of rain removal images, especially under rainfall conditions, which significantly reduces the image quality, affecting the effectiveness and accuracy of subsequent algorithms and human eye vision.

Method used

Using an image rain removal method based on an improved diffusion model, the noise estimation network structure is optimized by introducing a frequency dependency strategy, balance high-frequency and low-frequency feature information, and perform zero-domain decomposition during the reverse diffusion process to improve the quality of image rain removal and consistency.

Benefits of technology

It realizes high-quality image rain removal processing, ensures the clarity and visual quality of rain removal images, improves the accuracy and consistency of image rain removal, and is suitable for real-time processing on mobile devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006326A_ABST
    Figure CN119006326A_ABST
Patent Text Reader

Abstract

The invention discloses an image rain removal method and system based on an improved diffusion model, and relates to the technical field of image processing. According to the invention, the noise estimation network of the diffusion model is used to predict the noise such as rain stripes in the rain image. Specifically, a frequency dependence strategy is introduced into a noise estimation network of a diffusion model to enhance the noise distribution feature extraction capability of the noise estimation network. Under the guidance of a frequency dependence strategy, two scale factors are introduced to balance the weights of the high-frequency features and the low-frequency features, and the features of the rain stripes are effectively obtained. Furthermore, in the back diffusion process of the diffusion model, zero-value domain decomposition and zero-domain information learning are added, so that the image consistency in the back diffusion process is improved, and the image rain removal capability is improved. In addition, a rain removal system is designed based on the rain removal method, and the system is deployed to a mobile device end to display an image rain removal result in real time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image rain removal method and system based on an improved diffusion model. Background Art

[0002] Currently, images are widely used in many fields, including computer vision, medical imaging, security monitoring, fire warning, etc.

[0003] However, images taken under rainy conditions are often affected by raindrops or rain streaks. These raindrops or rain streaks reflect the light of the surrounding environment, resulting in light spots and noise in the image, which significantly degrades the image quality. This will seriously affect the effectiveness and accuracy of subsequent algorithms and human vision. Therefore, it is crucial to develop an image deraining method that can effectively remove rain and restore image background information with high quality.

[0004] Early traditional image deraining methods mainly performed deraining based on prior knowledge, such as Gaussian mixture models, discriminative sparse coding, etc. However, these methods based on prior knowledge have limited representation capabilities and are difficult to process complex rainy images. In recent years, deep learning has been applied to the problem of single image deraining with its powerful learning and representation capabilities. This type of method mainly uses convolutional neural networks and generative adversarial networks to learn the mapping relationship between rainy images and clean images. Compared with early traditional methods, deep learning methods have improved performance in processing rainy images. However, this type of method still has the problem of unstable training and will generate low-quality derained images.

[0005] In order to further improve the image deraining effect, researchers have proposed image deraining methods based on diffusion models, such as using the Markov chain principle to gradually add Gaussian noise to the image during the forward diffusion process, and gradually remove the noise during the reverse diffusion process to restore the image. However, when such methods are applied to image deraining, there is residual rain and loss of texture details, which leads to an over-smoothing effect on the restored background and reduces the quality of the derained image. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the present invention provides an image deraining method and system based on an improved diffusion model, aiming to solve the problems of rainwater residue and incomplete local details in the existing image deraining methods, improve the quality of image deraining, and design a deraining system suitable for mobile devices that matches the method of the present invention, so as to realize the application of the method of the present invention in mobile devices such as mobile phones and tablets.

[0007] The technical solution of the present invention is:

[0008] A first aspect of the present invention provides an image deraining method based on an improved diffusion model, the method comprising the following steps:

[0009] Step 1: Prepare a dataset consisting of image pairs and divide it into a training set and a test set; the image pairs consist of rainy images y j , and y j The corresponding clean image x j , where j = 0, 1, 2, ..., q, ..., n, n is the total number of image pairs in the dataset, and q is the total number of image pairs in the training set;

[0010] Step 2: Set the time step sequence length T of the forward diffusion process of the diffusion model;

[0011] Step 3: Select a set of image pairs from the training set and randomly select a time step t from T; add Gaussian noise to the clean image in the set of image pairs based on time step t to perform a forward diffusion process to obtain the noisy image x at time step t t ;

[0012] Step 4: Introduce a frequency-dependent strategy into the noise estimation network of the diffusion model to improve the noise estimation network structure and enhance the ability of the noise estimation network to extract noise distribution characteristics. t The input is sent to the improved noise estimation network to perform the reverse diffusion process to extract the noise distribution characteristics and predict a more accurate noise distribution.

[0013] Step 5: Set the loss function, iteratively train and improve the noise estimation network, and determine whether the stop training condition is met at the end of each training. If not, go to step 3. If yes, use the noise estimation network at the end of the current training as the trained noise estimation network model;

[0014] Step 6: Randomly select rainy images y from the test set s , s = q+1,q+2,…,n, and input it into the trained noise estimation network model to predict the noise distribution ε of the rainy image θ (x t ,t), and according to ε θ (x t ,t) get the preliminary rain image denoising result x st , and then perform zero-value domain decomposition to restore the clean image x s , completing the image deraining reasoning process.

[0015] Furthermore, the frequency-dependent strategy is introduced to improve the noise estimation network structure, specifically setting two scaling factors b f and f , for the noisy image x t The low frequency characteristics x f, high frequency features h f These two characteristics are balanced.

[0016] Furthermore, two scaling factors b are set f and f , for the noisy image x t The low frequency characteristics x f , high frequency features h f The way to balance these two characteristics is:

[0017] First, the noisy image x t The low frequency characteristics x f Adjust and calculate the average feature map along the channel dimension

[0018]

[0019] Among them, x f,i Represents the low-frequency feature x f The feature of the ith channel of f The total number of channels in ; the scaling factor of the low-frequency feature information is then determined:

[0020]

[0021] Among them, a f is a scalar constant; then the low-frequency feature x f With the scale factor b f Multiply, for low frequency features x f Perform detail enhancement;

[0022] As the details of low-frequency features are enhanced, the image becomes smoother and smoother. In order to further alleviate the problem caused by low-frequency features x f The over-smoothing problem caused by this requires high-frequency features h f To alleviate the high-frequency components in the f First, the features in each channel are converted to the frequency domain through Fourier transform, the original value of its high-frequency components is retained, and its low-frequency components are suppressed. The specific operations are as follows:

[0023]

[0024] Among them, h f,i Represents high-frequency features h f The characteristics of the ith channel; FFT(·) and IFFT(·) are Fourier transform and inverse Fourier transform respectively; h' f,i is the feature after inverse Fourier transform; ⊙ represents element-wise multiplication; β f,i is a Fourier mask designed as a function of the magnitude of the Fourier coefficients:

[0025]

[0026] Among them, s f is related to the high frequency feature h f The relevant scaling factor; r thresh represents the radius threshold; r represents the radius after frequency domain conversion.

[0027] Furthermore, the training stopping condition described in step 5 is completing a preset number of iterations or the loss function reaching convergence.

[0028] Furthermore, the step 6 comprises the following steps:

[0029] Step 6.1: Input the rainy image y that needs to be derained s , and sample a value from the Gaussian distribution that matches the rainy image y s Pure Gaussian noise image x of uniform size T Use it as the initial noise of the reasoning process, and the time step t is equal to the length of the time step sequence T;

[0030] Step 6.2: The rain image y s , pure Gaussian noise image x T And time step t is input into the trained noise estimation network model to obtain the predicted noise distribution ε θ (x t ,t), and then according to the predicted noise distribution ε θ (x t ,t), and use formula (7) to obtain the preliminary rain image denoising result x st ;

[0031]

[0032] Among them, α t =1-β t , α t is experience; t Scheduling for variance;

[0033] Step 6.3: De-noise the initial rainy image x st Perform zero-value domain decomposition, refine zero-domain information, and obtain consistent deraining results

[0034] Step 6.4: Use formula (13) to update the deraining result of the current time step t to obtain the deraining image corresponding to the next time step t-1:

[0035]

[0036] Step 6.5: Update the current time step t to t-1, and determine whether the current time step is 0. If so, the reasoning is complete, and the current x t-1 That is the predicted rain-free image x s ; If not, go to step 6.1.

[0037] A second aspect of the present invention provides an image deraining system based on an improved diffusion model, the system being applied to a mobile device, the system comprising:

[0038] The image collection module is responsible for collecting rainy images that need to be processed; saving the collected rainy images to the system album of the mobile device; using the image selector provided by the mobile device system to select the saved rainy images from the system album of the mobile device;

[0039] The image preprocessing module is responsible for scaling and normalizing the rainy images selected in the image collection module to ensure that the images have a uniform size and that the pixel values ​​of the images are more evenly distributed;

[0040] The image inference module is responsible for receiving the preprocessed image processed by the image preprocessing module; deploying the trained noise estimation network model described in claim 1 on the mobile device; performing rain removal inference on the preprocessed image according to the deployed model to obtain a rain removal image;

[0041] The result processing and display module is responsible for denormalizing the derained image output by the image inference module to map the image back to the original pixel value range; it also visualizes the processed derained image so that users can view and compare the images before and after deraining in real time.

[0042] Furthermore, in the image inference module, the trained noise estimation network model is first optimized and then deployed on the mobile device; the optimization of the trained noise estimation network model is as follows: first, the trained noise estimation network model is converted from a derain.pth format to a derain.onnx format; then the derain.onnx format is simplified to a derain_sim.onnx format; and finally, the derain_sim.onnx format is converted to a derain_sim.param and a derain_sim.bin format.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. The method of the present invention uses a deep learning algorithm based on a diffusion model as the core to perform image deraining processing. Through the forward and reverse diffusion processes, a high-quality clean image is gradually restored from the noisy image, ensuring the clarity and visual quality of the derained image.

[0045] 2. The present invention adopts a frequency-dependent strategy to optimize the noise estimation network structure of the diffusion model. This strategy can adaptively balance the high-frequency feature information and low-frequency feature information of the image in the noise estimation network, thereby obtaining a derained image with high fidelity and high detail expression capability.

[0046] 3. The present invention introduces zero-domain decomposition into the reverse diffusion process of the diffusion model to improve the quality of image deraining. Image deraining is an inverse problem. After the action of the degradation operator, the image information is lost. Zero-domain decomposition is performed on the rainy image to obtain domain information and zero-domain information. The domain information is retained, while the zero-domain information is lost, and the domain information retains valid information that meets the data consistency requirements. In order to solve the zero-domain information, the present invention uses the intermediate result generated by the noise estimation network as the zero-domain information, extracts the valid domain information in the rainy image, and refines the zero-domain information to make the derained image closer to the clean image, so as to improve the consistency and authenticity of the derained image.

[0047] 4. The present invention also designs a rain removal system suitable for mobile devices that matches the method of the present invention, which can perform rain removal processing on rainy images and display the image rain removal effect in real time on mobile devices. It can not only help users to remove rain from images anytime and anywhere and intuitively observe the rain removal effect, but also provide convenience for subsequent computer vision systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flow chart of the image deraining method based on the improved diffusion model in this embodiment;

[0049] Figure 2 This is a network structure diagram involved in the image deraining method based on the improved diffusion model in this embodiment;

[0050] Figure 3 This is a structural block diagram of the image deraining system based on the improved diffusion model in this embodiment;

[0051] Figure 4 This is a flow chart of the model optimization process in this implementation mode;

[0052] Figure 5 A screenshot showing the effect of image rain removal achieved on a mobile device according to this embodiment. DETAILED DESCRIPTION

[0053] To facilitate understanding of the present application, the present application will be described more comprehensively below with reference to the relevant drawings.

[0054] Image deraining is a challenging task because images obtained in rainy weather usually have severe degradation and artifacts. In order to enable a deep learning model to remove rain streaks from an image, the present invention provides an image deraining method and system based on an improved diffusion model. The basic idea of ​​the present invention is as follows: Although the existing image deraining methods have made certain progress, there are still problems with imperfect details and residual rain. In response to this problem, the present invention uses a noise estimation network of a diffusion model to predict noise such as rain streaks in rainy images. Specifically, a frequency-dependent strategy is introduced into the noise estimation network of the diffusion model to enhance the ability of the noise estimation network to extract noise distribution features. Under the guidance of the frequency-dependent strategy, two scaling factors are introduced to balance the weights of high-frequency features and low-frequency features, and the features of rain streaks are effectively obtained. Furthermore, in the reverse diffusion process of the diffusion model, zero-value domain decomposition is added to learn zero-domain information to improve the image consistency in the reverse diffusion process and improve the image deraining ability. In addition, the deraining method is deployed on a mobile device to display the image deraining results in real time.

[0055] Figure 1 FIG. 1 is a flowchart of the image deraining method based on the improved diffusion model in this embodiment. Figure 1 As shown, the image deraining method based on the improved diffusion model includes the following steps:

[0056] Step 1: Prepare a dataset consisting of image pairs and divide it into a training set and a test set; the image pairs consist of rainy images y j , and y j The corresponding clean image x j , where j = 0, 1, 2, ..., q, ..., n, n is the total number of image pairs in the dataset, and q is the total number of image pairs in the training set;

[0057] When preparing a dataset consisting of image pairs in this step, you can take images yourself to create image pairs and compose a dataset according to your needs, or you can use the public synthetic datasets Rain100H, Rain100L, Rain200H, and Rain200L.

[0058] The data set used in this embodiment is Rain200L, which contains 2000 pairs of matching rainy images and clean images. This embodiment obtains 1800 pairs of images to establish a training set, and uses the remaining 200 pairs of images to establish a test set.

[0059] Step 2: Set the time step sequence length T of the forward diffusion process of the diffusion model;

[0060] In this embodiment, the time step sequence length T of the forward diffusion process is set to 1000, that is, the clean image x obtained from the data set at t=00 , it needs to go through 1000 steps of noise addition to become a noise image x with pure Gaussian noise distribution T .

[0061] Step 3: Select a set of image pairs from the training set and randomly select a time step t from T; add Gaussian noise to the clean image in the set of image pairs based on time step t to perform a forward diffusion process to obtain the noisy image x at time step t t ;

[0062] In this embodiment, a set of image pairs is selected from the training set, assuming that the image pair is a rainy image y 0 and the corresponding clean image x 0 , use the random number function to randomly select a time step t, where the range of t is within the time step sequence length T, and add Gaussian noise ε~N(0,I) to the clean image x at this time step t 0 In the above example, we get the noise image x at this time step. t ,Right now:

[0063]

[0064] In formula (1), α t =1-β t , α t is an empirical constant; β t is the variance schedule, which determines the amount of noise added at time step t and increases linearly from 0.0001 to 0.02 during the forward diffusion process.

[0065] Step 4: Introduce a frequency-dependent strategy into the noise estimation network of the diffusion model to improve the noise estimation network structure and enhance the ability of the noise estimation network to extract noise distribution characteristics. t Input into the improved noise estimation network to perform the reverse diffusion process to extract the noise distribution characteristics and predict a more accurate noise distribution ε θ (x t ,t);

[0066] In this embodiment, the rain image y is converted into 0 and the noisy image x t The channel dimension is concatenated to obtain a six-channel tensor, which is then input into the noise estimation network. At the same time, the time step t is embedded into the noise estimation network using sinusoidal position encoding to extract noise distribution features. In order to optimize the performance of the noise estimation network, the frequency-dependent strategy is introduced into the noise estimation network to predict a more accurate noise distribution ε θ (x t ,t), such as Figure 2 shown.

[0067] The noise estimation network is a U-Net structure with ResNets residual blocks as the main components. The typical feature of the U-Net network architecture is a symmetrical fully convolutional neural network. The role of the noise estimation network is to t It predicts the noise distribution at time step t. It includes six layers of downsampling, an intermediate layer, and six layers of upsampling, and is an encoder-decoder architecture. Each stage of the encoder includes downsampling, and convolution and pooling are used to reduce the spatial size and number of channels of the noise image. The high-frequency features obtained after each layer of downsampling are recorded as h f , where f represents the downsampling of the fth layer. Then, in the decoder, the upsampling layer is used to gradually restore the low-frequency features x corresponding to each layer. f , where f represents the fth layer upsampling, and uses skip connections to connect x f The feature h corresponding to the encoder level f Make stitching.

[0068] However, the features directly transmitted from the earlier layers of the encoder to the decoder through skip connections contain a large amount of high-frequency feature information, which will affect the ability of the noise estimation network to extract noise distribution features, thereby affecting the image deraining ability. Therefore, this embodiment introduces a frequency-dependent strategy to improve the noise estimation network structure. Specifically, two scaling factors b are set. f and f , for x f 、h f These two features are balanced to obtain higher quality derained images.

[0069] First, the noisy image x t The low frequency characteristics x f Adjust and calculate the average feature map along the channel dimension

[0070]

[0071] In formula (2), x f,i Represents the low-frequency feature x f The feature of the ith channel of f The total number of channels in . The scaling factor of the low-frequency feature information is then determined:

[0072]

[0073] In formula (3), a f is a scalar constant. Then the low-frequency feature x f With the scale factor b f Multiply, for low frequency features xf Enhance details.

[0074] As the details of low-frequency features are enhanced, the image becomes smoother and smoother. In order to further alleviate the problem caused by low-frequency features x f The over-smoothing problem caused by this requires high-frequency features h f To alleviate the high-frequency components in f In this implementation, the features in each channel are first converted to the frequency domain through Fourier transform, the original value of the high-frequency component is retained, and the low-frequency component is suppressed, so as to extract the noise distribution features more accurately. The specific operations are as follows:

[0075]

[0076] In formula (4), h f,i Represents high-frequency features h f The characteristics of the ith channel; FFT(·) and IFFT(·) are Fourier transform and inverse Fourier transform respectively; h' f,i is the feature after inverse Fourier transform; ⊙ represents element-wise multiplication; β f,i is a Fourier mask designed as a function of the magnitude of the Fourier coefficients:

[0077]

[0078] In formula (5), s f is the proportional factor related to high frequency features; r thresh represents the radius threshold; r represents the radius after frequency domain conversion.

[0079] Step 5: Set the loss function, iteratively train the improved noise estimation network, and determine whether the stop training condition is met at the end of each training. If not, go to step 3, select the next set of image pairs from the training set, and repeat steps 3-5 for the next training. If yes, the noise estimation network at the end of the current training is used as the trained noise estimation network;

[0080] In order to optimize the performance of the noise estimation network in predicting noise distribution characteristics and make the deraining result close to the corresponding clean image, the loss function L is set 1 To constrain the noise estimation network:

[0081]

[0082] Then, the noise estimation network is iteratively trained, and back propagation is used to update the weights of the noise estimation network during the training process. The variable iteration is set to record the number of iterations of the current noise estimation network training. The variable iteration is updated to iteration+1 after each training. In this embodiment, the preset number of iterations for training is 2,000,000. Determine whether the preset number of iterations is completed and whether the loss function converges. If one of the two conditions is met, stop the training and save the trained noise estimation network model as derain.pth. If both conditions are not met, continue with the next training, that is, select the next set of image pairs, and repeat steps 3-5 until the training is completed.

[0083] Step 6: Randomly select rainy images y from the test set s , s=q+1,q+2,···,n, and input it into the trained noise estimation network model to predict the noise distribution ε of the rainy image θ (x t ,t), according to the predicted noise distribution ε θ (x t ,t) get the preliminary rain image denoising result x st , and then perform zero-value domain decomposition to restore the clean image x s , completing the image deraining reasoning process.

[0084] Step 6.1: Input the rainy image y that needs to be derained s , and sample a value from the Gaussian distribution that matches the rainy image y s Pure Gaussian noise image x of uniform size T It is used as the initial noise of the inference process, and the time step t is equal to the time step sequence length T, which is 1000;

[0085] Step 6.2: The rain image y s , pure Gaussian noise image x T And time step t is input into the noise estimation network trained in step 5 to obtain the predicted noise distribution ε θ (x t ,t), and then according to the predicted noise distribution ε θ (x t ,t), and use formula (7) to obtain the preliminary rain image denoising result x st ;

[0086]

[0087] Step 6.3: In order to further meet the requirements of de-raining image consistency, the initial rain image denoising result x st Perform zero-value domain decomposition, refine zero-domain information, and obtain consistent deraining results

[0088] For any image, it can be decomposed into:

[0089] x=Φ + Φx+(I-Φ + Φ)x (8)

[0090] In formula (8), Φ is a linear matrix, Φ + is the pseudo-inverse matrix of Φ, satisfying Φ + Φ=I. Multiplying both sides of equation (8) by Φ, we can get Φx=ΦΦ + Φx+Φ(I-Φ + Φ)x=Φx+0=y, after the matrix Φ transformation, the first term of formula (8) is equal to Φx, which is called the value range, and the second term becomes 0, which is called the zero domain. This shows that in the image deraining problem, after the degradation operator Φ, the part of image x retained is the value range information, and the zero domain information is lost, and the retained value range part meets the data consistency requirement, that is, ΦΦ + Φx=Φx=y.

[0091] For rainy images taken under real rainfall conditions, the above degradation operator Φ is unknown. In order to find the unknown Φ, we start from the commonly used rain model to solve it:

[0092] R=m·B+H·(1-m) (9)

[0093] In formula (9), B represents the clean image; R represents the degraded rainy image; H represents the global atmospheric light value; and m represents the atmospheric transmittance. According to the dark channel prior principle, that is, some pixels in the image always have at least one color channel with a very low value and the local air is homogeneous, the dark channel calculation is performed on R:

[0094]

[0095] In formula (10), k represents the coordinates of a pixel in each image; Ω k represents the image area centered on k; c represents each channel in the three RGB channels of the image; represents the transmittance of the image area centered on k. According to the dark channel prior principle, the first term in the above formula is 0, and we can get:

[0096]

[0097] For each pixel in the rainy image, we can calculate R c / H c Then we get the value of result. has a unique solution, which is taken as the degenerate operator Φ in the zero-valued range decomposition.

[0098] In order to solve the zero-domain information lost in the rainy image, we use x in step 6.2 st As zero-domain information, the derained image is made closer to the clean image, and a consistent deraining result is obtained.

[0099]

[0100] Step 6.4: Use formula (13) to update the deraining result of the current time step t to obtain the deraining image corresponding to the next time step:

[0101]

[0102] Step 6.5: Update the current time step t to t-1, and determine whether the current time step is 0. If so, the reasoning is complete, and the current x t-1 That is the predicted rain-free image x s ; If not, go to step 6.1.

[0103] In order to further verify the effectiveness of the method of the present invention, an evaluation was performed on the Rain200L dataset. The 1800 image pairs contained in Rain200L were used for training and 200 pairs of images were used for testing. After step 6, the restored rain-free image was obtained, and the PSNR and SSIM were calculated with the paired clean image. In addition, this embodiment compares the performance of the method of the present invention with the existing deep learning-based deraining methods such as PreNet, MSPFN, CycleGAN, Weatherdiff and NLCL, and the experimental results are shown in Table 1. It can be seen from the table that the method of the present invention achieves higher values ​​in both PSNR and SSIM, which proves the effectiveness of the method of the present invention.

[0104] Table 1 Objective evaluation comparison results of different deraining methods on the Rain200L dataset

[0105] Method PSNR↑ SSIM↑ PreNet 34.45 0.969 MSPFN 33.58 0.957 CycleGAN 28.95 0.890 Weatherdiff 30.11 0.817 NLCL 31.74 0.935 Method of the present invention 36.86 0.982

[0106] The present invention also provides an image deraining system based on an improved diffusion model, and its structural block diagram is as follows: Figure 3 The system aims to deploy the derain.pth model trained in the method of the present invention to a mobile device to achieve real-time visualization of deraining results, including: an image collection module, an image preprocessing module, an image reasoning module, and a result processing and display module.

[0107] The image collection module is responsible for collecting rainy images that need to be processed. These images can be existing synthetic rain images, or images of real rainy scenes taken by the camera of the mobile device. The collected synthetic rain images or the taken rainy images are saved to the system album of the mobile device. Then, the image selector provided by the mobile device system is used to select the saved rainy images from the system album of the mobile device.

[0108] The image preprocessing module is responsible for processing the rainy images selected in the image collection module. Its main task is to perform a variety of preprocessing operations on the images to ensure that they meet the input requirements of the model. During the preprocessing process, the image is first scaled to adjust the size of the image to make it compatible with the input layer of the model. The scaling operation ensures that the image has a uniform size before being input into the model, thereby improving the stability and efficiency of the model. Secondly, the image is normalized, that is, the pixel value range of the image is adjusted, the pixel value is reduced to between 0 and 1, and the pixel value distribution of the image is more uniform. Normalization is not only conducive to the accurate extraction of image features by the model, but also can improve the model's adaptability and generalization ability to input images.

[0109] The image reasoning module is responsible for receiving the image processed by the image preprocessing module and performing the reasoning process according to the model deployed on the mobile device to realize the image deraining function. First, the derain.pth model trained in the image deraining method based on the improved diffusion model is selected for model optimization. Secondly, the optimized model is deployed on the mobile device. Finally, the deraining reasoning is performed on the preprocessed image obtained by the image preprocessing module according to the deployed model to obtain the derained image.

[0110] Figure 4It is a model optimization process. In the present invention, the existing NCNN framework is used to optimize the model structure, so as to achieve the effect of model compression and lightweight, and reduce the complexity of the model. The noise estimation network model derain.pth trained in the image deraining method based on the improved diffusion model is selected, and it is converted into derain.onnx in ONNX (Open Neural Network Exchange) format by the torch.onnx tool provided by PyTorch. ONNX has its cross-platform characteristics. Therefore, converting the model to ONNX format can realize cross-platform deployment of the model. However, in some cases, the ONNX model may contain a large number of redundant parameters and complex calculations, which makes the model difficult to understand and optimize. The ONNX model is simplified using the ONNXSIM tool to obtain derain_sim.onnx, which eliminates redundant parameters and calculations in the model by performing a series of graph transformations and constant folding operations on the model. These operations can significantly reduce the complexity and size of the model while maintaining the performance and accuracy of the model. Use ONNX to NCNN conversion tools, such as the onnx2ncnn tool provided by NCNN, to convert the ONNX format derain_sim.onnx file to derain_sim.param and derain_sim.bin. NCNN uses some parameter compression techniques, such as weight pruning and quantization, to reduce the complexity and number of parameters of the model during the model conversion process, and improve the computational and memory efficiency of the model. At the same time, NCNN's optimization for mobile device hardware enables the model to achieve high-performance computing in practical applications.

[0111] In the image deraining inference process, the GPU acceleration technology of mobile devices is used to accelerate the inference of the model and realize the deraining of rainy images. The purpose of using the GPU acceleration technology of mobile devices to accelerate the inference process of the model is to improve the running speed and responsiveness of the model, so that the image deraining task can be performed more smoothly and efficiently on mobile devices. By making full use of the GPU acceleration technology of mobile devices, the inference time can be significantly reduced.

[0112] The result processing and display module performs a denormalization process on the output result of the image reasoning module to map the image back to the original pixel value range. Subsequently, the processed derained image is visualized so that the user can view and compare the image before and after the deraining process in real time. Figure 5 What is shown is a screenshot showing the effect of image deraining on a mobile device. The upper part of the screenshot is an input image with rain, and the lower part of the screenshot is an image after deraining.

[0113] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. Image deraining method based on improved diffusion model, It is characterized in that The method comprises the following steps: Step 1: Prepare a dataset consisting of image pairs and divide it into a training set and a test set; the image pairs consist of rainy images y j , and y j The corresponding clean image x j Composition, where j = 0, 1, 2, (, q, (, n, n is the total number of image pairs in the dataset, q is the total number of image pairs in the training set; Step 2: Set the time step sequence length T of the forward diffusion process of the diffusion model; Step 3: Select a set of image pairs from the training set and randomly select a time step t from T; add Gaussian noise to the clean image in the set of image pairs based on time step t to perform a forward diffusion process to obtain the noisy image x at time step t t ; Step 4: Introduce a frequency-dependent strategy into the noise estimation network of the diffusion model to improve the noise estimation network structure and enhance the ability of the noise estimation network to extract noise distribution characteristics. t The input is sent to the improved noise estimation network to perform the reverse diffusion process to extract the noise distribution characteristics and predict a more accurate noise distribution. Step 5: Set the loss function, iteratively train and improve the noise estimation network, and determine whether the stop training condition is met at the end of each training. If not, go to step 3. If yes, use the noise estimation network at the end of the current training as the trained noise estimation network model; Step 6: Randomly select rainy images y from the test set s , s = q+1,q+2,…,n, and input it into the trained noise estimation network model to predict the noise distribution ε of the rainy image θ (x t ,t), and according to ε θ (x t ,t) get the preliminary rain image denoising result x st , and then perform zero-value domain decomposition to restore the clean image x s , completing the image deraining reasoning process.

2. The image deraining method based on the improved diffusion model according to claim 1, It is characterized in that The frequency-dependent strategy is introduced to improve the noise estimation network structure, specifically setting two scaling factors b f and f , for the noisy image x t The low frequency characteristics x f , high frequency features h f These two characteristics are balanced.

3. The image deraining method based on the improved diffusion model according to claim 2, It is characterized in that Set two scale factors b f and f , for the noisy image x t The low frequency characteristics x f , high frequency features h f The way to balance these two characteristics is: First, the noisy image x t The low frequency characteristics x f Adjust and calculate the average feature map along the channel dimension Among them, x f,i Represents the low-frequency feature x f The feature of the ith channel of f The total number of channels in ; the scaling factor of the low-frequency feature information is then determined: Among them, a f is a scalar constant; then the low-frequency feature x f With the scale factor b f Multiply, for low frequency features x f Perform detail enhancement; As the details of low-frequency features are enhanced, the image becomes smoother and smoother. In order to further alleviate the problem caused by low-frequency features x f The over-smoothing problem caused by this requires high-frequency features h f To alleviate the high-frequency components in the f First, the features in each channel are converted to the frequency domain through Fourier transform, the original value of its high-frequency components is retained, and its low-frequency components are suppressed. The specific operations are as follows: Among them, h f,i Represents high-frequency features h f The characteristics of the ith channel; FFT(·) and IFFT(·) are Fourier transform and inverse Fourier transform respectively; h' f,i is the feature after inverse Fourier transform; ⊙ represents element-wise multiplication; β f,i is a Fourier mask designed as a function of the magnitude of the Fourier coefficients: Among them, s f is related to the high frequency feature h f The relevant scaling factor; r thresh represents the radius threshold; r represents the radius after frequency domain conversion.

4. The image deraining method based on the improved diffusion model according to claim 1, It is characterized in that The training stop condition described in step 5 is to complete the preset number of iterations or the loss function reaches convergence.

5. The image deraining method based on the improved diffusion model according to claim 1, It is characterized in that The step 6 comprises the following steps: Step 6.1: Input the rainy image y that needs to be derained s , and sample a value from the Gaussian distribution that matches the rainy image y s Pure Gaussian noise image x of uniform size T Use it as the initial noise of the reasoning process, and the time step t is equal to the length of the time step sequence T; Step 6.2: The rain image y s , pure Gaussian noise image x T And time step t is input into the trained noise estimation network model to obtain the predicted noise distribution ε θ (x t ,t), and then according to the predicted noise distribution ε θ (x t ,t), and use formula (7) to obtain the preliminary rain image denoising result x st ; Among them, α t =1-β t , α t is an empirical constant, β t Scheduling for variance; Step 6.3: De-noise the initial rainy image x st Perform zero-value domain decomposition, refine zero-domain information, and obtain consistent deraining results Step 6.4: Use formula (13) to update the deraining result of the current time step t to obtain the deraining image corresponding to the next time step t-1: Step 6.5: Update the current time step t to t-1, and determine whether the current time step is 0. If so, the reasoning is complete, and the current x t-1 That is the predicted rain-free image x s ; If not, go to step 6.

1.

6. Image deraining system based on improved diffusion model, It is characterized in that The system is applied to a mobile device, and the system includes: The image collection module is responsible for collecting rainy images that need to be processed; saving the collected rainy images to the system album of the mobile device; using the image selector provided by the mobile device system to select the saved rainy images from the system album of the mobile device; The image preprocessing module is responsible for scaling and normalizing the rainy images selected in the image collection module to ensure that the images have a uniform size and that the pixel values ​​of the images are more evenly distributed; The image inference module is responsible for receiving the preprocessed image processed by the image preprocessing module; deploying the trained noise estimation network model described in claim 1 on the mobile device; performing rain removal inference on the preprocessed image according to the deployed model to obtain a rain removal image; The result processing and display module is responsible for denormalizing the derained image output by the image inference module to map the image back to the original pixel value range; it also visualizes the processed derained image so that users can view and compare the images before and after deraining in real time.

7. The image deraining system based on the improved diffusion model according to claim 6, It is characterized in that In the image inference module, the trained noise estimation network model is first optimized and then deployed on the mobile device; the optimization of the trained noise estimation network model is as follows: first, the trained noise estimation network model is converted from the derain.pth format to the derain.onnx format; Then simplify the derain.onnx format to the derain_sim.onnx format; finally convert the derain_sim.onnx format to the derain_sim.param and derain_sim.bin formats.

Citation Information

Cited By

  • Image denoising method and system based on adversarial frequency mixing technology

    CN119251091A

  • Image rain removal method based on posterior diffusion sampling

    CN119741233A

  • Defogging method based on combination of stochastic differential equation and Gaussian pyramid

    CN119919304A

  • Protein residue motion coordinate prediction method, equipment, medium and program product

    CN120656557A

  • Method, apparatus, medium, and program product for predicting protein residue motion coordinates

    CN120656557B