Image deraining method based on posterior diffusion sampling

Abstract

The application discloses an image rain removal method based on posterior diffusion sampling, and relates to the field of image processing. The method comprises the following steps: inputting a rain-degraded image into an initial rain removal model to obtain a preliminary rain removal result; inputting the preliminary rain removal result into the initial rain removal model and combining the preliminary rain removal result to obtain a rain streak result; adopting an image mask technology to convert the rain streak result into a mask image; and inputting the mask image into a diffusion model to obtain an image rain removal processing result. The application can improve the image rain removal effect, and solve the problems of high training cost, many required training resources, long training time and relatively weak generalization ability when the diffusion model is used for rain removal.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to an image rain removal method based on posterior diffusion sampling. BACKGROUND

[0002] Image rain removal refers to recovering a clear background image from a degraded image after the observed image is affected by rain degradation. This problem involves the estimation of rain streaks and background information. Known rain removal methods start from the properties of rain streaks, and use dictionary learning, sparse representation and other methods to approximate rain streaks to achieve image rain removal. The known rain removal methods have strong universality, but have the disadvantage of poor rain removal effect.

[0003] Recently, diffusion denoising probability models and their continuous form, fractional-based models, have developed rapidly in the field of image processing. With their pre-training framework for various tasks, powerful deep generative priors, interpretable generation process and stable training process, they have quickly generalized a large number of new achievements. As a new emerging generative model, diffusion models have shown great potential in image generation tasks and can generate diverse and high perceptual images. However, the current idea of applying diffusion models to image rain removal mainly focuses on using diffusion models for direct training, using existing data sets to try to guide the generation process of diffusion models by inputting degraded images, and directly generating clear background images. However, this idea of applying diffusion models to rain removal has the problems of high training cost, large required training resources, long training time and relatively weak generalization ability. SUMMARY

[0004] The purpose of the present application is to provide an image rain removal method based on posterior diffusion sampling, which can improve the image rain removal effect while solving the problems of high training cost, large required training resources, long training time and relatively weak generalization ability when applying diffusion models to rain removal.

[0005] To achieve the above-mentioned purpose, the present application provides the following solutions:

[0006] In a first aspect, the present application provides an image rain removal method based on posterior diffusion sampling, comprising:

[0007] inputting a rain-degraded image into an initial rain removal model to obtain a preliminary rain removal result;

[0008] inputting the preliminary rain removal result into the initial rain removal model and combining the preliminary rain removal result to obtain a rain streak result;

[0009] using image mask technology to convert the rain streak result into a mask image;

[0010] inputting the mask image into a diffusion model to obtain an image rain removal processing result.

[0011] Optionally, the initial rain removal model comprises a stepwise optimization residual network model, an interpretable network structure model or a diffusion model.

[0012] Optionally, the preliminary rain removal result is input into the initial rain removal model, and a rain streak result is obtained in combination with the preliminary rain removal result, comprising:

[0013] The preliminary rain removal result is input into the initial rain removal model to obtain a secondary rain removal result.

[0014] The rain streak result is obtained based on the preliminary rain removal result and the secondary rain removal result.

[0015] Optionally, the rain streak result is converted into a mask image by using an image mask technology, comprising:

[0016] The rain streak result is subjected to a binarization operation to obtain a mask.

[0017] The mask image is generated based on the mask.

[0018] Optionally, the mask image is input into a diffusion model to obtain an image rain removal processing result, comprising:

[0019] A plurality of initial noises are sampled from a standard normal distribution, and the diffusion model is used to obtain a predicted noise corresponding to each initial noise based on the initial noise;

[0020] A posterior sampling mean value is obtained based on the predicted noise;

[0021] Diffusion sampling update is performed based on the posterior sampling mean value to obtain a diffusion sampling update result;

[0022] The mask image is corrected based on the mask image, the diffusion sampling update result and the posterior sampling mean value to obtain a correction result;

[0023] A background image is generated based on the correction result, and the background image is taken as the image rain removal processing result.

[0024] Optionally, the correction result is expressed as:

[0025]

[0026] In the formula, x i-1 represents the correction result, x′ i-1 represents the diffusion sampling update result, ζ i represents a step size, x i represents the i-th correction result, represents represents the i-th correction result x iTake the derivative, i = N,..., 3, 2, 1, y represents a mask image, represents the posterior sampling mean, and |||2 represents a two norm.

[0027] According to the specific embodiments provided in the application, the application has the following technical effects:

[0028] The application provides an image rain removal method based on posterior diffusion sampling. The initial rain removal model is used to perform twice rain removal processing on the rain degraded image, which can convert the image rain removal problem into an image repair problem. Then, the rain stripe result obtained based on the twice rain removal result is subjected to mask processing, and then the diffusion model is used to realize repair, which can cover the problematic part in the preliminary rain removal result, thereby improving the image rain removal effect. In addition, by converting the image rain removal problem into an image repair problem and using the diffusion model to solve the repair problem, the diffusion model does not need to be trained by using labeled image samples, which can solve the problems of high training cost, large required training resources, long training time and relatively weak generalization ability when the diffusion model is used for rain removal. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0030] Figure 1 A flowchart of an image rain removal method based on posterior diffusion sampling provided by an embodiment of the application is shown in the figure.

[0031] Figure 2 A mask diagram provided by another embodiment of the application is shown in the figure.

[0032] Figure 3 A twice rain removal result diagram provided by another embodiment of the application is shown in the figure.

[0033] Figure 4 A mask image diagram provided by another embodiment of the application is shown in the figure.

[0034] Figure 5 A comparison diagram of the optimization results of the application and PreNet (Progressive Optimized Recurrent Network) provided by another embodiment of the application is shown in the figure.

[0035] Figure 6 A comparison diagram of the optimization results of the application and RCDNet (Rain Convolutional Dictionary Network) provided by another embodiment of the application is shown in the figure.

[0036] Figure 7 A functional module schematic diagram of an image rain removal system provided by an embodiment of the present application is shown in the figure.

[0037] Figure 8 A structural schematic diagram of a computer device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0039] In order to make the above objectives, characteristics and advantages of the present application more apparent, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0040] The image rain removal method based on posterior diffusion sampling provided by the embodiments of the present application can be executed by a computer device, specifically, can be executed by a terminal or a server, or can be executed by a terminal and a server together. In the embodiments of the present application, the method is taken as an example applied to a server. As shown in the figure, the image rain removal method based on posterior diffusion sampling provided by the present application includes the following steps. Figure 1

[0041] Step 100: inputting a rain-degraded image into an initial rain removal model to obtain a preliminary rain removal result.

[0042] Step 101: inputting the preliminary rain removal result into the initial rain removal model and combining the preliminary rain removal result to obtain a rain streak result. In actual application, the initial rain removal model used in this step can be a step-by-step optimization residual network model, an interpretable network structure model and a diffusion model, etc.

[0043] Step 102: converting the rain streak result into a mask image by using an image mask technology.

[0044] Step 103: inputting the mask image into a diffusion model to obtain an image rain removal processing result.

[0045] In another exemplary embodiment of the present application, in order to improve the universality of the image rain removal method based on posterior diffusion sampling provided by the present application, the initial rain removal model used in the above step 100 can be a traditional rain removal method, for example, a method of removing rain from an image by using a step-by-step optimization residual network, a method of removing rain from an image by using an interpretable network structure, and a method of removing rain from an image by using a diffusion model, etc. ​

[0046] Further, the rain-degraded image input in the present application can be a color image with three channels and a size of 256x256. The pixel value of the color image ranges from 0 to 255. For example, the color image is as shown in Figure 5 (a) of FIG. 1 or Figure 6 (a) of FIG. 2. The rain-degraded image is first input into an initial rain removal model (i.e., a conventional rain removal model), to obtain a preliminary rain removal result. The initial rain removal model selected in the present application adopts an additive composite model assumption, i.e., assuming that the rain-degraded image O is:

[0047] O = B + R.

[0048] In the formula, B is a background image, and R is a rain streak. The preliminary rain removal result obtained by the conventional rain removal model is a predicted background image B.

[0049] The conventional rain removal method herein is not limited and can be arbitrarily selected from conventional rain removal models based on the additive composite model assumption.

[0050] In another exemplary embodiment of the present application, in order to mask the problematic part in the preliminary rain removal result, the present application adopts a secondary rain removal manner to obtain a rain streak result. Based on this, the implementation process of step 101 of the present application can include:

[0051] (1) inputting the preliminary rain removal result into the initial rain removal model to obtain a secondary rain removal result. For example, inputting the obtained preliminary rain removal result (i.e., the background image B) into the initial rain removal model (i.e., the model adopted by the conventional rain removal method) to obtain a secondary rain removal result B', which has:

[0052] B' = Derain(B).

[0053] In the formula, Derain(·) represents the model adopted by the conventional rain removal method.

[0054] The obtained secondary rain removal result can be seen from Figure 3 .

[0055] (2) obtaining a rain streak result based on the preliminary rain removal result and the secondary rain removal result. For example, using the difference between the two rain removal results (i.e., the preliminary rain removal result and the secondary rain removal result) to obtain the rain streak of the secondary rain removal as R:

[0056] R = B - B'.

[0057] In another exemplary embodiment of the present application, the image mask technology can be used to convert the preliminary rain removal result into a mask image to be repaired, which is input into a diffusion module to optimize the preliminary rain removal result. Based on this, the implementation process of step 102 of the present application can be described as:

[0058] (1) Binaryzation operation is performed on the rain streak result to obtain a mask. For example, binaryzation operation is performed on the rain streak R to obtain a mask K:

[0059]

[0060] In the formula, R(x, y) is the pixel gray value of the original image at position (x, y), T is a pre-set threshold value, and K(x, y) is the pixel value of the binaryzation image at position (x, y).

[0061] The obtained mask is as shown in Figure 2 .

[0062] (2) A mask image (as shown in Figure 4 ) is generated based on the mask. For example, the mask image is represented as y:

[0063] y = Kx.

[0064] In the formula, x is in the form of a vector.

[0065] In another exemplary embodiment of the present application, based on the above description, the present application uses a diffusion model to realize optimization of the preliminary rain removal result. Based on this, the implementation process of the above step 103 includes:

[0066] (1) Initial noise is sampled from a standard normal distribution, and a diffusion model is used to obtain corresponding predicted noise based on the initial noise. The initial noise is denoted as M N ~ N(0, I). N(0, I) represents a standard normal distribution. The subscript N of M N represents the total number of iteration steps. A series of hyperparameters and are also manually designed to describe the update result of diffusion sampling. There is a relationship between the hyperparameters: a i : = 1 - b i , The predicted noise is denoted as

[0067]

[0068] In the formula, s θ (M i , i) represents a diffusion model, M i represents the i-th predicted noise, and i = N,..., 3, 2, 1.

[0069] (2) Posterior sampling mean is obtained based on the predicted noise. The posterior sampling mean is denoted as has:

[0070]

[0071] (3) updating the diffusion sampling based on the posterior sampling mean value to obtain a diffusion sampling update result. The diffusion sampling update result is represented as x' i-1 , has:

[0072]

[0073] (4) correcting the mask image based on the mask image, the diffusion sampling update result and the posterior sampling mean value to obtain a correction result. The correction result is represented as:

[0074]

[0075] In the formula, x i-1 represents the correction result, ζ i represents a step size, represents derivation on the i-th correction result x i , and represents a two-norm.

[0076] (5) generating a background image based on the correction result, and taking the background image as an image rain removal processing result.

[0077] The implementation process of the above step 103 is iterative calculation from i=N to i=1, and the final clear background image can be denoted as x0.

[0078] Based on the above description, the image rain removal method based on posterior diffusion sampling provided by the application, under the image inpainting algorithm framework, first removes the problematic part of the rain removal result obtained by the traditional rain removal model using image mask technology, and then uses a diffusion model to repair the masked image to optimize the rain removal result. The application not only has better restoration effect in visual effect, but also shows higher universality and better technical index in solving the image rain removal problem.

[0079] In another exemplary embodiment of the application, two traditional rain removal methods and the image rain removal method based on posterior diffusion sampling provided by the application are selected for comparison to show the rain removal effect. The two traditional rain removal methods used are PreNet based on step-by-step optimization residual network and RCDNet based on dictionary learning. The rain removal effect comparison results are as follows Figure 5 and Figure 6In this embodiment, four quantitative indicators are selected to measure the effect of the method, namely Peak Signal-to-Noise Ratio (PSNR), Structure Similarity IndexMeasure (SSIM), Fréchet Inception Distance (FID) and Learned Perceptual Image Patch Similarity (LPISP). The specific indicator values ​​are shown in Table 1 and Table 2. Among them, Figure 5 Part (a) is the rain-degraded image of the first scene. Figure 5 Part (b) is a schematic diagram of the deraining results of the rain-degraded image of the first scene obtained using PreNet. Figure 5 Part (c) is a schematic diagram of the rain-degraded image of the first scene obtained by using the image rain removal method based on posterior diffusion sampling provided by this application. Figure 5 Part (d) is the original image of the first scene. Figure 6 Part (a) is the rain-degraded image of the second scene. Figure 6 Part (b) is a schematic diagram of the deraining results of the rain-degraded image of the second scene obtained using PreNet. Figure 6 Part (c) is a schematic diagram of the rain-degraded image of the second scene obtained by using the image rain removal method based on posterior diffusion sampling provided by this application. Figure 6 Part (d) is the original image of the second scene.

[0080] Table 1 PreNet refinement quantitative results

[0081] Method PSNR↑ SSIM↑ FID↓ LPISP↓ PreNet 25.82 0.8727 3.24 0.024 The present application 25.91 0.9301 2.98 0.023

[0082] Table 2 RCDNet refined quantitative results

[0083] Method PSNR↑ SSIM↑ FID↓ LPISP↓ RCDNet 26.15 0.9200 5.75 0.051 The present application 26.21 0.9337 5.03 0.040

[0084] Among them, ↑ indicates an increase and ↓ indicates a decrease.

[0085] from Figure 5 and Figure 6 It can also be clearly seen from the comparison that this application successfully removed some rain streaks (shown in the red box) that PreNet and RCDNet failed to remove, and restored the background image. In addition, through comparison, it can be found that the performance of this application surpassed the traditional rain removal method in all four indicators, especially the FID indicator, which showed a significant improvement, proving the effectiveness of this application in improving the visual effect of rain removal results.

[0086] In summary, the present application designs a new post-sampling diffusion model for the image rain removal method based on post-sampling diffusion, uses the technical idea of image inpainting to preprocess the rain removal result, and then uses the post-sampling diffusion method to realize the background image recovery. The effect of image rain removal is improved, the image background details are preserved, and compared with other traditional image rain removal methods based on post-sampling diffusion, the indicators are improved.

[0087] Based on the same inventive concept, the embodiments of the present application also provide an image rain removal system for implementing the above-mentioned image rain removal method based on post-sampling diffusion. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme described in the above method, so the specific limitations in one or more image rain removal system embodiments provided below can refer to the limitations of the image rain removal method based on post-sampling diffusion described above, which will not be repeated here.

[0088] In an exemplary embodiment, as shown in Figure 7 An image rain removal system is provided, including a rain removal module, a mask generation module, and a diffusion module. The mask generation module is connected to the rain removal module. The diffusion module is connected to the mask generation module.

[0089] The rain removal module is configured to input a rain-degraded image into an initial rain removal model to obtain a preliminary rain removal result, and to input the preliminary rain removal result into the initial rain removal model to obtain a rain streak result in combination with the preliminary rain removal result. The mask generation module is configured to convert the rain streak result into a mask image using an image mask technique. The diffusion module is configured to input the mask image into a diffusion model to obtain an image rain removal processing result.

[0090] Based on the above description, the present application provides a post-sampling diffusion framework for image rain removal, which is used to improve the effect of image rain removal algorithm under complex background conditions and improve the visual effect of rain removal result and the application range of rain removal algorithm.

[0091] In an exemplary embodiment, a computer device is provided, which can be a server or a terminal, and its internal structure diagram can be as shown in Figure 8As shown in the figure. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image rain removal data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement an image rain removal method based on posterior diffusion sampling.

[0092] Those skilled in the art can understand that, Figure 8 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement. In one exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in each of the above method embodiments.

[0093] In one exemplary embodiment, a computer readable storage medium is provided, storing a computer program, which is executed by a processor to implement the steps in each of the above method embodiments.

[0094] In one exemplary embodiment, a computer program product is provided, including a computer program, which is executed by a processor to implement the steps in each of the above method embodiments.

[0095] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0096] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc.

[0097] The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0098] The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.

[0099] The principles and implementation modes of the present application are described by applying specific examples in the present application. The above embodiment description is only used to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed. In conclusion, the content of the present application should not be understood as a limitation.

Claims

1. An image deraining method based on posterior diffusion sampling, characterized in that: The image deraining method based on posterior diffusion sampling includes: Input the rain-degraded image into the initial deraining model to obtain the preliminary deraining result; Inputting the preliminary rain removal result into the initial rain removal model, and combining the preliminary rain removal result to obtain a rain streak result; Using image masking technology, the rain streak result is converted into a mask image; The mask image is input into a diffusion model to obtain an image deraining processing result, including: sampling multiple initial noises from a standard normal distribution, and using the diffusion model to obtain a predicted noise corresponding to each initial noise based on the initial noise; obtaining a posterior sampling mean based on the predicted noise; performing a diffusion sampling update based on the posterior sampling mean to obtain a diffusion sampling update result; correcting the mask image based on the mask image, the diffusion sampling update result, and the posterior sampling mean to obtain a corrected result; generating a background image based on the corrected result, and using the background image as the image deraining processing result.

2. The image deraining method based on posterior diffusion sampling according to claim 1, characterized in that: The initial rain removal model includes: gradually optimizing a residual network model or an interpretable network structure model.

3. The image deraining method based on posterior diffusion sampling according to claim 1, characterized in that: Inputting the preliminary rain removal result into the initial rain removal model, and combining the preliminary rain removal result to obtain a rain streak result, including: Inputting the preliminary rain removal result into the initial rain removal model to obtain a secondary rain removal result; The rain streak result is obtained based on the preliminary rain removal result and the secondary rain removal result.

4. The image deraining method based on posterior diffusion sampling according to claim 1, characterized in that: The rain streak result is converted into a mask image using image masking technology, including: Performing a binarization operation on the rain streak result to obtain a mask; The mask image is generated based on the mask.

5. The image deraining method based on posterior diffusion sampling according to claim 1, characterized in that: The correction result is expressed as: Where x i-1 represents the correction result, x′ i-1 represents the diffusion sampling update result, ζ i Indicates the step size, x i represents the i-th correction result, Indicates the i-th correction result x i Perform derivation, i=N,...,3,2,1, y represents the mask image, represents the posterior sampling mean, and ||||2 represents the two-norm.

Images