Image rain removal method based on posterior diffusion sampling

Abstract

The invention discloses an image rain removal method based on posterior diffusion sampling, and relates to the field of image processing, and the method comprises the steps: inputting a rain degradation image into an initial rain removal model, and obtaining a preliminary rain removal result; inputting the initial rain removal result into the initial rain removal model, and combining the initial rain removal result to obtain a rain stripe result; converting the rain stripe result into a mask image by adopting an image mask technology; and inputting the mask image into the diffusion model to obtain an image rain removal processing result. According to the method, the image rain removal effect can be improved, and meanwhile, the problems of high training cost, more required training resources, long training time, relatively weak generalization ability and the like when a diffusion model is used for rain removal are solved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image deraining method based on posterior diffusion sampling. Background Art

[0002] Image deraining refers to restoring a clear background image from a degraded image after the observed image is degraded by rainfall. This problem involves the estimation of rain streaks and background information. Known deraining methods start from the properties of rain streaks and use dictionary learning, sparse representation and other methods to approximate rain streaks to achieve image deraining. Known deraining methods are highly versatile, but have the disadvantage of poor deraining effect.

[0003] Recently, diffusion denoising probabilistic models and their continuous form of score-based models have flourished in the field of image processing. With their pre-training framework that can be used for multiple tasks, powerful deep generative priors, interpretable generation process, and stable training process, they have rapidly generalized a large number of new results. As an emerging generative model, the diffusion model has shown great potential in image generation tasks and can generate diverse and highly perceptual images. However, the current idea of ​​applying the diffusion model to image deraining mainly focuses on using the diffusion model for direct training. Using existing datasets, attempts are made to directly generate clear background images by guiding the generation process of the diffusion model through inputting degraded images. However, this idea of ​​applying the diffusion model to deraining has problems such as high training cost, large number of training resources required, long training time, and relatively weak generalization ability. Summary of the invention

[0004] The purpose of this application is to provide an image deraining method based on posterior diffusion sampling, which can improve the image deraining effect while solving the problems of high training cost, large number of training resources required, long training time and relatively weak generalization ability when applying diffusion model for deraining.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides an image deraining method based on posterior diffusion sampling, comprising:

[0007] Input the rain-degraded image into the initial deraining model to obtain the preliminary deraining result;

[0008] Inputting the preliminary rain removal result into the initial rain removal model, and combining the preliminary rain removal result to obtain a rain streak result;

[0009] Using image masking technology, the rain streak result is converted into a mask image;

[0010] The mask image is input into a diffusion model to obtain an image deraining processing result.

[0011] Optionally, the initial rain removal model includes: a step-by-step optimization residual network model, an interpretable network structure model or a diffusion model.

[0012] Optionally, inputting the preliminary rain removal result into the initial rain removal model, and combining the preliminary rain removal result to obtain a rain streak result, including:

[0013] Inputting the preliminary rain removal result into the initial rain removal model to obtain a secondary rain removal result;

[0014] The rain streak result is obtained based on the preliminary rain removal result and the secondary rain removal result.

[0015] Optionally, an image masking technique is used to convert the rain streak result into a mask image, including:

[0016] Performing a binarization operation on the rain streak result to obtain a mask;

[0017] The mask image is generated based on the mask.

[0018] Optionally, the mask image is input into a diffusion model to obtain an image deraining processing result, including:

[0019] Sampling a plurality of initial noises from a standard normal distribution, and using the diffusion model to obtain a predicted noise corresponding to each of the initial noises;

[0020] Obtaining a posterior sampling mean based on the predicted noise;

[0021] Perform diffusion sampling update based on the posterior sampling mean to obtain a diffusion sampling update result;

[0022] Correcting the mask image based on the mask image, the diffusion sampling update result and the posterior sampling mean to obtain a correction result;

[0023] A background image is generated based on the correction result, and the background image is used as the image deraining processing result.

[0024] Optionally, the correction result is expressed as:

[0025]

[0026] In the formula, x i-1 represents the correction result, x i ' -1 represents the diffusion sampling update result, ζ i represents the step length, express Represents the initial noise x for the i-th i Take the derivative, xi ,i=N,...,3,2,1 represents the i-th initial noise, y represents the mask image, represents the posterior sampling mean, and ||||2 represents the second norm.

[0027] According to the specific embodiments provided in this application, this application has the following technical effects:

[0028] The present application provides an image deraining method based on posterior diffusion sampling, which uses an initial deraining model to perform two deraining processes on a rain-degraded image, and can transform the image deraining problem into an image restoration problem. Then, by masking the rain streak results obtained based on the two deraining results, and then using the diffusion model to restore them, the problematic parts of the initial deraining results can be covered up, thereby improving the image deraining effect. In addition, by transforming the image deraining problem into an image restoration problem and using a diffusion model to solve this restoration problem, there is no need to use labeled image samples to train the diffusion model, and can solve the problems of high training cost, large number of required training resources, long training time, and relatively weak generalization ability when applying the diffusion model for deraining. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 A schematic diagram of a process of an image rain removal method based on posterior diffusion sampling provided in one embodiment of the present application;

[0031] Figure 2 A mask schematic diagram provided for another embodiment of the present application;

[0032] Figure 3 A schematic diagram of two rain removal results provided in another embodiment of the present application;

[0033] Figure 4 A schematic diagram of a mask image provided by another embodiment of the present application;

[0034] Figure 5 A schematic diagram comparing the optimization results of the present application and PreNet (progressive optimization recurrent network) provided in another embodiment of the present application;

[0035] Figure 6A schematic diagram comparing the optimization results of the present application and RCDNet (an interpretable network structure, rainconvolutional dictionary network) provided in another embodiment of the present application;

[0036] Figure 7 A schematic diagram of the functional modules of an image rain removal system provided in one embodiment of the present application;

[0037] Figure 8 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0039] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0040] The image deraining method based on posterior diffusion sampling provided in the embodiment of the present application can be executed by a computer device, specifically, it can be executed by a computer device such as a terminal or a server alone, or it can be executed by a terminal and a server together. In the embodiment of the present application, the method is applied to a server as an example for explanation. Figure 1 As shown, the image deraining method based on posterior diffusion sampling provided by the present application includes:

[0041] Step 100: Input the rain-degraded image into the initial deraining model to obtain a preliminary deraining result.

[0042] Step 101: Input the preliminary rain removal result into the initial rain removal model, and combine the preliminary rain removal result to obtain the rain streak result. In actual application, the initial rain removal model used in this step can be a stepwise optimized residual network model, an interpretable network structure model, a diffusion model, etc.

[0043] Step 102: Using image masking technology, the rain streak result is converted into a mask image.

[0044] Step 103: input the mask image into the diffusion model to obtain the image deraining processing result.

[0045] In another exemplary embodiment of the present application, in order to improve the versatility of the image deraining method based on posterior diffusion sampling provided by the present application, the initial deraining model used in the above step 100 can be a traditional deraining method, for example: a method for image deraining using a stepwise optimized residual network, a method for image deraining using an interpretable network structure, and a method for image deraining using a diffusion model.

[0046] Furthermore, the rain-degraded image input in the present application may be a three-channel color image with a size of 256×256. The pixel value range of this color image is (0, 255). For example, this color image is Figure 5 Part (a) or Figure 6 As shown in part (a) of . The rain-degraded image will first be input into the initial rain removal model (i.e., the existing traditional rain removal model) to obtain a preliminary rain removal result. The initial rain removal model selected in this application must adopt the additive composite model assumption, that is, assuming that the rain-degraded image O is:

[0047] O=B+R.

[0048] Where B is the background image and R is the rain streak. The initial rain removal result obtained by the traditional rain removal model is the predicted background image B.

[0049] The traditional rain removal method here is not limited, and any traditional rain removal model based on the additive composite model assumption can be selected for implementation.

[0050] In another exemplary embodiment of the present application, in order to cover up the problematic part in the preliminary rain removal result, the present application adopts a secondary rain removal method to obtain the rain streak result. Based on this, the implementation process of the above step 101 of the present application may include:

[0051] (1) Input the initial rain removal result into the initial rain removal model to obtain the secondary rain removal result. For example, input the initial rain removal result (i.e., the background image B) into the initial rain removal model (i.e., the model used by the traditional rain removal method) to obtain the secondary rain removal result B', which is:

[0052] B' = Derain(B).

[0053] Where Derain(·) represents the model used by traditional rain removal methods.

[0054] The secondary deraining results can be found in Figure 3 shown.

[0055] (2) Based on the initial rain removal result and the secondary rain removal result, the rain streak result is obtained. For example, the rain streak of the secondary rain removal is obtained by taking the difference between the two rain removal results (i.e., the initial rain removal result and the secondary rain removal result) as R:

[0056] R=B-B'.

[0057] In another exemplary embodiment of the present application, the image masking technology can be used to transform the preliminary rain removal result into a mask image to be repaired and input into the diffusion module to optimize the preliminary rain removal result. Based on this, the implementation process of the above step 102 of the present application can be described as:

[0058] (1) Binarize the rain streak result to obtain a mask. For example, binarize the rain streak R to obtain a mask K:

[0059]

[0060] Where R(x,y) is the pixel grayscale value of the original image at position (x,y), T is the preset threshold, and K(x,y) is the pixel value of the binarized image at position (x,y).

[0061] The mask obtained is as follows Figure 2 shown.

[0062] (2) Generate a mask image based on the mask (such as Figure 4 For example, the mask image is represented as y:

[0063] y=Kx.

[0064] Where x is the vector form of B.

[0065] In another exemplary embodiment of the present application, based on the above description, the present application adopts a diffusion model to optimize the preliminary rain removal result. Based on this, the implementation process of the above step 103 includes:

[0066] (1) Sample the initial noise from the standard normal distribution and use the diffusion model to obtain the corresponding predicted noise based on the initial noise. The initial noise is denoted by x N ~N(0,I). N(0,I) represents the standard normal distribution. x N The subscript N represents the total number of iterations. A series of hyperparameters need to be manually designed. and It is used to describe the diffusion sampling update results. The hyperparameters are related by: i :=1-β i , The prediction noise is expressed as

[0067]

[0068] In the formula, s θ (x i ,i) represents the diffusion model, x i,i=N,...,3,2,1 represents the i-th prediction noise.

[0069] (2) Obtain the posterior sampling mean based on the predicted noise. The posterior sampling mean is denoted as have:

[0070]

[0071] (3) Perform diffusion sampling update based on the posterior sampling mean to obtain the diffusion sampling update result. The diffusion sampling update result is represented by x i ' -1 ,have:

[0072]

[0073] (4) Based on the mask image, the diffusion sampling update result and the posterior sampling mean, the mask image is corrected to obtain a corrected result. The corrected result is expressed as:

[0074]

[0075] In the formula, x i-1 represents the correction result, ζ i represents the step length, Represents the update result x for the i-th i To conduct derivation, represents the two-norm.

[0076] (5) Generate a background image based on the correction result, and use the background image as the image deraining result.

[0077] The implementation process of the above step 103 is to perform iterative calculation from i=N to i=1, and the clear background image finally obtained can be recorded as x0.

[0078] Based on the above description, the image deraining method based on posterior diffusion sampling provided by the present application first uses the image masking technology to remove the problematic parts of the deraining results obtained by the traditional deraining model under the framework of the image restoration algorithm, and then uses the diffusion model to repair the masked image to achieve the optimization of the deraining results. The present application not only has a better restoration effect in visual effect, but also shows higher versatility and better technical indicators in solving the image deraining problem.

[0079] In another exemplary embodiment of the present application, two traditional rain removal methods are selected for comparison with the image rain removal method based on posterior diffusion sampling provided by the present application to demonstrate the rain removal effect. The two traditional rain removal methods used are: PreNet based on stepwise optimization residual network and RCDNet based on dictionary learning. The comparison results of rain removal effect are shown in Figure 2. Figure 5 and Figure 6 In this embodiment, four quantitative indicators are selected to measure the effect of the method, namely Peak Signal-to-Noise Ratio (PSNR), Structure Similarity Index Measure (SSIM), Fréchet Inception Distance (FID) and Learned Perceptual Image Patch Similarity (LPISP). The specific indicator values ​​are shown in Tables 1 and 2. Among them, Figure 5 Part (a) is the rain-degraded image of the first scene. Figure 5 Part (b) is a schematic diagram of the deraining result of the rain-degraded image of the first scene obtained by using PreNet. Figure 5 Part (c) is a schematic diagram of a rain-degraded image of the first scene obtained by using the image rain removal method based on posterior diffusion sampling provided by the present application. Figure 5 Part (d) is the original image of the first scene. Figure 6 Part (a) is the rain-degraded image of the second scene. Figure 6 Part (b) is a schematic diagram of the deraining result of the rain-degraded image of the second scene obtained by using PreNet. Figure 6 Part (c) is a schematic diagram of a rain-degraded image of a second scene obtained by using the image rain removal method based on posterior diffusion sampling provided by the present application. Figure 6 Part (d) is the original image of the second scene.

[0080] Table 1 PreNet refinement quantitative results

[0081] method PSNR↑ SSIM↑ FID↓ LPISP↓ PreNet 25.82 0.8727 3.24 0.024 This application 25.91 0.9301 2.98 0.023

[0082] Table 2 RCDNet refined quantitative results

[0083] method PSNR↑ SSIM↑ FID↓ LPISP↓ RCDNet 26.15 0.9200 5.75 0.051 This application 26.21 0.9337 5.03 0.040

[0084] Among them, ↑ indicates an increase, and ↓ indicates a decrease.

[0085] from Figure 5 and Figure 6 It can also be clearly seen from the comparison that the present application successfully removed some rain streaks (shown in the red box) that PreNet and RCDNet failed to remove, and restored the background image. In addition, through comparison, it can be found that the performance of the present application in all four indicators has surpassed the traditional rain removal method, especially in the FID indicator, which proves the effectiveness of the present application in improving the visual effect of the rain removal result.

[0086] In summary, this application designs a new posterior sampling diffusion model for image deraining methods based on posterior diffusion sampling, uses the technical idea of ​​image restoration to preprocess the deraining results, and then uses the posterior diffusion sampling method to restore the background image. It not only improves the image deraining effect and retains the image background details, but also has improvements in various indicators compared to other traditional image deraining methods based on posterior diffusion sampling.

[0087] Based on the same inventive concept, the embodiment of the present application also provides an image deraining system for implementing the above-mentioned image deraining method based on posterior diffusion sampling. The implementation solution provided by the system to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image deraining system embodiments provided below can refer to the limitations of the image deraining method based on posterior diffusion sampling above, and will not be repeated here.

[0088] In an exemplary embodiment, Figure 7 As shown, an image deraining system is provided, comprising: a deraining module, a mask generating module and a diffusion module. The mask generating module is connected to the deraining module. The diffusion module is connected to the mask generating module.

[0089] The rain removal module is used to input the rain-degraded image into the initial rain removal model to obtain the initial rain removal result, and to input the initial rain removal result into the initial rain removal model to obtain the rain streak result in combination with the initial rain removal result. The mask generation module is used to convert the rain streak result into a mask image using image mask technology. The diffusion module is used to input the mask image into the diffusion model to obtain the image rain removal processing result.

[0090] Based on the above description, the present application provides a posterior sampling diffusion framework for image deraining, which is used to improve the effect of image deraining algorithm under complex background conditions, improve the visual effect of deraining results and the application scope of deraining algorithm.

[0091] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 8As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image deraining data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image deraining method based on posterior diffusion sampling is implemented.

[0092] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0093] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0094] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0095] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0096] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0097] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0098] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0099] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. An image deraining method based on posterior diffusion sampling, characterized in that: The image deraining method based on posterior diffusion sampling comprises: Input the rain-degraded image into the initial deraining model to obtain the preliminary deraining result; Inputting the preliminary rain removal result into the initial rain removal model, and combining the preliminary rain removal result to obtain a rain streak result; Using image masking technology, the rain streak result is converted into a mask image; The mask image is input into a diffusion model to obtain an image deraining processing result.

2. The image deraining method based on posterior diffusion sampling according to claim 1, characterized in that: The initial rain removal model includes: gradually optimizing a residual network model or an interpretable network structure model.

3. The image deraining method based on posterior diffusion sampling according to claim 1, characterized in that: Inputting the preliminary rain removal result into the initial rain removal model, and combining the preliminary rain removal result to obtain a rain streak result, including: Inputting the preliminary rain removal result into the initial rain removal model to obtain a secondary rain removal result; The rain streak result is obtained based on the preliminary rain removal result and the secondary rain removal result.

4. The image deraining method based on posterior diffusion sampling according to claim 1, characterized in that: The rain streak result is converted into a mask image by using an image masking technique, including: Performing a binarization operation on the rain streak result to obtain a mask; The mask image is generated based on the mask.

5. The image deraining method based on posterior diffusion sampling according to claim 1, characterized in that: The mask image is input into the diffusion model to obtain the image deraining processing result, including: Sampling a plurality of initial noises from a standard normal distribution, and using the diffusion model to obtain a predicted noise corresponding to each of the initial noises; Obtaining a posterior sampling mean based on the predicted noise; Perform diffusion sampling update based on the posterior sampling mean to obtain a diffusion sampling update result; Correcting the mask image based on the mask image, the diffusion sampling update result and the posterior sampling mean to obtain a correction result; A background image is generated based on the correction result, and the background image is used as the image deraining processing result.

6. The image deraining method based on posterior diffusion sampling according to claim 5, characterized in that: The correction result is expressed as: In the formula, x i-1 represents the correction result, x i ' -1 represents the diffusion sampling update result, ζ i represents the step length, express Represents the i-th initial noise x i Take the derivative, x i ,i=N,...,3,2,1 represents the i-th initial noise, y represents the mask image, represents the posterior sampling mean, and ||||2 represents the second norm.

Images