Unified raindrop and reflection removal method based on diffusion model
By employing a diffusion-based image processing method, utilizing variational autoencoders and cross-attention modulation techniques, combined with a structure-fidelity feature encoder, the problem of removing raindrops and reflections in images was solved, generating high-quality, clear images and improving the accuracy of computer vision tasks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2026-01-30
- Publication Date
- 2026-05-15
AI Technical Summary
Existing technologies are not ideal for processing raindrops and reflections in images, and it is difficult to balance removal effectiveness with image quality preservation. Especially in complex scenes, they are prone to false detection, false negatives, or over-processing, which affects the accuracy of advanced computer vision tasks.
A diffusion-based approach is adopted, which performs initial repair through a repair network, uses a variational autoencoder to encode image features, combines cross-attention and latent variable modulation, adaptively allocates spatial weights, and introduces a structure-fidelity feature encoder for image reconstruction, gradually restoring a clear image.
It achieves efficient removal of raindrops and reflections in complex scenes, and the generated images are consistent with real images in terms of detail and structure, thus improving the accuracy of advanced computer vision tasks.
Smart Images
Figure CN122048730A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology, specifically relating to a unified method for removing raindrops and reflections based on a diffusion model. Background Technology
[0002] When capturing images through glass or windshields, raindrops and reflections are two common types of visual interference. When raindrops adhere to the windshield or the lens of a camera, their curved liquid surface often refracts and reflects light, causing the raindrop area in the image to exhibit significantly different brightness, color, and sharpness characteristics compared to the background area. This results in localized blurring, distortion, and uneven brightness in the image. Simultaneously, ambient light from the shooting side creates reflections on the glass surface, further reducing image sharpness and visibility. This degradation not only affects human visual perception but also severely interferes with subsequent computer vision tasks such as object detection, scene understanding, and autonomous driving perception.
[0003] Raindrops and reflections can significantly impact image quality, posing a range of challenges for advanced computer vision tasks. This is particularly true in applications such as traffic monitoring, driver assistance systems, and outdoor security, where the need for clear images is paramount. Image degradation caused by raindrop occlusion can severely affect the accuracy of object detection, object recognition, and other advanced computer vision tasks. Therefore, to acquire high-quality, raindrop-free images under adverse conditions like rain, it is crucial to design an efficient raindrop removal algorithm to restore image sharpness and detail, thus meeting the high-quality input requirements of advanced computer vision tasks.
[0004] Early methods for handling raindrops and reflections in images mostly focused on raindrop detection rather than direct removal. Furthermore, some methods were primarily designed for raindrop or reflection removal within video sequences, making them difficult to apply directly to single-image scenes. For raindrops, traditional methods are largely based on image processing techniques, typically utilizing features such as shape, color, or texture, employing methods like filtering, edge detection, or image segmentation to identify and attempt to remove raindrops. For reflections, some traditional methods emphasize analyzing the optical characteristics of reflective areas, such as their high brightness and low color saturation, using histogram equalization and thresholding to locate reflective areas before attempting repair.
[0005] However, these methods perform poorly in complex scenes. When raindrops are irregularly shaped, or when they are highly similar to the background in color and texture, algorithms targeting raindrops are prone to false positives or false negatives. Similarly, in terms of reflection processing, if the reflective area is similar in color and texture to surrounding objects, or if there are multiple light sources in the scene causing complex and variable reflection conditions, traditional methods struggle to accurately identify reflective boundaries and effectively remove reflections, often resulting in incomplete removal or overprocessing that leads to loss of image information.
[0006] With the leapfrog development of deep learning technology in high-order computer vision tasks such as image classification and object detection, generative methods based on diffusion models are bringing revolutionary breakthroughs to the fields of raindrop removal and reflection restoration. Compared with traditional optimization algorithms based on physical priors and restoration schemes relying on discriminative networks, diffusion models, through a progressive denoising generation mechanism, demonstrate stronger modeling capabilities for complex degraded scenes. In particular, when dealing with complex degradation problems such as color space distortion and high-frequency texture breaks caused by raindrops, as well as local information occlusion and global color shift caused by reflections, their generation quality and generalization performance are significantly better than existing methods. Summary of the Invention
[0007] The purpose of this invention is to solve the problems of unsatisfactory raindrop and reflection removal effects in existing technologies, and the difficulty in balancing removal effect with image quality preservation. This invention proposes an image restoration method based on a diffusion model for unified removal of raindrops and reflections.
[0008] The objective of this invention is achieved through the following technical solution: a unified method for removing raindrops and reflections based on a diffusion model, the method comprising: Receive low-quality images and use a restoration network to remove the degradation of raindrops and reflections to obtain an initial restoration result; The low-quality image and the initial restoration result are encoded as latent representations as conditional latent variables. A noisy latent variable is generated at each diffusion time step. The conditional latent variables are modulated using the noisy latent variable. Spatial weights are assigned to each conditional latent variable after modulation. The processed features are input into the diffusion model to denoise the noisy latent variables. The latent variables are input into the structure-fidelity feature encoder. The reconstruction of the latent variables is guided by extracting the structured features of the low-quality image and the initial result, and the final restoration result is obtained.
[0009] Furthermore, the step of encoding the low-quality image and the initial restoration result into a latent representation specifically involves: using a pre-trained variational autoencoder as a latent space mapping tool to encode the low-quality image and the initial restoration result, thereby obtaining the conditional latent variables of the low-quality image and the initial restoration result, respectively.
[0010] Furthermore, the modulation of the conditional latent variable using the noise latent variable specifically involves: firstly, extracting the feature vectors of the noise latent variable and the conditional latent variable through a convolutional layer; then concatenating the two feature vectors; performing cross-attention operation using a Transformer network to obtain an interaction feature with the same feature dimension as the noise latent variable; combining the interaction feature with the noise latent variable feature vector; and then reducing the dimensionality through a convolutional layer to obtain the modulated conditional variable.
[0011] Furthermore, the allocation of spatial weights based on each conditional latent variable specifically involves: The modulated low-quality image is concatenated with the conditional latent variables of the initial restoration result and input into the spatial attention module to generate a spatial weight matrix. The spatial attention module includes two convolutional layers, one activation layer and one sigmoid layer, which are used to calculate the weight distribution at different spatial locations and adjust the conditional latent variables of the low-quality image and the initial restoration result based on the weight distribution.
[0012] Furthermore, the step of inputting the processed features into the diffusion model to denoise the noisy latent variables specifically involves: inputting the conditional information and the noisy latent variables into the control branch, receiving the conditional latent variables and the noisy latent variables and generating a control signal, and injecting the control signal into the main denoising U-Net network through zero convolution to guide the denoising process. The main denoising U-Net estimates and denoises the noisy latent variables at each time step, thereby gradually recovering the clean latent variables.
[0013] Furthermore, the control branch is a trainable copy of the pre-trained U-Net encoder and intermediate layers.
[0014] Furthermore, the training process of the structure-fidelity feature encoder includes: The low-quality image is gated and fused with the initial restoration result to obtain fidelity features. These fidelity features are then added to the corresponding positions in the VAE decoder using zero convolution to provide fidelity constraints during the decoding process. Real images The input is fed into a pre-trained VAE encoder, which encodes it into the latent space to obtain compressed latent variables. This is used to simulate the latent representation after denoising; then, guided by the fidelity features, the pre-trained VAE decoder converts the latent variables... Convert to reconstructed image ; During the training phase, by minimizing the reconstructed image With real images The mean absolute error between the two is used to optimize the structure-fidelity feature encoder, thereby ensuring that the generated image has higher fidelity at the texture and structure levels.
[0015] According to another aspect of the specification, a unified raindrop and reflection removal device based on a diffusion model is also provided, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the aforementioned unified raindrop and reflection removal method based on a diffusion model.
[0016] According to another aspect of the specification, a computer-readable storage medium is also provided, on which a program is stored, which, when executed by a processor, implements the aforementioned method for unified removal of raindrops and reflections based on a diffusion model.
[0017] According to another aspect of the specification, a computer program product is also provided, including a computer program that, when executed by a processor, implements the aforementioned method for unified removal of raindrops and reflections based on a diffusion model.
[0018] The beneficial effects of this invention are: This invention uses cross-attention-driven feature modulation, which modulates conditional latent variables using noise latent variables to achieve information interaction between conditional features and time-step related noise. By using gated fusion of latent variables, spatial weights are adaptively assigned to each conditional latent variable based on the feature importance of different spatial regions, thereby selectively enhancing useful information and suppressing redundant features.
[0019] A structure-fidelity feature encoder is introduced, which extracts the structured features of low-quality images and initial results, and guides the reconstruction of latent variables during the decoding process, so that the generated image maintains structural consistency with the real image while restoring details. Attached Figure Description
[0020] Figure 1 This is a schematic diagram of a two-stage image restoration method based on a diffusion model according to an embodiment of the present invention.
[0021] Figure 2 This is a schematic diagram of the latent feature modulation and gating module according to an embodiment of the present invention.
[0022] Figure 3 This is a schematic diagram of a structure-fidelity feature encoder according to an embodiment of the present invention.
[0023] Figure 4 This is a schematic diagram of a raindrop and reflection unified removal device based on a diffusion model according to an embodiment of the present invention. Detailed Implementation
[0024] Embodiments of the present invention will now be described in detail with reference to the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as limiting the present invention.
[0025] Figure 1 This invention is based on a unified method for removing raindrops and reflections using a diffusion model. This example includes the following two stages: Basic degradation suppression stage: In this stage, the method receives the input low-quality image. The image is then degraded using a restoration network. The goal of this stage is to remove interfering information such as raindrops and reflections, while avoiding the introduction of new artifacts into the image, thereby generating a highly reliable initial restoration result. . in This indicates the use of a repair network. In its specific implementation, this invention preferably employs the DRSformer network as the repair model (RM), which exhibits strong generalization ability and reconstruction performance in handling complex degradations such as raindrops and reflections. The resulting results... It will serve as the conditional input for the multi-condition driven diffusion generation stage.
[0026] Multi-condition driven diffusion generation stage: In this stage, the initial results processed in the first stage are further optimized based on the diffusion generation model to recover severely degraded or missing regions in the low-quality image. Specifically, this stage uses a pre-trained variational autoencoder (VAE) as a latent space mapping tool to transform the low-quality image... The initial result of the first stage output The encoding is a latent representation. This latent representation avoids the inefficiency and training instability caused by direct computation in pixel space.
[0027] Specifically, low-quality images Compared with the initial repair results Conditional latent variables are obtained by passing them through the VAE encoder. and ,in: The aforementioned conditional latent variables correspond to the feature information of the low-quality image and the initial restoration result, respectively. Furthermore, this stage generates a noise latent variable at each diffusion time step. This enhances the detail and realism of the generated images.
[0028] To effectively integrate conditional information and latent noise variables, the system sets up control branches. The control branch is a trainable copy of the pre-trained U-Net encoder and intermediate layers, used to receive conditional latent variables and noise latent variables and generate control signals. These control signals are injected into the main denoising U-Net network via zero-convolution to guide the denoising process.
[0029] Unlike the simple addition or splicing methods in existing technologies, this embodiment sets up a latent feature modulation and gating module before entering the control branch. This module consists of a cross-attention driven feature modulation module and a latent variable gating fusion module: The cross-attention-driven feature modulation module modulates the conditional latent variables using noisy latent variables to achieve information interaction between conditional features and time-step correlated noise. The latent variable gating fusion module adaptively assigns spatial weights to each conditional latent variable according to the feature importance of different spatial regions, thereby selectively enhancing useful information and suppressing redundant features.
[0030] The latent features processed by the modulation and gating modules described above are then input to the control branch. The control branch is a trainable copy of the pre-trained U-Net, capable of generating time-step-related control signals, which are injected into the main denoising U-Net via zero-convolution. The main denoising U-Net estimates and denoises the noisy latent variables at each time step, thereby gradually recovering the clean latent variables. .
[0031] After completing the diffusion denoising process, the clean latent variable The input is fed into a pre-trained VAE decoder to obtain the reconstructed image. To further improve the structural fidelity of the result, a structural fidelity feature encoder is introduced in this stage. It extracts the structural features of the low-quality image and the initial result, and guides the reconstruction of latent variables during the decoding process, so that the generated image maintains structural consistency with the real image while restoring details.
[0032] The structure-fidelity feature encoder, during the inference phase, will convert low-quality images... Compared with the initial reconstruction results The input is fed into the gated fusion module, which selectively fuses multi-source information to extract representative, high-fidelity features. Simultaneously, clean latent variables are obtained through diffusion denoising. As the master latent representation in the decoding stage, it carries global semantic and texture distribution information. The extracted fidelity features do not directly replace or destroy it. Instead of using a distributed structure, it injects residual data into the corresponding positions of the VAE decoder via zero convolution. The reconstruction process provides fidelity constraints.
[0033] Specifically, Figure 2 The latent feature modulation and gating module of the present invention includes the following: Cross-attention driven feature modulation module: This module uses noise latent variables to modulate conditional latent variables, realizing information interaction between conditional features and time-step correlated noise.
[0034] Input by conditions For example, first input the conditions. With noise latent variables The inputs are fed into the convolutional layer to extract their corresponding feature vectors: in, , ,and This indicates the number of feature channels, which is set to 32 in this embodiment.
[0035] Subsequently, the above features and The concatenation is performed and fed into two consecutive Transformer layers to perform cross-attention operations: in, This indicates a cross-attention operation. This indicates a concatenation operation. This cross-attention mechanism enables information exchange between different features and displays the interaction results in the output stage. Dimensions and Alignment.
[0036] Furthermore, interactive features With features The channels are summed, and then another convolutional layer is used to reduce the dimensionality of the channels, resulting in the modulated conditional latent variables: For low-quality input conditions By using the above process Replace with This allows us to obtain the modulated low-quality conditional latent variables. .
[0037] The gated fusion module for latent variables: This module adaptively assigns spatial weights to each conditional latent variable based on the feature importance of different spatial regions, thereby selectively enhancing useful information and suppressing redundant features.
[0038] Obtaining the modulated conditional latent variable and Next, the components that are beneficial to the diffusion and reconstruction process need to be selectively extracted. Specifically, the components that are beneficial to the diffusion and reconstruction process need to be selectively extracted. and The data is concatenated and input into the spatial attention module to generate a spatial weight matrix. The spatial attention module includes two convolutional layers, one activation layer, and one sigmoid layer, used to calculate the weight distribution at different spatial locations.
[0039] Its mathematical expression is as follows: in , indicating spatial attention operation, This indicates a splicing operation. and These are the conditional latent variables for the output.
[0040] Figure 3 The structure-fidelity feature encoder of the present invention includes the following: To reduce texture and structural distortion that may occur during decoding and improve the fidelity of the generated result, a structure-fidelity feature encoder is proposed. The training process of the structure-fidelity feature encoder is as follows: First, remove the low-quality image. Compared with the initial reconstruction results The input is fed into a gated fusion module to extract high-fidelity features. This module is structurally consistent with the aforementioned "latent variable gated fusion module," but its workspace is switched from the latent space to the image space. The gated fusion module is used to selectively fuse multi-source input information to obtain more representative high-fidelity features.
[0041] Subsequently, the aforementioned fidelity features are added to the corresponding positions in the VAE decoder using zero convolution to provide fidelity constraints during the decoding process.
[0042] Furthermore, real images The input is fed into a pre-trained VAE encoder, which encodes it into the latent space to obtain compressed latent variables. This latent variable is used to simulate the denoised latent representation. Then, guided by the fidelity features, the pre-trained VAE decoder uses the latent variable... Convert to reconstructed image .
[0043] During the training phase, by minimizing the reconstructed image With real images The average absolute error (i.e., L1 loss) between the two is used to optimize the structure-fidelity feature encoder, thereby ensuring that the generated image has higher fidelity at the texture and structure levels.
[0044] Corresponding to the aforementioned embodiment of a unified raindrop and reflection removal method based on a diffusion model, the present invention also provides an embodiment of a unified raindrop and reflection removal device based on a diffusion model.
[0045] See Figure 4 The present invention provides a unified raindrop and reflection removal device based on a diffusion model, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a unified raindrop and reflection removal method based on a diffusion model as described in the above embodiment.
[0046] The embodiment of the raindrop and reflection unified removal device based on a diffusion model provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 4 The diagram shown is a hardware structure diagram of any data processing device, including the raindrop and reflection unified removal device based on a diffusion model provided by this invention. (Except for...) Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.
[0047] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.
[0048] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0049] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements a method for unified removal of raindrops and reflections based on a diffusion model as described in the above embodiments.
[0050] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.
[0051] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method for unified removal of raindrops and reflections based on a diffusion model.
[0052] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.
[0053] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.
Claims
1. A unified method for removing raindrops and reflections based on a diffusion model, characterized in that, The method includes: Receive low-quality images and use a restoration network to remove the degradation of raindrops and reflections to obtain an initial restoration result; The low-quality image and the initial restoration result are encoded as latent representations as conditional latent variables. A noisy latent variable is generated at each diffusion time step. The conditional latent variables are modulated using the noisy latent variable. Spatial weights are assigned to each conditional latent variable after modulation. The processed features are input into the diffusion model to denoise the noisy latent variables. The latent variables are input into the structure-fidelity feature encoder. The reconstruction of the latent variables is guided by extracting the structured features of the low-quality image and the initial result, and the final restoration result is obtained.
2. The method for unified removal of raindrops and reflections based on a diffusion model according to claim 1, characterized in that, The specific method of encoding the low-quality image and the initial restoration result into a latent representation is as follows: a pre-trained variational autoencoder is used as a latent space mapping tool to encode the low-quality image and the initial restoration result, thereby obtaining the conditional latent variables of the low-quality image and the initial restoration result respectively.
3. The method for unified removal of raindrops and reflections based on a diffusion model according to claim 1, characterized in that, The modulation of the conditional latent variable using the noise latent variable specifically involves: firstly, extracting the feature vectors of the noise latent variable and the conditional latent variable through a convolutional layer; then concatenating the two feature vectors; using a Transformer network to perform cross-attention operations to obtain an interaction feature with the same feature dimension as the noise latent variable; combining the interaction feature with the noise latent variable feature vector; and then using a convolutional layer to reduce the dimensionality to obtain the modulated conditional variable.
4. The method for unified removal of raindrops and reflections based on a diffusion model according to claim 1, characterized in that, The specific method for allocating spatial weights based on each conditional latent variable is as follows: The modulated low-quality image is concatenated with the conditional latent variables of the initial restoration result and input into the spatial attention module to generate a spatial weight matrix. The spatial attention module includes two convolutional layers, one activation layer and one sigmoid layer, which are used to calculate the weight distribution at different spatial locations and adjust the conditional latent variables of the low-quality image and the initial restoration result based on the weight distribution.
5. The method for unified removal of raindrops and reflections based on a diffusion model according to claim 1, characterized in that, The process of inputting the processed features into the diffusion model to denoise the noisy latent variables specifically involves: inputting the conditional information and the noisy latent variables into the control branch, receiving the conditional latent variables and the noisy latent variables and generating a control signal, and injecting the control signal into the main denoising U-Net network through zero convolution to guide the denoising process. The main denoising U-Net estimates and denoises the noisy latent variables at each time step, thereby gradually recovering the clean latent variables.
6. The method for unified removal of raindrops and reflections based on a diffusion model according to claim 1, characterized in that, The control branch is a pre-trained U-Net encoder and a trainable copy of the intermediate layer.
7. The method for unified removal of raindrops and reflections based on a diffusion model according to claim 1, characterized in that, The training process of the structure-fidelity feature encoder includes: The low-quality image is gated and fused with the initial restoration result to obtain fidelity features. These fidelity features are then added to the corresponding positions in the VAE decoder using zero convolution to provide fidelity constraints during the decoding process. Real images The input is fed into a pre-trained VAE encoder, which encodes it into the latent space to obtain compressed latent variables. This is used to simulate the latent representation after denoising; then, guided by the fidelity features, the pre-trained VAE decoder converts the latent variables... Convert to reconstructed image ; During the training phase, by minimizing the reconstructed image With real images The mean absolute error between the two is used to optimize the structure-fidelity feature encoder, thereby ensuring that the generated image has higher fidelity at the texture and structure levels.
8. A unified raindrop and reflection removal device based on a diffusion model, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a method for unified removal of raindrops and reflections based on a diffusion model as described in any one of claims 1-7.
9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a unified method for removing raindrops and reflections based on a diffusion model as described in any one of claims 1-7.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements a unified method for removing raindrops and reflections based on a diffusion model as described in any one of claims 1-7.