Method and device for restoring images degraded in severe weather based on dynamic degradation generation

Abstract

The present invention discloses a method and apparatus for restoring images degraded by severe weather based on dynamic degradation generation, relating to the field of image processing. The method comprises: during the training of a noise predictor and a dynamic degradation generator, optimizing latent variables using the E-step of an expectation-maximization algorithm to obtain optimized latent variables; in the M-step of the expectation-maximization algorithm, inputting the optimized latent variables and state variables into the dynamic degradation generator to generate a degradation layer for semi-supervised training of the noise predictor and the dynamic degradation generator, thereby obtaining a trained noise predictor; and during the image restoration phase, inputting a degraded image degraded by severe weather and random Gaussian noise into the trained noise predictor, executing an iterative denoising step that follows a Markov chain, and obtaining a clean restored image. The present invention addresses the problems of limited generalization and unstable restoration results of existing severe weather image restoration models.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method and device for restoring an image degraded in severe weather based on dynamic degradation generation. Background Art

[0002] In the field of computer vision, adverse weather conditions (such as fog, rain, snow, and dust) can severely impact image quality, reducing the accuracy of tasks such as autonomous driving, video surveillance, and object detection. Therefore, research on efficient image restoration methods to address image degradation under various weather conditions is of great practical significance.

[0003] Traditional image restoration methods are primarily based on physical models, such as atmospheric scattering models and rain streak removal models. These methods require accurate estimation of physical parameters, such as transmittance and aerosol concentration. However, obtaining precise physical information in complex environments often makes it difficult to obtain this information, resulting in unstable restoration results. In recent years, data-driven deep learning methods have made significant progress in image restoration. Supervised learning methods rely on large amounts of paired data for training, but high-quality, real-world paired data is difficult to obtain, and the model's generalization ability is limited. Furthermore, a single model can typically only handle specific types of weather degradation and is difficult to adapt to a wide range of severe weather scenarios. Summary of the Invention

[0004] The purpose of this application is to propose a method and device for restoring images degraded in severe weather based on dynamic degradation generation in response to the above-mentioned technical problems.

[0005] In a first aspect, the present invention provides a method for restoring an image degraded in severe weather based on dynamic degradation generation, comprising the following steps:

[0006] Construct a conditional diffusion model and a dynamic degradation generator. The diffusion model includes a forward denoising process and a reverse denoising process. The reverse denoising process uses a noise predictor for iterative denoising. The dynamic degradation generator includes a sequentially connected transfer model, a fully connected layer, and an emission model.

[0007] Labeled and unlabeled datasets are constructed. During the training process of the noise predictor and dynamic degradation generator, the introduced latent variables are optimized using the E-step of the expectation-maximization algorithm to obtain the optimized latent variables. In the M-step of the expectation-maximization algorithm, the optimized latent variables and the introduced state variables are input into the dynamic degradation generator to generate a degradation layer. The noise predictor and the dynamic degradation generator are semi-supervisedly trained using the labeled and unlabeled datasets in combination with the degradation layer to obtain a trained noise predictor and a trained dynamic degradation generator. The inverse denoising process of the diffusion model including the trained noise predictor is used as the restoration model for degraded images in severe weather.

[0008] A severe weather-degraded image to be restored and random noise conforming to a standard Gaussian distribution are simultaneously input into the severe weather-degraded image restoration model. A denoising process of T diffusion time steps obeying a Markov chain is performed through a trained noise predictor to obtain a clean restored image.

[0009] As a preference, the E-step of the expectation maximization algorithm is used to optimize the introduced latent variables to obtain the optimized latent variables, specifically including:

[0010] In the E-step of the expectation maximization algorithm, Langevin dynamics is used to calculate the hidden variables Perform iterative optimization to obtain the optimized latent variables , the iterative optimization process is expressed as follows:

[0011] ;

[0012] where τ represents the τth iteration of Langevin dynamics, δ represents the step size factor, and ξ (τ) represents the Gaussian noise used in the τth iteration of the Langevin dynamics, Represents the latent variable The objective function, represents the hidden variables after the τth iteration optimization of Langevin dynamics, represents the hidden variables after the τ+1th iteration optimization of Langevin dynamics, Represents the latent variable The objective function is The gradient at

[0013] Repeat the above iterative optimization process to obtain the hidden variables after the last iterative optimization of Langevin dynamics and use them as the optimized hidden variables. .

[0014] Preferably, the optimized latent variables and the introduced state variables are input into a dynamic degradation generator to generate a degradation layer, which specifically includes:

[0015] Introduce the i-1th state variable in the i-th dynamic degradation generation process corresponding to the dynamic degradation generator and the i-th optimized latent variable , and , represents a normal distribution, represents the identity matrix, To submit to;

[0016] The i-1th state variable and the i-th optimized latent variable Input into the transfer model to obtain the i-th state variable , as shown below:

[0017] ;

[0018] Among them, T( ) represents the function corresponding to the transfer model, represents the parameters of the transfer model;

[0019] The i-th state variable Input to the fully connected layer and reshape to get the i-th tensor, input the i-th tensor into the emission model to get the i-th degradation layer , as shown below:

[0020] =E( ; );

[0021] in, represents the i-th tensor, E( ) represents the function corresponding to the emission model, Represents the parameters of the emission model.

[0022] Preferably, a labeled dataset and an unlabeled dataset are used in combination with a degradation layer to perform semi-supervised training on the noise predictor and the dynamic degradation generator to obtain a trained noise predictor and a trained dynamic degradation generator, specifically including:

[0023] In the forward denoising process of the diffusion model, a fixed Markov chain is used to transform the original clean image into The perturbation is gradually changed to a noise image. The perturbation process follows the preset variance scheduling parameters. In a diffusion time step, the state transition is defined by:

[0024] ;

[0025] in, Indicates the first The conditional transition probability distribution of diffusion time steps, Indicates the Noise image with diffusion time steps, Indicates the Noise image with diffusion time steps, Indicates the The variance scheduling parameter of the diffusion time step, , represents the identity matrix, represents the probability distribution of the original clean image, To submit to;

[0026] By using the properties of Markov chain, we can directly Sampling the first Noise image of diffusion time steps , as shown below:

[0027] ;

[0028] in, Indicates the original clean image Direct sampling under the condition of Noise image of diffusion time steps The forward diffusion conditional probability distribution of , and , Indicates passing The cumulative signal retention rate of diffusion time steps, Indicates the The signal retention coefficient for each diffusion time step, Indicates the The signal retention coefficient for each diffusion time step, , represents the noise term;

[0029] In the inverse denoising process of the diffusion model, Start sampling, and with the degraded image y as the condition, gradually perform the denoising sampling steps to restore the clean image. The inverse denoising process constitutes a parameterized Markov chain, and its joint probability distribution can be expressed as:

[0030] ;

[0031] in, Represents the diffusion time from the 0th time step to the The state sequence of diffusion time steps, Indicates that under the degraded image y The conditional probability distribution of denotes the parameters of the noise predictor, Indicates the Noise image of diffusion time steps The probability distribution of , represents a normal distribution, Denotes subject to, I denotes the identity matrix, Indicates that the degraded image y and Noise image of diffusion time steps Get the next Noise image of diffusion time steps The conditional probability distribution of ;

[0032] In the reverse denoising process, the following formula is used from the first Noise image of diffusion time steps Get the estimated clean image of the skip reconstruction :

[0033] ;

[0034] in, Indicates that the Noise image of diffusion time steps The predicted noise obtained by inputting into the noise predictor;

[0035] The estimated clean image reconstructed by skipping Linearly superimpose each degraded layer to obtain the estimated degraded image, as shown in the following formula:

[0036] ;

[0037] in, represents the estimated degraded image;

[0038] Loss function used in the semi-supervised training of the noise predictor and dynamic degradation generator for:

[0039] ;

[0040] in, represents the L2 norm, denotes the parameters of the noise predictor, represents the parameters of the dynamic degradation generator, represents the degraded image, Represents the estimated degraded image. If the degraded image comes from a labeled dataset, then ; If the degraded image comes from an unlabeled dataset, then ; Represents the function corresponding to the bad weather degradation image restoration model; represents the function corresponding to the dynamic degradation generator, represents the optimized latent variable, represents the initial state variables, represents the variance of the residual term, represents the Kullback-Leibler divergence, Indicates that given the original clean image Under the condition of Noise image of diffusion time steps The posterior probability distribution of Represents the noise image at the first diffusion time step The original clean image is obtained under the condition The conditional probability distribution of Indicates in Noise image of diffusion time steps and the original clean image Under the condition of Noise image of diffusion time steps The true posterior probability distribution of Indicates in Noise image of diffusion time steps Under the condition of Noise image of diffusion time steps The conditional probability distribution of represents the real clean images in the labeled dataset, represents the hyperparameter, Represents the probability distribution q( ) take expectation;

[0041] During each round of training, the E-step and M-step of the expectation-maximization algorithm are executed alternately. By minimizing the loss function, the optimized noise predictor and the optimized dynamic degradation generator are obtained in each round.

[0042] The noise predictor optimized in the current round and the dynamic degradation generator optimized in the current round are used to perform the E-step and M-step of the expectation-maximization algorithm in the next round of training. The above training process is repeated until convergence to obtain the trained noise predictor and the trained dynamic degradation generator.

[0043] Preferably, the transfer model includes a first fully connected layer, a second fully connected layer and a third fully connected layer connected in sequence, wherein the first fully connected layer and the second fully connected layer have a ReLU activation function, and the third fully connected layer has a Tanh activation function.

[0044] Preferably, the emission model includes a first convolutional layer, a second convolutional layer, an upsampling layer and a third convolutional layer connected in sequence, wherein the first convolutional layer, the second convolutional layer and the third convolutional layer all have a ReLU activation function.

[0045] In a second aspect, the present invention provides a device for restoring an image degraded in severe weather based on dynamic degradation generation, comprising:

[0046] a model building module configured to construct a conditional diffusion model and a dynamic degradation generator, wherein the diffusion model includes a forward denoising process and a reverse denoising process, wherein the reverse denoising process uses a noise predictor for iterative denoising, and the dynamic degradation generator includes a transfer model, a fully connected layer, and an emission model connected in sequence;

[0047] The model training module is configured to construct a labeled dataset and an unlabeled dataset. During the training process of the noise predictor and the dynamic degradation generator, the introduced latent variables are optimized using the E-step of the expectation maximization algorithm to obtain the optimized latent variables. In the M-step of the expectation maximization algorithm, the optimized latent variables and the introduced state variables are input into the dynamic degradation generator to generate a degradation layer. The noise predictor and the dynamic degradation generator are semi-supervisedly trained using the labeled dataset and the unlabeled dataset in combination with the degradation layer to obtain a trained noise predictor and a trained dynamic degradation generator. The inverse denoising process of the diffusion model including the trained noise predictor is used as a severe weather degradation image restoration model.

[0048] The restoration module is configured to obtain a severe weather-degraded image to be restored and random noise that conforms to a standard Gaussian distribution, and simultaneously input them into the severe weather-degraded image restoration model. Through a trained noise predictor, a denoising process of T diffusion time steps obeying a Markov chain is performed to obtain a clean restored image.

[0049] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0050] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0051] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which implements the method described in any implementation manner in the first aspect when the computer program is executed by a processor.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] (1) The proposed method for restoration of degraded images in severe weather conditions based on dynamic degradation generation achieves efficient optimization of latent variables and joint estimation of model parameters by designing a degradation layer generation process, constructing a conditional denoising process based on degradation generation, and combining an expectation maximization algorithm based on Monte Carlo, thereby achieving high-quality restoration of degraded images under different severe weather conditions under a semi-supervised learning framework.

[0054] (2) The proposed method for restoring images degraded in severe weather conditions based on dynamic degradation generation enables semi-supervised learning of the diffusion model through the Monte Carlo-based expectation maximization algorithm, which makes the model better generalized in real scenes. In addition, by optimizing the latent variables in the E-step of the expectation maximization algorithm and combining the emission model and the transfer model, the characteristics of degraded layers such as rain and snow layers can be realistically simulated, thereby improving the image restoration quality.

[0055] (3) The method for restoring images degraded in severe weather conditions based on dynamic degradation generation proposed in this paper utilizes the inverse denoising process of the diffusion model to improve the restoration performance, and implements the semi-supervised learning paradigm of the diffusion model based on the Monte Carlo expectation maximization algorithm, thereby effectively improving the robustness and generalization ability of visual tasks. It has the characteristics of high robustness, strong adaptability and excellent generalization ability, and is suitable for image restoration and enhancement in various visual tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1 This is a flowchart of a method for restoring a severe weather-degraded image based on dynamic degradation generation according to an embodiment of the present application;

[0058] Figure 2 A schematic diagram of the training process of the noise predictor and the dynamic degradation generator of the severe weather degraded image restoration method based on dynamic degradation generation according to an embodiment of the present application;

[0059] Figure 3 A schematic diagram of a transfer model of a method for restoring a severe weather-degraded image based on dynamic degradation generation according to an embodiment of the present application;

[0060] Figure 4 A schematic diagram of an emission model of a method for restoring a severe weather-degraded image based on dynamic degradation generation according to an embodiment of the present application;

[0061] Figure 5 Schematic diagram of a device for restoring an image degraded in severe weather based on dynamic degradation generation according to an embodiment of the present application;

[0062] Figure 6 A schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0064] Figure 1 The embodiment of the present application provides a method for restoring an image degraded in severe weather based on dynamic degradation generation, comprising the following steps:

[0065] S1, construct a conditional diffusion model and a dynamic degradation generator. The diffusion model includes a forward denoising process and a reverse denoising process. In the reverse denoising process, a noise predictor is used for iterative denoising. The dynamic degradation generator includes a sequentially connected transfer model, a fully connected layer, and an emission model.

[0066] In a specific embodiment, the transfer model includes a first fully connected layer, a second fully connected layer, and a third fully connected layer connected in sequence, wherein the first fully connected layer and the second fully connected layer have a ReLU activation function, and the third fully connected layer has a Tanh activation function.

[0067] In a specific embodiment, the emission model includes a first convolutional layer, a second convolutional layer, an upsampling layer, and a third convolutional layer connected in sequence, wherein the first convolutional layer, the second convolutional layer, and the third convolutional layer all have a ReLU activation function.

[0068] Specifically, refer to Figure 2 The embodiment of the present application adopts a conditional diffusion model, which includes a forward denoising process and a reverse denoising process. In the forward denoising process, Gaussian noise is gradually added to the randomly generated original clean image to obtain a noisy image at each diffusion time step; in the reverse denoising process, the degraded image is used as a condition, that is, the degraded image and the first The noise image of the diffusion time step is input to the noise predictor to obtain the The prediction noise of the diffusion time step is calculated by The prediction noise of the diffusion time step is The noise image of the diffusion time step is denoised and the The noise image of diffusion time steps is repeated, and the noise is gradually removed to obtain the original clean image.

[0069] The dynamic degradation generator is used in the training process of the noise predictor in the diffusion model, and the parameters of the dynamic degradation generator are also optimized synchronously during the training process of the noise predictor in the diffusion model. The dynamic degradation generator consists of a transfer model, a fully connected layer and an emission model. Figure 3 and Figure 4 The transfer model is modeled using a multi-layer perceptron (MLP). In one embodiment, the transfer model consists of a first fully connected layer, a second fully connected layer, and a third fully connected layer, which are connected in sequence. The first and second fully connected layers have ReLU activation functions, and the third fully connected layer has a Tanh activation function. The emission model consists of a first convolutional layer, a second convolutional layer, an upsampling layer, and a third convolutional layer, which are connected in sequence. Each convolutional layer has a ReLU activation function.

[0070] S2, construct labeled datasets and unlabeled datasets. During the training process of the noise predictor and dynamic degradation generator, the E-step of the expectation-maximization algorithm is used to optimize the introduced latent variables to obtain the optimized latent variables; in the M-step of the expectation-maximization algorithm, the optimized latent variables and the introduced state variables are input into the dynamic degradation generator to generate degradation layers. The labeled dataset and the unlabeled dataset are used in combination with the degradation layers to perform semi-supervised training on the noise predictor and the dynamic degradation generator to obtain the trained noise predictor and the trained dynamic degradation generator; the inverse denoising process of the diffusion model is used as the restoration model for severe weather degraded images.

[0071] In a specific embodiment, the E-step of the expectation maximization algorithm is used to optimize the introduced latent variables to obtain the optimized latent variables, which specifically includes:

[0072] In the E-step of the expectation maximization algorithm, Langevin dynamics is used to calculate the hidden variables Perform iterative optimization to obtain the optimized latent variables , the iterative optimization process is expressed as follows:

[0073] ;

[0074] where τ represents the τth iteration of Langevin dynamics, δ represents the step size factor, and ξ (τ) represents the Gaussian noise used in the τth iteration of the Langevin dynamics, Represents the latent variable The objective function gradient is represents the hidden variables after the τth iteration optimization of Langevin dynamics, represents the hidden variables after the τ+1th iteration optimization of Langevin dynamics, Represents the latent variable The objective function is The gradient at

[0075] Repeat the above iterative optimization process to obtain the hidden variables after the last iterative optimization of Langevin dynamics and use them as the optimized hidden variables. .

[0076] In a specific embodiment, the optimized latent variables and the introduced state variables are input into a dynamic degradation generator to generate a degradation layer, which specifically includes:

[0077] Introduce the i-1th state variable in the i-th dynamic degradation generation process corresponding to the dynamic degradation generator and the i-th optimized latent variable ,in, and , represents a normal distribution, represents the identity matrix, To submit to;

[0078] The i-1th state variable and the i-th optimized latent variable Input into the transfer model to obtain the i-th state variable , as shown below:

[0079] ;

[0080] Among them, T( ) represents the function corresponding to the transfer model, represents the parameters of the transfer model;

[0081] The i-th state variable Input to the fully connected layer and reshape to get the i-th tensor, input the i-th tensor into the emission model to get the i-th degradation layer , as shown below:

[0082] =E( ; );

[0083] in, represents the i-th tensor, E( ) represents the function corresponding to the emission model, Represents the parameters of the emission model.

[0084] In a specific embodiment, a noise predictor and a dynamic degradation generator are semi-supervisedly trained using a labeled dataset and an unlabeled dataset in combination with a degradation layer to obtain a trained noise predictor and a trained dynamic degradation generator, specifically including:

[0085] In the forward denoising process of the diffusion model, a fixed Markov chain is used to transform the original clean image into The perturbation is gradually changed to a noise image. The perturbation process follows the preset variance scheduling parameters. In a diffusion time step, the state transition is defined by:

[0086] ;

[0087] in, Indicates the first The conditional transition probability distribution of diffusion time steps, Indicates the Noise image with diffusion time steps, Indicates the Noise image with diffusion time steps, Indicates the The variance scheduling parameter of the diffusion time step, , represents the identity matrix, represents the probability distribution of the original clean image, To submit to;

[0088] By using the properties of Markov chain, we can directly Sampling the first Noise image of diffusion time steps , as shown below:

[0089] ;

[0090] in, Indicates the original clean image Direct sampling under the condition of Noise image of diffusion time steps The forward diffusion conditional probability distribution of , and , Indicates passing The cumulative signal retention rate of diffusion time steps, Indicates the The signal retention coefficient for each diffusion time step, Indicates the The signal retention coefficient for each diffusion time step, , represents the noise term;

[0091] In the inverse denoising process of the diffusion model, Start sampling, and with the degraded image y as the condition, gradually perform the denoising sampling steps to restore the clean image. The inverse denoising process constitutes a parameterized Markov chain, and its joint probability distribution can be expressed as:

[0092] ;

[0093] in, Represents the diffusion time from the 0th time step to the The state sequence of diffusion time steps, Indicates that under the degraded image y The conditional probability distribution of denotes the parameters of the noise predictor, Indicates the Noise image of diffusion time steps The probability distribution of , represents a normal distribution, Denotes subject to, I denotes the identity matrix, Indicates that the degraded image y and Noise image of diffusion time steps Get the next Noise image of diffusion time steps The conditional probability distribution of ;

[0094] In the reverse denoising process, the following formula is used from the first Noise image of diffusion time steps Get the estimated clean image of the skip reconstruction :

[0095] ;

[0096] in, Indicates that the Noise image of diffusion time steps The predicted noise obtained by inputting into the noise predictor;

[0097] The estimated clean image reconstructed by skipping Linearly superimpose each degraded layer to obtain the estimated degraded image, as shown in the following formula:

[0098] ;

[0099] in, represents the estimated degraded image;

[0100] Loss function used in the semi-supervised training of the noise predictor and dynamic degradation generator for:

[0101] ;

[0102] in, represents the L2 norm, denotes the parameters of the noise predictor, represents the parameters of the dynamic degradation generator, represents the degraded image, Represents the estimated degraded image. If the degraded image comes from a labeled dataset, then ; If the degraded image comes from an unlabeled dataset, then ; Represents the function corresponding to the bad weather degradation image restoration model; represents the function corresponding to the dynamic degradation generator, represents the optimized latent variable, represents the initial state variables, represents the variance of the residual term, represents the Kullback-Leibler divergence, Indicates that given the original clean image Under the condition of Noise image of diffusion time steps The posterior probability distribution of Represents the noise image at the first diffusion time step The original clean image is obtained under the condition The conditional probability distribution of Indicates in Noise image of diffusion time steps and the original clean image Under the condition of Noise image of diffusion time steps The true posterior probability distribution of Indicates in Noise image of diffusion time steps Under the condition of Noise image of diffusion time steps The conditional probability distribution of represents the real clean images in the labeled dataset, represents the hyperparameter, Represents the probability distribution q( ) take expectation;

[0103] During each round of training, the E-step and M-step of the expectation-maximization algorithm are executed alternately. By minimizing the loss function, the optimized noise predictor and the optimized dynamic degradation generator are obtained in each round.

[0104] The noise predictor optimized in the current round and the dynamic degradation generator optimized in the current round are used to perform the E-step and M-step of the expectation-maximization algorithm in the next round of training. The above training process is repeated until convergence to obtain the trained noise predictor and the trained dynamic degradation generator.

[0105] Specifically, in the training process of the noise predictor and the dynamic degradation generator in the diffusion model, the E-step of the expectation maximization algorithm is first used, and the latent variable z is iteratively optimized through Langevin Dynamics, so that the latent variable gradually approaches the real data distribution during the update process. The purpose of introducing Gaussian noise in the optimization iterative formula of the latent variable z is to prevent falling into the local optimal solution during the optimization iteration process. After obtaining the optimized latent variables, the introduced state variables and the optimized latent variables are used in each dynamic degradation generation process corresponding to the dynamic degradation generator. After multiple dynamic degradation generation processes, the degradation layer generated by each dynamic degradation generation process is obtained. The degradation layer generated by each dynamic degradation generation process is compared with the estimated clean image reconstructed by the jump step. By performing linear superposition, we can obtain each corresponding estimated degraded image.

[0106] Furthermore, in order to reflect the estimated degraded image The matching between the real degraded image y is used to introduce the mean square error (MSE) loss function:

[0107] 2;

[0108] in, represents the mean square error loss function;

[0109] In the M-step of the expectation-maximization algorithm, the model parameters are estimated by maximum likelihood from the probability model containing latent variables to further optimize the overall performance of the model. Specifically, by maximizing the following log-likelihood function:

[0110] ;

[0111] in, represents the parameters of the noise predictor under the degraded image y and the parameters of the dynamic degradation generator The joint posterior probability distribution of represents the parameters of the noise predictor and the parameters of the dynamic degradation generator The likelihood function of generating the degraded image y is: represents the parameters of the noise predictor The prior probability distribution of represents a constant term, It means that the expression on the left is defined as equivalent to the symbol on the right. represents the objective function to be optimized, represents the parameters of the dynamic degradation generator, ~ , Denotes subject to, and I denotes the identity matrix.

[0112] For the parameters of the noise predictor included in the diffusion model , the embodiment of the present application adopts the variational lower bound of negative log-likelihood for training, and its corresponding KL divergence loss function is defined as follows:

[0113] ;

[0114] in, Represents the KL divergence loss function, which uses KL divergence to measure the distance between two probability distributions.

[0115] Furthermore, a labeled dataset and an unlabeled dataset are constructed. In the labeled dataset, each sample has a real clear image corresponding to the degraded image. ; However, each sample in the unlabeled dataset does not contain a real clear image corresponding to the degraded image Therefore, a semi-supervised loss function is introduced for the cases of labeled and unlabeled datasets. In each training cycle (epoch), the model will alternate between the labeled and unlabeled datasets. The training of the labeled and unlabeled datasets is carried out simultaneously, alternating between them.

[0116] Combining the above four loss functions or objective functions, the loss function is derived , using the loss function minimized , the parameters of the noise predictor and the parameters of the dynamic degradation generator Optimization and adjustment are performed. During the above training process, the E-step and M-step of the expectation-maximization algorithm are performed simultaneously. Iterations are repeated until convergence, resulting in a trained noise predictor and a trained dynamic degradation generator. By minimizing this loss function, the model can gradually approximate the distribution of real data during the optimization process, ensuring higher stability and accuracy in the complex inverse denoising process. The pseudocode for the specific training process is shown in Table 1.

[0117] Table 1:

[0118]

[0119] In the embodiments of the present application, a dynamic degradation generator is introduced during the training of the noise predictor in the diffusion model to guide and synchronously optimize the training process. An expectation maximization algorithm is also embedded in the training process, so that the trained severe weather-degraded image restoration model (i.e., the inverse denoising process of the diffusion model including the trained noise predictor) can restore severe weather-degraded images under various conditions.

[0120] S3, obtaining a severe weather-degraded image to be restored and random noise conforming to a standard Gaussian distribution, and simultaneously inputting them into the severe weather-degraded image restoration model. Through the trained noise predictor, a denoising process of T diffusion time steps obeying a Markov chain is performed to obtain a clean restored image.

[0121] Specifically, the inverse denoising process containing the trained noise predictor in the diffusion model is used as the severe weather degraded image restoration model. The severe weather degraded image to be restored and the random noise vector obtained that conforms to the standard Gaussian distribution are input into the severe weather degraded image restoration model. With the severe weather degraded image to be restored as a condition, the trained noise predictor is used to predict the predicted noise of each diffusion time step, and the severe weather degraded image to be restored is gradually denoised to obtain the corresponding restored clean image.

[0122] The method for restoring images degraded by severe weather conditions based on dynamic degradation generation, proposed in the embodiments of this application, introduces semi-supervised learning methods to the image restoration task. Semi-supervised learning can utilize a small amount of labeled data and a large amount of unlabeled data for training, reducing reliance on large-scale annotated data and improving the model's generalization capabilities. Furthermore, by designing a unified framework, image restoration tasks under different weather conditions are integrated into a single model, enabling the integrated restoration of a variety of images degraded by severe weather conditions, better meeting practical application needs.

[0123] The technical effects of the embodiments of the present application are described below through specific experiments.

[0124] The experimental platform used in the embodiments of this application is as follows:

[0125] 1) Hardware:

[0126] GPU: NVIDIA GeForce RTX 3090;

[0127] CPU: Intel Xeon(R)CPU E5-2697A v4@ 2.60 GHz×64;

[0128] Memory: 256 GB RAM

[0129] 2) Software:

[0130] Operating system: Ubuntu 20.04.6 LTS;

[0131] Framework: PyTorch 2.1.0 + CUDA 12.1, Python 3.8;

[0132] Dependencies: NumPy, OpenCV, SciPy, torchvision;

[0133] The specific parameter settings are shown in Table 2.

[0134] Table 2:

[0135]

[0136] The performance of the severe weather degradation image restoration model proposed in the embodiments of this application is compared with other existing fully supervised image restoration models (including Transweather, AirNet, Gridformer*, and PromptIR*). The comparison results are shown in Table 3. According to the comparison results, it can be seen that:

[0137] 1. The severe weather-degraded image restoration model proposed in the examples of this application significantly outperforms Transweather in terms of PSNR, but still lags behind fully supervised methods (such as AirNet and Gridformer*). This gap is consistent with the expected performance of semi-supervised learning. In particular, fully supervised methods have an advantage in pixel-level reconstruction tasks because they directly utilize label information.

[0138] 2. The PSNR of the weather-degraded image restoration model proposed in this embodiment (30.95) differs by approximately 5.66 dB from the optimal fully supervised method (Gridformer*, 36.61). The difference in structural similarity (SSIM) is 0.069 (0.902 vs. 0.971). This difference is within the typical range for semi-supervised learning and is an acceptable result, especially when data annotation is limited.

[0139] 3. Semi-supervised methods are weaker than fully supervised methods in terms of PSNR / SSIM, primarily due to insufficient supervisory signal. However, they offer significant advantages in annotation cost and generalization. If the test data contains unknown degradation types or low-quality input, the performance of semi-supervised methods may degrade less than that of fully supervised methods.

[0140] Table 3:

[0141]

[0142] Further references Figure 5 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a device for restoring images degraded in severe weather based on dynamic degradation generation. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0143] The embodiment of the present application provides a device for restoring an image degraded in severe weather based on dynamic degradation generation, comprising:

[0144] Model building module 1 is configured to build a conditional diffusion model and a dynamic degradation generator. The diffusion model includes a forward denoising process and a reverse denoising process. The reverse denoising process uses a noise predictor for iterative denoising. The dynamic degradation generator includes a transfer model, a fully connected layer, and an emission model connected in sequence.

[0145] Model training module 2 is configured to construct a labeled dataset and an unlabeled dataset. During the training process of the noise predictor and the dynamic degradation generator, the introduced latent variables are optimized using the E-step of the expectation-maximization algorithm to obtain the optimized latent variables. In the M-step of the expectation-maximization algorithm, the optimized latent variables and the introduced state variables are input into the dynamic degradation generator to generate a degradation layer. The noise predictor and the dynamic degradation generator are semi-supervisedly trained using the labeled dataset and the unlabeled dataset in combination with the degradation layer to obtain a trained noise predictor and a trained dynamic degradation generator. The inverse denoising process of the diffusion model including the trained noise predictor is used as a severe weather degradation image restoration model.

[0146] Restoration module 3 is configured to obtain the severe weather-degraded image to be restored and random noise that conforms to the standard Gaussian distribution, and simultaneously input them into the severe weather-degraded image restoration model. Through the trained noise predictor, a denoising process of T diffusion time steps obeying the Markov chain is performed to obtain a clean restored image.

[0147] Figure 6 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Figure 6 As shown, the electronic device of this embodiment includes: a processor 601 and a memory 602; wherein the memory 602 is used to store computer-executable instructions; and the processor 601 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description of the above method embodiment.

[0148] Optionally, the memory 602 may be independent or integrated with the processor 601 .

[0149] When the memory 602 is independently provided, the electronic device further includes a bus 603 for connecting the memory 602 and the processor 601 .

[0150] An embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 601 executes the computer execution instructions, the above method is implemented.

[0151] An embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 601, the above method is implemented.

[0152] In the embodiments provided herein, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical functional division. In actual implementation, other division methods may be used. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not implemented. In addition, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or module, which may be electrical, mechanical or other forms.

[0153] Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of these modules may be selected to implement the solution of this embodiment based on actual needs.

[0154] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each module may exist physically separately, or two or more modules may be integrated into a single unit. The units formed by the above modules may be implemented in the form of hardware or hardware plus software functional units.

[0155] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or processor 601 to perform some steps of the methods of various embodiments of the present application.

[0156] It should be understood that the processor 601 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), or application-specific integrated circuits (ASIC). A general-purpose processor may be a microprocessor, or the processor 601 may be any conventional processor 601. The steps of the method disclosed in the present invention may be directly implemented as being executed by the hardware processor 601, or may be implemented by a combination of hardware and software modules in the processor 601.

[0157] The memory 602 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0158] Bus 603 can be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Bus 603 can be classified as an address bus, a data bus, a control bus, etc. For ease of illustration, the bus 603 in the drawings of this application is not limited to a single bus 603 or a single type of bus 603.

[0159] The storage medium may be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0160] An exemplary storage medium is coupled to the processor 601, so that the processor 601 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor 601. The processor 601 and the storage medium can be located in an application-specific integrated circuit (ASIC). Of course, the processor 601 and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0161] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for restoring images degraded by severe weather conditions based on dynamic degradation generation, characterized in that: The following steps are involved: Constructing a conditional diffusion model and a dynamic degradation generator, wherein the diffusion model includes a forward denoising process and a reverse denoising process, wherein the reverse denoising process uses a noise predictor for iterative denoising, and the dynamic degradation generator includes a transfer model, a fully connected layer, and an emission model connected in sequence; A labeled dataset and an unlabeled dataset are constructed. During the training process of the noise predictor and the dynamic degradation generator, the E-step of the expectation maximization algorithm is used to optimize the introduced latent variables to obtain the optimized latent variables, specifically including: In the E-step of the expectation-maximization algorithm, the latent variable z is iteratively optimized using Langevin dynamics to obtain the optimized latent variable z′. The iterative optimization process is expressed as follows: where τ represents the τth iteration of Langevin dynamics, δ represents the step size factor, and ξ (τ) represents the Gaussian noise used in the τth iteration of Langevin dynamics, g(z) represents the objective function with respect to the latent variable z, and z (τ) represents the hidden variable after the τth iteration optimization of Langevin dynamics, z (τ+1) represents the hidden variables after the τ+1th iteration optimization of Langevin dynamics, Represents the objective function of the latent variable z at z (τ) The gradient at Repeat the above iterative optimization process, and use the latent variable obtained in the last iteration as the optimized latent variable z′; in the M-step of the expectation-maximization algorithm, input the optimized latent variable and the introduced state variable into the dynamic degradation generator to generate a degradation layer; use the labeled data set and the unlabeled data set in combination with the degradation layer to perform semi-supervised training on the noise predictor and the dynamic degradation generator to obtain a trained noise predictor and a trained dynamic degradation generator; use the inverse denoising process of the diffusion model as a severe weather degraded image restoration model; A severe weather-degraded image to be restored and random noise conforming to a standard Gaussian distribution are simultaneously input into the severe weather-degraded image restoration model. A denoising process of T diffusion time steps obeying a Markov chain is performed through a trained noise predictor to obtain a clean restored image.

2. The method for restoring an image degraded by severe weather based on dynamic degradation generation according to claim 1, characterized in that: Inputting the optimized latent variables and the introduced state variables into the dynamic degradation generator to generate a degradation layer, specifically including: In the i-th dynamic degradation generation process corresponding to the dynamic degradation generator, the i-1th state variable s is introduced i-1 and the i-th optimized latent variable z′ i , s i-1 ~N(0,I) and z′ i ~N(0,I), N represents normal distribution, I represents the identity matrix, ~ represents subject to; The i-1th state variable s i-1 and the i-th optimized latent variable z′ i Input into the transfer model to obtain the i-th state variable s i , as shown below: in, represents the function corresponding to the transfer model, represents the parameters of the transfer model; The i-th state variable s i Input to the fully connected layer and reshape to get the i-th tensor, input the i-th tensor into the emission model to get the i-th degradation layer R i , as shown below: R i =E(S i ;f); Among them, S i represents the i-th tensor, E(·;φ) represents the function corresponding to the emission model, and φ represents the parameters of the emission model.

3. The method for restoring an image degraded by severe weather based on dynamic degradation generation according to claim 1, characterized in that: Semi-supervised training of the noise predictor and the dynamic degradation generator is performed using the labeled dataset and the unlabeled dataset in combination with the degradation layer to obtain a trained noise predictor and a trained dynamic degradation generator, specifically comprising: In the forward noisy process of the diffusion model, a fixed Markov chain is used to gradually perturb the original clean image x0~q(x0) into a noisy image within T diffusion time steps. The perturbation process follows the preset variance scheduling parameters. In the tth diffusion time step, the state transition is defined by the following formula: Among them, q(x t |x t-1 ) represents the conditional transfer probability distribution of the t-th diffusion time step in the forward noise addition process, x t represents the noise image at the t-th diffusion time step, x t-1 represents the noise image at the t-1th diffusion time step, β t represents the variance scheduling parameter of the t-th diffusion time step, t = 1, 2, ..., T, I represents the identity matrix, q(x0) represents the probability distribution of the original clean image, N represents the normal distribution, and ~ represents compliance with; By using the properties of the Markov chain, the noise image x of the t-th diffusion time step can be directly sampled from the original clean image x0 t , as shown below: Among them, q(x t |x0) represents the noise image x obtained by directly sampling the original clean image x0 at the t-th diffusion time step. t The forward diffusion conditional probability distribution of ∈ t ~N(0,I), ~ means subject to, I means the unit matrix, α t =1-β t and represents the cumulative signal retention rate after t diffusion time steps, α t represents the signal retention coefficient at the t-th diffusion time step, α j represents the signal retention coefficient of the jth diffusion time step, j = 1, 2, ..., t, ∈ t represents the noise term; In the inverse denoising process of the diffusion model, from p(x T ) starts sampling, and takes the degraded image y as the condition, gradually performs the denoising sampling steps to restore the clean image. The inverse denoising process constitutes a parameterized Markov chain, and its joint probability distribution can be expressed as: Among them, x 0:T represents the state sequence from the 0th diffusion time step to the Tth diffusion time step, p θ (x 0:T |y) means that x is obtained under the degraded image y 0:T The conditional probability distribution of θ represents the parameters of the noise predictor, p(x T ) represents the noise image x at the T-th diffusion time step T The probability distribution of p(x T )~N(x T ; 0,I), p θ (x t-1 |x t ,y) represents the noise image x at the degraded image y and the t-th diffusion time step t The noise image x of the t-1th diffusion time step is obtained t-1 The conditional probability distribution of ; In the inverse denoising process, the following formula is used to calculate the noise image x from the t-th diffusion time step: t Get the estimated clean image of the skip reconstruction : Among them, ∈ θ (x t ,t) represents the noise image x of the t-th diffusion time step t and the predicted noise obtained by inputting it into the noise predictor at the t-th diffusion time step; The estimated clean image reconstructed by the jump step Linearly superimpose each degraded layer to obtain the estimated degraded image, as shown in the following formula: in, represents the estimated degraded image; The loss function used in the semi-supervised training of the noise predictor and dynamic degradation generator for: Wherein, ‖·‖2 represents the L2 norm, ω represents the parameters of the dynamic degradation generator, if the degraded image comes from the labeled dataset, then h = 1; if the degraded image comes from the unlabeled dataset, then h = 0; f(·; θ) represents the function corresponding to the severe weather degradation image restoration model; G(·; ω) represents the function corresponding to the dynamic degradation generator, z′ represents the optimized latent variable, s0 represents the initial state variable, σ 2 Denotes the variance of the residual term, D KL (·||·) represents the Kullback-Leibler divergence, q(x T |x0) represents the noise image x obtained at the Tth diffusion time step given the original clean image x0 T The posterior probability distribution, p θ (x0|x1) represents the conditional probability distribution of the original clean image x0 under the condition of the noise image x1 at the first diffusion time step, q(x t-1 |x t ,x0) represents the noise image x at the tth diffusion time step t The noise image x of the t-1 diffusion time step is obtained under the condition of the original clean image x0 t-1 The true posterior probability distribution, p θ (x t-1 |x t ) represents the noise image x at the tth diffusion time step t The noise image x of the t-1th diffusion time step is obtained under the condition t-1 The conditional probability distribution of represents the real clean images in the labeled dataset, represents the hyperparameter, represents the expectation of the probability distribution q(·); During each round of training, the E step and the M step of the expectation-maximization algorithm are alternately performed once, and the noise predictor optimized in each round and the dynamic degradation generator optimized in each round are obtained by minimizing the loss function; The noise predictor optimized in the current round and the dynamic degradation generator optimized in the current round are used to perform the E step and the M step of the expectation-maximization algorithm in the next round of training. The above training process is repeated until convergence to obtain a trained noise predictor and a trained dynamic degradation generator.

4. The method for restoring an image degraded by severe weather based on dynamic degradation generation according to claim 1, characterized in that: The transfer model includes a first fully connected layer, a second fully connected layer and a third fully connected layer connected in sequence, wherein the first fully connected layer and the second fully connected layer have a ReLU activation function, and the third fully connected layer has a Tanh activation function.

5. The method for restoring an image degraded by severe weather based on dynamic degradation generation according to claim 1, characterized in that: The emission model includes a first convolutional layer, a second convolutional layer, an upsampling layer and a third convolutional layer connected in sequence, wherein the first convolutional layer, the second convolutional layer and the third convolutional layer all have a ReLU activation function.

6. A device for restoring images degraded by severe weather conditions based on dynamic degradation generation, characterized in that: include: a model building module configured to build a conditional diffusion model and a dynamic degradation generator, wherein the diffusion model includes a forward denoising process and a reverse denoising process, wherein the reverse denoising process uses a noise predictor for iterative denoising, and the dynamic degradation generator includes a transfer model, a fully connected layer, and an emission model connected in sequence; The model training module is configured to construct a labeled data set and an unlabeled data set. During the training process of the noise predictor and the dynamic degradation generator, the E-step of the expectation maximization algorithm is used to optimize the introduced latent variables to obtain the optimized latent variables, specifically including: In the E-step of the expectation-maximization algorithm, the latent variable z is iteratively optimized using Langevin dynamics to obtain the optimized latent variable z′. The iterative optimization process is expressed as follows: where τ represents the τth iteration of Langevin dynamics, δ represents the step size factor, and ξ (τ) represents the Gaussian noise used in the τth iteration of Langevin dynamics, g(z) represents the objective function with respect to the latent variable z, and z (τ) represents the hidden variable after the τth iteration optimization of Langevin dynamics, z (τ+1) represents the hidden variables after the τ+1th iteration optimization of Langevin dynamics, Represents the objective function of the latent variable z at z (τ) The gradient at Repeat the above iterative optimization process, and use the latent variable obtained in the last iteration as the optimized latent variable z′; in the M-step of the expectation-maximization algorithm, input the optimized latent variable and the introduced state variable into the dynamic degradation generator to generate a degradation layer; use the labeled data set and the unlabeled data set in combination with the degradation layer to perform semi-supervised training on the noise predictor and the dynamic degradation generator to obtain a trained noise predictor and a trained dynamic degradation generator; use the inverse denoising process of the diffusion model including the trained noise predictor as the severe weather degraded image restoration model; The restoration module obtains the severe weather-degraded image to be restored and random noise that conforms to the standard Gaussian distribution, and simultaneously inputs them into the severe weather-degraded image restoration model. Through the trained noise predictor, a denoising process of T diffusion time steps obeying the Markov chain is performed to obtain a clean restored image.

7. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images