Image restoration method and system suitable for various severe weathers

Abstract

The invention discloses an image restoration method suitable for various severe weathers, and the method comprises the steps: obtaining a degraded image in the severe weather, inputting the degraded image into a severe image restoration model based on a WaveDiff network for training, cutting the degraded image through a mask, and obtaining a plurality of image blocks; performing discrete wavelet transform on each image block to obtain a plurality of transformed image blocks, performing random noise generation on each transformed image block to obtain a plurality of noise generation results, obtaining a plurality of noise addition results, inputting each noise generation result and the corresponding image block into a generator for prediction to obtain a plurality of generation results, and outputting the plurality of generation results to the generator; sampling each generation result by using a sampling formula to obtain a plurality of de-noising results; and performing inverse discrete wavelet transform on each denoising result to obtain a plurality of restored image blocks, and reconstructing each restored image block to obtain a restored image. According to the method, the degraded images in various severe weathers can be restored.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image restoration method and system applicable to various severe weather conditions. Background Art

[0002] In daily life, bad weather phenomena such as rain, snow and haze are not uncommon, and these weather conditions often have a negative impact on the quality of captured images. At present, many image processing-related downstream task algorithms, such as object detection, object tracking and pedestrian recognition, are usually designed and optimized under ideal weather conditions. However, in practical applications, the occurrence of bad weather can significantly reduce the performance of these algorithms, resulting in their inability to effectively complete the intended tasks.

[0003] Existing image restoration techniques have made significant progress in addressing image degradation issues under multiple types of severe weather. These techniques can be divided into two main categories: single-task severe weather restoration algorithms that focus on a single weather condition and integrated severe weather restoration algorithms that can address different severe weather conditions. Single-task severe weather restoration algorithms are trained on a specific type of weather degradation to restore image degradation under that specific weather condition. While such methods perform well in dealing with image degradation under a single weather type, they are generally unable to cope with the effects of other types of severe weather. Summary of the invention

[0004] In order to solve the above technical problems, the embodiments of the present invention provide an image restoration method and system applicable to various severe weather conditions, thereby restoring degraded images under various severe weather conditions.

[0005] A first aspect of an embodiment of the present invention provides an image restoration method applicable to various severe weather conditions, the method comprising:

[0006] Obtain a degraded image under severe weather conditions, and input the degraded image into a severe image restoration model, so that the severe image restoration model uses a mask to crop the degraded image to obtain a plurality of image blocks;

[0007] Performing discrete wavelet transform on each image block to obtain a plurality of transformed image blocks, performing random noise generation on each transformed image block to obtain a plurality of noise generation results, obtaining a plurality of noise addition results, inputting each noise generation result and the corresponding image block into a generator for prediction to obtain a plurality of generation results, and sampling each generation result using a sampling formula to obtain a plurality of denoising results;

[0008] An inverse discrete wavelet transform is performed on each denoising result to obtain multiple restored image blocks, and each restored image block is reconstructed to obtain a restored image, wherein the severe weather image restoration model is trained based on the WaveDiff network.

[0009] In a possible implementation manner of the first aspect, the severe weather image restoration model is obtained by training based on a WaveDiff network, including:

[0010] A degraded sample image set under bad weather is obtained, and each degraded sample image in the degraded sample image set is cropped to obtain a sample image set;

[0011] Performing wavelet transform on sample images in the sample image set to obtain transformed sample images, and adding noise to the transformed target sample images according to the forward denoising formula of the accelerated diffusion model to obtain a denoising result;

[0012] The noise-added result and the sample image are input into the generator of the accelerated diffusion model for prediction to obtain the prediction result, and the prediction result is sampled using the sampling formula to obtain the sample denoising result;

[0013] If the sample image corresponding to the sample denoising result is a labeled sample image, the supervised loss value is calculated using the adversarial loss function, the perceptual loss function, and the reconstruction loss function. If the sample data corresponding to the denoising result is an unlabeled sample image, the unsupervised loss value is calculated using the adversarial loss function, the L1 loss function, and the contrast loss function. The final loss value is calculated using the supervised loss value and the unsupervised loss value. The parameters of the accelerated diffusion model are adjusted according to the final loss value to obtain the adjusted accelerated diffusion model, and the next sample image set is returned to train the adjusted accelerated diffusion model until the end condition is reached to stop training and obtain the severe weather image restoration model. The calculation formula of the perceptual loss function is:

[0014]

[0015] In the formula, y′ c is the denoising result, y c is the true label, φ j is the activation map of the jth pooling layer;

[0016] The expression of the contrast loss function is:

[0017]

[0018] In the formula, x c is an unlabeled sample image, is the denoising output of the student network for the unlabeled sample image, is a pseudo label, φj is the jth hidden layer, ω j is the weight coefficient.

[0019] In a possible implementation manner of the first aspect, the sample images in the sample image set include labeled sample images and unlabeled sample images.

[0020] The label corresponding to the labeled sample image is a real label, and the label corresponding to the unlabeled sample image is a pseudo label, and the pseudo label is obtained from a pseudo label repository of a teacher network.

[0021] In a possible implementation manner of the first aspect, the pseudo-label repository is obtained by comparing output results of a teacher network and a student network, including:

[0022] Input the labeled sample image into the student network for prediction, obtain the first output result, and use the first output result as the initial pseudo label;

[0023] Input the unlabeled degraded sample image and the corresponding initial pseudo label into the teacher network for prediction to obtain a second output result, and input the unlabeled degraded sample image and the corresponding initial pseudo label into the student network for prediction to obtain a third output result;

[0024] The first output result, the second output result, and the third output result are calculated using a non-reference image quality evaluation index to obtain a first quality score, a second quality score, and a third quality score, and a distance between the second output result and the third output result is calculated, wherein the expression of the non-reference image quality evaluation index is:

[0025]

[0026] Where N is the number of scales, ω i is the weight coefficient of the i-th scale, q i (x) is the quality score calculated at the i-th scale, Q(x) is the quality score;

[0027] If the second quality score is greater than a first preset value and the distance is less than a second preset value, the second output result is determined to be an optimal pseudo label, and the optimal pseudo label is stored in a pseudo label repository, wherein the first preset value is obtained by the first quality score and the third quality score.

[0028] In a possible implementation manner of the first aspect, cropping each degraded sample image in the degraded sample image set includes:

[0029] If the degraded sample image is a labeled sample image, the degraded sample image is randomly cropped; if the degraded sample image is an unlabeled sample image, the center of the degraded sample image is cropped.

[0030] A second aspect of an embodiment of the present invention provides an image restoration system applicable to various severe weather conditions, including an acquisition module, a denoising module and a restoration module.

[0031] The acquisition module is used to acquire a degraded image under severe weather conditions, and input the degraded image into a severe image restoration model, so that the severe image restoration model uses a mask to crop the degraded image to obtain a plurality of image blocks;

[0032] The denoising module is used to perform discrete wavelet transform on each image block to obtain multiple transformed image blocks, generate random noise on each transformed image block to obtain multiple noise generation results, obtain multiple noise addition results, input each noise generation result and the corresponding image block into the generator for prediction to obtain multiple generation results, and sample each generation result using a sampling formula to obtain multiple denoising results;

[0033] The restoration module is used to perform inverse discrete wavelet transform on each denoising result to obtain multiple restored image blocks, and reconstruct each restored image block to obtain a restored image. Among them, the severe weather image restoration model is trained based on the WaveDiff network.

[0034] In a possible implementation manner of the second aspect, the severe weather image restoration model is obtained by training based on a WaveDiff network, including:

[0035] A degraded sample image set under bad weather is obtained, and each degraded sample image in the degraded sample image set is cropped to obtain a sample image set;

[0036] Performing wavelet transform on sample images in the sample image set to obtain transformed sample images, and adding noise to the transformed target sample images according to the forward denoising formula of the accelerated diffusion model to obtain a denoising result;

[0037] The noise-added result and the sample image are input into the generator of the accelerated diffusion model for prediction to obtain the prediction result, and the prediction result is sampled using the sampling formula to obtain the sample denoising result;

[0038] If the sample image corresponding to the sample denoising result is a labeled sample image, the supervised loss value is calculated using the adversarial loss function, the perceptual loss function, and the reconstruction loss function. If the sample data corresponding to the denoising result is an unlabeled sample image, the unsupervised loss value is calculated using the adversarial loss function, the L1 loss function, and the contrast loss function. The final loss value is calculated using the supervised loss value and the unsupervised loss value. The parameters of the accelerated diffusion model are adjusted according to the final loss value to obtain the adjusted accelerated diffusion model, and the next sample image set is returned to train the adjusted accelerated diffusion model until the end condition is reached to stop training and obtain the severe weather image restoration model. The calculation formula of the perceptual loss function is:

[0039]

[0040] In the formula, y′ c is the denoising result, y c is the true label, φ j is the activation map of the jth pooling layer;

[0041] The expression of the contrast loss function is:

[0042]

[0043] In the formula, x c is an unlabeled sample image, is the denoising output of the student network for the unlabeled sample image, is a pseudo label, φ j is the jth hidden layer, ω j is the weight coefficient.

[0044] In a possible implementation manner of the second aspect, the sample images in the sample image set include labeled sample images and unlabeled sample images.

[0045] The label corresponding to the labeled sample image is the real label, and the label corresponding to the unlabeled sample image is the pseudo label, which is obtained from the pseudo label repository of the teacher network.

[0046] A third aspect of an embodiment of the present invention provides a computer device, including:

[0047] Memory for storing computer programs;

[0048] The processor is used to implement the image restoration method applicable to various severe weather conditions as described in the first aspect when executing the computer program.

[0049] A fourth aspect of an embodiment of the present invention provides a storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the image restoration method applicable to various severe weather conditions as in the first aspect are implemented.

[0050] The image restoration method applicable to various severe weather conditions provided by the embodiment of the present invention obtains a degraded image under severe weather conditions, inputs the degraded image into a severe image restoration model obtained by training based on a WaveDiff network, and enables the severe image restoration model to use a mask to crop the degraded image to obtain multiple image blocks; performs discrete wavelet transform on each image block to obtain multiple transformed image blocks, generates random noise on each transformed image block to obtain multiple noise generation results, obtains multiple noise addition results, inputs each noise generation result and the corresponding image block into a generator for prediction to obtain multiple generation results, samples each generation result using a sampling formula to obtain multiple denoising results; performs inverse discrete wavelet transform on each denoising result to obtain multiple restored image blocks, and reconstructs each restored image block to obtain a restored image. Through the above method, it is possible to restore degraded images under various severe weather conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 : A restoration process diagram of an embodiment of an image restoration method applicable to various severe weather conditions provided by the present invention;

[0052] Figure 2 : A schematic diagram of a dense regular grid clipping strategy for an embodiment of an image restoration method applicable to various severe weather conditions provided by the present invention;

[0053] Figure 3 : A schematic diagram of the inference process of a bad image restoration model for an embodiment of an image restoration method applicable to various bad weather conditions provided by the present invention;

[0054] Figure 4 : A schematic diagram of a calculation process of a bad image restoration model for an embodiment of an image restoration method applicable to various bad weather conditions provided by the present invention;

[0055] Figure 5 : A schematic diagram of the inference algorithm process of a bad image restoration model for an embodiment of an image restoration method applicable to various bad weather conditions provided by the present invention;

[0056] Figure 6 : A schematic diagram of the training process of an embodiment of the image restoration method applicable to various severe weather conditions provided by the present invention;

[0057] Figure 7: A schematic diagram of a training algorithm for an embodiment of an image restoration method applicable to various severe weather conditions provided by the present invention;

[0058] Figure 8 : A schematic diagram of a pseudo-label acquisition algorithm process for an embodiment of an image restoration method applicable to various severe weather conditions provided by the present invention;

[0059] Fig. 9 : A schematic diagram showing the visual quality comparison of the image raindrop removal effect on the Raindrop dataset according to an embodiment of the image restoration method applicable to various severe weather conditions provided by the present invention;

[0060] Fig.10 : A schematic diagram showing a visual quality comparison of an image deraining effect on an Outdoor-Rain dataset according to an embodiment of an image restoration method applicable to various severe weather conditions provided by the present invention; Fig.11 : A schematic diagram showing the visual quality comparison of snow removal effects of an image on the Snow00K-2000 dataset according to an embodiment of an image restoration method applicable to various severe weather conditions provided by the present invention; Fig.12 : A schematic diagram of visual quality comparison of image deraining effects on the IVIPC-DQA dataset for an embodiment of an image restoration method applicable to various severe weather conditions provided by the present invention; Fig.13 : A schematic diagram showing the visual quality comparison of snow removal effects of an image on the Snow00K-2000 dataset according to an embodiment of an image restoration method applicable to various severe weather conditions provided by the present invention; Fig.14 : A system block diagram of an embodiment of an image restoration method applicable to various severe weather conditions provided by the present invention. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] Please refer to Figure 1 , is a flow chart of an embodiment of an image restoration method applicable to various severe weather conditions provided by an embodiment of the present invention, including steps S101 to S103, each of which is specifically as follows:

[0063] S101. Obtain a degraded image under severe weather conditions, and input the degraded image into a severe image restoration model, so that the severe image restoration model uses a mask to crop the degraded image to obtain a plurality of image blocks.

[0064] In this embodiment, a degraded image x under bad weather is obtained, and the degraded image x is input into the bad image restoration model. The degraded image x is cropped into K overlapping image patches of 64×64 pixels using a mask. These image patches are cropped according to a dense regular grid with an interval of 4 pixels and collected into a dictionary P. The dense regular grid cropping strategy is as follows: Figure 2 shown.

[0065] This severe image restoration model is a model designed called SemiDDM-Weather, which can solve the image degradation problem caused by various severe weather conditions. The model is based on the WaveDiff network with high-quality image generation capabilities and fast inference speed, and integrates the accelerated DDM (denoising diffusion model) with the standard teacher-student network to restore images affected by different types of severe weather under the framework of semi-supervised learning.

[0066] It should be noted that degraded images refer to images whose quality is reduced in rainy, snowy and foggy weather conditions, and the objects in the images are blurred and difficult to recognize.

[0067] S102, performing discrete wavelet transform on each image block to obtain multiple transformed image blocks, performing random noise generation on each transformed image block to obtain multiple noise generation results, obtaining multiple noise addition results, inputting each noise generation result and the corresponding image block into a generator for prediction to obtain multiple generation results, sampling each generation result using a sampling formula to obtain multiple denoising results, wherein the severe weather image restoration model is obtained by training based on the WaveDiff network.

[0068] In this embodiment, after the image blocks are obtained, these image blocks are transformed by wavelet transform and then input into the student network for denoising to obtain denoising results. After that, the denoising results are subjected to inverse discrete wavelet transform to restore these image blocks. Finally, the restored image blocks are recombined to reconstruct the entire image. It should be noted that for the overlapping areas of the image blocks, an averaging method is applied to ensure visual consistency, and the edges of the overlapping areas are effectively smoothed, thereby achieving a coherent and natural image restoration effect. The reasoning process is as follows Figure 3 As shown in the figure, the algorithm flow of the reasoning process is as follows Figure 4 and Figure 5 As shown in Algorithm 3.

[0069] In some embodiments, in step S102, “the severe weather image restoration model is obtained by training based on the WaveDiff network” includes but is not limited to the following steps:

[0070] A degraded sample image set under bad weather is obtained, and each degraded sample image in the degraded sample image set is cropped to obtain a sample image set;

[0071] Performing wavelet transform on sample images in the sample image set to obtain transformed sample images, and adding noise to the transformed target sample images according to the forward denoising formula of the accelerated diffusion model to obtain a denoising result;

[0072] The noise-added result and the sample image are input into the generator of the accelerated diffusion model for prediction to obtain the prediction result, and the prediction result is sampled using the sampling formula to obtain the sample denoising result;

[0073] If the sample image corresponding to the sample denoising result is a labeled sample image, the supervised loss value is calculated using the adversarial loss function, the perceptual loss function, and the reconstruction loss function. If the sample data corresponding to the denoising result is an unlabeled sample image, the unsupervised loss value is calculated using the adversarial loss function, the L1 loss function, and the contrast loss function. The final loss value is calculated using the supervised loss value and the unsupervised loss value. The parameters of the accelerated diffusion model are adjusted according to the final loss value to obtain the adjusted accelerated diffusion model, and the next sample image set is returned to train the adjusted accelerated diffusion model until the end condition is reached to stop training and obtain the severe weather image restoration model. The calculation formula of the perceptual loss function is:

[0074]

[0075] In the formula, y′ c is the denoising result, y c is the true label, φ j is the activation map of the jth pooling layer;

[0076] The expression of the contrast loss function is:

[0077]

[0078] In the formula, x c is an unlabeled sample image, is the denoising output of the student network for the unlabeled sample image, is a pseudo label, φ j is the jth hidden layer, ω j is the weight coefficient.

[0079] In this embodiment, the degraded image is used as a conditional input and input into the generator of the conditional diffusion model to ensure that the restored image and the degraded input image are consistent in background, but this inevitably consumes more computing resources. In addition, the generator of WaveDiff only uses loss to constrain the generated content, which may not be enough to ensure that the generated restored image is consistent with the input degraded image in background. In response to these problems, a series of specific improvements have been made to the WaveDiff network.

[0080] During training, Figure 6 As shown, first given a degraded image and its corresponding tags If x is labeled data, y represents its corresponding true label. If x is unlabeled data, y represents its corresponding reliability pseudo label. First, crop the image into 64*64 blocks to obtain Then the cropped blocks are subjected to wavelet transform to obtain Add noise to y0 and get the noise results y at step t and step t-1 t and t-1 ,like Figure 7 Next, replace y t The degraded image x0 is input into the generator in WaveDiff to obtain the predicted generated result y′0, which is then used in the sampling formula: t-1 ~q(y′ t-1 ∣∣∣y t ,y′0) to produce the denoised output y′ t-1 ,like Figure 7 As described in lines 10-11 of , these steps are iterated until the entire denoising process is completed to obtain the final generated result.

[0081] It should be noted that choosing to crop a 64×64 pixel patch from the input image instead of using the entire image can effectively reduce the computational burden.

[0082] In order to solve the problem that only using the L1 loss function may lead to insufficient constraints on the content of the generated image, the perceptual loss is introduced to guide the training of the severe weather image restoration model, which can better preserve the texture and structural information of the image. Specifically, the original L1 loss function is expanded into two loss functions: a combination of the L1 loss function and the L2 loss function. By combining the advantages of the L1 loss function in maintaining pixel-level accuracy and the L2 loss function's insensitivity to noise, the content of the generated restoration results can be better constrained. In addition, similar to the traditional semi-supervised learning framework, a consistency constraint is also introduced. This constraint is implemented by using a contrast loss function instead of a traditional loss function, thereby reducing the possibility of the model overfitting to incorrect predictions and reducing the risk of cognitive bias, which can improve the performance and applicability of the severe weather image restoration model.

[0083] According to the final generated result, the loss function is used for calculation to obtain the loss function value. According to the loss function value, the parameters of the initial severe weather image restoration model are adjusted, and the next degraded sample image is returned to train the adjusted initial severe weather image until the end condition is reached and the training is stopped to obtain the severe weather image restoration model. The loss function includes the perceptual loss function and the contrast loss function. The calculation formula of the perceptual loss function is:

[0084]

[0085] In the formula, y′ c is the prediction result of the generator, y c is the true label, φ j is the activation map of the jth pooling layer in VGG-16;

[0086] The expression of the contrast loss function is:

[0087]

[0088] In the formula, x c is a degraded sample image without a label, For student networks for x c The denoised output of is a pseudo label, φ j is the jth hidden layer of the pre-trained VGG-19 network, ω j is the weight coefficient.

[0089] In some embodiments, the “sample image in the sample image set” in step S102 includes but is not limited to the following steps:

[0090] The sample images in the sample image set include labeled sample images and unlabeled sample images.

[0091] The label corresponding to the labeled sample image is the real label, and the label corresponding to the unlabeled sample image is the pseudo label, which is obtained from the pseudo label repository of the teacher network.

[0092] In this embodiment, given a degraded image and its corresponding tags If x is labeled data, y represents its corresponding true label. If x is unlabeled data, y represents its corresponding reliability pseudo label. The reliability pseudo label can be referred to as a pseudo label.

[0093] This pseudo label is the best output of the teacher network. In order to train an integrated severe weather image restoration model to remove multiple severe weather conditions and maintain high visual quality with a limited labeled dataset, a teacher-student network is used for semi-supervised learning. The task of the teacher network is to generate pseudo labels for unlabeled images, which are then used to guide the student network to train with unlabeled images. The weights of the teacher network are updated from the student network using the EMA (exponential moving average) strategy, i.e., θ t =ηθ t +(1-η)θ s , where η∈(0,1) represents the momentum that controls the update rate and is used to smooth the parameters over time.

[0094] In some embodiments, in step S102, “the pseudo-label repository is obtained by comparing the output results of the teacher network and the student network” includes but is not limited to the following steps:

[0095] Input the labeled sample image into the student network for prediction, obtain the first output result, and use the first output result as the initial pseudo label;

[0096] Input the unlabeled degraded sample image and the corresponding initial pseudo label into the teacher network for prediction to obtain a second output result, and input the unlabeled degraded sample image and the corresponding initial pseudo label into the student network for prediction to obtain a third output result;

[0097] The first output result, the second output result, and the third output result are calculated using the non-reference image quality evaluation index to obtain a first quality score, a second quality score, and a third quality score, and the distance between the second output result and the third output result is calculated, wherein the expression of the non-reference image quality evaluation index is:

[0098]

[0099] Where N is the number of scales, ω i is the weight coefficient of the i-th scale, q i(x) is the quality score calculated at the i-th scale, Q(x) is the quality score;

[0100] If the second quality score is greater than the first preset value and the distance is less than the second preset value, the second output result is determined to be the best pseudo label, and the best pseudo label is stored in the pseudo label repository, wherein the first preset value is obtained by the first quality score and the third quality score.

[0101] In this embodiment, considering that the restoration results of the teacher network are not always better than those of the student network, directly using its output as a pseudo-label may bring accuracy problems and may even mislead the training process of the student network. In order to reduce this risk, a reliability "repository" is designed to retain the "optimal" outputs generated by the teacher network during training. In this way, these selected best outputs are used as pseudo-labels to more effectively guide the learning of the student network when processing unlabeled images.

[0102] The parameters of the student network, denoted as θ s , trained by labeled data and unlabeled data. Specifically, the training of the student network can be expressed as minimizing the loss function L = L sup +λ t L unsup , where L sup and L unsup Denote supervised loss and unsupervised loss respectively, t is a time-varying hyperparameter of the trade-off between the two.

[0103] Use the model trained only on labeled data to create initial pseudo-signatures for each unlabeled data and store them in a reliability repository. Figure 8 As shown in Figure 2, in each training iteration, the output of the current teacher network is compared with the output of the student network and the corresponding pseudo-label in the reliability repository. If the current output of the teacher is the best in quality, it is used to update the pseudo-label in the reliability repository, and then the reliability pseudo-label is used to guide the training of the student network on unlabeled degraded images.

[0104] To evaluate the quality of pseudo labels, we use two criteria: (1) We introduce a non-reference image quality evaluation index MUSIQ to evaluate the quality of generated images, where the quality of pseudo labels is proportional to the MUSIQ score, i.e., the better the quality, the higher the score. Figure 8 As shown in the second row of , Q(x) represents the MUSIQ score of input x. The formula for calculating the MUSIQ score is:

[0105]

[0106] Where N represents the number of scales used for calculation, ωi is the weight coefficient of the i-th scale, q i (x) is the image quality score calculated at the i-th scale.

[0107] Specifically, firstly, the labeled degraded sample image is input into the student network for prediction to obtain the first prediction result, and the unlabeled degraded sample image is input into the teacher network and the student network for prediction to obtain the second prediction result and the third prediction result. The first prediction result, the second prediction result and the third prediction result are calculated using the non-reference image quality evaluation index to obtain the first image quality score, the second image quality score and the third image quality score. And the L1 distance between the second prediction result and the third prediction result is calculated.

[0108] If the second image quality score is greater than the maximum of the first image quality score and the third image quality score, and the distance is less than a predefined threshold Then it is determined that the first prediction result is a pseudo label.

[0109] It should be noted that The preferred setting is 0.1.

[0110] In some embodiments, the step S102 of “cropping each degraded sample image in the degraded sample image set” includes but is not limited to the following steps:

[0111] If the degraded sample image is a labeled sample image, the degraded sample image is randomly cropped; if the degraded sample image is an unlabeled sample image, the center of the degraded sample image is cropped.

[0112] In this embodiment, for labeled data, this cropping is performed randomly, but for unlabeled data, we perform center cropping on it and crop it into blocks of size 64*64 to ensure that the cropped content is consistent with the reliability pseudo-label content in the reliability "repository".

[0113] S103, performing inverse discrete wavelet transform on each denoising result to obtain a plurality of restored image blocks, and reconstructing each restored image block to obtain a restored image.

[0114] In this embodiment, after the denoising result is obtained, the denoising result is subjected to an inverse discrete wavelet transform to restore the image blocks, and then the restored image blocks are recombined to reconstruct the entire image. It should be noted that for the overlapping areas between the image blocks, an averaging method is applied to ensure visual consistency, effectively smoothing the edges of these areas, thereby achieving a coherent and natural image restoration effect.

[0115] The present application also conducted a large number of experiments on the technical solution provided in the embodiments of the present application, and selected three data sets, namely, Raildrop, Outdoor-Rai n and Snow100K, for training. Considering that Snow100K contains a large amount of training data and is quite different from the other two data sets, 9,000 images were randomly selected for training, and the three data sets were divided into two, as labeled data and unlabeled data respectively. In addition, a large number of experiments were conducted on the test sets of these three data sets, as well as the real-world rain image data set IVIPC-DQA and the snow image test set Snow100K-Real, to verify the effectiveness of this method. In view of the difficulty of obtaining a large amount of labeled data in the real world, a non-reference image quality evaluation index was selected for objective performance evaluation. Specifically, five non-reference image quality evaluation indicators, namely, PAQ2PIQ, DBCNN, CLIPIQA, MUSIQ-SPAQ and NIQE, were used.

[0116] The experimental objective evaluation results are presented in Tables 1, 2, and 3, respectively, showing the performance of this method on the three datasets Raindrop, Outdoor-Rain, and Snow100K-2000 compared with other SOTA algorithms in terms of objective evaluation indicators. Among them, the best results are highlighted in bold, and the suboptimal results are underlined. Fig. 9 , Fig.10 , Fig.11 , Fig.12 , Fig.13 The comparison of subjective visual quality between this method and other SOTA algorithms is shown. It is obvious from these results that this method can achieve excellent restoration results with limited labeled datasets, even surpassing many supervised methods.

[0117] Table 1 Objective comparison results of different raindrop removal methods on the Raindrop dataset

[0118]

[0119] Table 2 Objective comparison results of different deraining methods on the Outdoor-Rain dataset

[0120]

[0121] Table 3 Objective comparison results of different snow removal methods on the Snow100K-2000 dataset

[0122]

[0123]

[0124] It should be understood that although Figure 1The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0125] In some embodiments, Fig.14 As shown, it shows a block diagram of an image restoration system 1400 applicable to various severe weather conditions provided by an embodiment of the present application, including: an acquisition module 1401, a denoising module 1402 and a restoration module 1403, wherein:

[0126] The acquisition module 1401 is used to acquire a degraded image under severe weather conditions, and input the degraded image into a severe image restoration model, so that the severe image restoration model uses a mask to crop the degraded image to obtain a plurality of image blocks;

[0127] The denoising module 1402 is used to perform discrete wavelet transform on each image block to obtain multiple transformed image blocks, generate random noise on each transformed image block to obtain multiple noise generation results, obtain multiple noise addition results, input each noise generation result and the corresponding image block into the generator for prediction to obtain multiple generation results, and sample each generation result using a sampling formula to obtain multiple denoising results, wherein the severe weather image restoration model is obtained by training based on the WaveDiff network;

[0128] The restoration module 1403 is used to perform inverse discrete wavelet transform on each denoising result to obtain a plurality of restored image blocks, and reconstruct each restored image block to obtain a restored image.

[0129] The specific implementation of the image restoration system applicable to various severe weather conditions is substantially the same as the specific implementation of the image restoration method applicable to various severe weather conditions described above, and will not be described in detail herein.

[0130] In one embodiment of the present application, a computer device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and the above steps are implemented when the processor executes the computer program; the computer device provided in this embodiment has an implementation principle and technical effects similar to those of the above method embodiments, and will not be repeated here.

[0131] In one embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and the above steps are implemented when the computer program is executed by a processor; the computer-readable storage medium provided in this embodiment has an implementation principle and technical effects similar to those of the above method embodiment, and will not be repeated here.

[0132] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0133] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0134] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image restoration method applicable to various severe weather conditions, characterized in that: include: Acquire a degraded image under severe weather conditions, and input the degraded image into a severe image restoration model, so that the severe image restoration model uses a mask to crop the degraded image to obtain a plurality of image blocks; Performing discrete wavelet transform on each of the image blocks to obtain a plurality of transformed image blocks, performing random noise generation on each of the transformed image blocks to obtain a plurality of noise generation results, inputting each of the noise generation results and the corresponding image block into a generator for prediction to obtain a plurality of generation results, and sampling each of the generation results using a sampling formula to obtain a plurality of denoising results, wherein the severe weather image restoration model is obtained by training based on a WaveDiff network; Performing inverse discrete wavelet transform on each of the denoising results to obtain a plurality of restored image blocks, and reconstructing each of the restored image blocks to obtain a restored image.

2. The image restoration method applicable to various severe weather conditions as claimed in claim 1, characterized in that: The severe weather image restoration model is obtained by training based on the WaveDiff network, including: Acquire a degraded sample image set under bad weather conditions, and crop each degraded sample image in the degraded sample image set to obtain a sample image set; Performing wavelet transform on sample images in the sample image set to obtain transformed sample images, and adding noise to the transformed target sample images according to a forward noisy formula of an accelerated diffusion model to obtain a noisy result; Inputting the noise-added result and the sample image into the generator of the accelerated diffusion model for prediction to obtain a prediction result, and sampling the prediction result using a sampling formula to obtain a sample denoising result; If the sample image corresponding to the sample denoising result is a labeled sample image, the supervised loss value is calculated using the adversarial loss function, the perceptual loss function and the reconstruction loss function. If the sample data corresponding to the denoising result is an unlabeled sample image, the unsupervised loss value is calculated using the adversarial loss function, the L1 loss function and the contrast loss function. The final loss value is calculated using the supervised loss value and the unsupervised loss value. The parameters of the accelerated diffusion model are adjusted according to the final loss value to obtain the adjusted accelerated diffusion model, and the adjusted accelerated diffusion model is trained by returning to the next sample image set until the end condition is reached to stop the training and obtain the severe weather image restoration model. The calculation formula of the perceptual loss function is: In the formula, y′ c is the denoising result, y c is the true label, φ j is the activation map of the jth pooling layer; The expression of the contrast loss function is: In the formula, x c is an unlabeled sample image, is the denoising output of the student network for the unlabeled sample image, is a pseudo label, φ j is the jth hidden layer, ω j is the weight coefficient.

3. The image restoration method applicable to various severe weather conditions as claimed in claim 2, characterized in that: The sample images in the sample image set include labeled sample images and unlabeled sample images. The label corresponding to the labeled sample image is a real label, and the label corresponding to the unlabeled sample image is a pseudo label, and the pseudo label is obtained from a pseudo label repository of a teacher network.

4. The image restoration method applicable to various severe weather conditions as claimed in claim 3, characterized in that: The pseudo-label repository is obtained by comparing the output results of the teacher network and the student network, including: Input the labeled sample image into the student network for prediction, obtain the first output result, and use the first output result as the initial pseudo label; Input the unlabeled degraded sample image and the corresponding initial pseudo label into the teacher network for prediction to obtain a second output result, and input the unlabeled degraded sample image and the corresponding initial pseudo label into the student network for prediction to obtain a third output result; The first output result, the second output result, and the third output result are calculated using a non-reference image quality evaluation index to obtain a first quality score, a second quality score, and a third quality score, and a distance between the second output result and the third output result is calculated, wherein the expression of the non-reference image quality evaluation index is: Where N is the number of scales, ω i is the weight coefficient of the i-th scale, q i (x) is the quality score calculated at the i-th scale, Q(x) is the quality score; If the second quality score is greater than a first preset value and the distance is less than a second preset value, the second output result is determined to be an optimal pseudo label, and the optimal pseudo label is stored in a pseudo label repository, wherein the first preset value is obtained by the first quality score and the third quality score.

5. The image restoration method applicable to various inclement weather conditions as claimed in any one of claims 2 to 3, characterized in that: The step of cropping each degraded sample image in the degraded sample image set comprises: If the degraded sample image is a labeled sample image, the degraded sample image is randomly cropped; if the degraded sample image is an unlabeled sample image, the center of the degraded sample image is cropped.

6. An image restoration system suitable for various severe weather conditions, characterized in that: Including acquisition module, denoising module and restoration module, The acquisition module is used to acquire a degraded image under severe weather conditions, and input the degraded image into a severe image restoration model, so that the severe image restoration model uses a mask to crop the degraded image to obtain a plurality of image blocks; The denoising module is used to perform discrete wavelet transform on each of the image blocks to obtain a plurality of transformed image blocks, generate random noise on each of the transformed image blocks using a forward denoising formula to obtain a plurality of noise generation results, input each of the noise generation results and the corresponding image block into a generator for prediction to obtain a plurality of generation results, and sample each of the generation results using a sampling formula to obtain a plurality of denoising results, wherein the severe weather image restoration model is obtained by training based on a WaveDiff network; The restoration module is used to perform inverse discrete wavelet transform on each of the denoising results to obtain a plurality of restored image blocks, and reconstruct each of the restored image blocks to obtain a restored image.

7. The image restoration system applicable to various severe weather conditions as claimed in claim 6, characterized in that: The severe weather image restoration model is obtained by training based on the WaveDiff network, including: Acquire a degraded sample image set under bad weather conditions, and crop each degraded sample image in the degraded sample image set to obtain a sample image set; Performing wavelet transform on sample images in the sample image set to obtain transformed sample images, and adding noise to the transformed target sample images according to a forward noisy formula of an accelerated diffusion model to obtain a noisy result; Inputting the noise-added result and the sample image into the generator of the accelerated diffusion model for prediction to obtain a prediction result, and sampling the prediction result using a sampling formula to obtain a sample denoising result; If the sample image corresponding to the sample denoising result is a labeled sample image, the supervised loss value is calculated using the adversarial loss function, the perceptual loss function and the reconstruction loss function. If the sample data corresponding to the denoising result is an unlabeled sample image, the unsupervised loss value is calculated using the adversarial loss function, the L1 loss function and the contrast loss function. The final loss value is calculated using the supervised loss value and the unsupervised loss value. The parameters of the accelerated diffusion model are adjusted according to the final loss value to obtain the adjusted accelerated diffusion model, and the adjusted accelerated diffusion model is trained by returning to the next sample image set until the end condition is reached to stop the training and obtain the severe weather image restoration model. The calculation formula of the perceptual loss function is: In the formula, y′ c is the denoising result, y c is the true label, φ j is the activation map of the jth pooling layer; The expression of the contrast loss function is: In the formula, x c is an unlabeled sample image, is the denoising output of the student network for the unlabeled sample image, is a pseudo label, φ j is the jth hidden layer, ω j is the weight coefficient.

8. The severe weather image restoration system according to claim 6, characterized in that: The sample images in the sample image set include labeled sample images and unlabeled sample images. The label corresponding to the labeled sample image is a real label, and the label corresponding to the unlabeled sample image is a pseudo label, and the pseudo label is obtained from a pseudo label repository of a teacher network.

9. A computer device, characterized in that: include: Memory for storing computer programs; A processor is used to implement the image restoration method applicable to various severe weather conditions as described in any one of claims 1 to 5 when executing the computer program.

10. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, implements the steps of the image restoration method applicable to various severe weather conditions as described in any one of claims 1 to 5.

Images