A method of measuring and exploiting different degradation correlations to restore an image

Abstract

The application discloses a method for measuring and restoring images by using different degradation correlations. The application can find appropriate auxiliary degradation and mixing ratio. In the case of not changing the model structure and training strategy, the model is trained by using mixed data of anchor point degradation and auxiliary degradation, so as to improve the performance of the model for the anchor point degradation task and improve the ability of restoring visually friendly images. Compared with the traditional method, the strategy does not increase the operation amount of the model and does not increase the training times of the model, and therefore is more suitable for the actual situation that the model running time is strictly required and the hardware condition is difficult to improve. The application can find appropriate anchor point degradation data and auxiliary degradation data ratio in the early stage of training to improve the performance of the model, and can restore visually friendly clean images when facing different degradation low-quality pictures.

Description

Technical Field

[0001] This invention belongs to the field of low-level vision, specifically relating to a method for measuring and utilizing the interrelationships of different degradations to recover images. Background Technology

[0002] Image restoration methods, as an important research approach in the field of low-level vision, aim to recover visually friendly, high-quality images from low-quality images affected by degradation. In reality, the image imaging process is inevitably affected by various types of degradation. Early works often designed specific one-to-one models for a particular degradation, but these methods are usually not applicable to real-world situations. For example, outdoor surveillance systems are often affected by various severe weather conditions (rain, snow, smog, etc.), and using a one-to-one model designed for a specific degradation cannot recover clean surveillance images in all cases; in fact, when the degradation does not match the model, it may even reduce the quality of the surveillance image. Therefore, more and more researchers are turning their attention to multi-degradation research, hoping to use an integrated model to handle multiple degradations. Existing multi-degradation methods achieve this goal by training different degradations together and designing new network structures, objective functions, and training strategies. For example, Hanting Chen proposed a Transformer-based method, IPT, which uses specific head and tail network structures for specific degradations, enabling the network to recover the corresponding clean image when it knows the specific type of image degradation. Boyun Li further proposed a novel all-in-one method based on contrastive learning, AirNet, which can handle multiple types of degradation and recover visually friendly images even without knowing the specific type of image degradation.

[0003] Current integrated image restoration models typically use images with multiple degradation types as input during training, hoping that the model can recover visually friendly clean images even when faced with low-quality images of varying degradation levels. While this approach has achieved good performance in handling multi-degradation problems, it neglects the interactions between different degradation types during model training. This means that while the model can handle multiple degradation types, it may sacrifice performance for a single image restoration task. For example, experiments with the AirNet network indicate that adding rain or haze data during training will reduce the network's denoising performance for denoising tasks. However, this is not always the case. For some image restoration tasks, training with data of multiple degradation types may actually improve the model's performance on that task. For example, for dehazing tasks, adding noisy or rain images during AirNet training improves the model's dehazing performance. Although this is a common problem in multi-degradation methods, no researchers have further explored and revealed the interactions between different degradation types to effectively improve model performance. Therefore, in the context of multi-degradation problems, measuring and utilizing the interactions between different degradation types has high application prospects and practical value. Summary of the Invention

[0004] To address the aforementioned shortcomings in existing technologies, this invention provides a method for measuring and utilizing the interrelationships of different degradations to restore images. This method solves the problem of neglecting the mutual influence between different degradations during model training when processing multi-type degradation data.

[0005] To achieve the aforementioned objective, the technical solution adopted by this invention is as follows: a method for measuring and utilizing the interrelationships of different degradation processes to restore an image, comprising the following steps:

[0006] S1. Obtain the dataset and assign the training step value to 1;

[0007] S2. Sample anchor point degradation data and auxiliary degradation data from the dataset, and obtain mixed degradation data according to the first proportion threshold;

[0008] S3. Update the model parameters based on the anchor point degradation data and the mixed degradation data, and calculate the difference in the rate of decrease in validation loss after the model update;

[0009] S4. Determine if the current training step is greater than the maximum training step. If yes, proceed to S5. If no, increment the training step by 1 and return to S2.

[0010] S5. Calculate the relationship index between anchor degradation and auxiliary degradation based on the difference in the reduction rate of validation loss after model update;

[0011] S6. Obtain hybrid degradation data to improve model performance based on the relationship index between anchor point degradation and auxiliary degradation;

[0012] S7. Update the model parameters based on the anchor point degradation data and the mixed degradation data to improve model performance, and input the dataset into the updated model to obtain the recovered data.

[0013] Further: In S1, the dataset includes anchor point degradation data and auxiliary degradation data corresponding to clean images.

[0014] Furthermore: in S2, the clean images corresponding to the sampled anchor point degradation data and the auxiliary degradation data are the same;

[0015] The specific method for generating mixed degradation data is as follows:

[0016] The anchor point degradation data and the auxiliary degradation data are mixed to obtain the mixed degradation data, wherein the proportion of the auxiliary degradation data is the first proportion threshold.

[0017] Further: S3 includes the following sub-steps:

[0018] S31. Train the model based on the mixed degradation data and the anchor point degradation data, and then update the model parameters to obtain the first updated model parameters and the second updated model parameters.

[0019] S32. Calculate the difference in the rate of decrease of validation loss after model update based on the first updated model parameters and the second updated model parameters.

[0020] Further: In step S31, the first updated model parameters are obtained. The specific expression is:

[0021]

[0022] In the formula, θ t Here are the model parameters, t is the training step number, and X is the training parameter. 1 For anchor point degradation data, Let η be the first loss function, and ...;

[0023] Obtain the second updated model parameters The specific expression is:

[0024]

[0025] In the formula, X 1,2 For mixed degradation data, This is the second loss function.

[0026] Further: In step S32, the difference D in the rate of decrease of the validation loss after the model update is calculated. tThe expression is as follows:

[0027]

[0028] In the formula, For a validation set with anchor point degradation, The first verification loss, For the second verification loss, This is the third verification loss.

[0029] Further: In S5, the expression for calculating the relationship index DRI between anchor point degradation and auxiliary degradation is specifically as follows:

[0030]

[0031] In the formula, D t t represents the difference in the rate of decrease in validation loss after the model update, t is the training step number, and T is the maximum number of training steps.

[0032] Further: In step S6, the method for obtaining hybrid degradation data that improves model performance is specifically as follows:

[0033] Determine whether the value of the indicator for the relationship between anchor point degradation and auxiliary degradation is positive;

[0034] If so, model performance is improved when the proportion of auxiliary degraded data in the mixed degraded data is at the first proportion threshold; otherwise, model performance is impaired when the proportion of auxiliary degraded data in the mixed degraded data is at the first proportion threshold.

[0035] The beneficial effects of this invention are as follows:

[0036] (1) The present invention provides a method for measuring and utilizing the interrelationships of different degradations to restore images. This method can find a suitable auxiliary degradation and mixing ratio. Without changing the model structure and training strategy, the model is trained by using mixed data of anchor degradation and auxiliary degradation, thereby improving the model's performance on anchor degradation tasks and enhancing its ability to recover visually friendly images. Compared with traditional methods, this strategy does not increase the computational load of the model or the number of training iterations. Therefore, it is more suitable for practical situations where there are strict requirements for model runtime and where hardware conditions are difficult to improve.

[0037] (2) This invention does not require fully training the model before calculating the relationship between anchor degradation data and auxiliary degradation data. Experiments show that the relationship between auxiliary degradation and anchor degradation is determined in the early stages of training and does not change significantly in the later stages. Therefore, a suitable ratio of anchor degradation data to auxiliary degradation data can be found in the early stages of training to improve the model's performance, thus enabling the recovery of visually friendly clean images from low-quality images with different degradation levels. Attached Figure Description

[0038] Figure 1 This is a flowchart of a method for measuring and utilizing the interrelationships of different degradations to restore an image according to the present invention.

[0039] Figure 2 This is a graph showing the performance changes during model training in Example 4. Detailed Implementation

[0040] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0041] Example 1:

[0042] like Figure 1 As shown, in one embodiment of the present invention, a method for measuring and utilizing different degradation relationships to restore an image includes the following steps:

[0043] S1. Obtain the dataset and assign the training step value to 1;

[0044] S2. Sample anchor point degradation data and auxiliary degradation data from the dataset, and obtain mixed degradation data according to the first proportion threshold;

[0045] S3. Update the model parameters based on the anchor point degradation data and the mixed degradation data, and calculate the difference in the rate of decrease in validation loss after the model update;

[0046] S4. Determine if the current training step is greater than the maximum training step. If yes, proceed to S5. If no, increment the training step by 1 and return to S2.

[0047] S5. Calculate the relationship index between anchor degradation and auxiliary degradation based on the difference in the reduction rate of validation loss after model update;

[0048] S6. Obtain hybrid degradation data to improve model performance based on the relationship index between anchor point degradation and auxiliary degradation;

[0049] S7. Update the model parameters based on the anchor point degradation data and the mixed degradation data to improve model performance, and input the dataset into the updated model to obtain the recovered data.

[0050] In this embodiment, the composition of the mixed degenerate data has a significant impact on model performance. When the proportion of auxiliary degenerate data is too large, the model focuses too much on learning how to handle auxiliary degeneracy while neglecting anchor point degeneracy, leading to a sharp decline in model performance. Conversely, when the proportion of auxiliary degenerate data is too small, the model performance does not change much. To find an appropriate ratio of anchor point degeneracy to auxiliary degeneracy in the mixed data to improve model performance, this invention proposes a degradation ratio determination strategy to evaluate the performance of the mixed degenerate data composed of auxiliary degenerate data under a first ratio threshold.

[0051] In step S1, the dataset includes anchor point degradation data and auxiliary degradation data corresponding to clean images.

[0052] In this embodiment, a clean image may correspond to multiple types of degraded images. This invention divides degraded images into anchor point degraded data and auxiliary degraded data. Auxiliary degraded data is degraded data that helps the model improve its performance in handling anchor point degraded data.

[0053] In S2, the clean images corresponding to the sampled anchor point degradation data and auxiliary degradation data are the same;

[0054] The specific method for generating mixed degradation data is as follows:

[0055] The anchor point degradation data and the auxiliary degradation data are mixed to obtain the mixed degradation data, wherein the proportion of the auxiliary degradation data is the first proportion threshold.

[0056] In this embodiment, the composition of the mixed degraded data is determined by a first proportion threshold, thereby better evaluating and improving model performance.

[0057] S3 includes the following steps:

[0058] S31. Train the model based on the mixed degradation data and the anchor point degradation data, and then update the model parameters to obtain the first updated model parameters and the second updated model parameters.

[0059] S32. Calculate the difference in the rate of decrease of validation loss after model update based on the first updated model parameters and the second updated model parameters.

[0060] In this embodiment, the model is trained using a mixture of degenerate data and anchor point degenerate data. The changes in model performance can be reflected by the changes in the loss value during training. During model training, when auxiliary degradation can improve model performance, the loss value of the model trained with mixed degenerate data decreases faster than that of the model trained with only anchor point degenerate data. At the same time, in order to avoid the effects of overfitting and to reflect the robustness of the model, the loss value should not be calculated from the training data, but should be obtained from the validation data. Therefore, this invention calculates the difference in the rate of decrease of the validation loss after the model update by using the validation loss. When the rate of decrease of the validation loss value of the model trained with mixed degenerate data is faster than that trained with only anchor point degenerate data, then auxiliary degradation helps to improve model performance.

[0061] In step S31, the first updated model parameters are obtained. The specific expression is:

[0062]

[0063] In the formula, t t Here are the model parameters, t is the training step number, and X is the training parameter. 1 For anchor point degradation data, Let η be the first loss function and η be the learning rate.

[0064] Obtain the second updated model parameters The specific expression is:

[0065]

[0066] In the formula, X 1, 2 represents mixed degradation data. This is the second loss function.

[0067] In step S32, the difference D in the rate of decrease of the validation loss after the model update is calculated. t The expression is as follows:

[0068]

[0069] In the formula, For a validation set with anchor point degradation, The first verification loss, For the second verification loss, This is the third verification loss.

[0070] In this embodiment, the difference D in the rate of decrease of the validation loss after the model update t If positive, then at training step t, the model uses the mixed degenerate data X. 1,2 The validation loss obtained after training decreases faster than the model using only anchor degenerate data X.1 The training results show that auxiliary degenerate data has a positive effect on the model's handling of anchor tasks, while auxiliary degenerate data has a negative inhibitory effect on the model's handling of anchor tasks.

[0071] In step S6, the method for obtaining hybrid degradation data that improves model performance is as follows:

[0072] Determine whether the value of the indicator for the relationship between anchor point degradation and auxiliary degradation is positive;

[0073] If so, model performance is improved when the proportion of auxiliary degraded data in the mixed degraded data is at the first proportion threshold; otherwise, model performance is impaired when the proportion of auxiliary degraded data in the mixed degraded data is at the first proportion threshold.

[0074] In S5, the expression for calculating the relationship index DRI between anchor point degradation and auxiliary degradation is as follows:

[0075]

[0076] In the formula, D t t represents the difference in the rate of decrease in validation loss after the model update, t is the training step number, and T is the maximum number of training steps.

[0077] In this embodiment, when the relationship index DRI between anchor point degradation and auxiliary degradation is positive, it indicates that the auxiliary degradation data has a positive effect on the anchor point degradation data as a whole, which can effectively enhance the model's ability to handle anchor point degradation. Conversely, it indicates that the anchor point degradation data has an inhibitory effect on the auxiliary degradation data.

[0078] Example 2:

[0079] This example explores the impact of the proportion of auxiliary degradation on model performance:

[0080] We set the anchor point degradation to haze and used noise as an auxiliary degradation factor to explore the impact of the auxiliary degradation ratio on model performance. During model training, the input was a mixture of haze and noise data, while during testing, the input was only a haze image. The auxiliary degradation ratio r was set to increase from 0.1 to 0.9 with a spacing of 0.2.

[0081] PSNR, or Peak Signal-to-Noise Ratio, is an objective standard for evaluating images. By quantifying the PSNR of the restored image and the clean image, the performance of image restoration algorithms can be directly compared. The specific calculation process for PSNR is shown below:

[0082] Given a clear image I of size m×n and a noisy image K, the mean square error (MSE) is defined as:

[0083]

[0084] Therefore, the expression for PSNR can be obtained:

[0085]

[0086] in, It is the square of the maximum possible pixel value of the image. For data stored in uint8 format, the maximum pixel value is 255, and for data stored in floating-point type, the maximum pixel value is 1.0.

[0087] SSIM stands for Structural Similarity, a metric that quantifies the degree of structural similarity between two images. If the two images are a restored image and a clean image, the SSIM algorithm can be used to quantify the performance of image restoration algorithms and to determine the quality of the restored image.

[0088] The SSIM formula is based on three comparison measures between image I and K: brightness, contrast, and structure. The following formula is given:

[0089]

[0090]

[0091]

[0092] Where: μ I K is the mean of I; K Let K be the mean; Let I be the variance; Let σ be the variance of K; IK Let c1 be the covariance of I and K; c1 = (k1L) 2 c2=(k2L) 2 To avoid division by zero for two constants; L is the range of pixel values, L = 2 B -1, k1 = 0.01; k2 = 0.03. Therefore, we obtain the definition of SSIM:

[0093] SSIM(I,K)=[l(I,K) α ·c(I,K) β ·s(x,y) γ ]

[0094] Setting α, β, and γ to 1, we get:

[0095]

[0096] The experiment revealed the relationship between anchor point degradation and auxiliary degradation, as well as the changes in the indices DRI, PSNR, and SSIM. The specific results are shown in Table 1.

[0097] Table 1

[0098] r DRI PSNR ΔPSNR SSIM ΔSSIM 0％ 0 33.84 0 0.9849 0 10％ 0.00090 33.96 0.12 0.9850 0.0001 30％ -0.00301 33.00 -0.84 0.9828 -0.0021 50％ -0.01222 32.81 -1.03 0.9827 -0.0022 70％ -0.02784 32.10 -1.74 0.9804 -0.0045 90％ -0.05140 30.96 -2.88 0.9764 -0.0085

[0099] As shown in Table 1, the auxiliary degradation ratio is crucial to model performance. Only by setting an appropriate auxiliary degradation ratio *r* can auxiliary degradation help improve model performance. For example, when the noise ratio is 10%, a positive DRI improves the model performance by 0.12 dB in PSNR and 0.0001 in SSIM. However, when the auxiliary degradation ratio is inappropriate, model performance is severely affected. In particular, when noise data accounts for 90%, the model performance decreases by -2.88 dB in PSNR and -0.0085 in SSIM. Furthermore, DRI is highly correlated with the final model performance. A positive DRI always indicates improved model performance, while a negative DRI indicates a decline in performance.

[0100] Example 3:

[0101] This example explores the impact of different auxiliary degradation methods on anchor point degradation:

[0102] In Example 2, when the auxiliary degradation ratio is 10%, adding noise degradation to the training data helps improve the network's ability to process haze. Therefore, this example adds two additional degradations, rain and snow, and fixes their ratio at 10% to observe the changes in DRI and network performance. The specific results are shown in Table 2.

[0103] Table 2

[0104] Setting Original 10% Noise 10% Rain 10% Snow DRI 0 0.00090 0.00093 0.00091 PSNR 33.84 33.96 34.76 34.50 SSIM 0.9849 0.9850 0.9857 0.9851

[0105] As shown in Table 2, noise, rain, and snowflakes, when in appropriate proportions, can all help improve the model's ability to handle smog, and the higher the DRI, the greater the improvement in the model's performance.

[0106] Example 4:

[0107] This embodiment is to verify whether the hybrid degradation data proposed in this invention can universally improve the performance of various models.

[0108] In this embodiment, to verify whether the method proposed in this invention has generalization ability, seven representative models were selected as baselines, and the performance changes after training with a suitable hybrid model were observed. The specific results are as follows: Figure 2As shown in the diagram. Indoor and Outdoor represent two subsets of the dataset; Indoor represents indoor scenes, and Outdoor represents outdoor scenes. Original indicates that the model was trained using only fog-clean image pairs. Noise, Rain, and Snow represent the addition of noise-clean image pairs, rain-clean image pairs, and snowflake-clean image pairs to the training data, respectively. 10% indicates that the added images comprise 10% of the total training data. The remaining text in the first line represents the model name.

[0109] Using mixed data significantly improved model performance. For indoor scenes, adding 10% noise, rain, and snow to the training data improved PSNR by an average of 0.48 dB, 0.83 dB, and 0.62 dB, respectively, and SSIM by an average of 0.0023, 0.0039, and 0.0029, respectively. For more complex outdoor scenes, this approach also effectively improved model performance, increasing PSNR by 0.36 dB, 0.71 dB, and 0.64 dB, respectively, and SSIM by 0.0009, 0.0014, and 0.0012, respectively.

[0110] The beneficial effects of this invention are as follows: This invention provides a method for measuring and utilizing the interrelationships of different degradation processes to restore images. It can find suitable auxiliary degradation and blending ratios. Without changing the model structure or training strategy, it trains the model using mixed data of anchor point degradation and auxiliary degradation, thereby improving the model's performance on anchor point degradation tasks and enhancing its ability to recover visually friendly images. Compared to traditional methods, this strategy neither increases the model's computational load nor the number of training iterations, making it more suitable for situations with strict requirements on model runtime and where hardware improvements are difficult.

[0111] This invention eliminates the need to fully train the model before calculating the relationship between anchor degradation data and auxiliary degradation data. Experiments show that the relationship between auxiliary degradation and anchor degradation is already determined in the early stages of training and does not change significantly in the later stages. Therefore, by finding an appropriate ratio of anchor degradation data to auxiliary degradation data in the early stages of training, the model's performance can be improved, enabling the recovery of visually appealing clean images from low-quality images with varying degrees of degradation.

[0112] In the description of this invention, it should be understood that the terms "center," "thickness," "upper," "lower," "horizontal," "top," "bottom," "inner," "outer," and "radial," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying the relative importance or the number of technical features implicitly specified. Therefore, a feature defined by "first," "second," and "third" may explicitly or implicitly include one or more of that feature.

Claims

1. A method of measuring and restoring an image using different degradation correlations, characterized by, The method comprises the following steps: S1, obtaining a data set, and setting a training step to 1; S2, sampling anchor point degradation data and auxiliary degradation data from the data set, mixing the anchor point degradation data and the auxiliary degradation data according to a first proportion threshold to obtain mixed degradation data, wherein the proportion of the auxiliary degradation data is the first proportion threshold; S3, updating model parameters according to the anchor point degradation data and the mixed degradation data, and calculating a difference value of a verification loss reduction rate of the updated model; S4, determining whether the current training step is greater than a maximum training step, if yes, proceeding to S5, and if no, adding 1 to the training step and returning to S2; S5, calculating a relationship index between anchor point degradation and auxiliary degradation according to the difference value of the verification loss reduction rate of the updated model; S6, obtaining mixed degradation data for improving model performance according to the relationship index between anchor point degradation and auxiliary degradation; S7, updating model parameters according to the anchor point degradation data and the mixed degradation data for improving model performance, inputting the data set into the updated model, and obtaining recovered data; The S3 comprises the following steps: S31, training a model according to the mixed degradation data and the anchor point degradation data, updating model parameters, and obtaining first updated model parameters and second updated model parameters; S32, calculating a difference value of a verification loss reduction rate of the updated model according to the first updated model parameters and the second updated model parameters; In the S31, the first updated model parameter is obtained The expression is specifically: wherein is a model parameter, t is a training step number, is anchor degenerate data, is a first loss function, is a learning rate; obtaining a second updated model parameter The expression of the second updated model parameter is specifically: In the formula, is mixed degenerate data, is a second loss function; In the S32, the difference of the validation loss reduction rate of the updated calculation model is calculated The expression is specifically: In the formula, is a verification set with anchor degradation, is a first verification loss, is a second verification loss, is a third verification loss; In the S5, a relationship index between the anchor degradation and the auxiliary degradation is calculated The expression of the relationship index is specifically: In the formula, is the difference of the validation loss reduction rate after model update, t is the training step number, T is the maximum training step.

2. The method of claim 1 wherein, In the S1, the data set comprises anchor point degradation data and auxiliary degradation data corresponding to clean pictures.

3. The method of claim 1 wherein the different degradation relationships are determined by, In the S6, the method for obtaining the mixed degradation data for improving model performance is specifically: determining whether the value of the relationship index between anchor point degradation and auxiliary degradation is a positive number; if yes, the mixed degradation data improves model performance when the proportion of the auxiliary degradation data is the first proportion threshold; and if no, the mixed degradation data damages model performance when the proportion of the auxiliary degradation data is the first proportion threshold.

Images