Image restoration method and device, electronic equipment and storage medium

Abstract

The invention provides an image restoration method and device, electronic equipment and a storage medium, and relates to the field of computer vision. The method comprises the steps that a training image set is acquired, the training image set comprises a plurality of training samples corresponding to various severe weathers, and each training sample comprises a training image collected in the severe weathers and a clean reference image corresponding to the training image; and training a neural network model by using the training image set until a training stop condition is satisfied, and obtaining an image recovery model which is used for recovering an image collected in severe weather. When the neural network model is trained, a weather prompt is determined according to the weather degradation feature prompt pool, a scene prompt is determined according to the scene features, then a comparison prompt loss value is determined, finally the total loss value of the neural network model is determined, model training is performed according to the total loss value, and then an image restoration model is obtained. The image restoration model can cope with various weather degradation combinations, and the accuracy of image restoration is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method, device, electronic equipment and storage medium for image restoration. Background Art

[0002] In order to reduce the impact of bad weather on image acquisition, images acquired under bad weather (which may be referred to as bad weather images) may be restored.

[0003] At present, the transformer method is generally used to restore severe weather images. For example, TransWeather is a transformer-based image restoration library. TransWeather introduces a weather type decoder that can explain multiple weather degradations. However, when restoring severe weather images, TransWeather uses a fixed query method, and each severe weather corresponds to a fixed restoration method, resulting in low accuracy in severe weather image restoration. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for image restoration, so as to improve the restoration accuracy of images collected in severe weather.

[0005] In a first aspect, an embodiment of the present application provides a method for image restoration, the method comprising:

[0006] Acquire a training image set, the training image set including multiple training samples corresponding to multiple types of severe weather conditions, each training sample including a training image collected in severe weather conditions, and a clean reference image corresponding to the training image;

[0007] The neural network model is trained using the training image set until the training stop condition is met, thereby obtaining an image restoration model, which is used to restore images collected in bad weather;

[0008] Wherein, when the neural network model is trained using the i-th training sample in the training image set of the neural network model, the following steps are included:

[0009] Determine the weather prompt corresponding to the i-th training image according to the weather degradation feature prompt pool and the degradation residual of the i-th training image of the i-th training sample; the weather degradation feature prompt pool includes sub-prompts of multiple weather degradation features for characterizing multiple severe weather conditions; the degradation residual is the residual of the i-th training image minus the clean reference image corresponding to the i-th training image;

[0010] Determine the scene prompt of the i-th training image according to the scene feature of the i-th training image;

[0011] According to the weather prompt and the scene prompt, determine the contrast prompt loss value of the i-th training image; the contrast prompt loss value is used to characterize the distinction between the weather prompt and the scene prompt;

[0012] Determine the total loss value of the neural network model corresponding to the i-th training image according to the comparison prompt loss value and the noise loss value of the training image;

[0013] If the total loss value does not meet the training stop condition, adjust the model parameters of the neural network model, and use the i+1th training image in the training image set to train the adjusted neural network model until the total loss value of the neural network model meets the training stop condition to obtain an image restoration model, where i is a positive integer.

[0014] In some embodiments, determining a total loss value of the neural network model corresponding to the i-th training image based on the comparison hint loss value and the noise loss value of the training image includes:

[0015] Determine a model restoration image corresponding to the ith training image according to the degradation residual and the ith training image;

[0016] Determine the peak signal-to-noise ratio of the model restored image and the clean reference image corresponding to the i-th training image;

[0017] The total loss value of the neural network model corresponding to the i-th training image is determined according to the contrast cue loss value, the noise loss value of the training image, and the peak signal-to-noise ratio.

[0018] In some embodiments, the total loss value satisfies formula (1):

[0019]

[0020] Among them, L total represents the total loss value, λ1, λ2, λ3 represent weights, L res Indicates the noise loss value, L cp Indicates the comparison prompt loss value, represents the degraded residual in the restoration process, y represents the i-th training image, x represents the clean reference image corresponding to the i-th training image, Represents the model recovery image, Indicates peak signal-to-noise ratio.

[0021] In some embodiments, determining a weather prompt corresponding to the i-th training image according to the weather degradation feature prompt pool and the degradation residual of the i-th training image of the i-th training sample includes:

[0022] Convert the degraded residual of the ith training image of the ith training sample into the embedded features of the ith training image;

[0023] For each sub-cue in the weather degradation feature cue pool, determine the similarity between the sub-cue and the embedded feature of the i-th training image;

[0024] Based on the order of similarity from high to low, m sub-prompts are selected, where m is a positive integer;

[0025] The m sub-cues are concatenated to obtain the weather prompt corresponding to the i-th training image.

[0026] In some implementations, the scene prompt satisfies formula (2):

[0027]

[0028] Among them, P gd In the case where the scene features are deep features, Q g A query function representing the initial scene prompt, The key representing the transposed deep feature, V d represents the value of the deep feature, and D represents the feature dimension of the initial scene prompt.

[0029] In some implementations, determining the contrast cue loss value of the i-th training image according to the weather cue and the scene cue includes:

[0030] When the scene feature is a depth feature, obtaining a mean value of the depth feature;

[0031] Determine a first similarity between a target key and a mean of the deep features, the target key being a key of a scene prompt formed by the deep feature constraints;

[0032] Obtaining the second similarity between the key of each sub-prompt of the weather prompt and the target key respectively;

[0033] Based on the first similarity, the second similarity between the key of each sub-cue and the target key, the contrast cue loss value of the i-th training image is determined.

[0034] In some embodiments, the comparison prompt loss value satisfies formula (3):

[0035]

[0036] Among them, L CP represents the comparison prompt loss value, b represents the batch parameter, m represents the preset number, K gd Keys representing scene cues formed by deep feature constraints, represents the mean of deep features, Represents the key of the i-th subprompt.

[0037] In a second aspect, an embodiment of the present application provides an image restoration device, the device comprising:

[0038] An acquisition module is used to acquire a training image set, the training image set includes a plurality of training samples corresponding to various types of severe weather conditions, each training sample includes a training image collected in severe weather conditions, and a clean reference image corresponding to the training image;

[0039] A training module is used to train a neural network model using a training image set until a training stop condition is met to obtain an image restoration model, and the image restoration model is used to restore images collected in bad weather;

[0040] Wherein, when the neural network model is trained using the i-th training sample in the training image set of the neural network model, the following steps are included:

[0041] Determine the weather prompt corresponding to the i-th training image according to the weather degradation feature prompt pool and the degradation residual of the i-th training image of the i-th training sample; the weather degradation feature prompt pool includes sub-prompts of multiple weather degradation features for characterizing multiple severe weather conditions; the degradation residual is the residual of the i-th training image minus the clean reference image corresponding to the i-th training image;

[0042] Determine the scene prompt of the i-th training image according to the scene feature of the i-th training image;

[0043] Determine the contrast prompt loss value of the i-th training image according to the weather prompt and the scene prompt; the contrast prompt loss value is used to characterize the degree of distinction between the weather prompt and the scene prompt;

[0044] Determine the total loss value of the neural network model corresponding to the i-th training image according to the comparison prompt loss value and the noise loss value of the training image;

[0045] If the total loss value does not meet the training stop condition, adjust the model parameters of the neural network model, and use the i+1th training image in the training image set to train the adjusted neural network model until the total loss value of the neural network model meets the training stop condition to obtain an image restoration model, where i is a positive integer.

[0046] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0047] a memory configured to store instructions; and

[0048] The processor is configured to call instructions from the memory and implement the image recovery method provided in the first aspect of the embodiment of the present application when executing the instructions.

[0049] In a fourth aspect, an embodiment of the present application provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the above-mentioned image restoration method.

[0050] In an embodiment of the present application, a training image set is obtained, and the training image set includes multiple training samples corresponding to various severe weather conditions, each training sample includes a training image collected in severe weather, and a clean reference image corresponding to the training image. According to the weather degradation feature prompt pool and the degradation residual of the i-th training image of the i-th training sample, the weather prompt corresponding to the i-th training image is determined. Then, according to the scene features of the i-th training image, the scene prompt of the i-th training image is determined. After obtaining the weather prompt and the scene prompt, the contrast prompt loss value of the i-th training image can be determined according to the weather prompt and the scene prompt, and the contrast prompt loss value is used to characterize the distinction between the weather prompt and the scene prompt. Then, according to the contrast prompt loss value and the noise loss value of the training image, the total loss value of the neural network model corresponding to the i-th training image is determined. If the total loss value does not meet the training stop condition, the model parameters of the neural network model are adjusted, and the adjusted neural network model is trained using the i+1th training image in the training image set until the total loss value of the neural network model meets the training stop condition, and an image restoration model is obtained, and the image restoration model is used to restore images collected in severe weather.

[0051] In this way, the image restoration model is obtained by training according to the training samples corresponding to various severe weather conditions, and the degradation residual of the training image is trained during the model training to combine the sub-prompts that characterize the weather degradation characteristics to obtain the weather prompt. Combined with the scene prompt, the contrast prompt loss value and the total loss value of the training image are determined. Finally, the model parameters of the neural network model are adjusted according to the total loss value to obtain the image restoration model, so that the image restoration model can flexibly cope with a variety of complex and unpredictable weather degradation combinations, thereby improving the accuracy of image restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flowchart of the method for image restoration provided by an embodiment of the present application;

[0053] Figure 2 is a schematic diagram of an image restoration method under adverse weather conditions provided by a specific embodiment of the present application;

[0054] Figure 3 is a schematic diagram of the structure of an image restoration device provided in an embodiment of the present application;

[0055] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0057] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0058] The following, in conjunction with the accompanying drawings, describes in detail the method, device, electronic device and storage medium for image restoration provided by the embodiments of the present application through specific embodiments and their application scenarios.

[0059] See also Figure 1 , is a flow chart of an image restoration method provided in an embodiment of the present application, and the method is applied to an electronic device. Figure 1 As shown, the image restoration method includes the following steps S100 to S200.

[0060] Step S100: Acquire a training image set, where the training image set includes multiple training samples corresponding to various types of severe weather conditions, each training sample including a training image collected in severe weather conditions, and a clean reference image corresponding to the training image.

[0061] In the embodiment of the present application, the training image set can be understood as a collection of images used to train the neural network model. The training image set may include multiple training samples corresponding to multiple severe weather conditions. For each severe weather condition, the training image set may include multiple corresponding training samples. Each training sample includes a training image collected in severe weather conditions and a clean reference image corresponding to the training image. The training image may be referred to as a weather-degraded image or a degraded image.

[0062] Step S200: Train the neural network model using the training image set until the training stop condition is met, and obtain an image restoration model, which is used to restore images collected in bad weather.

[0063] In an embodiment of the present application, after obtaining the training image set, the neural network model is trained using the training image set until the training stop condition is met. The training stop condition can be understood as a judgment condition when the training of the neural network model is completed. In one embodiment, the training stop condition can be that the total loss value of the neural network model corresponding to the i-th training image is less than a preset value.

[0064] After the training of the neural network model is completed and the training stop condition is met, an image restoration model can be obtained. The image restoration model can be understood as a model used to restore images captured in bad weather. Images captured under bad weather conditions are usually affected by various factors, such as raindrops, haze, and snowflakes, which can significantly reduce the quality of the image. Therefore, the image restoration model obtained through training can be used to improve the clarity of images captured in bad weather.

[0065] In an embodiment of the present application, when training a neural network model using the i-th training sample in the training image set of the neural network model, the following steps are included:

[0066] According to the weather degradation feature prompt pool and the degradation residual of the i-th training image of the i-th training sample, the weather prompt corresponding to the i-th training image is determined; the weather degradation feature prompt pool includes sub-prompts of multiple weather degradation features for characterizing multiple severe weathers; the degradation residual is the residual of the i-th training image minus the clean reference image corresponding to the i-th training image.

[0067] According to the scene feature of the ith training image, the scene prompt of the ith training image is determined.

[0068] According to the weather prompts and scene prompts, the contrast prompt loss value of the i-th training image is determined; the contrast prompt loss value is used to characterize the distinction between weather prompts and scene prompts.

[0069] Determine the total loss value of the neural network model corresponding to the i-th training image according to the comparison prompt loss value and the noise loss value of the training image;

[0070] If the total loss value does not meet the training stop condition, adjust the model parameters of the neural network model, and use the i+1th training image in the training image set to train the adjusted neural network model until the total loss value of the neural network model meets the training stop condition to obtain an image restoration model, where i is a positive integer.

[0071] Specifically, the weather degradation feature prompt pool is composed of multiple sub-cues. Each sub-cue in the weather degradation feature prompt pool represents a specific weather degradation feature. The weather degradation feature can be understood as a feature or pattern that appears when weather conditions deteriorate. The degradation residual of the i-th training image can be understood as the residual obtained by subtracting the clean reference image corresponding to the i-th training image from the i-th training image.

[0072] Specifically, the weather prompts adapted to the weather conditions in the i-th training image can be obtained by autonomously selecting and combining the various sub-prompts in the weather degradation feature prompt pool according to the degradation residual of the i-th training image of the i-th training sample. By autonomously selecting and combining the sub-prompts in the weather degradation feature prompt pool, specific weather prompts can be constructed that can flexibly respond to a variety of weather degradation conditions to cope with unforeseen weather inputs.

[0073] In one example, 20 sub-cues are designed in the weather degradation feature prompt pool, each of which consists of 64 tokens and each of which has a learnable key. Through the query-key value mechanism, the similarity between the embedded features and the key is calculated, and the top 5 sub-cues with the highest similarity are selected to form the final weather prompt, and the sub-cues that meet the preset conditions are selected to construct the weather prompt. For example, the diversity and representativeness of the prompts can be ensured by selecting the top 5 sub-cues with the highest similarity to form the final weather prompt.

[0074] In some embodiments, the reconstruction target can be set as the degraded residual, and the total loss value can be calculated based on the degraded residual to achieve training of the neural network model.

[0075] In some embodiments, a prompt generation method based on a generative adversarial network (GAN) or a variational autoencoder (VAE) may also be used to generate prompts to further improve the diversity and adaptability of the prompts.

[0076] Specifically, in order to improve the accuracy of scene modeling, corresponding scene prompts can also be constructed according to the scene features of the i-th training image during the model training process. In one example, the scene features may include but are not limited to at least one of depth features, semantic segmentation features, and optical flow features. By introducing scene features, the model can better understand and restore background information in complex scenes, thereby improving the accuracy of image restoration.

[0077] Specifically, after obtaining the weather prompt and scene prompt corresponding to the i-th training image, the contrast prompt loss value of the i-th training image is determined according to the weather prompt and the scene prompt. The contrast prompt loss value can be understood as a value used to characterize the discrimination between the weather prompt and the scene prompt.

[0078] Specifically, in one example, when determining the contrast cue loss value of the i-th training image, a graph convolutional network (GCN) or a contrast learning method based on a self-attention mechanism can be used to determine the distinction between weather cues and scene cues, and then obtain the contrast cue loss value of the i-th training image.

[0079] In another example, when determining the contrast cue loss value of the i-th training image, a mutual push-pull strategy can also be adopted, that is, the similarity between the weather cues and the background cues is calculated using cosine similarity, and based on the similarity between the weather cues and the background cues, the two cues are moved away from each other in the high-dimensional feature space through contrast learning, thereby obtaining the contrast cue loss value of the i-th training image.

[0080] Specifically, noise loss is a loss function used to estimate and reduce the impact of noise in model input or prediction. The noise loss value can be used to optimize model parameters so that the model can better cope with noise in the data, thereby improving the stability and accuracy of the prediction. Noise can include but is not limited to Gaussian noise and mixed noise distribution.

[0081] Specifically, the contrast cue loss value can be used to optimize the representation of the cue, and the noise loss value can be used to optimize the parameters of the model. According to the contrast cue loss value and the noise loss value of the training image, the total loss value of the neural network model corresponding to the i-th training image can be determined. In one embodiment, the total loss value can be a weighted combination of the contrast cue loss value, the noise loss value of the training image, etc.

[0082] Specifically, when the total loss value corresponding to the i-th training image does not meet the training stop condition, the model parameters of the neural network model are adjusted, and the adjusted neural network model is trained using the i+1-th training image in the training image set until the total loss value of the neural network model meets the training stop condition. When the total loss value meets the training stop condition, the neural network model completes the training to obtain the image restoration model. The image restoration model can be used to restore images collected in bad weather.

[0083] Through the above steps S100 to S200, a training image set is obtained, and the training image set includes multiple training samples corresponding to various severe weather conditions, each training sample includes a training image collected in severe weather, and a clean reference image corresponding to the training image. According to the weather degradation feature prompt pool and the degradation residual of the i-th training image of the i-th training sample, the weather prompt corresponding to the i-th training image is determined. Then, according to the scene feature of the i-th training image, the scene prompt of the i-th training image is determined. After obtaining the weather prompt and the scene prompt, the contrast prompt loss value of the i-th training image can be determined according to the weather prompt and the scene prompt, and the contrast prompt loss value is used to characterize the distinction between the weather prompt and the scene prompt. Then, according to the contrast prompt loss value and the noise loss value of the training image, the total loss value of the neural network model corresponding to the i-th training image is determined. If the total loss value does not meet the training stop condition, the model parameters of the neural network model are adjusted, and the adjusted neural network model is trained using the i+1-th training image in the training image set until the total loss value of the neural network model meets the training stop condition, and the image restoration model is obtained, and the image restoration model is used to restore the image collected in severe weather.

[0084] In this way, the image restoration model is obtained by training according to the training samples corresponding to various severe weather conditions, and the weather prompts are obtained by freely combining the sub-prompts that characterize the weather degradation characteristics according to the degradation residual of the training image during model training. Combined with the scene prompts, the contrast prompt loss value and the total loss value of the training image are determined. Finally, the model parameters of the neural network model are adjusted according to the total loss value to obtain the image restoration model, so that the image restoration model can flexibly cope with a variety of complex and unpredictable weather degradation combinations, thereby improving the accuracy of image restoration.

[0085] In some embodiments, determining a total loss value of the neural network model corresponding to the i-th training image based on the comparison hint loss value and the noise loss value of the training image includes:

[0086] Determine a model restoration image corresponding to the ith training image according to the degradation residual and the ith training image;

[0087] Determine the peak signal-to-noise ratio of the model restored image and the clean reference image corresponding to the i-th training image;

[0088] The total loss value of the neural network model corresponding to the i-th training image is determined according to the contrast cue loss value, the noise loss value of the training image, and the peak signal-to-noise ratio.

[0089] Specifically, the degraded residual is the residual of the i-th training image minus the clean reference image corresponding to the i-th training image. The sum is calculated by summing the i-th training image y, and the obtained sum value can be used to determine the model recovery image corresponding to the i-th training image, that is, Then determine the peak signal-to-noise ratio of the model restored image and the clean reference image corresponding to the i-th training image. Peak signal-to-noise ratio (PSNR) can be understood as an indicator for measuring image quality, and is often used to compare the difference between a compressed or processed image and the original image.

[0090] The model restored image can be understood as the restored image of the training image output by the neural network model. Specifically, after respectively determining the contrast prompt loss value, the noise loss value of the training image, and the peak signal-to-noise ratio, the contrast prompt loss value, the noise loss value of the training image, and the peak signal-to-noise ratio can be weighted and summed, and then the total loss value corresponding to the i-th training image can be obtained. By determining the total loss value corresponding to the i-th training image, it can be used to judge the training result of the neural network model.

[0091] Specifically, the noise loss value can be expressed as:

[0092]

[0093] Among them, L res represents the noise loss value, E represents the expectation, which is used to calculate the mean square error, ∈ represents Gaussian noise, ∈ θ represents the noise estimation network, is the diffusion step size parameter, y represents the training image, x represents the clean reference image corresponding to the training image, c represents the condition generated by the prompt, and r d represents the degenerate residual.

[0094] In some embodiments, the total loss value satisfies formula (1):

[0095]

[0096] Among them, L total represents the total loss value, λ1, λ2, λ3 represent weights, L res Indicates the noise loss value, L cp Indicates the comparison prompt loss value, represents the degraded residual in the restoration process, y represents the i-th training image, x represents the clean reference image corresponding to the i-th training image, Represents the model recovery image, Indicates peak signal-to-noise ratio.

[0097] Specifically, L total Represents the total loss value, which is the weighted sum of the comparison prompt loss value, the noise loss value of the training image, and the peak signal-to-noise ratio. Lres Indicates the noise loss value. L cp Indicates the comparison prompt loss value, L cp It may include but is not limited to the contrast prompt loss value in the denoising process and the contrast prompt loss value in the restoration process. The denoising process can be understood as a process of reducing or eliminating unnecessary noise. The restoration process can be understood as a process of restoring the training image. represents the degraded residual in the restoration process, y represents the i-th training image, It means that the i-th training image passes through the neural network model and the output is the model restored image. x represents the clean reference image corresponding to the i-th training image. represents the peak signal-to-noise ratio of the model-restored image and the clean reference image corresponding to the i-th training image. λ1, λ2, and λ3 represent weights, i.e., the contrast prompt loss value, the noise loss value of the training image, and the weighted parameters of the peak signal-to-noise ratio. By calculating the peak signal-to-noise ratio of the model-restored image and the clean reference image corresponding to the training image, the model can be trained according to the peak signal-to-noise ratio to improve the accuracy of image restoration.

[0098] In some embodiments, determining a weather prompt corresponding to the i-th training image according to the weather degradation feature prompt pool and the degradation residual of the i-th training image of the i-th training sample includes:

[0099] Convert the degraded residual of the ith training image of the ith training sample into the embedded features of the ith training image;

[0100] For each sub-cue in the weather degradation feature cue pool, determine the similarity between the sub-cue and the embedded feature of the i-th training image;

[0101] Based on the order of similarity from high to low, m sub-prompts are selected, where m is a positive integer;

[0102] The m sub-cues are concatenated to obtain the weather prompt corresponding to the i-th training image.

[0103] Specifically, the degraded residual is the residual of the i-th training image minus the clean reference image corresponding to the i-th training image. For the i-th training image, the i-th training image degraded residual can be converted into the embedded features of the i-th training image. During the conversion, the conversion can be performed by methods such as convolution conversion.

[0104] Specifically, the process of screening sub-cues from the weather degradation feature cue pool can be expressed as: Pw = Ψ(P, Fe; θ).

[0105] Among them, Pw represents the m sub-cues obtained by screening, P represents all sub-cues in the weather degradation feature cue pool, Fe represents the embedded features of the i-th training image, and θ represents the parameters in the selection process.

[0106] Specifically, for each sub-prompt in the weather degradation feature prompt pool, determine the similarity between each sub-prompt and the embedded feature of the i-th training image. Then sort the obtained similarities from high to low, and select the first m sub-prompts, where m is a positive integer. Then concatenate the m sub-prompts to obtain the weather prompt corresponding to the i-th training image.

[0107] By selecting the top m sub-prompts with the highest similarity, the accuracy of sub-prompt selection can be improved, thereby improving the accuracy of image restoration. When splicing the sub-prompts, the splicing is performed using the keys of each sub-prompt. The spliced ​​sub-prompt, i.e., the weather prompt, can be expressed as:

[0108]

[0109] Among them, P w Indicates weather reminder, m indicates the preset number, represents the key of the ith subprompt, The concatenation of the keys representing the preset number of sub-cues. The autonomous combination mechanism enables the network to flexibly cope with a variety of weather degradation conditions, improving the adaptability and generalization ability of the image restoration model.

[0110] In some implementations, the scene prompt satisfies formula (2):

[0111]

[0112] Among them, P gd In the case where the scene features are deep features, Q g A query function representing the initial scene prompt, The key representing the transposed deep feature, V d represents the value of the deep feature, and D represents the feature dimension of the initial scene prompt.

[0113] Specifically, scene features may include but are not limited to depth features, semantic segmentation features, and optical flow features. When the scene features are depth features, the scene hints are scene hints formed by depth feature constraints. By using depth features as constraints, the hints can not only focus on weather degradation features, but also better understand and restore background information in complex scenes, thereby achieving higher quality image restoration.

[0114] Specifically, when forming a scene prompt based on the deep feature constraint, first obtain the deep feature and the initial scene prompt. The deep feature can be generated by a depth-anything feature generator, for example, the depth-anything feature generator can use the ViT-S-14 model. Then, the deep feature is integrated into the prompt through the cross-attention mechanism, that is, the initial scene prompt is constrained by the deep feature to obtain the scene prompt formed by the deep feature constraint.

[0115] In some embodiments, in addition to depth features, other types of scene features may be used as constraints, such as semantic segmentation features, optical flow features, etc. These features may provide scene information from different angles, further improving the quality of image reconstruction.

[0116] In some embodiments, scene hints may be generated by combining multiple scene features, such as depth features combined with semantic segmentation features, optical flow features, etc. The combination of multiple scene features may enable a more comprehensive understanding and reconstruction of complex scenes in an image, thereby improving the image restoration effect.

[0117] Scene cues are formed through scene feature constraints, so that scene cues can not only focus on weather degradation characteristics, but also consider the overall information of the scene background, thereby achieving higher quality image reconstruction.

[0118] In some embodiments, the comparison prompt loss value satisfies formula (3):

[0119]

[0120] Among them, L CP represents the comparison prompt loss value, b represents the batch parameter, m represents the preset number, K gd Keys representing scene cues formed by deep feature constraints, represents the mean of deep features, Represents the key of the i-th subprompt.

[0121] Specifically, L CP K represents the comparison prompt loss value, b represents the batch parameter, and m represents the preset number of sub-prompts screened out. gd The key representing the scene cue formed by the deep feature constraints, i.e., the target key, represents the mean of deep features, Represents the key of the i-th subprompt.

[0122] Specifically, the comparison prompts the loss value L CPIt may include but is not limited to the contrast prompt loss value in the denoising process and the contrast prompt loss value in the restoration process. For different contrast prompt loss values, formula (3) may be used for calculation. When calculating, the parameters in formula (3) may be adjusted according to the specific situation.

[0123] In some implementations, determining the contrast cue loss value of the i-th training image according to the weather cue and the scene cue includes:

[0124] When the scene feature is a depth feature, obtaining a mean value of the depth feature;

[0125] Determine a first similarity between a target key and a mean of the deep features, the target key being a key of a scene prompt formed by the deep feature constraints;

[0126] Obtaining the second similarity between the key of each sub-prompt of the weather prompt and the target key respectively;

[0127] Based on the first similarity, the second similarity between the key of each sub-cue and the target key, the contrast cue loss value of the i-th training image is determined.

[0128] Specifically, based on the weather cues and scene cues, the contrast cue loss value of the i-th training image can be determined. The introduction of the contrast cue loss value ensures the distinguishability and uniqueness of the weather cues and background cues in representation through a mutual push-pull strategy. The cosine similarity can be used to calculate the similarity between the weather cues and the background cues. Through contrast learning, the two cues are kept away from each other in the high-dimensional feature space to ensure their uniqueness and representativeness.

[0129] Specifically, when calculating the comparison prompt loss value, when the scene feature is a depth feature, the mean of the depth feature is obtained. The mean of the deep features can be understood as the average of multiple deep features. Then calculate the target key K gd And the deep feature mean The first similarity and the keys for each sub-tip of the weather tip With the target key K gd The second similarity The target key is the key of the scene prompt formed by the deep feature constraint. Finally, according to the first similarity, the second similarity between the key of each sub-prompt and the target key, the batch parameter and the preset number, the contrast prompt loss value L of the i-th training image can be determined. CP .

[0130] In some embodiments, other forms of contrastive learning strategies may also be used, such as contrastive learning methods based on graph convolutional networks (GCN) or self-attention mechanisms, to further optimize the expression effect of prompts and the generalization ability of the model.

[0131] In some embodiments, a combination of multiple contrastive learning methods can also be used. By combining different contrastive learning methods, for example, a mutual push-pull strategy combined with a graph convolutional network (GCN) or a contrastive learning method based on a self-attention mechanism, the representation of prompts can be more effectively optimized, so that the model can perform well under various weather conditions.

[0132] By using cosine similarity to calculate the similarity between weather cues and background cues, contrastive learning is used to keep these two cues away from each other in the high-dimensional feature space to ensure their uniqueness and representativeness.

[0133] In some embodiments, the neural network model can be trained on the AllWeather dataset, including multiple sub-datasets such as Snow100K, Outdoor-Rain, and RainDrop, and trained on two NVIDIA A800 GPUs using the PyTorch framework.

[0134] First, data preprocessing is performed to resize the image to 256x256 pixels, and data enhancement such as horizontal flipping and fixed-angle random rotation is performed. The Adam optimizer is used, the momentum parameters are set to 0.9 and 0.995, the initial learning rate is 1.5×10^-4, and the cosine annealing scheduling strategy is used to gradually reduce the learning rate. The cosine annealing scheduling strategy is a method of dynamically adjusting the learning rate. It uses the characteristics of the cosine function to gradually reduce the learning rate during the training process to help the model converge better. The number of training rounds is set to 800K training iterations, and the batch size used for each iteration is 16. The diffusion process can contain 1000 time steps, and the noise schedule increases from 0.0001 to 0.02.

[0135] The above is only a preferred solution of this specific embodiment. In model training, the parameters are not limited to the above parameter settings.

[0136] See also Figure 2 , is a schematic diagram of an image restoration method under adverse weather conditions provided by a specific embodiment of the present application. Figure 2 As shown, a specific embodiment of the present application provides a method for image restoration under severe weather conditions. The method proposes a new model: T3-DiffWeather Pipeline, which is based on T3-DiffWeather Pipeline to perform image restoration under severe weather conditions.

[0137] In the specific embodiment of the present application, a prompt pool for weather reporting is introduced. In the prompt pool, each sub-prompt Ps has a key K that matches s By computing the key K of each sub-prompt s and the embedded feature F e The similarity f x Then, k sub-prompts are selected in descending order of similarity, and the k sub-prompts are concatenated to obtain weather prompts (Weather Prompts, P w ).

[0138] In the specific embodiment of the present application, the depth-anything feature is obtained, and the depth-anything feature is used as motivation to directly guide the generation of general prompts (General Prompts, P gd , or background cues).

[0139] In the specific embodiment of the present application, after obtaining the weather prompt and the general prompt, a mutual push-pull strategy is adopted to ensure the distinction and uniqueness of the weather prompt and the background prompt in the representation, thereby optimizing the expression effect of the prompt. Specifically, we use cosine similarity to calculate the similarity between the weather prompt and the background prompt, and through contrastive learning, these two prompts are kept away from each other in the high-dimensional feature space.

[0140] In a specific embodiment of the present application, the first similarity between each sub-prompt in the weather prompt and the key of the scene prompt formed by the deep feature constraint is first determined. Then the key of the regular prompt corresponding to the regular prompt is matched, and the key of the regular prompt and the deep feature F of the regular prompt are calculated. d Perform "pull", that is, calculate the second similarity of the two. Finally, perform "push" through contrast learning based on the first similarity and the second similarity, so that the two prompts are far away from each other in the high-dimensional feature space.

[0141] In a specific embodiment of the present application, a degraded image y (i.e., a training image in the present application) and a degradation residual r corresponding to the degraded image are obtained. r . And combined with weather prompts, general prompts, and noise estimation network ∈ θ , embedded features F of degraded images e , train the model and obtain the degraded residual r0 that meets the preset conditions. Combining the degraded residual r0 that meets the preset conditions with the degraded image y, the restored image x can be obtained.

[0142] See also Figure 3 , is a schematic diagram of the structure of an image restoration device provided in an embodiment of the present application. In a second aspect, an image restoration device 30 is provided in an embodiment of the present application. The device 30 includes:

[0143] An acquisition module 31 is used to acquire a training image set, where the training image set includes multiple training samples corresponding to various types of severe weather conditions, each training sample includes a training image collected in severe weather conditions, and a clean reference image corresponding to the training image;

[0144] A training module 32 is used to train the neural network model using the training image set until a training stop condition is met to obtain an image restoration model, wherein the image restoration model is used to restore images collected in bad weather;

[0145] Wherein, when the neural network model is trained using the i-th training sample in the training image set of the neural network model, the following steps are included:

[0146] Determine the weather prompt corresponding to the i-th training image according to the weather degradation feature prompt pool and the degradation residual of the i-th training image of the i-th training sample; the weather degradation feature prompt pool includes sub-prompts of multiple weather degradation features for characterizing multiple severe weather conditions; the degradation residual is the residual of the i-th training image minus the clean reference image corresponding to the i-th training image;

[0147] Determine the scene prompt of the i-th training image according to the scene feature of the i-th training image;

[0148] Determine the contrast prompt loss value of the i-th training image according to the weather prompt and the scene prompt; the contrast prompt loss value is used to characterize the degree of distinction between the weather prompt and the scene prompt;

[0149] Determine the total loss value of the neural network model corresponding to the i-th training image according to the comparison prompt loss value and the noise loss value of the training image;

[0150] If the total loss value does not meet the training stop condition, adjust the model parameters of the neural network model, and use the i+1th training image in the training image set to train the adjusted neural network model until the total loss value of the neural network model meets the training stop condition to obtain an image restoration model, where i is a positive integer.

[0151] The image restoration device 30 provided in the second aspect of the embodiment of the present application can implement each process implemented in the above method embodiment and achieve the same beneficial effects. To avoid repetition, it will not be described here.

[0152] See also Figure 4 , is a structural diagram of an electronic device provided in an embodiment of the present application. The third aspect of the embodiment of the present application provides an electronic device 4000, including a processor 4100 and a memory 4200, the memory 4200 stores machine executable instructions that can be executed by the processor 4100, and the processor 4100 can execute the machine executable instructions to implement the above-mentioned image restoration method.

[0153] A fourth aspect of an embodiment of the present application provides a machine-readable storage medium, on which instructions are stored. When the instructions are executed by a processor, the processor implements the above-mentioned image restoration method.

[0154] In some implementations, the embodiments of the present application further provide a computer program product, including a computer program, which implements the image restoration method according to the above implementation when executed by a processor.

[0155] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0156] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0157] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0158] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0159] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0160] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0161] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0162] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

[0163] In addition, various embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the present invention, they should also be regarded as the contents disclosed by the present invention.

Claims

1. A method for image restoration, characterized in that: The method comprises: Acquire a training image set, wherein the training image set includes a plurality of training samples corresponding to various types of severe weather, each of the training samples includes a training image collected in severe weather, and a clean reference image corresponding to the training image; Using the training image set to train a neural network model until a training stop condition is met, thereby obtaining an image restoration model, wherein the image restoration model is used to restore images collected in severe weather; Wherein, when the neural network model is trained using the i-th training sample in the training image set of the neural network model, the following steps are included: Determine the weather prompt corresponding to the i-th training image according to the weather degradation feature prompt pool and the degradation residual of the i-th training image of the i-th training sample; the weather degradation feature prompt pool includes sub-prompts of multiple weather degradation features for characterizing multiple severe weather conditions; the degradation residual is the residual of the i-th training image minus the clean reference image corresponding to the i-th training image; Determining a scene prompt of the i-th training image according to the scene feature of the i-th training image; Determining a contrast prompt loss value of the i-th training image according to the weather prompt and the scene prompt; the contrast prompt loss value is used to characterize the degree of distinction between the weather prompt and the scene prompt; Determining a total loss value of the neural network model corresponding to the i-th training image according to the contrast prompt loss value and the noise loss value of the training image; If the total loss value does not meet the training stop condition, adjust the model parameters of the neural network model, and use the i+1th training image in the training image set to train the adjusted neural network model until the total loss value of the neural network model meets the training stop condition, so as to obtain the image restoration model, where i is a positive integer.

2. The method according to claim 1, characterized in that: The determining, according to the contrast prompt loss value and the noise loss value of the training image, a total loss value of the neural network model corresponding to the i-th training image comprises: Determining a model restoration image corresponding to the ith training image according to the degradation residual and the ith training image; Determining a peak signal-to-noise ratio of the model restored image and a clean reference image corresponding to the i-th training image; The total loss value of the neural network model corresponding to the i-th training image is determined according to the contrast prompt loss value, the noise loss value of the training image and the peak signal-to-noise ratio.

3. The method according to claim 2, characterized in that The total loss value satisfies formula (1): Among them, L total represents the total loss value, λ1, λ2, λ3 represent weights, L res represents the noise loss value, L cp represents the comparison prompt loss value, represents the degraded residual in the restoration process, y represents the i-th training image, x represents the clean reference image corresponding to the i-th training image, represents the model restored image, represents the peak signal-to-noise ratio.

4. The method according to claim 1, characterized in that: The step of determining the weather prompt corresponding to the i-th training image according to the weather degradation feature prompt pool and the degradation residual of the i-th training image of the i-th training sample comprises: Converting the degraded residual of the i-th training image of the i-th training sample into an embedded feature of the i-th training image; For each sub-prompt in the weather degradation feature prompt pool, determining a similarity between the sub-prompt and an embedded feature of the i-th training image; Based on the order of the similarity from high to low, m sub-prompts are selected, where m is a positive integer; The m sub-prompts are spliced ​​to obtain the weather prompt corresponding to the i-th training image.

5. The method according to claim 1, characterized in that The scene prompt satisfies formula (2): Among them, P gd In the case where the scene feature is a depth feature, the scene hint formed by the depth feature constraint, Q g A query function representing the initial scene prompt, The key representing the transposed deep feature, V d represents the value of the depth feature, and D represents the feature dimension of the initial scene prompt.

6. The method according to claim 1, characterized in that The step of determining the contrast prompt loss value of the i-th training image according to the weather prompt and the scene prompt comprises: When the scene feature is a depth feature, obtaining a depth feature mean; Determining a first similarity between a target key and the mean of the deep features, the target key being a key of a scene prompt formed by the deep feature constraints; Respectively obtaining a second similarity between a key of each sub-prompt of the weather prompt and the target key; The contrast prompt loss value of the i-th training image is determined according to the first similarity and the second similarity between the key of each sub-prompt and the target key.

7. The method according to claim 1, characterized in that The comparison prompt loss value satisfies formula (3): Among them, L CP represents the comparison prompt loss value, b represents the batch parameter, m represents the preset number, K gd Keys representing scene cues formed by deep feature constraints, represents the mean of deep features, Represents the key of the i-th subprompt.

8. An image restoration device, characterized in that: The device comprises: An acquisition module is used to acquire a training image set, wherein the training image set includes a plurality of training samples corresponding to various types of severe weather conditions, each of the training samples includes a training image collected in severe weather conditions, and a clean reference image corresponding to the training image; A training module, used to train a neural network model using the training image set until a training stop condition is met, thereby obtaining an image restoration model, wherein the image restoration model is used to restore images collected in bad weather; Wherein, when the neural network model is trained using the i-th training sample in the training image set of the neural network model, the following steps are included: Determine the weather prompt corresponding to the i-th training image according to the weather degradation feature prompt pool and the degradation residual of the i-th training image of the i-th training sample; the weather degradation feature prompt pool includes sub-prompts of multiple weather degradation features for characterizing multiple severe weather conditions; the degradation residual is the residual of the i-th training image minus the clean reference image corresponding to the i-th training image; Determining a scene prompt of the i-th training image according to the scene feature of the i-th training image; Determining a contrast prompt loss value of the i-th training image according to the weather prompt and the scene prompt; the contrast prompt loss value is used to characterize the degree of distinction between the weather prompt and the scene prompt; Determining a total loss value of the neural network model corresponding to the i-th training image according to the contrast prompt loss value and the noise loss value of the training image; If the total loss value does not meet the training stop condition, adjust the model parameters of the neural network model, and use the i+1th training image in the training image set to train the adjusted neural network model until the total loss value of the neural network model meets the training stop condition, so as to obtain the image restoration model, where i is a positive integer.

9. An electronic device, characterized in that: include: a memory configured to store instructions; as well as A processor is configured to call the instructions from the memory and implement the image restoration method according to any one of claims 1 to 8 when executing the instructions.

10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions for causing a machine to execute the image restoration method according to any one of claims 1 to 8.

Images