An image restoration method under various adverse weather conditions

Abstract

The present invention relates to an image restoration method under various adverse weather conditions, which is used to solve the problem that the existing unified model processing methods are all trained on the fitting data set, and the trained model has insufficient generalization ability. The technical solution of the present invention uses a real weather data set and adopts a two-stage training strategy to obtain an image restoration model. In the first training stage, a first deep learning model is used to learn the common features under different weather types; in the second training stage, after fixing the parameters of the first deep learning model, specific parameters representing different weather types are introduced into the first deep learning model to obtain a second deep learning model, and the second deep learning model is used to learn the specific features under different weather types; the trained second deep learning model is the image restoration model, which can remove the distortion interference caused by various weathers and better restore the image detail information.

Description

Technical Field

[0001] The present invention relates to the field of computer vision in artificial intelligence, and particularly to an image restoration method under various adverse weather conditions. Background Art

[0002] Adverse weather conditions such as rain, haze, snow, etc. are common climate phenomena in daily life. They usually lead to poor visual quality of captured images and deteriorate many outdoor vision systems, such as outdoor security cameras, autonomous driving systems, etc. Due to requirements such as deployment and storage, image restoration methods under various adverse weather conditions usually only use a set of network parameters to eliminate image degradation caused by shooting under various weather conditions (see the first row of pictures in Figure 1 ), which greatly increases the difficulty of restoring images captured under adverse weather conditions.

[0003] With the development of deep learning technology, there are currently many methods for dealing with single weather conditions. The most direct idea is to use a combination of multiple models for specific weather to handle various weather conditions. For example, when haze needs to be removed, an algorithm for specifically removing haze degradation of images is used. When rain line interference needs to be removed, an algorithm for specifically removing rain line degradation of images is used. For different weather types, a combination of multiple models is used to process pictures. Although the multi-model combination scheme can effectively handle different adverse weather degradations, this scheme paradigm will additionally bring computational and storage burdens to the system.

[0004] In addition, there is a unified model processing method that enables clear images to be restored using a set of network parameters under various weather conditions. For example, considering the similarities and differences in distortion under different adverse weather conditions, an integrated network architecture based on multiple encoders and a single decoder is designed. Although such methods design an integrated network architecture based on multiple encoders and a single decoder considering the similarities and differences in distortion under different adverse weather conditions, a fixed specific encoder is used for the differences under different weathers. When the number of weather scenarios to be processed increases, it often brings a large increase in model parameters, affecting the efficiency of model processing. For example, using the same network structure in combination with a two-stage knowledge distillation technique and a contrastive learning strategy to handle image degradation problems caused by various weathers. Or a unified transformer-based network with learnable weather-specific representations is proposed to handle image degradation problems caused by various weathers. Such methods only consider the common characteristics between different weathers, that is, the same parameters are used when dealing with different adverse weather conditions, and are easily interfered with by each other under different weather restoration tasks, thus affecting the restoration performance of the model in the case of a single specific weather distortion. At the same time, the unified model processing methods are often trained on the fitting dataset. Due to the difference between the fitting domain and the real domain, when applied to various weather degradation scenarios in the real world, there is often a problem of insufficient model generalization ability. Summary of the Invention

[0005] To solve the above problems existing in the prior art, the object of the present invention is to propose an image restoration method under various adverse weather conditions, which uses an efficient unified framework to achieve image restoration under various adverse weather conditions.

[0006] To achieve the above technical objectives, the technical solution of the present invention is as follows.

[0007] In a first aspect, the present invention proposes an image restoration method under multiple adverse weather conditions. The method uses a real weather dataset and adopts a two-stage training strategy to obtain an image restoration model, which can restore images under various adverse weather conditions;

[0008] The two-stage training strategy is as follows:

[0009] In the first training stage, a first deep learning model is used to learn the common features under different weather types;

[0010] In the second training stage, after fixing the parameters of the first deep learning model, specific parameters representing different weather types are introduced into the first deep learning model to obtain a second deep learning model, which is used to learn the specific features under different weather types;

[0011] The trained second deep learning model is the image restoration model.

[0012] In an implementation of the above technical method, the first deep learning model is a Unet network model composed of N convolutional layers, and the training process expression is as follows:

[0013]

[0014] In the formula:

[0015] I ρ refers to the input pictures under different weather distortions, ρ is the set of weather types, and F 1 (·) is the first deep learning model, and θ share is the shared parameter of the first deep learning model under different weathers. The shared parameter is the common feature under different weathers implicitly learned by the first deep learning model F 1 (·).

[0016] In the above technical solution, the optimization objective in the learning of the common feature by the first deep learning model is as follows:

[0017]

[0018] In the formula:

[0019] is the output picture corresponding to the input picture under different weather distortions, and Y ρ refers to the distortion-free reference picture consistent with the weather type, and ρ is the weather type.

[0020] In an implementation of the above technical method, the second deep learning model is obtained through the following method:

[0021] For the i-th feature extraction layer of the first deep learning model, add the specific parameter corresponding to the weather type ρ Then the parameters of the i-th feature extraction layer are reorganized into

[0022]

[0023] In the formula:

[0024] refers to the shared parameter in the i-th feature extraction layer of the first deep learning model, i ∈ [1, 2,..., N], and N is the number of the maximum convolutional layer;

[0025] is the specific parameter corresponding to the weather type ρ;

[0026] T τtakes a value of 0 or 1 and is used to indicate whether to add specific parameters in the i-th feature extraction layer

[0027] In a specific implementation of the above technical solution, T τ has the following value-taking principle:

[0028] Perform an importance assessment on the newly added specific parameters of the weather type ρ in the i-th feature extraction layer and record the evaluation result as

[0029] If is greater than or equal to the set threshold τ, then the corresponding T τ takes a value of 1, otherwise the corresponding T τ takes a value of 0.

[0030] In an implementation of the above technical solution, the optimization objective of the second deep learning model is expressed as follows:

[0031]

[0032] In the formula:

[0033] is the output image corresponding to the input image under different weather distortions, Y ρ refers to the undistorted reference image consistent with the weather type, ρ is the weather type, α reg is the hyperparameter coefficient, is the variable for evaluating the importance of the specific parameters added in each layer, is the regularization term constraint.

[0034] In an implementation of the above technical solution, the first deep learning model and the second deep learning model are optimized and adjusted using the cosine annealing scheme during training.

[0035] Second, the present invention proposes a computer-readable storage medium storing a computer program that can be loaded and executed by a processor to perform any of the above methods.

[0036] Third, according to the above method, the present invention proposes a corresponding system, that is: an image restoration system under multiple severe weather conditions, the system uses a real weather data set and adopts a two-stage training strategy to obtain an image restoration model, and the image restoration model can restore images under multiple severe weather conditions;

[0037] The system includes a common feature acquisition module and a specific feature acquisition module; where:

[0038] A common feature acquisition module, configured to learn common features under different weather types using a first deep learning model;

[0039] A specific feature acquisition module, configured to introduce specific parameters representing different weather types into the first deep learning model after fixing the parameters of the first deep learning model, to obtain a second deep learning model, and the second deep learning model is used to learn specific features under different weather types;

[0040] The trained second deep learning model is used as an image restoration model. Brief Description of the Drawings

[0041] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0042] Figure 1 、 one Schematic diagram of the two-stage training strategy in one implementation manner;

[0043] Figure 2 、 one Image restoration comparison diagram in one implementation manner;

[0044] Figure 3 、 one Unet structure diagram used in one implementation manner

[0045] Wherein: the first column is the degraded images under multiple input weathers, the second column is the result processed by the TSNet method, the third column is the result processed by the TransWeather method, and the fourth column is the result processed by the image restoration model obtained by the technical solution of the present invention. Detailed Implementation Manner

[0046] Considering that the existing unified model processing method has insufficient generalization ability, the technical solution of the present invention explores the common features and specific features during different weather degradations, and designs a brand-new and efficient unified framework for removing multiple image degradations related to bad weather. The key idea of this framework is to design a two-stage training strategy.

[0047] Specifically: The technical solution of the present invention uses a real weather dataset and adopts a two-stage training strategy to obtain an image restoration model, which can restore images under various adverse weather conditions. In the two-stage training strategy, in the first training stage, a first deep learning model is used to learn the common features under different weather types. In the second training stage, after fixing the parameters of the first deep learning model, specific parameters representing different weather types are introduced into the first deep learning model to obtain a second deep learning model, which is used to learn the specific features under different weather types; the trained second deep learning model is the image restoration model, which can remove the distortion interference caused by various weathers and better restore the image detail information. It can be seen from the above process that in the first training stage, the aim is to learn the common features under different adverse weathers by taking the degraded images under various weather conditions as the input of the model and outputting a roughly restored result from the first-stage model. In the second training stage, efficient learning of the specific features under different weather types is achieved through the adaptive expansion of the model. During the second-stage training process, by introducing a regularization term constraint for model expansion, it is realized to automatically learn which necessary positions in the model need to expand the specific parameters corresponding to the weather type, thus avoiding the redundancy caused by presetting specific weather parameters in advance. This also enables the technical solution of the present invention to maintain the efficiency of the model while considering the common features and specific features of different weather degradations.

[0048] Next, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments.

[0049] The terms "first", "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first", "second" may explicitly or implicitly include one or more of such features.

[0050] In one implementation, an image restoration system framework for various weather distortions is designed and referred to as the GSFNet system framework, as Figure 1 shown. For convenience of description, in the implementation, the restoration of distorted images under three different weather types, namely fog, rain, and snow, is considered. That is: for GSFNet, its input is the color-distorted pictures under different weather types where the value range of ρ is {fog, rain, snow}, is the set of color-distorted pictures, is the height × width of the picture. During the optimization process of GSFNet, different colors and symbols are used to represent different weather conditions. The training and inference processes of GSFNet are as follows.

[0051] 1. Learning of common features of image distortion under different adverse weather conditions

[0052] Construct a real-world benchmark dataset with multiple weather conditions to better handle various types of distortion under different adverse weather conditions in the real world.

[0053] To learn the common features of distorted images under different adverse weather conditions, first, a unified architecture network based on the pure convolutional experiment network UNet was constructed as the first deep learning model. The used Unet network is as Figure 2 shown. It contains four scales. Each scale consists of four residual blocks for feature extraction. In the encoder part, each scale undergoes a downsampling operation, reducing the size of the feature map by half. In the decoder part, each scale undergoes an upsampling operation, doubling the size of the feature map. In the decoder of each scale, the feature map will be concatenated with the feature map of the corresponding scale in the encoder along the channel dimension of the feature to help the network recover more refined spatial information. Finally, the feature map is mapped to the predicted result image without various weather distortions through a convolutional layer.

[0054] In the first stage, distorted images under different weather conditions are input into the first deep learning model for training, so as to jointly optimize the first deep learning model using different weather distortion data. This training process can be expressed as:

[0055]

[0056] In Equation (1): I ρ is the input image under different weather distortions, and the exemplary value range of ρ is {fog, rain, snow}; the function F 1 (·) is the first deep learning model for extracting the common features of different weather distortions. θ share refers to the shared parameters of the first deep learning model under different weather conditions, and the shared parameter θ share during the restoration of distorted images under different weather conditions 1 can represent the common features under different weather conditions implicitly learned through the first deep learning model F

[0057] The optimization objective in the learning of common features in the first stage is:

[0058]

[0059] In Equation (2): The output image corresponding to the input image under different weather distortions, Y ρ refers to the distortion-free reference image consistent with the weather type; α depth refers to the hyperparameter coefficient, and an exemplary value is 0.2.

[0060] 2. Learning the characteristics of image distortion under different adverse weather conditions

[0061] Using the first deep learning model trained in the previous part, preliminary image restoration results can already be achieved through the shared parameter network of image distortion under various adverse weather conditions. In this part, further processing of various distorted images is considered by adaptively expanding and processing the specific parameters of different weather distortions.

[0062] As Figure 1 shown, based on the first deep learning model F 1 (·) learned in the first stage for processing various weather distortions, for the shared parameter θ 1 of the first deep learning model F share is fixed in the second stage and no longer updated.

[0063] In the second stage, the second deep learning model is obtained in the following way:

[0064] For the i-th feature extraction layer of the first deep learning model, adding specific parameters reorganizes the parameters of the i-th feature extraction layer into

[0065]

[0066] where:

[0067] refers to the shared parameter in the i-th feature extraction layer of the first deep learning model, i ∈ [1, 2,..., N], and N is the number of the maximum convolutional layer;

[0068] is the specific parameter corresponding to the weather type ρ;

[0069] T τ takes a value of 0 or 1, used to indicate whether to add specific parameters in the i-th feature extraction layer

[0070] Specifically for the Figure 1 framework, the feature extraction layer is a convolutional layer. For each layer of the neural network model, corresponding specific parameters are expanded for three different distorted weather types (To distinguish different types of parameters, in Figure 1are marked with different colors and symbols). Taking the parameters of the i-th convolutional layer of the neural network model as an example, the parameters of the i-th convolutional layer for weather type ρ can be reorganized as follows:

[0071]

[0072] For equation (3), refers to the shared parameters in the i-th convolutional layer of the model, where i ∈ [1, 2,..., N], and N is the number of the maximum convolutional layer; refers to the newly expanded parameters for a specific weather type ρ, and the exemplary weather types of ρ are three kinds of weather: rain, snow, and fog; is the reorganized parameter after combining the shared parameters for multiple weather distortions and the newly expanded specific parameters; is a variable used to evaluate the importance of the newly amplified specific parameters in each layer. Combining with the threshold function T τ (·) can control whether new amplified parameters are needed in the i-th convolutional layer, and can be defined as:

[0073]

[0074] For equation (4), τ is a set hyperparameter, and the exemplary value is 0.1. When the importance evaluation variable of the newly added specific parameters for weather type ρ in the i-th convolutional layer learned through network optimization is greater than the preset threshold τ, then specific parameters for weather type ρ are amplified in the i-th convolutional layer. Otherwise, no specific parameters are introduced in the i-th convolutional layer.

[0075] In the second training stage, after fixing the parameters of the first deep learning model, specific parameters representing different weather types are introduced into the first deep learning model to obtain the second deep learning model, and the second deep learning model is used to learn specific features under different weather types. In the second stage, the optimization objective in the learning of characteristic features is:

[0076]

[0077] In equation (5), Y ρ refers to the undistorted reference image consistent with the weather type; α reg refers to the hyperparameter coefficient, and the exemplary value is 0.08.

[0078] Under the guidance of the regularization term constraint , the second deep learning model can learn some sparse but important location features, and these location features have specific features of image distortion of different severe weather types, thereby further improving the restoration performance of the model.

[0079] 3. Reasoning process for image distortion under different severe weather conditions

[0080] The network model after two-stage learning consists of two parts: a shared parameter θ that handles various weather distortions share and as specific parameters expanded for specific weather The trained second deep learning model is the image restoration model, which can be used for reasoning. Taking foggy image input as an example, the forward information flow of the network model will pass through those θ share and those specific parameters Δθ of the adaptive expansion 雾 , and restore the distortion-free image with the fog effect removed.

[0081] In the above implementation process, in addition to the UNet structure used, other network structures can also be used. When learning the specific features of different severe weather conditions, in addition to using convolutional layers, other submodules can also be used, such as Transformer structures or MLP layers.

[0082] In another embodiment, the PyTorch 1.8 platform is used to implement the deep learning model obtained by the two-stage training strategy of the method of the present invention, which can realize image restoration under various severe weather conditions. The Adam optimizer is used to optimize the two-stage deep learning model during the training process. The specific training stage is divided into two parts:

[0083] (1) The batch size of a single iteration is 4, and the image size is 224 × 224. The initial learning rate is 2e-4, and the cosine annealing scheme with a period of 50 is used for adjustment. A total of 100 training data passes. The optimization objective function of the first stage network is shown in formula (2).

[0084] (2) The batch size of a single iteration is 4, and the image size is 224 × 224. The initial learning rate is 1e-4, and the cosine annealing scheme with a period of 40 is used for adjustment. A total of 100 training data passes. The optimization objective function of the second-stage network is shown in formula (5).

[0085] In the test, the second deep learning model can directly use the input original distorted picture for image restoration.

[0086] In one embodiment, training is performed on a constructed real dataset that includes real-world fog distortion, rain distortion, and snow distortion. The technical solution of the present invention is compared with the latest image restoration methods for multiple weather conditions, including All-in-one, TransWeather, and TSNet.

[0087] In terms of measurement metrics, three quantitative evaluation metrics, Peak Signal to Noise Ratio (PSNR) and Structural Similarity (SSIM), are adopted to evaluate the performance of the model during restoration. Among them, PNSR and SSIM belong to pixel-level similarity metrics, and the larger the data, the better the restoration effect. To evaluate the efficiency of the model, the average inference time of different methods and the number of network parameters used by different methods are compared on the same device platform.

[0088] In quantitative evaluation, the proposed two-stage training network achieved the best restoration effect on each sub-test dataset with different weather distortions. Among them, for the defogging test dataset REVIDE (PSNR: 20.44, SSIM: 0.87), RealSnow (PSNR: 29.46, SSIM: 0.85), SPA+ (PSNR: 38.94, SSIM: 0.98). Among them, the number of model parameters in the second deep learning model is only 5.97 million, far less than 29 million of the best method. On the NVIDIA 1080Ti GPU device, the model inference time on the degraded image with a resolution of 256×256 is 0.03s, which is also less than 0.067 seconds of the best method. These quantitative evaluation metrics verify the superiority of the method in this case. In addition, competitive fidelity scores (PSNR and SSIM) are obtained. It proves the best detail recovery ability of the method in this case. Considering the measurement metrics and the average inference time of the model, the method of the present invention can obtain the best performance results in all benchmark tests. As Figure 3 shown, the image restoration model obtained in this case can excellently handle image distortions brought by various adverse weather conditions. Compared with other methods, it can better remove the distortion interference brought by various weather conditions and better restore the image detail information.

[0089] According to the method of the technical solution of the present invention, a corresponding system can be realized. Exemplarily, the system uses a real weather dataset and adopts a two-stage training strategy to obtain an image restoration model, which can restore images under various adverse weather conditions. The system includes a common feature acquisition module and a specific feature acquisition module; where:

[0090] The common feature acquisition module is configured to learn common features under different weather types using a first deep learning model;

[0091] The specific feature acquisition module is configured to introduce specific parameters representing different weather types into the first deep learning model after fixing the parameters of the first deep learning model to obtain a second deep learning model, and the second deep learning model is used to learn specific features under different weather types;

[0092] The trained second deep learning model serves as an image restoration model.

[0093] Through the description of the above embodiments, those skilled in the art can clearly understand that the present disclosure can be implemented by means of software plus necessary general hardware. Of course, it can also be implemented by dedicated hardware including application specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. Generally, functions accomplished by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures for implementing the same function can also be diverse, such as analog circuits, digital circuits or dedicated circuits. However, in more cases for the present disclosure, implementation by software programs is a better embodiment.

[0094] Although the embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields. The above specific embodiments are merely illustrative and guiding, rather than restrictive. Those of ordinary skill in the art can also make many forms under the inspiration of this specification and without departing from the scope protected by the claims of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. An image restoration method under multiple adverse weather conditions, Characterized in that: The method uses a real weather dataset and adopts a two-stage training strategy to obtain an image restoration model, which can restore images under multiple adverse weather conditions; The two-stage training strategy is as follows: In the first training stage, a first deep learning model is used to learn the common features under different weather types; In the second training stage, after fixing the parameters of the first deep learning model, specific parameters representing different weather types are introduced into the first deep learning model to obtain a second deep learning model, which is used to learn the specific features under different weather types; The trained second deep learning model is the image restoration model; Wherein: The first deep learning model is a Unet network model composed of N convolutional layers, and the training process expression is as follows: In the formula: Refers to the input images under different weather distortions, is the set of weather types, is the first deep learning model, is the shared parameter of the first deep learning model under different weathers. The shared parameter is the common feature implicitly learned by the first deep learning model under different weathers; The second deep learning model is obtained through the following method: For the i feature extraction layer of the first deep learning model, add the corresponding specific parameters of the weather type to it, then the parameters of the feature extraction layer are reorganized into i : ​ In the formula: is the parameter of the i-th feature extraction layer that is reorganized; Refers to the shared parameters in the feature extraction layer of the first deep learning model i , where , is the number of maximum convolutional layers; takes a value of 0 or 1 and is used to indicate whether to increase a specific parameter ; The optimization objective of the second deep learning model is expressed as follows: In the formula: L2 is the optimization objective parameter of the second deep learning model; The output images corresponding to the input images under different weather distortions refers to the undistorted reference images with the same weather type is the weather type is the hyperparameter coefficient is a variable for evaluating the importance of specific parameters added in each layer is the regularization term constraint 2. The method according to claim 1, Characterized in that: The optimization objective in the learning of the common features by the first deep learning model is as follows: In the formula: It is the optimization target parameter in the learning of common features by the first deep learning model; The output images corresponding to the input images under different weather distortions refer to the undistorted reference images with the same weather type which is the corresponding weather type.

3. The method according to claim 1, Characterized in that: Evaluate the importance of specific parameters and record the evaluation result as ; If is greater than or equal to the set threshold , then the corresponding value is 1, otherwise the corresponding value is 0.

4. The method according to claim 1, Characterized in that: The first deep learning model and the second deep learning model are optimized and adjusted using a cosine annealing scheme during the training process.

5. A computer-readable storage medium, Characterized in that: It stores a computer program that can be loaded and executed by a processor and perform any one of the methods according to claims 1 to 4.

6. An image restoration system under multiple adverse weather conditions, Characterized in that: The system uses a real weather dataset and adopts a two-stage training strategy to obtain an image restoration model, which can restore images under multiple adverse weather conditions; The system includes a common feature acquisition module and a specific feature acquisition module; wherein: The common feature acquisition module is configured to use a first deep learning model to learn the common features under different weather types; The specific feature acquisition module is configured to, after fixing the parameters of the first deep learning model, introduce specific parameters representing different weather types into the first deep learning model to obtain a second deep learning model, which is used to learn the specific features under different weather types; The trained second deep learning model is used as the image restoration model; Wherein: The first deep learning model is a Unet network model composed of N convolutional layers, and the training process expression is as follows: In the formula: Refers to the input images under different weather distortions, which is a set of weather types, is the first deep learning model, and is the shared parameter of the first deep learning model under different weathers. The shared parameter is the common feature implicitly learned by the first deep learning model under different weathers; The second deep learning model is obtained through the following method: For the i feature extraction layer of the first deep learning model, add the corresponding specific parameters of the weather type , then the i feature extraction layer parameters are reorganized into : In the formula: is the i feature extraction layer parameter is the reorganized parameter; refers to the shared parameters in the feature extraction layer of the first deep learning model i , , is the number of maximum convolutional layers; takes a value of 0 or 1 and is used to indicate whether to increase a specific parameter ; The optimization objective of the second deep learning model is expressed as follows: In the formula: L2 is the optimization objective parameter of the second deep learning model; The output images corresponding to the input images under different weather distortions refers to the distortion-free reference image with the same weather type where is the weather type is the hyperparameter coefficient is a variable for evaluating the importance of specific parameters added in each layer is the regularization term constraint

Images