A single image weather effect removal method based on generative adversarial network

Abstract

The present invention provides a method for removing weather effects from a single image based on a generative adversarial network. The method includes obtaining a data set for training, wherein the data set includes blurred images affected by weather and clear images not affected by weather; constructing a deep learning network for learning the relationship between blurred images and clear images, wherein the deep learning network is based on a generative adversarial network, and the generative adversarial network includes a first generator for generating blurred images affected by weather and a second generator for generating clear images not affected by weather; training the constructed deep learning network using the obtained data set for training to obtain the optimal second generator as the image removal model; inputting the image to be processed into the image removal model; and processing the input image and outputting a clear image corresponding to the input image.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and more specifically, to a method and system for removing weather influence from a single image based on a generative adversarial network. Background Art

[0002] With the maturity and continuous development of computer vision and image processing technologies, various imaging systems are widely used in transportation and security monitoring. However, due to inclement weather, images captured outdoors often suffer from poor quality, exhibiting blur, reduced contrast, and loss of detail. This impacts subsequent image processing and reduces the accuracy and reliability of image analysis and processing. Therefore, clarifying the features of foggy and rainy images is of great research significance.

[0003] Common severe weather conditions can be divided into two categories based on the size of their constituent particles and their visual characteristics: steady-state adverse weather (such as fog and haze) and dynamic adverse weather (such as rain, snow, and sandstorms). Unlike steady-state adverse weather, the constituent particles of dynamic adverse weather, such as rain, are larger, and their motion trajectories are affected by wind, resulting in motion blur, making scene recovery more challenging. Due to different processing methods for scene recovery in steady-state adverse weather and dynamic adverse weather, most existing restoration systems can only demonstrate good performance in defogging or deraining. Therefore, when faced with complex weather, existing systems must first classify the weather and then process rain and fog separately.

[0004] Currently, the main methods for image dehazing include histogram equalization, dark channel priors, DehazeNet, and MSCNN; the main methods for image deraining include color priors, frequency domain filtering, DerainNet, and DID-MDN. Frequency domain filtering restores the image by removing low-frequency information from the original image while retaining its high-frequency information. This method ignores the underlying cause of image degradation and therefore fails to improve image quality for complex scenes, and may even lose some image information. Deep learning-based methods such as DehazeNet and DerainNet primarily learn the mapping between blurred and clear images to achieve better image restoration. However, these methods have high dataset requirements and cannot directly generate a clear image from a blurred image; subsequent operations are still required, making them relatively cumbersome. Summary of the Invention

[0005] This Summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0006] In response to the above-mentioned deficiencies in the existing technologies, the present invention proposes a method and system for removing weather effects from a single image based on deep learning, which solves many problems in the existing technologies, such as the need to first classify images under various adverse weather conditions, the inability to improve image quality for complex scenes, the loss of detailed information, high requirements for data sets, the inability to directly obtain clear images from blurred images, and complex operations.

[0007] According to one embodiment of the present invention, a method for training a model for removing weather effects from images is provided, wherein the method comprises:

[0008] Acquire a data set for training, wherein the data set includes blurred images affected by weather and clear images not affected by weather;

[0009] Constructing a deep learning network for learning the relationship between blurred images and clear images, wherein the deep learning network is based on a generative adversarial network, the generative adversarial network including a first generator that generates blurred images affected by weather and a second generator that generates clear images not affected by weather; and

[0010] The constructed deep learning network is trained using the acquired data set for training to obtain the optimal second generator as the image weather impact removal model.

[0011] According to a further embodiment of the present invention, constructing a deep learning network for learning the relationship between blurry images and clear images further includes:

[0012] Constructing a first encoder E1 and a second encoder E2, wherein the input of the first encoder E1 is a blurred picture affected by weather, the input of the second encoder E2 is a clear picture not affected by weather, and the outputs of the first encoder E1 and the second encoder E2 are latent codes of a common latent space;

[0013] Constructing a first generator G1 and a second generator G2, wherein the inputs of the first generator G1 and the second generator G2 are the latent codes of the common latent space, the output of the first generator G1 is a generated blurred picture affected by weather, and the output of the second generator G2 is a generated clear picture not affected by weather; and

[0014] A first discriminator D1 and a second discriminator D2 are constructed, wherein the input of the first discriminator D1 is a real blurred picture affected by weather or a blurred picture affected by weather generated by the first generator G1, and the output is the probability of whether the input picture is a real picture or a generated picture, and the input of the second discriminator D2 is a real clear picture not affected by weather or a clear picture not affected by weather generated by the second generator G2, and the output is the probability of whether the input picture is a real picture or a generated picture.

[0015] According to a further embodiment of the present invention, each of the first generator G1 and the second generator G2 is configured to consist of three parts: an encoder, a converter and a decoder, wherein:

[0016] The encoder consists of convolutional layers to extract shallow features in the latent code;

[0017] The converter consists of a residual network that converts the feature vector of the image in the source domain to the feature vector in the target domain; and

[0018] The decoder consists of deconvolution layers to recover low-level features from the feature vector.

[0019] According to a further embodiment of the present invention, training the constructed deep learning network using the acquired data set for training further includes:

[0020] The blurred picture x1 affected by weather is input into the first encoder E1 to obtain the latent code z1, and the latent code z1 is transmitted as input to the first generator G1 and the second generator G2 respectively to obtain the pictures and Will sent to the first discriminator D1, Transmit to the second discriminator D2;

[0021] The clear picture x2 not affected by the weather is input into the second encoder E2 to obtain the latent code z2, and the latent code z2 is transmitted as input to the first generator G1 and the second generator G2 respectively to obtain the pictures and Will sent to the first discriminator D1, Transmit to the second discriminator D2; and

[0022] The first encoder E1, the second encoder E2, the first generator G1, the second generator G2, the first discriminator D1, and the second discriminator D2 are adjusted to minimize the loss function.

[0023] According to a further embodiment of the present invention, the loss function is designed as follows:

[0024]

[0025] in:

[0026]

[0027]

[0028]

[0029]

[0030]

[0031]

[0032]

[0033] Among them, λ0, λ1, λ2, λ3, and λ4 are configurable parameters. is the variational autoencoding loss, To generate adversarial loss, is the cycle consistency loss, and For perception loss.

[0034] According to a further embodiment of the present invention, λ0=10, λ1=0.1, λ2=100, λ3=0.1, and λ4=100.

[0035] According to a further embodiment of the present invention, the weather impact includes at least foggy weather impact and rainy weather impact.

[0036] According to a further embodiment of the present invention, the blurred pictures affected by weather and the clear pictures not affected by weather included in the data set are not in one-to-one correspondence.

[0037] According to another aspect of the present invention, a method for removing weather effects from an image is provided, wherein the method comprises:

[0038] Get the image to be processed;

[0039] Inputting the image to be processed into a model for removing the weathering effect of the image, wherein the model for removing the weathering effect of the image is a generator pre-trained according to the method of each embodiment of the present invention for generating clear images not affected by weather; and

[0040] Process the input image and output a clear image corresponding to the input image.

[0041] According to another aspect of the present invention, there is provided an image deblurring system, wherein the system comprises:

[0042] a storage unit storing a model for removing weather effects from images and computer-executable instructions, wherein the model for removing weather effects from images is a generator pre-trained according to the method of each embodiment of the present invention for generating clear images not affected by weather; and

[0043] At least one processing unit, wherein the computer executable instructions, when executed, may cause the at least one processing unit to perform a method for removing weather effects from an image, the method comprising:

[0044] Get the image to be processed;

[0045] Inputting the image to be processed into the image weather impact model; and

[0046] Process the input image and output a clear image corresponding to the input image.

[0047] These and other features and advantages will become apparent from reading the following detailed description and referring to the associated drawings.It is to be understood that both the foregoing general description and the following detailed description are illustrative only and are not restrictive of the aspects claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order that the manner in which the above-mentioned features of the present invention are understood in detail, a more particular description of the contents briefly summarized above may be given with reference to various embodiments, some aspects of which are illustrated in the accompanying drawings. It should be noted, however, that the accompanying drawings illustrate only certain typical aspects of the invention and are not to be considered limiting of its scope, as the description may admit to other equally effective aspects.

[0049] Figure 1 This is an example flow chart of a method for training a model for removing weather effects from images according to an embodiment of the present invention.

[0050] Figure 2 The figure shows a schematic structural diagram of a UNIT network that can be used in the present invention.

[0051] Figure 3 This is an example flow chart of a method for training a deep learning network for an image weather removal model according to an embodiment of the present invention.

[0052] Figure 4 is an exemplary flow chart of a method for removing weather effects from an image according to an embodiment of the present invention.

[0053] Figure 5FIG. 4 is a diagram showing an example structure of an image deblurring system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The present invention will be described in detail below with reference to the accompanying drawings, and the features of the present invention will be further apparent in the following specific description.

[0055] Figure 1 FIG. 1 is an example flow chart of a method for training a weather-removal model for an image according to an embodiment of the present invention. Figure 1 As shown in , the method starts at step 101, and a data set for training is obtained. Weather influences may include the influence of steady-state adverse weather (such as fog and haze) as well as dynamic adverse weather (such as rain, snow, and sandstorms). In one example, the data set used for training may include a data set for each specific weather, for example, it may include a data set for foggy days and a data set for rainy days. The data set may be an existing public data set, such as the foggy day data set RESIDE and the rainy day data set DID-MDN, or it may be any other suitable image data set for training. The image data set may also be divided into a training set for training and a test set for testing the effect, wherein the training set contains both blurred images that can reflect the typical features of images taken under adverse weather conditions such as foggy days and rainy days, and clear images without these features. According to one embodiment of the present invention, there does not need to be a one-to-one correspondence between blurred images and clear images, which can reduce the requirements for the data set.

[0056] In step 102, a deep learning network is constructed for learning the relationship between blurred images and clear images. As an example, a UNIT (Unsupervised Image-to-Image Translation Networks) network consisting of an encoder, a generator, and a discriminator can be constructed. The UNIT network is an extension of the generative adversarial network GAN (Generative Adversarial Network). The encoder can be used to map the input image to a latent space and convert it into a code. The generator can be further composed of three parts: an encoder, a converter, and a decoder, wherein the encoder uses a convolutional layer to extract shallow features in the latent code, and the converter is composed of a residual network to convert the feature vector of the image in the source domain to the feature vector in the target domain. The decoder is composed of a deconvolution layer to restore low-level features from the feature vector. The discriminator is used to determine whether the input image is a real image or a generated image.

[0057] Figure 2 Schematic diagram of the UNIT network that can be used in the present invention is shown. Figure 2 In the examples, foggy or rainy days are used as examples for illustration, but those skilled in the art will appreciate that the embodiments of the present invention are applicable to various adverse weather conditions. Figure 2 As shown in, first, the first encoder E1 and the second encoder E2, where the input of the first encoder E1 is a blurred picture x1 with fog or rain streaks, and the output is the latent code z1 of the shared latent space, and the input of the second encoder E2 is a clear picture x2, and the output is the latent code z2 of the shared latent space.

[0058] Then, the first generator G1 and the second generator G2 are constructed, where the input of the first generator G1 is the latent code z1 or z2 of the common latent space, and the output is a blurred picture with fog or rain streaks. For example, the corresponding output can be a picture with fog or rain streaks that is highly similar to x1. Images of the same scene with fog or rain streaks as the x1 The input of the second generator G2 is also the latent code z1 or z2 of the common latent space, and the output is a clear picture, that is, the corresponding output is the clear picture corresponding to x1 A clear picture that is highly similar to x2

[0059] Next, we construct the first discriminator D1 and the second discriminator D2. The first discriminator D1 takes as input a real image with fog or rain streaks, or a fog or rain streak image generated by the first generator G1, and outputs the probability of the input image being real or fake. The second discriminator D2 takes as input a real, clear image, or a clear image generated by the second generator G2, and outputs the probability of the input image being real or fake.

[0060] In the UNIT network above, E1 and G1 form a variational auto-encoder (VAE1). Similarly, E2 and G2 form a variational auto-encoder (VAE2). G1 and D1 form a generative adversarial network (GAN), denoted as GAN1. Similarly, G2 and D2 form a generative adversarial network (GAN2). G1, G2, D1, and D2 together form a paired GAN, also known as a CoupledGAN.

[0061] Finally, we design the loss function. This loss function consists of four components: a variational autoencoding component, a generative adversarial network component, a cycle consistency component, and a perceptual loss component. By constructing this loss function, we achieve more realistic generated images and significantly improve the discriminator's ability to distinguish real from fake. Furthermore, by training to minimize the loss function, we continuously adjust the encoder, generator, and discriminator to ultimately achieve the optimal generator model.

[0062] Back to Figure 1 In step 103, the deep learning network constructed is trained using the acquired training dataset. For example, the blurred image and the clear image in the training set are used as the input of the constructed UNIT network, and the network is trained to minimize the loss function, and finally the optimal generator model is obtained. Figure 3 Further details of the training process are described.

[0063] Figure 3 This is an example flow chart of a method for training a deep learning network for a weather-removal model for images according to an embodiment of the present invention. Figure 3 As shown in FIG, the method starts at step 301, where a blurred image x1 affected by weather (e.g., with fog or rain streaks) is input into the first encoder E1 to obtain a latent code z1, which is then transmitted as input to the first generator G1 and the second generator G2 to obtain images x1 and z1, respectively. and Will Sent to the first discriminator D1, It is transmitted to the second discriminator D2. It can be understood that when the input blurred image x1 is compared with the generated blurred image The closer they are, the better the effect of the first generator G1 is.

[0064] In step 302, the clear image x2 is input into the second encoder E2 to obtain the latent code z2, and the latent code z2 is transmitted as input to the first generator G1 and the second generator G2 respectively to obtain the images x2 and z2. and Will Sent to the first discriminator D1, is transmitted to the second discriminator D2. Similarly, when the input clear image x2 is compared with the generated clear image The closer the two images are, the better the effect of the second generator G2. It can be understood that based on the function of the second generator G2, that is, converting the input blurred image into a clear image, the second generator G2 can be used as an image defogging and deraining model to clear blurred images affected by weather (e.g., with fog or rain streaks).

[0065] In step 303 , the first encoder E1 and the second encoder E2 , the first generator G1 and the second generator G2 , and the first discriminator D1 and the second discriminator D2 are adjusted to minimize the loss function.

[0066] As an example, the loss function can be designed as:

[0067]

[0068] in:

[0069]

[0070]

[0071]

[0072]

[0073]

[0074]

[0075]

[0076] In the above formula, λ0, λ1, λ2, λ3, λ4 are configurable parameters. is the variational autoencoding loss, To generate adversarial loss, is the cycle consistency loss, is the perceptual loss, and the KL distance represents the relative entropy between two distributions. represents the activation of the i-th layer, p(z) represents the distribution of real samples, and q(z|x) represents the predicted distribution. In a preferred embodiment, when λ0 = 10, λ1 = 0.1, λ2 = 100, λ3 = 0.1, and λ4 = 100, a better image restoration effect can be achieved.

[0077] Figure 4 is an example flow chart of a method for removing weather effects from images according to one embodiment of the present invention. Method 400 starts at step 401, obtaining the image to be processed. The image to be processed may include blurred images affected by weather. As mentioned earlier, weather influences may include the influence of both steady-state bad weather (such as fog and haze) and dynamic bad weather (such as rain, snow, and sandstorms). According to another embodiment of the present invention, the image to be processed may also include images that are not affected by weather. As mentioned earlier, the generator of the present invention can generate clear images that are very close to the original images when inputting clear images that are not affected by weather, thereby eliminating the step of pre-judging the images to distinguish which are images affected by weather and which are not affected.

[0078] In step 402, the image to be processed is input into a pre-trained image weather removal model. As an example, the image weather removal model can be obtained by combining Figure 2 - Figure 3 The generator for generating clear images in the described image weather removal model is the second generator G2.

[0079] In step 403, the generator processes the input image and outputs a clear image corresponding to the input image. As described above, the trained generator can remove blur caused by weather effects such as rain or fog in the input image and obtain a corresponding clear image.

[0080] Figure 5 FIG. 5 is an example structural diagram of an image deblurring system 500 according to an embodiment of the present invention. Figure 5 As shown in FIG, the image deblurring system 500 may include a storage unit 501 and at least one processing unit 502. The storage unit 501 stores a pre-trained image de-weathering model and computer executable instructions. The generator may be a combination of Figure 2 - Figure 3 The generator for generating a clear image in the described image weather removal model is the second generator G2. When the computer executable instructions are executed by the processing unit 502, the following can be executed: Figure 4 The described method for removing weather effects from an image includes obtaining an image to be processed, inputting the image to be processed into a pre-trained generator, processing the input image, and outputting a clear image corresponding to the input image.

[0081] The above description of the method and system for removing weather effects from images of the present invention has at least the following technical effects compared with the prior art:

[0082] (1) The first encoder and the second encoder are designed to map the input image into the latent space and convert it into an encoding. The optimal discriminator model is trained by inputting foggy or rainy images into the network.

[0083] (2) Blurred images can be made clearer through the second generative model; and

[0084] (3) It solves the problems in existing technologies such as the inability to achieve end-to-end, high data set requirements, complex operations, and the need to classify and process foggy and rainy images.

[0085] What has been described above includes examples of various aspects of the claimed subject matter. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the claimed subject matter, but one skilled in the art will recognize that many further combinations and permutations of the claimed subject matter are possible. Thus, the disclosed subject matter is intended to embrace all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims.

Claims

1. A method for training a model to remove weather effects from images, characterized in that: The method comprises: Acquire a data set for training, wherein the data set includes blurred images affected by weather and clear images not affected by weather; Constructing a deep learning network for learning the relationship between blurred images and clear images, wherein the deep learning network is based on a generative adversarial network, the generative adversarial network including a first generator that generates blurred images affected by weather and a second generator that generates clear images not affected by weather; and The constructed deep learning network is trained using the acquired training data set to obtain the optimal second generator as the image weather effect removal model. The deep learning network constructed to learn the relationship between blurry images and clear images further includes: Constructing a first encoder E1 and a second encoder E2, wherein the input of the first encoder E1 is a blurred picture affected by weather, the input of the second encoder E2 is a clear picture not affected by weather, and the outputs of the first encoder E1 and the second encoder E2 are latent codes of a common latent space; Constructing a first generator G1 and a second generator G2, wherein the inputs of the first generator G1 and the second generator G2 are the latent codes of the common latent space, the output of the first generator G1 is a generated blurred picture affected by weather, and the output of the second generator G2 is a generated clear picture not affected by weather; and A first discriminator D1 and a second discriminator D2 are constructed, wherein the input of the first discriminator D1 is a real blurred picture affected by weather or a blurred picture affected by weather generated by the first generator G1, and the output is the probability of whether the input picture is a real picture or a generated picture, and the input of the second discriminator D2 is a real clear picture not affected by weather or a clear picture not affected by weather generated by the second generator G2, and the output is the probability of whether the input picture is a real picture or a generated picture.

2. The method according to claim 1, wherein Each of the first generator G1 and the second generator G2 is configured to consist of three parts: an encoder, a converter, and a decoder, wherein: The encoder consists of convolutional layers to extract shallow features in the latent code; The converter consists of a residual network that converts the feature vector of the image in the source domain to the feature vector in the target domain; and The decoder consists of deconvolution layers to recover low-level features from the feature vector.

3. The method according to claim 1, wherein Training the constructed deep learning network using the acquired training data set further includes: The blurred picture x1 affected by weather is input into the first encoder E1 to obtain the latent code z1, and the latent code z1 is transmitted as input to the first generator G1 and the second generator G2 respectively to obtain the pictures and Will sent to the first discriminator D1, Transmit to the second discriminator D2; The clear picture x2 not affected by the weather is input into the second encoder E2 to obtain the latent code z2, and the latent code z2 is transmitted as input to the first generator G1 and the second generator G2 respectively to obtain the pictures and Will sent to the first discriminator D1, Transmit to the second discriminator D2; and The first encoder E1, the second encoder E2, the first generator G1, the second generator G2, the first discriminator D1, and the second discriminator D2 are adjusted to minimize the loss function.

4. The method according to claim 3, wherein The loss function is designed as: in: Among them, λ0, λ1, λ2, λ3, and λ4 are configurable parameters. is the variational autoencoding loss, To generate adversarial loss, is the cycle consistency loss, and For perception loss.

5. The method according to claim 4, wherein λ0=10, λ1=0.1, λ2=100, λ3=0.1, and λ4=100.

6. The method according to claim 1, wherein The weather impact includes at least foggy weather impact and rainy weather impact.

7. The method according to claim 1, wherein The blurred pictures affected by weather and the clear pictures not affected by weather included in the data set do not correspond one to one.

8. A method for removing weather effects from an image, characterized in that: The method comprises: Get the image to be processed; Inputting the image to be processed into a model for removing the weathering effect of the image, wherein the model for removing the weathering effect of the image is a generator for generating clear images not affected by weather, which is pre-trained according to the method according to any one of claims 1 to 7; and Process the input image and output a clear image corresponding to the input image.

9. An image deblurring system, characterized in that: include: a storage unit storing a model for removing weather effects from images and computer-executable instructions, wherein the model for removing weather effects from images is a generator for generating clear images not affected by weather, pre-trained according to the method according to any one of claims 1 to 7; and At least one processing unit, wherein the computer executable instructions, when executed, may cause the at least one processing unit to perform a method for removing weather effects from an image, the method comprising: Get the image to be processed; Inputting the image to be processed into the image weather impact model; and Process the input image and output a clear image corresponding to the input image.

Images