A method for converting image light field information under the guidance of weather clues

Abstract

The present invention relates to a method for converting image light field information under the guidance of weather clues, and belongs to the field of graphics processing technology. It is mainly composed of four networks, namely, an image light field matching network M, a light field weather semantic structure segmentation module F, a light field weather clue extraction module W and a global weather light field generation module S, and each network performs different image processing on the source image. Together, they constitute a unified framework for multi-weather light field information conversion, and realize the mutual conversion between five types of weather light fields in the same generative adversarial network. During the entire process of image light field information conversion, the method of the present invention can retain the images of objects such as buildings and people in the image that are not related to the weather light field as much as possible, and perform corresponding accurate and diverse weather light field information conversion for light field weather clue-related areas such as the sky and the ground according to the attributes and requirements of the target weather.

Description

Technical Field

[0001] The present invention belongs to the technical fields of computer vision and graphics processing, and particularly relates to an image light field information conversion method guided by weather clues. Background Art

[0002] The conversion of image light field information guided by weather clues is a complex task of changing the weather light field in a picture, which has certain connections and differences with tasks such as image restoration and style conversion.

[0003] Image restoration methods can only perform light field operations such as de-raining and de-fogging on the rainy-day and foggy-day light fields of an image to make the picture clearer. For example, in Document 1 "K. He, J. Sun, and X. Tang, 'Single image haze removal using dark channel prior,' in Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 1956–1963.", a low-rank model was proposed to remove raindrops in the picture. In Document 2 "Y. Chang, L. Yan, and S. Zhong, 'Transformed low-rank model for line pattern noise removal,' in Proc. IEEE International Conference on Computer Vision, 2017, pp. 1735–1743.", a dark channel prior model was established to remove the fog element light field in the picture. De-raining and de-fogging algorithms usually need to establish complex mathematical models to estimate the noise in the picture, and cannot convert foggy days and rainy days to other weather light fields (such as sunny days, snowy days, cloudy days, etc.), nor can they convert other weather types to foggy or rainy-day light fields.

[0004] The picture style conversion algorithm uses a generative adversarial model to convert the overall light field of a picture, but it cannot and does not deal with the conversion between light fields of multiple weathers. In Document 3, "J. Zhu, T. Park, P. Isola, and A. A. Efros, 'Unpaired image-to-image translation using cycle-consistent adversarial networks,' in Proc. IEEE International Conference on Computer Vision, 2017, pp. 2242–2251.", by combining two generative adversarial networks and proposing a loss training strategy of cycle consistency, it becomes a general picture style transfer framework. In the picture style transfer method based on the generative adversarial network, generally one network can only complete the conversion between two weather light fields, and it cannot precisely edit the weather light field in the picture. It can only convert the overall light field of the picture, and the conversion result is false, unrealistic, and unnatural. In addition, most of these methods are restricted by the influence of the unbalanced data distribution of the weather data set, and the converted pictures are often severely overfitted, looking false and having a low availability rate. Summary of the Invention

[0005] Technical Problems to be Solved

[0006] In order to avoid the problem of unbalanced distribution of the existing weather data set and the problem that the existing methods cannot deal with the conversion of multiple weather light fields and precisely edit the image light field, the present invention proposes a method for converting image light field information guided by weather clues.

[0007] Technical Solution

[0008] A method for converting image light field information guided by weather clues, characterized by the following steps:

[0009] Step B1: The source picture is represented as x, the light field weather type of the source picture is y, and the target light field weather for the weather light field conversion is Randomly initialize the random latent encodings z1 and z2, and input them into the image light field matching network M to generate two different light field weather control encodings for the target light field weather of and

[0010] Step B2: Use the source picture x in Step B1 as the input of the light field weather semantic structure segmentation module F to obtain the weather light field clue segmentation map F y (x);

[0011] Step B3: The light field weather clue extraction module W uses the The light field weather light field segmentation map F output in step B2 y and the source picture x in step B1 as inputs, and outputs the corresponding light field weather clue map

[0012] Step B4: The global weather light field generation module S takes the output in step B1 and the source picture x as inputs, and outputs the global weather light field conversion map and uses the light field weather clue map output in step B3 as a guide to synthesize the final weather light field conversion map Defined as:

[0013]

[0014] The described image light field matching network M consists of three fully connected network layers.

[0015] The described weather semantic structure segmentation module F consists of an encoder-decoder network.

[0016] The described weather clue extraction module W consists of a regularized encoder-decoder network with parameters.

[0017] The described global weather light field generation module S consists of a regularized encoder-decoder network with parameters.

[0018] A computer system, characterized in that it includes: one or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0019] A computer-readable storage medium, characterized in that it stores computer-executable instructions, and the instructions are used to implement the above method when executed.

[0020] Beneficial effects

[0021] The present invention provides a method for converting image light field information under the guidance of weather clues. It can retain as many objects as possible in the image that are not related to the weather light field, such as buildings and people, in the whole process of image light field information conversion, and perform corresponding accurate and diverse weather light field information conversion for light field weather clue-related areas such as the sky and the ground according to the properties and requirements of the target weather. It effectively avoids the imbalance problem of existing data sets, and completes 20 types of weather light field conversions between five types of light field weather (sunny, rainy, cloudy, snowy, foggy) through a unified weather light field converter. The authenticity and diversity index FID of image light field conversion reaches 48.998, and the accuracy rate of image light field weather conversion reaches 78.83%. In addition, this method can not only be applied to weather light field conversion tasks, but also can be extended to extreme environment light field conversion and accurate image editing, image restoration, scene generation, style migration and other visual tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0023] Figure 1 Training flow chart of the present invention;

[0024] Figure 2 Test flow chart of the present invention;

[0025] Figure 3 Weather conversion effect diagram of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0027] A method for converting image light field information guided by weather clues mainly consists of four networks, namely an image light field converter G, an image light field matching network M, a weather light field encoder E, and a weather light field discriminator D. The light field converter consists of three parts, namely a light field weather semantic structure segmentation module F, a light field weather clue extraction module W, and a global weather light field generation module S. The image light field converter, the image light field matching network, the weather light field encoder, and the weather light field discriminator jointly constitute a unified framework for multi-weather light field information conversion, realizing the mutual conversion between 5 weather light fields in the same generative adversarial network. The image light field converter first completes the recognition and processing of weather light field semantic structure information, and completes the conversion of image light field information under multiple weathers with the help of the unified framework of multi-weather light field information conversion.

[0028] A method for converting image light field information guided by weather clues, including two parts: the recognition and processing of weather light field semantic structure information and the unified framework of multi-weather light field information conversion. The five light field weathers are sunny, rainy, foggy, snowy, and cloudy. The steps are as follows:

[0029] A) Training stage

[0030] Step A1: The source image is represented as x, the light field weather type of the source image is y, and the target light field weather for weather conversion is The purpose of the image light field matching network M is to generate a light field weather control code w = M y (z) from the random latent code z. The random latent code z is a randomly initialized one-dimensional vector, and its random property brings the possibility of generating diverse weather light field pictures. The image light field matching network M can generate a light field weather control code w for a specific target light field as the input of the global weather light field generation module S and the light field weather clue extraction module W, controlling the light field weather type and the process of light field information conversion. In addition, p is introduced to control the weather light field conversion intensity, defined as:

[0031]

[0032] where w i represents a set of light field weather control codes calculated from different z. Randomly initialize the random latent codes z1 and z2, and input them into the image light field matching network M to generate two different light field weather control codes for the target light field weather and p ∈ (0, 1) represents the intensity control factor;

[0033] Step A2: The weather light field encoder E is used to extract the weather control code w e = E y(x), the weather control code w generated by the weather light field encoder E e reflects the light field weather type of x and can be used to control the target light field weather of the finally converted image; the w generated by E e diversity comes from the input of different light field weather images x; the weather light field encoder E is a typical encoder structure; the weather control code w e is used for setting the loss function in training;

[0034] Step A3: The weather semantic structure segmentation module F takes the picture x with the source weather category as y as the input and outputs the weather light field clue segmentation map F y (x); the weather semantic structure segmentation module F is composed of a typical encoder-decoder network;

[0035] Step A4: The weather clue extraction module W takes the output in Step A1 the weather light field segmentation map F output in Step A3 y (x) and the source picture x in Step A1 as the input and outputs the corresponding weather clue map The weather clue extraction module W is composed of a regularized encoder-decoder network with parameters;

[0036] Step A5: The global weather light field generation module S takes the output in Step A1 and the source picture x as the input and outputs the global weather light field conversion map and synthesizes the final weather light field conversion map under the guidance of the weather clue map The global weather light field generation module S is composed of a regularized encoder-decoder network with parameters;

[0037] Step A6: The weather light field discriminator D discriminates whether the weather light field conversion picture in Step A5 is successfully converted into the target light field weather The output of the weather light field discriminator corresponding to the source picture x is D y (x), the weather light field conversion picture the output of the weather light field discriminator is The weather light field discriminator D is composed of an encoder and a fully connected network layer;

[0038] B) Testing phase:

[0039] Step B1: The source picture is represented as x, the light field weather type of the source picture is y, and the target light field weather for weather light field conversion is Randomly initialize the random latent codes z1 and z2 and input them into the image light field matching network M to generate two different light field weather control codes for the target light field weather and and

[0040] Step B2: Use the source image x in Step B1 as the input of the light field weather semantic structure segmentation module F to obtain the weather light field clue segmentation map F y (x);

[0041] Step B3: The light field weather clue extraction module W uses the output in Step B1 The weather light field segmentation map F of the light field output in Step B2 y (x) and the source image x in Step B1 as the input, and outputs the corresponding light field weather clue map

[0042] Step B4: The global weather light field generation module S uses the output in Step B1 and the source image x as the input, and outputs the global weather light field conversion map and uses the light field weather clue map output in Step B3 as a guide to synthesize the final weather light field conversion map Defined as:

[0043]

[0044]

[0045] Example 1

[0046] Training stage:

[0047] Step 1: Dataset enhancement and preprocessing. Randomly divide 8,878 pictures in the dataset for network training. Each picture is resized randomly with a scale multiple of 0.8 - 1.0, and the aspect ratio of the size is randomly changed to 0.9 - 1.1, and the source image x with a picture size of 256×256 is cropped.

[0048] Step 2: Randomly initialize the random latent codes z1 and z2, and input them into the image light field matching network M to generate two different light field weather control codes for the target light field weather and

[0049] Step 3: The light field weather semantic structure segmentation module F adopts a weakly supervised multi-task training strategy, uses the weather light field segmentation label and weather light field category label annotated at the box level as the supervision information, and outputs the weather light field clue segmentation map F of the picture x with the source light field weather category of y y (x). The light field weather classification loss function is:

[0050]

[0051] The loss function of light field weather semantic segmentation is defined as:

[0052]

[0053] Among them, s i,j represents the pixel segmentation prediction label at (i, j). The total loss function of the weakly supervised multi-task segmentation network is defined as:

[0054]

[0055] Step 4: The light field weather clue extraction module W takes the light field weather light field segmentation map F output in Step 3 and the source picture x in Step 1 as inputs, and outputs the corresponding light field weather clue map y (x) and the source picture x in Step 1 as inputs, and outputs the corresponding light field weather clue map

[0056] Step 5: The global weather light field generation module S takes the output in Step 2 and the source picture x as inputs, and outputs the global weather light field conversion picture and synthesizes the final weather light field conversion map under the guidance of the light field weather clue map The calculation process is:

[0057]

[0058]

[0059] Step 6: The image light field converter G needs to ensure that the weather light field conversion picture after the light field weather conversion is as similar as possible to the source picture. The weather control code of the source picture is w e , and uses the cyclic structure perceptual consistency loss function:

[0060]

[0061] Among them, SSIM(·) and VGG(·) respectively represent the structural similarity measurement and the L1 perceptual loss of the VGG network.

[0062] Step 7: The weather light field discriminator D needs to ensure that the weather type of the image after the weather conversion is the target light field weather and uses the adversarial loss function:

[0063]

[0064] Step 8: The light field conversion semantic consistency loss function is to make the generated result more consistent with the source picture semantically. Taking the in Step 5 and the source picture x in Step 1 as inputs, it is defined as:

[0065]

[0066] Step 9: The light field diversity loss function is to make the result of light field conversion have more diversity. Using the final weather light field conversion map in Step 5 as the input, it is defined as:

[0067]

[0068] Step 10: The weather light field reconstruction loss is to make the target picture of light field weather conversion have a weather category of Using the weather conversion result in Step 5 as the input, the weather of the picture is represented by the weather light field encoder as The weather light field reconstruction loss is expressed as:

[0069]

[0070] Step 11: The loss functions defined in Step 4, Step 6, Step 7, Step 8, Step 9, and Step 10 together constitute the total loss function:

[0071]

[0072] where λ cyc , λ seg , λ wrc , λ inv , and λ wd are the weight parameters of each loss function respectively.

[0073] Step 11: In each training iteration, the weather light field discriminator D is first updated 2 times by the adversarial loss function, then the image light field converter G, the image light field matching network M, and the weather light field encoder E are updated 1 time by the total loss function, and finally the image light field converter G is updated 1 time by the total loss function. All hyperparameters are set to 1 by λ cyc , λ seg , λ wrc , λ inv , and λ wd and the training is iterated 10,000 times in total.

[0074] Testing phase:

[0075] Step 12: Fix the network parameters trained in Step 11. The test picture is denoted as x t , the light field weather type of the source picture is y t , and the target light field weather for weather conversion is Randomly initialize the random latent code z t1and z t2 are input into the image light field matching network M to generate the target weather for two different light field weather control encodings and

[0076] Step 13: Using the test image x in Step 12 t as the input of the light field weather semantic structure segmentation module F to obtain the weather light field segmentation map

[0077] Step 14: The light field weather clue extraction module W uses the outputs in Step 12 and and the weather light field segmentation map output in Step 13 and the test image x in Step 12 t as the input and outputs the corresponding weather light field clue map

[0078] Step 15: The global weather light field generation module S uses the outputs in Step 12 and the test image x t as the input, outputs the global weather light field conversion map and uses the weather light field clue map output in Step 14 as a guide to synthesize the final weather light field conversion map Defined as:

[0079]

[0080]

[0081] The effects of the present invention can be further illustrated by the following experimental results.

[0082] 1. Experimental environment and settings

[0083] The present invention conducts experiments and simulations based on the PyTorch framework on an operation with an Intel(R) Xeon(R) Gold 6133 CPU @ 2.50 GHz, 64G of memory, an NVIDIA V100 GPU, and Ubuntu 18.04.

[0084] The dataset used in the experiment is Multi-Task Weather, which was made public by Zhao et al. at "https: / / github.com / wzgwzg / Multitask_Weather" and contains 8,878 training images and 2,500 test images. The dataset includes five weather categories: sunny, cloudy, rainy, foggy, and snowy, and has box-level weather semantic label annotation information.

[0085] 2. Simulation content

[0086] First, train a deep model with the training data; then, generate and save the results of a total of 20 weather conversions on the test set, and calculate the Fréchet Inception Distance (FID) score and the light field weather classification accuracy on all test results.

[0087] To prove the effectiveness of the algorithm, CycleGAN, which can only handle the transformation between two domains, the multi-domain transfer model MUNIT and Drit++ based on the shared content space assumption, and the multi-domain transfer model StarGANv2 based on the matching network are selected as comparison methods. CycleGAN is introduced in detail in the literature "J. Zhu, T. Park, P. Isola, and A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks, In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017." MUNIT is a multi-domain style transfer method proposed in the literature "X. Huang, M. Liu, S. J. Belongie, and J. Kautz. Multimodal Unsupervised Image-to-Image Translation, in Proceedings of the European Conference on Computer Vision, pp. 179–196, 2018." Drit++ is a multi-domain style transfer method proposed in the literature "H. Lee, H. Tseng, Q. Mao, J. Huang, Y. Lu, M. Singh, and M. Yang, “Drit++: Diverse Image-to-Image Translation via Disentangled Representations,” International Journal of Computer Vision, vol. 128, no. 10, pp. 2402–2417, 2020." StarGANv2 is a multi-domain transfer model based on the matching network proposed in the literature "Y. Choi, Y. Uh, J. Yoo, and J.-W. Ha, “StargGAN v2: Diverse image synthesis for multiple domains,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2020, pp. 8188–8197."

[0088] The FID score is shown in Table 1, which can reflect the distribution consistency and diversity between the generated images and real weather images. Since the present invention preserves the weather-light-field-independent regions in the image during the image light field conversion process, and the generated results are real and diverse, the FID score is in the first echelon.

[0089] Table 1 FID score

[0090] Target weather Cloudy Foggy day Rainy day Snowy day Sunny day Drit++ 95.00 104.73 117.75 109.85 99.13 Munit 75.95 99.03 135.48 124.60 82.63 CycleGAN 73.10 65.80 78.55 58.20 70.83 StarGAN v2 40.50 41.55 44.23 43.20 38.95 The present invention 44.73 45.94 45.48 47.50 49.20

[0091] The accuracy rate of light field weather classification is shown in Table 2, which can reflect the completion degree of successfully converting the weather image after conversion into the target light field weather. It can be seen that the accuracy rate of light field weather classification of the present invention is excellent under each target conversion weather.

[0092] Table 2 Accuracy rate of light field weather classification

[0093] Target weather Cloudy Foggy day Rainy day Snowy day Sunny day Drit++ 18.80 13.30 54.90 60.20 23.00 Munit 68.30 24.80 63.80 47.80 15.90 CycleGAN 54.30 3.30 42.80 64.60 13.60 StarGAN v2 17.40 49.30 16.50 50.40 92.10 The present invention 53.60 95.00 57.60 53.20 96.50

[0094] The visualization result of the experimental results is as Figure 3 shown. The image in the black frame is the input image, and the weather type of the input image is marked by the black characters above. The four unframed photos in each column are the results of weather conversion, and the target weather is marked on the left side of the image. From Figure 3 it can be seen that the weather conversion effect of the present invention is real and natural, and the conversion of a total of 20 light field weathers is effectively completed.

[0095] Under the guidance of weather clues, the present invention precisely edits the weather light field region of the image, and uses a unified multi-weather light field conversion framework to realize the mutual conversion between multiple light field weathers under an image light field converter. The conversion result preserves the original features of the scene, only changing the weather light field region of the image. The present invention is carried out under the image light field conversion task, but is not limited to this type of scene. There is no specific scene constraint in the algorithm design, and it can be extended to applications such as extreme environment light field conversion, precise image editing, image restoration, scene generation, and style transfer.

[0096] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered within the protection scope of the present invention.

Claims

1. A method for converting image light field information under the guidance of weather clues, characterized in that, Composed of four networks, namely the image light field converter , the image light field matching network , the weather light field encoder and the weather light field discriminator ; among them, the image light field converter is composed of three parts, namely the light field weather semantic structure segmentation module , the light field weather clue extraction module and the global weather light field generation module ; the image light field converter, the image light field matching network, the weather light field encoder and the weather light field discriminator together constitute a unified framework for multi-weather light field information conversion, realizing the mutual conversion between 5 weather light fields in the same generative adversarial network; the steps are as follows: Step B1: The source image is represented as , the light field weather type of the source image is , the target light field weather for the weather light field conversion is ; Randomly initialize the random latent code and , and input them into the image light field matching network to generate two different light field weather control codes for the target light field weather and ; The described image light field matching network consists of three fully connected network layers; Step B2: Using the source image in Step B1 as the input of the light field weather semantic structure segmentation module to obtain the weather light field clue segmentation map ; The weather semantic structure segmentation module is composed of an encoder-decoder network; Step B3: Light Field Weather Cue Extraction Module Using the output in Step B1 and , the light field weather light field segmentation map output in Step B2 and the source image in Step B1 as inputs, and output the corresponding light field weather cue map and ; the weather cue extraction module is composed of a regularized codec network with parameters; Step B4: Global Weather Light Field Generation Module Using the output in Step B1 、 and the source image as inputs, output the global weather light field conversion map 、 . The global weather light field generation module consists of a regularized codec network with parameters; and using the light field weather clue map 、 output in Step B3 as a guide, synthesize the final weather light field conversion map , which is defined as: , Weather light field discriminator Discriminate weather light field conversion pictures Whether it is successfully converted into the target light field weather ; Source picture The output of the corresponding weather light field discriminator is , Weather light field conversion picture The output of the weather light field discriminator of is ; The weather light field discriminator Consists of an encoder and a fully connected network layer.

2. A computer system, characterized in that Comprising: One or more processors, a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to claim 1.

3. A computer-readable storage medium, characterized in that Stored with computer-executable instructions which are used to implement the method according to claim 1 when executed.

Images