A network model for generating data sets and a fog map generation method

Abstract

The present application relates to the field of image processing technology, and in particular to a network model for generating a data set and a method for generating a fog image, including: an image encoding module, a coupled generative adversarial network module, an adversarial discriminant module and a synthesis module; the image encoding module includes a clean image data set input port and a fog image data set input port, the coupled generative adversarial network module includes a first generation unit and a second generation unit, and the adversarial discriminant module includes a first discriminator and a second discriminator; the first generation unit includes a first encoder, a shared potential image domain and a first decoder, and the second generation unit includes a second encoder, a shared potential image domain and a second decoder. In the actual application process, the present application proposes a neural network framework based on a coupled generative adversarial network, and adopts a semi-supervised learning strategy and a specific combined loss function in the network iterative training process, so that the image generated by the network can learn the depth information of the real image, making the image more realistic.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method for generating a data set network model and a fog image generation method. Background Art

[0002] With the development and maturity of neural network technology, the image application fields of neural network technology are increasing. However, neural networks require a large amount of image data for training to achieve good performance. Therefore, collecting a large amount of high-quality image data sets is indispensable for deep learning of neural networks.

[0003] Traditionally, foggy image datasets are usually obtained by artificial synthesis. To generate foggy image datasets, it is first necessary to obtain clean image features through the dark channel method, and then use the atmospheric scattering model to simulate foggy images, so as to create an artificially synthesized foggy image dataset based on the clean images.

[0004] However, traditional artificial synthetic image datasets cannot truly reflect the distribution of actual foggy images. Due to the processing superposition error and the failure to consider the randomness of the combination of various degradation factors and their interactions, the results of reconstructing synthetic fog images are not ideal. The neural network trained using artificial synthetic foggy image datasets has poor target detection performance in real foggy environments. Summary of the invention

[0005] In order to solve the problems in the prior art that the traditional artificial synthetic image dataset cannot truly reflect the distribution of world images, the trained neural network has poor target detection effect in the real environment, and the production process is relatively time-consuming, the embodiment of the present application provides a dataset generation network model, including: an image encoding module, a coupled generative adversarial network module, an adversarial discrimination module and a synthesis module;

[0006] The image encoding module includes a clean image data set input port and a fog image data set input port, the coupled generative adversarial network module includes a first generation unit and a second generation unit, and the adversarial discriminant module includes a first discriminator and a second discriminator; the first generation unit includes a first encoder, a shared potential image domain and a first decoder, and the second generation unit includes a second encoder, the shared potential image domain and a second decoder;

[0007] The clean image data set input port is used to read the clean image data set, the first encoder is used to extract features from the clean image data set, the shared potential image domain is used to store the extracted features, the first decoder is used to read the features stored in the shared potential image domain and transmit the features to the first discriminator, and the first discriminator is used to reconstruct the clean image data set after the feature extraction;

[0008] The fog image data set input port is used to read the fog image data set, the second encoder is used to extract features from the fog image data set, the shared potential image domain is used to store the extracted features, the second decoder is used to read the features stored in the shared potential image domain and transmit the features to the second discriminator, and the second discriminator is used to reconstruct the fog image data set after the feature extraction;

[0009] The synthesis module combines the reconstructed clean image dataset and the reconstructed fog image dataset into a synthetic fog image.

[0010] Furthermore, the generated dataset network model is optimized through the following loss functions: coupled generative adversarial loss function, reconstruction loss function, perceptual loss function, L1 loss function and total variational loss function.

[0011] Furthermore, a network model for generating a dataset, the coupled generation of the adversarial loss function is specifically:

[0012] L 生成 =L GAN (D h ,J h ,G h (I c ))+L GAN (D c ,J c ,G c (I h ));

[0013] Among them, L 生成 is the coupled generative adversarial loss function, L GAN is the classic generative adversarial loss model function, D h To identify the clean image domain, J h The original real clean image dataset G is the input of the network. h Clean image dataset generated based on latent feature distribution, I c is the fog image domain with random noise distribution, D c To distinguish the foggy image domain, J c The original real fog image dataset G is the input of the network. c is a fog image dataset generated based on latent feature distribution. h A clean image dataset with random noise distribution;

[0014] The classic generative adversarial loss model function is specifically:

[0015] L GAN =P J [logD(J)]+P I[log(1-D(Gen(I))];

[0016] Among them, L GAN is the classic generative adversarial loss model function, P J is the expected value of the distribution function of the original real image data set input to the network, D is the discriminant network, J is the original real image data set input to the network, P I is the expected value of the distribution function of random noise distribution, Gen(I) is the sample generated based on random noise;

[0017] The reconstruction loss function is specifically:

[0018] L 重建 =||G C (E C (J C )-J C )||+||G h (E h (J h )-J h )||;

[0019] Among them, L 重建 is the reconstruction loss function, G C To generate the fog image based on the features of the generated network, E C To encode the fog image, J C Input the original real fog image to the network, G h To generate clean images based on features of the generated network, E h To encode the clean image, J h Input the original real clean image to the network;

[0020] The perceptual loss function is specifically:

[0021]

[0022] Among them, L p is the perceptual loss function, The features of the network generated image and the original control image extracted from the convolutional neural network VGG16, G C To generate the fog image based on the features of the generated network, I c is the fog image domain with random noise distribution, J h The original real clean image dataset G is the input of the network. h To generate clean images based on features of the generated network, I h is a clean image dataset with random noise distribution, J C Input the original real fog image to the network;

[0023] The L1 loss model function is specifically:

[0024] L1=||G c (I c )-J h ||±||G h (I h )-J c ||;

[0025] Among them, L1 is the L1 loss model function, G C To generate the fog image based on the features of the generated network, I c is the fog image domain with random noise distribution, J h The original real clean image dataset G is the input of the network. h To generate clean images based on features of the generated network, I h is a clean image dataset with random noise distribution, J C Input the original real fog image to the network;

[0026] The total variation loss function is specifically:

[0027]

[0028] Among them, L 变分 is the total variational loss function, represents the horizontal gradient difference, Represents the vertical gradient difference, G C To generate the fog image based on the features of the generated network, I c is the fog image domain with random noise distribution.

[0029] Furthermore, the first encoder and the second encoder respectively include a plurality of first convolution blocks and a plurality of first residual blocks connected in sequence, and the first decoder and the second decoder respectively include a plurality of second residual blocks and a plurality of second convolution blocks connected in sequence; the output end of the first residual block is connected to the input end of the shared potential image domain, and the output end of the shared potential image domain is connected to the input end of the second residual block; the output end of the first convolution block is jump-connected to the input end of the second convolution block; the output end of the second convolution block is jump-connected to the input end of the first convolution block; the convolution block is used to extract high-level features of the image, and the residual block is used to learn detailed information of different features of the image.

[0030] Furthermore, the first encoder and the second encoder include 3 first convolution blocks and 4 first residual blocks respectively; the first decoder and the second decoder include 4 second residual blocks and 2 second convolution blocks respectively.

[0031] Furthermore, the first residual block and the second residual block are respectively composed of a plurality of sub-residual blocks cascaded in series; the sub-residual blocks at any level include a first convolution block and a second convolution block connected in sequence, which are used to process the output results of the sub-residual block of the previous level and the output results of the sub-residual block of the previous level, and output the processing results to the sub-residual block of the next level and the sub-residual block of the next-next level.

[0032] A fog image generation method based on a generated data set network model includes:

[0033] Obtain the image dataset to be processed;

[0034] Preprocess the image dataset;

[0035] Input the preprocessed image dataset into the dataset generation network model as described above;

[0036] Obtain a synthetic fog image output by the network model for generating the dataset.

[0037] Furthermore, the preprocessing is specifically as follows:

[0038] Convert the image dataset into an HSV format image dataset;

[0039] Extract the brightness and saturation of the HSV format image data set;

[0040] Generating depth information corresponding to different pixels in the image data set according to the brightness and saturation;

[0041] The depth information in the original image is combined with the original image to obtain an image dataset.

[0042] A computer device comprising:

[0043] Memory for storing computer programs;

[0044] A processor is used to implement the steps of a fog graph generation method based on generating a data set network model when executing the computer program.

[0045] From the above technical solution, it can be known that a network model for generating a data set includes: an image encoding module, a coupled generative adversarial network module, an adversarial discriminant module and a synthesis module; the image encoding module includes a clean image data set input port and a fog image data set input port, the coupled generative adversarial network module includes a first generation unit and a second generation unit, and the adversarial discriminant module includes a first discriminator and a second discriminator; the first generation unit includes a first encoder, a shared potential image domain and a first decoder, and the second generation unit includes a second encoder, the shared potential image domain and a second decoder; the clean image data set input port is used to read a clean image data set, the first encoder is used to extract features from the clean image data set, and the shared The shared latent image domain is used to store the extracted features, the first decoder is used to read the features stored in the shared latent image domain and transmit the features to the first discriminator, and the first discriminator is used to reconstruct the clean image data set after feature extraction; the fog image data set input port is used to read the fog image data set, the second encoder is used to extract features from the fog image data set, the shared latent image domain is used to store the extracted features, the second decoder is used to read the features stored in the shared latent image domain and transmit the features to the second discriminator, and the second discriminator is used to reconstruct the fog image data set after feature extraction; the synthesis module combines the reconstructed clean image data set and the reconstructed fog image data set into a synthetic fog image.

[0046] In actual application, this application proposes a neural network framework based on a coupled generative adversarial network. By adopting a semi-supervised learning strategy and a specific combined loss function in the network iterative training process, the images generated by the network can learn the depth information of the real image, while simulating the feature distribution of images in real scenes, realizing image generation across image domains, and making the images more realistic. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0048] Figure 1 A schematic diagram of a process for generating a data set network model according to an embodiment of the present application;

[0049] Figure 2 A schematic diagram of another process of generating a data set network model according to an embodiment of the present application;

[0050] Figure 3 A schematic diagram of a shared potential image domain for generating a dataset network model according to an embodiment of the present application;

[0051] Figure 4 A schematic diagram of a coupled generative adversarial network module for generating a data set network model according to an embodiment of the present application;

[0052] Figure 5 A schematic diagram of a workflow for generating a data set network model according to an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0054] In the description of this application, it should also be noted that, unless otherwise clearly specified and limited, the term "connection" should be understood in a broad sense, for example, it can be an electrical connection or a communication connection. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0055] In order to solve the problem that the traditional artificial synthetic image dataset in the prior art cannot truly reflect the distribution of world images, the trained neural network has poor target detection effect in the real environment, and the production process is relatively time-consuming, see Figure 1, is a flow chart of a network model for generating a data set according to an embodiment of the present application. The first aspect of an embodiment of the present application provides a network model for generating a data set, including: an image encoding module, a coupled generative adversarial network module, an adversarial discrimination module and a synthesis module; the image encoding module includes a clean image data set input port and a fog image data set input port, the coupled generative adversarial network module includes a first generation unit and a second generation unit, the adversarial discrimination module includes a first discriminator and a second discriminator; the first generation unit includes a first encoder, a shared potential image domain and a first decoder, the second generation unit includes a second encoder, the shared potential image domain and a second decoder; the clean image data set input port is used to read a clean image data set, and the first encoder is used to convert a clean image data set into a fog image data set. The data set is used for feature extraction, the shared latent image domain is used to store the extracted features, the first decoder is used to read the features stored in the shared latent image domain and transmit the features to the first discriminator, and the first discriminator is used to reconstruct the clean image data set after feature extraction; the fog image data set input port is used to read the fog image data set, the second encoder is used to extract features from the fog image data set, the shared latent image domain is used to store the extracted features, the second decoder is used to read the features stored in the shared latent image domain and transmit the features to the second discriminator, and the second discriminator is used to reconstruct the fog image data set after feature extraction; the synthesis module combines the reconstructed clean image data set and the reconstructed fog image data set into a synthetic fog image.

[0056] See also Figure 2 , is a flow chart of another data set generation network model according to an embodiment of the present application, wherein the data set generation network model includes an image encoding module, a coupled generative adversarial network module and an adversarial discriminant module; the image encoding module includes a clean image data set input port and a fog image data set input port; the coupled generative adversarial network module includes a first generation unit and a second generation unit; the adversarial discriminant module includes a first discriminator and a second discriminator; the clean image input port, the first generation unit and the first discriminator are connected in sequence for feature extraction and reconstruction of the clean image, and the fog image input port, the second generation unit and the second discriminator are connected in sequence for feature extraction and reconstruction of the fog image.

[0057] In some embodiments of this application, Figure 2 The network model for generating a dataset shown in the figure extracts features of the fog image and the clean image, and then reconstructs the extracted features. At the same time, it can better constrain the conversion effects between each other, making the synthetic fog image more natural in specific applications and enhancing the effect of synthetic fog on the real clean image.

[0058] In this embodiment, it is assumed that a latent space is shared between the foggy image and the clear image, so the feature connection from the foggy image to the clear image can be completed through the latent space. The shared latent image domain is used to store features and output the features to the first decoder and the second decoder. The features include the features extracted by the first encoder for the clean image and the features extracted by the second encoder for the foggy image; the training set and the validation set of the generated data set network model both include paired clean images and foggy images, and the synthetic foggy image of the generated data set network model is obtained.

[0059] See also Figure 3 , is a schematic diagram of a shared potential image domain of a network model for generating a data set in an embodiment of the present application. In some embodiments of the present application, it is assumed that a potential space is shared between the fog image and the clean image, so the feature connection from the fog image to the clean image can be completed through the potential space. The shared potential image domain assumes that different image domains share a potential space. Based on the generative adversarial network, this point is used to train and iterate the input vector through weight sharing, and the potential connection and image joint distribution of different image domains are learned, so that data with the characteristics of the corresponding image domain can be generated. In some embodiments of the present application, taking the generation of fog image datasets as an example, a potential space is shared between the fog image and the clean image, and the conversion from the fog image to the clean image can be completed through the potential space. That is, there are fog image domains and foggy image domains, and the encoded joint distribution of images can be learned in the shared potential image domain, and the generation network can restore the images in the fog image and the clean image through the image domain. The network model for generating a data set disclosed in the present application can well process real fog images and synthetic fog images, and restore and reconstruct fog images by first changing the encoding and then decoding, combined with the generative adversarial network.

[0060] It can be seen from the above technical scheme that a data set generation network model provided by the present application includes: an image coding module, a coupled generative adversarial network module, an adversarial discriminant module and a synthesis module; the image coding module includes a clean image data set input port and a fog image data set input port, the coupled generative adversarial network module includes a first generation unit and a second generation unit, and the adversarial discriminant module includes a first discriminator and a second discriminator; the first generation unit includes a first encoder, a shared latent image domain and a first decoder, and the second generation unit includes a second encoder, the shared latent image domain and a second decoder; the clean image data set input port is used to read a clean image data set, and the first encoder is used to extract features from the clean image data set. The shared latent image domain is used to store the extracted features, the first decoder is used to read the features stored in the shared latent image domain and transmit the features to the first discriminator, and the first discriminator is used to reconstruct the clean image data set after feature extraction; the fog image data set input port is used to read the fog image data set, the second encoder is used to extract features from the fog image data set, the shared latent image domain is used to store the extracted features, the second decoder is used to read the features stored in the shared latent image domain and transmit the features to the second discriminator, and the second discriminator is used to reconstruct the fog image data set after feature extraction; the synthesis module combines the reconstructed clean image data set and the reconstructed fog image data set into a synthetic fog image.

[0061] In the actual application process; This application is based on the neural network framework of the coupled generative adversarial network, and adopts a semi-supervised learning strategy in the network iterative training process. The network-generated pictures can learn the depth information of the real pictures, simulate the feature distribution of the images in the real scene, realize the image generation across the image domain, and the images are more realistic. At the same time, this application helps to generate paired data sets for deep learning network training, which helps to improve the effect of the target task in the real scene. After the network training obtains the generative model, it can make good use of the existing clean and clear data sets to generate the required data sets. For example, a paired clean-fog data set is made. Thereby, the problem of poor model effect caused by the lack of paired training data and the lack of real training data can be reduced. When it is necessary to perform target recognition in a foggy environment, the recognition network often uses an artificially synthesized data set for training. The inconsistency between the data set and the real scene will lead to the problem of low recognition rate of the trained model in the real scene.

[0062] In some embodiments of the present application, the generated dataset network model is optimized by the following loss functions: coupled generative adversarial loss function, reconstruction loss function, perceptual loss function, L1 loss function and total variational loss function.

[0063] In some embodiments of the present application, the coupling generates an adversarial loss function, specifically:

[0064] L 生成 =L GAN (D h ,J h ,G h (I c ))+L GAN (D c ,J c ,G c (I h ));

[0065] Among them, L 生成 is the coupled generative adversarial loss function, L GAN is the classic generative adversarial loss model function, D h To identify the clean image domain, J h The original real clean image dataset G is the input of the network. h Clean image dataset generated based on latent feature distribution, I c is the fog image domain with random noise distribution, D c To distinguish the foggy image domain, J c The original real fog image dataset G is the input of the network. c is a fog image dataset generated based on latent feature distribution. h A clean image dataset with random noise distribution.

[0066] The classic generative adversarial loss model function is specifically:

[0067] L GAN =P J [logD(J)]+P I [log(1-D(Gen(I))];

[0068] Among them, L GAN is the classic generative adversarial loss model function, P J is the expected value of the distribution function of the original real image data set input to the network, D is the discriminant network, J is the original real image data set input to the network, P I is the expected value of the distribution function of the random noise distribution, and Gen(I) is the sample generated based on the random noise.

[0069] Specifically, the loss function of the Generative Adversarial Networks (GAN). The loss calculation is generated at the output of D (discriminator), and the output is usually a false / true judgment, so the overall optimization process is similar to the binary cross entropy function. During the training process, the generator and the discriminator are trained separately. First, the generator Gen is fixed, and the discriminator D is trained. The optimization requirements of the model are that the larger the D(J), the more accurately the discriminator can identify the real image data set as the real image data set, and the smaller the D(Gen(I)), the more correctly the discriminator can distinguish false samples. When it comes to the stage of training the generator, the discriminator is fixed, and the model optimization requires that the generator's generation is more realistic, so that the discriminator makes a misjudgment, that is, the D(Gen(I)) result is close to 1, so that the total value is minimized.

[0070] In some embodiments of the present application, the reconstruction loss function is specifically:

[0071] L 重建 =||G C (E C (J C )-J C )||+||G h (E h (J h )-J h )||;

[0072] Among them, L 重建 is the reconstruction loss function, G C To generate the fog image based on the features of the generated network, E C To encode the fog image, J C Input the original real fog image to the network, G h To generate clean images based on features of the generated network, E h To encode the clean image, J h Input the original real clean image to the network;

[0073] Specifically, by using generative adversarial loss and reconstruction loss, we can ensure the authenticity of the reconstructed image and the similarity of the results of cross-domain image conversion as much as possible.

[0074] In some embodiments of the present application, the perceptual loss function is specifically:

[0075]

[0076] Among them, L p is the perceptual loss function, The features of the network generated image and the original control image extracted from the convolutional neural network VGG16, G C To generate the fog image based on the features of the generated network, I c is the fog image domain with random noise distribution, J h The original real clean image dataset G is the input of the network. h To generate clean images based on features of the generated network, I h is a clean image dataset with random noise distribution, J C Input the original real fog image to the network.

[0077] Specifically, the perceptual loss can use the features extracted by the convolutional neural network as part of the objective function. By comparing the feature values ​​of the image to be generated and the original reference image after passing through the convolutional neural network, the image to be generated and the reference image can be more similar in semantics and perception.

[0078] In some embodiments of the present application, the L1 loss model function is specifically:

[0079] L1=||G c (I c )-J h ||±||G h (I h )-J c ||;

[0080] Among them, L1 is the L1 loss model function, G C To generate the fog image based on the features of the generated network, I c is the fog image domain with random noise distribution, J h The original real clean image dataset G is the input of the network. h To generate clean images based on features of the generated network, I h is a clean image dataset with random noise distribution, J C The original real fog image is input to the network. Specifically, in the supervised learning part, the network is iteratively trained using paired synthetic images, and an L1 loss model function is additionally used in the supervised training part to ensure the similarity between the image G generated by the network of the present application and the real control image J.

[0081] In some embodiments of this application, the total variation loss function is specifically:

[0082]

[0083] Among them, L 变分 is the total variational loss function, represents the horizontal gradient difference, Represents the vertical gradient difference, G C To generate the fog image based on the features of the generated network, I c is the fog image domain with random noise distribution.

[0084] Specifically, in the unsupervised learning part, the network is trained using an unpaired real dataset. In the unsupervised training, we use the total variation loss to make the visual effect of the image better and closer to reality by focusing on the texture and details of the image.

[0085] By combining the above loss functions, the advantages of supervised learning and unsupervised learning can be brought into play in the semi-supervised training process, not only ensuring that the generated image meets the characteristics of the target image domain, but also learning the characteristic distribution of the real image. After the network is trained iteratively, when the model converges to a good accuracy, the obtained training weights can be saved, and then our generative model can be obtained, so that the network can generate image datasets that simulate the real distribution based on the existing clean dataset in an end-to-end manner. It can be used to generate paired datasets required for deep learning image training tasks, such as image datasets of foggy weather conditions.

[0086] In an embodiment of the present application, the first encoder and the second encoder respectively include a plurality of first convolution blocks and a plurality of first residual blocks connected in sequence, and the first decoder and the second decoder respectively include a plurality of second residual blocks and a plurality of second convolution blocks connected in sequence; the output end of the first residual block is connected to the input end of the shared potential image domain, and the output end of the shared potential image domain is connected to the input end of the second residual block; the output end of the first convolution block is jump-connected to the input end of the second convolution block; the output end of the second convolution block is jump-connected to the input end of the first convolution block; the convolution block is used to extract high-level features of the image, and the residual block is used to learn detailed information of different features of the image.

[0087] Specifically, in some embodiments of the present application, the first convolution block and the second convolution block are used to extract high-level features of the input image, and the first residual block and the second residual block are responsible for learning the detailed information of different features of the image, which is convenient for the restoration and reconstruction of the image. The first convolution block and the second convolution block complete the reconstruction process of the image from the high-level features to the output image (clean image or fog image). The encoder and the decoder are closely connected through jump connections. After each layer of convolution blocks, there will be a jump link connected to the corresponding convolution block. The jump connection links the output feature map of the encoder's convolution block to the input of the corresponding decoder, and splices it at the dimensional level, so that the feature map output by the decoder can include the feature information of the original image. The jump connection enables the detailed texture of the image to be better learned. Through the connection of two layers of jump connections in high and low dimensions, different position information of the image can be processed separately, further ensuring the consistency of the output image and the input image, as well as the authenticity of the reconstruction.

[0088] In some embodiments of the present application, the first encoder and the second encoder include 3 first convolution blocks and 4 first residual blocks respectively; the first decoder and the second decoder include 4 second residual blocks and 2 second convolution blocks respectively. The first residual block and the second residual block are respectively composed of a plurality of sub-residual blocks cascaded; the sub-residual blocks at any level include the first convolution block and the second convolution block connected in sequence, which are used to process the output results of the sub-residual blocks of the previous level and the output results of the sub-residual blocks of the previous level, and output the processing results to the sub-residual blocks of the next level and the sub-residual blocks of the next next level.

[0089] Specifically, in some embodiments of the present application, after encoding and decoding by the encoder and decoder, a generated image is obtained, and the generated image is input into the discrimination module for discrimination. The first encoder and the second encoder include 3 first convolution blocks and 4 first residual blocks, respectively; the first decoder and the second decoder include 4 second residual blocks and 2 second convolution blocks, respectively. Compare the output results of the discriminant network at different granularities with the actual feature map to determine whether the network output meets expectations, apply weight sharing to the training results that do not meet expectations, calculate the discrimination loss, and return it to the network to update the parameters. A multi-level discriminant network is used to sample and compare the output results generated by the network model of the generated data set, and make coarse-grained and fine-grained comparisons, thereby expanding the discrimination accuracy of the adversarial discriminant module.

[0090] In some embodiments of the present application, in the coupled generative adversarial network module, the stride convolution used by the convolution block is changed to other upsampling methods (such as bilinear interpolation) to reconstruct the image, but the visual effect achieved is not as good as the reconstruction method of stride convolution, and it is difficult to more finely reflect the original details of the image, which easily causes blurred edges or unclear textures. Use stride convolution in upsampling: In deep learning, it is represented as a reverse process of convolution. According to the size of the convolution kernel and the size of the output, the image size before convolution can be restored instead of the original value. The specific implementation steps are: pad the input with zeros on all four sides; the number of zero padding on one side depends on the size of the convolution kernel; then rotate the convolution kernel 180 degrees; and perform direct convolution on the new input. Bilinear interpolation is used in upsampling: Linear interpolation refers to a method of using a straight line connecting two known quantities to determine the value of an unknown quantity between the two known quantities. Bilinear interpolation is a type of interpolation algorithm and an extension of linear interpolation. The four real pixel values ​​around the target point in the original image are used to jointly determine a pixel value in the target image. The core content is to perform linear interpolation in two directions respectively.

[0091] In some embodiments of the present application, when the model converges to a good accuracy, the obtained training weights can be saved to obtain the final generated data set network model. At this time, the model can synthesize a new target fog image from the existing fog image and clean image in an end-to-end manner.

[0092] In some embodiments of the present application, for example, a foggy image target detection network requires several paired image data sets for end-to-end training. In order to improve the detection effect of the network in a real environment, a control data set that conforms to the distribution of real images needs to be produced. The existing clean image data set A can be used to produce a paired fog image data set A* based on A through our network model. Specifically, our network must first be trained, and the training strategy is semi-supervised training, that is, a mixed data set is used. The training data set consists of two parts, one of which is image data C under foggy conditions in real scenes (which does not need to correspond to the images in data set A). The purpose of this part is to learn the image distribution of real fog in the network to improve the accuracy of image tasks in real scenes; the other part is a synthetic fog image A' based on data set A, and the purpose of this part is to ensure the similarity of the reconstructed image and the clean image in pixels. Through the semi-supervised training strategy, it can be ensured that under the condition of the similarity between the reconstructed image and the reference image, the synthesized reconstructed image simulates the distribution of the real fog image. The input part of the network includes clean data and control data, a clean data set A, and a control data set of image data C and synthetic data A', and the network output is the generated data set A*. A and A* are control data sets, which can provide paired training data sets for specific image tasks such as image target detection networks under foggy weather conditions.

[0093] See also Figure 4 , is a schematic diagram of a coupled generative adversarial network module for generating a dataset network model in an embodiment of the present application. In some embodiments of the present application, the coupled generative adversarial network module includes a first generator and a second generator, the first generator and the second generator correspond to an image domain respectively, and through the constraint of weight sharing, the coupled generative network, the first generator and the second generator can learn the joint distribution of images in two different domains during training. The coupled generative adversarial network uses the idea of ​​hierarchical feature representation of deep neural networks to share the weights of semantic decoding of the generative adversarial network, and uses the details of the two image domains to confuse their respective identification models. After sufficient training iterations at a certain learning rate, the first generator and the second generator can transform the input random image data set into a fog image and a clean image that conform to the distribution of the corresponding image domain.

[0094] In order to realize the practical application of the above model, the second aspect of the embodiment of the present application also provides a fog map generation method based on the network model of the generated data set, see Figure 5 , is a schematic diagram of a workflow for generating a data set network model according to an embodiment of the present application, which includes:

[0095] Step S101: Obtain an image data set to be processed.

[0096] In step S101, the image data set in the embodiment of the present application is a "film library", which collects meteorological scene data of a foggy weather database as a basic meteorological image data set. For example, the meteorological scene data includes collected real foggy weather scenes, simulated virtual foggy weather scenes, movie foggy weather scenes, and various typical foggy weather scenes on the Internet. The basic foggy weather scenes are expanded by covering different lighting conditions, different fog amounts, rainfall influence conditions, and characteristics of different seasons.

[0097] Step S102: pre-processing the image data set.

[0098] In step S102, the preprocessing provided by the embodiment of the present application is specifically to convert the image data set into an HSV format image data set; extract the brightness and saturation in the HSV format image data set; use a linear calculation formula to convert the brightness and saturation into depth information corresponding to different pixels in the image data set; combine the depth information in the original image with the original image to obtain the image data set. Combining the depth information in the original image with the original image can highlight the depth information related to the brightness and saturation in the image, thereby serving as an input source for the network.

[0099] In some embodiments of the present application, the linear calculation formula is specifically:

[0100] p(i)=a1v(i)+a2s(i)+z;

[0101] Among them, p is the depth information, i is the image dataset, a1 and a2 are known parameters, v is the brightness, s is the saturation, and z is random noise.

[0102] Step S103: input the preprocessed image data set into a pre-built data set generation network model.

[0103] Step S104: Obtaining the synthetic fog image output by the network model for generating the data set.

[0104] In step S104, the specific function of the adversarial discrimination module in some embodiments of the present application is to use the adversarial discrimination module to process the output of the generated part, sample it to three different image scales, and calculate its loss through layering, and continuously reduce the loss through training iterations, so that the image generated by the network is consistent with the target image domain characteristics, and the target image data set is obtained.

[0105] The network model for generating a dataset disclosed in the present application interprets the image into a shared potential image domain by first transforming the encoding and then decoding, and then reconstructs the image. The reconstructed foggy image and clean image are then input into an adversarial discrimination module. The adversarial discrimination module can effectively recognize the image to achieve the purpose of identifying the foggy image and the clean image, thereby synthesizing a new foggy image.

[0106] A third aspect of some embodiments of the present application provides a computer device, including:

[0107] Memory, used to store computer programs.

[0108] A processor is used to implement the steps of the fog map generation method based on generating a data set network model as described in the second aspect of the embodiment of the present application when executing the computer program.

[0109] It can be seen from the above technical solution that a network model for generating a data set includes: an image encoding module, a coupled generative adversarial network module, an adversarial discriminant module and a synthesis module; the image encoding module includes a clean image data set input port and a fog image data set input port, the coupled generative adversarial network module includes a first generation unit and a second generation unit, and the adversarial discriminant module includes a first discriminator and a second discriminator; the first generation unit includes a first encoder, a shared potential image domain and a first decoder, and the second generation unit includes a second encoder, the shared potential image domain and a second decoder; the clean image data set input port is used to read a clean image data set, the first encoder is used to extract features from the clean image data set, and the shared The latent image domain is used to store the extracted features, the first decoder is used to read the features stored in the shared latent image domain and transmit the features to the first discriminator, and the first discriminator is used to reconstruct the clean image data set after feature extraction; the fog image data set input port is used to read the fog image data set, the second encoder is used to extract features from the fog image data set, the shared latent image domain is used to store the extracted features, the second decoder is used to read the features stored in the shared latent image domain and transmit the features to the second discriminator, and the second discriminator is used to reconstruct the fog image data set after feature extraction; the synthesis module combines the reconstructed clean image data set and the reconstructed fog image data set into a synthetic fog image.

[0110] In actual application, this application proposes a neural network framework based on a coupled generative adversarial network. By adopting a semi-supervised learning strategy and a specific combined loss function in the network iterative training process, the images generated by the network can learn the depth information of the real image, while simulating the feature distribution of images in real scenes, realizing image generation across image domains, and making the images more realistic.

[0111] The present application is described in detail above in conjunction with specific implementation methods and exemplary examples, but these descriptions cannot be understood as limiting the present application. Those skilled in the art understand that, without departing from the spirit and scope of the present application, a variety of equivalent replacements, modifications or improvements can be made to the technical solution of the present application and its implementation methods, all of which fall within the scope of the present application. The scope of protection of the present application shall be subject to the attached claims.

Claims

1. A network model for generating a data set, characterized in that: include: Image encoding module, coupled generative adversarial network module, adversarial discrimination module and synthesis module; The image encoding module includes a clean image data set input port and a fog image data set input port, the coupled generative adversarial network module includes a first generation unit and a second generation unit, and the adversarial discriminant module includes a first discriminator and a second discriminator; the first generation unit includes a first encoder, a shared potential image domain and a first decoder, and the second generation unit includes a second encoder, the shared potential image domain and a second decoder; The clean image data set input port is used to read the clean image data set, the first encoder is used to extract features from the clean image data set, the shared potential image domain is used to store the extracted features, the first decoder is used to read the features stored in the shared potential image domain and transmit the features to the first discriminator, and the first discriminator is used to reconstruct the clean image data set after the feature extraction; The fog image dataset input port is used to read the fog image dataset, the second encoder is used to extract features from the fog image dataset, the shared potential image domain is used to store the extracted features, the second decoder is used to read the features stored in the shared potential image domain and transmit the features to the second discriminator, and the second discriminator is used to reconstruct the fog image dataset after feature extraction; the synthesis module combines the reconstructed clean image dataset and the reconstructed fog image dataset into a synthetic fog image; the generated dataset network model is optimized by the following loss functions: coupled generation adversarial loss function, reconstruction loss function, perceptual loss function, L1 loss function and total variation loss function; The coupling generates an adversarial loss function, specifically: L 生成 =L GAN (D h ,J h ,G h (I c ))+L GAN (D c ,J c ,G c (I h )); Among them, L 生成 is the coupled generative adversarial loss function, L GAN is the classic generative adversarial loss model function, D h To identify the clean image domain, J h The original real clean image dataset G is the input of the network. h Clean image dataset generated based on latent feature distribution, I c is the fog image domain with random noise distribution, D c To distinguish the foggy image domain, J c The original real fog image dataset G is the input of the network. c is a fog image dataset generated based on latent feature distribution. h A clean image dataset with random noise distribution; The classic generative adversarial loss model function is specifically: L GAN =P J [logD(J)]+P I [log(1-D(Gen(I))]; Among them, L GAN is the classic generative adversarial loss model function, P J is the expected value of the distribution function of the original real image data set input to the network, D is the discriminant network, J is the original real image data set input to the network, P I is the expected value of the distribution function of random noise distribution, Gen(I) is the sample generated based on random noise; The reconstruction loss function is specifically: HIM 重建 =||G C (HE C (J C )-J C )||+||G h (HE h (J h )-J h )||; Among them, L 重建 is the reconstruction loss function, G C To generate the fog image based on the features of the generated network, E C To encode the fog image, J C Input the original real fog image to the network, G h To generate clean images based on features of the generated network, E h To encode the clean image, J h Input the original real clean image to the network; The perceptual loss function is specifically: Among them, L p is the perceptual loss function, The features of the network generated image and the original control image extracted from the convolutional neural network VGG16, G C To generate the fog image based on the features of the generated network, I c is the fog image domain with random noise distribution, J h The original real clean image dataset G is the input of the network. h To generate clean images based on features of the generated network, I h is a clean image dataset with random noise distribution, J C Input the original real fog image to the network; The L1 loss function is specifically: L1=||G c (I c )-J h ||±||G h (I h )-J c ||; Among them, L1 is the L1 loss function, G C To generate the fog image based on the features of the generated network, I c is the fog image domain with random noise distribution, J h The original real clean image dataset G is the input of the network. h To generate clean images based on features of the generated network, I h is a clean image dataset with random noise distribution, J C Input the original real fog image to the network; The total variation loss function is specifically: Among them, L 变分 is the total variational loss function, represents the horizontal gradient difference, Represents the vertical gradient difference, G C To generate the fog image based on the features of the generated network, I c is the fog image domain with random noise distribution.

2. A network model for generating a data set according to claim 1, characterized in that: The first encoder and the second encoder respectively include a plurality of first convolution blocks and a plurality of first residual blocks connected in sequence, and the first decoder and the second decoder respectively include a plurality of second residual blocks and a plurality of second convolution blocks connected in sequence; the output end of the first residual block is connected to the input end of the shared potential image domain, and the output end of the shared potential image domain is connected to the input end of the second residual block; the output end of the first convolution block is jump-connected to the input end of the second convolution block; the output end of the second convolution block is jump-connected to the input end of the first convolution block; the convolution block is used to extract high-level features of the image, and the residual block is used to learn detailed information of different features of the image.

3. A network model for generating a data set according to claim 2, characterized in that: The first encoder and the second encoder include 3 first convolution blocks and 4 first residual blocks respectively; the first decoder and the second decoder include 4 second residual blocks and 2 second convolution blocks respectively.

4. A network model for generating a data set according to claim 2, characterized in that: The first residual block and the second residual block are respectively composed of a plurality of sub-residual blocks cascaded in series; the sub-residual blocks at any level include a first convolution block and a second convolution block connected in sequence, which are used to process the output results of the sub-residual block of the previous level and the output results of the sub-residual block of the previous level, and output the processing results to the sub-residual block of the next level and the sub-residual block of the next-next level.

5. A fog image generation method based on a data set generation network model, characterized in that: include: Obtain the image dataset to be processed; Preprocess the image dataset; The preprocessing is specifically as follows: Convert the image dataset into an HSV format image dataset; Extract the brightness and saturation of the HSV format image data set; Generating depth information corresponding to different pixels in the image data set according to the brightness and saturation; Combining the depth information in the original image with the original image to obtain an image dataset; Inputting the preprocessed image data set into the data set generation network model as claimed in claim 1; Obtain a synthetic fog image output by the network model for generating the dataset.

6. A computer device, characterized in that: include: Memory for storing computer programs; A processor is used to implement the steps of the fog map generation method based on generating a data set network model as claimed in claim 5 when executing the computer program.

Images