A method for generating image dataset

Abstract

The present invention proposes a simple and effective method for generating a comparison data set of haze-containing images and haze-removed images based on a generative adversarial network (GAN). The method can be specifically used to solve the bottleneck and difficulty of the current training of deep learning networks for haze removal, which is the lack of image comparison data sets. The main features are: (1) a texture-focused generative adversarial network is proposed to solve the problem of being unable to automatically generate data sets; (2) in view of the low training efficiency of the generative adversarial network and the easy distortion of the generated images, a feature enhancement generation model based on transfer learning is proposed in the generator design to solve the problem of low generation efficiency; (3) in the network discriminator design, a dual discriminator discrimination model with a global discriminator and a local discriminator is proposed to solve the problem of distortion of texture details in the generated images.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method for generating a comparison image data set based on a generative adversarial network. Background Art

[0002] High-resolution remote sensing images are widely used in agriculture, cities, resources, forests, mine surveys and monitoring, etc., but the widespread use of high-resolution remote sensing images is based on high-quality images. As the global haze situation worsens, haze reduces atmospheric visibility, obscures a lot of ground object information, and causes serious loss of useful image information, which has a great impact on applications such as ground object extraction, classification, and subsequent analysis and understanding based on images. High-resolution remote sensing image datasets are currently lacking overall. The mainstream ones are UC-Merced dataset, WHU-RS dataset, RSSCN7 dataset, and the AID dataset, which is currently the largest dataset produced by Wuhan University in 2017 and includes 10,000 high-resolution images of 30 categories. Due to the lack of corresponding effective high-resolution remote sensing image datasets, it is necessary to create datasets for research. Therefore, there is an urgent need for an efficient, high-quality, and fully automated method for generating high-resolution remote sensing image datasets to break through the bottleneck of high-resolution remote sensing image dehazing. Summary of the invention

[0003] In order to solve the problem of efficiently, high-quality and fully automated generation of high-resolution remote sensing image datasets, the present invention proposes a method for generating high-resolution remote sensing image datasets based on a generative adversarial network.

[0004] The purpose of the present invention is to implement a color image edge detection method through the following technical solution, comprising the following steps:

[0005] The following steps are involved:

[0006] Step 1: Establish a texture-attention-based generative adversarial network model (Texture Attention-GAN) to enhance image texture features and achieve fully automated processing of high-resolution remote sensing images;

[0007] Step 2: In the design of the generative adversarial network generator, a feature-enhanced generation model based on transfer learning is proposed to solve the problem of low generation efficiency;

[0008] Step 3: In the discriminator design of the network, a dual discriminator model with a global discriminator and a local discriminator is proposed to solve the problem of distortion of texture details in the generated image.

[0009] Step 4: Use the generative adversarial network to construct a high-resolution remote sensing image dataset with haze and clear contrast.

[0010] The specific content is: In order to solve the problem that the currently commonly used remote sensing image dehazing methods are overly dependent on prior information and cannot achieve fully automated dehazing of remote sensing images, a generative adversarial network model that can achieve unsupervised learning is used to automatically dehaze high-resolution remote sensing images. In order to solve the problem that the noise caused by haze will highlight the low-frequency information of the image, resulting in the loss of useful information such as target texture, it is proposed to introduce a texture attention map generation module into the generative adversarial network and establish a texture attention-based generative adversarial network model (Texture Attention-GAN) to ensure that the high-resolution remote sensing image dehazing method in this study can achieve both automated processing and good preservation of ground object detail information.

[0011] The network can be divided into three parts: the texture attention map generation module, the generator part and the discriminator part. This design framework generates a texture attention map before the generator. The texture attention map is a crucial part of the designed network, because usually the useful information such as texture in an image is high-frequency, which can enable the generator to focus on the texture area of ​​the target area of ​​the image. During the training process, the texture attention map pays more attention to the texture details of the relevant structure, thereby improving the generation efficiency of the generator and improving the quality of the details of the generated image. The texture attention map uses an improved cellular neural network (CNN) to extract the high-frequency part of the clear image corresponding to the original haze high-resolution remote sensing image, and generates a binary image. If the pixel position has a high-frequency part, the value in the binary map is 1, otherwise it is 0. Here, the eigenvalue statistical setting is used for the setting of the texture extraction threshold. The generated texture attention map and the input image with haze are spliced ​​as the input of the generator.

[0012] The objective function of Texture Attention-GAN is:

[0013]

[0014] Among them, G represents the generative network and re-predicts the discriminator; I x It is a sample extracted from a high-resolution remote sensing image dataset with haze; clear is a sample corresponding to the no-haze dataset.

[0015] During the training process, the values ​​of the parameters directly determine the quality of the network operation. Here we try to use the initial learning rate set to 0.0002 and the batch size to 1. In each iteration, the discriminator is first trained with the clear image and the dehazed image generated in the previous iteration to minimize the discriminator's loss function, and then the generator is trained with the hazy image to minimize the generator's loss function.

[0016] First, the input of this generator is the splicing of the texture attention map and the image with haze. The addition of the texture attention map plays a role in feature enhancement. In order to solve the problem of low generation efficiency of the generative adversarial network, the transfer learning theory is adopted here, and the parameters of the mature VGG-16 model are used. The main key problem of introducing the VGG-16 model in the texture-attention-based generative adversarial network is to choose the appropriate migration strategy and parameter transplantation problem. Here we adopt the migration strategy of model splicing, and mainly use the VGG-16 model as a feature extraction device. The specific approach is to remove the output layer, and then treat the remaining entire network as a fixed feature extraction machine, and combine it with other structures of the generator, so as to apply it to the new data set. In this way, using the already trained parameters of VGG-16 can reduce the work of underlying feature extraction, thereby improving generation efficiency.

[0017] In addition to the VGG-16 structure, the other parts of this generator model use two lower convolutional layers with a stride of 2, connecting 5 ResNet blocks and 2 upper convolutional layers. The addition of skip connections is motivated by observing the detailed structure of the texture attention map corresponding to the input image. Skip connection is a residual network structure introduced in ResNet. The difference between it and the ordinary convolutional network is that an additional transmission line is connected directly from the input source to the output source. This is an identity mapping used for residual calculation. Its effect is to prevent the gradient diffusion and degradation caused by the increase in the number of network layers, resulting in blurred output.

[0018] Texture Attention-GAN's autoencoder uses a multi-scale loss function

[58] The multi-scale loss extracts features from different decoder layers to form outputs of different sizes. By adopting this approach, more information can be obtained from different scales.

[0019] The loss function of the encoder is defined here as:

[0020]

[0021] Among them, S i represents the i-th output extracted from the decoding layer, T i Indicates that S i The same scale of true value annotation, are weights at different scales. We put more weight on larger scales.

[0022] In general, the loss function of the generator can be expressed as:

[0023]

[0024] in,

[0025] The entire discriminant model consists of two discriminators, the two discriminators are the global discriminator D 1 and the local discriminator D 2 . Global Discriminator D 1 The entire image is inspected to check for inconsistencies, while the local discriminator D 2 Then check the specific texture area, here we use the texture attention map of clear high-resolution remote sensing images x The generator generates a texture attention map of the high-resolution remote sensing image after dehazing. G(x) To check whether the local texture is distorted, the global discriminator D is used. Both discriminators use a 5-layer convolutional neural network (CNN), including 4 3x3 convolutional layers and 1 fully connected layer, and use a sigmoid activation function. 1 The remote sensing image generated from the generator is compared with the clear remote sensing image. Local discriminator D 2 The remote sensing image generated from the generator is processed for texture extraction and then compared with the previously generated texture attention map.

[0026] In order to achieve better convergence, the discriminator D 1 The loss function used is:

[0027]

[0028] Among them, I clear is a sample from a haze-free dataset, I Dehaze is the image sample after dehazing. is the difference between the texture attention map of the generated image and the texture attention map of the clear image in the discriminator D2.

[0029]

[0030] Among them, D map Represents the process of CNN generating texture attention map.

[0031] In the construction of the entire data set, since this method requires a relatively large number of paired high-resolution remote sensing image data with haze and clear contrast for training, firstly, the conventional generative adversarial network model is used to generate 3200 pairs of high-resolution remote sensing images with haze and clear contrast as the training data set for the texture-attention-based generative adversarial network with the high-resolution remote sensing image of 0.8m of the GF-2 panchromatic. The image size is unified as 896×896. The conventional generative adversarial network GAN has two networks, G (Generator) and D (Discriminator). G is a generator for generating images. It receives a random noise z and generates images through this noise, which is recorded as G (z). D is a discriminator that determines whether a picture is "real". Its input parameter is x, x represents a picture, and the output D (x) represents the probability that x is a real picture. If it is 1, it means that it is 100% a real picture, and if the output is 0, it means that it cannot be a real picture. Using this principle, more high-resolution remote sensing image data can be generated. Taking into account the generalization ability of the model, the corresponding real scene images are selected based on the principle that the gap between classes should be small and the gap within classes should be large.

[0032] Then, we used the trained dehazing generative adversarial network model to process real remote sensing images containing haze, and continued to improve the construction of high-resolution remote sensing image datasets, so that the remote sensing datasets can eventually have more than 8,000 pairs of haze and clear high-resolution remote sensing images. The ratio of real image data to effective simulated image data in the dataset is at least 6:4. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of a dataset generation method (taking high-resolution remote sensing images as an example);

[0034] Figure 2 These are some images of the generated clear and haze comparison dataset. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Combination Figure 1-Figure 2 The present invention proposes a color image edge detection method, comprising the following steps:

[0037] The purpose of the present invention is to implement a color image edge detection method through the following technical solution, comprising the following steps:

[0038] The following steps are involved:

[0039] Step 1: Establish a texture-attention-based generative adversarial network model (Texture Attention-GAN) to enhance the image texture features and achieve fully automated processing of high-resolution remote sensing image dehazing without relying on prior knowledge;

[0040] Step 2: In the design of the generative adversarial network generator, a feature-enhanced generation model based on transfer learning is proposed to solve the problem of low generation efficiency;

[0041] Step 3: In the discriminator design of the network, a dual discriminator model with a global discriminator and a local discriminator is proposed to solve the problem of distortion of texture details in the generated image.

[0042] Step 4: Use the generative adversarial network to construct a high-resolution remote sensing image dataset with haze and clear contrast.

[0043] The specific content is: In order to solve the problem that the currently commonly used remote sensing image dehazing methods are overly dependent on prior information and cannot achieve fully automated dehazing of remote sensing images, a generative adversarial network model that can achieve unsupervised learning is used to automatically dehaze high-resolution remote sensing images. In order to solve the problem that the noise caused by haze will highlight the low-frequency information of the image, resulting in the loss of useful information such as target texture, it is proposed to introduce a texture attention map generation module into the generative adversarial network and establish a texture attention-based generative adversarial network model (Texture Attention-GAN) to ensure that the high-resolution remote sensing image dehazing method in this study can achieve both automated processing and good preservation of ground object detail information.

[0044] The network can be divided into three parts: the texture attention map generation module, the generator part and the discriminator part. This design framework generates a texture attention map before the generator. The texture attention map is a crucial part of the designed network, because usually the useful information such as texture in an image is high-frequency, which can enable the generator to focus on the texture area of ​​the target area of ​​the image. During the training process, the texture attention map pays more attention to the texture details of the relevant structure, thereby improving the generation efficiency of the generator and improving the quality of the details of the generated image. The texture attention map uses an improved cellular neural network (CNN) to extract the high-frequency part of the clear image corresponding to the original haze high-resolution remote sensing image, and generates a binary image. If the pixel position has a high-frequency part, the value in the binary map is 1, otherwise it is 0. Here, the eigenvalue statistical setting is used for the setting of the texture extraction threshold. The generated texture attention map and the input image with haze are spliced ​​as the input of the generator.

[0045] The objective function of Texture Attention-GAN is:

[0046]

[0047] Among them, G represents the generative network and re-predicts the discriminator; I x It is a sample extracted from a high-resolution remote sensing image dataset with haze; clear is a sample corresponding to the no-haze dataset.

[0048] During the training process, the values ​​of the parameters directly determine the quality of the network operation. Here we try to use the initial learning rate set to 0.0002 and the batch size to 1. In each iteration, the discriminator is first trained with the clear image and the dehazed image generated in the previous iteration to minimize the discriminator's loss function, and then the generator is trained with the hazy image to minimize the generator's loss function.

[0049] First, the input of this generator is the splicing of the texture attention map and the image with haze. The addition of the texture attention map plays a role in feature enhancement. In order to solve the problem of low generation efficiency of the generative adversarial network, the transfer learning theory is adopted here, and the parameters of the mature VGG-16 model are used. The main key problem of introducing the VGG-16 model in the texture-attention-based generative adversarial network is to choose the appropriate migration strategy and parameter transplantation problem. Here we adopt the migration strategy of model splicing, and mainly use the VGG-16 model as a feature extraction device. The specific approach is to remove the output layer, and then treat the remaining entire network as a fixed feature extraction machine, and combine it with other structures of the generator, so as to apply it to the new data set. In this way, using the already trained parameters of VGG-16 can reduce the work of underlying feature extraction, thereby improving generation efficiency.

[0050] In addition to the VGG-16 structure, the other parts of this generator model use two lower convolutional layers with a stride of 2, connecting 5 ResNet blocks and 2 upper convolutional layers. The addition of skip connections is motivated by observing the detailed structure of the texture attention map corresponding to the input image. Skip connection is a residual network structure introduced in ResNet. The difference between it and the ordinary convolutional network is that an additional transmission line is connected directly from the input source to the output source. This is an identity mapping used for residual calculation. Its effect is to prevent the gradient diffusion and degradation caused by the increase in the number of network layers, resulting in blurred output.

[0051] Texture Attention-GAN's autoencoder uses a multi-scale loss function

[58] The multi-scale loss extracts features from different decoder layers to form outputs of different sizes. By adopting this approach, more information can be obtained from different scales.

[0052] The loss function of the encoder is defined here as:

[0053]

[0054] Among them, S i represents the i-th output extracted from the decoding layer, T i Indicates that S i The same scale of true value annotation, are weights at different scales. We put more weight on larger scales.

[0055] In general, the loss function of the generator can be expressed as:

[0056]

[0057] in,

[0058] The entire discriminant model consists of two discriminators, the two discriminators are the global discriminator D 1 and the local discriminator D 2 . Global Discriminator D 1 The entire image is inspected to check for inconsistencies, while the local discriminator D 2 Then check the specific texture area, here we use the texture attention map of clear high-resolution remote sensing images x The generator generates a texture attention map of the high-resolution remote sensing image after dehazing. G(x) To check whether the local texture is distorted, the global discriminator D is used. Both discriminators use a 5-layer convolutional neural network (CNN), including 4 3x3 convolutional layers and 1 fully connected layer, and use a sigmoid activation function. 1 The remote sensing image generated from the generator is compared with the clear remote sensing image. Local discriminator D 2 The remote sensing image generated from the generator is processed for texture extraction and then compared with the previously generated texture attention map.

[0059] In order to achieve better convergence, the discriminator D 1 The loss function used is:

[0060]

[0061] Among them, I clear is a sample from a haze-free dataset, I Dehazeis the image sample after dehazing. is the difference between the texture attention map of the generated image and the texture attention map of the clear image in the discriminator D2.

[0062]

[0063] Among them, D map Represents the process of CNN generating texture attention map.

[0064] In the construction of the entire data set, since this method requires a relatively large number of paired high-resolution remote sensing image data with haze and clear contrast for training, firstly, the conventional generative adversarial network model is used to generate 3200 pairs of high-resolution remote sensing images with haze and clear contrast as the training data set for the texture-attention-based generative adversarial network with the high-resolution remote sensing image of 0.8m of the GF-2 panchromatic. The image size is unified as 896×896. The conventional generative adversarial network GAN has two networks, G (Generator) and D (Discriminator). G is a generator for generating images. It receives a random noise z and generates images through this noise, which is recorded as G (z). D is a discriminator that determines whether a picture is "real". Its input parameter is x, x represents a picture, and the output D (x) represents the probability that x is a real picture. If it is 1, it means that it is 100% a real picture, and if the output is 0, it means that it cannot be a real picture. Using this principle, more high-resolution remote sensing image data can be generated. Taking into account the generalization ability of the model, the corresponding real scene images are selected based on the principle that the gap between classes should be small and the gap within classes should be large.

[0065] Then, we used the trained dehazing generative adversarial network model to process real remote sensing images containing haze, and continued to improve the construction of high-resolution remote sensing image datasets, so that the remote sensing datasets can eventually have more than 8,000 pairs of haze and clear high-resolution remote sensing images. The ratio of real image data to effective simulated image data in the dataset is at least 6:4.

[0066] The above is a detailed introduction to a color image edge detection method provided by the present invention. In this method, specific examples are applied to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for generating an image dataset, characterized in that: The following steps are involved: Step 1: Establish a texture-focused generative adversarial network model to enhance the image texture features and achieve fully automated processing of generated images; Step 2: In the design of the generative adversarial network generator, a feature-enhanced generation model based on transfer learning is proposed to solve the problem of low generation efficiency; Step 3: In the discriminator design of the network, a dual discriminator model with a global discriminator and a local discriminator is proposed to solve the problem of distortion of texture details in the generated image; Step 4: Use the generative adversarial network to construct a high-resolution remote sensing image dataset with haze and clear contrast; It is characterized in that, in view of the problem that the current image generation method cannot realize fully automated image processing, a generative adversarial network model that can realize unsupervised learning is used to automatically generate high-resolution remote sensing images. In view of the problem that the noise caused by haze will highlight the low-frequency information of the image and cause the loss of useful information such as target texture, it is proposed to introduce a texture attention map generation module into the generative adversarial network, and establish a texture-attention-based generative adversarial network model to ensure that the high-resolution remote sensing image generation in this study can be automated and the detailed information of the ground objects can be well preserved. The network can be divided into three parts: a texture attention map generation module part, a generator part and a discriminator part. The design framework generates a texture attention map before the generator. The texture attention map is crucial in the designed network. The important part is that usually the useful information such as texture in an image is high-frequency, which can make the generator focus on the texture area of ​​the target area of ​​the image. During the training process, the texture attention map pays more attention to the texture details of the relevant structure, thereby improving the generation efficiency of the generator and improving the quality of the generated image details. The texture attention map uses the improved cellular neural network to extract the high-frequency part of the clear image corresponding to the original haze high-resolution remote sensing image to generate a binary image. If there is a high-frequency part at the pixel position, the value in the binary map is 1, otherwise it is 0. The setting of the texture extraction threshold adopts the eigenvalue statistical setting. The generated texture attention map and the input image with haze are spliced ​​as the input of the generator; the objective function of the generative adversarial network model based on texture attention is: , Among them, G represents the generative network and re-predicts the discriminator; I x It is a sample extracted from a high-resolution remote sensing image dataset with haze; clear is a sample in the haze-free dataset. During the training process, the value of the parameter directly determines the quality of the network operation. Here we try to use the initial learning rate set to 0.0002 and the batch size to 1. In each iteration, the discriminator is first trained with the clear image and the dehazed image generated in the previous iteration to minimize the loss function of the discriminator, and then the generator is trained with the haze image to minimize the generator loss function. The input of the generator is the concatenation of the texture attention map and the haze image. The addition of the texture attention map plays a role in feature enhancement. To address the problem of low generation efficiency of the generative adversarial network, the transfer learning theory is adopted here. The parameters of the mature VGG-16 model are used. The main key problem of introducing the VGG-16 model into the texture-attention-based generative adversarial network is to choose the appropriate migration strategy and parameter transplantation. Here we adopt the migration strategy of model concatenation, mainly using the VGG-16 model as a feature extraction device. The specific approach is to remove the output layer and then treat the remaining entire network as a fixed feature extraction machine, which is combined into other structures of the generator and applied to the new data set. In this way, the trained parameters of VGG-16 can reduce the work of underlying feature extraction, thereby improving generation efficiency. In addition to the VGG-16 structure, the other parts of this generator model use two lower convolutional layers with a stride of 2, connected to 5 ResNet blocks and 2 upper convolutional layers. By observing the detailed structure of the texture attention map corresponding to the input image, the skip connection is motivated to add. The skip connection is a residual network structure introduced in ResNet. The difference between it and the ordinary convolutional network is that it connects an additional transmission line directly from the input source to the output source. This is an identity mapping used for residual calculation. Its effect is to prevent the gradient diffusion and degradation caused by the increase in the number of network layers, resulting in blurred output; The autoencoder of the texture-attention-based generative adversarial network model adopts a multi-scale loss function. The multi-scale loss extracts features from different decoder layers to form outputs of different sizes. By adopting this method, more information can be obtained from different scales. The loss function of the encoder is defined here as: Among them, S i represents the i-th output extracted from the decoding layer, T i Indicates that S i The same scale of true value annotation, are weights of different scales. We put more weights on larger scales. In general, the loss function of the generator can be expressed as: in, The entire discriminant model consists of two discriminators, namely the global discriminator D1 and the local discriminator D 2， The global discriminator D1 looks at the entire image to check for inconsistencies, while the local discriminator D2 looks at specific texture areas, using the texture attention map of a clear high-resolution remote sensing image. x The generator generates a texture attention map of the high-resolution remote sensing image after dehazing. G(x) To check whether the local texture is distorted, both discriminators use a 5-layer convolutional neural network, including 4 3x3 convolutional layers and 1 fully connected layer, and use a sigmoid activation function. Specifically, the global discriminator D1 compares the remote sensing image generated by the generator with the clear remote sensing image, and the local discriminator D2 performs texture extraction on the remote sensing image generated by the generator, and then compares it with the previously generated texture attention map. In order to better achieve convergence, the loss function used by the discriminator D1 is: Among them, I clear is a sample from a haze-free dataset, I Dehaze is the image sample after dehazing. is the difference between the texture attention map of the generated image and the texture attention map of the clear image in the discriminator D2, Among them, D map Represents the process of CNN generating texture attention map.

2. The method for generating an image dataset according to claim 1, characterized in that: In the construction of the entire data set, since this method requires a relatively large amount of paired high-resolution remote sensing image data of haze and clear contrast for training, firstly, the conventional generative adversarial network model is used to generate 3200 pairs of high-resolution remote sensing images of haze and clear contrast as the training data set for the texture-attention-based generative adversarial network with the GF-2 panchromatic 0.8m high-resolution remote sensing image. The image size is unified as 896×896. The conventional generative adversarial network GAN has two networks, the generator G and the discriminator D. G is a generator that generates images. It receives a random noise z and generates images through this noise, denoted as G(z). D is a discriminator that determines whether a picture is "real". Its input parameter is x. , x represents a picture, and the output D(x) represents the probability that x is a real picture. If it is 1, it means that it is 100% a real picture, and if the output is 0, it means that it cannot be a real picture. Using this principle, more high-resolution remote sensing image data can be generated. Considering the generalization ability of the model, the corresponding real scene images are selected based on the principle that the gap between classes should be small and the gap within classes should be large. Then, the trained dehaze generative adversarial network model is used to process the real remote sensing images containing haze, and the high-resolution remote sensing image data set is continuously improved and constructed, so that the remote sensing data set can eventually have more than 8,000 pairs of high-resolution remote sensing images with haze and clear contrast, and the ratio of real image data to effective simulated image data in the data set is at least 6:4.

Images