A method for enhancing target samples in remote sensing images

By using generative adversarial networks and local-global discriminators in remote sensing image processing to generate high-resolution images, combined with copy-paste strategy, the problems of difficulty in collecting tag data and low image resolution in remote sensing image target detection are solved, and the problem of improved Dataset quality and model performance.

CN114758123BActive Publication Date: 2024-11-12TONGJI UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210288788.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-22
Publication Date
2024-11-12
Estimated Expiration
2042-03-22

AI Technical Summary

Technical Problem

In the detection of remote sensing image object, label data is difficult to collect and the images generated by traditional data enhancement methods have low resolution and blurred images, which cannot meet training needs.

Method used

The pix2pix model and local-global discriminator based on the generative adversarial network are used to generate high-resolution images through polygon annotations and rectangular box annotations, and the images are mixed with copy-paste strategies to generate detailed target samples.

Benefits of technology

It improves the quantity and quality of the data set, reduces sample annotation work, reduces labor and time costs, enhances the identification accuracy and stability of the target detection model, and avoids overfitting.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114758123B_ABST
    Figure CN114758123B_ABST
Patent Text Reader

Abstract

The present invention relates to a method for enhancing target samples of remote sensing images, comprising: step S1, obtaining a remote sensing image, and annotating the target in the remote sensing image with polygons and rectangular frames, and dividing the data set into a training set and a test set; step S2, inputting the segmented image and the original image after polygon annotation into a pix2pix model based on a generative adversarial network for cyclic training, generating a first remote sensing image containing the target and the probability that the image is a real image; step S3, inputting the segmented image and the original image after polygon annotation into a pix2pix model based on a local-global discriminator for training, generating a second remote sensing image; step S4, randomly selecting an image from the second remote sensing image data set and the original data set for transformation processing, using a copy-paste strategy for image mixing, and obtaining a final target enhanced image. Compared with the prior art, the present invention can generate high-resolution images with rich detail information, and improves the richness and diversity of samples.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the field of image processing, and in particular to a remote sensing image target sample enhancement method. Background Art

[0002] In recent years, with the development of deep learning, convolutional neural networks have made revolutionary breakthroughs in target detection tasks. Detection algorithms based on convolutional neural networks need to have sufficient labeled samples as training data to achieve high accuracy. However, in some scenarios, especially in remote sensing image target detection, labeled data is difficult to collect or the cost of obtaining these data is high.

[0003] In order to obtain high-quality labels, the common method at this stage is to ask professional labelers to label each image. However, the tedious labeling process not only consumes a lot of manpower, but also requires a long time and a large amount of labeling to meet the required number of data sets. Therefore, data augmentation is usually used to expand it to meet the training requirements.

[0004] The common traditional data augmentation methods have limited expansion and are insufficient to meet the training requirements. The images generated by the existing pix2pix model have the defects of low resolution and blur. Summary of the invention

[0005] The purpose of the present invention is to provide a remote sensing image target sample enhancement method in order to overcome the defects of the above-mentioned prior art, solve the problem that the original pix2pix model generates images with low resolution and blur, and the generated images improve the quantity and quality of the data set, making the detail information richer.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] The present invention provides a remote sensing image target sample enhancement method, the method comprising the following steps:

[0008] Step S1, obtaining a remote sensing image, and annotating the target in the remote sensing image with a polygon and a rectangular frame, and dividing the data set into a training set and a test set;

[0009] Step S2, inputting the polygon-annotated segmented image and the original image into a pix2pix model based on a generative adversarial network for cyclic training, generating a first remote sensing image containing the target and the probability that the image is a real image;

[0010] Step S3, inputting the polygon-annotated segmented image and the original image into a pix2pix model based on a local-global discriminator for training to generate a second remote sensing image;

[0011] Step S4: randomly select one image from the second remote sensing image dataset and the original dataset for transformation processing, and use the copy-paste strategy based on the IOU paste position determination algorithm to paste the target of one image to the other image, perform image mixing, and obtain the final target enhanced image.

[0012] Preferably, in step S1, polygonal marking and rectangular frame marking are performed on the target in the remote sensing image, specifically:

[0013] Perform polygon annotation on the target in the remote sensing image and obtain the segmented image of the target and background;

[0014] Mark the remote sensing image with a rectangular frame to obtain the rectangular frame surrounding the target.

[0015] Preferably, the improved pix2pix model based on generative adversarial network in step S2 includes a generator for image conversion and a discriminator for judging whether the input image is a real image or an image generated by the generator; the discriminator is a Markov discriminator PatchGAN.

[0016] Preferably, the generator comprises 8 convolutional layers and 7 deconvolutional layers with skip connections.

[0017] Preferably, the loss function expression of the pix2pix model based on the generative adversarial network is:

[0018]

[0019] Where, L cGAN (G, D) is the adversarial network loss function, L1(G) is the image difference loss function, and λ is the set coefficient.

[0020] Preferably, step S3 includes the following sub-steps:

[0021] Step S31, downsampling the generated first remote sensing image and the real image by k1 times and k2 times respectively;

[0022] Step S32, constructing original scale, k1-fold downsampled, and k2-fold downsampled image pyramids of the first remote sensing image and the real image respectively;

[0023] Step S33, input each layer of the image pyramid into a discriminator; wherein the discriminator structure corresponding to each layer is the same;

[0024] Step S33: The coarse-scale discriminator guides the generator to generate an image with global consistency, and the fine-scale discriminator supervises the generator to generate detail information within the local range of the image, and finally obtains a second remote sensing image containing detail information and global consistency information.

[0025] More preferably, k1 and k2 are 2 and 4 respectively.

[0026] Preferably, the loss function expression of the pix2pix model based on the local-global discriminator is:

[0027]

[0028] Where, L cGAN (G,D i ) is the adversarial network loss function; L L1 (G) is the image difference loss function; , λ are the corresponding weight coefficients respectively.

[0029] Preferably, in step S4, a copy-paste strategy based on an IOU pasting position determination algorithm is used to paste the target of one image onto another image, and image mixing is performed to obtain a final target enhanced image, specifically:

[0030] 1) Randomly select any number of object subsets in one image and paste them to random positions in another image;

[0031] 2) Adjust the real frame of the image and calculate the ratio threshold of the intersection of the pasted target and the target on the image to the target on the image. The expression is:

[0032]

[0033] In the formula, bbox1 is the pasted target, bbox2 is the target on the image, and intersection is the area intersection function;

[0034] 3) When the ratio threshold is greater than the set threshold, the occluded object is removed and the bounding box of the partially occluded object is updated to obtain the final target enhanced image.

[0035] More preferably, the transformation processing performed on each image in step S41 includes random flipping, random rotation and scale scaling.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] 1) The present invention uses the pix2pix model based on the generative adversarial network to generate images, constructs an image pyramid, and uses a local-global discriminator to supervise the generator to generate high-resolution images rich in detail information, which solves the problem that the original pix2pix model generates images with low resolution and blur. The generated images improve the quantity and quality of the data set and make the detail information richer;

[0038] 2) Use the copy-paste strategy to randomly rotate, randomly flip horizontally, and arbitrarily scale any image in the dataset, and paste the target at any position on another image in the dataset; by randomly combining the images in the dataset, the complexity of the dataset is increased, and the number of various targets is balanced, which improves the accuracy and stability of the target detection model recognition and reduces overfitting;

[0039] 3) The method of the present invention avoids a large amount of sample labeling work and reduces manpower and time costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of the method of the present invention;

[0041] Figure 2 The training process of the pix2pix model based on the generative adversarial network;

[0042] Figure 3 It is a local-global discriminator structure;

[0043] Figure 4 The image generation process is based on the copy-paste strategy;

[0044] Figure 5 Images generated by original pix2pix and pix2pix based on local-global discriminator. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0046] Working principle and workflow of the present invention:

[0047] The targets in the remote sensing images are annotated with polygons and rectangular boxes, and the dataset is divided into a training set and a test set.

[0048] The pix2pix model based on the generative adversarial network is used to generate remote sensing images containing targets to achieve data enhancement. As the images generated by the original pix2pix model have low resolution and lack local detail information, a local-global discriminator is constructed to guide the generator to generate higher quality remote sensing images.

[0049] Use the copy-paste strategy to paste different objects of different scales on the image to the new remote sensing image, outputting a new rich and novel image.

[0050] Example

[0051] like Figure 1 As shown, this embodiment takes solid waste remote sensing detection as an example and provides a remote sensing image target sample enhancement method that combines an improved pix2pix model with a copy-paste strategy, including four main processes: data set preparation, data enhancement of a pix2pix model based on a generative adversarial network, data enhancement of a pix2pix model based on a local-global discriminator, and copy-paste data enhancement.

[0052] 1. Preparation of solid waste dataset

[0053] Perform polygon annotation on the solid waste remote sensing image to obtain the segmented image of the solid waste target and background. The segmented image and the original image are used as the influence pair to input into the pix2pix model for training; after the training is completed, load the training model and input a segmented image to generate a remote sensing image containing the solid waste target. Perform rectangular box annotation on the image to obtain the rectangular box surrounding the target and save it as an XML file. The XML file and the image are used as the input of the copy-paste strategy to realize the combination of the two images and obtain a new solid waste image.

[0054] 2. Solid waste image generation based on the pix2pix model of generative adversarial network

[0055] The pix2pix model consists of two parts: the generator and the discriminator. The generator transforms the input segmented image to generate the corresponding solid waste image. The discriminator uses PatchGAN to determine whether the input image is a real image or an image generated by the generator. After the generator and the discriminator are trained in a loop, a solid waste image and the probability that the image is a real image are generated. Figure 2 The training process of the pix2pix model for generating adversarial networks.

[0056] The generator is essentially a U-shaped network, similar to a decoder-encoder. Its encoder contains 8 convolutional layers, and the decoder contains 7 deconvolutional layers, a total of 15 convolutional layers. On this basis, a skip connection is introduced, so that the input of each deconvolutional layer is the output of the previous convolutional layer, so that the features are continuously passed down, retaining more image feature information;

[0057] The discriminator uses the Markov discriminator PatchGAN to determine whether it is a generated image. That is, an image is cut into patches of different N×N sizes, and the discriminator makes a true or false judgment on each patch, and the results of all patches of an image are averaged as the final discriminator output.

[0058] The input of the conditional generation adversarial loss function in the Pix2pix network is x, the output is y, and the random noise z is the dropout layer in the network. The loss function expression of the pix2pix model based on the generation adversarial network is:

[0059]

[0060] Where, L CGAN (G,D) is the adversarial network loss function, L1(G) is the image difference loss function, and λ is the set coefficient.

[0061] 3. Solid waste image generation based on the pix2pix model of local-global discriminator

[0062] In view of the low resolution and lack of local detail information of the images generated by the original pix2pix model, the present invention adopts an improved pix2pix model and generates high-quality images through a local-global discriminator to supervise the generator, specifically:

[0063] First, the generated image and the real image are downsampled by 2 and 4 times respectively; then, the generated image and the real image of the original scale, 2 times downsampled, and 4 times downsampled are used to construct an image pyramid respectively; finally, each layer of the pyramid is input into a discriminator, and the discriminator structures of these three scales are the same. The coarse-scale discriminator guides the generator to generate images with global consistency, and the fine-scale discriminator supervises the generator to generate detailed information within the local range of the image. Finally, an image with rich detailed information and strong global consistency is generated. Figure 3 shown.

[0064] 4. Solid waste image generation based on copy-paste strategy

[0065] like Figure 4 As shown in the figure, two images are randomly selected from the solid waste image dataset generated by the pix2pix model based on the local-global discriminator in the solid waste dataset, and the two images are first transformed as follows: a scale is randomly selected between 0.5 and 2.0 for scaling; horizontal flipping, vertical flipping, and diagonal flipping are randomly selected, and flipping is performed with a probability of 0.5; an angle is randomly selected between -25° and 25° for rotation; then any number of target subsets are randomly selected from one image and pasted to a random position in another image, and the ratio of the intersection of the pasted target and the target on the image to the target on the image is calculated. When the ratio is greater than the threshold, any position is selected for pasting, and the cycle is repeated 20 times. If the last position adjustment is still greater than the threshold, the occluded object is removed and the bounding box of the partially occluded object is updated.

[0066] Among them, the ratio expression of the intersection of the pasted target and the target on the image to the target on the image is:

[0067]

[0068] Where bbox1 is the pasted target, bbox2 is the target on the image, and intersection is the area intersection function and area function.

[0069] Figure 5 The images are generated by the original pix2pix and the pix2pix based on the local-global discriminator. From the results, it can be seen that the remote sensing image data enhancement method of the present invention, by combining the pix2pix model based on the local-global discriminator with the copy-paste strategy, can generate images with high resolution, rich detail information, and sufficient texture information, which greatly improves the richness and diversity of samples and avoids a large amount of sample labeling work. At the same time, the diversity of the data set avoids overfitting of the model training and improves the target detection performance based on deep learning.

[0070] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A remote sensing image target sample enhancement method, characterized in that: The method comprises the following steps: Step S1, obtaining a remote sensing image, and annotating the target in the remote sensing image with a polygon and a rectangular frame, and dividing the data set into a training set and a test set; Step S2, inputting the polygon-annotated segmented image and the original image into a pix2pix model based on a generative adversarial network for cyclic training, generating a first remote sensing image containing the target and the probability that the image is a real image; Step S3, inputting the segmented image and the original image after polygon annotation into the pix2pix model based on the local-global discriminator for training to generate a second remote sensing image, including the following sub-steps: Step S31, downsampling the generated first remote sensing image and the real image by k1 times and k2 times respectively; Step S32, constructing original scale, k1-fold downsampled, and k2-fold downsampled image pyramids of the first remote sensing image and the real image respectively; Step S33, input each layer of the image pyramid into a discriminator; wherein the discriminator structure corresponding to each layer is the same; Step S33: the coarse-scale discriminator guides the generator to generate an image with global consistency, and the fine-scale discriminator supervises the generator to generate detail information within the local range of the image, and finally obtains a second remote sensing image containing detail information and global consistency information; Step S4: randomly select one image from the second remote sensing image dataset and the original dataset for transformation processing, and use the copy-paste strategy based on the IOU paste position determination algorithm to paste the target of one image to the other image, perform image mixing, and obtain the final target enhanced image, specifically: 1) Randomly select any number of object subsets in one image and paste them to random positions in another image; 2) Adjust the real frame of the image and calculate the ratio threshold of the intersection of the pasted target and the target on the image to the target on the image. The expression is: In the formula, bbox1 is the pasted target, bbox2 is the target on the image, and intersection is the area intersection function; 3) When the ratio threshold is greater than the set threshold, the occluded object is removed and the bounding box of the partially occluded object is updated to obtain the final target enhanced image.

2. A remote sensing image target sample enhancement method according to claim 1, characterized in that: In step S1, polygonal marking and rectangular frame marking are performed on the target in the remote sensing image, specifically: Perform polygon annotation on the target in the remote sensing image and obtain the segmented image of the target and background; Mark the remote sensing image with a rectangular frame to obtain the rectangular frame surrounding the target.

3. A remote sensing image target sample enhancement method according to claim 1, characterized in that: The improved pix2pix model based on the generative adversarial network in step S2 includes a generator for image conversion and a discriminator for judging whether the input image is a real image or an image generated by the generator; the discriminator is a Markov discriminator PatchGAN.

4. A remote sensing image target sample enhancement method according to claim 3, characterized in that: The generator consists of 8 convolutional layers and 7 deconvolutional layers with skip connections.

5. The remote sensing image target sample enhancement method according to claim 1, characterized in that: The loss function expression of the pix2pix model based on the generative adversarial network is: Where, L cGAN (G,D) is the adversarial network loss function, L1(G) is the image difference loss function, and λ is the set coefficient.

6. A remote sensing image target sample enhancement method according to claim 1, characterized in that: The k1 and k2 are 2 and 4 respectively.

7. The remote sensing image target sample enhancement method according to claim 1, characterized in that: The loss function expression of the pix2pix model based on the local-global discriminator is: Where, L cGAN (G,D i ) is the adversarial network loss function; L L1 (G) is the image difference loss function; λ are the corresponding weight coefficients.

8. The remote sensing image target sample enhancement method according to claim 1, characterized in that: Transformations performed on each image include random flipping, random rotation, and scaling.