A sample data enhancement method based on generative adversarial network

Abstract

This invention discloses a sample data enhancement method based on a generative adversarial network. The method comprises: creating a real target image dataset R-Dataset1 without background information, generating a generated sample dataset G-Dataset1 without background information; performing similarity matching using a cosine-aware hashing similarity matching method, directly overwriting the real sample with the generated target image in G-Dataset1 with the generated target image of the same size, thereby creating a generated sample dataset G-Dataset2 with background information and a real sample dataset R-Dataset2 with background information; and training a cyclic generative adversarial network (CycleGAN) and performing style conversion on the data in G-Dataset2. The method can generate images with rich backgrounds and clear objects, and solves the problem of unlabeled data generation in the GAN algorithm.

Description

Technical Field

[0001] The present invention relates to the field of computer image processing technology, and more particularly to using a generative adversarial network to learn features of an existing dataset and generate high-quality images to achieve effective image data enhancement in the case of small sample target detection. Background Art

[0002] Data augmentation is currently an important method for expanding dataset size. It enriches datasets and improves the performance of corresponding classification and detection models. Data augmentation can be categorized into offline and online augmentation based on the time required to process the dataset. Offline augmentation involves processing the dataset before the network model loads the training data. This is suitable for datasets with a small number of images. Online augmentation involves real-time augmentation of the dataset using the network's own augmentation algorithms after the training data has been loaded. This method is suitable for larger datasets. Currently, mainstream deep learning frameworks all have their own online augmentation strategies.

[0003] If divided by the method of data augmentation, it can be divided into supervised data augmentation and unsupervised data augmentation. Supervised data augmentation can also be further divided into single-sample augmentation and multi-sample augmentation. Supervised data augmentation is common in traditional data augmentation. For a single sample, the main enhancement methods include geometric transformation (rotation, flipping, cropping, scaling deformation and affine, etc.) and color transformation (noise, color transformation, blurring, random erasing and superpixel method, etc.). These methods are all operations around the sample itself. Excessive use of such methods will lead to a lack of diversity in the dataset, resulting in a large number of useless data samples, which will make it easier for the network model to overfit. Summary of the Invention

[0004] In view of this, the present invention proposes a sample data enhancement method based on a generative adversarial network, which uses a generative adversarial network to implement an image enhancement method for small sample target recognition.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A sample data enhancement method based on a generative adversarial network comprises the following steps:

[0007] (1) Based on the annotation information of the DOTA dataset, the targets contained in the dataset images are cropped to produce a real target image dataset R-Dataset1 without background information. The targets are divided into three categories: aircraft, ships, and small vehicles;

[0008] (2) Use the least squares loss function to replace the original loss function in the deep convolutional generative adversarial network DCGAN to obtain the least squares deep convolutional neural network L-DCGAN, train the L-DCGAN with R-Dataset1, and use the trained L-DCGAN network to generate a generated sample dataset G-Dataset1 without background information;

[0009] (3) The cosine-aware hashing similarity matching method is used for similarity matching. For each image in G-Dataset1, the image with the highest matching degree is searched in R-Dataset1. At this time, some images in R-Dataset1 may not be the images with the highest matching degree for any image in G-Dataset1, and some images may be the images with the highest matching degree for multiple images in G-Dataset1.

[0010] (4) Based on the matching results obtained in step (3), on the original large image of the DOTA dataset, the generated target image in G-Dataset1 is used to directly overwrite the real sample with the highest similarity to it, and the overlay position information is recorded; at this time, the same target on some original large images may be overwritten multiple times. In this case, the kth original large image will be copied n times. k copies, n k Indicates the maximum number of times all targets need to be covered in the kth original image, and the number of times a single target is covered is less than n k Randomly select the extra number of original large images without covering them, n k In the original large image, there is no corresponding relationship between different targets, and they are all covered randomly independently;

[0011] (5) According to the coverage position information of step (4), the coverage range of the picture without background information is randomly expanded by 3 to 5 times, and the expanded coverage range is used as the cropping range to crop the original large picture, record the target label information in the cropped picture, and use the cropped picture to create a generated sample dataset G-Dataset2 containing background information; then, on the original DOTA dataset, crop the uncovered target pictures of the same range, and use the cropped pictures to create a real sample dataset R-Dataset2 containing background information;

[0012] (6) The generated sample dataset G-Dataset2 containing background information and the real sample dataset R-Dataset2 containing background information are input into the cyclic generative adversarial network CycleGAN for training, and then the trained model is used to perform style conversion on the data in G-Dataset2 to obtain the final generated sample dataset containing background information.

[0013] Furthermore, in step (3), the specific method of the cosine-aware hash similarity matching method is as follows:

[0014] The similarity is calculated according to the following formula:

[0015] ξ=αξ cosin +βξ pHash +γ(1-ξ Barcol )

[0016] in:

[0017]

[0018]

[0019] Where ξ represents the similarity calculated by the cosine-aware hash similarity matching method, ξ cosin represents the cosine similarity, ξ Barcol represents the similarity calculated by the Bhattacharya coefficient, ξ pHash represents the similarity calculated by the perceptual hash algorithm, α, β, and γ are normalized weight coefficients, Represents two n-dimensional vectors, where p and p′ represent the three-channel color histogram data of the two images to be matched, respectively. i 、p' i Represents a data point in the histogram, that is, the number of pixels with a certain pixel value, and m represents the number of possible pixel values.

[0020] The beneficial effects of the present invention are:

[0021] 1. This invention can generate images with rich backgrounds and clear objects. Furthermore, this invention solves the unlabeled problem in the data generation process of the GAN algorithm through an improved cosine-aware hashing similarity matching method, making the image dataset generated by this invention more suitable for target detection training.

[0022] 2. This paper uses two generative adversarial networks: the least-squares deep convolutional generative adversarial network (L-DCGAN) for image generation and the cyclic generative adversarial network (CycleGAN) for style transfer. Compared to other methods, GANs only require random noise input to generate fake images that are as realistic as the original, effectively increasing the diversity of datasets.

[0023] 3. The present invention proposes an image similarity matching method based on perceptual hashing, which has good matching effect.

[0024] In short, although the quality of samples generated by the existing deep convolutional generative adversarial network (DCGAN) is good enough, it lacks scene information. The samples generated by the least squares deep convolutional neural network (L-DCGAN) only have a single object itself and lack background data, such as Figure 1 To meet the needs of target detection networks for sufficient scene information to improve detection performance, the present invention adopts a sample data enhancement method based on a generative adversarial network. This method can generate images with rich backgrounds and clear objects, providing an artificial dataset of sufficient magnitude for small-sample target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Effect diagram of samples generated for L-DCGAN.

[0026] Figure 2 Flowchart of the sample data enhancement method of the present invention.

[0027] Figure 3 is the similarity matching result, where (a) is the real sample and (b) is the generated sample.

[0028] Figure 4 are samples containing background information, where (a) is a real sample and (b) is a generated sample.

[0029] Figure 5 This is the CycleGAN style conversion effect, where (a) is the input image and (b) is the output image.

[0030] Figure 6 This is the improved CycleGAN style conversion effect diagram. DETAILED DESCRIPTION

[0031] A sample data augmentation method based on a generative adversarial network (GAN). The GAN in this method consists of a generator and a discriminator. During GAN training, the generator and the discriminator compete with each other. The generator constructs new fake images using random noise input. The discriminator then uses the image's authenticity to determine whether it is real or fake, and feeds the result back to the generator. Based on the discriminator's feedback, the generator continuously adjusts its parameters to achieve a Nash equilibrium, allowing the generator to produce images that even the discriminator cannot distinguish.

[0032] like Figure 2 As shown, the specific implementation steps of the present invention are as follows:

[0033] 1) Based on the annotation information of the DOTA dataset, a real sample dataset R-Dataset1 without background information is created. This sample set has three categories: airplanes, ships, and small vehicles, with 7152, 9177, and 10293 images in each category respectively.

[0034] 2) Use the dataset prepared in step 1 to train the least squares deep convolutional neural network (L-DCGAN). After 100 training cycles, use the trained generative network to generate a generative sample dataset G-Dataset1 without background information, as shown in the following example: Figure 1 As shown, the dataset contains about 1200 images per category.

[0035] 3) Perform similarity matching and use the cosine-aware hashing similarity matching algorithm. For each image in G-Dataset1, find the image with the highest matching degree in R-Dataset1. Figure 3 The pairing results are shown. At this point, some images in R-Dataset1 may not be the highest matching images for any image in G-Dataset1, and some images may be the highest matching images for multiple images in G-Dataset1.

[0036] 4) Based on the matching information obtained in step 3, the generated image in G-Dataset1 is used to directly overwrite the real sample with the same size as the original image in the DOTA dataset, and the overlay position information is recorded. At this time, the same target in the original image may need to be overwritten multiple times. In this case, the original image will be copied multiple times, and each copy of the original image will be overwritten by a different image in G-Dataset1.

[0037] 5) Create a sample set containing background information. According to the coverage position information in step 4, randomly expand the coverage range of the picture without background information by 3 to 5 times, and perform random position cropping. Record the sample label information in the cropped picture to create a generated sample dataset G-Dataset2 containing background information. Subsequently, randomly select real samples from the original large image of the DOTA dataset, and create a real sample dataset R-Dataset2 containing background information in the same way. It is worth noting that the pictures in R-Dataset2 will not contain generated samples, but the pictures in G-Dataset2 may contain real samples. The dataset sample is as follows: Figure 4 shown.

[0038] 6) Perform style conversion on G-Dataset2. Since the generated images in G-Dataset1 are directly copied to the original large images of the DOTA dataset, the texture and semantic information of the generated images do not correspond to the background information of the real images, such as Figure 4In order to eliminate the traces of artificial coverage and to make the generated image without background information perfectly integrated into the real image, the generated image with background information and the real image are input into the CycleGAN network for style conversion and training to obtain the final generated image with background information, as shown in Figure 5 shown.

[0039] 7) Create three datasets for target detection: R-Dataset2, G-Dataset2, and R-Dataset2+G-Dataset2.

[0040] 8) Use the three datasets produced in step 7 to train the three target detection networks YOLOv5, SSD, and Faster R-CNN, and then use the test set to verify the effect of data augmentation.

[0041] The existing perceptual hashing algorithm generates a "fingerprint" string for each photo and then compares the fingerprints of different images to determine their similarity. It does this by first reducing the image size and simplifying the colors, then performing a discrete cosine transform (DCT) on the image. The DCT works as follows:

[0042]

[0043]

[0044] Among them, f(i, j) is the original image, F(u, v) is the result after DCT transformation, N is the pixel point of the input image, c(u) and c(v) can be considered as a compensation coefficient, which can make the DCT transformation matrix an orthogonal matrix.

[0045] The DCT is then scaled down to retain only the 8x8 matrix in the upper left corner and the average is calculated. Next, the DCT is further scaled down to calculate a hash value to obtain the image's "fingerprint." Finally, the fingerprints are compared.

[0046] In order to address the shortcomings of the existing perceptual hashing algorithm, the present invention introduces a cosine similarity algorithm based on the existing perceptual hashing algorithm. Cosine similarity measures the similarity between two vectors by measuring the cosine value of the angle between them. The closer the cosine value is to 1, the closer the angle is to 0 degrees, that is, the more similar the two vectors are. The present invention represents the image as a vector and characterizes the similarity between the two images by calculating the cosine distance between the vectors. The specific method is as follows:

[0047] Suppose there are two image vectors A and B, and their cosine similarity is defined as follows:

[0048]

[0049] The range of the calculated similarity value is [-1, 1]. cosin =-1 means the two vectors point in opposite directions, ξ cosin =1 means that the two vectors point in the same direction, ξ cosin =0 indicates that the two vectors are independent of each other, while other values in the interval [-1, 1] indicate that there is a certain similarity or difference between them.

[0050] Cosine similarity is applicable to high-dimensional spatial vectors, and digital images contain many features, which are high-dimensional features. At the same time, because the perceptual hashing algorithm calculates similarity through grayscale images, it will lose color details. Therefore, the improved similarity matching algorithm will add color histogram calculations and calculate the similarity value of two images by calculating the Bhattacharyya coefficient (also known as the Bhattacharya coefficient). Its definition is as follows:

[0051]

[0052] Where p and p' represent the three-channel color histogram data of the original image and the candidate image respectively. The result of adding the square root of the product of each identical data point is the image similarity value, which ranges from [0, 1]. The closer the value is to 0, the higher the image similarity.

[0053] In summary, the present invention proposes a cosine-aware hash similarity matching algorithm based on perceptual hashing, cosine distance, and Bhattacharya coefficient, and its calculation formula is:

[0054] ξ=αξ cosin +βξ pHash +γ(1-ξ Barcol )

[0055] The similarity calculated by the cosine perceptual hashing similarity matching algorithm ranges from [0 to 1]. After extensive experimental comparisons, the best matching results were achieved when α = 0.6, β = 0.3, and γ = 0.1. This means that the cosine similarity accounts for 60% of the total similarity, the similarity calculated by the perceptual hashing algorithm accounts for 30%, and the similarity calculated by the three-channel color histogram accounts for 10%. Furthermore, the present invention sets a threshold of 0.8 for the two images to be considered similar.

[0056] To validate the effectiveness of the sample augmentation algorithm, three object detection networks, YOLOv5x, SSD300, and Faster R-CNN, were trained on the original dataset (R-Dataset), the control dataset (G-Dataset), and the sample augmentation dataset (RG-Dataset). The mAP@0.5 and mAP@0.5:0.95 detection results of the three trained object detection networks on the same test set were statistically analyzed. mAP@0.5 represents the average precision of each class label when the Intersection over Union (IoU) threshold is set to 0.5, and then the average is calculated over all class labels to obtain mAP@0.5; mAP@0.5:0.95 represents the average mAP at different IoU thresholds (from 0.5 to 0.95, with a step size of 0.05).

[0057] like Figure 6 As shown in Figure 1, the generated sample enhanced dataset is compared with the original dataset. The specific data is shown in Table 1:

[0058] Table 1 Comparison of sample enhancement effects of three target detection networks on the same test set

[0059]

[0060] As can be seen, the mAP at mAP@0.5 is improved by up to 5.2%, with an average improvement of 3.75%. At mAP@0.5:0.95, the mAP is improved by up to 5.5%, with an average improvement of 3.83%. It can be seen that the images generated by the sample enhancement method of the present invention are very excellent and can be indistinguishable from real images.

[0061] In summary, the method of the present invention can generate images with rich backgrounds and clear objects, and also solves the problem of unlabeled data generation in the GAN algorithm.

Claims

1. A sample data enhancement method based on generative adversarial network, characterized in that: The following steps are involved: (1) Based on the annotation information of the DOTA dataset, the targets contained in the dataset images are cropped to produce a real target image dataset R-Dataset1 without background information. The targets are divided into three categories: aircraft, ships, and small vehicles; (2) Use the least squares loss function to replace the original loss function in the deep convolutional generative adversarial network DCGAN to obtain the least squares deep convolutional generative adversarial network L-DCGAN, train L-DCGAN with R-Dataset1, and use the trained L-DCGAN network to generate a generated sample dataset G-Dataset1 without background information; (3) The cosine-aware hash similarity matching method is used for similarity matching. For each image in G-Dataset1, the image with the highest matching degree is searched in R-Dataset1. At this time, some images in R-Dataset1 may not be used as the images with the highest matching degree for any image in G-Dataset1, and some images may be used as the images with the highest matching degree for multiple images in G-Dataset1. The specific method of the cosine-aware hash similarity matching method is as follows: The similarity is calculated according to the following formula: ； in: ； ； Where ξ represents the similarity calculated by the cosine-aware hash similarity matching method, ξ cosin represents the cosine similarity, ξ Barcol represents the similarity calculated by the Bhattacharya coefficient, ξ pHash represents the similarity calculated by the perceptual hashing algorithm, α 、 β 、 γ is the normalized weight coefficient, Indicates two dimensional vector, where , Represent the three-channel color histogram data of the two images to be matched, p i 、 p ' i Represents a data point in the histogram, that is, the number of pixels with a certain pixel value, and m represents the number of possible pixel values; (4) Based on the matching results obtained in step (3), on the original image of the DOTA dataset, the generated target image in G-Dataset1 is used to directly overwrite the real sample with the highest similarity to it, and the overlay position information is recorded; at this time, the same target on some original images may be overwritten multiple times. In this case, the kth original image will be copied n times. k copies, n k Indicates the maximum number of times all targets need to be covered in the kth original image, and the number of times a single target is covered is less than n k Randomly select the extra number of original large images without covering them, n k In the original large image, there is no corresponding relationship between different targets, and they are all covered randomly independently; (5) According to the coverage position information of step (4), the coverage range of the picture without background information is randomly expanded by 3 to 5 times, and the expanded coverage range is used as the cropping range to crop the original large picture, record the target label information in the cropped picture, and use the cropped picture to create a generated sample dataset G-Dataset2 containing background information; then, on the original DOTA dataset, crop the uncovered target pictures of the same range, and use the cropped pictures to create a real sample dataset R-Dataset2 containing background information; (6) The generated sample dataset G-Dataset2 containing background information and the real sample dataset R-Dataset2 containing background information are input into the cyclic generative adversarial network CycleGAN for training, and then the trained model is used to perform style conversion on the data in G-Dataset2 to obtain the final generated sample dataset containing background information.

Images