Training set augmentation techniques and multi-class recurrent generative adversarial methods

Abstract

This invention relates to the field of training set augmentation technology, specifically a multi-class cyclic generative adversarial method for training set augmentation, including: using a generative model to augment training samples; introducing external information can reduce overfitting in the generative model and improve the classification accuracy of the classification neural network; however, due to the scarcity of underwater data, traditional generative methods, which only augment existing data, are unlikely to achieve good results; to better augment the data, this technology proposes an M-CycleGAN network to augment the training set; constructing a 3D model image dataset, introducing external classification target information, and assisting the generative model in generating new data similar to the classification dataset.

Description

Technical Field

[0001] This invention relates to the field of training set augmentation technology, specifically a multi-class cyclical adversarial generation method for training set augmentation. Background Technology

[0002] Generative Adversarial Networks (GANs) are deep learning models consisting of a generator and a discriminator. The generator is a function that transforms random noise (often Gaussian noise) into an image, while the discriminator is another function that distinguishes between real and generated images. GANs are trained by having these two functions compete against each other: the generator continuously tries to generate better images, and the discriminator continuously improves its discrimination ability until the generated images are indistinguishable from real images.

[0003] The core idea of ​​GANs is to establish a game between the generator and the discriminator. The generator aims to generate images that can fool the discriminator, making it unable to distinguish between real and generated images, while the discriminator aims to be able to identify which images are real. In this way, during training, the two models continuously compete with each other, gradually improving the generator's ability to generate realistic images, until the images generated by the generator can be comparable to real images.

[0004] Traditional generative models extract information from the original data in different ways and process it to generate new samples similar to the original samples. However, due to the scarcity of underwater data, traditional generative methods, which only expand the original data, are unlikely to achieve good results.

[0005] In summary, this application proposes a training set augmentation technique and a multi-class cyclical adversarial generation method to improve any of the technical problems mentioned above. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention proposes a multi-class cyclic generative adversarial method using training set augmentation techniques. This adversarial method employs a CycleGAN network to transfer a constructed 3D image dataset into a sonar image style. The CycleGAN network is designed based on the number of different categories contained in the classification data, and the classification data includes M-CycleGAN... 2 The model includes classification dataset A and classification dataset B.

[0007] The M-CycleG AN 2 The model mainly consists of two CycleGAN networks and a class discriminator. The model structure of the class discriminator is the same as that of the discriminator in the CycleGAN network, and its purpose is to calculate the differences between different classes.

[0008] As a preferred embodiment of this application, each CycleGAN network consists of two generators and two discriminators.

[0009] As a preferred embodiment of this application, the generator consists of three parts: encoding, conversion, and decoding.

[0010] As a preferred embodiment of this application; the encoding utilizes a convolutional neural network to extract features from the input image; the transformation converts the feature vectors of two different regions of the image by combining dissimilar features of the image; the decoding utilizes a deconvolutional layer to restore low-level features from the feature vectors, and finally obtains the generated image.

[0011] As a preferred embodiment of this application, the conversion also includes a six-layer Reset module, each Reset module being a neural network layer consisting of two convolutional layers, which can achieve the goal of preserving the original image features during conversion.

[0012] As a preferred embodiment of this application, the discriminator takes an image as input and predicts whether the image is the original image or the output image of the generator.

[0013] The beneficial effects of this invention are as follows:

[0014] 1. The training set expansion technique described in this invention is a multi-class cyclic generative adversarial method. This technique uses a generative model to expand the training samples. Introducing external information can reduce the overfitting phenomenon of the generative model and improve the classification accuracy of the classification neural network. However, due to the scarcity of underwater data, the traditional generation method, which only expands based on the original data, is difficult to achieve good results. In order to better expand the data, this technique proposes an M-CycleGAN network to expand the training set. A 3D model image dataset is constructed, and external classification target information is introduced to assist the generative model in generating new data similar to the classification dataset. Attached Figure Description

[0015] Figure 1 M-CycleGAN in this invention 2 Model diagram;

[0016] Figure 2 This is a selection diagram of the SCTD-master dataset used in this invention;

[0017] Figure 3 This is a selection image of the 3D model image dataset in this invention;

[0018] Figure 4 M-CycleGAN in this invention 3 Model diagram;

[0019] Figure 5 M-CycleGAN in this invention 3 Images after model transfer; Detailed Implementation

[0020] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0021] Generative Adversarial Networks (GANs) are deep learning models consisting of a generator and a discriminator. The generator is a function that transforms random noise (often Gaussian noise) into an image, while the discriminator is another function that distinguishes between real and generated images. GANs are trained by having these two functions compete against each other: the generator continuously tries to generate better images, and the discriminator continuously improves its discrimination ability until the generated images are indistinguishable from real images.

[0022] The core idea of ​​GANs is to establish a game between the generator and the discriminator. The generator aims to generate images that can fool the discriminator, making it unable to distinguish between real and generated images, while the discriminator aims to be able to identify which images are real. In this way, during training, the two models continuously compete with each other, gradually improving the generator's ability to generate realistic images, until the images generated by the generator can be comparable to real images.

[0023] Traditional generative models extract information from the original data in different ways and process it to generate new samples similar to the original samples. However, due to the scarcity of underwater data, traditional generative methods, which only expand the original data, are unlikely to achieve good results.

[0024] This includes: the adversarial method uses a CycleGAN network to transfer a constructed 3D image dataset into a sonar image style. The CycleGAN network is designed based on the number of different categories contained in the classification data, and the classification data includes M-CycleGAN. 2 Model;

[0025] The M-CycleGAN 2 The model mainly consists of two CycleGAN networks and a class discriminator. The model structure of the class discriminator is the same as that of the discriminator in the CycleGAN network, and its purpose is to calculate the differences between different classes.

[0026] Given a target classification dataset A containing two classes: AX and AY, we construct a similar 3D model image dataset B, also containing two classes: BX and BY. We then build an M-CycleGAN2 model with two classes to transform dataset B, generating a dataset A′ similar in style to dataset A. First, we input images of class X (AX from dataset A) and BX from dataset B into a CycleGAN network for training, calculating the loss L for class X. X (G X ,F X D AX D BX );

[0027] L X (G X ,F X D AX D BX ) = L GAN (G X D BX ,AX,BX)+L GAN (F X D AX (BX,AX)

[0028] +λL cyc-X (G X ,F X )+L class-X (AX,A B X)

[0029] Where L GAN (G X D BX ,AX,BX) and L GAN (F X D AX (AX, BX) represent the adversarial style loss between datasets A and B, respectively; L cyc-x (G X ,F X ) represents the cycle consistency loss; L class-X (AX, A) B X) represents the class loss of class X data, i.e., the real image AX and the generated image ABX. It is calculated as follows:

[0030]

[0031] Among them and D class This is a class discriminator used to determine whether the input image belongs to class X or class Y. The class loss for class Y data is:

[0032]

[0033] Next, we solve the following equation to make the total loss L of class X equal to... X (G X ,F X D AX D BX Minimum:

[0034]

[0035] Among them G X * and F X * This is the optimal mapping function for class X; similarly, the total loss L for class Y can be obtained. Y (G Y ,F Y D AY D BY ), and minimize:

[0036]

[0037] Among them G Y * and F Y * It is the optimal mapping function for class Y.

[0038] Each CycleGAN network consists of two generators and two discriminators;

[0039] The generator consists of three parts: encoding, conversion, and decoding.

[0040] The encoding process uses a convolutional neural network to extract features from the input image; the transformation process combines dissimilar features of the image to transform the feature vectors of two different regions; the decoding process uses a deconvolutional layer to recover low-level features from the feature vectors, and finally obtains the generated image.

[0041] The conversion also includes a six-layer Reset module, each of which is a neural network layer consisting of two convolutional layers, which can achieve the goal of preserving the original image features during the conversion.

[0042] The discriminator takes an image as input and predicts whether the image is the original image or the output image of the generator.

[0043] Selection of experimental data

[0044] This experiment uses the SCTD-master dataset, which, based on extensive collection of sonar images, constructs a completely public dataset of common sonar target detection that can be used for sonar image detection and classification research, such as... Figure 1 As shown; the dataset currently contains three typical targets: a) underwater shipwrecks, b) aircraft wreckage, and c) victims, totaling 596 samples; this paper sets 30 images for each class in the training set, and the remaining images are used for testing;

[0045] Building a 3D model image dataset

[0046] This experiment constructed a 3D model image dataset for style transfer. This dataset is similar to the SCTD-master dataset, containing three object classes: d (ship), e (airplane), and f (person), with 100 images for each class. Figure 2 As shown;

[0047] Constructing M-CycleGAN 3 network

[0048] CycleGAN 3 Network structure, such as Figure 4 As shown, compared with M-CycleGAN 2 The models are basically the same, the difference being that M-CycleGAN3 contains three CycleGAN networks and three class discriminators. The structure of the class discriminators and the generators and discriminators in each CycleGAN network is consistent with the M-CycleGAN2 model.

[0049] like Figure 4 As shown, X, Y, and Z represent three target classes: d (ship), e (airplane), and f (person), respectively; A represents the SCTD-master dataset; B represents the 3D model image dataset; in the cycleGAN1 network, G1 represents the generator that transforms AX-type data into BX-type data for X-class data (ship); F1 represents the generator that transforms BX-type data into AX-type data for data (ship); the discriminator DAX is used to determine whether the data is real data AX or data generated by F1 (BX); the discriminator DBX is used to determine whether the data is real data BX or data generated by G1 (AX). Similarly, in the cycleGAN2 and cycleGAN3 networks, the model structure is similar; the class discriminators DclassA and DclassB are used to distinguish different categories of data (including real data and generated data) in the SCTD-master dataset and the 3D model image dataset.

[0050] Training M-CycleGAN 3 network

[0051] Input the SCTD-master training set (30 images per class) and the 3D model image dataset (100 images per class), and iteratively train M-CycleGAN. 32The network is trained for 1000 rounds, and the M-CycleGAN is saved after training. 3 network;

[0052] Input the 3D model image dataset into the saved M-CycleGAN 3 In the corresponding generator in the network, the style-transferred generated image is obtained, and the experimental results are as follows. Figure 5 As shown;

[0053] in Figure 5 In this context, gi represents the SCTD-master training set, jl represents the 3D model image dataset, and mo represents M-CycleGAN. 3 Network-generated results;

[0054] from Figure 5 It can be seen that the model transfers 3D model image datasets to generate images with a style similar to the SCTD-master dataset; subjectively, M-CycleGAN... 3 Networks can effectively perform style transfer, retaining appropriate external information to assist in the training of subsequent classification models;

[0055] Classification network training

[0056] In this experiment, a comparative experiment was conducted on three advanced classification networks, ResNet18, ResNet34, and DenseNet121. The effectiveness of the data generated by the M-CycleGAN3 network was demonstrated by comparing the classification performance of the network obtained by whether the input training data contained images generated by the M-CycleGAN3 network.

[0057] Note: ResNet-18 (Residual network-18) is an 18-layer residual network; ResNet-34 (Residual network-34) is a 34-layer residual network; Desenet121 (densely connected convolutional network121) is a 121-layer densely connected convolutional neural network.

[0058] First, in ResNet18, the network was trained iteratively 200 times using a pre-defined training set of SCTD-master. After training, the results were saved and tested on the test set to obtain the classification results of the baseline model ResNet18. Then, the training set was reset, including the previously defined training set of SCTD-master and data generated by the M-CycleGAN3 network, and the network was trained iteratively 200 times. After training, the results were saved and tested on the test set to obtain the optimized results with the M-CycleGAN3 network-generated data. Similarly, the same strategy was used for ResNet34 and DenseNet121.

[0059] Classification network experimental results

[0060] Compared to the baseline model, the optimal accuracy of our method has been improved by 5%, as shown in the table below.

[0061] Table 1. Experimental Results of Classification Networks

[0062] Resnet18 0.7266 <![CDATA[Resnet18+M-cycleGAN 3 ]]> 0.7566(+3％) Resnet34 0.6779 <![CDATA[Resnet34+M-cycleGAN 3 ]]> 0.7041(+2.62％) Densenet121 0.7453 <![CDATA[Densenet121+M-cycleGAN 3 ]]> 0.8015(+5.62％)

[0063] The values ​​in parentheses represent the improvement of the proposed method compared to the corresponding benchmark methods; Resnet18, Resnet34, and Densenet121 are different base classification models, and Resnet18+M-cycleGAN... 3 Resnet34+M-cycleGAN 3 Densenet121+M-cycleGAN 3 This is the method proposed in this study; as can be seen, this study uses M-cycleGAN. 3 After data augmentation, there were improvements on different base classifiers, with the largest improvement of 5.62% on the Densenet121 network, achieving the optimal classification accuracy of 0.8015.

[0064] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A training set augmentation technique multi-class cyclic generative adversarial method; characterized in that The application relates to a CycleGAN network for image translation, comprising: The adversarial method adopts a CycleGAN network to migrate the constructed 3D picture dataset into a sonar picture style, the CycleGAN network is designed according to the number of different categories contained in classification data, and the classification data includes M-CycleGAN 2 and the classification data includes classification dataset A and classification dataset B; The M-Cycle GAN 2 The model contains two CycleGAN networks and a category discriminator whose model structure is the same as the discriminator structure in the CycleGAN network, and its purpose is to calculate the difference between different categories; Each of the CycleGAN networks is composed of two generators and two discriminators; The generator is composed of three parts of encoding, conversion and decoding; The encoding extracts features from an input image by using a convolutional neural network; The conversion transforms feature vectors of two different regions of an image by combining non-close features of the image; the decoding restores low-level features from the feature vectors by using a deconvolution layer, and finally generates an image; The conversion contains six layers of Reset modules, each of which is a neural network layer composed of two convolution layers, and can achieve the purpose of preserving original image features during conversion; Target classification dataset A, which contains 2 classes: AX and AY; construct a 3D model image dataset B similar to dataset A, which also contains 2 classes: BX and BY; construct an M-CycleGAN2 model with 2 categories to convert dataset B and generate dataset A' similar in style to dataset A; input the images AX and BX, the X-class images in dataset A, and the X-class images in dataset B into a CycleGAN network for training to calculate the category X loss ; L X (G X ,F X ,D AX ,D BX ) = L GAN (G X ,D BX ,AX,BX) + L GAN (F X ,D AX ,BX,AX) + λL cyc-X (G X ,F X )+L class-X (AX,A B X); wherein and are adversarial losses of style between data sets A and B, respectively; is a cycle consistency loss; is a class loss of X-class data, i.e., real image AXand generated image ABX; it is calculated in the following manner: ； wherein is a class discriminator to determine whether the input image is of class X or Y; and the class loss for Y-class data is: Next, the total loss for X-class is minimized by solving the following equation: ;​ where G X * and F X * is the optimal mapping function for class X, and similarly for class Y and minimize: ; where G Y * and F Y * are the optimal mapping functions of class Y. 2.The training set augmentation technique multi-category cyclic generative adversarial method according to claim 1, wherein: The discriminator takes an image as input and predicts whether the image is an original image or a generated image.

Images