An image data augmentation method based on transfer learning and a generative adversarial network

Abstract

The application discloses an image data amplification method based on a transfer learning and a generative adversarial network, and comprises the following steps: collecting sample pair data sets of input hidden vectors and output images by using a first image generator of a GAN, and the sample pair data sets are referred to as a first data set; fusing each pair of data in the first data set to obtain a second data set; constructing a first encoder, wherein the input and output formats of the first encoder are the same as those of the images and the hidden vectors of the first data set; constructing a first vector discriminator, wherein the input format of the first vector discriminator is the same as that of the data of the second data set; training the first encoder and the first vector discriminator by using the first data set and the second data set; collecting a third data set; pre-processing the third data set to obtain a fourth data set; inputting the images of the fourth data set into the trained first encoder in sequence to obtain the same number of hidden vector data sets, which are referred to as a fifth data set; training a GAN network model; and performing data amplification by using the trained GAN network model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image generation adversarial network, and particularly relates to an image data augmentation method based on transfer learning and a generation adversarial network. BACKGROUND

[0002] In recent years, deep learning has developed rapidly, bringing great performance improvement to the fields of image classification, target detection, image segmentation, etc. However, the use of deep learning to complete these tasks often needs to rely on large-scale training data sets, such as the commonly used ImageNet and COCO data sets, which have about 14 million and 330,000 images, respectively. In many real-world scenarios, due to factors such as time, cost, and difficulty of acquisition, it is difficult to collect such large-scale data. Therefore, many data augmentation methods (i.e., increasing the number of images available for training) have been proposed, such as simple pixel-level operations such as translation, flipping, cropping, noise addition, and color jittering, as well as methods using generative adversarial networks (GANs) to generate new samples. Among them, the pixel-level data augmentation methods bring relatively limited improvement because they do not effectively bring more and more comprehensive information. In the case of limited data quantity and samples, existing methods are difficult to produce high-quality images. How to effectively augment small data sets is a widely concerned problem in the industry.

[0003] In existing methods, there is an idea that uses a network (which the authors call Miner) to transform the input latent vector, which is equivalent to transforming the original input space into another space, so that the input latent vector is more conducive to the pre-trained generator to generate images similar to the target data set. Miner and the pre-trained GAN are further jointly trained on the target data set to improve the generation ability of the target image. Unfortunately, this approach is relatively simple and difficult to achieve satisfactory performance. The present application is based on the same starting point and discloses a more effective way to generate input latent vectors and use them to train GANs on target data sets, ultimately effectively augmenting the target data set. SUMMARY

[0004] The purpose of the present application is to use pre-trained GANs (Generative Adversarial Networks) and existing small amounts of target data samples, to more effectively transform the input space using the disclosed method, to provide better input latent vectors to the generator, to improve the generation ability of the target data set images, and to achieve the goal of effectively augmenting the target data set.

[0005] The application is implemented by at least one of the following technical solutions.

[0006] An image data augmentation method based on transfer learning and a generative adversarial network, comprising the following steps:

[0007] A first GAN is constructed, and the generator and the discriminator thereof are respectively referred to as a first image generator and a first image discriminator;

[0008] A sample pair data set of an input hidden vector and an output image is collected by using the first image generator, and the data set is referred to as a first data set;

[0009] Each pair of data in the first data set is fused, and the obtained data set is referred to as a second data set;

[0010] A first encoder is constructed, and the input and output formats thereof are the same as the image and the hidden vector of the first data set;

[0011] A first vector discriminator is constructed, and the input format thereof is the same as the data of the second data set;

[0012] The first encoder and the first vector discriminator are trained by using the first data set and the second data set;

[0013] A target data set is collected as a third data set; the third data set is preprocessed to obtain a fourth data set; and the images of the fourth data set are sequentially input into the trained first encoder to obtain a same number of hidden vector data sets, which are referred to as a fifth data set;

[0014] The GAN network model is trained by using the fourth data set and the fifth data set; and data augmentation is performed by using the trained GAN network model, so that the effect of image data augmentation is effectively achieved.

[0015] Further, the fusion comprises the following steps:

[0016] A network pre-trained on a data set and having the capability of extracting features from image data is obtained, and a plurality of layers at the end of the network are removed, and only the front-end feature extraction part is retained, and the retained part is referred to as a first feature extractor;

[0017] For a sample pair in the first data set, the image in the sample pair is input into the first feature extractor to obtain a feature map corresponding to the image; the elements of the feature map are rearranged into a one-dimensional vector, and the one-dimensional vector is spliced with the hidden vector in the sample pair, so that the fusion of the sample pair is completed.

[0018] Further, the step of pre-processing the third data set comprises:

[0019] scaling, cropping the data to match the size of the output image of the first image generator;

[0020] normalizing the data to have a value range of -1 to 1, inclusive;

[0021] creating a copy of each data and horizontally flipping the copy, and adding the flipped copy to the dataset.

[0022] Further, the first encoder is trained in a supervised manner combined with an adversarial manner, and the first vector discriminator is trained in an adversarial manner;

[0023] The first encoder includes a loss function of supervised training and a loss function of adversarial training, and the loss function of adversarial training includes the vector output by the first encoder and the image input to the first encoder, the fused image-implicit vector fusion data is attributed to true fusion data by the first vector discriminator;

[0024] The first vector discriminator includes a loss function of adversarial training, and the data in the second dataset is attributed to true fusion data, and the fusion data obtained by the first encoder is attributed to false fusion data; attribution means that the first vector discriminator receives a sample pair and outputs a value close to the expected output value;

[0025] During the training process, the first encoder and the first vector discriminator are alternately optimized according to the above loss functions.

[0026] Further, the loss function of supervised training for training the first encoder is an L2 loss function, and the specific form is:

[0027]

[0028] Wherein, m is the number of sample batches in the training process, and i is the serial number of the sample in the batch; z_01_pred is the predicted implicit vector obtained after the image in the first data set is input into the first encoder; z_01 is the implicit vector corresponding to the corresponding image in the first data set.

[0029] Further, the loss function for adversarial training of the first encoder and the first vector discriminator has the specific form:

[0030]

[0031]

[0032] Wherein, L enc is the adversarial loss function acting on the first encoder, L disc_vecis the adversarial loss function for the first vector discriminator; m is the number of samples in an iteration batch during the training process, i, j, k are the serial numbers of samples in the batch; img_01 is an image in the first data set; z_01 is a latent vector in the first data set; E() represents the forward propagation process of the first vector discriminator; F() represents the fusion mode; D vec E() represents the forward propagation process of the first vector discriminator.

[0033] Further, the GAN network model comprises a modifier and a second GAN, and the step of training the GAN network model on the fourth data set comprises:

[0034] The modifier is constructed, and the format of the input and the output of the modifier is the format of a latent vector; a second GAN with the same structure as the first GAN is established, and the second GAN is initialized with the values of the trainable parameters of the first GAN; the generator part of the second GAN is referred to as a second image generator, and the discriminator part is referred to as a second image discriminator.

[0035] Further, the forward propagation process of the modifier and the second image generator comprises the following steps:

[0036] Firstly, the input of the modifier is obtained: a latent vector is sampled from the fifth data set, referred to as z_enc; a latent vector is randomly sampled and is identically distributed with the input during the pre-training of the first image generator, referred to as z_noise01; a weighted sum of z_enc and z_noise01 is obtained, and the input latent vector z_inp01 of the modifier is obtained, that is, z_inp01 = a * z_enc + b * z_noise01, wherein a and b are arbitrary real numbers;

[0037] Secondly, the input of the second image generator is obtained: z_inp01 is input into the modifier to obtain a modified latent vector z_ref, and a latent vector is again randomly sampled and is identically distributed with the input during the pre-training of the first image generator, referred to as z_noise02; a weighted sum of z_ref and z_noise02 is obtained, and the input latent vector z_inp02 of the second image generator is obtained, that is, z_inp02 = c * z_ref + d * z_noise02, wherein c and d are arbitrary real numbers;

[0038] Finally, z_inp02 is input into the second image generator, and an image generated is output, and the forward propagation of the modifier and the second image generator is completed.

[0039] Further, the loss function for jointly training the modifier, the second image generator and the second image discriminator has the following specific form:

[0040]

[0041]

[0042] wherein, L gen is the adversarial loss function acting on the modifier and the second image generator, L disc_img is the adversarial loss function acting on the second image discriminator; m is the number of samples in the iteration batch during the training process, i, j, k are the serial numbers of the samples in the batch; img_04 is a certain image in the fourth data set; z_inp02 is the input latent vector of the second image generator; G() represents the forward propagation process of the second image generator; D img () represents the forward propagation process of the second image discriminator.

[0043] Further, the step of using the trained modifier and the second image generator to perform data augmentation on the target data set comprises:

[0044] 1) using the trained modifier and the second image generator to generate images, specifically: sampling a latent vector from the fifth data set, called z_enc; randomly sampling a latent vector that is identically distributed with the input during the pre-training of the first image generator, called z_noise01; calculating the weighted sum of z_enc and z_noise01 to obtain the input latent vector z_inp01 of the modifier, i.e. z_inp01=a*z_enc+b*z_noise01, wherein a and b take consistent values with the training; inputting z_inp01 into the modifier to obtain a modified latent vector z_ref, and again randomly sampling a latent vector that is identically distributed with the input during the pre-training of the first image generator, called z_noise02, calculating the weighted sum of z_ref and z_noise02 to obtain the input latent vector z_inp02 of the second image generator, i.e. z_inp02=c*z_ref+d*z_noise02, wherein c and d take consistent values with the training; inputting z_inp02 into the second image generator and obtaining the generated image;

[0045] 2) restoring the value range of the generated image from -1 to 1 to the image pixel range of 0 to 255; scaling and cutting it according to the use requirements;

[0046] 3) selecting the image that is close to the visual perception of the fourth data set image;

[0047] According to the actual augmentation quantity requirement, steps 1) to 3) are repeated until the requirements of image data augmentation are met.

[0048] Compared with the existing technology, the beneficial effects of the present application are:

[0049] The present application focuses on how to effectively use GAN for data augmentation in the case of only a small amount of training samples, that is, to generate higher quality images of the same category as the training samples. In order to avoid the problem of overfitting and mode collapse caused by a small amount of training data, the present application is based on the framework of transfer learning, and with the help of a GAN that has been pre-trained on a large-scale dataset, to assist in GAN training on the target dataset. Under this framework, the present application focuses on better adapting the input latent space to the target dataset, thereby helping the second generator to more easily learn to generate high-quality target data images during the joint adversarial training process, and thus achieving the goal of effectively augmenting the target dataset. In order to achieve this goal, the present application effectively encodes the target dataset through the first encoder, and these encodings correspond to the relevant regions in the latent space associated with the target dataset; the forward propagation process of generating images focuses on these relevant regions, and through the modifier, the latent vectors are transformed to further produce input latent vectors that are adapted to the second image generator and the target dataset, helping to improve the quality of generated images; in addition, by introducing random sampling noise and weighting it with the relevant latent vectors, a certain amount of random disturbance is introduced to help generate more diverse images, further optimizing the effect of data augmentation. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 A step flowchart of an image data augmentation method based on transfer learning and a generative adversarial network in an embodiment of the present application;

[0051] Figure 2 A schematic diagram of a first data set collection method in an embodiment of the present application;

[0052] Figure 3 A schematic diagram of a joint training method of a first encoder and a first vector discriminator in an embodiment of the present application;

[0053] Figure 4 A schematic diagram of a fifth data set collection method in an embodiment of the present application;

[0054] Figure 5 A schematic diagram of a joint training method of a modifier, a second image generator, and a second image discriminator in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to enable personnel in the technical field to better understand the present application scheme, the present application will be further described in detail below in conjunction with the drawings and specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0056] The method of the present application is committed to better adapting the input latent space to the target data set, so as to help the second image generator learn to generate high-quality target data images more easily in the process of joint adversarial training, and thus achieve the goal of effectively expanding the target data set. In order to achieve this goal, the method includes: constructing a first GAN, the generator and the discriminator of which are called the first image generator and the first image discriminator respectively; collecting a sample pair data set of input latent vectors and output images using the first image generator, called the first data set; fusing each pair of data in the first data set to obtain a data set, called the second data set; constructing a first encoder, the input and output formats of which are the same as the image and the latent vector of the first data set respectively; constructing a first vector discriminator, the input format of which is the same as the data of the second data set; training the first encoder and the first vector discriminator using the first data set and the second data set; collecting the target data set as a third data set; preprocessing the third data set to obtain a fourth data set; inputting the images of the fourth data set into the trained first encoder in turn to obtain a latent vector data set of the same number, called the fifth data set; training the GAN network model using the fourth data set and the fifth data set; and using the trained GAN network model for data augmentation, thereby effectively achieving the effect of image data augmentation.

[0057] The first encoder described in the specification can effectively encode the target data set, and these encodings can correspond to the relevant regions in the latent space associated with the target data set; the forward propagation process of generating images focuses on these relevant regions, and the latent vectors are transformed by the modifier, further producing input latent vectors that are adapted to the second image generator and the target data set, helping to improve the quality of generated images; in addition, by introducing random sampling noise and weighting with relevant latent vectors, a certain random disturbance is introduced to help generate more images and further optimize the effect of data augmentation.

[0058] As shown in Figure 1 The embodiment provides an image data augmentation method based on transfer learning and a generative adversarial network, which comprises the following steps:

[0059] S1, a first GAN is obtained, the generator and the discriminator of which are called the first image generator and the first image discriminator respectively. It should be pre-trained on a suitable large-scale data set that has a certain relevance to the target data sample, and the first GAN has the ability to generate images that are visually similar to the selected data set images after pre-training is completed;

[0060] As a preferred embodiment, the first GAN mentioned above includes, but is not limited to, WGAN, WGAN-GP, LSGAN, BiGAN, BigGAN, PGGAN, StyleGAN, etc.

[0061] As a preferred embodiment, the above datasets include, but are not limited to, ImageNet, COCO, LSUN, FFHQ, etc.

[0062] S2, such as Figure 2 As shown, a large-scale dataset of input latent vectors and output images is collected using the first image generator, referred to as the first dataset. Specifically, a large number of latent vectors adapted to the first image generator are first obtained (one possible scenario is that the first image generator receives Gaussian noise vectors during pre-training, in which case Gaussian noise vectors with the same distribution as those received during pre-training are randomly collected during the first dataset collection process), and these latent vectors are input into the first image generator to obtain the corresponding generated images. The input latent vectors and output images are recorded in pairs to form the first dataset.

[0063] As a preferred embodiment, the above-mentioned paired records are implemented in a Python environment using a serial number correspondence method: First, each pair of latent vectors and images is uniquely numbered; second, a dictionary with the number as the key and the latent vector as the value is created, and the `savez` function of the NumPy package is used to store the dictionary as an npz file; then, using the lmdb package, the `open` function is used to create a key-value database, the `begin` function is used to start a transaction, and for each number, the `put` function is used to temporarily store the key-value pair with the number as the key and the image as the value, and the `commit` function is used to write the temporarily stored key-value pair to the database file. The above operations store the number-latent vector and number-image pairs in the npz file and the lmdb database file respectively. In subsequent use, the corresponding latent vector and image are retrieved from the npz file and the lmdb database file respectively using the same number as the key, thus obtaining the paired latent vector and image.

[0064] S3. Preprocess the first dataset by fusing each pair of data in the first dataset to obtain the second dataset. The fusion process involves: obtaining a network pre-trained on a large-scale dataset that is capable of extracting features from image data; removing several layers at the end of the network and retaining only the feature extraction part at the front end, which is called the first feature extractor; for each sample pair in the first dataset, inputting the image from the sample pair into the first feature extractor to obtain the feature map corresponding to the image; rearranging the elements of this feature map into a one-dimensional vector and concatenating it with the latent vector in the sample pair to complete the fusion of the sample pair.

[0065] As a preferred embodiment, the aforementioned networks capable of extracting features from image data include, but are not limited to, the Inception series of networks, ResNet networks and their various variants, GoogLeNet networks, etc.

[0066] S4. Obtain an encoder network whose input and output formats are the same as the images and latent vectors of the first dataset, and call it the first encoder;

[0067] Obtain a discriminator network whose input format is the same as that of the second dataset; this is called the first vector discriminator.

[0068] In a preferred embodiment, the first encoder and the first vector discriminator described above are implemented using a multilayer perceptron network;

[0069] S5, such as Figure 3 As shown, the first encoder and the first vector discriminator are trained using a combination of supervised and adversarial training on the first and second datasets. The goal is to enable the first encoder to encode image information into a format easily received by the first image generator. The first encoder includes two loss functions: one for supervised training, which minimizes the error between the vector output by the first encoder and the corresponding latent vector in the first dataset for any image in the first dataset; and the other for adversarial training, which ensures that the image-latent vector fusion data, processed by the first fusion method, should be classified as "true fusion data" by the first vector discriminator. "Classification" means that the output value of the first vector discriminator after receiving a sample pair should be as close as possible to the preset expected output value. The first vector discriminator includes one loss function, the adversarial training loss function, which aims to classify the data in the second dataset as "true fusion data" and the fusion data obtained by the first encoder as "false fusion data." During training, the first encoder and the first vector discriminator are alternately optimized according to the loss function described above;

[0070] The loss function used for supervised training of the first encoder is the L2 loss function, which has the following specific form:

[0071]

[0072] Wherein, m is the number of sample batches in the training process, i is the serial number of the sample in the batch; z_01_pred is the predicted hidden vector obtained after the image in the first data set is input into the first encoder; z_01 is the hidden vector corresponding to the corresponding image in the first data set;

[0073] The specific form of the loss function for the adversarial training of the first encoder and the first vector discriminator is as follows:

[0074]

[0075]

[0076] Wherein, L enc is the adversarial loss function acting on the first encoder, L disc_vec is the adversarial loss function acting on the first vector discriminator; m is the number of sample batches in the training process, i, j, k are the serial numbers of the samples in the batch; img_01 is the image in the first data set; z_01 is the hidden vector in the first data set; E() represents the forward propagation process of the first vector discriminator; F() represents the fusion mode; D vec () represents the forward propagation process of the first vector discriminator;

[0077] S6, collect the target data set, that is, the data set that is expected to be image data augmented, referred to as the third data set;

[0078] Preprocess the third data set to make it suitable for subsequent training. The preprocessed data set is referred to as the fourth data set. The preprocessing method is: scale and crop the data to make its size consistent with the size of the image output by the first image generator; normalize the data to make its value range between -1 and 1, inclusive; create a copy for each data and horizontally flip the copy, and add the flipped copy to the data set;

[0079] S7, as Figure 4 shown, input the images of the fourth data set into the trained first encoder one by one to obtain a hidden vector data set with the same number, referred to as the fifth data set. Since the first encoder has the ability to predict hidden vectors from images after training, the hidden vectors in the fifth data set encode the information of the images in the fourth data set;

[0080] S8, obtain a network whose input and output formats are both hidden vector formats, referred to as a modifier; establish a second GAN with the same structure as the first GAN and initialize the second GAN with the values of the trainable parameters of the first GAN; the generator part of the second GAN is referred to as a second image generator, and the discriminator part is referred to as a second image discriminator;

[0081] In a preferred embodiment, the above-described modifier is implemented using a multilayer perceptron network;

[0082] S9, such as Figure 5 As shown, joint adversarial training is performed on the above-mentioned modifier, second image generator, and second image discriminator;

[0083] For the modifier and the second image generator, the forward propagation process is as follows: First, obtain the input of the modifier. Sample the latent vector from the fifth dataset, called z_enc; randomly sample the latent vector that is in the same distribution as the input during the pre-training of the first image generator, called z_noise01; calculate the weighted sum of z_enc and z_noise01 to obtain the input latent vector z_inp01 of the modifier, i.e., z_inp01 = a*z_enc + b*z_noise01, where a and b are arbitrary real numbers; Second, obtain the input of the second image generator. The modifier z_inp01 is input into the modifier to obtain a modified latent vector z_ref. Then, a latent vector z_noise02, which is in the same distribution as the input of the first image generator during pre-training, is randomly sampled again. A weighted sum of z_ref and z_noise02 is calculated to obtain the input latent vector z_inp02 of the second image generator, i.e., z_inp02 = c * z_ref + d * z_noise02, where c and d are arbitrary real numbers. Finally, z_inp02 is input into the second image generator, and the generated image is output, completing the forward propagation of the modifier and the second image generator.

[0084] Logically, the modifier and the second image generator form a unified whole for generating images. This unified whole is trained adversarially against the second image discriminator on the fourth dataset. The optimization objective for the adversarial loss function of the modifier and the second image generator is to make the second image discriminator classify the images generated by the modifier and the second image generator as real images. The optimization objective for the adversarial loss function of the second image discriminator is to make the second image discriminator classify the images generated by the modifier and the second image generator as fake images, and classify the images from the fourth dataset as real images. During training, the modifier and the second image generator are updated as a whole, and they are alternately optimized with the second image discriminator according to their respective adversarial loss functions.

[0085] The loss function used for joint adversarial training of the modifier, the second image generator, and the second image discriminator is as follows:

[0086]

[0087]

[0088] Among them, L genL is an adversarial loss function acting on the modifier and the second image generator disc_img L is an adversarial loss function acting on the second image discriminator; m is the number of samples in an iteration batch during the training process, i, j, k are the serial numbers of samples in the batch; img_04 is an image in the fourth dataset; z_inp02 is the input latent vector of the second image generator; G() represents the forward propagation process of the second image generator; D img () represents the forward propagation process of the second image discriminator.

[0089] S10, using the trained modifier and the second image generator to generate images, the specific method is: sampling a latent vector from the fifth dataset, called z_enc; randomly sampling a latent vector with the same distribution as the input of the first image generator during pre-training, called z_noise01; calculating the weighted sum of z_enc and z_noise01 to obtain the input latent vector z_inp01 of the modifier, i.e. z_inp01 = a*z_enc + b*z_noise01, wherein a and b take consistent values with the training; inputting z_inp01 into the modifier to obtain a modified latent vector z_ref, and again randomly sampling a latent vector with the same distribution as the input of the first image generator during pre-training, called z_noise02, calculating the weighted sum of z_ref and z_noise02 to obtain the input latent vector z_inp02 of the second image generator, i.e. z_inp02 = c*z_ref + d*z_noise02, wherein c and d take consistent values with the training; inputting z_inp02 into the second image generator and obtaining the generated image;

[0090] Restoring the value range of the generated image from -1 to 1 to the image pixel range of 0 to 255; according to the needs of use, scaling and cutting it; selecting the image, manually selecting the image with a visual observation close to the fourth dataset image;

[0091] According to the actual number of expansion requirements, repeat the above generation, processing and selecting image process until the image data expansion requirements are met.

[0092] It can be seen from the embodiment that the application focuses on better adapting the input latent space to the target data set, thereby helping the second generator to learn to generate high-quality target data images more easily in the process of joint adversarial training, and thereby achieving the goal of effectively expanding the target data set. In order to achieve this goal, the application effectively encodes the target data set through the first encoder, and the encodings correspond to the relevant regions in the latent space associated with the target data set; the forward propagation process of the generated image focuses on these relevant regions, and the modifier is used to transform the latent vectors, further generating input latent vectors that are adapted to the second image generator and the target data set, thereby helping to improve the quality of the generated image; in addition, by introducing random sampling noise and weighting the relevant latent vectors, a certain random disturbance is introduced to help generate more diverse images and further optimize the data augmentation effect. In the test experiment in the embodiment, high-quality and diversified new samples of the same type as the target data can be effectively generated, and the target data set can be effectively expanded.

[0093] The preferred embodiments of the application disclosed above are only used to help explain the application. The preferred embodiments do not describe all the details and limit the application to the specific embodiments described. Obviously, many modifications and changes can be made according to the content of the specification. The specification selects and describes these embodiments in order to better explain the principles and practical applications of the application, so that those skilled in the art can well understand and use the application. The application is limited only by the claims and their entire scope and equivalents.

Claims

1. An image data augmentation method based on transfer learning and generative adversarial networks, characterized in that, Includes the following steps: The first GAN is constructed, and its generator and discriminator are called the first image generator and the first image discriminator, respectively. The dataset of sample pairs of input latent vectors and output images collected using the first image generator is called the first dataset; The dataset obtained by fusing each pair of data in the first dataset is called the second dataset. Construct a first encoder whose input and output formats are the same as the images and latent vectors of the first dataset, respectively. Construct a first vector discriminator whose input format is the same as that of the second dataset; The first encoder and the first vector discriminator are trained using the first dataset and the second dataset; Collect the target dataset as a third dataset; The third dataset is preprocessed to obtain the fourth dataset; The images from the fourth dataset are sequentially input into the first encoder after training to obtain the same number of latent vector datasets, which are called the fifth dataset. The GAN network model was trained using the fourth and fifth datasets; Data augmentation is achieved by using a trained GAN network model, thus effectively enhancing image data.

2. The image data augmentation method based on transfer learning and generative adversarial networks according to claim 1, characterized in that, The fusion includes the following steps: Obtain a network that has been pre-trained on the dataset and has the ability to extract features from image data, and remove several layers at the end of the network, keeping only the feature extraction part at the front end. The retained part is called the first feature extractor. For a sample pair in the first dataset, the image in the sample pair is input into the first feature extractor to obtain the feature map corresponding to the image; the elements of this feature map are rearranged into a one-dimensional vector and concatenated with the latent vector in the sample pair to complete the fusion of the sample pair.

3. The image data augmentation method based on transfer learning and generative adversarial networks according to claim 2, characterized in that, The steps for preprocessing the third dataset include: The data is scaled and cropped to match the size of the image output by the first image generator; The data is normalized so that its value ranges from -1 to 1, including both -1 and 1; Create a copy of each data point, flip the copy horizontally, and add the flipped copy to the dataset.

4. The image data augmentation method based on transfer learning and generative adversarial networks according to claim 3, characterized in that, The first encoder is trained using a combination of supervised training and adversarial training, while the first vector discriminator is trained using adversarial training. The first encoder includes a loss function for supervised training and a loss function for adversarial training; The first vector discriminator includes a training-resistant loss function, which classifies the data in the second dataset as true fused data and classifies the fused data obtained by the first encoder as false fused data; classification means that after the first vector discriminator receives a sample pair, the output value is close to the preset expected output value; During training, the first encoder and the first vector discriminator are alternately optimized according to the above loss function.

5. The image data augmentation method based on transfer learning and generative adversarial networks according to claim 4, characterized in that, The loss function used for supervised training of the first encoder is the L2 loss function, which has the following form: Where m is the number of samples in the iterative batch during training, and i is the sequence number of the sample in the batch; z_01_pred is the predicted latent vector obtained after the image in the first dataset is input into the first encoder; z_01 is the latent vector corresponding to the image in the first dataset.

6. The image data augmentation method based on transfer learning and generative adversarial networks according to claim 5, characterized in that, The loss function used for adversarial training of the first encoder and the first vector discriminator is as follows: Among them, L enc L is the adversarial loss function applied to the first encoder. disc_vec The adversarial loss function applied to the first vector discriminator; m is the number of samples in the iterative batch during training, and i, j, and k are the sample indices in the batch; img_01 is the image in the first dataset; z_01 is the latent vector in the first dataset; E() represents the forward propagation process of the first vector discriminator; F() represents the fusion method; D vec () represents the forward propagation process of the first vector discriminator.

7. The image data augmentation method based on transfer learning and generative adversarial networks according to claim 6, characterized in that, The GAN network model includes a modifier and a second GAN. The steps for training the GAN network model on the fourth dataset include: Construct a modifier whose input and output formats are both latent vectors; build a second GAN with the same structure as the first GAN and initialize the second GAN with the values ​​of the trainable parameters of the first GAN; the generator part of the second GAN is called the second image generator, and the discriminator part is called the second image discriminator.

8. The image data augmentation method based on transfer learning and generative adversarial networks according to claim 7, characterized in that, The forward propagation process of the modifier and the second image generator includes the following steps: First, obtain the input of the modifier: sample the latent vector from the fifth dataset, called z_enc; randomly sample the latent vector that is in the same distribution as the input during the pre-training of the first image generator, called z_noise01; calculate the weighted sum of z_enc and z_noise01 to obtain the input latent vector of the modifier z_inp01, that is, z_inp01=a*z_enc+b*z_noise01, where a and b are arbitrary real numbers; Next, the input of the second image generator is obtained: z_inp01 is input into the modifier to obtain a modified latent vector z_ref, and the latent vector z_noise02, which is in the same distribution as the input during the pre-training of the first image generator, is randomly sampled again. The weighted sum of z_ref and z_noise02 is calculated to obtain the input latent vector z_inp02 of the second image generator, that is, z_inp02=c*z_ref+d*z_noise02, where c and d are arbitrary real numbers; Finally, z_inp02 is input into the second image generator, and the generated image is output, completing the forward propagation of the modifier and the second image generator.

9. The image data augmentation method based on transfer learning and generative adversarial networks according to claim 8, characterized in that, The loss function used for joint adversarial training of the modifier, the second image generator, and the second image discriminator is as follows: Among them, L gen L is the adversarial loss function applied to both the modifier and the second image generator. disc_img The adversarial loss function applied to the second image discriminator; m is the number of samples in the iterative batch during training, and i, j, and k are the sample indices in the batch; img_04 is an image in the fourth dataset; z_inp02 is the input latent vector of the second image generator; G() represents the forward propagation process of the second image generator; D img () indicates the forward propagation process of the second image discriminator.

10. The image data augmentation method based on transfer learning and generative adversarial networks according to claim 9, characterized in that, The steps for augmenting the target dataset using the trained modifier and second image generator include: 1) Image generation is performed using the trained modifier and second image generator. Specifically, latent vectors are sampled from the fifth dataset, denoted as z_enc; latent vectors with the same distribution as the input during the pre-training of the first image generator are randomly sampled, denoted as z_noise01; a weighted sum of z_enc and z_noise01 is calculated to obtain the input latent vector z_inp01 of the modifier, i.e., z_inp01 = a*z_enc + b*z_noise01, where the values ​​of a and b remain consistent with those used during training; z_ The modifier z_ref is input to inp01, resulting in a modified latent vector z_ref. This modified latent vector z_ref is then randomly sampled again and compared to the latent vector z_noise02, which is identically distributed to the input of the first image generator during pre-training. A weighted sum of z_ref and z_noise02 is then calculated to obtain the input latent vector z_inp02 for the second image generator, i.e., z_inp02 = c * z_ref + d * z_noise02, where c and d maintain the same values ​​as during training. z_inp02 is then input into the second image generator to obtain the generated image. 2) Restore the generated image's value range from -1 to 1 to the image pixel range of 0 to 255; scale and crop it as needed. 3) Select images that are visually similar to those in the fourth dataset; Repeat steps 1) to 3) according to the actual amplification requirements until the image data amplification requirements are met.

Images