Image data augmentation method based on conditional generative adversarial network

Abstract

The invention discloses an image data augmentation method based on a conditional generative adversarial network, and the method comprises the steps: S1, obtaining an image data sample, carrying out the preprocessing of an original sample, and determining the dependent condition information during the augmentation of the image data; s2, inputting the conditional information and the random noise vector into a conditional generative adversarial network model under a double-projection mechanism, performing image generation on an original sample by the model based on the conditional information, and outputting a probability that an image is a real image; s3, performing gradient normalization on a loss function of the model by adopting a continuous differentiable function, dynamically adjusting the weight of the loss function to obtain a loss value and an optimal learning rate, and stopping model training; and S4, augmenting the image data sample through the trained double-projection condition generative adversarial network model. According to the invention, the diversity and quality of the image data set can be improved, and diversified image data meeting specific conditions can be effectively generated.

Description

Technical Field

[0001] The present invention relates to the technical field of image data augmentation, and in particular to an image data augmentation method based on a conditional generative adversarial network. Background Art

[0002] With the widespread application of deep learning in the field of image processing, the scale and diversity of image data play a vital role in model performance. Large quantities of high-quality training data can often significantly improve the generalization ability of the model, but in many practical scenarios, it is costly to obtain sufficient image data, especially in data-scarce fields such as medical imaging and remote sensing images. Therefore, data augmentation technology came into being. Image data augmentation is an important technology in the field of computer vision. It aims to generate more training samples by performing various forms of transformations on existing data, thereby improving the generalization ability of the model and reducing overfitting problems.

[0003] Traditional image data augmentation methods mainly include simple operations such as geometric transformation, color transformation and adding noise. Geometric transformation includes image rotation, scaling, translation, flipping, cropping, reflection, etc. These transformations do not change the semantic content of the image, but change the geometric structure of the image, thereby increasing the robustness of the model to different scales and perspectives. Color transformation includes image brightness adjustment, contrast adjustment, saturation adjustment and hue change. These operations are designed to enhance the adaptability of the model to different lighting conditions. Adding noise enables the model to adapt to noise interference and image blur by adding random noise (such as Gaussian noise, salt and pepper noise) or blurring (such as Gaussian blur) to the image.

[0004] Although the above methods can achieve the expansion of small sample data sets, they are limited by manual settings and cannot fundamentally solve the problem of insufficient data diversity. The implicit generative model Generative Adversarial Networks (GANs) can capture the high-dimensional features of existing data, synthesize new data from a given prior distribution, and generate images similar to the training data. In recent years, it has made significant progress in the field of image data augmentation. However, ordinary GANs-based methods also face some shortcomings and challenges. Since the adversarial process between the generator and the discriminator can easily lead to oscillation or non-convergence during training, model training instability often occurs; when generating high-resolution images, it is easy to have problems such as image quality degradation and increased training difficulty, making it difficult to generate high-resolution images.

[0005] These problems limit the diversity, sample quality, performance improvement of downstream tasks, and controllability and breadth of practical applications of image data augmentation. It is necessary to further improve the model structure and training methods to overcome these limitations so as to better apply them to data augmentation tasks. Summary of the invention

[0006] In order to solve the problem of unstable generation of adversarial network models and poor quality of generated images in the prior art, the present invention proposes an image data augmentation method based on conditional generative adversarial networks.

[0007] The specific technical solutions are as follows:

[0008] A method for augmenting image data based on a conditional generative adversarial network, comprising:

[0009] S1: Obtain image data samples, preprocess the original samples, and determine the conditional information that the image data augmentation depends on;

[0010] S2: Input the conditional information and random noise vector into the conditional generative adversarial network model under the double projection mechanism. The model generates images for the original samples based on the conditional information and outputs the probability that the image is a real image.

[0011] S3: Use a continuously differentiable function to normalize the gradient of the model's loss function, dynamically adjust the weight of the loss function, obtain the loss value and the optimal learning rate, and stop model training;

[0012] S4: Augment image data samples using the trained dual-projection conditional generative adversarial network model.

[0013] Furthermore, the determination of the conditional information is specifically as follows: the conditional information adopts a category label, the category label information is embedded to represent the potential features, and the category label information is embedded and converted into a high-dimensional vector as a model input.

[0014] Furthermore, the conditional generative adversarial network adopts a double projection mechanism to simultaneously process data matching and label matching. The double projection mechanism uses minimizing f-loss to align the real data distribution and the generated data distribution, and adds the distribution of the real data and the weight of label matching to the generator network.

[0015] Furthermore, the conditional generative adversarial network includes a generator G and a discriminator D;

[0016] Generator G: The generator network takes a random noise vector and category label information as input and outputs a generated image;

[0017] Discriminator D: The discriminator network takes the generated fake image, real image and corresponding conditional information as input, compares and identifies them, and outputs the probability that the image is a real image.

[0018] Furthermore, the generator G includes an attention module, which is inserted into a traditional convolutional layer and replaces one layer of the traditional convolutional layer.

[0019] Furthermore, the step S3 specifically includes:

[0020] A continuously differentiable function is used to normalize the gradient of the loss function of the discriminator D, and a gradient norm constraint is imposed to keep the gradient of the discriminator D within a controllable range during the training process;

[0021] By dynamically adjusting the weight of the loss function, the training process under the dual projection mechanism balances the weight deviation caused by processing data matching and label matching at the same time;

[0022] The model adaptively learns weights, obtains the loss value and the optimal learning rate, stops model training, and obtains the trained image data augmentation model.

[0023] Furthermore, the step S4 specifically includes:

[0024] The convolutional layer of the generator G extracts features from the image to be processed and provides conditional information;

[0025] The deconvolution layer of the generator G is used to restore the feature vector from the extracted features, generate new image data, expand the original image, and complete the image data augmentation requirements under specific conditions.

[0026] Furthermore, the image data sample acquisition is specifically: using an industrial camera to collect image data, inputting the data into image processing software, and determining the original sample of the image data that needs to be augmented.

[0027] Furthermore, the preprocessing of the original sample includes cropping, scaling, rotating and flipping.

[0028] The above technical solution has the following advantages or technical effects:

[0029] 1. The present invention can perform data augmentation for specific small or low-quality image data samples, has very strong universality, and can effectively improve the diversity and quality of image data while maintaining the stability of model training.

[0030] 2. The present invention improves the diversity and quality of image data, effectively replacing the tedious process of collecting and annotating image data, and providing data support for solving specific machine vision problems.

[0031] 3. The present invention adopts conditional generative adversarial network technology, and by designing methods such as a dual projection mechanism, it can generate more diverse image data that meets specific conditions, avoiding problems such as low sample quality, unstable training, and lack of extensiveness in traditional methods.

[0032] 4. The present invention can achieve better matching of data and labels in generative adversarial networks, improve the quality and diversity of image data samples, and significantly enhance the stability in model training.

[0033] 5. The present invention can be applied to various types of small samples or low-quality image data samples, achieve high-quality, multi-dimensional augmentation of image data, significantly improve the stability and accuracy of data augmentation, and has considerable application value in image data augmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flow chart of the method of the present invention;

[0035] Figure 2 It is a dual-projection conditional generative adversarial network model diagram of the present invention;

[0036] Figure 3 is a visual learning flow chart of the discriminator loss of the present invention;

[0037] Figure 4 is a schematic diagram of the attention mechanism of the present invention;

[0038] Figure 5 It is a schematic diagram of data image augmentation of the present invention. DETAILED DESCRIPTION

[0039] In order to make the technical solution of the present invention clearer, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] like Figure 1 As shown, a method for image data augmentation based on conditional generative adversarial network comprises the following steps:

[0041] S1: Obtain image data samples, preprocess the original samples, and determine the conditional information that the image data augmentation depends on;

[0042] S2: Input the conditional information and random noise vector into the conditional generative adversarial network model under the double projection mechanism. The model generates images for the original samples based on the conditional information and outputs the probability that the image is a real image.

[0043] S3: Use a continuously differentiable function to normalize the gradient of the model's loss function, dynamically adjust the weight of the loss function, obtain the loss value and the optimal learning rate, and stop model training;

[0044] S4: Augment image data samples using the trained dual-projection conditional generative adversarial network model.

[0045] Conditional Generative Adversarial Networks (cGANs) are an extended model that introduces conditional information based on traditional GANs. By introducing labels or other auxiliary information, cGANs can generate images of specific categories or samples that meet specific attributes. This conditional generation capability makes cGANs show great potential in image data augmentation.

[0046] The present invention proposes an image data augmentation method based on a conditional generative adversarial network, and the specific steps include:

[0047] According to the actual scene, industrial cameras and other equipment are used for image acquisition, and combined with corresponding image processing tools, a set of clear and unprocessed original images are obtained, which will serve as the basis of the training data set. The original images are preprocessed through operations such as cropping, scaling, rotating and flipping to ensure that there is enough rich data for learning during subsequent model training, and finally form image data samples.

[0048] Determine the conditional information for generating specific targets in the image data augmentation stage, and determine the conditional information that you want to rely on when augmenting the image data, so as to generate images in the target direction. Use one-hot encoding of category labels to embed and convert category label information into high-dimensional vectors, providing richer information to guide the generator to generate images that match the conditions.

[0049] like Figure 2 As shown in the figure, a conditional generative adversarial network model is built. The conditional generative adversarial network adopts a dual projection mechanism to process data matching and label matching at the same time. It can align the real data distribution and the generated data distribution, effectively measure the difference between the two distributions, and guide the generator to generate samples that are more in line with the real distribution. When generating images, it can take into account the distribution and label information of the real image data, bind the category embedding parameters for learning, simplify the learning process, better capture the relationship between data and category labels, and improve the quality and diversity of generated images. The dual projection mechanism uses the minimized f-loss to align the real data distribution and the generated data distribution, and adds the weight of the real data distribution and label matching to the generator network. The conditional generative adversarial network includes a generator G and a discriminator D:

[0050] Generator G: The generator network takes a random noise vector and category label information as input and outputs a generated image;

[0051] Discriminator D: The discriminator network takes the generated fake image, real image and corresponding conditional information as input for comparison and identification. The probability that the output image is a real image determines the objective function of the generator G and the discriminator D, and clarifies the different matching strategies of the discriminator D to distinguish between real data and generated data.

[0052] The conditional information and random noise vector are used as inputs, and the generator G and the discriminator D learn alternately to finally generate image data. The generator G that incorporates the conditional information can generate images in a predetermined direction through learning. The discriminator D takes the generated fake image, the real image, and the matching conditional information as inputs, learns, compares, and identifies the output image, and then outputs the probability that the image is a real image.

[0053] In one embodiment of the present invention, the conditional generative adversarial network uses x to represent image data, y is its corresponding label, and the label pair is represented as Train the generator G to extract Z The samples z are transformed into samples that match the true data distribution under the given label conditions.

[0054] P represents the real distribution, Q represents the generated distribution, and the real data and fake data are represented by x + ~P X and x - ~Q X Represents. The discriminator D learns to distinguish the XY and Q XY The objective functions of the generator G and the discriminator D are described as follows:

[0055]

[0056]

[0057] In the formula, is the activation function, is the output of the discriminator D before the activation function. By choosing the activation function The original GAN ​​formula can be restored. Under this activation function, the logit of the optimal discriminator D can be decomposed in two ways: marginal matching and label matching, data matching and label distribution matching:

[0058]

[0059]

[0060] According to the marginal matching and label matching equations, the logit of the projection discriminator can be obtained:

[0061]

[0062] Where τ(·) is the image embedding function, v y is the embedding of category y, Collect the residuals.

[0063] like Figure 3As shown, represents the difference between the real and fake category embeddings. However, there is no explicit constraint to enforce this property, so we first untie v y , then let V p and V q The conditional distributions p(y|x) and q(y|x) are learned separately by minimizing the cross entropy loss with a softmax function. The loss functions are and

[0064]

[0065] To avoid the difficulty of studying alone To solve this problem, this embodiment introduces a dual projection mechanism to embed the category of the discriminator D into the parameter and Bind to a single v y In this case, the model learning process is simplified. At this time, the model learning process becomes to perform two alternating steps: the discriminator D and (τ(x + )-τ(x - )) aligned, the generator G is along Direction movement τ(x - ). By binding parameters, the model can directly match data, that is, align Q(x|y) and P(x|y). Finally, the loss function of the generator G and the discriminator D under the dual projection mechanism is described as:

[0066]

[0067]

[0068]

[0069] The attention mechanism is essentially a weight distribution method that assigns different weights according to the importance of information. In order to enhance the image generation quality of the generator, an attention mechanism module is used in the generator network to replace a traditional convolutional layer. The calculation and distribution of the attention coefficient is used to enhance the influence of important information. The module structure is as follows: Figure 4 shown.

[0070] During the training process, the generator G is responsible for generating new image data based on the learned feature information, and the discriminator D is responsible for judging the authenticity of the image generated by the generator G. This process will cause training instability due to the sharp gradient space of the model. To this end, the present invention imposes a gradient norm constraint on the discriminator function, thereby increasing the capacity of the discriminator and enhancing the stability of the training. Define a continuous differentiable function f(x): Rn →R, the gradient norm is normalized by the following formula:

[0071]

[0072] In the formula, is a general term that can be related to f(x) or a constant to avoid Approaching infinity or When the discriminator D is saturated due to overfitting, the normalized gradient norm is close to 0. Set to final, Expressed as:

[0073]

[0074] Gradient normalization makes the discriminator's gradient space smoother, reducing the mode collapse that may occur during training, making it less likely for the generator to easily deceive the discriminator. Finally, by not limiting the discriminator's model capacity, gradient normalization improves the discriminator's ability to more effectively learn to distinguish between real and generated samples, helping the generator better learn to generate realistic samples.

[0075] Through this self-control mechanism, the generator G is prevented from obtaining gradient explosion, thus stabilizing the model training process. On this basis, in order to more effectively balance the impact of data matching and label matching at different training stages, the weight of the loss function is dynamically adjusted, and the model can perform adaptive weight learning to avoid falling into the local optimal solution and improve the robustness of the model. Finally, the loss value and the optimal learning rate are obtained, and the model training is finally terminated.

[0076] The obtained image data samples are augmented by constructing and training an optimized dual-projection conditional generative adversarial network model, and the convolution layer of the generator G is used to extract features from the image to be processed; the deconvolution layer of the generator G is used to restore the feature vector from the extracted features to generate new image data, thereby completing the image data augmentation requirements in specific scenarios. The schematic diagram of image data augmentation is shown in the figure. Figure 5 shown.

[0077] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A method for image data augmentation based on conditional generative adversarial network, characterized in that the steps include: S1: Obtain image data samples, preprocess the original samples, and determine the conditional information that the image data augmentation depends on; S2: Input the conditional information and random noise vector into the conditional generative adversarial network model under the double projection mechanism. The model generates images for the original samples based on the conditional information and outputs the probability that the image is a real image. S3: Use a continuously differentiable function to normalize the gradient of the model's loss function, dynamically adjust the weight of the loss function, obtain the loss value and the optimal learning rate, and stop model training; S4: Augment image data samples using the trained dual-projection conditional generative adversarial network model.

2. The image data augmentation method based on conditional generative adversarial network according to claim 1, characterized in that: The determination of the conditional information is specifically as follows: the conditional information adopts a category label, the category label information is embedded to represent the potential features, and the category label information is embedded and converted into a high-dimensional vector as a model input.

3. The image data augmentation method based on conditional generative adversarial network according to claim 1, characterized in that: The conditional generative adversarial network adopts a double projection mechanism to simultaneously process data matching and label matching. The double projection mechanism uses minimizing f-loss to align the real data distribution and the generated data distribution, and adds the distribution of the real data and the weight of the label matching to the generator network.

4. The image data augmentation method based on conditional generative adversarial network according to claim 1, characterized in that: The conditional generative adversarial network includes a generator G and a discriminator D; Generator G: The generator network takes a random noise vector and category label information as input and outputs a generated image; Discriminator D: The discriminator network takes the generated fake image, real image and corresponding conditional information as input, compares and identifies them, and outputs the probability that the image is a real image.

5. The method for image data augmentation based on conditional generative adversarial network according to claim 4, characterized in that: The generator G includes an attention module, which is inserted into a traditional convolutional layer and replaces one layer of the traditional convolutional layer.

6. The method for image data augmentation based on conditional generative adversarial network according to claim 4, characterized in that: The step S3 specifically includes: A continuously differentiable function is used to normalize the gradient of the loss function of the discriminator D, and a gradient norm constraint is imposed to keep the gradient of the discriminator D within a controllable range during the training process; By dynamically adjusting the weight of the loss function, the training process under the dual projection mechanism balances the weight deviation caused by processing data matching and label matching at the same time; The model adaptively learns weights, obtains the loss value and the optimal learning rate, stops model training, and obtains the trained image data augmentation model.

7. The method for image data augmentation based on conditional generative adversarial network according to claim 4, characterized in that: The step S4 specifically includes: The convolutional layer of the generator G extracts features from the image to be processed and provides conditional information; The deconvolution layer of the generator G is used to restore the feature vector from the extracted features, generate new image data, expand the original image, and complete the image data augmentation requirements under specific conditions.

8. The method for image data augmentation based on conditional generative adversarial network according to claim 1, characterized in that: The image data sample acquisition specifically includes: using an industrial camera to collect image data, inputting the data into image processing software, and determining the original sample of the image data that needs to be augmented.

9. The method for image data augmentation based on conditional generative adversarial network according to claim 8, characterized in that: The preprocessing of the original sample includes cropping, scaling, rotating and flipping.

Images