A method for generating high-quality small-sample data based on multi-scale generative adversarial networks

Abstract

This invention relates to the field of computer vision and proposes a method for generating high-resolution images from small datasets using multi-scale generative adversarial networks (GANs). First, a generator based on a fully convolutional neural network (WCNN) is used to generate images. Then, a discriminator based on the same WCNN implicitly augments the images, calculating distance metrics between the generated and real images. The generator and discriminator form a single-layer image generation model. A multi-scale structure is then constructed using this model, progressively increasing the image size from low to high. Finally, a dynamic game is performed layer by layer to fit the generated image distribution to the real image distribution. This invention addresses the problems of limited dataset size and insufficient image clarity in small datasets.

Description

Technical fields:

[0001] This invention belongs to the field of computer vision, and in particular relates to a method for generating high-quality images based on small-scale sample datasets. Background technology:

[0002] With the advancement of image processing technology and computer capabilities, high-quality images can significantly improve the accuracy and predictive power of applications, changing the traditional model where using small images resulted in low precision. High-precision imaging equipment acquires increasingly higher-quality images, containing a wealth of information and more detail. In fields such as remote sensing and medicine, high-quality image annotation datasets are extremely scarce and expensive, often leading to problems such as limited samples and lack of resources. Therefore, developing a classifier or recognizer applicable to small sample domains is a very challenging task.

[0003] In recent years, with the widespread use of neural networks, various types of datasets have been explored in depth. Early methods for augmenting small datasets involved random flipping, rotation, cropping, scaling, adding noise, and color perturbation. Currently, research on learning methods based on small datasets mainly falls into three directions: model fine-tuning-based small-sample learning, data augmentation-based small-sample learning, and transfer learning-based small-sample learning. In model fine-tuning-based small-sample learning, the model is typically pre-trained on large-scale data, and the parameters of the fully connected layers or the top few layers of the neural network model are fine-tuned on the target small-sample dataset. However, while model fine-tuning methods are relatively simple, in real-world scenarios, the target dataset and the source dataset are often dissimilar, and the resulting model is prone to overfitting on the target dataset. To address the overfitting problem, data augmentation-based and transfer learning-based methods have been proposed. Data augmentation-based small-sample learning methods primarily utilize auxiliary data or information to augment or enhance the features of the original dataset. However, traditional data augmentation methods have limitations in improving sample diversity. In few-shot learning based on transfer learning, the goal is to use existing knowledge to learn new knowledge, quickly transferring learned knowledge to a new domain. However, the accuracy of transfer learning methods relies excessively on the correlation between the source and target domains.

[0004] Considering the three challenges faced by small-sample fields such as remote sensing and medicine—limited data volume, poor cross-center generalization ability of algorithms, and high accuracy requirements—this invention proposes a high-quality small-sample generation method based on multi-scale generative adversarial networks (GANs). First, an image generation model based on GANs is constructed. Then, a multi-scale model is built using the image generation model, taking images of different sizes as input and performing generative adversarial operations at different scales to learn the image distribution at different scales. Finally, a clear generated image is obtained, achieving high-quality generation of small-sample images. Summary of the Invention

[0005] This invention proposes a small-sample high-quality generation method based on multi-scale generative adversarial networks. Multiple generative adversarial network structures are used to learn the distribution of image patches in images of different sizes. Each image patch is regarded as an image. Therefore, realistic images can be generated step by step from coarse to fine and from low resolution to high resolution using small-sample datasets.

[0006] This invention presents a few-shot generation method based on multi-scale generative adversarial networks. First, an image generation model is trained using a generative adversarial network. The generator is a fully convolutional neural network (CNN) composed of multiple convolutional layers. CNNs are more efficient than traditional fully connected networks, avoiding the problems of redundant storage and convolution computation caused by using pixel blocks. The discriminator is also a CNN, also known as a Markov discriminator. Each data point in the output matrix obtained after passing the input image through multiple convolutional layers represents a receptive field in the original input image, corresponding to a region of the image. The discriminator can thus narrow the distance between the generated image and the real image in various regions, achieving overall distribution fitting, and thus achieving image augmentation. Then, a multi-scale model is constructed using the image generation model. Each scale receives images of different sizes, increasing sequentially from low to high. After the first scale is trained, the image generation model learns the contour details of the image size at the first scale. The image generated at the first scale is then magnified and fed into the second scale. The magnified image will be distorted and less clear compared to the training dataset images at the second scale. Therefore, the image generation model at the second scale learns the missing details in the magnified image. By gradually increasing the scale, a clear generated image is finally obtained. The main process of this method is shown in the attached figure. Figure 1 As shown, it can be divided into three steps: image generation model construction, multi-scale model construction, and layer-by-layer image generation.

[0007] (1) Image generation model construction

[0008] The image generation model consists of a generator and a discriminator. The input to each generator layer includes two parts: random noise from the current layer and a magnified image from the previous layer. Utilizing the residual concept, the magnified input image is not only used as input but also added to the network output as the final output, allowing the generator to learn only the missing details, indirectly reducing the learning difficulty. The Markov discriminator, composed of a fully convolutional neural network, is trained using this structure, which expands the number of samples while preserving the correlation between individual samples.

[0009] (2) Multiscale model construction

[0010] Each layer consists of a separate image generation model. The image generated at the first scale is the same size as the original image reduced by a certain factor. Since not much detail can be obtained in the small image, the image generated at the second scale is larger than the first scale, until the required size is reached.

[0011] (3) Generating images layer by layer

[0012] Except for the first-scale image generation model, which learns to generate a complete image, the image generation models at other scales learn to fill in the details lost in the generated image from the previous scale. This progressive image generation allows the multi-scale model to generate larger, higher-quality images.

[0013] Compared with the prior art, the present invention has the following obvious advantages and beneficial effects:

[0014] 1. Generators built using fully convolutional neural networks can handle images of any size without requiring separate model design for each layer scale. Discriminators built using fully convolutional neural networks, and trained using this structure, theoretically expand the number of samples while preserving the correlation between individual samples.

[0015] 2. The constructed multi-scale structure learns only the missing details from the previous scale after image magnification at each scale, making it feasible to generate high-resolution images through this progressive approach.

[0016] Experiments have shown that training with a multi-scale generative adversarial network can achieve an FID (Fréchet Inception Distance) of 38.41593 on the BreakHis dataset. This metric represents the diversity and quality of generated images; a smaller FID indicates better image diversity and quality. Therefore, this method has significant application value in tasks involving small sample datasets. Attached image description:

[0017] Figure 1 This is a flowchart of the method involved in the present invention;

[0018] Figure 2 This is a diagram of the generator structure at the first-level scale.

[0019] Figure 3 This is a diagram of the discriminator structure at the first-level scale.

[0020] Figure 4 Network structure diagram of image generation model at a single scale;

[0021] Figure 5 A network structure diagram for multi-scale generative adversarial networks;

[0022] Figure 6 Example of the final generated image; Detailed implementation method:

[0023] The present invention will be further described in detail below with reference to specific embodiments and the accompanying drawings, and the dataset is selected from the BreakHis database.

[0024] Step 1: Construction of the first-level scale image generation model

[0025] Step 1.1: Generator Construction Based on Fully Convolutional Neural Network

[0026] The proposed method for generating high-quality few-shot data based on multi-scale generative adversarial networks is implemented using the current mainstream deep learning framework PyTorch. A detailed diagram of the generator structure in this step is attached. Figure 2 The input to the first scale generator is a noise image that conforms to a normal distribution with an expected value of 0 and a standard deviation of 1. Before convolution, the noise image is padded with 5 rows on each side. It then undergoes five convolutional layers, each with a 3×3 kernel and a stride of 1. No padding is applied during convolution. The first four convolutional layers all have 32 channels and employ batch normalization, which normalizes the input to subsequent activation functions, making the values ​​more stable. After batch normalization, a Leaky Rectified Linear Unit (ReLU) is used as the activation function, with the negative slope angle set to 0.2 for faster model convergence. The fifth convolutional layer has 3 channels and uses the hyperbolic tangent function (Tanh) as the activation function, setting the output value range to between -1 and 1. After five convolutional layers, the noise image produces a generated image of the same size, which serves as the input to the first scale discriminator and the next scale generator.

[0027] Step 1.2: Construction of a discriminator based on a fully convolutional neural network

[0028] The detailed structural diagram of the discriminator proposed in this step is attached. Figure 3 The discriminator takes either a real or generated image as input. The image passes through five convolutional layers, all with a 3×3 kernel and a stride of 1. No padding is applied during convolution. The first four convolutional layers have 32 channels and use batch normalization. After batch normalization, a linear unit with leakage correction is used as the activation function, with the negative slope angle set to 0.2. The fifth convolutional layer has 1 channel and no activation function is applied. After passing through these five convolutional layers, the image yields a one-dimensional matrix. The mean of all elements in this matrix is ​​used as the distance metric and as the input to the subsequent loss function.

[0029] Step 2: Multi-scale model construction

[0030] The structure of the multi-scale model is attached. Figure 5 , where G represents the generator, D represents the discriminator, Z represents the noise map, F represents the generated image, and R represents the real image.

[0031] Step 2.1: Determine the number of scales N

[0032] In the BreakHis dataset, the original image dataset has a resolution of 700×460. In the model, the minimum input size of the image at each scale cannot be lower than 32, and the maximum size cannot be higher than 256. Therefore, the original dataset images are scaled proportionally to 256×168, and this size will also be used as the input size of the highest scale. The scaling factor r between each scale in the model is set to 0.75. The image input sizes at each scale are calculated from high to low using the scaling factor r: 256×168, 192×126, 144×95, 108×72, 81×54, 61×41. Continuing downwards, the minimum size will be lower than 32. There are a total of 6 scales, so the scale N is set to 6.

[0033] Step 2.2: Constructing image generation models for all scales except the first scale.

[0034] The second layer and all subsequent layers require receiving generated images from the previous layer. See the appendix for the specific image generation model structure. Figure 4 The input to the generator at the nth level is a noise map Z that conforms to a standard normal distribution. n and the image F generated from the n-1 layer scale generator n-1 Before inputting into the network, multiply by F by 1 / r. n-1 The image is magnified to obtain the magnified generated image F. n-1 r F n-1 r and Z nFive rows are filled in each of the top, bottom, left, and right sides. The sum of these rows is then fed into a 5-layer convolutional network. All five convolutional layers have a 3×3 kernel size and a stride of 1, and no padding is applied during convolution. The first four convolutional layers have a channel count of 32x(pow(2,n / / 4)), where " / / " is an arithmetic operator representing integer division, returning the integer part of the quotient rounded down. The pow function is used for exponentiation, with 2 as the base and n / / 4 as the exponent. This means the channel count doubles every four scales, so the channel counts from scale 1 to 7 are 32, 32, 32, 32, 64, 64, and 64 respectively. The fifth convolutional layer has 3 channels and uses the hyperbolic tangent function as the activation function, limiting the output value to between -1 and 1. Finally, the network output will be compared with F... n-1 r The resulting image F obtained by addition n F n It will be used as input to the discriminator and the next scale generator.

[0035] The discriminator at the nth scale takes either a real or generated image as input and passes through five convolutional layers. Each convolutional layer has a 3×3 kernel size and a stride of 1. No padding is applied during convolution. The first four convolutional layers have 32x (pow(2,n / / 4)) channels and use batch normalization. After batch normalization, a linear unit with leakage correction is used as the activation function. The fifth convolutional layer has 1 channel and no activation function is applied. After passing through these five convolutional layers, the image yields a 1D matrix. The mean of all elements in this matrix is ​​used as the distance metric and as the input to the subsequent loss function.

[0036] Step 3: Generate images layer by layer

[0037] Step 3.1: Dynamic game theory is performed using a loss function. Before the game begins, the discriminator is pre-trained for 2000 rounds. During the game phase, the discriminator is trained three times consecutively, and the generator is trained three times consecutively. This avoids the problem of the generator having too small a gradient during optimization. The optimization process is performed using a loss function. The loss function consists of two parts: adversarial loss and reconstruction loss. The overall loss function formula is as follows:

[0038]

[0039] Where D is the discriminator and G is the generator. To minimize the loss function when optimizing the generator, and maximize the loss function when optimizing the discriminator. adv (G,D) represents the adversarial loss, as shown in formula (2). L rec (G) represents the reconstruction loss, as shown in formula (4). In this invention, the reconstruction loss weighting coefficient is ∝ rec Setting it to 10 provides versatility.

[0040] Combat loss L adv The specific formula for (G,D) is as follows:

[0041]

[0042] The adversarial loss employs WGAN-GP loss to improve model stability and prevent mode collapse. D(x) represents the probability that the image is true when the input image to the discriminator is image x. Let D(x) be the mathematical expectation of the image data, where x follows a probability distribution p of the real image data. data (x); D(G(z)) is the probability that the discriminator classifies the generated image as real when G(z) is input into a certain latent space. Let D(G(z)) be the mathematical expectation of the noise, where z follows a normal distribution and the probability distribution p is the noise. z (z); Follows the distribution X, The formula is shown in (3), where α is a random number between -1 and 1. It involves performing a linear interpolation between the real image and the generated image. for exist gradient on, That is, the L2 norm of the gradient relative to the original input should be constrained to a mathematical expectation around 1; λ is the gradient penalty factor, and in this invention, λ is taken as 10, which has universality.

[0043]

[0044] Reconstruction loss L rec (G) The specific formula is as follows:

[0045]

[0046]

[0047] Where n represents the layer number at the current scale, x1 and x n These are the real images at the first and nth scales, respectively, and they remain unchanged throughout the training process. and These are the reconstructed images at the first and nth scales, respectively, as shown in formula (5), where z * This is the noise map input at the first scale. For all real images, there will be a fixed z-axis. * It remains constant throughout the training process. G1(z) * (,0) represents the generator input z at the first-level scale. * Reconstructed image obtained after noise map This indicates that the generator outputs a reconstructed image at the nth scale after taking the reconstructed image at the (n-1)th scale as input and magnifying it by 1 / r. and The L2 norm represents the difference between the reconstructed image and the ground truth image, essentially comparing the reconstructed image with the ground truth image using the mean squared error. Minimizing the reconstruction loss can, to some extent, allow the randomly generated image to fit the ground truth image as closely as possible, controlling the divergence of the generated image. The divergence is determined by the reconstruction loss weighting coefficient ∝ rec control.

[0048] Subsequently, the generator is trained by minimizing the loss function, and the discriminator is trained by minimizing the loss after taking the negative of the loss function. Both employ the Adam (Adaptive Moment Estimation) optimization algorithm. In this invention, the learning rate is set to 0.0005, the exponential decay rate for first-moment estimation is set to 0.5, the exponential decay rate for second-moment estimation is set to 0.999, and the dynamic learning rate is adjusted so that it becomes 0.1 times its previous value every 2000 iterations. A total of 24000 iterations are performed.

[0049] Step 3.2: Then train from low to high scales one by one until all scales are trained, forming a small sample high-quality generation framework based on multi-scale generative adversarial networks.

[0050] Step 3.3: Evaluation of Test Results

[0051] The generated images are evaluated using the FID (Fréchet Inception Distance) metric. Mathematically, FID represents the distance between two distributions; a smaller distance indicates that the generated image distribution closely resembles the real image distribution. The best possible score is 0.0, indicating that the two sets of images are identical. This metric reflects the diversity and quality of the generated images; a smaller FID indicates better image diversity and quality. The FID calculation formula is shown in (6).

[0052] FID = ||μ r -μ g || 2 +Tr(Σ r +Σ g -2(Σ r Σ g ) 1 / 2 (6)

[0053] In FID, the 2048-dimensional vector before the fully connected layer is extracted as the image feature using a pre-trained Inception V3. The Inception V3 model and its weights in this invention are provided by PyTorch 1.12.1. In equation (6), μ r μ represents the mean of features in a real image. g Σ represents the feature mean of the generated image. r The covariance matrix of features of a real image, Σ g Generates the covariance matrix of the features of the image. Tr() calculates the sum of the eigenvalues ​​of the matrix.

[0054] Nine generated images were randomly selected for display; see attached. Figure 6 .

[0055] The above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the present invention. The scope of protection of the present invention is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to the present invention within its spirit and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of the present invention.

Claims

1. A method for small sample high-quality generation based on a multi-scale generative adversarial network, characterized in that: First, a generative image model is trained using a generative adversarial network, wherein the generator is a fully convolutional neural network composed of multiple convolutional layers; The discriminator is also a fully convolutional neural network composed of multiple convolutional layers. The output matrix obtained by inputting the image through multiple convolutional layers represents a receptive field in the original input image, corresponding to a region of the image. The discriminator can thus narrow the distance between the generated image and the real image in each region to achieve overall distribution fitting, thereby achieving image augmentation. Then, a multi-scale model is constructed using multiple image generation models. Each layer of the model receives images of different sizes, which increase from low to high. After the first layer is trained, the image generation model learns the outline details of the image at the first layer; The generated image at the first layer is enlarged and input into the second layer. The enlarged image may be distorted and unclear compared to the training data set image at the second layer. Therefore, the image generation model at the second layer learns the missing detail information of the enlarged image. The scale is gradually increased, and finally a clear generated image is obtained. Step 1: Construction of the first layer scale image generation model Step 1.1: Generator construction based on a fully convolutional neural network The input of the first layer generator is a noise image with a normal distribution of 0 expected value and 1 standard deviation. The noise image is padded with 5 rows on the top, bottom, left, and right before convolution. After 5 convolutional layers, the convolution kernel size is 3x3, the step is 1, and no padding is performed during convolution. The first four convolutional layers have 32 channels, and batch normalization is used. Batch normalization normalizes the input of the subsequent activation function, making the values more stable. After batch normalization, the leaky rectified linear unit is used as the activation function, with a negative slope angle of 0.2 to make the model converge faster. The fifth convolutional layer has 3 channels and uses the hyperbolic tangent function as the activation function to limit the output value range to -1 to 1. The noise image is processed through 5 convolutional layers to obtain a generated image of the same size as the input, which is used as the input of the first layer discriminator and the input of the next layer generator. Step 1.2: Discriminator construction based on a fully convolutional neural network The input of the discriminator is either a real image or a generated image. After 5 convolutional layers, the convolution kernel size is 3x3, the step is 1, and no padding is performed during convolution. The first four convolutional layers have 32 channels, and batch normalization is used. After batch normalization, the leaky rectified linear unit is used as the activation function, with a negative slope angle of 0.

2. The fifth convolutional layer has 1 channel and does not use an activation function. After 5 convolutional layers, a 1-dimensional matrix is obtained, and the mean of all elements in the matrix is used as the distance indicator, which is input into the subsequent loss function. Step 2: Multi-scale model construction Step 2.1: Determine the number of scales N The resolution size of the original image dataset is 700x460; the minimum input size of each scale image in the model cannot be lower than 32, and the maximum size cannot be higher than 256; therefore, the original dataset image length and width are scaled in proportion to 256x168 by scaling, and this size will also be used as the input size of the highest scale; the scaling factor r between each scale in the model is set to 0.75; the image input size of each scale is calculated from high to low by the scaling factor r, which is 256x168, 192x126, 144x95, 108x72, 81x54, 61x41, and the minimum size will be lower than 32, so there are 6 layers of scales, and therefore the scale N is set to 6; Step 2.2: Generating model construction for the rest of the scale images except the first layer scale The second layer includes a second layer above the scale needs to receive generated images from the previous scale; the input of the generator of the n-th layer scale is noise map Z conforming to the standard normal distribution n and the image F generated from the n-1-th layer scale generator n-1 ; before inputting into the network, F is enlarged by 1 / r n-1 ; the enlarged generated image F is obtained by enlarging the image n-1 r ; F n-1 r and Z n are respectively filled with 5 rows above and below and left and right, and then added and input into a 5-layer convolutional network, wherein the convolution kernel size of the 5 convolutional layers is 3x3, the step is 1, and no padding is performed in the convolution process; the channel number of the first four convolutional layers is all 32x(pow(2,n / / 4)), wherein " / / " is an arithmetic operator, which represents integer division, and can return the integer part of the quotient down, and the pow function is used to perform power operation with 2 as the base and n / / 4 as the exponent; the whole indicates that every 4 scales, the channel number is doubled, so the channel numbers from the 2nd to the 7th scales are 32, 32, 32, 64, 64, and 64 respectively; the channel number of the fifth convolutional layer is 3, and the hyperbolic tangent function is used as the activation function to limit the output value range to-1 to 1; the output result of the last network is added to F n-1 r to obtain the generated image F n , and F n will be used as the input of the discriminator and the input of the next scale generator; The input of the discriminator of the nth layer scale is a real image or a generated image, which passes through 5 convolutional layers, the convolution kernel size is 3x3, the step is 1, and no padding is performed during the convolution process. The channel number of the first four convolutional layers is 32x(pow(2, n / / 4)), and batch normalization is used. After batch normalization, a leaky rectified linear unit is used as the activation function. The channel number of the fifth convolutional layer is 1, and no activation function is added. After the image passes through the 5 convolutional layers, a 1-dimensional matrix is obtained, and the mean of all elements in the matrix is used as the distance index, which is used as the input of the subsequent loss function.

2. The method of claim 1, wherein: Step 3: Generating images layer by layer Step 3.1: Dynamic game through loss function, pre-training the discriminator for 2000 rounds before the game starts, and training the discriminator for 3 times and the generator for 3 times in the game stage, which can avoid the problem of too small gradient in the optimization process of the generator; the optimization process is through the loss function; the loss function is divided into two parts: adversarial loss and reconstruction loss; the overall loss function formula is as follows: where D is the discriminator, G is the generator; min G max D V(D,G) is to minimize the loss function when optimizing the generator, and to maximize the loss function when optimizing the discriminator; L adv (G,D) is the adversarial loss, as shown in equation (2); L rec (G) is the reconstruction loss, as shown in equation (4); wherein the reconstruction loss weight coefficient ∝ rec is set to 10, which is universal; Adversarial loss L adv (G, D) are given by the following equations: where the adversarial loss adopts WGAN-GP loss to improve the stability of the model and prevent mode collapse; D(x) is the probability that the input picture of the discriminator is x, the picture is real, denotes the mathematical expectation of D(x), where x obeys the probability distribution p of the real image data data (x); D(G(z)) is the probability that the discriminator judges the generated image as real when the generator inputs a certain hidden space G(z), denotes the mathematical expectation of D(G(z)), where z obeys the probability distribution p of the normal distribution noise z (z); obeys the distribution The formula is shown in equation (3), and a is a random number between -1 and 1, is a linear difference between the real image and the generated image; is the gradient on , the mathematical expectation of the L2 norm of the gradient with respect to the original input is constrained around 1, that is, the mathematical expectation of the L2 norm of the gradient with respect to the original input is constrained around 1. λ is the gradient penalty factor, where λ is 10, which is universal; reconstruction loss L rec (G) The specific formula is as follows: where n represents the number of layers at the current scale, x1 and x n are the real images at the 1st scale and the n-th scale respectively, which remain unchanged throughout the training process; and are the reconstructed images at the 1st scale and the n-th scale respectively, as shown in equation (5), where z * is the input noise image at the 1st scale, which remains unchanged throughout the training process for all real images; G1(z * , 0) represents the reconstructed image obtained after inputting the z * noise image into the generator at the 1st scale * and represents the 2-norm of the difference between the reconstructed image and the real image, which is equivalent to comparing the reconstructed image with the real image using mean square error; minimizing the reconstruction loss can to some extent make the randomly generated image fit the real image as much as possible, thereby controlling the divergence of the generated image; the divergence is controlled by the reconstruction loss weight coefficient ∝ rec ;​​ Subsequently, the generator is trained by minimizing the loss function, and the discriminator is trained by minimizing the loss after taking the opposite number of the loss function; both use the Adam optimization algorithm, where the learning rate is set to 0.0005, the exponential decay rate of the first moment estimate is set to 0.5, the exponential decay rate of the second moment estimate is set to 0.999, and the dynamic learning rate adjustment is set to the learning rate being 0.1 times of the previous value every 2000 iterations; a total of 24000 iterations are performed; Step 3.2: Then train from low to high one by one until all scales are trained, forming a small sample generation framework based on the multi-scale generative adversarial network.

Images