Image generation method and system suitable for multi-class and scarce sample data scenarios

Abstract

The application discloses an image generation method and system suitable for multi-category and rare sample data scenes, relates to the technical field of image generation, and comprises the following steps: preprocessing original sample data to obtain image training samples; constructing and initializing a generative adversarial network, wherein the generative adversarial network comprises an evolutionary population composed of multiple generator network individuals and a discriminator network, preset hyperparameters are set, the fitness of each generator network is evaluated by using a fitness function to determine an optimal generator network individual; candidate offspring are generated by using different mutation strategies according to the fitness; next-generation individuals are selected from the candidate offspring; the error parameters of the discriminator network are adjusted, and the discriminator network parameters are updated; and when the maximum number of iterations is reached, the generator network with the highest fitness is obtained and used to generate images. Compared with the method for improving the loss function or enhancing the network structure to improve the image quality in the prior art, the mode collapse problem is effectively relieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image generation, and in particular to an image generation method and system suitable for multi-class and rare sample data scenarios. BACKGROUND

[0002] As one of the key ways to artificial intelligence, the core of deep generation technology is to estimate parameters by learning the probability distribution of the data to be generated, so as to realize data generation. Since the generative adversarial network (GAN) was proposed by Ian Goodfellow, it has become a research hotspot in the field of deep learning due to its excellent generation ability, but the problems such as mode collapse in its practical application still need to be solved.

[0003] The generative adversarial network (GAN) realizes high-quality sample generation through the adversarial training of the generator (G) and the discriminator (D): the goal of the generator is to generate realistic samples to deceive the discriminator, and the discriminator needs to distinguish between real samples and generated samples. In theory, the two can reach Nash equilibrium through continuous confrontation, so that the generator generates samples consistent with the real data distribution. Gradient algorithms such as Adam algorithm are widely used in GAN training, which combines the advantages of momentum method and RMSProp, and can accelerate training and improve convergence.

[0004] In order to overcome the GAN model collapse problem and improve the quality and diversity of generated samples, there are currently various methods to improve the technology of generating images by generative adversarial network model. The mainstream methods are as follows: first, design a new loss function (such as LSGAN), replace the cross-entropy loss with the least squares loss, and reduce the risk of gradient disappearance; second, improve the network structure of the generator and the discriminator (such as DCGAN), integrate the convolutional neural network (CNN) into the GAN, and improve the image generation fidelity; third, train multiple discriminators at the same time (such as UnrolledGAN, MIX-plusGAN, MCL-GAN), provide more gradient information and feedback through the cooperation of multiple discriminators; fourth, introduce evolutionary strategies (such as EGAN), break through the single generator training mode, and enhance the global search ability and exploration of the solution space.

[0005] In the prior art, the method based on loss function improvement still has the problems of insufficient diversity and quality defects of generated images when the training data is scarce or the sample categories are more; improving the network structure does not fundamentally solve the problem of mode collapse; training multiple discriminators greatly increases the computational complexity, and introducing evolutionary strategies requires a large number of fitness evaluations, which is slow in convergence speed, and the mode collapse problem still exists only relying on the transformation of the loss function as a mutation operator. SUMMARY

[0006] The application provides an image generation method and system suitable for multi-category and rare sample data scenarios, aiming to solve the mode collapse problem of the image generation technology based on a GAN model, in particular, the problems of insufficient diversity of generated images and significant decline in quality of generated images in application scenarios where training samples are significantly insufficient or the number of sample categories is large.

[0007] To achieve the above object, the application adopts the following technical solutions:

[0008] In a first aspect, the application provides an image generation method suitable for multi-category and rare sample data scenarios, comprising:

[0009] Obtaining original sample data and performing preprocessing to obtain image training samples;

[0010] Constructing and initializing a generative adversarial network, wherein the generative adversarial network comprises an evolutionary population composed of multiple generator network individuals and a discriminator network;

[0011] Predefining hyperparameters and training the generative adversarial network using the image training samples, determining the optimal generator network individual by evaluating the fitness of each generator network through a fitness function, generating candidate offspring using different mutation strategies according to the fitness, selecting the next generation individual from the candidate offspring using an accept-reject method, adjusting the error parameter of the discriminator network and updating the discriminator network parameter;

[0012] When the maximum number of iterations is reached, the generator network with the highest fitness is obtained and used to generate images.

[0013] In further technical solutions, the preprocessing comprises removing duplicate images and damaged images from the original sample data, uniformly scaling the original sample data in size and normalizing the pixel values, and then dividing the original sample data into several batches to obtain the image training samples.

[0014] In further technical solutions, the generative adversarial network is trained using an evolutionary method based on gradient guidance, and the predefining of the hyperparameters comprises learning rate, maximum number of rejections of the algorithm, error parameter, fitness threshold, mutation interval, number of mutations performed by an individual in one mutation strategy, and maximum number of iterations of the algorithm.

[0015] In further technical solutions, the specific method for generating candidate offspring using different mutation strategies according to the fitness is as follows: when the fitness of the generator network is greater than the lowest fitness threshold and less than the highest fitness threshold, the network parameters of the generator network are controlled to learn towards the optimal generator network individual, learning in a knowledge distillation manner to generate first mutated offspring; when the fitness of the generator network is less than the lowest fitness threshold, the network parameters of the generator network are controlled to explore towards a randomly selected individual to generate second mutated offspring.

[0016] Calculate the gradient estimates of the objective function for the first and second mutated offspring, and update the first and second mutated offspring to obtain candidate offspring.

[0017] A further technical solution is that the method for selecting the next generation individual is as follows: if the fitness of the candidate offspring is higher than that of the current generator network individual and exceeds a set threshold, then the candidate offspring is accepted as the next generation individual; otherwise, the candidate offspring is rejected from entering the next generation, and the current generator network individual is retained as the next generation individual.

[0018] If all candidate offspring are rejected to enter the next generation, the rejection count is increased by 1; otherwise, the rejection count remains unchanged. When the rejection count reaches the maximum set rejection count, training ends.

[0019] A further technical solution is that the method for adjusting the discriminator network error parameters is as follows: if a candidate offspring is accepted into the next generation, the discriminator network error parameters are adjusted, expressed as: ;in, This represents the adjusted error parameter. This represents the error parameter before adjustment. Indicates the learning rate. Indicates a constant coefficient; otherwise, the error parameter remains unchanged.

[0020] A further technical solution involves updating the discriminator network parameters by calculating the gradient of the objective function estimate corresponding to the discriminator network, and when the following conditions are met... At that time, the discriminator parameters are updated based on the stochastic gradient ascent algorithm, where, This indicates that the gradient of the objective function is estimated. This represents the optimal generator network individual after the update. express Discriminator network in the next iteration This indicates the adjusted error parameter.

[0021] Secondly, the present invention provides an image generation system suitable for scenarios with multiple categories and scarce sample data, including:

[0022] The training sample acquisition module is configured to: acquire raw sample data and preprocess it to obtain image training samples;

[0023] The Generative Adversarial Network (GAN) construction and initialization module is configured to: construct and initialize a GAN, wherein the GAN includes an evolutionary population composed of multiple generator network individuals and a discriminator network;

[0024] The training module is configured to preset hyperparameters, train the generative adversarial network by using image training samples, determine the optimal generator network individual by evaluating the fitness of each generator network through a fitness function, generate candidate offspring by using different mutation strategies according to the fitness, select the next generation individual in the candidate offspring by using an accept-reject method, and adjust the error parameters of the discriminator network and update the discriminator network parameters.

[0025] The image generation module is configured to obtain the generator network with the highest fitness when the maximum number of iterations is reached, and generate an image by using the generator network.

[0026] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0027] 1. The present application introduces an evolutionary population composed of multiple generator network individuals, cooperatively optimizes multiple generator network individuals, each individual corresponding to an independent generator network parameter, generates candidate offspring through a mutation mechanism, and guides the optimization direction combined with gradient information, thereby significantly improving the convergence speed and global search ability of the individual. Compared with the traditional method of improving image quality by improving the loss function or enhancing the network structure, the present application effectively alleviates the mode collapse problem.

[0028] 2. The present application uses a mutation strategy based on knowledge distillation, realizes experience sharing by introducing knowledge transfer between individuals, makes the generator network individual in the evolution process be able to consider sample quality and diversity, and to a great extent overcomes the problem of single mutation strategy and blind search direction of traditional methods.

[0029] 3. The present application starts from the balance between global search ability and local convergence, groups the generator individuals in the population according to the fitness, and focuses on rapid convergence for high fitness individuals and global exploration for low fitness individuals. At the same time, the present application designs an accept-reject strategy to select the next generation individual, which makes the designed method have theoretical convergence guarantee. BRIEF DESCRIPTION OF DRAWINGS

[0030] The drawings accompanying the specification of the present application serve to provide a further understanding of the present application, and the illustrative embodiments of the present application and their descriptions serve to explain the present application, and do not constitute a limitation of the present application.

[0031] Figure 1 is a flowchart of the image generation method of the present application applicable to multi-class and rare sample data scenarios;

[0032] Figure 2Figure 1 is a comparison chart of images generated by the method of the present application and current mainstream methods in the NotMNIST dataset, wherein (a) is a GAN generated image, (b) is a DCAN generated image, (c) is an LSGAN generated image, (d) is a WGAN generated image, (e) is a WGANGP generated image, and (f) is an image generated by the method of the present application.

[0033] Figure 3 Figure 2 is a comparison chart of images generated by the method of the present application and current mainstream methods in the Omniglot dataset, wherein (a) is a GAN generated image, (b) is a DCAN generated image, (c) is an LSGAN generated image, (d) is a WGAN generated image, (e) is a WGANGP generated image, and (f) is an image generated by the method of the present application. DETAILED DESCRIPTION

[0034] Embodiment 1

[0035] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0036] As shown in Figure 1, the present embodiment provides an image generation method suitable for multi-class and rare sample data scenarios, and the specific steps are as follows: Figure 1

[0037] S1: Obtain original sample data and perform preprocessing to obtain image training samples.

[0038] In step S1, four publicly available image datasets are used to obtain original sample data, namely MNIST, FashionMNIST, NotMNIST, and Omniglot, which are all publicly available general image datasets.

[0039] Preprocessing is to remove duplicate images and damaged images from the original sample data, uniformly scale the original sample data, normalize the pixel values, and then divide them into several batches to obtain image training samples.

[0040] S2: Construct and initialize a generative adversarial network, which includes an evolutionary population composed of multiple generator network individuals, a discriminator network, and a trained objective function form.

[0041] In step S2, the classical GAN training generative adversarial network objective function is used, which has the following form:

[0042] (1)

[0043] ​wherein, a generator and a discriminator network, a real sample randomly drawn from a data distribution , a random noise drawn based on a distribution , is an image sample generated by the generator.

[0044] In step S2, the network is constructed and initialized, and the constructed generative adversarial network is composed of an evolutionary population of multiple generator network individuals and a discriminator network , wherein denotes the evolutionary population, denotes a generator network, the superscript denotes the generation number of evolution (such as 0 representing the initial generation), and the subscript denotes the number of generator network individuals (a total of ), each generator network individual , wherein the subscript denotes the th individual, the superscript denotes the th iteration, and an image sample is generated by inputting Gaussian noise from a Gaussian distribution .

[0045] S3: Pre-set hyperparameters and train the generative adversarial network using image training samples, determine the optimal generator network individual by evaluating the fitness of each generator network through a fitness function; generate candidate offspring according to the fitness by using different mutation strategies; select the next generation individuals from the candidate offspring by using the acceptance-rejection method; adjust the error parameter of the discriminator network and update the discriminator network.

[0046] In step S3, the generative adversarial network is trained by using a gradient-guided evolutionary method, and the pre-set hyperparameters include a learning rate , in the embodiment, the generator parameter is updated by a stochastic gradient descent algorithm and the value is usually a positive number close to 0, in the embodiment, a certain constant value 0.001 between the interval is set; the maximum number of rejections , which is initially 0 and increases by 1 when all candidate offspring in the population are rejected into the next generation; an error parameter, in the embodiment, the error parameter takes a smaller positive value close to 0; a fitness threshold, including a highest fitness threshold and a lowest fitness threshold , in the embodiment, = 0.8, = 0.4; mutation interval In this embodiment, the positive integer is greater than 0 and not too large, in this embodiment, = 14; the number of mutations performed by an individual in one mutation strategy d = 3) and the maximum number of iterations of the algorithm = 4000 and above.

[0047] In this embodiment, the fitness function is used to evaluate the fitness of each generator network individual , and the generator network individual with the highest fitness is selected as the optimal generator network individual . Specifically, the loss function of the generator is used as the fitness function to evaluate the fitness value, and the fitness value is normalized to obtain the fitness , and the specific formula is as follows:

[0048] ; (2)

[0049] wherein, , are the fitness values of the generator with the highest fitness and the generator with the lowest fitness in the population, respectively.

[0050] In this embodiment, according to the fitness of the generator network individual , a mutation operator is applied to generate a mutated individual (mutated offspring) , wherein the superscript represents the iteration number of the current iteration k , the subscript represents the individual number, and the underline "-" represents the candidate individual. Among them, different fitness individuals use different mutation strategies, and the specific method of generating candidate offspring according to the fitness and using different mutation strategies is as follows: when the fitness of the generator network is greater than the lowest fitness threshold and less than the highest fitness threshold , the generator network is controlled to learn towards the optimal generator network individual , that is, to learn in the manner of knowledge distillation, to generate the first mutated offspring; specifically:

[0051] The generator network individual obtains the first mutated offspring by leaning towards the optimal generator network individual . Specifically, based on , the Adam update is continuously performed for steps to minimize the following loss function:

[0052] ​​​(3)

[0053] wherein is a noise sample, and are the outputs of the best individual and the higher-adaptation-value group individual before the generator activation layer, respectively.

[0054] When the adaptation of the generator network is less than the lowest adaptation threshold value (Amin ), the generator network is controlled to explore a randomly selected individual, to generate a second variation offspring, to promote the exploration of potential patterns, and to enhance the diversity. Specifically, the network parameters of the randomly selected individual

[0055] and the randomly selected individual are explored to obtain the second variation offspring. Specifically, the continuous step Adam update is used to minimize the following loss function:

[0056] (4)

[0057] wherein is a noise sample, and are the outputs of the randomly selected generator before the activation layer, and the output of the lowest-adaptation-value group individual before the generator activation layer.

[0058] The Adam algorithm is used to update in the negative gradient direction to accelerate the convergence, to calculate the gradient estimates of the objective functions of the first variation offspring and the second variation offspring , and to update the first variation offspring and the second variation offspring (x1 ), to obtain the candidate offspring ( x ), and to return the candidate offspring and the candidate population .

[0059] In the embodiment, the method for selecting the next generation individual from the candidate offspring by using the acceptance-rejection method is as follows: if the adaptation of the candidate offspring is higher than the adaptation of the current generator network individual , and exceeds the set threshold value δ (a positive value), the candidate offspring is accepted as the next generation individual ( x ); otherwise, the candidate offspring is rejected to enter the next generation ( x ), and the current generator network individual is still retained as the next generation individual . This is represented as: ​

[0060]

[0061] wherein the left arrow “ ” in the formula represents assignment, and the formula above represents that the corresponding individual or is assigned to the offspring individual .

[0062] If all candidate offspring are rejected to enter the next generation (N ), the number of rejections is increased by 1 (N ), otherwise the number of rejections remains unchanged (N ). When the number of rejections reaches the maximum number of rejections set (N ), the training is ended.

[0063] In this embodiment, the error parameter is dynamically adjusted by whether the candidate offspring is accepted as the next generation individual. The specific method of adjusting the error parameter of the discriminator network is that the candidate offspring is accepted to enter the next generation (N ), that is, the mutation operation produces a generator with better fitness, and the error parameter of the discriminator network is adjusted, which is represented as: ; wherein, represents the adjusted error parameter, represents the error parameter before adjustment, represents the learning rate, represents the Lipschitz constant; to enhance the ability of the discriminator to distinguish between true and false images, otherwise, the error parameter remains unchanged .

[0064] In this embodiment, the method of updating the discriminator network is: calculating the estimated gradient of the objective function of the discriminator network , when is satisfied, updating the discriminator parameters based on the stochastic gradient ascent algorithm, wherein, represents the estimated gradient of the objective function, represents the updated optimal generator network individual, represents the th iteration of the discriminator network.

[0065] S4: When the maximum number of iterations is reached, the generator network with the highest fitness is obtained and used to generate images.

[0066] In this embodiment, the detailed execution steps of the mutation update strategy are summarized as Algorithm 1, the detailed execution steps of the accept-reject strategy are summarized as Algorithm 2, and the detailed execution steps of the discriminator gradient ascent algorithm are summarized as Algorithm 3. The algorithm flow of each part is as follows:

[0067] Algorithm 1:

[0068] Input: population , the optimal individual in population , mutation interval , fitness threshold , and , the number of mutations .

[0069] (1) if 0 then (here "% " represents the remainder)

[0070] Calculate the fitness of individual ; if

[0071] then; The individual with a higher fitness value is updated to the optimal individual

[0072] using formula (3) times to obtain the mutated offspring ; Based on the parameters

[0073] , the Adam update is continuously performed steps to minimize the objective function defined by formula (3), and the updated individual is the candidate individual ;

[0074] else if then;

[0075] Based on the parameters , the Adam update is continuously performed steps to minimize the objective function defined by formula (4), and the updated individual is the candidate individual ;

[0076] else

[0077] (The highest fitness value individual is directly assigned as the mutation offspring )

[0078] end if

[0079] end if

[0080] ​​(2) Calculate the estimated gradient of the mutated individual based on the generator's objective function. ;

[0081] Update the mutated offspring using the Adam update strategy. Obtain candidate offspring ;

[0082] (3) Return to candidate offspring and candidate populations .

[0083] Algorithm 2:

[0084] Input: Parent individual and their corresponding candidate offspring Set a threshold for accepting candidate offspring into the next generation. and annealing parameters ;

[0085] (1) if then

[0086] set up ;

[0087] else

[0088] Based on the 0-1 distribution, set Its probability is ,

[0089] set up Its probability is ;

[0090] end if

[0091] (2) if then

[0092]

[0093] else

[0094]

[0095] end if

[0096] (3) Return to the next generation of parent individuals .

[0097] Algorithm 3:

[0098] Input: population The optimal individual Error parameters ;

[0099] (1) Settings ;

[0100] Random sampling based on Gaussian distribution Small batch noise ;

[0101] Randomly sampling from the training set Small batch real training samples ;

[0102] (2) Calculate the objective function gradient estimate of the optimal individual according to the objective function of the discriminator ;

[0103] (3) while do

[0104] Based on the gradient , the discriminator is updated using the Adam algorithm;

[0105] Set (reset to )

[0106] end while

[0107] Algorithm 1 implements mutual learning between individuals according to the fitness of the individual, that is, the individual with a higher fitness learns from the individual with the highest fitness, and the individual with a lower fitness learns from two randomly selected individuals in the population, thereby enhancing global exploration. Algorithm 2 uses the acceptance-rejection strategy selection operator to ensure that better individuals enter the next generation and ensure the convergence performance of the algorithm. The generator and discriminator of the generative adversarial network are iteratively trained until the iteration is terminated, and the optimal population is output . Algorithm 4 is as follows:

[0108] Algorithm 4:

[0109] Input: population , maximum number of rejections , maximum number of iterations of the algorithm , error parameter of the initial generation , learning rate of the gradient algorithm , objective function of GAN model training

[0110] (1) Set ;

[0111] (2) Initialize the evolutionary population ;

[0112] (3) Find the optimal individual with the highest fitness in the population ;

[0113] while do

[0114] for do

[0115]

[0116] based on Gaussian distribution random sampling a small batch of noise

[0117] generate candidate offspring using algorithm 1 ;

[0118] select the next generation of individuals using algorithm 2 ;

[0119] update the next generation population

[0120] end for

[0121] if (indicating that the next generation population is exactly the same as the last generation population, i.e. all candidate offspring are rejected)

[0122] then

[0123]

[0124] else

[0125] end if

[0126] find the individual with the highest fitness value in the next generation population as the optimal individual , update the discriminator using algorithm 3 set

[0127] (reset the number of iterations to )

[0128] end while

[0129] (4) return the optimal population and the optimal individual after the iterative training terminates . .

[0130] ​In this embodiment, the effectiveness of the disclosed method was tested using four publicly available image datasets: MNIST, FashionMNIST, NotMNIST, and Omniglot. Table 1 lists the total number of training samples and the key features of each dataset. It is worth noting that the Omniglot dataset contains an average of only 20 images per class, with 1623 training categories, making the training task more complex. All datasets were trained using mini-batch samples, batch by batch.

[0131] Table 1 Dataset

[0132]

[0133] On the MNIST dataset, the method disclosed in this embodiment is compared with images generated by current mainstream methods such as DCAN, LSGAN, WGAN, WGNOP, and GAN. The evaluation metrics are shown in Table 2. (The arrows in the table indicate...) "The larger the value of the corresponding indicator, the better the performance," indicated by the arrow. "This indicates that the larger the value of the corresponding indicator, the better the performance." As can be seen, compared with the comparative methods, the GEGAN method used in this embodiment achieves better quality metrics under three training samples. In terms of performance, GEGAN outperforms the best-in-class methods by 15.2%, 1.9%, and 4.9%; in terms of the quality metric FID, it outperforms the best-in-class methods by 51.8%, 40.5%, and 29.0%; and in terms of diversity, GEGAN outperforms the best-in-class methods. The results show that GEGAN outperforms the best comparison method by 3.4%, 1.1%, and 0.9%, respectively. For the diversity metric IS, GEGAN outperforms the best benchmark by 0.1% and 2.5% at training scales of 500 and 2000, respectively. Therefore, the method disclosed in this embodiment outperforms the comparison benchmark method on the tested dataset.

[0134] Table 2 Evaluation metrics for the MNIST dataset

[0135]

[0136] Note: Fr Chet Inception Distance (FID) : a method to measure the distance between two distributions, which takes into account the mean and covariance matrix of the two distributions. Focuses on quality measurement, generally the smaller the FID value, the better the generation effect. Inception Score (IS) : using the picture category classifier Inception Net-V3 to evaluate the quality of the generated pictures, focusing on diversity measurement, the larger the IS value, the better the generation effect. Image quality score ( ) and image diversity score ( ) : proposed in IE-GAN, calculated by the output of the trained generator and discriminator respectively, generally the larger the value, the better the generation effect. Maximum Mean Discrepancy (MMD) : used to measure the distance between two distributions, the smaller the MMD, the smaller the difference between the real samples and the generated samples. Its basic assumption is: if for all functions , if the corresponding image mean values of enough samples generated by two distributions on are equal, then the two distributions are considered to be the same distribution.

[0137] On the FashionMNIST dataset, the method GEGAN disclosed in the embodiment is compared with the current mainstream methods DCAN, LSGAN, WGAN, WGANGP and GAN to generate images. The evaluation index values are shown in Table 3. It can be seen that compared with the comparison methods, the method GEGAN adopted in the embodiment has: in the quality index , GEGAN is 3.6%, 14.8% and 3.9% higher than the best baseline on three training scales respectively; in the quality index FID, GEGAN is improved by 6.9% and 2.9% respectively for 500 and 2000 training samples per class; for the image diversity index , GEGAN is 2.1%, 2.3% and 0.1% higher than the best baseline method on three training scales respectively; for the diversity IS index, GEGAN is 2.9% lower than LSGAN for 2000 training samples, but for training scales of 500 and 5000, GEGAN is 10.2% and 1.2% higher than the best baseline respectively; for the MMD comprehensive index, GEGAN is 0.1% higher than the best baseline for 5000 samples. It can be seen that the method disclosed in the embodiment produces significantly higher quality and more diverse samples on the tested dataset, and has more significant advantages when the number of training samples per class is limited.

[0138] Table 3 Evaluation index values of FashionMNIST dataset

[0139]

[0140] The NotMNIST dataset contains artistic printed characters (A to J) with curled edges and obvious contours, different shapes, and stronger noise. Therefore, it is more difficult for the GAN model to generate these datasets, especially in terms of sample diversity. The method GEGAN disclosed in the present example is compared with the current mainstream methods DCAN, LSGAN, WGAN, WGANGP and GAN in generating images on the NotMNIST dataset. The evaluation index values are shown in Table 4, and the generated images are shown in Figure 2 It can be seen that the images generated by the GAN model have a serious mode collapse problem, and the problem of generating low-quality images after fine-tuning the parameters of WGAN and WGAN-GP still exists. As can be seen from Table 4, the method GEGAN of the present example has obvious advantages compared with the compared methods: GEGAN has obvious advantages in two image quality indicators, and GEGAN is 7.3% and 39.8% higher than the best baseline in and FID indicators, respectively, where GAN is excluded because its diversity indicator is poor; for the diversity IS indicator, GEGAN is 3.3% higher than the best baseline method. This indicates that the method also has good generation performance on this more difficult to train dataset.

[0141] Table 4 Evaluation index values of NotMNIST dataset

[0142]

[0143] The Omniglot dataset is an image dataset widely used for small sample learning, with 1623 categories, each with only 20 samples, and 105x105 pixels. The complexity of the Omniglot dataset increases the difficulty of generating images. The method GEGAN disclosed in the present example is compared with the current mainstream methods DCAN, LSGAN, WGAN, WGANGP and GAN in generating images on the Omniglot dataset. The evaluation index values are shown in Table 5, and the generated images are shown in Figure 3 As can be seen from Table 5, the method GEGAN used in the present example is compared with the compared methods, and GEGAN is optimal in , , IS and MMD indicators under three training scales. It can be seen that the method disclosed in the present example is superior to the compared benchmark methods on the tested datasets.

[0144] Table 5 Evaluation index values of Omniglot dataset

[0145]

[0146] Example 2

[0147] The embodiment provides an image generation system suitable for multi-class and rare sample data scenes, which specifically comprises the following modules:

[0148] The training sample acquisition module is configured to acquire original sample data, and pre-process the original sample data to obtain image training samples.

[0149] The generative adversarial network construction and initialization module is configured to construct and initialize a generative adversarial network, wherein the generative adversarial network comprises an evolutionary population composed of multiple generator network individuals and a discriminator network, and the generator network generates image samples by inputting Gaussian noise.

[0150] The training module is configured to pre-set hyperparameters, train the generative adversarial network by using the image training samples, determine an optimal generator network individual by evaluating the fitness of each generator network through a fitness function, generate candidate offspring by using different mutation strategies according to the fitness, select next-generation individuals from the candidate offspring by using an accept-reject method, adjust error parameters of the discriminator network and update discriminator network parameters.

[0151] The image generation module is configured to obtain a generator network with the highest fitness when a maximum number of iterations is reached, and use the generator network to generate images.

[0152] The implementation of the specific modules in the embodiment is described in the steps of the image generation method suitable for multi-class and rare sample data scenes in Embodiment 1, and will not be specifically described here.

[0153] The above-described embodiments only express several embodiments of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the present application. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. An image generation method suitable for multi-class and scarce sample data scenarios, characterized in that, The method comprises the following steps: obtaining original sample data and pre-processing to obtain image training samples; constructing and initializing a generative adversarial network, which comprises an evolutionary population composed of multiple generator network individuals and a discriminator network; pre-setting hyperparameters and training the generative adversarial network using the image training samples, determining the optimal generator network individual by evaluating the fitness of each generator network through a fitness function, generating candidate offspring according to the fitness using different mutation strategies, selecting the next generation individual from the candidate offspring using an accept-reject method, and adjusting the error parameters of the discriminator network and updating the discriminator network parameters; the specific method for generating candidate offspring according to the fitness using different mutation strategies is: when the fitness of the generator network is greater than the minimum fitness threshold and less than the maximum fitness threshold, controlling the network parameters of the generator network to learn towards the optimal generator network individual, learning in the manner of knowledge distillation to generate first mutated offspring; when the fitness of the generator network is less than the minimum fitness threshold, controlling the network parameters of the generator network to explore towards a randomly selected individual to generate second mutated offspring; calculating the gradient estimate of the objective function of the first and second mutated offspring and updating the first and second mutated offspring to obtain the candidate offspring; the method for selecting the next generation individual is: if the fitness of the candidate offspring is higher than the fitness of the current generator network individual and exceeds the set threshold, the candidate offspring is accepted as the next generation individual; otherwise, the candidate offspring is rejected to enter the next generation, and the current generator network individual is retained as the next generation individual; if all candidate offspring are rejected to enter the next generation, the number of rejections is increased by 1, otherwise the number of rejections remains unchanged, and when the number of rejections reaches the set maximum number of rejections, the training is ended; when the maximum number of iterations is reached, the generator network with the highest fitness is obtained and used to generate images.

2. The image generation method for multi-class and scarce sample data scenarios as claimed in claim 1 wherein, The pre-processing is to remove duplicate images and damaged images in the original sample data, uniformly scale the original sample data, normalize the pixel values, and then divide the original sample data into several batches to obtain the image training samples.

3. The image generation method for multi-class and scarce sample data scenarios as claimed in claim 1 wherein, The evolutionary method based on gradient guidance is used to train the generative adversarial network, and the pre-set hyperparameters include learning rate, maximum number of rejections, error parameter, fitness threshold, mutation interval, number of mutations performed by an individual in one mutation strategy, and maximum number of iterations.

4. The image generation method for multi-class and scarce sample data scenarios as claimed in claim 1 wherein, The method for adjusting the discriminator network error parameter is: if the candidate offspring is accepted into the next generation, the discriminator network error parameter is adjusted, denoted as: ; wherein, denotes the adjusted error parameter, denotes the error parameter before adjustment, denotes the learning rate, denotes a constant coefficient; otherwise, the error parameter remains unchanged.

5. The image generation method for multi-class and scarce sample data scenarios as claimed in claim 1 wherein, The method for updating the discriminator network parameters is: calculating the estimated gradient of the objective function corresponding to the discriminator network, when the following condition is met updating the discriminator parameters based on the stochastic gradient ascent algorithm, wherein, the estimated gradient of the objective function, the optimal generator network individual after updating, the estimated gradient of the objective function, the discriminator network in the iteration, the adjusted error parameter.

6. An image generation system suitable for multi-class and scarce sample data scenarios, characterized in that, The method comprises the following steps: a training sample acquisition module configured to obtain original sample data and pre-process to obtain image training samples; a generative adversarial network construction and initialization module configured to construct and initialize a generative adversarial network, which comprises an evolutionary population composed of multiple generator network individuals and a discriminator network; The training module is configured to preset hyperparameters, train the generative adversarial network using image training samples, determine an optimal generator network individual by evaluating the fitness of each generator network through a fitness function, generate candidate offspring according to the fitness using different mutation strategies, select next-generation individuals from the candidate offspring using an accept-reject method, and adjust error parameters of the discriminator network and update discriminator network parameters. The specific method of generating candidate offspring according to the fitness using different mutation strategies is that when the fitness of the generator network is greater than a minimum fitness threshold and less than a maximum fitness threshold, the network parameters of the generator network are controlled to learn towards the optimal generator network individual, learning in a knowledge distillation manner to generate first mutated offspring; and when the fitness of the generator network is less than the minimum fitness threshold, the network parameters of the generator network are controlled to explore towards a randomly selected individual to generate second mutated offspring. The target function gradient estimates of the first mutated offspring and the second mutated offspring are calculated, and the first mutated offspring and the second mutated offspring are updated to obtain candidate offspring. The method of selecting next-generation individuals is that if the fitness of the candidate offspring is higher than the fitness of the current generator network individual and exceeds a set threshold, the candidate offspring is accepted as a next-generation individual. Otherwise, the candidate offspring is rejected to enter the next generation, and the current generator network individual is retained as a next-generation individual. If all candidate offspring are rejected to enter the next generation, the number of rejections is increased by 1, otherwise the number of rejections remains unchanged, and when the number of rejections reaches a set maximum number of rejections, the training is ended. The image generation module is configured to obtain a generator network with the highest fitness when a maximum number of iterations is reached, and use the generator network to generate images.

7. A computer-readable storage medium having stored thereon a program, characterized in that, The program is executed by the processor to implement the steps of the image generation method for multi-class and rare sample data scenarios according to any one of claims 1-5.

8. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized by The processor executes the program to implement the steps of the image generation method for multi-class and rare sample data scenarios according to any one of claims 1-5.

Images