A sample picture generation method, apparatus and device

Abstract

This invention discloses a method, apparatus, and device for generating sample images. The method includes: selecting multiple reference images from a preset sample database based on an acquired target image, and obtaining model sample images based on a pre-built generator; inputting the target image, reference images, and model sample images into a discriminator corresponding to the generator, and training the discriminator based on a first discriminant loss, a second discriminant loss, and a third discriminant loss; inputting the model sample images into the trained discriminator, and training the generator based on the output results; saving model sample images whose similarity to the target image meets preset requirements; determining whether the number of saved model sample images reaches a preset threshold; if not, repeating the steps of obtaining model sample images, training the discriminator, and training the generator. The generated model sample images are influenced by both the target image and the reference images, enhancing the robustness of the model sample images.

Description

Technical Field

[0001] This invention relates to a method, apparatus, and device for generating sample images. Background Technology

[0002] In some scenarios, because real-world images are very small, it is often necessary to use technical means to generate images that closely resemble real-world images. For example, in the transportation field, obtaining traffic sign data for some uncommon traffic signs is extremely difficult. Conventional techniques typically use GAN neural networks to generate these hard-to-obtain images.

[0003] Generative Adversarial Networks (GANs), also known as Adversarial Networks, consist of a Generator Network and a Discriminator Network. The core logic of GANs is that the generator and discriminator compete and challenge each other. The Generator Network's task is to generate instances that appear natural, realistic, and similar to the original data. The Discriminator Network's task is to determine whether a given instance appears natural and realistic or is artificially created. During training, the generator aims to generate as many fake images as possible that closely resemble real images to deceive the discriminator, while the discriminator tries to distinguish the fake images generated by the generator from real images. Through alternating optimization training, both models improve, ultimately resulting in a highly efficient generative model. Theoretically, the final generated fake images can be indistinguishable from real images. Summary of the Invention

[0004] The inventors of this invention have discovered that existing GAN neural networks only use the discriminator twice during training, resulting in sample images that are biased towards the target or reference sample images. This fails to produce well-blended images, leading to a lack of robustness in the overall output. Therefore, the inventors propose this invention to provide a sample image generation method, apparatus, and device that overcomes or at least partially solves the aforementioned problems in the prior art.

[0005] In a first aspect, embodiments of the present invention provide a method for generating sample images, the method comprising:

[0006] Based on the acquired target image, multiple reference images are selected from a pre-set sample database, and model sample images are obtained based on a pre-built generator.

[0007] The target image, reference image, and model sample image are respectively input into the discriminator corresponding to the generator, and the discriminator is trained based on the obtained first discrimination loss, second discrimination loss, and third discrimination loss.

[0008] The model sample images are input into the trained discriminator, and the generator is trained based on the output results.

[0009] Save model sample images that meet preset requirements in terms of similarity to the target image;

[0010] Determine whether the number of all saved model sample images has reached a preset threshold. If not, repeat the steps above to obtain model sample images, train the discriminator, and the generator.

[0011] In one or more optional embodiments, the first discrimination loss, the second discrimination loss, and the third discrimination loss are obtained in the following manner:

[0012] The target image is input into the discriminator to obtain a first output matrix. A first discrimination loss is obtained based on the first output matrix and a preset first truth matrix. The first output matrix and the first truth matrix are both n*n matrices, where n is a positive integer. The values ​​of the elements representing the foreground region of the target image in the first truth matrix are greater than the values ​​of the elements representing the background region of the target image.

[0013] The reference image is input into the discriminator to obtain a second output matrix. A second discrimination loss is obtained based on the second output matrix and a preset second truth matrix. Both the second output matrix and the second truth matrix are n*n matrices. In the second truth matrix, the values ​​of elements representing the foreground region of the reference image are less than the values ​​of elements representing the background region of the reference image. Furthermore, the values ​​of elements representing the foreground region of the reference image in the second truth matrix are less than the values ​​of elements representing the foreground region of the target image in the first truth matrix, and the values ​​of elements representing the background region of the reference image in the second truth matrix are less than the values ​​of elements representing the background region of the target image in the first truth matrix.

[0014] The model sample image is input into the discriminator to obtain a third output matrix. Based on the third output matrix and a preset third truth matrix, a third discriminant loss is obtained. The third output matrix and the third truth matrix are both n*n matrices. The value of the element representing the foreground region of the model sample image in the third truth matrix is ​​equal to the value of the element representing the background region of the model sample image.

[0015] In one or more optional embodiments, the discriminator is trained based on the first discrimination loss, the second discrimination loss, and the third discrimination loss, including:

[0016] The discrimination loss result of the discriminator is calculated based on the first discrimination loss, the second discrimination loss, and the third discrimination loss;

[0017] The discriminator is trained based on the discrimination loss result to obtain the trained discriminator.

[0018] In one or more optional embodiments, the method further includes:

[0019] After obtaining the first discrimination loss, the second discrimination loss, and the third discrimination loss, the discriminator is trained by backpropagation.

[0020] In one or more optional embodiments, the step of inputting the model sample image into the trained discriminator and training the generator based on the obtained output includes:

[0021] The model sample image is input into the trained discriminator to obtain a fourth output matrix. Based on the fourth output matrix and a preset fourth ground truth matrix, a fourth discriminant loss is obtained. The generator is updated based on the fourth discriminant loss to obtain a trained generator. The fourth output matrix and the fourth ground truth matrix are both n*n matrices, and the values ​​of the elements representing the foreground region of the model sample image in the fourth ground truth matrix are equal to the values ​​of the elements representing the background region of the model sample image.

[0022] In one or more optional embodiments, the method further includes:

[0023] After obtaining the fourth discriminant loss, the generator is trained by backpropagation.

[0024] In one or more optional embodiments, the step of selecting multiple reference images from a preset sample database based on the acquired target image includes:

[0025] Obtain the target feature vector of the target image, compare the similarity of the target feature vector with the images in the sample database, and filter to obtain the multiple reference images.

[0026] In one or more optional embodiments, obtaining the target feature vector of the target image, comparing the target image with images in the sample database, and filtering to obtain the plurality of reference images includes:

[0027] Obtain the target feature vector of the target image, compare the target feature vector with the images in the sample database, and filter to obtain multiple reference candidate images;

[0028] The multiple reference images are randomly selected from the multiple candidate reference images.

[0029] In one or more optional embodiments, the model sample images are obtained based on a pre-built generator in the following manner:

[0030] The target image and each reference image are used to construct a model data pair. The reference image of the model data pair is input into the encoder of the generator for feature extraction to obtain the feature vector of the reference image.

[0031] The reference image feature vector and the target feature vector are input into the attention module of the generator for calculation to obtain the attention feature vector;

[0032] The attention feature vector is input into the decoder of the generator for processing to obtain the model sample image.

[0033] In one or more optional embodiments, when repeating the steps of obtaining model sample images, training the discriminator and the generator as described above, multiple reference images are randomly selected again from the multiple reference candidate images.

[0034] In one or more optional embodiments, the target feature vector of the target image is determined in the following manner:

[0035] The target image is input into a pre-trained feature extraction model to obtain the target feature vector.

[0036] In one or more optional embodiments, saving model sample images that meet preset similarity requirements with the target image includes:

[0037] Determine whether the similarity between the model sample image and the target image reaches a preset threshold;

[0038] If so, save the model sample image.

[0039] Secondly, embodiments of the present invention provide a sample image generation apparatus, the apparatus comprising:

[0040] The generation module is used to select multiple reference images from a pre-set sample database based on the acquired target image, and obtain model sample images based on a pre-built generator;

[0041] The discriminator training module is used to input the target image, reference image and model sample image into the discriminator corresponding to the generator, and train the discriminator based on the obtained first discrimination loss, second discrimination loss and third discrimination loss;

[0042] The generator training module is used to input the model sample images into the trained discriminator and train the generator based on the obtained output results.

[0043] The saving module is used to save model sample images that have a similarity to the target image that meets preset requirements;

[0044] The judgment module is used to determine whether the number of all saved model sample images has reached a preset threshold.

[0045] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the sample image generation method as described above.

[0046] Fourthly, embodiments of the present invention provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the sample image generation method as described above.

[0047] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following:

[0048] The sample image generation method provided by this invention adds two discriminator operations during the GAN neural network training process by simultaneously using target images, reference images, and model sample images. This ensures that the generated model sample images are influenced by both the target and reference images, enhancing the mixing effect of the model sample images, improving the quality of the generated images, and increasing the robustness of the model sample images generated by the GAN neural network. Furthermore, compared to other label-based semantic training images in existing technologies, the model sample images obtained by this method are semantically neutral, thus allowing for reuse in different application scenarios and providing greater flexibility.

[0049] The sample image generation method provided by this invention employs a ground truth matrix with masking properties during the three training discriminator steps to determine the discrimination loss. By setting the feature values ​​of the background and foreground regions of the ground truth matrices corresponding to the target image, reference image, and model sample image, respectively, the method controls the region of the generated image, including the background and foreground regions, which are influenced by the target image and reference image, respectively. This allows for better control over the quality of the generated model sample images.

[0050] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.

[0051] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0052] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0053] Figure 1 This is a schematic flowchart of the sample image generation method provided in an embodiment of the present invention;

[0054] Figure 2 A schematic diagram of a generator model for a GAN neural network provided in an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of another generator model for the GAN neural network provided in an embodiment of the present invention;

[0056] Figure 4 A schematic diagram of a discriminator model for a GAN neural network provided in an embodiment of the present invention;

[0057] Figure 5 This is a schematic diagram of the sample image generation device provided in an embodiment of the present invention. Detailed Implementation

[0058] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0059] Example 1

[0060] This invention provides a method for generating sample images, referring to... Figure 1 As shown, the method includes:

[0061] S101: Based on the acquired target image, select multiple reference images from the preset sample database, and obtain model sample images based on the pre-built generator;

[0062] S102: Input the target image, reference image and model sample image into the discriminator corresponding to the generator, and train the discriminator based on the obtained first discrimination loss, second discrimination loss and third discrimination loss;

[0063] S103: Input the model sample image into the trained discriminator, and train the generator based on the output result;

[0064] S104: Save model sample images that meet the preset requirements in terms of similarity to the target image;

[0065] S105: Determine whether the number of all saved model sample images has reached the preset threshold. If not, repeat steps S101 to S105 above. If yes, end.

[0066] The sample image generation method provided by this invention adds two discriminator operations during the GAN neural network training process by simultaneously using target images, reference images, and model sample images. This ensures that the generated model sample images are influenced by both the target and reference images, enhancing the mixing effect of the model sample images, improving the quality of the generated images, and increasing the robustness of the model sample images generated by the GAN neural network. Furthermore, compared to other label-based semantic training images in existing technologies, the model sample images obtained by this method are semantically neutral, thus allowing for reuse in different application scenarios and providing greater flexibility.

[0067] In this embodiment of the invention, the step S101 described above, which involves selecting multiple reference images from a preset sample database based on the acquired target image and obtaining model sample images based on a pre-built generator, specifically includes:

[0068] Based on the acquired target image, multiple reference images are selected from a pre-set sample database;

[0069] For the model data pairs constructed for the target image and each reference image, the model data pairs are input into a pre-built generator to obtain model sample images.

[0070] In one specific embodiment, the above-mentioned selection of multiple reference images from a preset sample database based on the acquired target image specifically includes:

[0071] Obtain the target feature vector of the target image, compare the similarity of the target feature vector with the images in the sample database, and filter to obtain the multiple reference images.

[0072] Specifically, it can be done by obtaining the target feature vector of the target image, comparing the similarity of the target feature vector with the images in the sample database, and filtering out multiple reference candidate images;

[0073] The multiple reference images are randomly selected from the multiple candidate reference images.

[0074] For example, assuming the target feature vector of the target image A is 'a', this target feature vector 'a' is used to retrieve the Top N candidate images from a pre-defined sample database that are close in similarity, have large differences in similarity, or are randomly selected. These Top N candidate images are then used as reference candidate images. Finally, a pre-set number of images are randomly selected from these Top N candidate images as reference images. A model data pair is constructed between the target image and each selected reference image. For example, if the pre-set number is three images, three reference images can be randomly selected, such as reference images B, C, and D. The target feature vector 'a' of the target image is then used to construct a model data pair with each reference image, resulting in the generated model data pairs Ba, Ca, and Da.

[0075] In this embodiment of the invention, the preset sample database is the image database of reference images. By repeatedly executing the above steps of randomly selecting a preset number of images from these Top N candidate images as reference images, and constructing model data pairs with the reference images and target feature vectors, a training batch of model data pairs can be generated.

[0076] In a specific example, the target feature vector of the target image is obtained in the following way:

[0077] The target image is input into a pre-trained feature extraction model to obtain the target feature vector.

[0078] In this embodiment of the invention, the pre-trained feature extraction model can use a network structure such as ResNet12 or ConvNet4 to obtain the target feature vector.

[0079] In this embodiment of the invention, when comparing the similarity between the target feature vector and the images in the sample database, the feature vector of the reference image can be obtained first, and the feature vector comparison method can be used. Of course, other methods for image similarity comparison described in the prior art can also be used. The specific implementation process can be referred to in the detailed description of the prior art, and will not be repeated here.

[0080] In one specific embodiment, reference is made to Figure 2As shown, the pre-built generator includes an encoder, an attention module, and a decoder. The process of inputting the model data into the pre-built generator to obtain model sample images specifically includes:

[0081] The reference image of the model data pair is input into the encoder of the generator for feature extraction to obtain the feature vector of the reference image;

[0082] The reference image feature vector and the target feature vector are input into the attention module of the generator to perform attention calculation, thereby obtaining the attention feature vector;

[0083] The attention feature vector is input into the decoder of the generator for processing, thereby obtaining the model sample image.

[0084] In this embodiment of the invention, the encoder and decoder of the generator can be implemented in accordance with the generation method of the encoder and decoder of the generator in the prior art. The attention module can adopt the residual attention module in the prior art. The specific implementation process will not be described in detail here.

[0085] As a specific implementation of an embodiment of the present invention, refer to Figure 3 As shown, the generator may also include a random vector module, which generates random vectors, merges (Concat) the random vectors with the attention feature vectors to obtain concatenated feature vectors, and then inputs the concatenated feature vectors into the decoder of the generator for processing to obtain model sample images.

[0086] In this embodiment of the invention, the inventors considered that merging the reference image feature vector and the target feature vector to obtain a merged feature vector, and then performing attention calculation on the merged feature vector, would result in a stronger image feature vector and a weaker image feature vector, with a greater relative difference. Based on this, the inventors proposed to generate a random vector through a random vector module and concatenate it with the attention feature vector, thereby increasing random noise and increasing the randomness of the generated model sample images, so as to better train the discriminator of the GAN neural network.

[0087] In this embodiment of the invention, the step S102 described above, which involves inputting the target image, the reference image, and the model sample image into the discriminator corresponding to the generator, and training the discriminator based on the obtained first discrimination loss, second discrimination loss, and third discrimination loss, specifically includes:

[0088] The target image, reference image, and model sample image are respectively input into the discriminator corresponding to the generator to obtain the corresponding first discrimination loss, second discrimination loss, and third discrimination loss;

[0089] The discriminator is trained based on the first discrimination loss, the second discrimination loss, and the third discrimination loss.

[0090] In one specific embodiment, the above-mentioned inputting the target image, reference image, and model sample image into the discriminator corresponding to the generator to obtain a first discrimination loss, a second discrimination loss, and a third discrimination loss specifically includes:

[0091] The target image is input into the discriminator to obtain a first output matrix. Based on the first output matrix and a preset first truth matrix, a first discrimination loss is obtained.

[0092] The reference image is input into the discriminator to obtain a second output matrix. Based on the second output matrix and a preset second truth matrix, a second discriminant loss is obtained.

[0093] The model sample image is input into the discriminator to obtain the third output matrix. The third discriminant loss is obtained based on the third output matrix and the preset third ground truth matrix.

[0094] In this embodiment of the invention, inputting the target image into the discriminator means inputting the target feature vector of the target image into the discriminator to obtain the first output matrix and calculate the first discriminant loss.

[0095] In this embodiment of the invention, reference is made to Figure 4 As shown, when the discrimination images—namely, the target image, the reference image, and the model sample image—are input into the discriminator, the discriminator's discrimination network will output corresponding discrimination result matrices, namely, the first output matrix, the second output matrix, and the third output matrix. It should be noted that, in this embodiment of the invention, the discriminator's discrimination network differs from discriminators in conventional techniques in that the output of the last convolutional layer of this discriminator is an n*n matrix with n dimensions and 1 channel, rather than a feature vector with 1 dimension and 1 channel.

[0096] In one specific embodiment, assuming n is 4, the first truth matrix, the second truth matrix, and the third truth matrix can be set as 4*4 matrices.

[0097] In this embodiment of the invention, the eigenvalues ​​of the elements representing the foreground region and the eigenvalues ​​of the elements representing the background region in each truth matrix can be set according to actual needs, but the following conditions should be met:

[0098] In this first truth matrix, the values ​​of the elements representing the foreground region of the target image are greater than the values ​​of the elements representing the background region of the target image;

[0099] In the second truth matrix, the values ​​of the elements representing the foreground region of the reference image are less than the values ​​of the elements representing the background region of the reference image; and the values ​​of the elements representing the foreground region of the reference image in the second truth matrix are less than the values ​​of the elements representing the foreground region of the target image in the first truth matrix, and the values ​​of the elements representing the background region of the reference image in the second truth matrix are less than the values ​​of the elements representing the background region of the target image in the first truth matrix;

[0100] In the third truth matrix, the values ​​of the elements representing the foreground region of the model sample image are equal to the values ​​of the elements representing the background region of the model sample image.

[0101] For example, if the eigenvalues ​​of the elements representing the foreground region in the first truth matrix GT1 are 1, and the eigenvalues ​​of the elements representing the background region are 0.5, then the first truth matrix GT1 is:

[0102] GT1 = [[0.5, 0.5, 0.5, 0.5],

[0103] [0.5, 1.0, 1.0, 0.5],

[0104] [0.5, 1.0, 1.0, 0.5],

[0105] [0.5, 0.5, 0.5, 0.5]];

[0106] The second truth matrix GT2 has an eigenvalue of 0.1 for the elements representing the foreground region and an eigenvalue of 0.2 for the elements representing the background region. Therefore, the second truth matrix GT2 is:

[0107] GT2=[[0.2, 0.2, 0.2, 0.2],

[0108] [0.2, 0.1, 0.1, 0.2],

[0109] [0.2, 0.1, 0.1, 0.2],

[0110] [0.2, 0.2, 0.2, 0.2]];

[0111] If the eigenvalues ​​of the elements representing the foreground region and the eigenvalues ​​of the elements representing the background region in the third truth matrix GT3 are both 0, then the third truth matrix GT3 is:

[0112] GT3=[[0.0, 0.0, 0.0, 0.0],

[0113] [0.0, 0.0, 0.0, 0.0],

[0114] [0.0, 0.0, 0.0, 0.0],

[0115] [0.0, 0.0, 0.0, 0.0]].

[0116] In one specific embodiment, training the discriminator based on the first discrimination loss, the second discrimination loss, and the third discrimination loss specifically includes:

[0117] The discrimination loss result of the discriminator is calculated based on the first discrimination loss, the second discrimination loss, and the third discrimination loss;

[0118] The discriminator is trained based on the discrimination loss result to obtain the trained discriminator.

[0119] In one specific embodiment, the calculation of the discrimination loss result based on the first discrimination loss, the second discrimination loss, and the third discrimination loss specifically includes:

[0120] Substituting the first discrimination loss, the second discrimination loss, and the third discrimination loss into the following formula 1, the discrimination loss result is obtained:

[0121] LossD=(1-α)*Loss1+α * Loss2+Loss3 Formula 1;

[0122] In the formula, LossD represents the discrimination loss result, Loss1 represents the first discrimination loss, Loss2 represents the second discrimination loss, Loss3 represents the third discrimination loss, α is a hyperparameter, and 0 < α < 1.

[0123] The sample image generation method provided by this invention employs a ground truth matrix with masking properties during the three training discriminator steps to determine the discrimination loss. By setting the feature values ​​of the background and foreground regions of the ground truth matrices corresponding to the target image, reference image, and model sample image, respectively, the method controls the region of the generated image, including the background and foreground regions, which are influenced by the target image and reference image, respectively. This allows for better control over the quality of the generated model sample images.

[0124] In one specific embodiment, the above method further includes: after obtaining the first discrimination loss, the second discrimination loss, and the third discrimination loss, performing backpropagation training on the discriminator respectively. Based on this, as a specific implementation of the present invention, the specific process of training the discriminator based on the above step S102 may include:

[0125] Fix the generator and reset the discriminator gradient;

[0126] The target image A is input into the discriminator to obtain the first output matrix. The first discrimination loss Loss1 is calculated using the corresponding first ground truth matrix GT1. Since the target image is a strong ground truth image, the first output matrix obtained by the discriminator at this time is the probability matrix of being classified as a strong ground truth image.

[0127] Perform the first backpropagation on the discriminator;

[0128] The reference image is input into the discriminator to obtain the second output matrix. The second discrimination loss Loss2 is calculated using the corresponding second ground truth matrix GT2. Since the reference image is a weak ground truth image that is close to the real target image, the second output matrix obtained by the discriminator at this time is the probability matrix of being classified as a weak ground truth image.

[0129] Perform a second backpropagation on the discriminator;

[0130] The model sample image, i.e. the generated image, is input into the discriminator to obtain the third output matrix. The third discriminant loss Loss is calculated using the corresponding third ground truth matrix GT3. Since the model sample image is a generated image, the third output matrix obtained by the discriminator at this time is the probability matrix of being classified as false.

[0131] Perform a third backpropagation on the discriminator;

[0132] Finally, the discriminant loss result is obtained through Formula 1 above. The discriminant parameters are updated based on the discriminant loss result to obtain the trained discriminant.

[0133] In this embodiment of the invention, the step S103 described above, which involves inputting the model sample image into the trained discriminator and training the generator based on the obtained output, includes:

[0134] The model sample image is input into the trained discriminator to obtain a fourth output matrix. A fourth discriminant loss is obtained based on the fourth output matrix and a preset fourth ground truth matrix. The generator is then updated based on the fourth discriminant loss to obtain a trained generator. Both the fourth output matrix and the fourth ground truth matrix are n*n matrices, and the values ​​of the elements representing the foreground region of the model sample image in the fourth ground truth matrix are equal to the values ​​of the elements representing the background region of the model sample image.

[0135] In this embodiment of the invention, during the generator training process, the trained discriminator should be fixed. This trained discriminator performs the process of determining the discrimination result as true. Therefore, the values ​​of the elements representing the foreground region of the model sample image and the elements representing the background region of the model sample image in the fourth truth matrix can both be 1. Thus, the fourth truth matrix GT_G is:

[0136] GT_G=[[1.0, 1.0, 1.0, 1.0],

[0137] [1.0, 1.0, 1.0, 1.0],

[0138] [1.0, 1.0, 1.0, 1.0],

[0139] [1.0, 1.0, 1.0, 1.0]].

[0140] As can be seen from the training process of the discriminator and generator described above, the discriminator is used three times during its own training process and once during the generator training process. That is, in the entire training process of a GAN network model, the discriminator is used four times and the generator is used once. Compared with the existing technology that only uses the discriminator twice and the generator once, this makes the generated model sample images simultaneously affected by the target image and the reference image, enhances the mixing effect of the model sample images, improves the quality of the generated images, and improves the robustness of the model sample images generated by the GAN neural network.

[0141] In one specific embodiment, the above method further includes: after obtaining the fourth discriminant loss, performing backpropagation training on the generator. Based on this, as a specific implementation of the present invention, the specific process of training the generator based on the above step S103 may include:

[0142] Fix the discriminator and train the generator and discriminator together;

[0143] Reset the generator gradient;

[0144] The model sample image is input into the discriminator to obtain the fourth output matrix. The fourth discriminant loss LossG is calculated using the corresponding fourth ground truth matrix GT_G. Since it is desired that the discriminator classifies the generated image as true, the fourth output matrix obtained by the discriminator at this time is the probability matrix of classifying as true.

[0145] Perform backpropagation on the generator;

[0146] The generator parameters are updated based on the fourth discriminant loss LossG to obtain the trained generator.

[0147] In this embodiment of the invention, saving model sample images that meet preset requirements in similarity to the target image, as described in step S104 above, specifically includes:

[0148] Determine whether the similarity between the model sample image and the target image reaches a preset threshold;

[0149] If so, save the model sample image;

[0150] If not, then discard the model sample image.

[0151] In this embodiment of the invention, the similarity between the model sample image and the target image can be determined by judging the Frechet Inception Distance (FID) value. Other feature vector or hash table comparison methods can also be used. Of course, other image similarity comparison methods described in the prior art can also be used; the specific implementation process can be found in the detailed description of the prior art, and will not be repeated here.

[0152] In this embodiment of the invention, when it is determined according to step S105 that the number of all saved model sample images has not reached the preset threshold and the steps of obtaining model sample images, training discriminator and generator need to be repeated, multiple reference images can be randomly selected from the multiple reference candidate images.

[0153] Example 2

[0154] Based on the same inventive concept, embodiments of the present invention also provide a sample image generation device, referring to... Figure 5 As shown, the device includes:

[0155] The generation module 101 is used to select multiple reference images from a preset sample database based on the acquired target image, and obtain model sample images based on a pre-built generator.

[0156] The discriminator training module 102 is used to input the target image, reference image and model sample image into the discriminator corresponding to the generator, and train the discriminator based on the obtained first discrimination loss, second discrimination loss and third discrimination loss;

[0157] The generator training module 103 is used to input the model sample images into the trained discriminator and train the generator based on the obtained output results.

[0158] The storage module 104 is used to store model sample images that have a similarity to the target image that meets preset requirements;

[0159] The judgment module 105 is used to determine whether the number of all saved model sample images has reached a preset threshold.

[0160] Example 3

[0161] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the sample image generation method as described in Embodiment 1 above.

[0162] Example 4

[0163] Based on the same inventive concept, this embodiment of the invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the sample image generation method as described in Embodiment 1 above.

[0164] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0165] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0166] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0167] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0168] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for generating sample images, characterized in that, include: Based on the acquired target image, multiple reference images are selected from a pre-set sample database, and model sample images are obtained based on a pre-built generator. The target image, reference image, and model sample image are respectively input into the discriminator corresponding to the generator, and the discriminator is trained based on the obtained first discrimination loss, second discrimination loss, and third discrimination loss. The model sample images are input into the trained discriminator, and the generator is trained based on the output results. Save model sample images that meet preset requirements in terms of similarity to the target image; Determine whether the number of all saved model sample images has reached a preset threshold. If not, repeat the above steps of obtaining model sample images, training the discriminator and generator. The first discriminant loss, the second discriminant loss, and the third discriminant loss are obtained in the following ways: The target image is input into the discriminator to obtain a first output matrix. Based on the first output matrix and a preset first truth matrix, a first discrimination loss is obtained. The reference image is input into the discriminator to obtain a second output matrix. Based on the second output matrix and a preset second truth matrix, a second discriminant loss is obtained. The model sample image is input into the discriminator to obtain the third output matrix. The third discriminant loss is obtained based on the third output matrix and the preset third ground truth matrix. The discriminator is trained based on the first discrimination loss, the second discrimination loss, and the third discrimination loss, including: The discrimination loss result of the discriminator is calculated based on the first discrimination loss, the second discrimination loss, and the third discrimination loss; The discriminator is trained based on the discrimination loss result to obtain the trained discriminator; The step of inputting the model sample images into the trained discriminator and training the generator based on the obtained output includes: The model sample image is input into the trained discriminator to obtain the fourth output matrix. The fourth discriminant loss is obtained based on the fourth output matrix and the preset fourth ground truth matrix. The generator is updated based on the fourth discriminant loss to obtain the trained generator.

2. The method as described in claim 1, characterized in that, Based on the acquired target image, multiple reference images are selected from a pre-defined sample database, including: Obtain the target feature vector of the target image, compare the similarity of the target feature vector with the images in the sample database, and filter to obtain the multiple reference images.

3. The method as described in claim 2, characterized in that, The process of obtaining the target feature vector of the target image, comparing the target image with images in the sample database, and filtering to obtain the plurality of reference images includes: Obtain the target feature vector of the target image, compare the target feature vector with the images in the sample database, and filter to obtain multiple reference candidate images; The multiple reference images are randomly selected from the multiple candidate reference images.

4. The method as described in claim 2, characterized in that, Based on a pre-built generator, the model sample images are obtained in the following ways: The target image and each reference image are used to construct a model data pair. The reference image of the model data pair is input into the encoder of the generator for feature extraction to obtain the feature vector of the reference image. The reference image feature vector and the target feature vector are input into the attention module of the generator for calculation to obtain the attention feature vector; The attention feature vector is input into the decoder of the generator for processing to obtain the model sample image.

5. The method as described in claim 3, characterized in that, When repeating the steps of obtaining model sample images, training the discriminator and the generator, multiple reference images are randomly selected from the multiple reference candidate images.

6. The method as described in claim 1, characterized in that, The step of saving model sample images whose similarity to the target image meets preset requirements includes: Determine whether the similarity between the model sample image and the target image reaches a preset threshold; If so, save the model sample image.

7. A sample image generation device, characterized in that, include: The generation module is used to select multiple reference images from a pre-set sample database based on the acquired target image, and obtain model sample images based on a pre-built generator; The discriminator training module is used to input the target image, reference image and model sample image into the discriminator corresponding to the generator, and train the discriminator based on the obtained first discrimination loss, second discrimination loss and third discrimination loss; The first discriminant loss, the second discriminant loss, and the third discriminant loss are obtained in the following ways: The target image is input into the discriminator to obtain a first output matrix. Based on the first output matrix and a preset first truth matrix, a first discrimination loss is obtained. The reference image is input into the discriminator to obtain a second output matrix. Based on the second output matrix and a preset second truth matrix, a second discriminant loss is obtained. The model sample image is input into the discriminator to obtain the third output matrix. The third discriminant loss is obtained based on the third output matrix and the preset third ground truth matrix. The discriminator is trained based on the first discrimination loss, the second discrimination loss, and the third discrimination loss, including: The discrimination loss result of the discriminator is calculated based on the first discrimination loss, the second discrimination loss, and the third discrimination loss; The discriminator is trained based on the discrimination loss result to obtain the trained discriminator; The generator training module is used to input the model sample images into the trained discriminator and train the generator based on the output results, including: The model sample image is input into the trained discriminator to obtain the fourth output matrix. The fourth discriminant loss is obtained based on the fourth output matrix and the preset fourth ground truth matrix. The generator is updated based on the fourth discriminant loss to obtain the trained generator. The saving module is used to save model sample images that have a similarity to the target image that meets preset requirements; The judgment module is used to determine whether the number of all saved model sample images has reached a preset threshold.

8. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the sample image generation method as described in any one of claims 1-6.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the sample image generation method as described in any one of claims 1-6.

Images