Data filling method, system, device and medium based on generative adversarial network
By employing a generative adversarial network (GAN) training method and utilizing a dual-output discriminator to calculate image similarity and realism, the generator is optimized. This addresses the issues of long training time and low image quality in GANs, achieving efficient data filling and high-quality image generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHANGSHA UNIVERSITY OF SCIENCE AND TECHNOLOGY
- Filing Date
- 2023-03-30
- Publication Date
- 2026-07-21
AI Technical Summary
Existing generative adversarial networks suffer from long training times and low image quality during data filling.
A generative adversarial network is adopted. By acquiring training image samples, a first real image, and a second real image, a discriminator with dual outputs is trained to calculate image similarity and authenticity, and optimize the generator to improve image generation quality and reduce training time.
It effectively reduces data filling time, improves image generation quality, and enhances the efficiency and effectiveness of data filling.
Smart Images
Figure CN116385594B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image generation technology, and in particular to a data filling method, system, device and medium based on generative adversarial networks. Background Technology
[0002] Generative Adversarial Networks (GANs) are commonly used for data imputation. Existing techniques can use well-trained GANs to fit the distribution of real data, generating a large number of artificial images to compensate for the lack of data in deep learning applications. However, the training process is essentially the process of the generator capturing the distribution domain of real data, so the training time is often very long. Even with long training times, the generator may still not be able to fit the distribution domain of real data well. In addition, the learning ability of GAN generators is limited, resulting in poor image quality. Therefore, using existing techniques for data imputation not only takes more time but also produces low-quality imputation. Summary of the Invention
[0003] This invention aims to address at least one of the technical problems existing in the prior art. To this end, this invention proposes a data imputation method, system, device, and medium based on generative adversarial networks, which can improve the quality of image generation while reducing model training time, thereby reducing data imputation time and improving data imputation quality.
[0004] In a first aspect, embodiments of the present invention provide a data imputation method based on generative adversarial networks, the data imputation method based on generative adversarial networks comprising:
[0005] Get the image to be filled;
[0006] The image to be filled is input into a preset generative adversarial network (GAN) for image filling, resulting in the filled image output by the GAN; wherein, the GAN includes a generator and a dual-output discriminator, and the GAN is trained in the following manner:
[0007] Obtain training image samples, the first real image, and the second real image;
[0008] The generator is pre-trained by inputting training image samples to obtain the first image output by the generator.
[0009] The first image, the first real image, and the second real image are input into the discriminator for training. The discriminator calculates the first image and the first real image samples, as well as the similarity between the first real image and the second real image, and the authenticity of the first image and the second real image as determined by the discriminator's second branch. The training of the discriminator is then completed based on the similarity and the authenticity.
[0010] The generator is trained by combining the trained discriminator with the training image samples to obtain the trained generator.
[0011] Compared with the prior art, the first aspect of the present invention has the following beneficial effects:
[0012] This method acquires an image to be filled and inputs it into a pre-defined generative adversarial network (GAN) for image filling, obtaining the filled image output by the GAN. Using the pre-defined GAN for image filling reduces data filling time and improves data filling quality. The GAN comprises a generator and a dual-output discriminator. The GAN is trained as follows: training image samples, a first real image, and a second real image are acquired; the training image samples are input into the generator for pre-training to obtain the first image output by the generator; the first image, the first real image, and the second real image are input into the discriminator for training to obtain the discriminator. The first branch of the discriminator calculates the similarity between the first image and the first real image sample, as well as the similarity between the first real image and the second real image, and the authenticity of the first image and the second real image as determined by the second branch of the discriminator. Based on the similarity and authenticity, the discriminator is trained. The trained discriminator can both distinguish between real and fake images and calculate the similarity between images. By using a single discriminator to simultaneously distinguish between real and fake images and calculate image similarity, the training time of the generative adversarial network can be reduced. The generator is then trained using the trained discriminator and training image samples to obtain a trained generator. Training the generator using the trained discriminator can improve the quality of image generation.
[0013] According to some embodiments of the present invention, the step of inputting the first image, the first real image, and the second real image into the discriminator for training, obtaining the first image and the first real image samples calculated by the first branch of the discriminator, the similarity between the first real image and the second real image, and the authenticity of the first image and the second real image as determined by the second branch of the discriminator, and completing the training of the discriminator based on the similarity and the authenticity, includes:
[0014] The discriminator is trained for the current round, wherein the current round training includes: inputting the first real image and the second real image into the discriminator for training, calculating the similarity between the first real image and the second real image through the first branch of the discriminator, and determining the authenticity of the second real image through the second branch of the discriminator; simultaneously inputting the first real image and the first image into the discriminator for training, calculating the similarity between the first real image and the first image through the first branch of the discriminator, and determining the authenticity of the first image through the second branch of the discriminator.
[0015] If the authenticity and similarity meet the preset requirements, the training of the discriminator is completed; if the authenticity and / or similarity do not meet the preset requirements, the next round of training of the discriminator is started until the authenticity and similarity output by the discriminator in the next round meet the preset requirements.
[0016] According to some embodiments of the present invention, the loss function of the generative adversarial network is constructed in the following manner:
[0017]
[0018] Wherein, G represents the generator, D represents the discriminator, x1 represents the first real image selected from the real image samples, x2 represents the second real image selected from the real image samples, x represents selecting the first real image or the second real image, and G(z) represents the first image of the generator.
[0019] According to some embodiments of the present invention, training the generator based on the trained discriminator combined with the training image samples to obtain the trained generator includes:
[0020] The generator is trained by the discriminator that has completed training, so that the generator minimizes log(1-D(G(z),x)) when training the training image samples, and the trained generator is obtained.
[0021] According to some embodiments of the present invention, the discriminator uses a Siamese neural network to calculate the similarity between two images.
[0022] According to some embodiments of the present invention, the calculation of the similarity between two images using a Siamese neural network includes:
[0023] The two images are convolved through a convolutional layer to obtain two convolved images;
[0024] The two convolutional images are connected using a fully connected layer to extract image features;
[0025] The distance between two images is calculated by extracting image features;
[0026] The distance between the two images is calculated and passed through a fully connected layer, and the similarity between the two images is calculated using the sigmoid activation function.
[0027] According to some embodiments of the present invention, the objective function of the Siamese neural network includes:
[0028]
[0029] Where N represents the number of samples, X1 and X2 represent the first real image and the second real image or the first real image and the first image, Y = 1 indicates that the first real image and the second real image are similar or the first real image and the first image are similar, Y = 0 indicates that the first real image and the first image are not similar, and E w The distance between the first real image and the second real image or the distance between the first real image and the first image is represented by m, where m represents a preset threshold.
[0030] Secondly, embodiments of the present invention also provide a data padding system based on generative adversarial networks, the data padding system based on generative adversarial networks comprising:
[0031] The data acquisition module is used to acquire the image to be filled.
[0032] A data infilling module is used to input the image to be infilled into a preset generative adversarial network (GAN) for image infilling, obtaining the infilled image output by the GAN; wherein, the GAN includes a generator and a dual-output discriminator, and the data infilling module includes a data acquisition submodule, a pre-training submodule, a first training submodule, and a second training submodule, wherein:
[0033] The data acquisition submodule is used to acquire training image samples, a first real image, and a second real image;
[0034] The pre-training submodule is used to input training image samples into the generator for pre-training to obtain the first image output by the generator;
[0035] The first training submodule is used to input the first image, the first real image, and the second real image into the discriminator for training, to obtain the first image and the first real image samples calculated by the first branch of the discriminator, the similarity between the first real image and the second real image, and the authenticity of the first image and the second real image as determined by the second branch of the discriminator, and to complete the training of the discriminator based on the similarity and the authenticity.
[0036] The second training submodule is used to train the generator based on the discriminator that has completed training and the training image samples, so as to obtain the generator that has completed training.
[0037] Thirdly, embodiments of the present invention also provide a data filling device based on a generative adversarial network, including at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, the instructions being executed by the at least one control processor to enable the at least one control processor to perform a data filling method based on a generative adversarial network as described above.
[0038] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions for causing a computer to execute a data filling method based on a generative adversarial network as described above.
[0039] It is understood that the beneficial effects of the second to fourth aspects compared with the related technologies are the same as the beneficial effects of the first aspect compared with the related technologies. Please refer to the relevant description in the first aspect above, which will not be repeated here. Attached Figure Description
[0040] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:
[0041] Figure 1 This is a flowchart of a data imputation method based on generative adversarial networks according to an embodiment of the present invention;
[0042] Figure 2 This is a flowchart of step S200 in an embodiment of the present invention;
[0043] Figure 3 This is a network structure diagram of a generative adversarial network according to an embodiment of the present invention;
[0044] Figure 4This is a structural diagram of a data filling system based on a generative adversarial network according to an embodiment of the present invention;
[0045] Figure 5 This is a structural diagram of a submodule in a data filling module according to an embodiment of the present invention. Detailed Implementation
[0046] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0047] In the description of this invention, the use of terms such as "first," "second," etc., is for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance, or implicitly indicating the number of technical features indicated, or implicitly indicating the order of the technical features indicated.
[0048] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the drawings and are only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.
[0049] In the description of this invention, it should be noted that, unless otherwise explicitly defined, terms such as "setting," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.
[0050] Generative Adversarial Networks (GANs) are commonly used for data imputation. Existing techniques can use well-trained GANs to fit the distribution of real data, generating a large number of artificial images to compensate for the lack of data in deep learning applications. However, the training process is essentially the process of the generator capturing the distribution domain of real data, so the training time is often very long. Even with long training times, the generator may still not be able to fit the distribution domain of real data well. In addition, the learning ability of GAN generators is limited, resulting in poor image quality. Therefore, using existing techniques for data imputation not only takes more time but also produces low-quality imputation.
[0051] To address the aforementioned problems, this invention acquires an image to be filled and inputs it into a pre-defined generative adversarial network (GAN) for image filling, obtaining the filled image output by the GAN. Using the pre-defined GAN for image filling reduces data filling time and improves data filling quality. The GAN comprises a generator and a dual-output discriminator. The GAN is trained as follows: training image samples, a first real image, and a second real image are acquired; the training image samples are input into the generator for pre-training to obtain the first image output by the generator; the first image, the first real image, and the second real image are input into the discriminator for training. The discriminator obtains the first image and the first real image sample calculated by the first branch, the similarity between the first real image and the second real image, and the authenticity of the first image and the second real image as determined by the second branch of the discriminator. Based on the similarity and authenticity, the discriminator is trained. The trained discriminator can both distinguish between real and fake images and calculate the similarity between images. By using a single discriminator to simultaneously distinguish between real and fake images and calculate image similarity, the training time of the generative adversarial network can be reduced. The generator is then trained using the trained discriminator and training image samples to obtain the trained generator. Training the generator using the trained discriminator can improve the quality of image generation.
[0052] Reference Figures 1 to 2 This invention provides a data imputation method based on generative adversarial networks. This data imputation method includes, but is not limited to, steps S100 to S200 and steps S210 to S240, wherein:
[0053] Step S100: Obtain the image to be filled;
[0054] Step S200: Input the image to be filled into a preset generative adversarial network (GAN) for image filling, and obtain the filled image output by the GAN; wherein, the GAN includes a generator and a dual-output discriminator, and the GAN is trained in the following manner:
[0055] Step S210: Obtain training image samples, the first real image, and the second real image;
[0056] Step S220: Input the training image samples into the generator for pre-training to obtain the first image output by the generator;
[0057] Step S230: Input the first image, the first real image, and the second real image into the discriminator for training, and obtain the first image and the first real image samples calculated by the first branch of the discriminator, the similarity between the first real image and the second real image, and the authenticity of the first image and the second real image judged by the second branch of the discriminator. Then, complete the training of the discriminator based on the similarity and authenticity.
[0058] Step S240: Train the generator based on the trained discriminator and training image samples to obtain the trained generator.
[0059] In steps S100 to S200 and S210 to S240 of some embodiments, in order to reduce data filling time and improve data filling quality, this embodiment acquires the image to be filled and inputs it into a preset generative adversarial network for image filling to obtain the filled image output by the generative adversarial network. In order to reduce the training time of the generative adversarial network, this embodiment acquires training image samples, a first real image, and a second real image, inputs the training image samples into the generator for pre-training, and obtains the first image output by the generator. The first image, the first real image, and the second real image are input into the discriminator for training to obtain the first image and the first real image samples calculated by the first branch of the discriminator, the similarity between the first real image and the second real image, and the authenticity of the first image and the second real image judged by the second branch of the discriminator. The training of the discriminator is completed based on the similarity and authenticity. In order to improve the quality of image generation, this embodiment trains the generator based on the trained discriminator and the training image samples to obtain the trained generator.
[0060] In some embodiments, a first image, a first real image, and a second real image are input into a discriminator for training. This yields the first image and first real image samples calculated by the first branch of the discriminator, the similarity between the first real image and the second real image, and the authenticity of the first image and the second real image as determined by the second branch of the discriminator. The training of the discriminator is then completed based on the similarity and authenticity, including:
[0061] The discriminator is trained in the current round, wherein the current round training includes: inputting a first real image and a second real image into the discriminator for training, calculating the similarity between the first real image and the second real image through the first branch of the discriminator, and judging the authenticity of the second real image through the second branch of the discriminator; simultaneously inputting a first real image and a first image into the discriminator for training, calculating the similarity between the first real image and the first image through the first branch of the discriminator, and judging the authenticity of the first image through the second branch of the discriminator.
[0062] If the authenticity and similarity meet the preset requirements, the training of the discriminator is completed; if the authenticity and / or similarity do not meet the preset requirements, the next round of training of the discriminator is started until the authenticity and similarity output of the discriminator in the next round meet the preset requirements.
[0063] In this embodiment, the trained discriminator can both distinguish between genuine and fake images and calculate the similarity between images. By using a single discriminator to simultaneously distinguish between genuine and fake images and calculate image similarity, the training time of the generative adversarial network can be reduced.
[0064] It should be noted that the preset requirements in this embodiment can be changed according to the actual situation, and this embodiment does not impose specific limitations.
[0065] In some embodiments, the loss function of a generative adversarial network is constructed as follows:
[0066]
[0067] Where G represents the generator, D represents the discriminator, x1 represents the first real image selected from the real image samples, x2 represents the second real image selected from the real image samples, x represents the selection of the first real image or the second real image, and G(z) represents the first image of the generator.
[0068] In some embodiments, the generator is trained using a trained discriminator combined with training image samples to obtain a trained generator, including:
[0069] The generator is trained by training the discriminator so that the generator minimizes log(1-D(G(z),x)) when training image samples, thus obtaining the trained generator.
[0070] In this embodiment, training the generator with a trained discriminator can improve the quality of image generation.
[0071] In some embodiments, the discriminator uses a Siamese neural network to calculate the similarity between two images.
[0072] In some embodiments, a Siamese neural network is used to calculate the similarity between two images, including:
[0073] Two images are convolved through a convolutional layer to obtain two convolved images;
[0074] The two convolutional images are connected using a fully connected layer to extract image features;
[0075] The distance between two images is calculated by extracting image features;
[0076] The distance between the two images is calculated and passed through a fully connected layer, and the similarity between the two images is calculated using the sigmoid activation function.
[0077] In some embodiments, the objective function of the Siamese neural network includes:
[0078]
[0079] Where N represents the number of samples, X1 and X2 represent the first real image and the second real image, or the first real image and the first image, Y = 1 indicates that the first real image and the second real image are similar, or the first real image and the first image are similar, Y = 0 indicates that the first real image and the first image are not similar, and E w This represents the distance between the first real image and the second real image, or the distance between the first real image and the first image, where m represents a preset threshold.
[0080] To facilitate understanding by those skilled in the art, a set of preferred embodiments is provided below:
[0081] A dual-output discriminator is constructed, capable of both distinguishing between genuine and fake images and calculating the similarity between images. Based on the generator and the dual-output discriminator, a generative adversarial network (GAN) is built. The GAN is then trained. Specifically:
[0082] Reference Figure 3 In the diagram, the generator G generates the first image using randomly generated noise data (i.e., noisy training image samples). The input data for the discriminator D mainly comes from the first image G(z) generated by the generator (labeled as gen imgs in the diagram), the selected first real image x1 (labeled as true imgs1 in the diagram), and the selected second real image x2 (labeled as true imgs2 in the diagram). These data sources are grouped and input into the discriminator to determine the authenticity of the images and calculate their similarity.
[0083] The discriminator in this embodiment is mainly composed of a Siamese neural network. The input to the discriminator is two paired images, and the output is the similarity between the two images and the image's realism. Specifically, two pairs of images (each pair labeled Input 1 and Input 2, where Input 1 can be a real image or a generated first image, and Input 2 is always a real image) are input into the discriminator. After convolution by the discriminator's convolutional layer, it outputs one-dimensional feature vectors of the two input images respectively. Figure 3(The image is marked as Img Feature). At this point, the authenticity of image Input 1 will be judged. If Input 1 is a real image, the judgment result is 1; if it is the first generated image, the judgment result is 0.
[0084] Furthermore, the discriminator uses a Siamese neural network to calculate the feature distance between two images, Input 1 and Input 2, and determines their similarity. If the two images are similar, the output is 1; if they are dissimilar, the output is 0. The similarity calculation is as follows:
[0085] The two images input to the Siamese neural network first pass through convolutional layers, where weights are shared. After the convolutional layers, the outputs of the two images are connected by a fully connected layer to extract their features. The distance between the two images (e.g., Euclidean distance) is calculated using these features, and finally, a fully connected layer is used to determine the similarity between the two images using the sigmoid activation function. The Siamese neural network is well-suited for calculating the similarity between two images.
[0086] The objective function of the Siamese neural network is shown below:
[0087]
[0088] Where N represents the number of samples, X1 and X2 represent two images, Y = 1 indicates that the two images are similar, Y = 0 indicates that the two images are dissimilar, and E w This represents the distance between two images, where m represents the preset threshold.
[0089] When Y = 1, it means that images X1 and X2 belong to the same category. The loss function of the Siamese neural network in this case is:
[0090]
[0091] The optimization direction is to minimize the loss function, that is, minimize This is equivalent to optimizing in the direction that minimizes the distance between two images.
[0092] When Y = 0, it means that images X1 and X2 do not belong to the same category. The loss function of the Siamese neural network in this case is:
[0093]
[0094] Optimize in the direction of minimizing the loss function, i.e., maximizing E. w The direction refers to the direction in which the distance between two images is increased. After training, the Siamese neural network can effectively determine whether two images belong to the same category, that is, whether the two images are similar.
[0095] In the backpropagation process of an image, there are two main optimization objectives: one is to make the generator G generate more realistic images, and the other is to generate images that are more similar to real images.
[0096] The loss function calculation formula for the generative adversarial network in this embodiment is as follows:
[0097]
[0098] When training the generator, the goal is to optimize the loss function towards log(1-D(G(z),x))→0, i.e., to generate images that are more realistic and similar to real images. When training the discriminator, the goal is to optimize the loss function towards log(D(x1,x2))+log(1-D(G(z),x)+D(x1,x2))→1, specifically towards D(G(z),x)→0 and D(x1,x2)→1. At this point, the discriminator can effectively distinguish the realism of the generated first image and its similarity to the real image. Through the interplay between the generator and the discriminator, the generator eventually produces highly realistic and high-quality images, while the discriminator effectively judges the realism of the images.
[0099] In the discriminator training process of this embodiment, the first image (G(z)) and the real images (first real image x1 or second real image x2) are grouped together, and the first real image x1 and the second real image x2 are grouped together, and input to the first branch and the second branch of the discriminator of the network, respectively. After feature extraction by the network's convolutional layers, the similarity between the two groups of images is calculated in the fully connected layers. In addition, the network uses the selected real image (first real image x1 or second real image x2) and the generated first image G(z) to determine the authenticity of the image.
[0100] The image to be filled is obtained, and the image is filled using a pre-trained generative adversarial network to obtain the filled image output by the generative adversarial network.
[0101] To better illustrate this, the following experiment was conducted in this embodiment:
[0102] GAN networks are commonly used for data imputation, often encountering situations of insufficient or unevenly distributed data. In such cases, neural network training is prone to underfitting or overfitting on certain categories. Therefore, artificially synthesized images can compensate for insufficient training data, but this method is time-consuming and the imputation quality is not high. Experiments demonstrate that the technical solution in this embodiment outperforms the original GAN network in the field of data imputation. Specifically:
[0103] This embodiment uses the MNIST handwritten digit dataset for experiments. The original MNIST dataset training set contains 60,000 handwritten digits from 0 to 9. Before the experiment, this embodiment extracts a portion of the data for category 6, leaving only 500 images of the digit 6 in the training set. At this point, the number of digit 6 images in the training set will be significantly less than other digits. Then, we observe the recognition rate of the digit 6 in several scenarios: directly training the network on the existing dataset (without data padding), using the original GAN network for data padding, and using the data padding method of this embodiment. (The number of correctly identified digit 6 images in the training set is compared with the total number of digit 6 images in the training set.) A higher recognition rate indicates that the model is more sensitive to the digit 6, i.e., the data padding is more effective. The final experimental results are shown in Table 1 below.
[0104] Table 1
[0105] Do not use data filling 77％ Fill with the original GAN 83％ Fill using the technical solution of this embodiment 88％
[0106] Experimental results show that using the technical solution of this embodiment to fill data can effectively compensate for the problem of uneven data distribution during neural network training. Furthermore, the technical solution of this embodiment outperforms the original GAN in the field of data filling, and has significant practical implications.
[0107] This embodiment also selected three datasets for experiments: the MNIST handwritten digit dataset, the FashionMnist dataset, and the Covid19 (Covid19 Radiography Database) to verify the feasibility of the technical solution of this embodiment in the medical field. The experiments mainly simulated the improvement in classification performance brought about by data imputation using the technical solution of this embodiment in real-life situations with imbalanced class numbers. The experimental group used the original dataset without any data imputation techniques. The control groups were those using traditional GAN networks for data imputation and those using the technical solution of this embodiment for data imputation. An imbalanced class was defined as a class whose number of members is significantly less than other classes. The effect of data imputation was measured by calculating the recognition rate of the neural network for the imbalanced class (the recognition rate of the neural network for this class = the number of members in the dataset). The categories in Table 2 below represent imbalanced classes, i.e., classes whose data volume is significantly less than other classes. In the FashionMnist and MNIST datasets, the data imputation method using the technical solution of this embodiment has significant advantages compared to the other two methods. However, the advantage is not so obvious in the Covid-19 dataset because medical images have more and more complex features, requiring more complex generators with stronger expressive capabilities to generate high-quality medical images.
[0108] Table 2
[0109]
[0110]
[0111] Reference Figures 4 to 5 This invention also provides a data filling system based on generative adversarial networks. This data filling system includes a data acquisition module 100 and a data filling module 200, wherein:
[0112] Data acquisition module 100 is used to acquire the image to be filled;
[0113] The data infill module 200 is used to input the image to be filled into a preset generative adversarial network (GAN) for image infilling, and obtain the filled image output by the GAN; wherein, the GAN includes a generator and a dual-output discriminator, and the data infill module includes a data acquisition submodule 210, a pre-training submodule 220, a first training submodule 230, and a second training submodule 240, wherein:
[0114] The data acquisition submodule 210 is used to acquire training image samples, a first real image, and a second real image;
[0115] The pre-training submodule 220 is used to input training image samples into the generator for pre-training to obtain the first image output by the generator.
[0116] The first training submodule 230 is used to input the first image, the first real image, and the second real image into the discriminator for training, to obtain the first image and the first real image samples calculated by the first branch of the discriminator, as well as the similarity between the first real image and the second real image and the authenticity of the first image and the second real image judged by the second branch of the discriminator, and to complete the training of the discriminator based on the similarity and authenticity.
[0117] The second training submodule 240 is used to train the generator based on the discriminator that has been trained and the training image samples, so as to obtain the trained generator.
[0118] It should be noted that since the data filling system based on generative adversarial networks in this embodiment is based on the same inventive concept as the data filling method based on generative adversarial networks described above, the corresponding content in the method embodiment is also applicable to this system embodiment, and will not be described in detail here.
[0119] This invention also provides a data stuffing device based on a generative adversarial network, comprising: at least one control processor and a memory for communicatively connecting to the at least one control processor.
[0120] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0121] The non-transitory software program and instructions required to implement the data imputation method based on generative adversarial networks in the above embodiments are stored in memory. When executed by a processor, the data imputation method based on generative adversarial networks in the above embodiments is executed, for example, the method described above is executed. Figure 1 Method steps S100 to S200 and Figure 2 The method steps S210 to S240.
[0122] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0123] This invention also provides a computer-readable storage medium storing computer-executable instructions. These instructions are executed by one or more control processors, causing the processors to perform a data padding method based on a generative adversarial network as described in the above-described method embodiments. For example, they may execute the methods described above. Figure 1 The functions of steps S100 to S200 in the method and Figure 2 The functions of steps S210 to S240 in the method.
[0124] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.
[0125] The above is a detailed description of the preferred embodiments of this application. However, the embodiments of this application are not limited to the above-described implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the embodiments of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of the embodiments of this application.
Claims
1. A data imputation method based on generative adversarial networks, characterized in that, The data imputation method based on generative adversarial networks includes: Get the image to be filled; The image to be filled is input into a preset generative adversarial network (GAN) for image filling, resulting in the filled image output by the GAN; wherein, the GAN includes a generator and a dual-output discriminator, and the GAN is trained in the following manner: Obtain training image samples, the first real image, and the second real image; The generator is pre-trained by inputting training image samples to obtain the first image output by the generator. The first image, the first real image, and the second real image are input into the discriminator for training. This yields the similarity between the first image and the first real image, and between the first real image and the second real image, calculated by the first branch of the discriminator, and the authenticity of the first image and the second real image as determined by the second branch of the discriminator. The training of the discriminator is then completed based on the similarity and the authenticity, including: The discriminator is trained for the current round, wherein the current round training includes: inputting the first real image and the second real image into the discriminator for training, calculating the similarity between the first real image and the second real image through the first branch of the discriminator, and determining the authenticity of the second real image through the second branch of the discriminator; simultaneously inputting the first real image and the first image into the discriminator for training, calculating the similarity between the first real image and the first image through the first branch of the discriminator, and determining the authenticity of the first image through the second branch of the discriminator. If the authenticity and similarity meet the preset requirements, the training of the discriminator is completed; if the authenticity and / or similarity do not meet the preset requirements, the next round of training of the discriminator is started until the authenticity and similarity output by the discriminator in the next round meet the preset requirements. The generator is trained by combining the trained discriminator with the training image samples to obtain the trained generator.
2. The data imputation method based on generative adversarial networks according to claim 1, characterized in that, The loss function of the generative adversarial network is constructed as follows: in, This refers to the generator. This refers to the discriminator. This represents the first real image selected from the real image samples. This represents a second real image selected from the real image samples. This indicates whether to select the first or second real image. This represents the first image of the generator.
3. The data imputation method based on generative adversarial networks according to claim 2, characterized in that, The step of training the generator based on the trained discriminator and the training image samples to obtain the trained generator includes: The generator is trained using the discriminator that has completed training, so that the generator minimizes the training image samples during training. The trained generator is then obtained.
4. The data imputation method based on generative adversarial networks according to claim 1, characterized in that, The discriminator uses a twin neural network to calculate the similarity between two images.
5. The data imputation method based on generative adversarial networks according to claim 4, characterized in that, The method of calculating the similarity between two images using a Siamese neural network includes: The two images are convolved through a convolutional layer to obtain two convolved images; The two convolutional images are connected using a fully connected layer to extract image features; The distance between two images is calculated by extracting image features; The distance between the two images is calculated and passed through a fully connected layer, and the similarity between the two images is calculated using the sigmoid activation function.
6. The data imputation method based on generative adversarial networks according to claim 4, characterized in that, The objective function of the twin neural network includes: in, Indicates the number of samples. This refers to either the first real image and the second real image, or the first real image and the first image. This indicates that the first real image and the second real image are similar, or that the first real image and the first image are similar. This indicates that the first real image and the first image are not similar. This represents the distance between the first real image and the second real image, or the distance between the first real image and the first image. This indicates the preset threshold.
7. A data filling system based on generative adversarial networks, characterized in that, The data padding system based on generative adversarial networks includes: The data acquisition module is used to acquire the image to be filled. A data infilling module is used to input the image to be infilled into a preset generative adversarial network (GAN) for image infilling, obtaining the infilled image output by the GAN; wherein, the GAN includes a generator and a dual-output discriminator, and the data infilling module includes a data acquisition submodule, a pre-training submodule, a first training submodule, and a second training submodule, wherein: The data acquisition submodule is used to acquire training image samples, a first real image, and a second real image; The pre-training submodule is used to input training image samples into the generator for pre-training to obtain the first image output by the generator; The first training submodule is used to input the first image, the first real image, and the second real image into the discriminator for training, to obtain the similarity between the first image and the first real image, and between the first real image and the second real image calculated by the first branch of the discriminator, and the authenticity of the first image and the second real image as determined by the second branch of the discriminator, and to complete the training of the discriminator based on the similarity and the authenticity, including: The discriminator is trained for the current round, wherein the current round training includes: inputting the first real image and the second real image into the discriminator for training, calculating the similarity between the first real image and the second real image through the first branch of the discriminator, and determining the authenticity of the second real image through the second branch of the discriminator; simultaneously inputting the first real image and the first image into the discriminator for training, calculating the similarity between the first real image and the first image through the first branch of the discriminator, and determining the authenticity of the first image through the second branch of the discriminator. If the authenticity and similarity meet the preset requirements, the training of the discriminator is completed; if the authenticity and / or similarity do not meet the preset requirements, the next round of training of the discriminator is started until the authenticity and similarity output by the discriminator in the next round meet the preset requirements. The second training submodule is used to train the generator based on the discriminator that has completed training and the training image samples, so as to obtain the generator that has completed training.
8. A data filling device based on generative adversarial networks, characterized in that, It includes at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, which, when executed by the at least one control processor, enable the at least one control processor to perform the data filling method based on a generative adversarial network as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the data filling method based on generative adversarial networks as described in any one of claims 1 to 6.