A handwritten character generation method, system and electronic device for single-sample classes

Abstract

The application discloses a handwritten character generation method and system for a single sample class and electronic equipment, relates to the technical field of handwritten character digitization, and comprises the following steps: training a variational autoencoder by using a sample real handwritten character image, obtaining a probability distribution of the variational autoencoder in a hidden space of the variational autoencoder, initializing a generator by using a decoder of the trained variational autoencoder, and initializing a discriminator by using an encoder; inputting a target real handwritten character image and a target computer printed character image into the above-mentioned pre-trained encoder to obtain corresponding hidden variables; aligning the mean and covariance of all hidden variables in the hidden space by using a CORAL transformation technology to obtain an initialized hidden variable set; and inputting the initialized hidden variable into the generator in the generative adversarial network to generate a new character image. The application can perform data augmentation on a sample extremely unbalanced handwritten character dataset, so that the generated small sample class sample has correctness and diversity.

Description

Technical Field

[0001] This invention relates to the field of handwritten character digitization technology, and in particular to a method, system and electronic device for generating handwritten characters for single sample classes. Background Technology

[0002] Optical character recognition (OCR) can digitize printed or handwritten text from images. Despite significant advancements in accuracy over the years, challenges remain in recognizing text in historical handwritten documents. One major challenge is the imbalance of sample numbers between character categories. This category imbalance is particularly prevalent in scripts such as Chinese and Tibetan, severely hindering further improvements in the accuracy of deep learning-based OCR systems.

[0003] One intuitive way to improve the accuracy of character classifiers is to add more synthetic data to smaller classes. Deep generative models, such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs), can automatically generate synthetic samples given labeled or unlabeled training datasets. GANs have achieved excellent results in generating realistic samples over the past decade. However, when dealing with imbalanced datasets—where samples for some classes are very scarce—there is a well-known “fidelity-diversity” tradeoff. That is, to optimize the loss function, the generator tends to generate identical samples because the discriminator will classify any generated sample that differs from the real sample as “fake,” which does not meet the requirements of data augmentation.

[0004] Some studies have focused on addressing the imbalanced learning problem by improving the loss function of GANs. For example, MFC-GAN introduces multiple "fake" labels on top of AC-GAN to ensure fine-grained generation of small-sample classes. Ren et al. proposed an entropy-based weighting strategy to characterize the importance of different classes, assigning higher importance to small-sample classes and lower importance to large-sample classes. The importance-weighted label vector and a random noise vector are concatenated and used as input to Wasserstein GAN. To bias the generator towards small-sample classes, Lin et al. proposed Rare-GAN, introducing a reweighting technique into the AC-GAN loss function, assigning two different weights to samples based on their class (large-sample or small-sample). Furthermore, to improve the classification performance of AC-GAN on imbalanced data, they also employed a confidence-based active learning strategy. However, this method of improving the loss function has limited data augmentation effect on small-sample classes because artificial neural networks follow a certain "laziest principle" when optimizing the loss function.

[0005] There are also studies combining GANs with other deep generative models. For example, Mariani et al. proposed BAGAN. They addressed the class imbalance problem by fitting the probability distribution of each class in the latent space of a pre-trained autoencoder (AE) and using it as input to the generator. Before adversarial training, the generator and discriminator are initialized by the decoder and encoder of the pre-trained AE, respectively. The goal of the pre-trained AE is to learn a low-dimensional manifold for each class in the latent space, enabling it to capture the conditional probability distribution of each class even with limited samples. However, the latent space trained by the AE often lacks regularity, failing to accurately capture class boundaries. To alleviate this problem, Yang et al. proposed IDA-GAN, replacing the AE pre-training in BAGAN with VAE pre-training. Furthermore, it incorporates class labels into the generator's input to guide the generation process. In BAGAN and IDA-GAN, the class conditional distribution is obtained by fitting a Gaussian distribution to the latent variables; however, the assumption that classes follow a Gaussian distribution in the latent space is controversial.

[0006] The research described above has achieved some success in imbalanced learning. However, when discussing "few-sample classes," they usually refer to classes with at least a few dozen samples. For the extreme case of few-sample classes—the generation of single-sample classes—there is still no effective solution. In languages ​​such as Chinese and Tibetan, single-sample classes are very frequent due to the large number of rare characters. Summary of the Invention

[0007] The purpose of this invention is to provide a method, system, and electronic device for generating handwritten characters in a single-sample class. This method augments extremely imbalanced handwritten character datasets, generating small-sample class samples that are both accurate and diverse, thereby significantly improving the recognition performance of the character recognizer.

[0008] To achieve the above objectives, the present invention provides the following solution:

[0009] In a first aspect, the present invention provides a method for generating handwritten characters for a single sample class, comprising:

[0010] The sample real handwritten character image is input into the variational autoencoder to pre-train the variational autoencoder, and the probability distribution of the sample real handwritten character image in the latent space of the variational autoencoder is obtained. The generator in the generative adversarial network is initialized with the decoder of the trained variational autoencoder, and the discriminator in the generative adversarial network is initialized with the encoder of the trained variational autoencoder.

[0011] The target real handwritten character images and corresponding target computer-printed character images from the target real dataset are respectively input into the encoder of the trained variational autoencoder to obtain the latent variables of each target real handwritten character image and the corresponding target computer-printed character image; each target real handwritten character image in the target real dataset belongs to a different category; the target real dataset is the real dataset to be augmented; the target real handwritten character images are the real handwritten character images to be augmented.

[0012] Using the CORAL transformation technique, the latent variables of all the target real handwritten character images and the corresponding target computer printed character images are aligned in the latent space for mean and covariance to obtain an initial latent variable set.

[0013] The initialization hidden variables in the set of initialization hidden variables are input into the generator in the generative adversarial network to generate a new character image.

[0014] Secondly, the present invention provides a handwritten character generation system for single-sample classes, comprising:

[0015] The training module is used to input real handwritten character images of the samples into the variational autoencoder to pre-train the variational autoencoder, obtain the probability distribution of the real handwritten character images of the samples in the latent space of the variational autoencoder, initialize the generator in the generative adversarial network with the decoder of the trained variational autoencoder, and initialize the discriminator in the generative adversarial network with the encoder of the trained variational autoencoder.

[0016] The latent variable determination module is used to input the target real handwritten character images and the corresponding target computer-printed character images from the target real dataset into the encoder of the trained variational autoencoder to obtain the latent variables of each target real handwritten character image and the corresponding target computer-printed character image; the category of each target real handwritten character image in the target real dataset is different; the target real dataset is the real dataset to be augmented; the target real handwritten character images are real handwritten character images to be augmented.

[0017] The latent variable initialization module is used to align the latent variables of all the target real handwritten character images and the corresponding target computer printed character images in the latent space using CORAL transformation technology to obtain an initialized latent variable set.

[0018] The character image generation module inputs the initialization hidden variables from the initialization hidden variable set into the generator in the generative adversarial network to generate a new character image.

[0019] Thirdly, the present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to cause the electronic device to perform the handwritten character generation method for a single sample class as described in the first aspect.

[0020] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0021] Deep generative models based on GANs face challenges in generating diverse and high-quality samples from a single character class. Since only one sample is available for reference per class, the discriminator inevitably focuses on irrelevant features of that class, leading to the generator outputting identical samples and failing to meet the needs of data augmentation. This invention proposes a method, system, and electronic device for generating handwritten characters from a single class, specifically providing a Character Rebalance GAN (CRB-GAN) model, capable of generating handwritten character images with both accuracy and a certain degree of diversity even when only one sample is available for a class. This method includes: First, pre-training a variational autoencoder on a real character dataset to map character images from a high-dimensional image space to a low-dimensional latent space; then, using CORAL transformation, aligning the mean and covariance of computer-printed character images of different fonts and sizes with real handwritten character images in the latent space to obtain initialized latent variables; next, using these initialized latent variables as input to the generator, and using the decoder from the pre-training stage as the generator and the encoder as the discriminator for adversarial training; to maintain the diversity of small sample classes as much as possible, the sampling frequency of the classes is redesigned during the discriminator training stage, ensuring that the expected loss for each class being classified as "false" does not exceed the expected loss for each class being classified as "true". This method is used for data augmentation on extremely imbalanced handwritten character datasets, generating small sample class samples that are both correct and diverse, thereby significantly improving the recognition performance of the character recognizer. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the handwritten character generation method for a single-sample class provided in an embodiment of the present invention.

[0024] Figure 2 This is a schematic diagram of the VAE pre-training process provided in an embodiment of the present invention;

[0025] Figure 3 This is a schematic diagram of the implicit variable initialization process provided in an embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram of the adversarial training process provided in an embodiment of the present invention;

[0027] Figure 5The first generated result image for a single sample category in the handwritten Chinese data provided in this embodiment of the invention;

[0028] Figure 6 This is a second generation effect diagram for a single sample category in the handwritten Chinese data set provided in the embodiments of the present invention;

[0029] Figure 7 This is the first generated result image for a single sample category in the handwritten Tibetan dataset provided in this embodiment of the invention;

[0030] Figure 8 This is a second generation effect diagram for a single sample category in the handwritten Tibetan data set provided in an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] This invention effectively augments data for single-sample categories by automatically generating synthetic samples that are both realistic and diverse.

[0033] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] Example 1

[0035] This embodiment provides a method for generating handwritten characters for a single sample class, including:

[0036] Step 100: Input the real handwritten character image of the sample into the variational autoencoder to pre-train the variational autoencoder, obtain the probability distribution of the real handwritten character image of the sample in the latent space of the variational autoencoder, and initialize the generator in the generative adversarial network with the decoder of the trained variational autoencoder, and initialize the discriminator in the generative adversarial network with the encoder of the trained variational autoencoder.

[0037] Step 200: Input the target real handwritten character images and the corresponding target computer-printed character images from the target real dataset into the encoder of the trained variational autoencoder to obtain the latent variables of each target real handwritten character image and the corresponding target computer-printed character image; the categories of each target real handwritten character image in the target real dataset are different; the target real dataset is the real dataset to be augmented; the target real handwritten character images are real handwritten character images to be augmented.

[0038] Step 300: Using the CORAL transformation technique, align the latent variables of all the target real handwritten character images and the corresponding target computer printed character images in the latent space with their mean and covariance to obtain an initialized set of latent variables.

[0039] Step 400: Input the initialization hidden variables from the initialization hidden variable set into the generator in the generative adversarial network to generate a new character image.

[0040] In this embodiment, the encoder of the variational autoencoder consists of multiple downsampling layers, and the decoder of the variational autoencoder consists of multiple upsampling layers; the variational autoencoder is used to map character images from a high-dimensional image space to a low-dimensional latent space.

[0041] The training process of this variational autoencoder is as follows: The variational autoencoder is trained using real handwritten character images from the sample dataset. Training stops when the overall loss value of the variational autoencoder is less than a set threshold, resulting in a trained variational autoencoder. The overall loss value of the variational autoencoder is the sum of its original loss value and the cross-entropy loss value.

[0042] In this embodiment, the specific process of step 200 is described in Embodiment 2, and will not be repeated here.

[0043] In this embodiment, during the training phase of the generative adversarial network, a new category sampling frequency is used to train the discriminator, so that the expected loss of the discriminator for each category being judged as false does not exceed the expected loss for each category being judged as true; wherein, the sampling frequency of the new category is described in Embodiment 2, and will not be repeated here.

[0044] Example 2

[0045] This embodiment is a specific example of Embodiment 1.

[0046] The proposed GRB-GAN model in this embodiment aims to synthesize diverse samples for character categories with few samples, especially when only a single sample is available. Since diversity is difficult to achieve through adversarial training alone when samples are scarce, latent variables initialized from a Variational Autoencoder (VAE) are used as input to the generator, and the generator is initialized using the VAE's decoder. The latent variables are initialized by aligning the mean and covariance between computer-printed character images and real handwritten character images, which is equivalent to performing a style transfer operation in the latent space. The intuitive understanding of this method is that for successful data augmentation, it is necessary to synthesize samples that are similar to but not identical to real samples. The style transfer operation brings the computer-printed character images closer to real samples in the latent space to achieve this goal. Furthermore, this simple technique allows the generator to start with a good initial solution, i.e., it can synthesize blurry but structurally correct samples at the beginning. Therefore, the goal of adversarial training is to correct the details of the generated images. The training process of the CRB-GAN model includes three steps: VAE pre-training, latent variable initialization, and adversarial training.

[0047] VAE pre-training: Train the VAE using all samples from the real dataset, such as Figure 2 As shown, the solid line represents the data flow, and the dashed line represents the calculation of the loss function. The VAE encoder consists of multiple downsampling layers, and the decoder consists of multiple upsampling layers. First, the real samples are input into the encoder to obtain the corresponding mean μ and variance σ, and the latent variable z is constructed using the reparameterization technique, i.e., z = μ + σ·ζ, where ζ ~ N(0, I), and I represents the identity matrix, i.e., ζ is sampled from a standard Gaussian distribution; then, the latent variable z is input into the decoder to obtain the reconstructed image. In addition, to encourage characters with the same category label to cluster together in the latent space, a classification head is added after the latent variable z. Let q(x) represent the marginal distribution of the real samples, and the pre-trained loss function is defined as follows:

[0048]

[0049]

[0050]

[0051] in, It is a VAE loss. This is the cross-entropy loss, where α is the weight balancing the two terms. c is the class label associated with the real handwritten character image x, and q is... E (z|x) is the approximate posterior probability represented by encoder E, P Δ(x|z) is the approximate conditional probability represented by the decoder Δ, p(z) is the probability density function of the standard Gaussian distribution; E represents Expectation, i.e., the expected value. Here, E x～q(x) [...] refers to taking the expected value of the relative probability distribution q(x). KL stands for Kullback-Leibler Divergence, which is an index for calculating the similarity between two probability distributions. cls(z;x) represents the classification prediction result for the latent variable z. Since the latent variable z needs to be obtained from the input image x, it is represented as xls(z;x). L CE () represents the cross-entropy loss function for a single sample. Let q(x) represent the relative probability distribution, which is the expected value of the cross-entropy loss of the entire dataset.

[0052] Latent variable initialization: To achieve successful data augmentation, the generated new character image should be similar to, but not identical to, the real character image. Guided by this idea, the latent variables of the computer-printed character image transformed in latent space are used as input to the generator, as follows: Figure 3 As shown, when obtaining latent variables through the encoder, the reparameterization process is ignored, that is, σ = 0 is taken for all input images.

[0053] CORAL (CORrelation Alignment) is a simple and widely used domain adaptation technique that minimizes domain shift by aligning the second-order statistics of the source and target domains. This embodiment treats a real handwritten character image as the target domain and a computer-printed character image as the source domain, aligning their first-order and second-order statistics. The computer-printed character image and the real handwritten character image are input into a pre-trained encoder to obtain the corresponding latent variables, specifically the latent variables of the computer-printed character image. Latent variables of real handwritten character images Furthermore, to provide diversity, different fonts and computer-printed character images of varying sizes are used. The goal of this step is to find a linear transformation A. * This minimizes the Frobenius norm between the covariance matrices of the source and target domains. In addition to the covariance matrix, the class means of the source domain are aligned to the class means of the target domain. Let C... s and C t Z s and Z t Let the covariance matrix be... and Z s and Z t The set of class means is then the set of latent variables used as input to the generator. Initialize according to the following formula.

[0054]

[0055]

[0056] Among them, among them, Let be the mean of the k-th category of the target real handwritten character in the latent space. The mean of the k-th category in the latent space for the target computer-printed character image. Let z be the position in latent space of the i-th target computer-printed character image belonging to the k-th category. (i) Let k be the value in the latent space of the i-th target computer-printed character image after transformation; k is the value of the latent variable. Associated category label, U s Σ s U s T and U t Σ t U t T They are C s and C r The singular value decomposition, r = min{rank(C s ), rank(C t )},Σ t[1:r] and U t[1:r] It is C t The first r largest singular values ​​and their corresponding singular vectors are given. After the above transformation, the diversity of real handwritten character images is rebalanced among the various categories, where T is the transpose.

[0057] Adversarial training: Use the initialized latent variables obtained in the previous step as input to the generator, and initialize the generator G with the decoder Δ and the discriminator D with the encoder E, as follows. Figure 4 As shown. In adversarial training, the sampling strategy for the real samples and generated fake samples used to discriminate the loss is as follows: Let q(x) = c1q1(x) + c2q2(x) + ... + c K q K (x) represents the distribution of the real data, where q k (x) and c k Let k = 1, ..., K represent the data distribution of category k in the image space and the corresponding category frequency, respectively. The sampling distribution of the real data is as follows: The sampling distribution of the generated data is Where, r k (z) is the distribution of the k-th class in the latent space in the initial latent variables.

[0058]

[0059] λ is a constant that satisfies . The objective function of the discriminator is defined as follows:

[0060]

[0061] where p D (x is k|x) represents the probability that when the input is the image x, the discriminator D determines that the image x belongs to the category k. p D (G(z) is fake|G(z)) represents the probability that when the input is the image G(z) generated by the generator, the discriminator D determines that the image G(z) is "fake", and E represents the expected value.

[0062] In this way, the expected value of the generation loss for each category will not exceed the expected value of its real loss, and the generation loss for the small-sample categories is penalized, while maintaining diversity as much as possible during adversarial training.

[0063] Chinese and Tibetan are two languages with complex glyph structures and a large number of rare characters. Therefore, they are selected to test the performance of the CRB-GAN model in generation and data augmentation. The handwritten Chinese character dataset used in the experiment comes from the Chinese Ancient Handwritten Character Database (CASIA-AHCDB), which was constructed by annotating 11,937 pages of Chinese ancient manuscript documents and is used for character recognition research. In the experiment, the enhanced category set (AHCDB-style1-enhanced) of the AHCDB-style1 sub-database is used. The training set of the dataset contains 84,201 samples, distributed in 3,052 categories, and the validation / test set contains 34,248 samples. Among them, there are 602 categories with fewer samples in the training set than in the test set, which makes it very suitable for testing the performance of data augmentation. The handwritten Tibetan character dataset used in the experiment is obtained by segmenting the characters in the Beijing edition of the Kangyur and manually annotating the category labels of the characters (PeK-212). This dataset contains 501 categories and 40,372 character samples. Among them, the training set contains 28,094 samples and the validation set contains 12,277 samples. The sample distribution between categories in this dataset is extremely unbalanced: there are 135 categories with single samples, and 109 categories with more than 60 samples.

[0064] Figure 5 The first generation effect diagram for single-sample categories in the handwritten Chinese dataset. (a) Generated samples of "佷" / U4F77, (b) "侏" / U4F8F, (c) "倞" / U501E, (d) "兗" / U5157 and the corresponding within-class SSIM scores. Figure 6The second generated effect diagram for single-sample categories in the handwritten Chinese character dataset: (a) "凭" / U51ED, (b) "嘖" / U5616, (c) "瘕" / U7615, (d) "碓" / U7893, and the corresponding within-class SSIM scores.

[0065] Figure 5 and Figure 6 show the generated effects on single-sample categories in the handwritten Chinese character dataset. Among them, the diversity of the generated samples of BAGAN and IDA-GAN is usually only reflected at the stroke level, and the overall structure changes little, which leads to noisy foreground pixels, such as Figure 5 parts (b) and (c) of Figure 5 and some irregular glyphs, such as Figure 6 part (d) of

[0066] Figure 7 The first generated effect diagram for single-sample categories in the handwritten Tibetan character dataset: (a) / U0F42+U0FB2+U0FAD, (b) / U0F42+U0FB3+U0F7C, (c) / U0F62+U0F97, (d) / U0F45+U0F74, and the corresponding within-class SSIM scores. Figure 8 The first generated effect diagram for single-sample categories in the handwritten Tibetan character dataset: (a) / U0F46+U0F72, (b) / U0F51+U0F71, (c) / U0F51+U0FB2+U0F71, (d) / U0F66+U0F75, and the corresponding within-class SSIM scores.

[0067] Figure 7 and Figure 8 show the generated effects on single-sample categories in the handwritten Tibetan character dataset. Except for Figure 8 parts (b) and (c) of Figure 8The high diversity in parts (b) and (c) is mainly due to high noise and irregular strokes, which can sometimes compromise the meaning of the characters. Overall, the CRB-GAN-generated samples exhibit a certain degree of intra-class variation while maintaining accuracy as much as possible.

[0068] This embodiment also tested the data augmentation effect of CRB-GAN, and the results are shown in Table 1. Except for the first row (which is the real handwritten character dataset used to train the generative model), each row in Table 1 corresponds to a specific generation method. The second row (Zhang et al.) is a synthetic Tibetan dataset created by recombining the segmented Tibetan components.

[11] .

[0069] In terms of handwritten Chinese characters, this embodiment achieved the best performance in terms of ACC accuracy, FID score, and SSIM score. By augmenting the training dataset with generated samples, the classification accuracy was improved by nearly 2.17% compared to using only real data. Neither BAGAN nor IDA-GAN improved the SSIM score, while CRB-GAN reduced the overall in-class SSIM score by nearly 0.015.

[0070] In the Tibetan handwritten character dataset, IDA-GAN achieved the best FID score of 15.22, while CRB-GAN only reached 17.30, similar to BAGAN (17.25). This can be explained as follows: the Pek-212 dataset is extremely imbalanced even compared to the CASIA-AHCDB dataset. In principle, the FID score is a measure of the difference between probability distributions. Therefore, even if the generated samples are of high quality and have high intra-class diversity, the FID score may still be high because we are calculating the distribution difference between an extremely imbalanced dataset and a class-balanced dataset. It is important to note that the SSIM score we mention here is the intra-class SSIM score averaged across all classes. CRB-GAN's SSIM score is generally better than other methods in a few classes, as discussed earlier, intra-class variability in most classes does not contribute much to classification performance. Therefore, even though IDA-GAN's SSIM score reaches 0.4979, its classification accuracy is the worst among all methods (89.69%). In comparison, CRB-GAN improved classification accuracy by 4.57%, 3.46%, and 5.15% compared to real data, BAGAN, and IDA-GAN, respectively.

[0071] Table 1 shows the model generated based on a comparison of classification accuracy (ACC), FID score, and overall within-class SSIM score.

[0072]

[0073] This embodiment is an improvement upon BAGAN and IDA-GAN. In BAGAN and IDA-GAN, after pre-training with AE / VAE, they utilize the latent variables of real samples to fit class-conditional probability functions in the latent space, and use these class-conditional probability functions as input to the generator. This embodiment, building upon BAGAN and IDA-GAN, employs a novel method to initialize latent variables: aligning the first- and second-order statistics of computer-printed characters and real characters in the latent space. Furthermore, a cross-entropy loss function is introduced during the pre-training phase to enhance class clarity, and a special sampling method is proposed during the adversarial training phase to better preserve diversity. Experimental results show that this embodiment outperforms BAGAN and IDA-GAN in generating and augmenting Chinese and Tibetan handwritten characters. For other GAN-based generative models, since they primarily improve by modifying the loss function, it is difficult to achieve sufficient diversity across single sample classes simply by improving the loss function. Therefore, currently, there is no other alternative method to achieve the goal of this invention: successful data augmentation on extremely imbalanced datasets.

[0074] Example 3

[0075] In order to implement the method corresponding to Embodiment 1 above and achieve the corresponding functions and technical effects, a handwritten character generation system for single sample classes is provided below.

[0076] This embodiment provides a handwritten character generation system for single-sample classes, including:

[0077] The training module is used to input real handwritten character images of the samples into the variational autoencoder to pre-train the variational autoencoder, obtain the probability distribution of the real handwritten character images of the samples in the latent space of the variational autoencoder, initialize the generator in the generative adversarial network with the decoder of the trained variational autoencoder, and initialize the discriminator in the generative adversarial network with the encoder of the trained variational autoencoder.

[0078] The latent variable determination module is used to input the target real handwritten character images and the corresponding target computer-printed character images from the target real dataset into the encoder of the trained variational autoencoder to obtain the latent variables of each target real handwritten character image and the corresponding target computer-printed character image; the category of each target real handwritten character image in the target real dataset is different; the target real dataset is the real dataset to be augmented; the target real handwritten character images are real handwritten character images to be augmented.

[0079] The latent variable initialization module is used to align the latent variables of all the target real handwritten character images and the corresponding target computer printed character images in the latent space using CORAL transformation technology to obtain an initialized latent variable set.

[0080] The character image generation module inputs the initialization hidden variables from the initialization hidden variable set into the generator in the generative adversarial network to generate a new character image.

[0081] Example 4

[0082] This invention provides an electronic device including a memory and a processor. The memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform the handwritten character generation method for a single sample class according to Embodiment 1.

[0083] Alternatively, the aforementioned electronic device may be a server.

[0084] In addition, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the handwritten character generation method for a single-sample class as described in Embodiment 1.

[0085] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0086] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for generating handwritten characters for a single sample class, characterized in that, include: The sample real handwritten character image is input into the variational autoencoder to pre-train the variational autoencoder, and the probability distribution of the sample real handwritten character image in the latent space of the variational autoencoder is obtained. The generator in the generative adversarial network is initialized with the decoder of the trained variational autoencoder, and the discriminator in the generative adversarial network is initialized with the encoder of the trained variational autoencoder. The target real handwritten character images and the corresponding target computer printed character images in the target real dataset are respectively input into the encoder in the trained variational autoencoder to obtain the latent variables of each target real handwritten character image and the latent variables of the corresponding target computer printed character image. Each target real handwritten character image in the target real dataset belongs to a different category; The target real dataset is the real dataset to be augmented; The target real handwritten character image is the real handwritten character image to be augmented; Using the CORAL transformation technique, the latent variables of all the target real handwritten character images and the corresponding target computer printed character images are aligned in the latent space for mean and covariance to obtain an initial latent variable set. Using the CORAL transform technique, the latent variables of all the target real handwritten character images and the corresponding target computer-printed character images are aligned in the latent space using mean and covariance to obtain an initialized set of latent variables, specifically including: and formula This yields the set of initialized implicit variables; where, Let be the mean of the k-th category of the target real handwritten character in the latent space. The mean of the k-th category in the latent space for the target computer-printed character image. Let be the position of the i-th target computer-printed character image belonging to the k-th category in the latent space. Let be the value of the i-th target computer-printed character image in the latent space after transformation; the latent variable of the computer-printed character image is . The latent variables of a real handwritten character image are , and They represent and The covariance matrix, and They are and Singular value decomposition, , and yes The first r largest singular values ​​and their corresponding singular vectors; During the training phase of the generative adversarial network, a new sampling frequency for each category is used to train the discriminator, ensuring that the expected loss for each category being classified as false does not exceed the expected loss for each category being classified as true. The initialization hidden variables in the set of initialization hidden variables are input into the generator in the generative adversarial network to generate a new character image.

2. The handwritten character generation method for a single-sample class according to claim 1, characterized in that, The encoder of the variational autoencoder consists of multiple downsampling layers, and the decoder of the variational autoencoder consists of multiple upsampling layers; the variational autoencoder is used to map character images from a high-dimensional image space to a low-dimensional latent space.

3. The handwritten character generation method for a single sample class according to claim 2, characterized in that, The training process of a variational autoencoder is as follows: The variational autoencoder is trained using real handwritten character images from the real sample dataset. Training stops when the overall loss value of the variational autoencoder is less than a set threshold, thus obtaining the trained variational autoencoder.

4. The handwritten character generation method for a single-sample class according to claim 3, characterized in that, The overall loss value of the variational autoencoder is the sum of the variational autoencoder's loss value and the cross-entropy loss value.

5. A handwritten character generation system for single-sample classes, characterized in that, include: The training module is used to input real handwritten character images into the variational autoencoder (VAE) for pre-training, obtain the probability distribution of real handwritten character images in the latent space of the VAE, initialize the generator in the generative adversarial network (GAN) with the decoder of the VAE, initialize the discriminator in the GAN with the encoder of the VAE, and train the discriminator with the sampling frequency of the new category during the GAN training phase, so that the expected loss of the discriminator for each category being false does not exceed the expected loss for each category being true. The latent variable determination module is used to input the target real handwritten character images and the corresponding target computer printed character images from the target real dataset into the encoder in the trained variational autoencoder to obtain the latent variables of each target real handwritten character image and the latent variables of the corresponding target computer printed character image. Each target real handwritten character image in the target real dataset belongs to a different category; The target real dataset is the real dataset to be augmented; The target real handwritten character image is the real handwritten character image to be augmented; The latent variable initialization module is used to align the latent variables of all the target real handwritten character images and the corresponding target computer printed character images in the latent space using CORAL transformation technology to obtain an initialized latent variable set. Using the CORAL transform technique, the latent variables of all the target real handwritten character images and the corresponding target computer-printed character images are aligned in the latent space using mean and covariance to obtain an initialized set of latent variables, specifically including: and formula This yields the set of initialized implicit variables; where, Let be the mean of the k-th category of the target real handwritten character in the latent space. The mean of the k-th category in the latent space for the target computer-printed character image. Let be the position of the i-th target computer-printed character image belonging to the k-th category in the latent space. Let be the value of the i-th target computer-printed character image in the latent space after transformation; the latent variable of the computer-printed character image is . The latent variables of a real handwritten character image are , and They represent and The covariance matrix, and They are and Singular value decomposition, , and yes The first r largest singular values ​​and their corresponding singular vectors; The character image generation module inputs the initialization hidden variables from the initialization hidden variable set into the generator in the generative adversarial network to generate a new character image.

6. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the handwritten character generation method for a single sample class according to any one of claims 1 to 4.

Images