A semi-supervised Chinese character image generation method based on semantically guided discriminator

Abstract

The present invention discloses a semi-supervised Chinese character image generation method based on a semantically guided discriminator. In a semi-supervised scenario, a small number of labeled text images are selected from a training set, and the rest are used as unlabeled text data. An auxiliary classifier is used to provide multiple semantic information for the unlabeled text. Semantic information is implicit in the classifier features, and the classifier features of the text image are introduced into the discriminator. A dual-head structure is adopted on the discriminator: one head receives the category label, and the other head uses a feature fusion module to fuse the discriminator features and the classifier features, so that the classifier features are used as the condition for distinguishing true from false. Finally, the generator, discriminator, and classifier jointly complete the training. The present invention is aimed at unlabeled real text images, and uses the pseudo-labels predicted by the classifier and the classifier features to provide rich semantic information, improve the discriminator's ability to distinguish true and false samples under category conditions, and enable the generator to generate text that is more in line with the real class conditional distribution and has high authenticity and diversity.

Description

Technical Field

[0001] The present invention relates to the technical field of text generation, and in particular to a semi-supervised conditional Chinese character image generation method based on a generative adversarial neural network. Background Art

[0002] In recent years, generative adversarial neural networks have shown excellent performance in generating diverse and high-fidelity images. To control the generation of class-conditional images, many works have been devoted to class-conditional image generation, which generates instances that are related to a specific class but have sufficient diversity to serve downstream tasks.

[0003] When the amount of data is sufficient, the effect of class-conditional text image generation is relatively ideal. However, in practical applications, the training of conditional generative adversarial neural networks is subject to some limitations. In reality, most data are unlabeled, and labeling each image requires a lot of manpower, material resources and financial resources, which is a difficult task. If a large amount of unlabeled data is ignored and only a small amount of labeled data is used, it will often lead to a reduction in the diversity of the generated text images and inaccurate semantics. Therefore, semi-supervised technology can be used to improve the quality of text generation. Most existing semi-supervised image generation methods introduce classifiers on the basis of generative adversarial neural networks to classify unlabeled data and assign pseudo labels to it, so that unlabeled data can be used like labeled data. However, in the present invention, considering that the classifier features contain more semantic information than category information, the classifier features of the text data are introduced into the discriminator to add it to the generative adversarial training process, thereby improving the class-conditional distribution alignment of the real data and the generated data, and generating text images with richer styles to serve downstream tasks such as text detection and text recognition.

[0004] Specifically, this paper employs a dual-head architecture to enhance the discriminator's capabilities: one head receives class-level semantic information, while the other fuses the discriminator and classifier features. Given the differences in training objectives between the discriminator and classifier, and the fact that classifier features contain richer instance-level semantic information than class labels, this paper employs a feature fusion module to introduce classifier features and fuse them with discriminator features as a condition for adversarial training. Summary of the Invention

[0005] The goal of this invention is to utilize unlabeled text data to improve the fidelity and diversity of generated text images in situations where only a small amount of labeled text data is available, making the generated text more consistent with the corresponding category semantics. This invention provides a semi-supervised class-conditional Chinese character image generation method based on a generative adversarial neural network. This method combines a small amount of labeled data with a large amount of unlabeled data to train a generator for class-conditional Chinese character generation, generating more realistic and diverse text data to serve downstream tasks.

[0006] In order to achieve the above object, the present invention provides a technical solution: a semi-supervised text generation method based on a semantically guided discriminator, comprising the following steps:

[0007] S1. Prepare a dataset where only a small portion of the training data is labeled. Let the labeled data and its corresponding label set be L = {(x l ,y l )}, the unlabeled data set is recorded as: U = {x u}.

[0008] S2. Prepare a generator G:P implemented by a neural network k ×Y→R h×w , the generator generates a high-fidelity text image x with a resolution of h×w through a k-dimensional random vector z and a random category label y z , random category labels are randomly sampled from the annotation space Y of the text dataset.

[0009] S3: Use the text generated in S2 and the labeled text in the dataset as labeled data, and the unlabeled text in the dataset as unlabeled data. Use this data to train a classifier C implemented by a neural network. Use classifier C to classify the unlabeled data, predict the label of the unlabeled data, and obtain the classifier features of the data.

[0010] S4. Take the data generated in S2 as false data, and the labeled data and unlabeled data in the dataset as real data, and train a discriminator D:P implemented by a neural network to distinguish between real data and false data. h×w ×Y×P M →[0,1]. The discriminator takes the class label y and the classifier feature f C (·)∈P M is the condition, where M represents the dimension of the feature map.

[0011] S5. The learning of the neural network is constrained through the adversarial learning among the generator G, the classifier C and the discriminator D. When the adversarial learning of the three reaches a balance, the generator can generate text images that conform to the true category conditional distribution and have richer styles.

[0012] Furthermore, in step S1, since the unlabeled data has no category labels, a classifier C is needed to predict its pseudo labels. The dataset used to train the generative adversarial neural network is in the form of text images with category labels.

[0013] Furthermore, in step S2, the generator needs to generate text images using random vectors, taking the category label as a condition. The generated text is expressed by the following formula:

[0014] x z =G(z,y z )

[0015] where y z ∈Y represents a random class label, z represents a random vector, z is usually drawn from an easy-to-sample distribution, such as a predefined Gaussian distribution

[0016] Furthermore, in step S3, ResNet50 is selected as the network structure of classifier C. In order to make classifier C more accurately predict the category of unlabeled data, the classifier neural network is trained using real data and generated data. The training data includes: labeled data-label pairs (x l ,y l ), unlabeled data x u , and generate data-label pairs (x z ,y z ), the corresponding loss function is:

[0017]

[0018] Among them, C is the classifier, L stands for expectation. CE (·,·) is the cross entropy loss function, which is defined as:

[0019] L CE (x,y)=-ylogC(x)

[0020] Furthermore, in step S4, in order to compete with the generator and improve the realism of the generated image, it is necessary to train a discriminator composed of a multi-layer convolutional neural network. The purpose of the discriminator is to distinguish between real text and generated text as much as possible under the condition of category labels. In order to make full use of the instance-level semantic information contained in the classifier features in S3, the present invention adopts a dual-head structure D = {F, T1, T2} for the discriminator, where F is a feature extraction network, T1 contains a learnable neural network layer for learning the mapping of category labels; T2 contains a feature fusion module for combining the features learned by the classifier C and the feature extraction network F. T1 and T2 will work together to help G capture the accurate category semantic information corresponding to each type of text. At the same time, since the classifier features of the image are fused in T2, the features encoded in the classifier are richer than the category labels. Therefore, more detailed and diverse semantic information can be encoded into the discriminator through this operation. For simplicity, the present invention uses x to represent a labeled or unlabeled text image sample, and its corresponding label is defined as:

[0021]

[0022] T1 distinguishes between real data and generated data based on the category label y. T1 distinguishes true and false data based on whether the feature distribution of the generated text is consistent with the feature distribution of the real text of the corresponding category. Its expression is:

[0023]

[0024] Where Embed(·) represents a function that can learn category label mapping, is a mapping function, represents the weight of the last linear representation layer. In the above formula, T1(·,·) represents the probability that the discriminator network identifies the input data as real data. The loss function on T1 is expressed as follows:

[0025]

[0026] Considering that the classifier feature f is compared with the class label C The encoded semantic features are richer. In this invention, the classifier features are added to the discriminator as a kind of instance-level prior knowledge. In order to make the discriminator pay more attention to the category semantic information and make better use of the instance-level semantic information, the feature extractor feature f F and classifier features f C The feature fusion based on channel attention is implemented by using a neural network to calculate the pixel-level weights Q loc and global weights and Q glob , and then fuse the two types of weights as follows:

[0027]

[0028] in Represents a broadcast addition operation. By introducing the attention weight Q, T2 obtains instance-level information by receiving the fused features, as follows:

[0029]

[0030] in Indicates element-by-element multiplication. Under this condition, the feature extractor F needs to learn the same features as the classifier f C Complementary features. T2 also needs to identify real data and generated data, but it does not need category labels because it contains classifier features f C , It can strengthen the class separation of the discriminator feature space and promote class conditional distribution alignment. The optimization formula on T2 is as follows:

[0031]

[0032] Furthermore, in S5, in order to ensure that the generated text contains accurate category semantics and maintains high fidelity and diversity, it is necessary to jointly train the generator, discriminator, and classifier. The discriminator competes with the generator, while the classifier works in conjunction with the generator to guide the generator to capture accurate class semantic information. Therefore, the loss function for the generator and discriminator during adversarial training is defined as follows:

[0033]

[0034] The weighting factors α and β are used to balance the influence of the additional T2 and semantic evaluation items.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] 1. This invention combines three neural networks: a classifier, a generator, and a discriminator. Through joint adversarial training among the three, it ultimately improves the authenticity, diversity, and category matching of the text images generated by the generator, making the generated text images better serve downstream tasks such as text detection and text recognition.

[0037] 2. This paper proposes a neural network that can encode semantics at different levels as a discriminator, and adopts a relatively compact dual-head structure. This discriminator is more suitable for semi-supervised generative learning tasks. It utilizes the instance-level semantic information of the classifier and, through the function of the feature fusion module, improves the class separability in the discriminator's feature space, thereby facilitating the alignment of the class-conditional distribution of true and false data.

[0038] 3. The label-independent discriminator head introduced in this invention allows for better utilization of unlabeled data. Because the discriminator encodes richer semantic information, it can guide the generator in synthesizing text images with richer styles. Furthermore, this approach eliminates the need for the classifier to assign pseudo-labels to unlabeled data, thus reducing the problem of classifier error accumulation.

[0039] 4. The present invention effectively combines the classifier and the discriminator, making full use of the rich semantic information contained in the classifier features to promote the class-conditional distribution alignment in the discriminator feature space, making up for the problem of inaccurate semantics of the generated text in the semi-supervised generative learning task, while improving the diversity and fidelity of the generated civilized text, providing a better solution for the semi-supervised class-conditional text generation task. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 4 is a flowchart of the method of the present invention.

[0041] Figure 2This is a structural diagram of the character generation and classification part of the method of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0043] like Figure 1 As shown, the semi-supervised conditional Chinese character generation method based on the semantically guided discriminator provided in this embodiment takes the implementation of the generation model on the handwritten Chinese character dataset CASIA Online and Offline Chinese HandwritingDatabases 1.0 (referred to as CASIA-HWDB1.0) as an example, and includes the following steps:

[0044] The S1.CASIA-HWDB1.0 dataset contains a total of 1,680,258 text images stored as grayscale images. These text images can be categorized into 4,037 categories, encompassing 3,866 Chinese characters and 171 English alphabet letters and symbols. The entire dataset was handwritten by 420 writers. Before training, 50 grayscale text images from each category were randomly selected and provided with corresponding category labels as labeled data. The remaining text images were used as unlabeled data to construct the semi-supervised text dataset.

[0045] S2. Train a generator G:P implemented by a neural network k ×Y→R h×w , the generator generates a high-fidelity text image x with a resolution of h×w through a k-dimensional random vector z and a random category label y z , random category labels are randomly sampled from the annotation space Y of the text dataset.

[0046] S3: Use the text generated in S2 and the labeled text in the dataset as labeled data, and the unlabeled text in the dataset as unlabeled data. Use this data to train a classifier C implemented by a neural network. Use classifier C to classify the unlabeled data, predict the label of the unlabeled data, and obtain the classifier features of the data.

[0047] S4. Take the data generated in S2 as false data, and the labeled data and unlabeled data in the dataset as real data, and train a discriminator D:P implemented by a neural network to distinguish between real data and false data. h×w ×Y×P M →[0,1]. The discriminator takes the class label y and the classifier feature f C (·)∈P M is the condition, where M represents the dimension of the feature map.

[0048] S5. The learning of the neural network is constrained through the confrontation between the generator G, the classifier C and the discriminator D. When the adversarial learning of the three reaches a balance, the generator can generate text that conforms to the real category conditional distribution and has high authenticity.

[0049] Furthermore, in step S1, since the unlabeled data has no category labels, a classifier C is needed to predict its pseudo labels. The dataset used to train the generative adversarial neural network is in the form of text images with category labels.

[0050] Furthermore, in step S2, the generator needs to generate text images using random vectors, taking the category label as a condition. The generated text is expressed by the following formula:

[0051] x z =G(z,y z )

[0052] where y z ∈Y represents a random category label, z represents a random vector, z is extracted from a distribution that is easy to sample. This paper uses a standard Gaussian distribution with a mean of 0 and a variance of 1.

[0053] Furthermore, in step S3, ResNet50 is selected as the network structure of classifier C. In order to make classifier C more accurately predict the category of unlabeled data, the classifier neural network is trained using real data and generated data. The training data includes: labeled data-label pairs (x l ,y l ), unlabeled data x u , and generate data-label pairs (x z ,y z ), the corresponding loss function is:

[0054]

[0055] Among them, C is the classifier, L stands for expectation. CE (·,·) is the cross entropy loss function, which is defined as:

[0056] L CE (x,y)=-ylogC(x)

[0057] Furthermore, in step S4, in order to compete with the generator and improve the realism of the generated image, it is necessary to train a discriminator composed of a multi-layer convolutional neural network. The purpose of the discriminator is to distinguish between real text and generated text as much as possible under the condition of category labels. In order to make full use of the instance-level semantic information contained in the classifier features in S3, the present invention adopts a dual-head structure D = {F, T1, T2} for the discriminator, where F is a feature extraction network, T1 contains a learnable neural network layer for learning the mapping of category labels; T2 contains a feature fusion module for combining the features learned by the classifier C and the feature extraction network F. T1 and T2 will work together to help G capture the accurate category semantic information corresponding to each type of text. At the same time, since the classifier features of the image are fused in T2, the features encoded in the classifier are richer than the category labels. Therefore, more detailed and diverse semantic information can be encoded into the discriminator through this operation. For simplicity, the present invention uses x to represent a labeled or unlabeled text image sample, and its corresponding label is defined as:

[0058]

[0059] T1 distinguishes between real data and generated data based on the category label y. T1 distinguishes true and false data based on whether the feature distribution of the generated text is consistent with the feature distribution of the real text of the corresponding category. Its expression is:

[0060]

[0061] Where Embed(·) represents a function that can learn category label mapping, is a mapping function, represents the weight of the last linear representation layer, Represents a matrix operation. In the above formula, T1(·,·) represents the probability that the discriminator network identifies the input data as real data. The loss function on T1 is expressed as follows:

[0062]

[0063] Considering that the classifier feature f is compared with the class label C The encoded semantic features are more, and in this invention, the classifier features are added to the discriminator as a kind of instance-level prior knowledge. In order to make the discriminator pay more attention to the category semantic information and make better use of the instance-level semantic information, the feature extractor feature f F and classifier features f C The feature fusion based on channel attention is implemented by using a neural network to calculate the pixel-level weights Q loc and global weights and Qglob , and then fuse the two types of weights as follows:

[0064]

[0065] in Represents a broadcast addition operation. By introducing the attention weight Q, T2 obtains instance-level information by receiving the fused features, as follows:

[0066]

[0067] in Indicates element-by-element multiplication. Under this condition, the feature extractor F needs to learn the same features as the classifier f C Complementary features. T2 also needs to identify real data and generated data, but it does not need category labels because it contains classifier features f C , It can strengthen the class separation of the discriminator feature space and promote class conditional distribution alignment. The optimization formula on T2 is as follows:

[0068]

[0069] Furthermore, in S5, in order to ensure that the generated text contains accurate category semantics and maintains high fidelity and diversity, it is necessary to jointly train the generator, discriminator, and classifier. The discriminator competes with the generator, while the classifier works in conjunction with the generator to guide the generator to capture accurate class semantic information. Therefore, the loss function for the generator and discriminator during adversarial training is defined as follows:

[0070]

[0071] The weighting factors α and β are used to balance the influence of the additional T2 and semantic evaluation items.

[0072] After training, the performance of this method was quantitatively evaluated on the CASIA-HWDB1.0 dataset. The evaluation metrics included Inception Score (IS), Fr′echet Inception Distance (FID), and Recognition Accuracy (RA%). FID represents the similarity between the feature distribution of the generated image and the real image; a lower value indicates a more realistic generated image. IS represents the overall distribution of the generated image; a higher value indicates a more realistic generated image and better diversity. RA represents the probability that the generated image can be correctly classified by the classifier; a higher value indicates a higher semantic accuracy in the generated image. The evaluation showed that the performance of this method on all three evaluation criteria was significantly higher than that of the baseline method, making it worthy of promotion.

[0073] The above-mentioned embodiments are only preferred embodiments of the present invention, but they do not limit the scope of application of the method of the present invention. Therefore, all changes made according to the shape and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A semi-supervised Chinese character image generation method based on semantically guided discriminator, characterized in that: The following steps are involved: S1. Prepare a data set, in which only a small part of the training data is labeled; the labeled data and its corresponding label set are denoted as L = {(x l ,y l )}, the unlabeled data set is recorded as: U = {x u }; S2. Prepare a generator G:P implemented by a neural network k ×Y→R h×w , the generator generates a high-fidelity text image x with a resolution of h×w through a k-dimensional random vector z and a random category label y z , random category labels are randomly sampled from the annotation space Y of the text dataset; S3: Use the text generated in S2 and the text with category labels in the dataset as labeled data, and the unlabeled text in the dataset as unlabeled data. Use this data to train a classifier C implemented by a neural network. Use classifier C to classify the unlabeled data, predict the label of the unlabeled data, and obtain the classifier features of the data. S4. Take the data generated in S2 as false data, and the labeled data and unlabeled data in the dataset as real data, and train a discriminator D:P implemented by a neural network to distinguish between real data and false data. h×w ×Y×P M →[0,1], the discriminator takes the class label y and the classifier feature f C (·)∈P M is a condition, where M represents the dimension of the feature map; S5. The neural network is constrained through the adversarial learning between the generator G, the classifier C, and the discriminator D. When the adversarial learning of the three reaches a balance, the generator will generate text that conforms to the conditional distribution of the true category. A dual-head structure D = {F, T1, T2} is adopted for the discriminator, where F is a feature extraction network, T1 contains a learnable neural network layer for learning the mapping of category labels; T2 contains a feature fusion module for combining the features learned by the classifier C and the feature extraction network F. At the same time, since T2 incorporates the classifier features of the image, T1 and T2 will work together to help G capture the accurate category semantic information corresponding to each type of text. Let x represent a labeled or unlabeled text image sample, and its corresponding label is defined as: T1 distinguishes between real data and generated data based on the category label y. T1 identifies true and false data based on whether the feature distribution of the generated text is consistent with the feature distribution of the real text of the corresponding category. Its expression is: Where Embed(·) represents a function that can learn category label mapping, is a mapping function, represents the weight of the last linear representation layer, Represents a matrix operation. In the above formula, T1(·,·) represents the probability that the discriminator network identifies the input data as real data. The loss function on T1 is expressed as follows: In the feature extractor feature f F and classifier features f C The feature fusion based on channel attention is implemented by using a neural network to calculate the pixel-level weights Q loc and global weights and Q glob , and then fuse the two types of weights as follows: in Represents a broadcast addition operation. By introducing the attention weight Q, T2 obtains instance-level information by receiving the fused features. The specific method is as follows: in Indicates element-by-element multiplication. Under this condition, the feature extractor F needs to learn the same features as the classifier f C Complementary features, T2 also needs to identify real data and generated data, but it does not need category labels, because it contains the classifier feature f C , It can enhance the category separation of the discriminator feature space and promote the alignment of class conditional distributions. The optimization formula on T2 is as follows:

2. The semi-supervised Chinese character image generation method based on semantically guided discriminator according to claim 1, characterized in that: In step S1, since the unlabeled data has no category labels, a classifier C is needed to predict its pseudo labels. The dataset used to train the generative adversarial neural network is in the form of text images plus category labels.

3. The semi-supervised Chinese character image generation method based on semantically guided discriminator according to claim 1, characterized in that: In step S2, the generator needs to generate a text image using a random vector, taking the category label as a condition. The generated text is expressed by the following formula: x z =G(z,y z ) where y z ∈Y represents a random category label, z represents a random vector, z is usually extracted from a distribution that is easy to sample, using a standard Gaussian distribution N(0,1) with a mean of 0 and a variance of 1.

4. The semi-supervised Chinese character image generation method based on semantically guided discriminator according to claim 1, characterized in that: In step S3, ResNet50 is selected as the network structure of classifier C. In order to make classifier C more accurately predict the category of unlabeled data, real data and generated data are used to train the classifier neural network. The training data includes: labeled data-label pairs (x l ,y l ), unlabeled data x u , and generate data-label pairs (x z ,y z ), the corresponding loss function is: Among them, C is the classifier, E represents the expectation, and L CE (·,·) is the cross entropy loss function, which is defined as: L CE (x,y)=-ylogC(x)。 5. The semi-supervised Chinese character image generation method based on semantically guided discriminator according to claim 1, characterized in that: In step S4, in order to compete with the generator to improve the realism of the generated text image, it is necessary to train a discriminator composed of a multi-layer convolutional neural network. The purpose of the discriminator is to distinguish real text from generated text as much as possible under the category label condition, in order to make full use of the instance-level semantic information contained in the classifier features in S3.

6. The semi-supervised Chinese character image generation method based on semantically guided discriminator according to claim 1, characterized in that: In step S5, the generator, discriminator, and classifier need to be jointly trained. The discriminator competes with the generator, and the classifier works with the generator to guide the generator to capture accurate class semantic information. Therefore, the loss function on the generator and discriminator during the adversarial training process is defined as follows: The weighting factors α and β are used to balance the influence of the additional T2 and semantic evaluation items.

Images