A method and device for recognizing Bai script based on zero-shot learning
By employing a zero-shot learning method, generative adversarial networks and variational autoencoders are used to generate Bai ethnic script features, which solves the problem of insufficient Bai ethnic script datasets and achieves high-precision Bai ethnic script recognition with an accuracy rate of 78.14%.
Patent Information
- Application Number
- CN202210666167.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-13
- Publication Date
- 2026-02-06
- Estimated Expiration
- 2042-06-13
AI Technical Summary
Existing deep learning models have low recognition rates when there is insufficient Bai language data, failing to achieve the expected results, and the cost of collecting the data is high.
We employ the zero-shot learning (ZSL) method, using the Chinese character set as the visible class. We generate Bai ethnic character features through generative adversarial networks (GANs) or variational autoencoders (VAEs), construct a classification model, realize the transfer of knowledge from the visible class to the invisible class, and generate a large number of Bai ethnic character datasets.
It improved the accuracy of Bai script recognition, especially when VAE was combined with the ResNet101 classification model, the recognition accuracy reached the highest of 78.14%.
Smart Images
Figure CN115293144B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of character recognition, in particular to a Bai character recognition method and device based on zero-shot learning. BACKGROUND
[0002] The Bai people have a long history and splendid culture. Most of them live in the Dali Bai Autonomous Prefecture in Yunnan Province, and the rest live in various parts of Yunnan, Bijie in Guizhou, Liangshan in Sichuan, Sangzhi County in Hunan, etc., and have their own unique language. The Bai language is the common communication language of the Bai people, an important link for the cohesion of national sentiment, and an important carrier for the development of Bai culture. This is the most basic national characteristic of the Bai people. Because the vocabulary, pronunciation and grammar of the Bai characters are all Chinese and Tibetan-Burmese, the language structure of the Bai characters has very important academic value, and has been widely concerned by the Chinese language community and the national language community at home and abroad for a long time. In order to carry forward the Bai culture and enable people who do not understand Bai characters to read Bai characters on historical documents or stone inscriptions, it is urgent to develop a model capable of recognizing Bai single characters. In this way, when we encounter an unknown Bai character, we can take a photo and then perform model recognition to finally give an explanation of the character.
[0003] With the rapid development of neural networks, great breakthroughs have been made in various visual tasks, but unfortunately, convolutional networks perform significantly worse under insufficient data sets. The Bai character set is not as large as traditional classification data sets, and the collection of the Bai character data set requires a lot of expert knowledge and is relatively costly. Therefore, the deep learning model trained directly using the Bai character set has a low recognition rate when recognizing Bai characters and cannot achieve the expected effect. SUMMARY
[0004] (I) Technical problem to be solved
[0005] To solve the above problems of the prior art, the present application provides a Bai character recognition method and device based on zero-shot learning, which can improve the accuracy of Bai character recognition.
[0006] (II) Technical scheme
[0007] In order to achieve the above purpose, the present application adopts a technical scheme:
[0008] A Bai character recognition method based on zero-shot learning, comprising the steps of:
[0009] S1, constructing a Chinese character set and a Bai character set;
[0010] S2, a generation network of zero-shot learning is constructed, the generation network is trained according to the Chinese character set, and the trained generation network is detected through the Bai character set until the trained generation network is obtained;
[0011] S3, a classification model is constructed, the synthesized sample set containing the Bai character features is output from the trained generation network to the classification model for training, and the trained classification model is obtained, and the trained classification model is used for identifying whether the to-be-recognized character is the Bai character.
[0012] In order to achieve the above purpose, another technical scheme adopted by the present application is:
[0013] A Bai character recognition device based on zero-shot learning, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the following steps when executing the program:
[0014] S1, a Chinese character set and a Bai character set are constructed;
[0015] S2, a generation network of zero-shot learning is constructed, the generation network is trained according to the Chinese character set, and the trained generation network is detected through the Bai character set until the trained generation network is obtained;
[0016] S3, a classification model is constructed, the synthesized sample set containing the Bai character features is output from the trained generation network to the classification model for training, and the trained classification model is obtained, and the trained classification model is used for identifying whether the to-be-recognized character is the Bai character.
[0017] (Three) beneficial effects
[0018] The beneficial effects of the present application are that: by constructing a Chinese character set and a Bai character set, constructing a generation network of zero-shot learning, training the generation network according to the Chinese character set, and detecting the trained generation network through the Bai character set until the trained generation network is obtained, constructing a classification model, outputting the synthesized sample set containing the Bai character features from the trained generation network to the classification model for training, obtaining the trained classification model, and using the trained classification model to identify whether the to-be-recognized character is the Bai character, the problem of insufficient data set is solved by using zero-shot learning (ZSL), the knowledge of the Chinese character set is transferred to the Bai character by using the GAN or VAE generation network, and a large number of Bai character data sets containing class labels are generated, the structural information of the character is automatically obtained, and therefore the accuracy of Bai character recognition is improved. BRIEF DESCRIPTION OF DRAWINGS
[0019] Figure 1 A flow chart of a Bai character recognition method based on zero-shot learning according to an embodiment of the present application;
[0020] Figure 2 A schematic diagram of the overall structure of a Bai character recognition device based on zero-shot learning according to an embodiment of the present application;
[0021] Figure 3 A comparison diagram of Chinese characters and Bai characters;
[0022] Figure 4 Basic strokes of 32 Chinese characters and Bai characters;
[0023] Figure 5 An example of a Bai character data set;
[0024] Figure 6 Visualization results of Bai character features by four generation methods of DAP, IAP, GAN and VAE;
[0025] Figure 7 Performance of GAN and VAE two generation methods when each Bai character class has different number of synthetic features.
[0026]
BRIEF DESCRIPTION OF DRAWINGS
[0027] 1: Bai character recognition device based on zero-shot learning;
[0028] 2: memory;
[0029] 3: processor. DETAILED DESCRIPTION
[0030] In order to better explain the present application, so as to be understood, the present application is described in detail by specific embodiments in combination with the drawings.
[0031] Embodiment one
[0032] Please refer to Figure 1 A Bai character recognition method based on zero-shot learning, characterized in that it comprises the steps of:
[0033] S1, constructing a Chinese character set and a Bai character set;
[0034] Step S1 is specifically:
[0035] Obtaining a data set, and dividing the data set into two mutually disjoint sets, i.e. a Chinese character set and a Bai character set;
[0036] Constructing a ZSL format of visible class Chinese character set and invisible class Bai character set;
[0037] The Chinese character set and the Bai character set each include an instance set, a label set of the instance, and an attribute set.
[0038] S2, constructing a generative network of zero-shot learning, training the generative network according to the Chinese character set, and detecting the trained generative network through the Bai character set until a trained generative network is obtained;
[0039] S3, constructing a classification model, outputting a synthetic sample set containing Bai character features from the trained generative network to the classification model for training, and obtaining a trained classification model, wherein the trained classification model is used for identifying whether a to-be-recognized character is a Bai character.
[0040] The generative network is a generative adversarial network.
[0041] The classification model is one of AlexNet, VGG19, and ResNet101.
[0042] Embodiment Two
[0043] The difference between this embodiment and Embodiment One is that the generative network of zero-shot learning is a variational autoencoder VAE.
[0044] Embodiment Three
[0045] This embodiment will further illustrate how the above-mentioned Bai character recognition method based on zero-shot learning is implemented in combination with a specific application scenario:
[0046] Zero-shot learning (ZSL) is used to solve the problem of difficulty in collecting a Bai character dataset. ZSL can transfer knowledge from visible classes to invisible classes through attributes, thereby alleviating the problem of lack or even loss of invisible class samples. The present application regards the Bai character as an invisible class and collects a dataset similar to the Bai character as a visible class, so as to better transfer knowledge. It is found that Chinese and Bai language have high similarity and belong to the Sino-Tibetan language family, see Figure 3 . In addition, the Chinese character dataset is easy to collect and has low cost without the need for expert knowledge. Therefore, a large Chinese character dataset is collected as a visible class in ZSL. In addition, ZSL needs attributes that can describe visible classes and invisible classes in order to better transfer knowledge through attributes. It is worth mentioning that, just like English is composed of 26 letters, Chinese characters and Bai characters are both composed of 32 basic strokes, see Figure 4 . Therefore, it is reasonable to take the number of different strokes of each word as an attribute.
[0047] 1. Obtain a dataset, and divide the dataset into two disjoint sets, i.e., a Chinese character set D s ={X s , Ys A s} and white character set D u ={X u ,Y u ,A u};
[0048] Constructing ZSL format visible Chinese character set and invisible white character set;
[0049] Wherein, X s and X u are instance sets, respectively for training and testing, Y s and Y u are label sets of instances, A s and A u are corresponding attribute sets.
[0050] 2. Learning intermediate attribute classifier
[0051] Direct attribute prediction (DAP) and indirect attribute prediction (IAP) are the earliest ZSL methods that use attributes to predict instance labels; attributes represent any trained Chinese character class, and c represents a vector of length x.
[0052] 3. Generation network
[0053] The generative adversarial network (GAN) is divided into two stages of feature generation and classification;
[0054] In feature generation, the conditional generator is trained to synthesize samples and attributes a conditioned on Gaussian noise z∈N(0,1), the discriminator is cross-iteratively trained with the generator, and the learner distinguishes between real pairs (x,a) and synthesized pairs The generator tries to synthesize more realistic features To confuse the discriminator during training. In addition, the generator expects to match the synthesized features with their corresponding attributes a.
[0055] The generative model adopts the structure of WGAN and introduces a gradient penalty term, and the adversarial training loss function of the generator and the discriminator is as follows:
[0056]
[0057] Wherein, E represents the expected value. The generator is trained on the Chinese character set, and the trained generator is used to output a synthesized sample set containing white character features.
[0058] In the classification stage, the classification model is trained using the synthetic sample set to obtain a trained classification model, which is used for identifying whether the to-be-identified character is a Bai character or not, and the cross-entropy loss function thereof is as follows:
[0059]
[0060] wherein, is the weight matrix of the full connection layer, and R represents the real number field. x is the dimension of x, and L is the class number of the Bai character, The prediction function formula is as follows:
[0061] f(x)=argmax b P(b|x;theta) (3)
[0062] In addition to the GAN-based generation framework, the application proposes a generation network based on a variational autoencoder (VAE), which is more suitable for character recognition tasks. Without a discriminator, the VAE-based framework is composed of an encoder E n and a decoder D e (generator).
[0063] In feature generation, the encoder is trained by a Kullback-Leibler divergence (KLD) loss function, and the formula is as follows:
[0064]
[0065] wherein mu and delta and are the outputs of the encoder, and represent the mean and variance, respectively. d is the dimension of mu and delta and. By resetting the parameters, the hidden variable coded by the encoder can be represented as:
[0066]
[0067] At the same time, the decoder D e is trained together with the encoder E n , and the loss function used is as follows:
[0068]
[0069] wherein The loss function can make the synthesized features have better structural information. Therefore, compared with GAN, VAE is more suitable for character recognition which pays more attention to structure. In feature generation, the total loss function formula of VAE is as follows:
[0070]
[0071] where λ is a hyper-parameter. After training, the decoder synthesizes visual features of Bai characters under the condition of satisfying the attributes and Gaussian noise, to obtain a large number of synthetic sample sets containing Bai character features. In terms of classification, based on the GAN classification method, the formula 2 is used to train the classifier by using the synthetic features, and the formula 3 is used for prediction.
[0072] 4. Experiment
[0073] Dataset: A large Bai character dataset is established by the present application, as shown in the following table. Figure 5 There are a total of 400 Bai characters. Since there is a certain overlap between Bai characters and Chinese characters, the present application only selects Bai characters that are greatly different from Chinese characters to establish this dataset. Each character has about 50 samples, which are written by Bai people and Bai culture enthusiasts. In addition, the present application also collects a large amount of Chinese character data. The dataset consists of 509 Chinese characters, each with about 1000 samples. In order to establish a ZSL format dataset, both the Chinese character and Bai character datasets are labeled with class-level attributes (32 basic strokes). The Chinese character dataset is used as the training set, and the Bai character dataset is used as the test set.
[0074] Evaluation protocol: The proposed method is evaluated from the perspective of average per-class Top1 accuracy (ACC).
[0075] Classification model: The present application uses three backbone models AlexNet, VGG19 and ResNet101. Through comparison of multiple models, it can be seen that the method is effective for all models and has strong generalization ability.
[0076] 5. Precision analysis
[0077] The accuracy comparison of different training strategies is shown in Table 1.
[0078] Method AlexNet VGG19 ResNet101 None 40.16 43.42 55.17 DAP 46.28 49.92 53.84 IAP 42.23 47.26 48.59 GAN 67.54 70.78 76.46 VAE 70.47 74.81 78.14
[0079] Table 1 Precision comparison of different methods, None indicates that the model is directly trained using the Bai character data set.
[0080] The accuracy of the first row represents the accuracy of the model trained directly using part of the Bai character dataset as the training set. It is not difficult to find that due to the small size of the training set, the model performance is poor and has no practical value.
[0081] The second and third rows used DAP and IAP prediction methods. Since the model was trained using only a Chinese character dataset, it had never encountered Bai characters. Therefore, although the model was improved by using knowledge transfer, it was still not ideal. This also shows that using ZSL for knowledge transfer can transfer knowledge from large-scale Chinese character datasets to Bai characters, thus enabling the model to achieve a high Bai character recognition rate even without Bai character data for training.
[0082] GAN is a generative method. Because this generator integrates a large number of Bai ethnic script features, it greatly alleviates the problem of missing Bai script data, ultimately significantly improving accuracy. This shows that although Chinese characters and Bai characters differ, the generator can reconstruct Bai script features well even when trained only on Chinese characters, through stroke attributes.
[0083] VAE is also a generative method that not only possesses the feature-generating capabilities of GANs but also the ability to reconstruct the original image. This capability plays a crucial role in the transfer of textual knowledge, enabling the method of this invention to achieve the highest accuracy.
[0084] 6. Visualization
[0085] To further illustrate the advantages and disadvantages of different methods, this invention directly extracted features of 30 Bai ethnic characters using a pre-trained network (ResNet101). These features were then visualized using the t-SNE algorithm, as shown below. Figure 5 , 6 As shown, networks trained directly with Bai characters suffer from low feature recognition rates due to the limited training set. However, with the support of a massive Chinese character dataset, Bai character features extracted using networks trained with DAP and IAP exhibit high discriminative power, demonstrating the effectiveness of knowledge transfer. Furthermore, Bai character features extracted through generative methods further improve the recognition rate. Finally, Bai character features extracted from networks trained with VAE achieve excellent discrimination, showcasing the superiority of VAE in character reconstruction.
[0086] 7. Hyperparameter Analysis
[0087] exist Figure 7 This invention reports the results of two generation methods, each with a different number of synthetic features for each Bai script class. It can be observed that the generation methods require a certain number of synthetic samples to achieve the desired results, but this does not mean that more synthetic samples are always better. The noise generated by synthetic samples is too great, limiting classification performance. Therefore, to obtain ideal Bai script recognition results, the number of synthetic samples for each Bai script class needs to be limited to a certain range.
[0088] Figure 7Performance of GAN-based and VAE-based generation methods when each Bai script character class has a different number of synthetic features. ACC is the average TOP-1 accuracy per class.
[0089] 8、Conclusion
[0090] As a nation with a long history, Bai has not only its own language, but also created a splendid culture. However, with the development of the times, as fewer and fewer people understand Bai script, Bai culture is on the verge of extinction. In order to enable people who love Bai culture and relevant researchers to read Bai literature smoothly, the present application mainly discloses how to train a high-precision Bai script recognition model. First of all, the present application establishes a Bai script data set, but due to the limitation of expert knowledge demand, the size of the data set is limited. Therefore, those deep learning models that need a large amount of data to drive cannot achieve satisfactory results on this data set. In order to solve this problem, zero-shot learning (ZSL) is used to solve the problem of insufficient data set, taking Chinese characters as the visible class and Bai script as the invisible class, using GAN or VAE generation network to transfer the knowledge of Chinese character set to Bai script, and generating a large number of Bai script data set containing class labels, realizing the automatic acquisition of character structure information, thereby improving the accuracy of Bai script recognition. Among them, when the generation network VAE is combined with the classification model ResNet101, the recognition accuracy is the highest.
[0091] Embodiment Four
[0092] Please refer to Figure 2 A Bai script recognition device 1 based on zero-shot learning, comprising a memory 2, a processor 3 and a computer program stored in the memory 2 and executable on the processor 3, wherein the processor 3 implements each step in embodiments one and two when executing the program.
[0093] The above description is only an embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent transformation or direct or indirect application in related technical fields based on the content of the present application specification and drawings is also included in the patent protection scope of the present application.
Claims
1. A method for recognizing Bai script based on zero-shot learning, characterized in that, The method comprises the steps of: S1, constructing a visible class Chinese character set in a zero-shot learning format and an invisible class Bai character set, and taking the number of strokes as the attribute of the visible class and the invisible class; S2, constructing a generation network, training the generation network according to the Chinese character set, and testing the generation network through the Bai character set until a trained generation network is obtained; S3, constructing a classification model, outputting a synthesized sample set containing Bai character features from the trained generation network to the classification model for training, and obtaining a trained classification model, wherein the trained classification model is used for identifying whether a to-be-recognized character is a Bai character. 2.The method of claim 1, wherein the method comprises: The Chinese character set and the Bai character set are two disjoint sets. 3.The method of claim 1, wherein the method further comprises: The generation network is a generative adversarial network. 4.The method of claim 1, wherein, The generation network is a variational autoencoder. 5.The method of claim 1, wherein the method further comprises: The classification model is one of AlexNet, VGG19 and ResNet101.
6. A zero-shot learning-based Bai script recognition device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the following steps when executing the program: S1, constructing a visible class Chinese character set in a zero-shot learning format and an invisible class Bai character set, and taking the number of strokes as the attribute of the visible class and the invisible class; S2, constructing a generation network, training the generation network according to the Chinese character set, and testing the generation network through the Bai character set until a trained generation network is obtained; S3, constructing a classification model, outputting a synthesized sample set containing Bai character features from the trained generation network to the classification model for training, and obtaining a trained classification model, wherein the trained classification model is used for identifying whether a to-be-recognized character is a Bai character. 7.The white character recognition device based on zero-shot learning according to claim 6, characterized in that, The Chinese character set and the Bai character set are two disjoint sets. 8.The white character recognition device based on zero-shot learning according to claim 6, characterized in that, The generation network is a generative adversarial network. 9.The white character recognition device based on zero-shot learning according to claim 6, characterized in that, Step S2 is specifically: The generation network is a variational autoencoder. 10.The white character recognition device based on zero-shot learning according to claim 6, characterized in that, The classification model is one of AlexNet, VGG19 and ResNet101.