A component-splitting-based zero-shot Chinese character recognition method
By employing a component-based zero-sample Chinese character recognition method, which utilizes hierarchical decomposition of Chinese character structure and neural network structure, the problems of sample imbalance and high complexity in Chinese character recognition are solved, achieving accurate recognition and efficient training of rare characters.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TONGJI UNIV
- Filing Date
- 2022-09-20
- Publication Date
- 2026-07-31
AI Technical Summary
Existing technologies for Chinese character recognition suffer from problems such as uneven sample distribution, large data requirements, high model complexity, and low recognition accuracy, especially in the recognition of rare characters.
We adopt a zero-shot recognition method based on component decomposition. By redefining the hierarchical decomposition strategy of Chinese character structure, we construct a Chinese character encoder, feature decoder and sequence converter. We use convolutional neural networks and recurrent neural networks to extract image and component features, and combine dictionary matching and nearest neighbor algorithms to recognize Chinese characters.
It enables the recognition of rare characters, improves the training efficiency and recognition accuracy of the model, eliminates the dependence on a large number of samples, simplifies the model structure, and increases the recognition speed.
Smart Images
Figure CN115512357B_ABST
Abstract
Description
Technical Field
[0001] It involves the field of Chinese character recognition. Background Technology
[0002] Chinese characters are an important part of the vast Chinese culture. As a carrier of information, each Chinese character possesses unique semantic information. As a pictographic script, Chinese characters are extremely numerous. According to the latest information technology Chinese character encoding set GB19030-2005, there are nearly 3,000 commonly used Chinese characters, and the entire dataset contains as many as 70,244 characters. Recognizing each of these characters is extremely difficult for the average person; therefore, the idea of using computers to recognize Chinese characters arose. Early methods relied on the number of database samples, but the existence of rare characters caused a serious imbalance in the distribution of Chinese character samples, leading to biases in the training results of the models. Under these circumstances, training a zero-shot model for recognizing Chinese characters became the inevitable trend.
[0003] Currently, Chinese character recognition methods are mainly divided into three categories. The first is to use traditional machine learning methods for recognition. The second is to use deep learning to recognize each Chinese character as a whole. The third is to use the idea of component decomposition to decompose Chinese characters and then use the first two methods for recognition. The three are described in detail below.
[0004] First, traditional machine learning methods. These methods primarily rely on manually extracting features from Chinese characters and then using mathematical methods and classifiers to categorize these features and obtain the recognition results. As one of the earliest types of machine learning algorithms, traditional machine learning methods are easy to implement and interpret, but they have significant drawbacks: 1) Traditional machine learning methods require manual extraction of Chinese character features, and the quality of feature design directly affects the accuracy of the final prediction. 2) Most classifiers struggle with large amounts of data, often leading to overfitting. 3) Traditional machine learning methods suffer from low classification accuracy in multi-class problems, which are particularly problematic for Chinese character recognition, which involves a large number of classifications.
[0005] Second, deep learning-based overall Chinese character recognition. This type of method mainly relies on the advantages of convolutional neural networks in image feature extraction, which allows deep learning to achieve better recognition accuracy than traditional methods. However, this method still has the following fatal drawbacks: 1) Deep learning depends on a large amount of data, requiring a high number of samples in each class of the dataset. As a multi-class classification problem, Chinese character recognition requires more data than other few-class classification problems. 2) Deep learning methods are easily affected by the distribution of classes in the dataset. Whether the number of samples in each class is balanced and whether the sample data are similar will affect the final recognition accuracy.
[0006] Third, there are recognition methods based on Chinese character decomposition. This type of method adds a special structure to break down each Chinese character into its radicals and even strokes. Then, based on the first and second methods mentioned above, it classifies the radicals and strokes, and finally, through recombination of the results, maps them to obtain the corresponding recognition result. Because the number of radicals is still extremely rich, some radicals may not appear in the training set during the decomposition process, ultimately leading to the inability to correctly recognize the entire Chinese character. Therefore, how to develop a radical dataset that can constitute all Chinese characters has become a major challenge for this method.
[0007] 1.1 Relevant Existing Technology
[0008] "A Zero-Sample Chinese Character Recognition Method Based on Character Roots", Patent Publication No.: CN112508108A.
[0009] CN112508108A utilizes convolutional neural network (CNN) and recurrent neural network (RNN) techniques to construct a bidirectional parallel network structure that extracts image features of Chinese characters and attribute features of character root sequences, respectively. These two features are then connected through an objective function, resulting in a Chinese character recognition model. The specific steps are as follows:
[0010] First, this invention is based on the Chinese Text Dataset in the Wild (CTW) dataset. Training and test sets are constructed according to the frequency of occurrence of Chinese characters, with the most frequent characters used as the training set and the least frequent as the test set. All radicals contained in the images are collected from the dataset, and each unique radical is uniquely encoded. Second, the invention uses two parallel network structures. One branch uses a convolutional neural network to extract image features from the Chinese character images, while the other uses a recurrent neural network to extract attribute features from the corresponding radical sequence. The two network structures are linked by a loss function, and the network is optimized during training to make their feature vectors similar. Finally, during testing, the input Chinese character image can be located to the relevant radical sequence through the network's feature extraction and nearest neighbor distance function, thus obtaining the corresponding result.
[0011] The first major drawback of the CN112508108A technology lies in its splitting method. This invention uses the five-stroke root table as the standard for root splitting. The relatively large number of similar roots in the five-stroke root table greatly increases the difficulty of root classification. Similarly, the five-stroke method is suitable for constructing Chinese characters rather than splitting them. Take the character '秦' as an example. Humans obtain its splitting sequence as '三、人、禾' through prior knowledge of five-stroke input, while a machine without corresponding prior knowledge may split it into the sequence '一、夫、禾'. Among them, '一', '夫', and '禾' are all legitimate roots in five-stroke input. This will undoubtedly lead to a decrease in recognition results and the loss of model interpretability.
[0012] The second drawback of this invention is that the method it uses is to match similar feature vectors in an embedded space matching mode. For this method, the model needs to store all vectors in advance, and this step requires a large amount of storage space, which in turn leads to the bloat of the model. The overly large number of embedded vectors will result in low efficiency and a reduced matching success rate during the model matching process. Summary of the Invention
[0013] Based on the research status presented in the above background technology, the present invention proposes a brand-new zero-shot Chinese character recognition method based on component splitting, and the summary of the invention is as follows:
[0014] In view of the current situation of a large number of Chinese characters and the limitation of uneven sample distribution, the present invention proposes a zero-shot recognition method based on component splitting. By defining a brand-new Chinese character splitting method and combining prior knowledge to design the network structure for component recognition, an end-to-end zero-shot Chinese character recognition model is obtained, which has obvious advantages in accuracy and efficiency compared with the existing methods.
[0015] Technical Solution
[0016] A zero-shot Chinese character recognition method based on component splitting proposed by the present invention is outlined as follows:
[0017] First, construct the datasets 'Chinese Character Dataset' and 'Prototype Dataset': The present invention selects one Chinese character picture from each category of Chinese characters in the offline handwritten sample database CASIA-HWDB (hereinafter referred to as HWDB) of the Institute of Automation, Chinese Academy of Sciences to form the 'Prototype Dataset', and uses the remaining part of the HWDB dataset as the 'Chinese Character Dataset', and divides it into a training set, a validation set, and a test set according to the ratio of 8:1:1.
[0018] Secondly, the components were encoded, and a mapping dictionary (hereinafter referred to as the dictionary) was constructed between Chinese characters and their components. This dictionary records the Chinese characters and their corresponding component decomposition sequences. Compared with the decomposition method based on Wubi input method, this invention proposes a hierarchical decomposition strategy based on the structure of Chinese characters. The specific strategy is as follows: Chinese characters are defined into twelve structures, and different decomposition methods are implemented for each structure. For a Chinese character component that can still be decomposed further, it will continue to be iteratively decomposed until it cannot be decomposed further. After all components cannot be further decomposed, the sequence of all components is the final Chinese character decomposition sequence.
[0019] Finally, a Chinese character recognition model is constructed: this model has three structures: a 'Chinese character encoder', a 'feature decoder', and a 'sequence converter', and their respective functions are as follows:
[0020] 1. The Chinese character encoder is used to extract the image feature map of the input Chinese character. The structure of the Chinese character encoder is composed of a convolutional neural network, and the feature matrix of the image is represented by θ(x).
[0021] 2. The feature decoder is used to output the feature matrix obtained by the Chinese character encoder as the corresponding sequence of Chinese character components. This part of the structure mainly consists of a recurrent neural network layer based on an attention mechanism: this layer extracts local structural features to convert the Chinese character image into a sequence of components.
[0022] During the training phase, the probabilistic features of the component sequence of Chinese characters are represented as P. x ={p1,p2,p3,……,p N}, where p i The vector represents the probability predicted by the model for each component, i∈[1,N], where N is the number of components in the sequence, and the component sequence corresponding to the input Chinese character image is represented as G. x ={g1,g2,g3,……,g N} where N is the number of components within the sequence. The model is trained using a loss function, which is calculated as follows: p ii This represents the probability of an accurate prediction in the sequence.
[0023] During the testing phase, the model will make predictions based on the probability features of the component sequences obtained from the Chinese character encoder and obtain the predicted sequences. Where N is the number of components within the sequence. This sequence will be used in subsequent sequence converters.
[0024] 3. The sequence converter is used only during the testing phase. This structure is used to convert the component sequences obtained from the feature decoder into specific Chinese characters according to a dictionary. The conversion process can be divided into three cases: 1) When the sequence can match a specific Chinese character in the dictionary, the converter outputs that Chinese character. 2) When the sequence cannot match a specific Chinese character in the dictionary, the vector nearest neighbor algorithm function is used to obtain the final result, calculated as follows: Where R dic Let D(x1,x2) be the set of candidate vectors in the dictionary, and let D(x1,x2) be the distance calculation formula. 3) When the sequence matches multiple Chinese characters, the feature maps of the candidate Chinese characters will be obtained through the Chinese character encoder, and the final prediction result will be obtained by comparing them with the feature maps of the test Chinese character images according to the nearest neighbor algorithm. The calculation method is as follows: Where R can This is the set of candidate vectors corresponding to the same sequence.
[0025] The distance calculation formula is as follows: This invention utilizes the cosine distance.
[0026] Therefore, the zero-shot Chinese character recognition method based on component decomposition proposed in this invention is significant in the following ways: First, it can identify rare Chinese characters that people do not recognize and obtain their semantic information; second, it overcomes the limitations of traditional datasets that are limited by a large number of balanced samples and cannot obtain rare character datasets; finally, the model architecture of decomposition before recognition greatly improves the training efficiency of the model while ensuring the accuracy of recognition. Attached Figure Description
[0027] Figure 1 These are the ten types of Chinese character structures in the hierarchical decomposition strategy proposed in this scheme.
[0028] Figure 2 This solution proposes a hierarchical decomposition strategy based on the structure of Chinese characters.
[0029] Figure 3 This is the model structure proposed in this scheme.
[0030] Figure 4 This is the heatmap obtained by the feature decoder proposed in this scheme.
[0031] Figure 5 This is the training process of this scheme.
[0032] Figure 6 This is a diagram illustrating the first scenario in the testing process of this solution.
[0033] Figure 7 This is a diagram illustrating the second scenario in the testing process of this solution.
[0034] Figure 8 This is a diagram illustrating the third scenario in the testing process of this solution. Detailed Implementation
[0035] The technical solution of the present invention will be further described below with reference to the accompanying drawings.
[0036] A zero-sample Chinese character recognition method based on component decomposition is characterized by redesigning the Chinese character decomposition strategy to perform Chinese character recognition by decomposing the Chinese characters.
[0037] The first part involves redesigning the hierarchical decomposition strategy based on Chinese character structure and constructing the dataset.
[0038] like Figure 1 , Figure 2 As shown.
[0039] like Figure 1 The present invention provides a schematic diagram of the structure of Chinese characters. It categorizes Chinese characters and their components into one basic structure and ten composite structures. The basic structure consists of indivisible single-component characters, while the composite structures are composed of multiple components. Each composite structure has its unique decomposition rules. When a Chinese character or its component is classified into one of these composite structures, the character or component will be decomposed into several components according to the decomposition rules. For example... Figure 2 The diagram illustrates a hierarchical character segmentation strategy. A Chinese character is recursively segmented using this strategy until all resulting components are indivisible basic structures. This hierarchical character segmentation strategy constructs a mapping dictionary from Chinese characters to sequences of components formed by those characters, laying the foundation for subsequent recognition steps.
[0040] The model proposed in this invention uses the HWDB dataset. We constructed a 'Chinese character dataset' and a 'prototype dataset' by partitioning the HWDB dataset: First, we selected one Chinese character image from each category of Chinese characters in the HWDB dataset to form the 'prototype dataset'. Second, we used the remaining part of the HWDB dataset as the 'Chinese character dataset' and divided it into training and test sets according to the specified proportions.
[0041] Part Two: Introduction to the Model Structure
[0042] like Figure 3 As shown in the model structure diagram, the model structure of this invention is mainly divided into three modules: 'Chinese character encoder', 'feature decoder', and 'sequence converter'.
[0043] The Chinese character encoder is used to extract the image feature map of the input Chinese character. The structure of the Chinese character encoder consists of three connected convolutional blocks with residual structures.
[0044] The feature decoder outputs the feature matrix obtained from the Chinese character encoder as a sequence of corresponding Chinese character components. This part of the structure mainly consists of a recurrent neural network layer based on an attention mechanism: this layer extracts local structural features to transform the Chinese character image into a sequence of components. The attention visualization obtained from this layer is shown below. Figure 4 As shown.
[0045] The sequence converter is used only during the testing phase. This structure is used to convert the component sequences obtained from the feature decoder into specific Chinese characters based on a dictionary. The conversion process can be divided into three cases: 1) When the sequence can match a specific Chinese character in the dictionary, the converter outputs that Chinese character. 2) When the sequence cannot match a specific Chinese character in the dictionary, the vector nearest neighbor algorithm is used to match the most similar component sequence in the dictionary. 3) When the sequence matches multiple Chinese characters, we send the candidate Chinese characters to the Chinese character encoder to obtain the feature maps of the candidate Chinese characters, and then compare the feature maps of the test Chinese characters with the feature maps of the candidate Chinese characters in turn according to the nearest neighbor algorithm to obtain the closest result.
[0046] Part Three: Model Process Introduction
[0047] Training process as follows Figure 5 The training flowchart shows that the training process involves two main modules: a Chinese character encoder and a feature decoder. For a training sample x, the Chinese character image is encoded by the Chinese character encoder to obtain its corresponding Chinese character feature map θ(x). The feature map θ(x) is then fed into the feature decoder to obtain the vector sequence P. x ={p1,p2,p3,……,p N}, where p i The vector represents the probability that component i is predicted as any component, and N is the number of components in the sequence. Simultaneously, the component sequence vector G corresponding to the input Chinese character in the dictionary is also represented. x ={g1,g2,g3,……,g N} as the actual label of the input image, and minimize the cross-entropy loss function. Based on this, the Chinese character encoder and feature decoder are trained and optimized using the backpropagation algorithm. In the cross-entropy loss function, P(g) k ) represents the predicted sequence p k The result is g k The probability of.
[0048] The testing process is as follows Figure 6 , Figure 7 , Figure 8As shown, the images above correspond to three different scenarios in the testing process. The common elements in all three scenarios are as follows: For an input test Chinese character sample, the sample is encoded by a character encoder to obtain its corresponding Chinese character feature map θ(x). The feature map θ(x) is then fed into the feature decoder to obtain the vector sequence P. x ={p1,p2,p3,……,p N}, based on the vector sequence P x For each value in the vector, select the component corresponding to the value with the highest probability and form a predicted component sequence. Where N is the number of components within the sequence.
[0049] The sequence converter is for predicting component sequences. There are three possible solutions:
[0050] Scenario 1: Sequence It can match specific Chinese characters in the dictionary. At this point, the sequence converter will use the specific Chinese character matched by the dictionary output as the test result for the test Chinese character image sample.
[0051] Scenario 2: Sequence No matching Chinese character can be found in the dictionary. In this case, the sequence converter will use the nearest neighbor algorithm to match component sequences of the same length and output the nearest component sequence. The nearest distance vector sequence is calculated as follows: Where R dic Let D(x1,x2) be a set of vectors of the same length in the dictionary, and let D(x1,x2) be the distance calculation formula. Finally, based on the number of matched Chinese characters in the dictionary, we are divided into two cases: when only one Chinese character is matched, the sequence converter will use the matched specific Chinese character as the test result of the test image sample; when more than one Chinese character is matched, the sequence converter will calculate and output the result according to the processing method of scenario three.
[0052] Scenario 3: Sequence It can match multiple Chinese characters in the dictionary. At this point, we use the matched Chinese characters as the candidate sequence and the corresponding prototype Chinese character image sequence c = (x′1, x′2, x′3, ..., x′) in the prototype dataset. M The candidate sequences are selected by M, where M is the length of the candidate sequence. Next, the corresponding prototype Chinese character images are fed into the already trained Chinese character encoder to obtain the feature map sequence C = (θ(x′1), θ(x′2), θ(x′3), ..., θ(x′...). M), where M is the length of the feature sequence. The sequence converter compares the distance between the candidate feature map sequence C and the feature map θ(x) of the test sample using the nearest neighbor algorithm, and outputs the Chinese character label corresponding to the nearest candidate feature map as the final output of the model.
[0053] In this invention, the distance calculation formula is as follows: The present invention utilizes the cosine distance.
[0054] Advantages of this technical solution
[0055] This solution can ultimately be used in the field of Chinese character recognition, and has the following two advantages:
[0056] 1. The model requires minimal training; only the feature extractor and feature decoder need to be trained. The model runs relatively quickly.
[0057] 2. By splitting Chinese characters into components, zero-shot recognition can be achieved. When the model encounters Chinese character samples not in the training set, it can still convert Chinese characters into component sequences based on the feature extractor and feature decoder in the model, and finally obtain the result based on the dictionary.
Claims
1. A zero-shot Chinese character recognition method based on component decomposition, characterized in that, First, construct the dataset 'Chinese character dataset' and the 'prototype dataset'; select one Chinese character image from each category of Chinese characters in the Chinese character dataset to form the 'prototype dataset'. Secondly, the components were encoded and a mapping dictionary (referred to as the dictionary) of Chinese characters and their components was constructed. This dictionary records the Chinese characters and their corresponding component decomposition sequences. A hierarchical decomposition strategy based on the structure of Chinese characters was adopted. The specific strategy is as follows: Chinese characters are defined as twelve structures. For each structure, a different decomposition method is implemented. For a Chinese character component that can still be decomposed further, it will continue to be iteratively decomposed until it cannot be decomposed further. After all components cannot be further decomposed, the sequence of all components is the final Chinese character decomposition sequence. Finally, a Chinese character recognition model is constructed for recognition: this model has three structures: a 'Chinese character encoder', a 'feature decoder', and a 'sequence converter'; the Chinese character encoder is used to extract the image feature map of the input Chinese character, and its structure is composed of a convolutional neural network, with the image feature matrix using... The feature decoder is used to output the feature matrix obtained by the Chinese character encoder as the corresponding sequence of Chinese character components. This part of the structure is mainly a recurrent neural network layer based on the attention mechanism: this layer extracts local structural features to convert Chinese character images into a sequence of components. During the training phase, the probabilistic features of the component sequences of Chinese characters are represented as follows: ,in The vector represents the probability predicted by the model for each component, i∈[1,N], where N is the number of components in the sequence, and the component sequence corresponding to the input Chinese character image is represented as... , where N is the number of components within the sequence; The model is trained by a loss function, which is calculated as , is the probability of the prediction being accurate in the sequence. In the test phase, the model will make predictions based on the component sequence probability features obtained by the Chinese character encoder and obtain the predicted sequence where N is the number of components in the sequence; the sequence will be used in the subsequent sequence converter; The sequence converter is used only during the testing phase. This structure is used to convert the component sequences obtained from the feature decoder into specific Chinese characters according to a dictionary. The conversion process is divided into three cases: 1) When a sequence matches a specific Chinese character in the dictionary, the converter outputs that Chinese character; 2) When the sequence cannot match a specific Chinese character in the dictionary, the vector nearest neighbor algorithm function is used to obtain the final result, and the calculation method is as follows: wherein is a candidate vector set in the dictionary, is a distance calculation formula; 3) When a sequence matches multiple Chinese characters, the feature maps of candidate Chinese characters are obtained through a Chinese character encoder. These feature maps are then compared with the feature maps of the test Chinese character images using the nearest neighbor algorithm to obtain the final prediction result. The matched Chinese characters are used as candidate sequences, along with their corresponding prototype Chinese character image sequences in the prototype dataset. The candidate sequences are selected, where M is the length of the candidate sequence; the corresponding prototype Chinese character images are fed into the already trained Chinese character encoder to obtain the feature map sequence. Where M is the length of the feature sequence; the sequence converter compares the candidate feature map sequence C with the feature map of the test sample using the nearest neighbor algorithm. The distance between them is compared, and the Chinese character label corresponding to the feature map of the nearest candidate is output as the final output of the model; The distance calculation formula , adopts the cosine distance therein; The prediction result is calculated as follows: wherein is a candidate vector set in corresponding to the same sequence.
2. The identification method of claim 1, wherein: One Chinese character image was selected from each category of Chinese characters in the CASIA-HWDB (hereinafter referred to as HWDB) offline handwritten sample database of the Institute of Automation, Chinese Academy of Sciences, to form a 'prototype dataset'. The remaining part of the HWDB dataset was used as the 'Chinese character dataset' and divided into training set, validation set and test set in a ratio of 8:1:1.