Font adaptive Chinese character recognition method based on radical detection and splitting
Through the font adaptive Chinese character recognition method based on radical detection and splitting, and combined with supervised learning and weakly supervised learning, the recognition problem caused by imbalance in frequency distribution of Chinese character radicals is solved, and the model's generalization ability and high recognition rate recognition effect of multiple fonts is achieved.
Patent Information
- Application Number
- CN202411589164.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-08
- Publication Date
- 2025-03-14
AI Technical Summary
In the prior art, the frequency distribution of Chinese characters radicals is unbalanced, resulting in inaccurate identification of low-frequency radicals. At the same time, the model's generalization ability and data annotation cost of multiple font data are difficult to take into account.
The font adaptive Chinese character recognition method based on radical detection and splitting is adopted. By constructing a multi-font Chinese character radical detection network model combining supervised learning and weak supervision learning, the radical classification module, radical detection module and weak supervision learning module are used, and the model is iteratively trained with improved loss functions, and the Chinese character character image similarity and radical position similarity are integrated to predict Chinese characters.
It effectively reduces the training cost of Chinese characters that do not appear in the training set, improves the model's generalization ability of multiple fonts, reduces the cost of manual data labeling, enhances the recognition ability of low-frequency radicals, and realizes high recognition rate Chinese character recognition.
Smart Images
Figure CN119625742A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of image recognition, and in particular relates to a font-adaptive Chinese character recognition method. Background Art
[0002] Existing Chinese character recognition methods based on neural network models can be roughly divided into three categories: classification-based methods, radical-based methods, and stroke-based methods. Classification-based methods treat Chinese character recognition as a multi-classification task, with each Chinese character corresponding to a category. This method is more suitable for the recognition of English characters, but there are too many Chinese characters, with more than 7,000 common Chinese characters. As a result, the parameters of the last fully connected layer of the classification model are too large, and the model migration cost is high for the recognition of Chinese characters that do not appear in the training data.
[0003] To address this shortcoming, radical-based methods and stroke-based methods deconstruct Chinese characters into radicals, or further decompose them into strokes. In this way, the number of categories involved in classification is greatly reduced. However, due to the introduction of the structural characteristics of the split, it is necessary to perform corresponding data annotation on the radicals or strokes. In order to improve the generalization ability of the model for different fonts, it is also necessary to use Chinese character images with different writing styles to construct a data set, which further increases the data annotation cost. In addition, since the frequency distribution of the radicals that make up Chinese characters in common Chinese characters is extremely unbalanced, the radical data presents a long-tail distribution. This property results in more high-frequency data and less low-frequency data, which seriously affects the model's recognition accuracy for low-frequency radicals. Summary of the invention
[0004] In order to solve the technical problem of inaccurate recognition of low-frequency radicals caused by unbalanced frequency distribution of Chinese character radicals, while taking into account the generalization ability of the model for a variety of font data and maintaining an acceptable data annotation cost, the present invention provides a "font-adaptive Chinese character recognition method based on radical detection and splitting".
[0005] Font-adaptive Chinese character recognition method based on radical detection and splitting, such as Figure 1 As shown, the following steps are included:
[0006] Step 1: construct a Chinese character image dataset, one part of which is used for supervised learning, and the image label is composed of Chinese character label information and radical label information, and the other part is used for weakly supervised learning, and the image label is composed of Chinese character label information, and the Chinese character image dataset is divided into multiple batch processing data according to the image label;
[0007] Step 2: Based on the Chinese character image dataset in step 1, a multi-font Chinese character radical detection network model combining supervised learning and weakly supervised learning is constructed, and batch data is input. The network model includes a model backbone module, a radical classification module, a radical detection module, and a weakly supervised learning module. The model backbone module is used to extract features of the input Chinese character image to obtain a high-dimensional feature representation of the Chinese character image; the radical classification module is used to predict the radical category for data with radical label information; the radical detection module is used to predict the radical position for data with radical label information; the weakly supervised learning module is used to compare the data with radical label information and the data without radical label information to achieve weakly supervised learning of data without radical label information;
[0008] Step 3: Iteratively train the model using the loss function improved for the distribution characteristics of Chinese character radicals;
[0009] Step 4: Detect radicals of Chinese characters and obtain radical category and position information;
[0010] Step 5. Integrate the Chinese character image similarity and radical position similarity to predict Chinese characters, select a font data set with radical label information to build a Chinese character category query library; use the radical category prediction results obtained in step 4 as a candidate set; convert the test image and the images of Chinese characters in the candidate set into single channels respectively, calculate the image similarity and radical position similarity and integrate them, and use the category corresponding to the Chinese character with the highest matching score as the Chinese character recognition result for the test character image.
[0011] Technical effects:
[0012] 1. A radical-based Chinese character recognition method is adopted. For new Chinese characters that do not appear in the training set, there is no need to retrain the model. New Chinese characters are predicted by known radical categories. Compared with the classification-based Chinese character recognition method, the training cost can be greatly reduced. At the same time, a weakly supervised learning module is added. Through comparative learning between data with radical position label information and data without radical position label information, weakly supervised learning for unlabeled data is achieved, avoiding radical position labeling for multiple fonts, reducing the cost of manual data labeling, and improving the model's generalization ability for multiple fonts.
[0013] 2. Improve the recognition ability of low-frequency radicals by improving the loss function L2. Due to the serious imbalance in the frequency distribution of different types of radicals in Chinese characters, the model can better recognize high-frequency radicals after learning, but the recognition accuracy of low-frequency radicals is poor. In order to increase the proportion of the loss of less frequent radicals in the loss and make up for the defect of insufficient data of low-frequency radicals, the weight term in the improved focus loss function of the present invention is combined with the characteristics of radicals: The technical problem of inaccurate recognition of low-frequency radicals caused by unbalanced frequency distribution of Chinese character radicals has been solved.
[0014] 3. Based on the similarity of Chinese character images and combined with the similarity measurement of radical positions, by executing the corresponding matching algorithm in the query database, it is possible to more accurately identify and match the query Chinese characters. Even Chinese characters that do not appear in the training set can be effectively recognized, thereby accurately identifying and achieving a high recognition rate in various application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 It is a schematic diagram of the overall process of the present invention.
[0016] Figure 2 Schematic diagram of the multi-font Chinese character radical detection network model in the present invention.
[0017] Figure 3 This is a schematic diagram of the result of performing radical detection on Chinese character images in step 4. DETAILED DESCRIPTION
[0018] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by ordinary technicians in the field without creative work by adopting the embodiments of the present invention are within the scope of protection of the present invention.
[0019] In step 1, Python's PIL library tool can be used to generate Chinese character images. A variety of different font files are used to generate corresponding Chinese character images based on Chinese character information. Each image includes only one Chinese character in one font.
[0020] The Chinese character information refers to the Chinese character corresponding to the Chinese character image, and the radical information includes the radical category of the Chinese character and the position information of the radical in the image; in terms of the radical category, according to "GF 0011-2009 Language and Character Standard-Chinese Character Radical Table", there are 201 main radicals and 100 auxiliary radicals constituting Chinese characters, and N represents the number of radical categories, N=301, c i It is one of the radical categories, represented by c i ∈{1,2,...,301}; the position information of the radical in the image p i Described by three coordinate points, represented by p i =(p i1 ,p i2 ,p i3 ), where p i1 is the center coordinate of the minimum enclosing rectangle of the radical in the image, p i2Indicates the coordinates of the upper left corner of the minimum enclosing rectangle, p i3 Indicates the coordinates of the lower right corner, and the radical label information is expressed as
[0021] It also includes data annotation: Chinese character images of a small number of fonts are used as data sets for supervised learning, and Chinese character images of other fonts are used as data sets for weakly supervised learning. Since the Chinese character label information is available in the image preparation stage, in the annotation stage, only the radical label information is annotated for the data used for supervised learning, and an identifier is added to indicate whether there is radical label information; batch data is composed: a batch data includes three types of data: image data d1 with Chinese character label information and radical label information, image data d2 without radical label information and the same Chinese character label information as d1, and image data d3 without radical label information and different Chinese character label information from d1.
[0022] like Figure 2 As shown, the multi-font Chinese character radical detection network model in step 2 includes a model backbone module, a radical classification module, a radical detection module and a weakly supervised learning module. The model backbone module is used to extract features from the input Chinese character image to obtain a high-dimensional feature representation of the Chinese character image. It is composed of a convolutional layer, a maximum pooling layer, and a nonlinear layer. Residual connections are set between some network layers. The data processing process is: input Chinese character image After feature extraction and downsampling, the feature map is obtained Among them, H and W represent the height and width of the image, C, G', and W' represent the number of channels, height, and width of the feature map respectively; the model backbone module uses the feature extraction backbone network of ResNet50, including a 7×7 convolutional layer, a nonlinear layer, and 4 residual modules. Each residual module consists of a 1×1 convolutional layer, a 3×3 convolutional layer, a 1×1 convolutional layer, and a nonlinear layer.
[0023] The radical classification module is used to predict the radical category of the radical label information data, and is composed of a convolution layer, a maximum pooling layer and an average pooling layer. The process is: the input feature map I' is compressed into a feature vector Among them, cls represents the number of radical categories.
[0024] The radical detection module is used to predict the radical position of the data with radical label information, and is composed of a deconvolution layer. The feature map I' is upsampled to
[0025] The weakly supervised learning module is used to compare the data with radical label information and the data without radical label information, so as to realize weakly supervised learning for the data without radical label information, so as to reduce the annotation cost of radical position and enhance the model's radical detection results for Chinese characters in various different fonts. The data processing process is: the input feature map I2 is compressed into a feature vector It includes feature vectors with radical label information and feature vectors without radical label information, which are combined into a feature vector combination for contrastive learning; specifically, d1, d2, and d3 in a batch of data are compressed into feature vectors after passing through the model backbone module, radical detection module, and weak supervision learning module. Then combine them into the feature vector combination {f1,f2}, {f1,f3} for contrastive learning.
[0026] The loss function L improved for the distribution characteristics of Chinese character radicals in step 3 consists of three parts:
[0027] Loss function L1, multi-classification cross entropy loss function for radical classification module, Where p1 represents the probability distribution of the feature map I' in the radical classification module after being processed by the softmax layer, and N represents the number of radical categories.
[0028] Loss function L2, the focal loss function used for the improved radical detection module,
[0029] Wherein, ∈ and β represent hyperparameters, ∈ is used to prevent division by 0, p2 represents the probability distribution of feature map I' in the radical detection module after being processed by the softmax layer, and N represents the number of radical categories; preferably, ∈ takes a value of 0.01 and β takes a value of 2.
[0030] Loss function L3, a contrast loss function used for weakly supervised learning modules,
[0031] Where d represents the Euclidean distance between feature vectors, m is the Euclidean distance threshold, y∈[0,1], y=1 means that the feature vectors have the same Chinese character label information, and are the same Chinese character, y=0 means that the feature vectors have different Chinese character label information, and are not the same Chinese character, that is, when calculating the feature vector combination {f1,f2}, y=1, When calculating the eigenvector combination {f1, f3}, y = 0,
[0032] Finally, the three types of loss functions are integrated as the improved loss function L, L = L1 + L2 + L3.
[0033] Step 5: Based on the radical category and position information obtained in step 4, the process of integrating image similarity and radical position similarity to predict Chinese characters is as follows: a font dataset with radical label information is selected to build a Chinese character category query library, and each item in the query library is stored in a key-value manner, wherein the key is the Chinese character category code, and the value includes the Chinese character image, the radical category constituting the Chinese character, and the position information corresponding to each radical;
[0034] Radical category prediction results for the test image obtained in step 4 Where k represents the number of predicted radicals, and Chinese characters with the same radical category are searched from the query library as a candidate set;
[0035] Convert the test image and the images of Chinese characters in the candidate set into single channels respectively, calculate the matrix similarity, and obtain the image similarity S1;
[0036] The prediction result of each radical position information is represented by (p j1 ,p j2 ,p j3 ) is adjusted to (x j ,y j ,w j ,h j ), where p j1 , p j2 , p j3 The center position, upper left corner coordinates, and upper right corner coordinates of the minimum circumscribed rectangle of the corresponding radical, x j ,y j are the horizontal and vertical coordinates of the center of the radical's smallest circumscribed rectangle, w j ,h j is the width and height of the minimum circumscribed rectangle. All k predicted results of radical position information are concatenated into a 4k-dimensional vector. The same operation is performed on the radical position information of Chinese characters in the candidate set. The KL divergence is then calculated for the two sets of vectors to obtain the radical position similarity S2.
[0037] The image similarity and radical position similarity are integrated as the matching score S between the test image and the Chinese characters in the query library, S=S1(1-S2), and the category corresponding to the Chinese character with the highest matching score is used as the Chinese character recognition result for the test character image.
[0038] The preferred embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, a variety of simple modifications can be made to the technical solution of the present invention, and these simple modifications all belong to the protection scope of the present invention.
Claims
1. A font-adaptive Chinese character recognition method based on radical detection and splitting, characterized in that: The steps include: Step 1: construct a Chinese character image dataset, one part of which is used for supervised learning, and the image label is composed of Chinese character label information and radical label information, and the other part is used for weakly supervised learning, and the image label is composed of Chinese character label information, and the Chinese character image dataset is divided into multiple batch processing data according to the image label; Step 2: Based on the Chinese character image dataset in step 1, a multi-font Chinese character radical detection network model combining supervised learning and weakly supervised learning is constructed, and batch data is input. The network model includes a model backbone module, a radical classification module, a radical detection module, and a weakly supervised learning module. The model backbone module is used to extract features from the input Chinese character image to obtain a high-dimensional feature representation of the Chinese character image. The radical classification module is used to predict the radical category of the data with radical label information; the radical detection module is used to predict the radical position of the data with radical label information; The weakly supervised learning module is used for comparative learning between data with radical label information and data without radical label information, so as to realize weakly supervised learning of data without radical label information. Step 3: Iteratively train the model using the loss function improved for the distribution characteristics of Chinese character radicals; Step 4: Detect radicals of Chinese characters and obtain radical category and position information; Step 5: Integrate the Chinese character image similarity and radical position similarity to predict Chinese characters, and select a font dataset with radical label information to build a Chinese character category query library; The radical category prediction results obtained in step 4 are used as the candidate set; the test image and the images of Chinese characters in the candidate set are converted into single channels respectively, the image similarity and radical position similarity are calculated and integrated, and the category corresponding to the Chinese character with the highest matching score is used as the Chinese character recognition result for the test character image.
2. The font-adaptive Chinese character recognition method based on radical detection and splitting according to claim 1, characterized in that: Among the multiple batch data, one batch data includes three types of data, image data d1 with Chinese character label information and radical label information, image data d2 without radical label information and the same Chinese character label information as d1, and image data d3 without radical label information and different Chinese character label information from d1.
3. The font-adaptive Chinese character recognition method based on radical detection and splitting according to claim 1, characterized in that: The model backbone module consists of a convolutional layer, a maximum pooling layer, and a nonlinear layer. The Chinese character image in the input batch data Output feature map after feature extraction and downsampling Where H and W represent the height and width of the Chinese character image, and C, H′, and W′ represent the number of channels, height, and width of the feature map, respectively; The radical classification module consists of a convolutional layer, a maximum pooling layer, and an average pooling layer. The input feature map I′ is compressed into an output feature vector Among them, cls represents the number of radical categories; The radical detection module is composed of a deconvolution layer. The input feature map I′ is upsampled to the output feature map by deconvolution. The data processing process of the weakly supervised learning module is: the input feature map I2 is compressed and output as a feature vector It includes feature vectors with radical label information and feature vectors without radical label information, which are combined into a feature vector combination for contrastive learning.
4. The font-adaptive Chinese character recognition method based on radical detection and splitting according to claim 3 is characterized in that: The model backbone module uses the feature extraction backbone network of ResNet50, including a 7×7 convolutional layer, a nonlinear layer, and 4 residual modules. Each residual module consists of a 1×1 convolutional layer, a 3×3 convolutional layer, a 1×1 convolutional layer, and a nonlinear layer.
5. The font-adaptive Chinese character recognition method based on radical detection and splitting according to claim 1, characterized in that: The loss function L improved for the distribution characteristics of Chinese character radicals in step 3 includes three parts: loss function L1, multi-classification cross entropy loss function for radical classification module, Where p1 represents the probability distribution of the feature map in the radical classification module after being processed by the softmax layer, and N represents the number of radical categories; Loss function L2, the focal loss function used for the improved radical detection module, Where v and β represent hyperparameters, v is used to prevent division by 0, p2 represents the probability distribution of the feature map in the radical detection module after being processed by the softmax layer, and N represents the number of radical categories; The loss function L3 is used as the contrast loss function of the weakly supervised learning module. Where d represents the Euclidean distance between feature vectors, m is the Euclidean distance threshold, and y∈[0,1]; Loss function L=L1+L2+L3.
6. The font-adaptive Chinese character recognition method based on radical detection and splitting according to claim 1, characterized in that: Step 5 is as follows: a font dataset with radical label information is selected to construct a Chinese character category query library, and each item in the query library is stored in a key-value manner, where the key is the Chinese character category code, and the value includes the Chinese character image, the radical category constituting the Chinese character, and the position information corresponding to each radical; Radical category prediction results for the test image obtained in step 4 Where k represents the number of predicted radicals, and Chinese characters with the same radical category are searched from the query library as the candidate set; Convert the test image and the images of Chinese characters in the candidate set into single channels respectively, calculate the matrix similarity, and obtain the image similarity S1; The prediction result of each radical position information is represented by (p j1 , p j2 , p j3 ) is adjusted to (x j ,y j , w j ,h j ), where p j1 , p j2 , p j3 The center position, upper left corner coordinates, and upper right corner coordinates of the minimum circumscribed rectangle of the corresponding radical, x j ,y j are the horizontal and vertical coordinates of the center of the radical's smallest circumscribed rectangle, w j ,h j is the width and height of the minimum circumscribed rectangle. All k predicted results of radical position information are concatenated into a 4k-dimensional vector. The same operation is performed on the radical position information of Chinese characters in the candidate set. The KL divergence is then calculated for the two sets of vectors to obtain the radical position similarity S2. The image similarity and radical position similarity are integrated as the matching score S between the test image and the Chinese characters in the query library, S=S1(1-S2), and the category corresponding to the Chinese character with the highest matching score is used as the Chinese character recognition result for the test character image.