Ancient Chinese character recognition method based on lightweight convolutional neural network

Abstract

The application discloses a kind of ancient book Chinese character recognition methods based on lightweight convolutional neural network.Integrating multiple expert models, in the training process, the prediction probability distribution difference item is added to the loss function, the prediction deviation and variance on the whole class are reduced, so as to improve the recognition accuracy on all classes, while introducing model lightweight technology reduces the calculation overhead and parameter quantity of integrated model, realizes the long-tail sample recognition model of lightweight ancient book Chinese character.The model obtained by the application can effectively improve the recognition accuracy on the rare word sample set on the seriously unbalanced ancient book Chinese character dataset, and the calculation complexity and parameter quantity of the conventional lightweight model are similar.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of Chinese character recognition, and particularly relates to a method for recognizing ancient Chinese characters based on a light convolutional neural network. BACKGROUND

[0002] The Chinese character recognition method based on deep learning has become the mainstream technical solution of optical character recognition (OCR), but it only performs well on a data set with balanced categories. In the collected Chinese character data set in the ancient text image, the commonly used characters with high frequency account for most of the samples, and only a small amount or even sporadic number of samples of rare Chinese characters can be obtained. Therefore, the CNN classification model based on the cross-entropy loss function will face the challenge of insufficient recognition accuracy of rare characters on the seriously unbalanced Chinese character sample set.

[0003] Although the existing long-tail recognition method can alleviate the above problems, it generally has high computational complexity and parameter quantity, and needs to be lightened to meet the deployment environment of storage resource limited devices. SUMMARY

[0004] The application provides a method for recognizing ancient Chinese characters based on a light convolutional neural network to solve the problems of recognition accuracy of unbalanced sample Chinese character data set and high parameter quantity of integrated neural network.

[0005] In order to achieve the above purpose, a method for recognizing ancient Chinese characters based on a light convolutional neural network comprises the following steps:

[0006] Step 1: an integrated model is composed of a feature extraction network and a plurality of expert models, and the integrated model is trained by a distribution difference loss and a classification loss;

[0007] Step 2: an expert model distribution module is added to the integrated model trained in step 1 to dynamically adjust the number of opened expert models to reduce the computational overhead of the model;

[0008] Step 3: the integrated model trained in step 2 outputs a pseudo-label sample set, and the labeled sample set is updated by class rebalancing sampling;

[0009] Step 4: the integrated model is trained by the unlabeled sample and the labeled sample set updated in step 3;

[0010] Step 5: steps 3 and 4 are repeated to realize the iterative training of the integrated model;

[0011] Step 6: the integrated model trained in step 5 is used to recognize ancient Chinese characters.

[0012] Moreover, the integrated model in step 1 contains a shared backbone network f θ and n independent expert models ψ θ = [ψ1, ψ2,..., ψ n ], where the shared backbone network f θ is a feature extraction network, and the input of each expert model is the output f θ (x) of the shared backbone network for a sample x. In the training process of the first stage of the integrated model, the classification loss and the distribution difference loss L D between different expert models are used to optimize f θ and ψ θ simultaneously, so as to minimize the classification loss to obtain higher accuracy of the integrated model, and maximize the distribution difference loss L D to obtain diversified expert models, reduce the prediction bias of the expert models, and improve the accuracy of the tail classes.

[0013] Suppose γ min = min{γ1,..., γ k} represents the minimum boundary distance of each class to the classification boundary, and class j has n j samples, that is, class 1 and class 2 have n1 and n2 samples respectively, in a binary classification model, there is a real number δ ∈ (-γ2, γ1), and γ1 and γ2 satisfy the following inequalities:

[0014]

[0015] In the formula, γ1 represents the minimum boundary distance of class 1 to the classification boundary, and γ2 represents the minimum boundary distance of class 2 to the classification boundary. Formula (1) is established when the following conditions are met, where C is an undetermined hyperparameter:

[0016]

[0017] When the above binary classification case is extended to a multi-classification task, the optimal distance Y j of class j to the boundary is:

[0018]

[0019] In the formula, C is an undetermined hyperparameter, and n j represents the number of samples of class j.

[0020] The classification loss function is calculated as follows:

[0021]

[0022] where (x, y) denotes a sample x and its corresponding label y, f is a classification model, z j represents the output of the classification model f for a sample x of class j j j is the margin distance, z y represents the output of the classification model f for a sample x of class y y .

[0023] To enable the trained ensemble model to effectively distinguish intra-class variations of the head classes and maintain sufficient robustness on the tail classes with insufficient samples, the output ψ θ (f θ (x)) of the expert models is soft-labeled, and the size of the confidence in the final prediction distribution is adjusted according to the number of samples in each class, so that the ensemble model can handle both head and tail classes well. In addition, the idea of temperature adjustment in knowledge distillation is introduced, and temperature adjustment is added in the first stage of training, so that the ensemble model learns relatively stable weights in the early stage of training; the temperature T k of the kth class is calculated as follows:

[0024] T k = a (β k + 1-max (β k )) (5)

[0025] where a is a constant value, usually 1; β k is obtained by normalizing the number of samples of all classes, and the specific calculation is as follows:

[0026]

[0027] where r is the weight term, representing the proportion of normalization; n k is the number of samples of the kth class; c is the total number of classes; n s is the number of samples of the s th class.

[0028] When β k = 1, T k = a, ensuring that the temperature of each class is the same on the balanced dataset; after temperature adjustment of the prediction distribution of the expert model, the output p (i) (x, y) of the ith expert model can be represented as:

[0029]

[0030] where ψ θi (f θ (x)) c ​The output of the i-th expert model before temperature adjustment for the c-th class, T c The temperature corresponding to the c-th class, the softmax function controls the value range of the output to [0, 1].

[0031] The KL divergence loss is used as the distribution difference loss The calculation method is as follows:

[0032]

[0033] In the formula, p (i) (x, y) is the output of the i-th expert model, D KL () is the KL divergence, and n is the number of classes.

[0034] For an integrated model composed of n expert models and a shared backbone network, the total loss function of the first stage of the integrated model training process is obtained by accumulating the classification loss L C and the distribution difference loss between the prediction distributions of different expert models The calculation formula of the total loss function of the first stage of the integrated model training process is as follows:

[0035]

[0036] In the formula, λ is a weight coefficient, and λ < 0 is set to maximize the KL divergence loss.

[0037] Moreover, the second stage of the integrated model training is completed by adding an expert model allocation module in step 2, and the expert model allocation module is composed of multiple routing switches. The number of opened expert models is dynamically adjusted according to the output features of the backbone network and the expert models, the number of opened expert models in the head class sample reasoning process is reduced, and the number of opened expert models in the tail class sample reasoning process is increased as much as possible. Specifically, the opening or closing state of the routing module is used as a label for the routing switch to learn. The parameters of the backbone network and the expert models are frozen and only responsible for feature output, and no longer participate in training. By training the expert model allocation module, it is determined whether to allocate the next expert model to participate in calculation according to the output of the backbone network and the previous expert model. When the model predicts that the sample number is sufficient for the head class, the number of opened expert models can be reduced, thereby reducing the calculation overhead of the integrated model.

[0038] The expert model allocation module is composed of a shared routing module W1 and k-1 routing switch modules W2. W1 and W2 are both fully connected layers. The shared routing W1 is used to reduce the calculation overhead of the backbone network f θThe dimension of the normalized output feature. First, the top s bits of the averaged logits output by the k opened expert models are obtained, and the output of W1 is concatenated, and then the concatenated feature is input into the routing switch W2. W2 outputs a value in the range of (0, 1) through a sigmoid activation function. If the output value of W2 is greater than η, the k+1th expert model is continued to be allocated to participate in the inference process. If the output value of W2 is less than η, the result of the current averaged logits after being activated by a softmax function is taken as the final prediction distribution.

[0039] The k-1 routing switches W2 are respectively responsible for controlling the opening state of the 2th to kth expert models. The calculation process of the output result r(x) of the kth routing switch W2 for the input sample x is represented as:

[0040]

[0041] In the formula, W2(k) represents the kth routing switch W2, f θ (x) is the output of the backbone network, Ψ θk (f θ (x)) is the output of the kth expert model before temperature adjustment, and top-s represents the top s bits of logits. The value of s is set according to the data set.

[0042] Loss function of the second stage training of the integrated model The calculation formula is:

[0043]

[0044] In the formula, w on is a weight term, which can control the difficulty of opening the routing switch; the label y on is given according to whether the prediction result of the expert model is correct or not. When the kth expert model predicts correctly, y on = 0; when the kth expert model predicts incorrectly, if the prediction of any one of the remaining n-k expert models is correct, y on = 1, otherwise y on = 0.

[0045] During the training process, the of each routing switch is accumulated. When the convergence is reached, the routing switch can determine whether to open the next expert model according to the prediction distribution of the expert model it is responsible for, thereby reducing the computational overhead of the integrated model.

[0046] Furthermore, when sampling from the pseudo-label dataset in step 3, different sampling rates μ(·) need to be set according to the number of pseudo-label samples N(·) corresponding to each category. The fewer the number of samples corresponding to a category, the more samples are collected for that category, thereby realizing category rebalancing sampling in the unlabeled dataset.

[0047] The formula for calculating the sampling rate μ(l) of category l is as follows:

[0048]

[0049] In the formula, α is a constant sampling rate hyperparameter, c is the total number of categories, and N() represents the number of pseudo-label samples corresponding to each category.

[0050] Furthermore, in step 4, the cross-entropy loss function is used as the supervised loss for learning labeled samples, while the consistency regularization method is used for learning unlabeled samples. The consistency regularization method assumes that the model should output similar predicted distributions for unlabeled data with different perturbations. This invention applies two data augmentation methods, one strong and one weak, to the same unlabeled image u. If the highest confidence level in the predicted distribution q of the weakly augmented image w(u) exceeds a preset fixed threshold τ, then it is treated as a pseudo-label in one-hot format for the unlabeled sample. The unsupervised cross-entropy loss is calculated together with the model's predicted distribution of the strongly enhanced image Ω(u) output. To enable learning from unlabeled data.

[0051] Unsupervised loss The calculation formula is as follows:

[0052]

[0053] q b =p m (w(u b (14)

[0054] In the formula, μ is the weight term of the unsupervised loss in the overall training loss, B is the number of training batch samples, τ is the set confidence threshold, and q b For the model to weakly enhanced image w(u) b The predicted distribution of ), max(q) b L(max(q)) represents the maximum confidence level in the prediction distribution. b )>τ) means selecting the maximum(q) b Learning is performed on prediction samples that are greater than the confidence threshold τ. This represents the one-hot pseudo-label corresponding to the maximum confidence score, Ω(u b ) indicates a strongly enhanced sample u b p m(Ω(u b )) is the prediction distribution of the ensemble model on the strongly augmented sample, and H represents the cross-entropy loss.

[0055] Supervised loss is calculated as follows:

[0056]

[0057] In the formula, B is the number of training batch samples, y b represents the true label of the sample batch, p m (w(x b )) is the prediction distribution of the ensemble model on the weakly augmented sample, and H represents the cross-entropy loss.

[0058] In the training of the semi-supervised model for balancing samples, the supervised loss and the unsupervised loss are used at the same time, and the overall loss function is calculated as follows:

[0059]

[0060] In the formula, λ u is the weight term of the unsupervised loss.

[0061] Moreover, the step 5 uses a consistency regularization framework based on a constant threshold to train the ensemble model on the labeled and unlabeled data sets to obtain a teacher model during the training process, the pseudo label of the unlabeled data set can be obtained through the teacher model, then the pseudo label data of each class is sampled at different sampling rates using a class rebalancing strategy, the pseudo label data set obtained by sampling is added to the labeled data set to generate a more balanced labeled data set, and finally the new labeled data set and the original unlabeled data set are continuously trained to obtain a new teacher model, and a higher-precision Chinese character recognition model can be obtained through repeated iteration.

[0062] Compared with the prior art, the present application has the following advantages:

[0063] 1) By integrating multiple expert models, the prediction probability distribution difference term is added to the loss function during the training process, the prediction deviation and variance on all classes are reduced, and thus the recognition accuracy on all classes is improved.

[0064] 2) The model lightening technology is introduced to reduce the calculation overhead and parameter quantity of the ensemble model, and a lightweight ancient Chinese character long-tail sample recognition model is realized.

[0065] 3) The recognition model obtained by the application can effectively improve the recognition accuracy of rare character sample sets on a seriously unbalanced ancient Chinese character data set, and has similar calculation complexity and parameter quantity to conventional lightweight models. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 is a flowchart of an embodiment of the application.

[0067] Figure 2 is a first-stage training structure of an integrated model of an embodiment of the application.

[0068] Figure 3 is a second-stage training structure of an integrated model of an embodiment of the application.

[0069] Figure 4 is a semi-supervised learning method based on consistency regularization of an embodiment of the application. DETAILED DESCRIPTION

[0070] The application provides an ancient Chinese character recognition method based on a lightweight convolutional neural network. The technical solutions of the application are further described below in conjunction with the drawings and embodiments.

[0071] As shown in Figure 1 , the flow of an embodiment of the application includes the following steps:

[0072] Step 1: An integrated model is composed of a feature extraction network and multiple expert models, and the integrated model is trained together by distribution difference loss and classification loss.

[0073] As shown in Figure 2 , the integrated model ENS-CNet includes a shared backbone network f θ and n independent expert models ψ θ =[ψ1,ψ2,...,ψ n ], the shared backbone network f θ is a feature extraction network, and the input of each expert model comes from the output f θ (x) of the shared backbone network for the sample x. In the first-stage training process of the integrated model, the classification loss of each expert model and the distribution difference loss L D between different expert models are used to optimize f θ and ψ θ at the same time, so as to minimize the classification loss to obtain higher integrated model accuracy, and maximize the distribution difference loss L D to obtain diversified expert models, reduce the prediction deviation of the expert models, and thus improve the tail class accuracy.

[0074] To improve the tail class accuracy of ensemble models on long-tail datasets, it is necessary to reduce the generalization error of expert models on the tail classes. However, on severely imbalanced datasets, expert models do not learn the tail class features sufficiently, which can easily lead to overfitting. Therefore, a more lenient classification boundary needs to be given to the tail classes. The classification loss used in this invention... Label-Distribution-Aware Margin Loss (LDAM) increases the distance between the separating boundaries of tail classes and head classes, reducing the difficulty of tail class classification. The distance between the tail class and the boundary is crucial for classification models; an excessively large distance may cause the boundary to shift towards the head class, affecting head class classification performance, while an excessively small distance will result in insufficient generalization ability on tail classes. Therefore, a balance point at the boundary position needs to be found. While a universal balance point for multi-class classification is difficult to determine, the optimal solution can be calculated based on binary classification tasks, thus extending to multi-class tasks.

[0075] Assuming Y min =min{γ1,...,γ k} represents the minimum boundary distance between each category and the classification boundary, and category j has n j When there are n1 and n2 samples for category 1 and category 2 respectively, in the binary classification model, there exists a real number δ∈(-Υ2,Υ1) such that Υ1 and Υ2 satisfy the following inequality:

[0076]

[0077] In the formula, γ1 represents the minimum boundary distance between category 1 and the classification boundary, and γ2 represents the minimum boundary distance between category 2 and the classification boundary. Equation (1) holds when the following conditions are met, where C is an undetermined hyperparameter:

[0078]

[0079] Extending the binary classification case described above to multi-class classification tasks, the optimal distance γ between category j and the boundary is... j for:

[0080]

[0081] In the formula, C is an undetermined hyperparameter, and n j This represents the number of samples in category j.

[0082] In this invention, (x, y) represents a sample x and its corresponding label y, and f is a classification model. The output f(x) of the classification model f for a sample x of class j is defined. j Simplified representation is z j The above boundary distance γ jThe hinge loss is jointly rewritten as a cross-entropy loss function including a softmax activation function to obtain a classification loss function L C :

[0083]

[0084] where z y represents the output f(x) of the classification model f for the sample x of the class y y .

[0085] As can be seen from formula (4), the calculation of the inter-class distance in the classification loss function takes into account the sample distribution and can solve the problem of long-tail sample distribution imbalance.

[0086] In the first phase of the training of the ensemble model, the classification loss needs to accumulate the difference between the predicted distribution ψ θ (f θ (x)) of each expert model and the label y, rather than only measuring the difference between the predicted distribution after averaging and the label, so that each expert model fully learns the distance between the head and tail classes, improves the prediction accuracy of the tail classes, and does not affect the accuracy of the head classes. In addition to using the classification loss to measure the difference between the predicted distribution and the label, the distribution difference loss is also used to measure the difference between the predicted distributions of the expert models. The difference is that the latter minimizes the LDAM loss and maximizes the distribution difference loss during the training process, so that different expert models output complementary predicted distributions and avoid outputting extremely extreme predicted distributions. For a long-tail data set, this can effectively reduce the bias of the expert models and improve the overall performance of the ensemble model.

[0087] In order to enable the trained ensemble model to effectively distinguish the intra-class variation of the head classes and maintain sufficient robustness in the tail classes with insufficient samples, the output ψ θ (f θ (x)) of the expert model is subjected to soft labeling operation, and the size of the confidence in the final predicted distribution is adjusted according to the number of samples of each class, so that the ensemble model can handle the head and tail classes well. In addition, the idea of temperature adjustment in knowledge distillation is introduced, and temperature adjustment is added during the first phase of training, so that the ensemble model can learn relatively stable weights in the early stage of training. The temperature T increases linearly with the number of samples of the class, and the calculation formula of the temperature T k corresponding to the kth class is as follows:

[0088] T k = α(β k +1-max(β k )) (5)

[0089] where α is a constant value, usually 1; β kThe normalization is performed by the number of samples of all categories, and the specific calculation is as follows:

[0090]

[0091] In the formula, r is a weight item, indicating the normalized proportion; n k is the number of samples of the kth category; c is the total number of categories; n s is the number of samples of the sth category.

[0092] When β k = 1, T k = α can be obtained, which ensures that the temperature of each category on the balanced dataset is the same, so the temperature processing mechanism of the present application can also be used for integrated model learning on a balanced dataset.

[0093] After temperature adjustment is performed on the prediction distribution of the expert model, the output p (i) (x, y) of the ith expert model can be represented as:

[0094]

[0095] In the formula, Ψ θi (f θ (x)) c is the output of the cth category of the ith expert model before temperature adjustment, T c is the temperature corresponding to the cth category, and the softmax function controls the value range of the output to [0, 1].

[0096] For the prediction of samples with long-tail distribution, it is necessary to further increase the prediction difference between expert models to achieve complementary effect. The KL divergence loss is used as the distribution difference loss together with the classification loss to guide the training of the integrated model in the first stage. The specific calculation of the distribution difference loss is as follows:

[0097]

[0098] In the formula, p (i) (x, y) is the output of the ith expert model, D KL () is the KL divergence, and n is the number of categories.

[0099] For an integrated model composed of n expert models and a shared backbone network, the classification loss L C of each expert model and the distribution difference loss between the prediction distributions of different expert models can be added to obtain the overall loss function of the integrated model in the first stage training process, and the calculation formula is as follows:

[0100]

[0101] In the formula, λ is the weighting coefficient, and λ<0 is set to maximize the KL divergence loss.

[0102] Step 2: Add an expert model allocation module to the ensemble model after training in Step 1, and dynamically adjust the number of expert models enabled to reduce the computational overhead of the model.

[0103] Because the head class has a large number of samples, expert models can learn the class features relatively easily, and it is not necessary to run too many expert models for prediction during the inference phase. However, for the tail class samples, which are difficult to learn, as many expert models as possible are needed to participate in the decision-making process. Therefore, an expert model allocation module is added to complete the second stage of training of the ensemble model, such as... Figure 3 As shown, the expert model allocation module consists of multiple routing switches, which dynamically adjust the number of expert models enabled based on the output characteristics of the backbone network and the expert models. This reduces the number of expert models enabled during the inference process of head class samples, while ensuring that the number of expert models enabled during the inference process of tail class samples is increased as much as possible.

[0104] In the second stage, the on / off state of the routing module is used as a label for the routing switch to learn. The parameters of the backbone network and expert models are frozen, and they are only responsible for feature output and no longer participate in training. By training the expert model allocation module, the system determines whether to allocate the next expert model to participate in the calculation based on the output of the backbone network and the previous expert model. When the model predicts the head class with a sufficient number of samples, the number of expert models that need to be activated can be reduced, thereby reducing the computational overhead of the ensemble model.

[0105] The expert model allocation module consists of a shared routing module W1 and k-1 routing switch modules W2, where W1 and W2 are both fully connected layers; the shared route W1 is used to reduce the backbone network f θ The dimension of the normalized output features: First, obtain the first s bits of the averaged logits from the outputs of the k activated expert models, and concatenate them with the output of W1. Then, input the concatenated features into the routing switch W2. W2 outputs values ​​in the range (0,1) through the sigmoid activation function, using η as the threshold. If the output value of W2 is greater than η, continue to allocate the (k+1)th expert model to participate in the inference process. If the output value of W2 is less than η, then the result of the current averaged logits after activation by the softmax function is used as the final prediction distribution.

[0106] These k-1 routing switches W2 are responsible for controlling the on / off state of the 2nd to kth expert models respectively. For the input sample x, the calculation process of the output result r(x) of the kth routing switch W2 is expressed as follows:

[0107]

[0108] wherein, represents the kth routing switch W2, f θ (x) is the output of the backbone network, Ψ θk (f θ (x)) is the output of the kth expert model before temperature adjustment, top-s represents the top s logits, and the value of s is set according to the data set.

[0109] The loss function for optimizing r(x) in the embodiment rewritten from the binary cross-entropy loss, the difference being that a weight term w on is added on , w on can control the difficulty of opening the routing switch, and the calculation formula of the loss function of the second stage training of the integrated model is

[0110]

[0111] wherein, the label y on in the training process is given according to whether the prediction result of the expert model is correct, when the kth expert model predicts correctly, y on = 0; when the kth expert model predicts incorrectly, if the prediction of any one of the remaining n-k expert models is correct, y on = 1, otherwise y on = 0. For example, in an integrated model containing 3 experts, when the first expert predicts correctly, y on = 1, otherwise y on = 0.

[0112] The of each routing switch is accumulated in the training process After convergence, the routing switch can determine whether to open the next expert model according to the prediction distribution of the expert model it is responsible for, thereby reducing the computational overhead of the integrated model, especially in the inference process of commonly used word samples.

[0113] Step 3, using the integrated model trained in step 2 to output the pseudo-label sample set, and updating the labeled sample set by class rebalancing sampling.

[0114] ​In actual scenarios, manual annotation of ancient Chinese character pictures has the problems of time-consuming and difficulty, so in the case of limited manpower, there are a large number of unlabeled images in the ancient Chinese character dataset. Through semi-supervised learning method, the model can learn these unlabeled samples, and further improve the model precision.

[0115] In Figure 1 the semi-supervised learning process, the semi-supervised model for unlabeled ancient Chinese characters adopts an iterative training method. In the process of iterative training, samples need to be selected from the pseudo-label dataset to make the updated labeled dataset more tend to a class-balanced dataset. Therefore, when sampling from the pseudo-label dataset, different sampling rates μ(·) need to be set according to the number of pseudo-label samples N(·) corresponding to each class. The fewer the number of samples corresponding to the class, the more samples are collected for the class, so as to realize class rebalancing sampling in the unlabeled dataset.

[0116] The calculation formula of the sampling rate μ(l) of the class l is as follows:

[0117]

[0118] In the formula, α is a constant sampling rate hyperparameter, c is the total number of classes, and N() represents the number of pseudo-label samples corresponding to the class.

[0119] In the iterative updating process of the labeled sample set, the rebalancing strategy ensures that the tail classes have a higher sampling rate than the head classes. More rare character samples will be added to the labeled sample set, and through the learning of the integrated model, the learning effect of the model on the rare character samples in the semi-supervised learning scenario will be enhanced.

[0120] Step 4: Train the integrated model through the unlabeled samples and the updated labeled samples in step 3.

[0121] The common cross-entropy loss function is used as the supervised loss for learning the labeled samples, and the consistency regularization method is used for learning the unlabeled samples. The consistency regularization method assumes that the model should output similar prediction distributions for unlabeled data with different perturbations. Based on this assumption, the same unlabeled image u is subjected to two data enhancement methods with strong and weak magnitudes. If the highest confidence in the prediction distribution q of the model for the weak enhanced image w(u) exceeds the preset fixed threshold τ, then it is regarded as the pseudo-label of the unlabeled sample in the one-hot form The prediction distribution output by the model for the strong enhanced image Ω(u) is used to calculate the unsupervised cross-entropy loss To learn the unlabeled data, the process of the consistency regularization method in the present application is as Figure 4 shown.

[0122] In the training of the semi-supervised model for balancing samples, a supervised loss and an unsupervised loss are simultaneously used and are cross-entropy loss functions.

[0123] The calculation formula of the unsupervised loss is as follows:

[0124]

[0125] q b =p m (w(u b )) (14)

[0126] In the formula, μ is the weight term of the unsupervised loss in the total training loss, B is the number of training batch samples, τ is the set confidence threshold, q b is the prediction distribution of the model for the weak augmented image w(u b ), max(q b ) is the maximum value of the confidence in the prediction distribution, L(max(q b )>τ) represents that the prediction samples with max(q b ) greater than the confidence threshold τ are filtered out for learning, max(q b ) represents the one-hot pseudo label corresponding to the maximum confidence, Ω(u b ) represents the strongly augmented sample u m , p b (Ω(u b )) is the prediction distribution of the ensemble model for the strongly augmented sample, and H represents the cross-entropy loss.

[0127] The calculation method of the supervised loss is as follows:

[0128]

[0129] In the formula, B is the number of training batch samples, y m represents the real label of the sample batch, p b (w(x u )) is the prediction distribution of the ensemble model for the weak augmented sample, and H represents the cross-entropy loss.

[0130] The total loss function of the semi-supervised model for balancing samples is calculated as follows:

[0131]

[0132] In the formula, λ u is the weight term of the unsupervised loss.

[0133] ​Step 5, repeat steps 3 and 4 to realize iterative training of the integrated model.

[0134] In the training process, firstly, a consistent regularization framework based on a constant threshold is used to train the integrated model on the labeled and unlabeled data sets to obtain a teacher model, through which the pseudo labels of the unlabeled data set can be obtained, then a class rebalancing strategy is used to sample the pseudo label data of each class at different sampling rates, the pseudo label data set obtained by sampling is added to the labeled data set, thereby generating a more balanced labeled data set, then the new labeled data set and the original unlabeled data set are used for continuous training to obtain a new teacher model, and through repeated iterations, a Chinese character recognition model with higher accuracy can be obtained.

[0135] Step 6, use the integrated model trained in step 5 to recognize ancient Chinese characters.

[0136] The specific embodiments described herein are merely illustrative of the spirit of the present application. Those skilled in the art of the present application can make various modifications or supplements to the described specific embodiments or use similar ways to replace them, without deviating from the spirit of the present application or exceeding the scope defined by the appended claims.

Claims

1. A method for ancient Chinese character recognition based on a lightweight convolutional neural network, characterized in that, The method comprises the following steps: Step 1, an integrated model is constituted by a feature extraction network and a plurality of expert models, and the integrated model is trained by a distribution difference loss and a classification loss; The ensemble model ENS-CNet contains a shared backbone network. and n independent expert models Shared backbone network For the feature extraction network, the input of each expert model comes from the shared backbone network's input of examples. Output During the first stage of training the ensemble model, each expert model uses its own independent classification loss. Distributional discrepancy loss between different expert models right and Simultaneously optimize to reduce classification loss To achieve higher ensemble model accuracy, the distribution difference loss is minimized. Maximize the diversity of expert models to reduce the prediction bias of expert models and thus improve the accuracy of tail classes; The classification loss function is calculated as follows: （4） wherein representing samples and their corresponding labels , is a classification model, representing a classification model for samples of class y , is a margin distance, representing a classification model for samples of class y ; Step 2, an expert model distribution module is added to the integrated model trained in step 1, and the number of opened expert models is dynamically adjusted to reduce the calculation overhead of the model; The second phase of integrated model training is completed by adding the expert model distribution module, the expert model distribution module is constituted by a plurality of routing switches, the number of opened expert models is dynamically adjusted according to the output features of the backbone network and the expert models, the number of opened expert models in the head class sample inference process is reduced, and the number of opened expert models in the tail class sample inference process is increased as much as possible, specifically, the opening or closing state of the routing module is taken as a label for the routing switch to learn, the parameters of the backbone network and the expert models are frozen and only responsible for feature output, and the training is no longer participated, by training the expert model distribution module, it is judged whether the next expert model is allocated to participate in calculation according to the output of the backbone network and the previous expert model, when the model predicts that the sample number of the head class is sufficient, the number of opened expert models can be reduced, and the effect of reducing the calculation overhead of the integrated model is achieved; Step 3, a pseudo-label sample set is output by the integrated model trained in step 2, and a labeled sample set is updated by class rebalancing sampling; Step 4, the integrated model is trained by the unlabeled sample and the labeled sample updated in step 3; Step 5, steps 3 and 4 are repeated to realize iterative training of the integrated model; Step 6, the integrated model trained in step 5 is used for recognition of ancient Chinese characters. 2.The ancient Chinese character recognition method based on a lightweight convolutional neural network according to claim 1, wherein: The assumption in step 1 denotes the minimum boundary distance of each class to the classification boundary, and class has samples, i.e., class and class have and samples, respectively, there exists a real number for and satisfying the following inequality: （1） wherein denotes the minimum boundary distance of class 1 and the classification boundary, denotes the minimum boundary distance of class 2 and the classification boundary, formula (1) is valid when the following conditions are met, wherein is an undetermined hyperparameter: （2） Extending the above binary classification case to the multi-classification task, the classes Optimal distance from the boundary are: （3） wherein is an undetermined hyperparameter, denotes the number of samples of the class . 3.The ancient Chinese character recognition method based on a lightweight convolutional neural network according to claim 2, wherein: In step 1, in order to make the trained ensemble model effectively distinguish the intra-class variation of the head class and maintain sufficient robustness on the tail class with lack of samples, the output of the expert model is soft-labeled, and the size of the confidence in the final prediction distribution is adjusted according to the number of samples in each class, so that the ensemble model can handle the head and tail classes well; in addition, the idea of temperature regulation in knowledge distillation is introduced, and temperature regulation is added in the first stage of training, so that the ensemble model learns relatively stable weights in the early stage of training; the calculation formula of the temperature of the first class is as follows: ​​ （5） wherein is a constant value, usually taken as ; The normalization is performed by the number of samples of all classes, and is calculated as follows: （6） wherein, is a weight term, representing the normalized proportion; is the number of samples in the k th class; is the total number of classes; is the number of samples in the s th class; When , , the temperature of each class is guaranteed to be the same on balanced datasets; after temperature adjustment of the prediction distribution of the expert model, the output of the first expert model is represented as: （7） In the formula, The first before temperature adjustment The first expert model c Output of each category For the first c For the temperature corresponding to each category, the softmax function controls the output value range to [0,1]. using a KL-divergence loss as the distribution difference loss in the following manner: （8） wherein is the output of the th expert model, is the KL divergence, n is the number of classes; for An ensemble model consisting of several expert models and a shared backbone network is formed by summing the independent classification losses of each expert model. Distribution difference loss between distributions predicted by different expert models The overall loss function for the first stage of training of the ensemble model is obtained, and the calculation formula is as follows: （9） wherein are weight coefficients, let to maximize the KL divergence loss. 4.The ancient Chinese character recognition method based on a lightweight convolutional neural network according to claim 3, wherein: In step 2, the expert model allocation module consists of a shared routing module. and One routing switch module constitute, and All are fully connected layers; shared routing Used to reduce backbone network The dimension of the normalized output feature; first obtain the enabled... k The top average logits of the expert model output Position, and The output is concatted, and then the characteristics resulting from the concat operation are input into the routing switch. , Output via sigmoid activation function Values ​​within the range, As a threshold, if Output value greater than Continue to allocate the first If an expert model participates in the reasoning process, if Output value less than Then the result of activating the current averaged logits using the softmax function will be used as the final prediction distribution; Router switch Responsible for controlling the 2nd to the 3rd The activation status of each expert model, for input samples , No. Router switch Output results The calculation process is expressed as follows: （10） wherein, represents the output of the th routing switch, , is the output of the backbone network, is the output of the k th expert model before temperature adjustment, represents the first bits of logits, the numerical value is set according to the dataset; Loss function for second stage training of ensemble model The formula is: （11） In the formula, is a weight term, which can control the difficulty level of the routing switch to open. Labels in the training process According to whether the prediction result of the expert model is correct or not, when the prediction of the th expert model is correct, ; when the prediction of the th expert model is wrong, if the prediction of any one of the remaining expert models is correct, , otherwise ; The cumulative number of times each routing switch is used during the training process When After convergence, the routing switch can determine whether to turn on the next expert model according to the prediction distribution of the expert model it is responsible for, thereby reducing the computational overhead of the integrated model. 5.The ancient Chinese character recognition method based on a lightweight convolutional neural network according to claim 4, wherein: When sampling from the pseudo-label dataset in step 3, the number of pseudo-label samples corresponding to each class needs to be considered Different sampling rates are set The fewer the number of samples corresponding to the class, the more samples are collected for the class, so that class rebalancing sampling is realized in the unlabeled dataset; the class The calculation formula of the sampling rate is as follows: （12） In the formula, is a constant sampling rate hyperparameter, is the total number of classes, and N( ) represents the number of pseudo-label samples corresponding to the class. 6.The ancient Chinese character recognition method based on a lightweight convolutional neural network according to claim 5, wherein: In step 4, the cross-entropy loss function is used as the supervised loss for learning labeled samples, while the consistency regularization method is used for learning unlabeled samples. The consistency regularization method assumes that the model should output similar predicted distributions for unlabeled data with different perturbations; for the same unlabeled image... Applying two data augmentation methods, one stronger and one weaker, to the image, if the model is weakly augmented... Predicted distribution The highest confidence level in the data exceeds a preset fixed threshold. Therefore, it is used as a pseudo-label in one-hot form for unlabeled samples. , and the model for strongly enhanced images The unsupervised cross-entropy loss is calculated together with the output prediction distribution. This enables learning from unlabeled data. Unsupervised loss The formula for the calculation is as follows: （13） （14） wherein, is a weight term of the unsupervised loss in the total training loss, is the number of training batch samples, is a set confidence threshold, is the prediction distribution of the model on the weakly augmented image , is the maximum value of the confidence in the prediction distribution, indicates that the prediction samples with confidence greater than the confidence threshold are filtered out for learning, indicates the one-hot pseudo label corresponding to the maximum value of the confidence, indicates the strongly augmented sample , is the prediction distribution of the ensemble model on the strongly augmented sample, indicates the cross-entropy loss;​ Supervised loss is calculated as follows: （15） In the formula, is the number of training batch samples, represents the true label of the sample batch, is the prediction distribution of the integrated model for the weakly enhanced sample, represents the cross-entropy loss; In the training of the semi-supervised model for balancing samples, a supervised loss is used simultaneously and an unsupervised loss The total loss function is calculated as follows: （16） In the formula, is a weight term for the unsupervised loss.

7. The ancient Chinese character recognition method based on a lightweight convolutional neural network according to claim 6, characterized in that: In the training process of step 5, firstly, an integrated model is trained on the labeled and unlabeled data sets based on a consistency regularization framework based on a constant threshold to obtain a teacher model, pseudo labels of the unlabeled data set are obtained through the teacher model, then a class rebalancing strategy is used to sample the pseudo label data of each class at different sampling rates, the pseudo label data set sampled is added to the labeled data set to generate a more balanced labeled data set, finally, the new labeled data set and the original unlabeled data set are continuously trained to obtain a new teacher model, and a higher-precision Chinese character recognition model is obtained through repeated iteration.

Images