A method for ancient Chinese character sample collection, detection and recognition based on optimized deep network

Abstract

The present invention provides a method for collecting, detecting and recognizing ancient character samples based on an optimized deep network. First, data collection and sample annotation are performed, and the database is expanded and updated through an iterative collection method to build an ancient character rubbing copy and a single-character database to make up for the lack of an ancient character image database. Secondly, CycleGAN is selected to perform cross-domain image conversion on ancient character copies and rubbings, expand the data from the perspective of style migration, and perform ancient character target detection on this basis. By performing target detection on batches of ancient character images, the recognition of ancient characters can be converted from single-character to whole-piece recognition, thereby improving the practicality of the technology. In addition, based on the constructed ancient character single-character database, the Mixup method is used to generate new sample data based on its long-tail distribution characteristics. Four deep network models are then used for training, and finally a decision-level fusion recognition model is established to achieve the goals of ancient character collection, detection and recognition.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method for collecting, detecting and recognizing ancient Chinese characters based on an optimized deep network. Background Art

[0002] The study of ancient characters has always been a crucial area of ​​cultural heritage preservation and historical research. However, traditional methods for studying ancient characters often require significant manpower and time, and are subject to subjectivity and errors. With the rapid development of computer vision and deep learning technologies, it has become possible to use computers to automatically collect, detect, and recognize ancient characters.

[0003] Currently, there are insufficient electronic datasets of ancient characters available for research and analysis, and most ancient character recognition efforts remain at the manual stage, relying on physical objects or paper documents. This results in a data shortage for deep network technology in ancient character recognition. The lack of large-scale, high-quality ancient character datasets limits the comprehensive understanding and application of these characters.

[0004] Furthermore, ancient characters, as natural data, have a long-tail distribution: most characters appear infrequently, while a few appear more frequently. This leads to data imbalance. While deep network models have excellent feature extraction and classification capabilities, enabling more efficient and accurate character recognition, data imbalance can easily lead to overfitting of low-frequency data and underfitting of high-frequency data, compromising recognition accuracy. Summary of the Invention

[0005] Purpose of the invention: In view of the above existing technologies, a method for collecting, detecting and recognizing ancient Chinese characters based on an optimized deep network is proposed;

[0006] Technical solution: A method for collecting, detecting and recognizing ancient Chinese characters based on an optimized deep network, characterized by comprising the following steps:

[0007] 1) Acquisition and expansion of data sets;

[0008] 2) Ancient Chinese characters detection;

[0009] 3) Ancient character recognition.

[0010] Preferably, the implementation process of step 1) is as follows: collecting pictures of original rubbings of ancient characters and pictures of copies of ancient characters, and storing them in a folder, and then using the LabelMe tool to label the ancient characters in the pictures of original rubbings of ancient characters and pictures of copies of ancient characters, wherein the labels of the original rubbings of ancient characters and pictures of copies of ancient characters are unified, and the labeled ancient characters are used as the initial data set, and the unlabeled, i.e., unknown ancient characters are placed in the "unknown" folder;

[0011] Perform deep network training on the initial dataset, and predict unknown ancient characters in the "unknown" folder. Add the correctly predicted unknown ancient characters to the initial dataset, and iterate the training to expand the initial dataset.

[0012] The frequency of known ancient characters on each ancient character copy image is counted and arranged in ascending order of frequency. The last 20% of the copy images are selected as training sets and validation sets, and the rest are used as test sets. The style transfer CycleGAN algorithm is used to perform image conversion between the two styles of the original ancient character rubbing images and the copy images, expanding the ancient character rubbing images and ancient character copy images collected in step 1).

[0013] Preferably, in step 2), the YOLOv7-tiny model is trained using the expanded copy image to obtain a trained YOLOv7-tiny model, and the ancient characters in the ancient character rubbings image are detected using the trained YOLOv7-tiny model.

[0014] Preferably, in step 3), pictures of ancient character single words are intercepted based on the expanded pictures of ancient character rubbings and ancient character facsimiles, and the pictures of the ancient character single words are preprocessed and segmented, wherein the preprocessing includes resizing, rotating, and optical color adjustment of the pictures of the ancient character single words; based on the characteristics of the long-tail distribution, the head class of the pictures of the ancient character single words is randomly sampled, and then the pictures of the ancient character single words are segmented into a training set, a validation set, and a test set according to a stratified sampling method;

[0015] Secondly, the long-tail distribution features are enhanced by the Mixup long-tail optimization method. Let λ be the mixing coefficient, where 0≤λ≤1, then the new sample x is expressed as:

[0016] x=λx1(1-λ)x2

[0017] The new label y is represented as:

[0018] y=λy1(1-λ)y2

[0019] The two randomly sampled image samples are x1 and x2, and the corresponding labels are y1 and y2;

[0020] Next, the generated (x, y) pairs are pre-trained in four deep network classifiers: AlexNet, ResNet, Transformer, and Cross Transformer to obtain four training weights.

[0021] Finally, the weights pre-trained from the four deep network classifiers are trained with soft voting fusion at the decision level to obtain the final recognition model, through which ancient characters are recognized.

[0022] Beneficial effects:

[0023] Targeting the long-tail data distribution characteristics of ancient characters, this paper proposes an optimized fusion model for ancient character recognition, effectively improving its efficiency. This model combines multiple techniques, including deep learning, transfer learning, and data augmentation. By modeling and processing the long-tail data distribution, the model of this paper is able to better handle the recognition of low-frequency ancient characters. The innovation of this fusion model lies in its effective utilization of the data distribution characteristics, improving the accuracy and robustness of ancient character recognition.

[0024] The present invention achieves the translation of entire ancient text copies through methods such as target detection and style transfer. Traditional ancient text translation methods can usually only handle the translation of single characters or phrases, but the method of the present invention can handle the translation task of entire ancient text copies. Through target detection technology, the present invention can accurately locate and extract ancient text areas, and then use deep network technology to convert the extracted ancient text areas into text in the target style. The innovation of this method is that it expands the scope of ancient text translation and provides a more comprehensive and accurate ancient text translation solution.

[0025] By proposing a fusion model for ancient Chinese character recognition based on long-tail optimization and enabling the translation of entire ancient Chinese character copies, this invention highlights the processing of long-tail data distribution and expands the scope of ancient Chinese character translation. These innovations and benefits make this invention of great practicality and promotional value in the field of ancient Chinese character research and application. Compared with existing classification methods, the method of this invention can effectively solve the problems of establishing ancient Chinese character databases and recognition, and has a certain degree of adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of the present invention;

[0027] Figure 2 is a flow chart of the iterative data acquisition method of the present invention;

[0028] Figure 3 This is a comparison chart of style transfer in the present invention;

[0029] Figure 4 This is the YOLOv7-tiny architecture diagram of the present invention;

[0030] Figure 5 This is a diagram of the algorithm structure of the present invention based on the long-tail method optimization;

[0031] Figure 6 It is a ResNet network structure diagram adopted by the present invention;

[0032] Figure 7It is a structural diagram of the multi-scale fusion model of the present invention;

[0033] Figure 8 Schematic diagram of the principle of the soft voting method of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further explained below with reference to the accompanying drawings.

[0035] Example 1:

[0036] Data collection and database construction: Classify, collect and annotate existing ancient character samples.

[0037] First, the images contained in existing ancient text documents were divided into three main types: ancient text rubbings, ancient text copies made by experts, and single characters. The numbers of the original ancient text rubbings and the copies made by experts correspond to each other.

[0038] Next, we collected image data from ancient Chinese character rubbings. We took screenshots to obtain sample images of the entire ancient Chinese character rubbing, named them according to their sample numbers, and stored them in folders. We then used the LabelMe tool to annotate each character in the ancient Chinese character rubbing and sample image, saving the annotation results as a JSON file with the corresponding name.

[0039] Finally, collect image data of ancient characters by taking screenshots. For known characters, place the images in the corresponding label folder; for unknown characters, temporarily store them in the "unknown" folder. Deep network training is then performed on the constructed character database, and predictions are made for the characters in the "unknown" folder. The predicted results are compared with the target results. After manual verification, the correctly predicted characters are added to the known dataset to update the dataset. This step is repeated continuously, continuously reducing the number of samples in the "unknown" folder and expanding the dataset through iteration.

[0040] Ancient character target detection: By analyzing the constructed database of ancient character copies and rubbings, the frequency of known single characters on each copy is counted, and ancient character copies with high frequencies of known single characters are selected as training sets and validation sets, and the rest are used as test sets. A style transfer algorithm is used to convert images of ancient character copies and rubbings, thereby expanding the dataset while enhancing the generalization ability of the model. The CycleGAN algorithm can achieve cross-domain image conversion through two mutually inverting mapping functions. Its loss function is a combination of adversarial loss and cycle consistency loss. By reducing the adversarial loss and cycle consistency loss, the style transfer of copies and rubbings is achieved.

[0041] The rubbing image is referred to as domain A and the copy image is referred to as domain B. The image style of domain A is relatively fuzzy and realistic, with cracks, blur and even incomplete content; the image style of domain B is clear and has obvious outlines. To achieve cross-domain image conversion, two generator functions are required. The function that converts the image of domain A to domain B is denoted as G A , the discriminator is D B . With G A The inverse function is G B ,, then G A The adversarial loss L adv It can be expressed as:

[0042]

[0043] Discriminator D B The loss function for:

[0044]

[0045] Among them, A is the real image in domain A, and B is the real image in domain B.

[0046] In order to keep the image style transfer and content consistent, the cycle consistency L is introduced. cycle , which means the difference between the image converted from domain A to domain B and then from domain B back to domain A and the original image, L cycle It can be expressed as:

[0047]

[0048] Therefore, the overall loss function can be defined as the weighted sum of the generator and discriminator losses:

[0049]

[0050] Among them, λ is the weight parameter of the cycle consistency loss.

[0051] Based on the style migration and expansion data set, the present invention adopts the YOLOv7-tiny algorithm to detect and identify the ancient character rubbings database.

[0052] Ancient Chinese Character Recognition: First, the data is segmented and preprocessed. During data segmentation, due to the data's long-tail distribution, random sampling is performed on the head class to reduce the impact of an overabundance of head-class samples. Then, using stratified sampling, the dataset is divided into training, validation, and test sets. During data preprocessing, due to the horizontal mirror image nature of some ancient characters, sample size scaling, degree rotation, and optical color adjustment are employed.

[0053] Secondly, a deep neural network algorithm optimized with four long-tail methods is pre-trained. The Mixup method is used as a long-tail optimization method, performing random linear interpolation on the input to generate training samples. Mixup mixes two different samples x1 and x2 in a certain ratio to obtain a new sample x, and simultaneously mixes the labels y1 and y2 corresponding to these two samples in the same ratio to obtain a new label y. Specifically, let λ be the mixing coefficient (0≤λ≤1), then the new sample x can be expressed as:

[0054] x=λx1(1-λ)x2

[0055] The new label y can be expressed as:

[0056] y=λy1(1-λ)y2

[0057] This method can generate an unlimited number of new training samples, each containing distinct features and labels. Its advantages include preventing overfitting and enhancing the model's robustness and generalization capabilities, particularly in cases of class imbalance. The generated (x, y) is then trained in four deep network classifiers: AlexNet, ResNet, Transformer, and Cross Transformer, yielding four training weights.

[0058] Next, the pre-trained weights of the four classifiers are trained using soft voting fusion at the decision level to produce the final recognition model. The voting method simultaneously trains multiple classification models and then merges and averages them to obtain the fusion model's results. Voting methods include hard voting and soft voting. Hard voting achieves a fusion result by voting on the predictions of various models, while soft voting uses the probability of each model's prediction as weight and performs weighted voting based on hard voting. Generally speaking, soft voting produces better results than hard voting. Therefore, the soft voting method is used to perform decision classifier fusion across AlexNet, ResNet, Transformer, and Cross Transformer.

[0059] Assume that Naive Bayes (NB) combination is used to integrate the classifier label output. Mathematically, given a test image with xN base classifiers, the nth classifier l n The predicted probability of class n is expressed as, Among them, conditional independence can be defined as l n ∈{1,…,C}

[0060]

[0061] where w∈{1,…,C} is the class label predicted by the classifier combination. The posterior probability required by the nth classifier can be interpreted as

[0062]

[0063] It should be noted that θ(w) does not return l n Therefore, the final probability x formula is,

[0064]

[0065] Among them, the max function is used to select the final class, that is, the maximum probability of all classes.

[0066] Integrating target detection and single-word classification: The present invention ultimately implements ancient Chinese character target detection for the entire input ancient Chinese character copy using the trained YOLOv7-tiny model; translates the detected single words using the trained fusion model optimized based on the long-tail method; and finally outputs the ancient Chinese character translation result for the entire rubbing.

[0067] Example 2:

[0068] The invented method for collecting, detecting and recognizing ancient characters based on optimized deep neural networks mainly includes four implementation steps: data collection and database construction, data set segmentation, ancient character detection and ancient character recognition. First, collect rubbings and single-character samples of ancient characters and perform data annotation. Secondly, perform target detection on the ancient character rubbings and copy samples. Furthermore, the deep network algorithm is optimized by the long-tail method, and the ancient character single-character samples are classified and recognized, and multiple optimized models are fused at the decision level. Thus, the final model is obtained. Finally, the target detection and single-character classification are integrated, that is, the input is the whole ancient character picture, and the output is the whole picture of the ancient character translation. The following is combined with Figure 1 The present invention is further described in detail with reference to specific embodiments. The present invention takes the ancient characters of Houma League Book as an example to complete the collection, detection and recognition of ancient characters.

[0069] The ancient characters from the Houma League Book are one of the ten great archaeological discoveries of New China and are hailed as a "national treasure." Taking the ancient characters from the Houma League Book as an example, the specific process for collecting, detecting, and identifying these characters involves the following steps:

[0070] Step 1: Data collection and database construction;

[0071] Step 2: Data segmentation;

[0072] Step 3: Ancient character detection;

[0073] Step 4: Ancient character recognition.

[0074] First, data collection and database construction were performed. Using the LabelMe tool, we collected rubbings and copies of the ancient text from the Houma League Book. The collected images were named x_y_z, where x represents the excavation pit, y represents the serial number from which the rubbing emerged, and z represents the sample type. When z = 1, the sample is a copy, and when z = 2, the sample is the corresponding rubbing.

[0075] In the collected rubbings and facsimile images, ancient characters were annotated. Single-character samples were annotated based on the 327 categories identified in the Houma League Book, and unidentified samples were labeled "unknown."

[0076] Through an iterative data collection method, the amount of data of known samples is expanded. Figure 2 As shown in Figure 1, the core of this method is to expand the training set through the data collection process of "data collection, model training, model prediction, test and proofreading, and update the data set", and obtain sufficient data set samples through multiple iterations.

[0077] Through this collection method, the constructed dataset contains 327 categories of ancient characters, 26,732 known single-character samples, 655 copies and 63 rubbings.

[0078] Furthermore, the data set is segmented. The present invention segments the rubbings and copies database and the ancient character single-word database respectively. For the rubbings and copies database, 120 copies with more than 80% of known single words are selected and divided into training sets and validation sets according to the ratio of 8:2. The remaining copies and rubbings images with a smaller proportion of known single words are used as test sets. For the single-word database, due to the long-tail data distribution characteristics of ancient characters, 200 samples are randomly selected for the ancient character single-word categories with more than 200 samples for segmentation. During segmentation, the ratio of training set, validation set and test set is 6:2:2 according to the stratified sampling method.

[0079] Furthermore, ancient characters detection is performed. Ancient characters detection is divided into two steps: first, the training set and test set are expanded by adopting the style transfer method; second, the YOLOv7-tiny algorithm is used to train and test ancient characters target detection. Style transfer is mainly used to expand the rubbings and copy data sets required for target detection. The present invention uses the method of adversarial generative network to transfer the style of copies and rubbings. The CycleGAN algorithm is used to realize the mutual conversion of two different styles of copies and rubbings. The results of CycleGAN style transfer are shown in the figure below. Figure 3 Then, the present invention uses the YOLOv7-tiny model to detect and identify the ancient character rubbings database, and its network structure is as follows Figure 4 As shown. The main confidence and non-maximum suppression thresholds are set to 0.1 and 0.5, and the following four indicators are used for evaluation:

[0080] (1) Precision, expressed as

[0081] (2) Recall, expressed as

[0082] (3) Average mean precision measured at the intersection over union (IOU);

[0083] The harmonic mean of precision and recall is F1,

[0084] Among them, TP, FP and FN respectively represent the number of Houma League Book data identified as Houma League Book samples, the number of non-Houma League Book data identified as Houma League Book samples and the number of Houma League Book data identified as non-Houma League Book samples.

[0085] After completing the ancient character target detection, the ancient character single words are cut out one by one according to the detection results, and input into the fourth trained ancient character recognition model to obtain the prediction results.

[0086] Furthermore, ancient Chinese character recognition is performed. This process can be divided into three parts: first, preprocessing and long-tail data generation at the data input stage; second, training a deep network model; and finally, decision-level fusion of multiple trained deep network models to obtain an ancient Chinese character recognition fusion model optimized using the long-tail method.

[0087] The input image is pre-processed by Tranforms, including scaling, vertical flipping, color adjustment, random angle flipping, and grayscale normalization, to perform preliminary data expansion. The process is as follows Figure 5 As shown in the figure, in the long-tail data augmentation module, Mixup is used to augment the data, mitigating the impact of the head class on overall recognition. Mixup balances the data by mixing images. In the experiment, λ in Mixup is set to 1.0.

[0088] The present invention uses four benchmark classifiers, Alexnet, ResNet, Vision Transformer, and Cross Transformer, for training. The trained weights are used for the ancient text image recognition and handwriting recognition functions of the software platform. The learning rate during training is set to 0.001. When training the AlexNet model, the parameters are updated by SGD gradient descent, which improves the memory usage efficiency and enhances the generalization ability of the model. When training the ResNet model, the transfer learning method is adopted. A fully connected layer with a length of 327 is added on the basis of ResNet34. The structure is as follows: Figure 6When training the Cross Transformer model, multi-scale dual-branch fusion is used to achieve the fusion of large patches and small patches. The multi-scale fusion model is shown in Figure 7 As shown in Figure 2, f(·) adjusts the CLS module of the large patch to the same dimension as the small patch. g(·) does the opposite, mapping the dimension back to the large patch.

[0089] Conduct decision-level network fusion and training. Conduct fusion training on the four classifiers at the decision level. Each time, four classifiers are trained simultaneously. The prediction result is the result obtained by voting of the four classifiers. Voting adopts the soft voting method as follows Figure 8 As shown, Naive Bayes (NB) combination is used to integrate the classifier label output. Mathematically, given a test image with xN base classifiers, the nth classifier l n The predicted probability of class n is expressed as, Among them, conditional independence can be defined as l n ∈{1,…,C}

[0090]

[0091] where w∈{1,…,C} is the class label predicted by the classifier combination. The posterior probability required by the nth classifier can be interpreted as

[0092]

[0093] It should be noted that θ(w) does not return l n Therefore, the final probability x formula is,

[0094]

[0095] Among them, the max function is used to select the final class, that is, the maximum probability of all classes.

[0096] After completing the ancient character recognition, the modern Chinese character translation corresponding to the ancient character word is scaled to an appropriate size and replaced with the original ancient character word on the original image input in step three.

[0097] The method was tested and experimented on a database of ancient characters from the Houma League Book. In actual testing, the target detection consisted of 192 training samples and 48 test samples. For single-character classification, the training, test, and validation sets contained 20,035, 3,278, and 3,419 samples, respectively. Based on this, the ancient character recognition model of the present invention achieved a prediction accuracy of over 90% for the ancient characters in the Houma League Book.

[0098] In summary, the present invention optimizes the deep network model based on the long-tail distribution characteristics of ancient characters, thereby improving the accuracy of ancient character recognition; and integrates the target detection algorithm to achieve the goal of translating the entire copy of the Houma League Book ancient characters.

[0099] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for collecting, detecting and recognizing ancient Chinese characters based on an optimized deep network, characterized in that: The following steps are involved: 1) Acquisition and expansion of data sets; 2) Ancient Character Detection: The YOLOv7-tiny model is trained using the expanded facsimile images to obtain a trained YOLOv7-tiny model. The trained YOLOv7-tiny model is then used to detect ancient characters in the original rubbings of ancient characters. After completing the ancient character target detection, the ancient character single words are cut out one by one according to the detection results and input into the trained ancient character recognition model; 3) Ancient Character Recognition: Based on the expanded images of original ancient character rubbings and copies, images of individual ancient characters are captured and preprocessed and segmented. The preprocessing includes scaling, rotation, and optical color adjustment of the images. Based on the characteristics of the long-tail distribution, the head classes of the images of individual ancient characters are randomly sampled, and then the images are segmented into training, validation, and test sets using a stratified sampling method. Secondly, the long-tail distribution features are enhanced by the Mixup long-tail optimization method. Let λ be the mixing coefficient, where 0≤λ≤1, then the new sample x is expressed as: x=λx1(1-λ)x2 The new label y is represented as: y=λy1(1-λ)y2 The two randomly sampled image samples are x1 and x2, and the corresponding labels are y1 and y2; Next, the generated (x, y) pairs are pre-trained in four deep network classifiers: AlexNet, ResNet, Transformer, and Cross Transformer to obtain four training weights. Finally, the weights pre-trained from the four deep network classifiers are trained with soft voting fusion at the decision level to obtain the final recognition model, which is then used to identify ancient characters.

2. The method for collecting, detecting and recognizing ancient Chinese characters based on an optimized deep network as claimed in claim 1, wherein: The implementation process of step 1) is as follows: collect the original rubbings and copies of ancient characters and save them in a folder. Then use the LabelMe tool to annotate the ancient characters in the original rubbings and copies. The labels of the original rubbings and copies are unified. The annotated ancient characters are used as the initial data set, and the unannotated, i.e., unknown, ancient characters are placed in the "unknown" folder. Perform deep network training on the initial dataset, and predict unknown ancient characters in the "unknown" folder. Add the correctly predicted unknown ancient characters to the initial dataset, and iterate the training to expand the initial dataset. The frequency of known ancient characters on each ancient character copy image is counted and arranged in ascending order of frequency. The last 20% of the copy images are selected as training sets and validation sets, and the rest are used as test sets. The style transfer CycleGAN algorithm is used to perform image conversion between the two styles of the original ancient character rubbing images and the copy images, expanding the original ancient character rubbing images and ancient character copy images collected in step 1).

Images