A method for detecting the order of ancient books and documents

By constructing the adjacency matrix and features of the graph, training character connection relationships and text line order prediction models, the error problem in the order detection of ancient documents intensive documents is solved, and efficient document digitization and information extraction are achieved.

CN114648770BActive Publication Date: 2024-11-19SOUTH CHINA UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210354142.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-06
Publication Date
2024-11-19
Estimated Expiration
2042-04-06

AI Technical Summary

Technical Problem

The existing document reading order detection method has errors in handling unconstrained arrangements and intensive ancient documents, resulting in errors in output tasks such as information extraction, and lacks a robust and general document order detection scheme.

Method used

A method for detecting order of ancient documents is proposed. By obtaining image data, character connection order annotation and text line order annotation, constructing the adjacency matrix and features of the graph, training character connection relationship prediction model and text line order prediction model, and using encoding-decoding sequence The model obtains document content that matches the reading order.

Benefits of technology

It realizes accurate sequential detection of intensive ancient documents, reduces the manpower labeling process, improves the efficiency of document digitization process, and is better than traditional methods in terms of accuracy, recall and harmonization average.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114648770B_ABST
    Figure CN114648770B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for detecting the order of ancient books and documents, comprising: obtaining image data, annotating the connection order of each character and the text line order of the entire document based on the image data, and obtaining a training data set; constructing an adjacency matrix of a graph through spatial geometric relationships based on the training data set, constructing the features of each node in the graph and the edge features, training a character connection relationship prediction model, and obtaining the text line after connecting the characters; constructing and training a text line order prediction model based on the training data set through a sequence model of encoding and decoding, and obtaining a text line order prediction result; according to the image data, obtaining the document content of the image data that conforms to the reading order based on the training character connection relationship prediction model and the training text line order prediction model. The present invention is aimed at document images of dense characters, performs sequence detection based on character detection results, and proposes a single-word connection relationship prediction model and a text line order prediction model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a method for detecting the order of documents, and in particular to a method for detecting the order of ancient book documents. Background Art

[0002] Document reading order detection, as a basic task for understanding visually rich document content, refers to obtaining sorted fragment output that readers can directly understand. Most existing reading order detection methods obtain results through heuristic methods, that is, the final content is output from the detected character results in a reading order from top to bottom and from left to right. However, for document content with unconstrained arrangement, especially multi-page and vertical layout structures such as ancient documents, the results obtained by this heuristic method are mostly wrong. Incorrect reading order output will lead to incorrect output results for other document understanding tasks, such as information extraction. Therefore, a robust and general document order detection scheme is indispensable for document understanding tasks.

[0003] In recent years, relevant scholars have proposed solutions for document sequence detection. Since the document sequence annotation dataset requires a lot of manual annotation, these methods only use a small number of datasets for training. Some relevant scholars have also proposed some heuristic rules to solve this problem. Recently, some scholars have proposed a method based on deep learning to solve the sequence detection problem in e-commerce scenarios. However, this method uses a dataset with fewer text fragments and does not face the challenge of dense characters. Therefore, there is an urgent need for a method for dense ancient document sequence detection, especially for dense ancient document image data, which can further transcribe paper documents, reduce the manual annotation process, and further accelerate the document digitization process. Summary of the invention

[0004] The purpose of the present invention is to provide a method for detecting the sequence of ancient book documents, so that the data of the document type can be accurately transcribed into paper documents.

[0005] To achieve the above object, the present invention provides a method for detecting the sequence of ancient book documents, comprising:

[0006] Acquire image data, and perform connection sequence annotation of each character and text line sequence annotation of the entire document based on the image data to obtain a training data set;

[0007] Based on the training data set, an adjacency matrix of a graph is constructed through spatial geometric relationships, features of each node and edge features of the graph are constructed, a character connection relationship prediction model is trained, and a text line after connecting the characters is obtained;

[0008] Based on the training data set, a text line order prediction model is constructed and trained through an encoding-decoding sequence model to obtain a text line order prediction result;

[0009] According to the image data, based on the training character connection relationship prediction model and the training text line order prediction model, the document content of the image data that conforms to the reading order is obtained.

[0010] Optionally, the image data includes: a handwritten unconstrained text line dataset CASIA, an ancient book document dataset with regular character arrangement TKH, and an ancient book document dataset with diverse single-word arrangement MTH; the text data includes: an ancient book dataset annotated with text lines.

[0011] Optionally, the continuous sequence marking includes: the next connecting character of each single word character.

[0012] Optionally, constructing a training character connection relationship prediction model includes: calculating the L2 distance of each character node, finding the nearest 8 character nodes for each character node, constructing a k-order subgraph network, constructing node features and edge features of the graph based on the geometric structure of each character, constructing an adjacency matrix of the graph based on spatial geometric relationships, and training to obtain a connection relationship graph network prediction model based on character nodes.

[0013] Optionally, constructing a training character connection relationship prediction model also includes a threshold, and the threshold is used to determine whether the image data can be trained; based on the spatial L2 distance, an 8-neighborhood subgraph of each node is constructed, and the IoU of the number of nodes is calculated. When the IoU is greater than the threshold, the image data is not trained, otherwise it is trained.

[0014] Optionally, in the constructed k-order subgraph network, the node features are the normalized center point coordinates and the normalized width and height of each character; the edge features are the aspect ratio and the horizontal and vertical coordinate distance difference between characters; the constructed adjacency matrix is ​​the first k nodes in the KNN matrix that two character nodes satisfy for each character node.

[0015] Optionally, the encoding-decoding sequence model includes: based on the layout coordinate information and position number information of the text line, obtaining the encoded feature sequence through the Transformer model, adding a classification branch through the Decoder, predicting the index of the input sequence corresponding to each time step, and calculating the loss based on the cross entropy loss to train the encoding-decoding sequence model according to the prediction results of each time step.

[0016] Optionally, based on the character input in the data set, a character connection relationship prediction model is used to obtain the connected text line output; through the text line output, a text line sequence prediction model is used to obtain the final document sequential output content.

[0017] Technical effect of the invention: The invention discloses a method for detecting the order of ancient books and documents. For dense character document images, the method performs order detection based on character detection results, and proposes a single-word connection relationship prediction model and a text line order prediction model. Combining modern computer information technology with document digitization content plays an important role in digital heritage protection, information discovery, and automated paper document transcription processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 Schematic diagram of the flow of the ancient book document sequence detection method according to an embodiment of the present invention;

[0020] Figure 2 A network diagram for predicting character connection relationships according to an embodiment of the present invention;

[0021] Figure 3 A network diagram for predicting the order of text lines in an embodiment of the present invention;

[0022] Figure 4 The following are visualization effect diagrams of the character connection relationship of the embodiment of the present invention, wherein (a) is the CASIA dataset, (b) is the TKH dataset, and (c) is the MTH dataset;

[0023] Figure 5 This is a visualization effect diagram of text line order prediction according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0026] like Figure 1-5 As shown, this embodiment provides a method for detecting the order of ancient book documents, including:

[0027] Acquire image data, and perform connection sequence annotation of each character and text line sequence annotation of the entire document based on the image data to obtain a training data set;

[0028] Based on the training data set, an adjacency matrix of a graph is constructed through spatial geometric relationships, features of each node and edge features of the graph are constructed, a character connection relationship prediction model is trained, and a text line after connecting the characters is obtained;

[0029] Based on the training data set, a text line order prediction model is constructed and trained through an encoding-decoding sequence model to obtain a text line order prediction result;

[0030] According to the image data, based on the training character connection relationship prediction model and the training text line order prediction model, the document content of the image data that conforms to the reading order is obtained.

[0031] Further optimizing the scheme, the image data includes: a handwritten unconstrained text line dataset CASIA, an ancient book document dataset with regular character arrangement TKH, and an ancient book document dataset with diverse single-word arrangement MTH; the text data includes: an ancient book dataset annotated with text lines.

[0032] Obtaining data: For single-word connection relationship prediction and text line order prediction, we divide the experimental data set construction into two parts. The single-word connection relationship prediction includes the handwritten unconstrained text line data set CASIA. The characteristic of this data set is that the single words in it are all handwritten unconstrained text lines, and they also contain punctuation marks; the TKH data set is an ancient book document data set, in which the characters of the data set are arranged regularly; the MTH data set is an ancient book document data set, in which the single words are arranged in a variety of ways, and there are multiple layout areas and double-column annotations, which makes training more difficult. Among them, the text line order prediction experiment uses an ancient book data set. Different from the MTH data set, this data set is annotated with text line annotations to predict the order index of the text line. Table 1 shows the distribution statistics of the single-word connection relationship prediction data set, and Table 2 shows the distribution statistics of the text line order prediction data set. Table 1 shows the distribution statistics of the data set used for single-word connection relationship prediction; the specific training and test data statistics are shown in Tables 1 and 2.

[0033] Table 1

[0034] Number of training set images Number of test set images CASIA Dataset 5091 300 TKH dataset 749 250 MTH Dataset 1649 551

[0035] Table 2

[0036] Number of training set images Number of test set images MTH textline 1062 119

[0037] Further optimizing the scheme, the continuous sequence marking includes: the next connecting character of each single word character.

[0038] Further optimization scheme, construction of training character connection relationship prediction model includes: by calculating the L2 distance of each character node, finding the nearest 8 character nodes for each character node, constructing a k-order subgraph network, constructing the node features and edge features of the graph based on the geometric structure of each character, constructing the adjacency matrix of the graph based on spatial geometric relationships, and training to obtain a connection relationship graph network prediction model based on character nodes.

[0039] A further optimization scheme is provided, wherein the training character connection relationship prediction model also includes a threshold value, wherein the threshold value is used to determine whether the image data can be trained; based on the spatial L2 distance, an 8-neighborhood subgraph of each node is constructed, and the IoU of the number of nodes is calculated. When the IoU is greater than the threshold value, the image data is not trained, otherwise it is trained.

[0040] Training character connection relationship model: For the input document image, since there are many characters in the document, especially the number of characters in ancient books is thousands, it requires a lot of calculation to construct a graph network directly based on all the characters in the document. Therefore, multiple subgraphs are constructed based on the L2 coordinate distance of the characters in the document. At the same time, there may be a large redundancy between subgraphs. In order to avoid repeated calculations and enable the network to learn effectively, the IoU between subgraphs is calculated. When the IoU is greater than the set threshold, the calculation of the subgraph is ignored. For each subgraph, we use characters as nodes of the graph network and construct the node features, edge features and adjacency matrix of the graph in turn.

[0041] For a certain node, its adjacency matrix is ​​whether the other character nodes meet the L2 distance in the nearest k nodes. If they meet the L2 distance, the adjacency matrix is ​​assigned a value of 1, where the value of k is 4. The node features of the graph network include visual features and geometric features. The visual features are the features of fixed dimension size obtained by passing the image through the ResNet18 network and the area at the corresponding coordinate position through RoIAlign; the geometric features are the normalized center point coordinates and width and height. The edge features of the graph network include the center point coordinate difference and aspect ratio between each character box.

[0042] After encoding the adjacent nodes, the encoded edge features are used to predict the connection relationship between the nodes. Due to the large difference in the ratio of positive and negative samples, the ratio of positive and negative samples is kept at 1:1 during training. The cross entropy loss is used to calculate the loss size and optimize the update of model parameters.

[0043] The experimental results on the three data sets are shown in Tables 3-5. Compared with the heuristic scheme and the method of directly using the fully connected layer classification, the method of the present invention has more obvious advantages in precision (Precision), recall rate (Recall), and harmonic mean (F-score). Table 3 is the comparison of single-word connection experiments on the CASIA data set, Table 4 is the comparison of single-word connection experiments on the TKH data set, and Table 5 is the comparison of single-word connection experiments on the MTH data set.

[0044] Table 3

[0045]

[0046] Table 4

[0047]

[0048] Table 5

[0049]

[0050] A further optimization scheme is to construct a k-order subgraph network, wherein the node features are the normalized center point coordinates and the normalized width and height of each character; the edge features are the aspect ratio and the horizontal and vertical coordinate distance difference between characters; the constructed adjacency matrix is ​​the first k nodes in the KNN matrix of each character node that are satisfied by two character nodes.

[0051] A further optimization scheme is provided, wherein the encoding-decoding sequence model includes: obtaining the encoded feature sequence through the Transformer model based on the layout coordinate information and position number information of the text line, adding a classification branch through the Decoder, predicting the index of the input sequence corresponding to each time step, and training the encoding-decoding sequence model based on the cross entropy loss calculation loss according to the prediction results of each time step.

[0052] Training text line order prediction model: The text line order prediction model consists of an encoding-decoding structure, in which the encoding model input includes the layout coordinate information (Layout Embedding) of the text line and the position number information, and each encoded sequence is obtained through the Transformer model; the decoding part adds a classification branch to predict the index of the input sequence corresponding to each time step. For the prediction results of each time step, the loss is calculated through the cross entropy loss to train the model. Specifically, we introduce data enhancement and data synthesis technology to further improve the performance of our model. For data enhancement, that is, by replacing and deleting text lines, the number of images in the original data set is increased; at the same time, for the document layout of the ancient book data set, additional document images with double-column annotations are synthesized.

[0053] To further optimize the solution, based on the character input in the data set, the connected text line output is obtained through the character connection relationship prediction model; through the output of the text line, the text line sequence prediction model is used to obtain the final document sequential output content.

[0054] In terms of indicator evaluation, the average relative distance is used to evaluate the performance of the model. Specifically, for the input sequence box, the index corresponding to the predicted sequence is found, the error between the predicted index and the target index is determined, and the absolute value of the relative difference is calculated for evaluation. As shown in Table 6, for the MTH text line sequence prediction experiment comparison, our solution is better than the heuristic rule and has a lower error result.

[0055] Table 6

[0056] Heuristic Solutions Proposed solution MTH Textline 0.0927 4.8269 +Data Augmentation 0.0927 0.5708 +Data Synthesis 0.0927 0.0894

[0057] Finally, the summary is as follows: by inputting an ancient book document image and the detected single-word results, after predicting the single-word connection relationship, the output single words are connected to obtain the output of the text line; after predicting the text line sequence, the input text lines are arranged in sequence to obtain the final sequential output result.

[0058] The present invention analyzes the shortcomings of existing methods and proposes a solution for document order detection, which mainly includes a single-word connection relationship prediction model and a text line order prediction model. Since character density is widely present in existing data sets, the solution of first connecting single words into text lines and then further predicting the text line order can greatly reduce the computational complexity and improve the effect of the model.

[0059] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for detecting the sequence of ancient book documents, characterized in that: include: Acquire image data, and perform connection sequence annotation of each character and text line sequence annotation of the entire document based on the image data to obtain a training data set; Based on the training data set, an adjacency matrix of a graph is constructed through spatial geometric relationships, features of each node and edge features of the graph are constructed, a character connection relationship prediction model is trained, and a text line after connecting the characters is obtained; Based on the training data set, a text line order prediction model is constructed and trained through an encoding-decoding sequence model to obtain a text line order prediction result; According to the image data, based on a character connection relationship prediction model and a text line order prediction model, obtaining document content of the image data that conforms to a reading order; Building a character connection relationship prediction model includes: calculating the L2 distance of each character node, finding the nearest 8 character nodes for each character node, constructing a k-order subgraph network, constructing the node features and edge features of the graph based on the geometric structure of each character, constructing the adjacency matrix of the graph based on the spatial geometric relationship, and training to obtain a connection relationship graph network prediction model based on character nodes; The character connection relationship prediction model is constructed, and a threshold is used to determine whether the image data can be trained. Based on the spatial L2 distance, an 8-neighborhood subgraph of each node is constructed, and the IoU of the number of nodes is calculated. When the IoU is greater than the threshold, the image data is not trained, otherwise it is trained. The encoding-decoding sequence model includes: obtaining the encoded feature sequence through the Transformer model based on the layout coordinate information and position number information of the text line, adding a classification branch through the Decoder, predicting the index of the input sequence corresponding to each time step, and training the encoding-decoding sequence model based on the cross entropy loss calculation loss according to the prediction results of each time step.

2. The ancient book document sequence detection method according to claim 1, characterized in that: The image data includes: a handwritten unconstrained text line dataset CASIA, an ancient book document dataset with regular character arrangement TKH, and an ancient book document dataset with diverse single-word arrangement MTH; the text line data includes: an ancient book dataset annotated with text lines.

3. The ancient book document sequence detection method as claimed in claim 2, characterized in that: The continuous sequence marking includes: the next connecting character of each single word character.

4. The ancient book document sequence detection method as claimed in claim 3, characterized in that: In the constructed k-order subgraph network, the node features are the normalized center point coordinates and the normalized width and height of each character; the edge features are the aspect ratio and the horizontal and vertical coordinate distance difference between characters; the constructed adjacency matrix is ​​the first k nodes in the KNN matrix of each character node that are satisfied by two character nodes.

5. The ancient book document sequence detection method according to claim 1, characterized in that: According to the character input in the data set, the connected text line output is obtained through the character connection relationship prediction model; through the output of the text line, the text line sequence prediction model is used to obtain the final document sequential output content.