A document layout recognition method and related apparatus

Abstract

The application discloses a document layout recognition method, comprising: extracting text box information and hierarchical relationship of original document data to obtain training data; constructing an Albert model containing an Embedding layer as an initial recognition model; wherein the Embedding layer is constructed by text features, layout features, image features and page number features; training the initial recognition model by using the training data to obtain a recognition model; and recognizing a to-be-recognized document by using the recognition model to obtain the area category and hierarchical relationship of each text box in the to-be-recognized document. The accuracy and precision of automatic analysis of the document layout are improved. The application also discloses a document layout recognition device, a terminal device and a computer readable storage medium, which have the above beneficial effects.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a document layout recognition method, a document layout recognition device, a terminal device, and a computer-readable storage medium. Background Technology

[0002] With the continuous development of electronic technology, documents today are all in rich text format, including titles, paragraphs, images, tables, headers, footers, etc., and the layouts are also different. For example, when columns are staggered, paragraph splitting or other layout analysis cannot be performed as easily as simple plain text documents using simple rules such as line breaks.

[0003] In related technologies, document layout analysis is generally performed from multiple perspectives. One approach, from an image perspective, can be viewed as an object detection task within the document image, where basic units such as headings, paragraphs, tables, illustrations, headers, and footers are the objects to be detected and identified. Another approach, from a text perspective, can be viewed as a semantic understanding task within the document text, determining whether a text box is a heading, paragraph, table heading, illustration heading, header, or footer based on the semantic information contained in the text lines. For example, "First Consumer Bill Service" can be identified as a heading based on the semantic meaning of "first". Additionally, some headings or paragraphs can be clearly identified by the size of their text boxes; the text box size for headings is often larger than that for paragraph text. However, relying solely on any one of these perspectives will be somewhat one-sided or unable to cover all document layout analysis needs, inevitably leading to errors or limitations in expansion, reducing the accuracy and precision of document layout analysis, and ultimately diminishing its effectiveness.

[0004] Therefore, how to improve the effectiveness of document layout analysis is a key issue of concern to those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a document layout recognition method, a document layout recognition device, a terminal device, and a computer-readable storage medium to improve the accuracy and precision of automated analysis of document layout.

[0006] To address the aforementioned technical problems, this application provides a document layout recognition method, comprising:

[0007] Text box information and hierarchical relationships are extracted from the original document data to obtain training data;

[0008] An Albert model containing an embedding layer is constructed and used as the initial recognition model; wherein the embedding layer is constructed from text features, layout features, image features, and page number features;

[0009] The initial recognition model is trained using the training data to obtain the recognition model;

[0010] The recognition model is used to identify the document to be identified, and the region category and hierarchical relationship of each text box in the document to be identified are obtained.

[0011] Optionally, text box information and hierarchical relationships are extracted from the original document data to obtain training data, including:

[0012] The original document data is processed to extract text boxes, thus obtaining text box information;

[0013] The training data is obtained by constructing a training corpus based on the text box information and the extracted hierarchical relationship information.

[0014] Optionally, an Albert model containing an embedding layer is constructed and used as the initial recognition model, including:

[0015] A fusion embedding layer is constructed based on text feature embedding layer, layout feature embedding layer, image feature embedding layer, and page number feature embedding layer;

[0016] An Albert model is constructed based on the fusion embedding layer, and the Albert model is used as the initial recognition model.

[0017] Optionally, the initial recognition model is trained using the training data to obtain a recognition model, including:

[0018] Construct loss functions for region categories and loss functions for hierarchy determination;

[0019] The initial recognition model is trained based on the loss function of the region category, the loss function of the hierarchy judgment, and the training data to obtain the recognition model.

[0020] This application also provides a document layout recognition device, including:

[0021] The training data acquisition module is used to extract text box information and hierarchical relationships from the original document data to obtain training data.

[0022] The model building module is used to build an Albert model containing an embedding layer, which serves as the initial recognition model; wherein the embedding layer is constructed from text features, layout features, image features, and page number features;

[0023] The model training module is used to train the initial recognition model using the training data to obtain the recognition model;

[0024] The document recognition module is used to recognize the document to be recognized using the recognition model, and to obtain the region category and hierarchical relationship of each text box in the document to be recognized.

[0025] Optionally, the training data acquisition module is specifically used to extract text boxes from the original document data to obtain text box information; and to construct training corpus based on the text box information and the extracted hierarchical relationship information to obtain the training data.

[0026] Optionally, the model building module is specifically used to construct a fusion embedding layer based on the text feature embedding layer, the layout feature embedding layer, the image feature embedding layer, and the page number feature embedding layer; construct an Albert model based on the fusion embedding layer, and use the Albert model as the initial recognition model.

[0027] Optionally, the model training module is specifically used to construct a loss function for region categories and a loss function for hierarchy determination; and to train the initial recognition model based on the loss function for region categories, the loss function for hierarchy determination, and the training data to obtain a recognition model.

[0028] This application also provides a terminal device, including:

[0029] Memory, used to store computer programs;

[0030] A processor is configured to implement the document layout recognition method as described above when executing the computer program.

[0031] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the document layout recognition method described above.

[0032] This application provides a document layout recognition method, comprising: extracting text box information and hierarchical relationships from original document data to obtain training data; constructing an Albert model containing an embedding layer as an initial recognition model; wherein the embedding layer is constructed from text features, layout features, image features, and page number features; training the initial recognition model using the training data to obtain a recognition model; and using the recognition model to recognize the document to be recognized, thereby obtaining the region category and hierarchical relationship of each text box in the document to be recognized.

[0033] By first extracting training data from the original data, and then constructing a model based on the embedding layer that integrates multiple features, the model is trained to effectively analyze various types of information in the document. The trained model is then used to identify the document to be identified, obtaining the region category and hierarchical relationship in the text box. This achieves simultaneous identification of category and relationship, rather than using multiple models for identification. In addition, the integration of multiple features improves the accuracy and precision of document identification, ultimately enhancing the identification effect.

[0034] This application also provides a document layout recognition device, a terminal device, and a computer-readable storage medium, which have the above-mentioned beneficial effects, and will not be elaborated here. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0036] Figure 1 A flowchart illustrating a document layout recognition method provided in an embodiment of this application;

[0037] Figure 2 A schematic diagram of the model structure of a document layout recognition method provided in an embodiment of this application;

[0038] Figure 3 A schematic diagram illustrating the hierarchical relationship of a document layout recognition method provided in an embodiment of this application;

[0039] Figure 4 This is a schematic diagram illustrating feature fusion in a document layout recognition method provided in an embodiment of this application.

[0040] Figure 5 A schematic diagram illustrating the page number encoding method of a document layout recognition method provided in an embodiment of this application;

[0041] Figure 6 This is a schematic diagram of the structure of a document layout recognition device provided in an embodiment of this application. Detailed Implementation

[0042] The core of this application is to provide a document layout recognition method, a document layout recognition device, a terminal device, and a computer-readable storage medium to improve the accuracy and precision of automated analysis of document layouts.

[0043] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] In related technologies, document layout analysis is generally performed from multiple perspectives. One approach, from an image perspective, can be viewed as an object detection task within the document image, where basic units such as headings, paragraphs, tables, illustrations, headers, and footers are the objects to be detected and identified. Another approach, from a text perspective, can be viewed as a semantic understanding task within the document text, determining whether a text box is a heading, paragraph, table heading, illustration heading, header, or footer based on the semantic information contained in the text lines. For example, "First Consumer Bill Service" can be identified as a heading based on the semantic meaning of "first". Additionally, some headings or paragraphs can be clearly identified by the size of their text boxes; the text box size for headings is often larger than that for paragraph text. However, relying solely on any one of these perspectives will be somewhat one-sided or unable to cover all document layout analysis needs, inevitably leading to errors or limitations in expansion, reducing the accuracy and precision of document layout analysis, and ultimately diminishing its effectiveness.

[0045] Therefore, this application provides a document layout recognition method. First, training data is extracted from the original data. Then, a model based on an embedding layer that integrates multiple features is constructed and trained to effectively analyze various types of information in the document. The trained model is then used to recognize the document to be recognized, obtaining the region categories and hierarchical relationships within the text boxes. This achieves simultaneous recognition of categories and relationships, rather than using multiple models. Furthermore, the integration of multiple features improves the accuracy and precision of document recognition, ultimately enhancing the recognition effect.

[0046] The following embodiment illustrates a document layout recognition method provided in this application.

[0047] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a document layout recognition method provided in an embodiment of this application.

[0048] In this embodiment, the method may include:

[0049] S101, Extract text box information and hierarchical relationships from the original document data to obtain training data;

[0050] As can be seen, this step mainly involves extracting training data from the original document data so that the model can be trained using this training data. In this embodiment, since multiple feature data in the document need to be identified and analyzed together, multiple types of data need to be included in the training data for training purposes.

[0051] Furthermore, this step may include:

[0052] Step 1: Extract text boxes from the original document data to obtain text box information;

[0053] Step 2: Construct training corpus based on text box information and extracted hierarchical relationship information to obtain training data.

[0054] As can be seen, this optional solution mainly describes how to specifically obtain training data. In this optional solution, text boxes are extracted from the original document data to obtain text box information. Training corpus is then constructed based on the text box information and the extracted hierarchical relationship information to obtain training data. That is, first, each text box in the document data is identified; then, the hierarchical relationship of each text box in the document is obtained; finally, the extracted information can be constructed into training corpus, i.e., training data, according to a preset format.

[0055] S102, Construct an Albert model containing an Embedding layer and use it as the initial recognition model; wherein, the Embedding layer is constructed from text features, layout features, image features, and page number features;

[0056] Building upon S101, this step aims to construct an Albert model containing an embedding layer, which will serve as the initial recognition model. The embedding layer is constructed from text features, layout features, image features, and page number features. In other words, it constructs the model required for use in this embodiment.

[0057] The Albert model is a Lite BERT, a lightweight version of BERT (Bidirectional Encoder Representations from Transformers, an autoencoder language model). Overall, BERT is an autoencoder language model, and it is designed with two tasks for pre-training.

[0058] The purpose of the embedding layer is to project relatively sparse data in each dimension of a high-dimensional dataset onto a relatively low-dimensional dataset where each dimension can operate on a set of real numbers. Essentially, it replaces a (quasi) discrete space with a continuous space to increase space utilization.

[0059] Furthermore, in order to improve the utilization rate of information in the document and to identify more information features, this embodiment mixes multiple feature categories in the Embedding layer so as to mix multiple types of information during the recognition process and improve the accuracy of page layout recognition.

[0060] Furthermore, this step may include:

[0061] Step 1: Construct a fusion embedding layer based on the text feature embedding layer, layout feature embedding layer, image feature embedding layer, and page number feature embedding layer;

[0062] Step 2: Construct the Albert model based on the fusion embedding layer, and use the Albert model as the initial recognition model.

[0063] As can be seen, this optional scheme mainly describes how to construct the initial recognition model. This optional scheme primarily constructs a fusion embedding layer based on text feature embedding layers, layout feature embedding layers, image feature embedding layers, and page number feature embedding layers; an Albert model is then constructed based on the fusion embedding layer, and this Albert model is used as the initial recognition model. Each feature fusion layer can process corresponding features during data processing for judgment, achieving processing of multiple features and improving the document recognition effect.

[0064] S103, The initial recognition model is trained using training data to obtain the recognition model;

[0065] Building upon S102, this step aims to train the initial recognition model using training data to obtain the recognition model.

[0066] Since this embodiment requires judgments on multiple tasks, multiple loss functions can be used to train the model. These multiple loss functions may include: a region category loss function and a hierarchy judgment loss function.

[0067] Furthermore, this step may include:

[0068] Step 1: Construct the loss function for region category and the loss function for hierarchy determination;

[0069] Step 2: Train the initial recognition model based on the loss function of region category, the loss function of hierarchy judgment, and the training data to obtain the recognition model.

[0070] As can be seen, this optional solution mainly describes how to train the recognition model. This optional solution constructs a loss function for region categories and a loss function for hierarchy judgment; then, based on the region category loss function, the hierarchy judgment loss function, and training data, the initial recognition model is trained to obtain the recognition model. Clearly, this embodiment does not train the recognition model for just one task, but rather for multiple tasks. These multiple tasks include the task of judging the region category of text boxes and the task of judging the hierarchy of text boxes. Obviously, through the training steps in this embodiment, the recognition model can simultaneously recognize and judge the region category and hierarchy relationship of text boxes, thereby improving the efficiency of document recognition.

[0071] S104, The recognition model is used to recognize the document to be recognized, and the region category and hierarchical relationship of each text box in the document to be recognized are obtained.

[0072] Building upon S103, this step aims to use a recognition model to identify the document to be identified, thereby obtaining the region category and hierarchical relationship of each text box in the document.

[0073] Based on the training of the model described above, this step uses the trained recognition model to identify the document. Since this recognition model can make judgments on different tasks, it can obtain the region categories and hierarchical relationships of each text box in the document.

[0074] In summary, this embodiment first extracts training data from the original data, then constructs a model based on the embedding layer that integrates multiple features, and trains it to effectively analyze various types of information in the document. The trained model is then used to identify the document to be recognized, obtaining the region categories and hierarchical relationships within the text boxes. This achieves simultaneous identification of categories and relationships, rather than using multiple models. Furthermore, the integration of multiple features improves the accuracy and precision of document recognition, ultimately enhancing the recognition effect.

[0075] The following specific embodiment will further illustrate the document layout recognition method provided in this application.

[0076] Please refer to Figure 2 , Figure 2This is a schematic diagram of the model structure of a document layout recognition method provided in an embodiment of this application.

[0077] In this embodiment, the method may include:

[0078] Step 10, Data Preparation.

[0079] In this embodiment, the identified documents are mostly PDF (Portable Document Format), Word (a document format), and images. Therefore, the specific steps for constructing training data may include:

[0080] Step 11, Obtain the document's text boxes: text box text, text box position coordinates, and the image on the current page.

[0081] When the document is a PDF, use a PDF parsing tool (if the document is a Word document, it needs to be converted to PDF) to extract the text and convert each page into an image; when the document is an image, use OCR technology to extract the text.

[0082] Step 12, Training corpus preparation: (T,S,V1,V2,L) area ,L relation ).

[0083] Where T represents the text information of a text box, S represents the position coordinates of the page where the text box is located, V1 represents the image information of the text box, and V2 represents the image information of the page where the text box is located.

[0084] Among them, L area This indicates the label of the area corresponding to the text box. Labels include: heading, table of contents, paragraph, table, table heading, illustration, illustration heading, header, and footer. relation This indicates the relationship between the text box and other samples in a batch: 0 represents no relationship, 1 represents an internal relationship, 2 represents a sibling relationship, 3 represents a positive hierarchical relationship, and 4 represents a negative hierarchical relationship.

[0085] Furthermore, the set of region labels is denoted as S. area The set of hierarchical relationships is denoted as S. relation .

[0086] For example, suppose a batch file has 10 text boxes, with "Service of Eight Notices and Legal Documents" as the first text box, which is a title. Then the sample data could be:

[0087] {

[0088] T: "Service of eight notices and legal documents"

[0089] S: [12, 22, 28, 26], where the top-left and bottom-right coordinates are respectively.

[0090] V1: numpy.array

[0091] V2: numpy.array

[0092] L area :"title",

[0093] L relation :{

[0094] "0": [6, 7, 8, 9], # This has no relation to the 6th, 7th, 8th, and 9th samples in this batch.

[0095] "1": [5], # and the 5th sample are siblings, both being titles.

[0096] "2": [1, 2, 3, 4], # and 1, 2, 3, 4 have a positive hierarchical relationship, and similarly, they have a negative hierarchical relationship for this sample.

[0097] “3”:[],

[0098] }}.

[0099] In the actual data, L area It is a label index, such as 0 (the index value corresponding to the area label); the relationship between two text boxes can only belong to one category, and a text box can only belong to one area category.

[0100] S relation Definitions and examples:

[0101] 0 represents no relation: all relationships except those listed below are no relation;

[0102] 1. Internal relation: If a paragraph (or sentence) is at the same level, the internal relation area category must be the same.

[0103] 2 indicates a sibling relationship: text boxes that are not paragraphs but are at the same level. The sibling relationship areas must be of the same category and have the same relationship with other text boxes.

[0104] 3 indicates a hierarchical positive relationship: it has a subordinate relationship, can skip levels, is positive, and the regional categories are definitely different;

[0105] 4 indicates a negative hierarchical relationship: it has a subordinate relationship, can skip levels, is negative, and the regional categories are definitely different;

[0106] Two text boxes with a positive or negative hierarchical relationship will definitely have different area categories; two text boxes with a sibling relationship will definitely have the same area category; and two text boxes with no relationship will definitely have different area categories. Furthermore, titles and titles, titles and paragraphs, tables and table titles, illustrations and illustration titles, titles and tables and table titles, and titles and illustrations and illustration titles have hierarchical relationships.

[0107] Please refer to Figure 3 , Figure 3 This is a schematic diagram illustrating the hierarchical relationship of a document layout recognition method provided in an embodiment of this application.

[0108] Based on the above explanation of the hierarchical relationship, it is evident that... Figure 3 In the first part, 1) and 2) are unrelated; 2) and 3), 2) and 5) are positive hierarchical relationships; 3) and 2), 5) and 2) are negative hierarchical relationships; 3) and 4) are internal relationships; 3) and 5) are sibling relationships; 3) and 7) are sibling relationships; 2) and 6) are sibling relationships; 2) and 7), 7) and 2) are unrelated; 7) and 8) are internal relationships.

[0109] Step 20: Construction of the multimodal embedding layer.

[0110] The Albert model's embedding consists of Segment-Embedding, Position-Embedding, and Token-Embedding. Token-Embedding is only a word embedding and does not contain image or layout representations. Therefore, this embodiment adds image embedding (Visual-Embedding) and layout embedding (Spatial-Embedding) to Token-Embedding to form Fusion-Embedding (replacing Token-Embedding), enriching the representational capabilities of the embedding.

[0111] Please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating feature fusion for a document layout recognition method provided in an embodiment of this application.

[0112] This embodiment mainly illustrates Visual-Embedding, Spatial-Embedding, and how to construct Fusion-Embedding. This process may include:

[0113] Step 21, layout embedding Spatial-Embedding.

[0114] Layout embedding is formed using the coordinates of the text box.

[0115] The coordinates of the text box consist of (x1, y1, x2, y2, w, h), where x1 and y1 are the coordinates of the top left corner of the text box, x2 and y2 are the coordinates of the bottom right corner of the text box, and w and h are the width and height of the text box (note that the coordinates are normalized to 0 to 1200 here).

[0116] Initialize an embedding dictionary for x, y, w, and h, corresponding to the x-coordinate, y-coordinate, width, and height, respectively.

[0117] The embeddings corresponding to the six coordinates of the text box are extracted and then concatenated to form the Spatial-Embedding.

[0118] The text frame coordinates for x2 (i.e., the page number) can be (0, 0, 1200, 1200, 1200, 1200).

[0119] Step 22, Visual Embedding of Images.

[0120] Image embedding involves creating an image corresponding to the area containing the text box. The image is encoded using a ResNeXt-FPN network, then subjected to row-wise average pooling, and finally linear projection to obtain the feature sequence corresponding to the image.

[0121] The image for x2 (i.e., the page number) is an image converted from the entire page.

[0122] Step 23, Fusion-Embedding.

[0123] After obtaining the spatial-embedded layout and visual-embedded image embeddings, they are concatenated with the token-embedded text embeddings, and then passed through a fully connected layer W. F This yields the fusion-embedding of the Chinese character.

[0124] As can be seen, Fusion-Embedding combines the layout of the text box and the features of the image. After passing through the Albert model, it also incorporates the information features of the entire page of images.

[0125] In the page number x2 token-embedding, to avoid page number sparsity and page number coverage, a page number encoding method is invented, which may include:

[0126] First, initialize an embedding dictionary from 0 to 9;

[0127] Second, use 4-digit page numbers, such as 0001 for page 1 and 0012 for page 12 (if it exceeds 9999 pages, truncate it, but it generally will not exceed 9999 pages).

[0128] The page number sequence is processed using a 2-dimensional convolutional CNN (Convolutional Neural Network) with max pooling to obtain the token-embedded page number.

[0129] Please refer to Figure 5 , Figure 5 This is a schematic diagram of the page number encoding method of a document layout recognition method provided in an embodiment of this application.

[0130] The CNN includes a one-dimensional convolutional layer conv1d and a max pooling layer maxpool1d. 0_vector to 9_vector are the representations of page numbers 0 to 9. The CNN then obtains the token-embedded page numbers.

[0131] Step 30, Model Building.

[0132] The document layout analysis model in this embodiment is built on a simplified version of the Albert model. It includes 4 Transformer units, and the embedding layer vector is 128-dimensional with approximately 3 million parameters. The embedding layer is obtained by adding Segment-Embedding, Position-Embedding, and the constructed Fusion-Embedding. The Fusion-Embedding is constructed from three features: text, layout, and image.

[0133] Step 40: Calculate the multi-task loss function.

[0134] This patent document layout analysis model has two training tasks: region category judgment and hierarchical relationship judgment.

[0135] Region category determination refers to determining which region a text box belongs to, such as the <heading> or <paragraph> region. In this case, the first character CLS is classified into multiple categories, which is a multi-category task.

[0136] Hierarchical relationship judgment refers to judging the hierarchical relationship between pairs of samples in a batch. In this case, it involves classifying the first characters of the two text boxes in a batch after concatenation, which is a multi-classification task.

[0137] Step 41, the loss function for region category determination is:

[0138]

[0139]

[0140] Where I is the indicator function, and tag is the set of region tags S. area A tag, P tag Predict the probability that this sample is a tag.

[0141] Step 42, the loss function for determining hierarchical relationships is:

[0142]

[0143]

[0144] Where I is the indicator function, batch is a group during training, and tag is the set of region labels S. relation A tag, P tag The probability of predicting the relationship between samples i and j as a tag. Label the relationship between samples i and j

[0145] Step 43: Summing the two loss functions above yields the overall loss function of the patent document layout analysis model:

[0146] Total loss = Sentence area loss + Sentence relation loss.

[0147] Step 50, Model Optimization.

[0148] Based on the prepared training corpus, multimodal embedding layers, and multi-task loss functions, a deep learning model optimization framework is used to train Albert-tiny (a simplified Albert model), ultimately yielding a document layout analysis model. Compared with single-task, single-modal trained models, it can reconstruct multi-level headings and paragraphs with fine granularity.

[0149] During training, a batch of data belongs to a single document (the text of a document is randomly selected). In each epoch (loop), the text boxes of a document are randomly shuffled and then put into training (therefore, there may be cases of spanning multiple pages in a batch, which supports the learning of hierarchical relationships across pages).

[0150] Step 60: Document layout analysis model reasoning.

[0151] The document layout analysis model in this embodiment has two main steps: region judgment and relationship judgment.

[0152] Step 61, Region Determination.

[0153] Input all the text boxes and image information of the resume document (i.e., according to the data preparation instructions in step 10, except that the label information does not need to be filled) into the trained document layout analysis model to obtain the region category of each text box;

[0154] Step 62, Relationship determination.

[0155] Step 621: Text boxes categorized as headings, table of contents, paragraphs, headers, and footers.

[0156] 1) Classify the relationships in pairs; 2) Combine the internal relationships into a paragraph or sentence (the combined area category and its relationship with other text boxes are also combined into one); 3) Combine the hierarchical relationships into paragraphs with different levels.

[0157] Step 622: Text boxes of the type Table and Table Title.

[0158] Lock onto the page number containing the table area, and then use table methods to structure the table data.

[0159] Step 623: Text box category is illustration, illustration title category.

[0160] The system locates the page number containing illustrations, then uses object detection or semantic segmentation methods to identify and save the illustration information.

[0161] As can be seen, this embodiment first extracts training data from the original data, then constructs a model based on the embedding layer that integrates multiple features, and trains it to effectively analyze various types of information in the document. Then, the trained model is used to identify the document to be identified, obtaining the region category and hierarchical relationship in the text box. This achieves simultaneous identification of category and relationship, rather than using multiple models for identification. In addition, the integration of multiple features improves the accuracy and precision of document identification, and finally improves the identification effect.

[0162] The document layout recognition device provided in the embodiments of this application is described below. The document layout recognition device described below can be referred to in correspondence with the document layout recognition method described above.

[0163] Please refer to Figure 6 , Figure 6This is a schematic diagram of the structure of a document layout recognition device provided in an embodiment of this application.

[0164] In this embodiment, the device may include:

[0165] The training data acquisition module 100 is used to extract text box information and hierarchical relationships from the original document data to obtain training data.

[0166] The model building module 200 is used to build an Albert model containing an embedding layer, which serves as the initial recognition model; the embedding layer is constructed from text features, layout features, image features, and page number features.

[0167] The model training module 300 is used to train the initial recognition model using training data to obtain the recognition model;

[0168] The document recognition module 400 is used to recognize the document to be recognized using a recognition model, and to obtain the region category and hierarchical relationship of each text box in the document to be recognized.

[0169] Optionally, the training data acquisition module 100 is specifically used to extract text boxes from the original document data to obtain text box information; and to construct training corpus based on the text box information and the extracted hierarchical relationship information to obtain training data.

[0170] Optionally, the model building module 200 is specifically used to construct a fusion embedding layer based on the text feature embedding layer, the layout feature embedding layer, the image feature embedding layer, and the page number feature embedding layer; construct an Albert model based on the fusion embedding layer, and use the Albert model as the initial recognition model.

[0171] Optionally, the model training module 300 is specifically used to construct the loss function for region category and the loss function for hierarchy judgment; the initial recognition model is trained based on the loss function for region category, the loss function for hierarchy judgment and the training data to obtain the recognition model.

[0172] This application also provides a terminal device, including:

[0173] Memory, used to store computer programs;

[0174] A processor is configured to implement the steps of the document layout recognition method as described in the above embodiments when executing the computer program.

[0175] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the document layout recognition method described in the above embodiments.

[0176] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0177] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0178] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly in hardware, in a software module executed by a processor, or in a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0179] The foregoing has provided a detailed description of a document layout recognition method, a document layout recognition device, a terminal device, and a computer-readable storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A document layout recognition method, characterized in that, include: Text box information and hierarchical relationships are extracted from the original document data to obtain training data; An Albert model containing an embedding layer is constructed and used as the initial recognition model; wherein the embedding layer is constructed from text features, layout features, image features, and page number features; The initial recognition model is trained using the training data to obtain the recognition model; The recognition model is used to identify the document to be identified, and the region category and hierarchical relationship of each text box in the document to be identified are obtained; The step of extracting text box information and hierarchical relationships from the original document data to obtain training data includes: The original document data is processed to extract text boxes, thus obtaining text box information; The training data is obtained by constructing a training corpus based on the text box information and the extracted hierarchical relationship information; The construction of the Albert model, which includes an embedding layer, and its use as the initial recognition model, includes: A fusion embedding layer is constructed based on text feature embedding layer, layout feature embedding layer, image feature embedding layer, and page number feature embedding layer; An Albert model is constructed based on the fusion embedding layer, and the Albert model is used as the initial recognition model.

2. The document layout recognition method according to claim 1, characterized in that, The initial recognition model is trained using the training data to obtain a recognition model, including: Construct loss functions for region categories and loss functions for hierarchy determination; The initial recognition model is trained based on the loss function of the region category, the loss function of the hierarchy judgment, and the training data to obtain the recognition model.

3. A document layout recognition device, characterized in that, include: The training data acquisition module is used to extract text box information and hierarchical relationships from the original document data to obtain training data. The model building module is used to build an Albert model containing an embedding layer, which serves as the initial recognition model; wherein the embedding layer is constructed from text features, layout features, image features, and page number features; The model training module is used to train the initial recognition model using the training data to obtain the recognition model; The document recognition module is used to recognize the document to be recognized using the recognition model, and to obtain the region category and hierarchical relationship of each text box in the document to be recognized. Specifically, the training data acquisition module is used to extract text boxes from the original document data to obtain text box information; and to construct training corpus based on the text box information and the extracted hierarchical relationship information to obtain the training data. Specifically, the model building module is used to construct a fusion embedding layer based on the text feature embedding layer, the layout feature embedding layer, the image feature embedding layer, and the page number feature embedding layer; construct an Albert model based on the fusion embedding layer, and use the Albert model as the initial recognition model.

4. The document layout recognition device according to claim 3, characterized in that, The model training module is specifically used to construct a loss function for region categories and a loss function for hierarchy determination; and to train the initial recognition model based on the loss function for region categories, the loss function for hierarchy determination, and the training data to obtain the recognition model.

5. A terminal device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the document layout recognition method as described in claim 1 or 2 when executing the computer program.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the document layout recognition method as described in claim 1 or 2.

Images