Document information extraction method, device, equipment and medium

By performing identification processing of documents and pre-training of named entity recognition models, combined with LayoutLM model and mask visual language model, the problems of high cost and poor adaptability of intelligent document extraction in the existing technology are solved, and the simplification of document information extraction is achieved. and efficient adaptability.

CN114821590BActive Publication Date: 2024-11-26CHINA PING AN LIFE INSURANCE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210440065.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-25
Publication Date
2024-11-26
Estimated Expiration
2042-04-25

AI Technical Summary

Technical Problem

In the case of diversified business demand scenarios, existing intelligent document extraction technology requires the development of different solutions for different scenarios, which leads to high work costs and difficulty in maintaining and optimizing. In addition, a general intelligent document solution requires a large number of applications in the migration process of application fields. Manual labeling is expensive.

Method used

A document information extraction method is proposed. By recognizing the document, text blocks are obtained, and inputting them into the pre-trained named entity recognition model to obtain semantic paragraph labels, semantic paragraphs are determined based on the labels, and layout types are determined based on the text reading order. Then, structured data is extracted, and self-supervised pre-training is used to use the LayoutLM model and the mask visual language model to simplify the information extraction process and improve adaptability.

Benefits of technology

The document information extraction process is simplified, the adaptability difficulty of different application fields is reduced, the cost of manual labeling is reduced, and the adaptability and accuracy of document information extraction is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114821590B_ABST
    Figure CN114821590B_ABST
Patent Text Reader

Abstract

The present application relates to the field of information processing technology, and provides a document information extraction method, device, equipment and medium, the method comprising: performing recognition processing on a document to obtain a plurality of text blocks, inputting the plurality of text blocks into a pre-trained named entity recognition model, so as to obtain semantic paragraph labels corresponding to each text block through the named entity recognition model, determining the semantic paragraphs of the document according to the semantic paragraph labels corresponding to each text block, and determining the layout type corresponding to each semantic paragraph in the document based on the text reading order, wherein the text reading order includes a left-right reading order and a top-bottom reading order, and the layout type includes a paragraph layout and a table layout, and extracting the structured data of each semantic paragraph according to the layout type corresponding to the semantic paragraph. The document information extraction method of the embodiment of the present application can simplify the extraction process of document information, facilitate the migration of application fields, and improve the adaptability to different document types.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to a document information extraction method, device, electronic device and computer-readable storage medium. Background Art

[0002] Document intelligence mainly refers to the understanding, classification, extraction and information summarization of the text and layout information contained in the scanned documents through artificial intelligence technology. The structured data obtained by document intelligence technology can also be used for higher-level intelligent applications, such as information mining and intelligent decision-making.

[0003] In the application of relevant document extraction, document intelligence technology generally converts document images into text and symbols through technologies such as Optical Character Recognition (OCR), and then applies Natural Language Processing (NLP) technology to convert the text and symbols of document images into structured information.

[0004] At present, with the diversification of business demand scenarios and templates, document intelligence tasks involve multiple scenarios, such as: medical records, invoices, lists, physical examination reports, insurance policies, bank notes, etc. Different extraction solutions need to be developed for different scenarios, which will obviously greatly increase the work cost and make maintenance and optimization difficult. For general document intelligence solutions, a large amount of manual annotation is required during the application field migration process, and the cost of switching application fields each time is high. Therefore, how to provide a document information extraction method with a simple extraction process and convenient application field migration has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a document information extraction method, device, electronic device and computer-readable storage medium, which can simplify the document information extraction process and improve the adaptability of document information extraction to different application fields.

[0006] To achieve the above objective, a first aspect of an embodiment of the present application proposes a document information extraction method, the method comprising:

[0007] Recognize and process the document to obtain multiple text blocks;

[0008] Inputting the plurality of text blocks into a pre-trained named entity recognition model to obtain semantic paragraph labels corresponding to each of the text blocks through the named entity recognition model;

[0009] Determining the semantic paragraphs of the document according to the semantic paragraph labels corresponding to the respective text blocks;

[0010] Determining a layout type corresponding to each semantic paragraph in the document based on a text reading order, wherein the text reading order includes a left-right reading order and a top-bottom reading order, and the layout types include a paragraph layout and a table layout;

[0011] According to the layout type corresponding to the semantic paragraph, the structured data of each semantic paragraph is extracted.

[0012] According to the document information extraction method provided by some embodiments of the present invention, determining the layout type corresponding to each semantic paragraph in the document based on the text reading order includes:

[0013] Obtaining text feature information corresponding to the semantic paragraph;

[0014] Determining a text reading order of the semantic paragraphs according to the text feature information;

[0015] When the text reading order of the semantic paragraph is a left-right reading order, determining that the layout type of the semantic paragraph is a paragraph layout;

[0016] When the text reading order of the semantic paragraph is a top-bottom reading order, it is determined that the layout type of the semantic paragraph is a table layout.

[0017] According to the document information extraction method provided by some embodiments of the present invention, determining the text reading order of the semantic paragraphs according to the text feature information includes:

[0018] Constructing a binary classification task for determining the text reading order of the semantic paragraph;

[0019] The text feature information is input into a classification model to perform the binary classification task through the classification model to obtain the text reading order of the semantic paragraph.

[0020] According to the document information extraction method provided by some embodiments of the present invention, the text feature information is a word vector matrix of the semantic paragraph, and the classification model is a TextCNN model;

[0021] The step of inputting the text feature information into a classification model to perform the binary classification task through the classification model to obtain the text reading order of the semantic paragraph includes:

[0022] The word vector matrix is ​​input into the TextCNN model to perform the binary classification task through the TextCNN model to obtain the text reading order of the semantic paragraph.

[0023] According to the document information extraction method provided by some embodiments of the present invention, the text feature information is a character feature matrix of the semantic paragraph, and the classification model is an Xgboost model;

[0024] The step of inputting the text feature information into a classification model to perform the binary classification task through the classification model to obtain the text reading order of the semantic paragraph includes:

[0025] Performing maximum pooling on each row of character vectors in the character feature matrix to obtain a row feature vector of the semantic paragraph;

[0026] The row feature vectors of the semantic paragraph are concatenated, and the concatenated row feature vectors are input into an Xgboost model, so as to perform the binary classification task through the Xgboost model and obtain the text reading order of the semantic paragraph.

[0027] According to the document information extraction method provided by some embodiments of the present invention, extracting structured data of each semantic paragraph according to the layout type includes:

[0028] When the layout type of the semantic paragraph is paragraph layout, the following processing is performed:

[0029] Obtaining field attributes and field values ​​in the semantic paragraph;

[0030] Standardizing the field attributes in the semantic paragraph by regular matching;

[0031] The field value is mapped to the standardized field attribute to obtain the structured data of the semantic paragraph.

[0032] According to the document information extraction method provided by some embodiments of the present invention, the named entity recognition model is a two-dimensional named entity recognition model based on the LayoutLM model;

[0033] Before inputting the plurality of text blocks into a pre-trained named entity recognition model to obtain semantic paragraph labels corresponding to each of the text blocks through the named entity recognition model, the method further includes:

[0034] Get the preset document training data set;

[0035] The two-dimensional named entity recognition model is augmented and pre-trained using the document training data set, wherein the augmented pre-training adopts a self-supervised pre-training method based on a masked visual language model.

[0036] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a document information extraction device, the device comprising:

[0037] A text block acquisition module is used to identify and process the document to obtain multiple text blocks;

[0038] A label classification module, used for inputting the plurality of text blocks into a pre-trained named entity recognition model, so as to obtain a semantic paragraph label corresponding to each of the text blocks through the named entity recognition model;

[0039] A semantic paragraph extraction module, used to determine the semantic paragraphs of the document according to the semantic paragraph tags corresponding to each of the text blocks;

[0040] A layout type classification module, used to determine the layout type corresponding to each semantic paragraph in the document based on a text reading order, wherein the text reading order includes a left-right reading order and a top-bottom reading order, and the layout types include a paragraph layout and a table layout;

[0041] The document information extraction module is used to extract the structured data of each semantic paragraph according to the layout type corresponding to the semantic paragraph.

[0042] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program implements the method described in the first aspect when executed by the processor.

[0043] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage, and the storage medium stores one or more computer programs, and the one or more computer programs can be executed by one or more processors to implement the method described in the first aspect above.

[0044] The present application proposes a document information extraction method, device, electronic device and computer-readable storage medium, wherein the document information extraction method obtains a plurality of text blocks by performing recognition processing on the document, and inputs the plurality of text blocks into a pre-trained named entity recognition model, so as to obtain semantic paragraph labels corresponding to each text block through the named entity recognition model, and determine the semantic paragraphs of the document according to the semantic paragraph labels corresponding to each text block, thereby determining the layout type corresponding to each semantic paragraph in the document based on the text reading order, wherein the text reading order includes a left-right reading order and a top-bottom reading order, and the layout type includes a paragraph layout and a table layout, and extracts the structured data of each semantic paragraph according to the layout type corresponding to the semantic paragraph. The embodiment of the present application divides the document according to the layout type on the basis of determining the semantic paragraph corresponding to the document, and performs information extraction for semantic paragraphs of different layout types respectively, which can simplify the document information extraction process, facilitate the migration of application fields, and improve the adaptability to different document types. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flowchart of a document information extraction method provided in an embodiment of the present application;

[0046] Figure 2 It is a flowchart of a document information extraction method provided in an embodiment of the present application;

[0047] Figure 3 It is a flowchart of a document information extraction method provided in an embodiment of the present application;

[0048] Figure 4 It is a flowchart of a document information extraction method provided in an embodiment of the present application;

[0049] Figure 5 It is a schematic diagram of a LayoutLm model structure provided in an embodiment of the present application;

[0050] Figure 6 It is a schematic diagram of an insurance policy provided in an embodiment of the present application;

[0051] Figure 7 yes Figure 6 Schematic diagram of semantic paragraphs in an insurance policy;

[0052] Figure 8 It is a structural schematic diagram of a document information extraction device provided in an embodiment of the present application;

[0053] Fig. 9 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0056] First, some nouns involved in this application are analyzed:

[0057] LayoutLM model. Currently, a large number of research results show that large-scale pre-trained language models can effectively capture the semantic information contained in the text in the pre-training stage through self-supervised tasks, and can effectively improve the model effect after fine-tuning in downstream tasks. However, the existing pre-trained language models mainly focus on the single modality of text, while ignoring the visual structure information of the document itself that is naturally aligned with the text. The general document pre-training model LayoutLM models the document structure information (Document Layout Information) and visual information (Visual Information), allowing the model to perform multi-modal alignment in the pre-training stage. The structure and visual information introduced in the pre-training can be effectively transferred to downstream tasks, and has achieved significant accuracy improvements in downstream tasks such as form understanding (Form Understanding), receipt understanding (Receipt Understanding), and document image classification (Document Image Classification). See Figure 5 , Figure 5 A schematic diagram of the LayoutLM model structure provided in an embodiment of the present application is shown. Figure 5 As shown in the figure, the LayoutLM model structure adds two new Embedding layers, 2-DPosition Embedding and ImageEmbedding, to the existing pre-trained model:

[0058] 2-D Position Embedding, based on the text block obtained by OCR technology, can obtain the specific position of the text in the document. After converting the corresponding coordinates into virtual coordinates, the representation of the coordinates corresponding to the four Embedding sub-layers x, y, w, and h is calculated. The final 2-D Position Embedding is the sum of the Embeddings of the four sub-layers.

[0059] Image Embedding, in the Image Embedding part, the text block corresponding to each text is treated as a candidate box (Proposal) in Faster R-CNN to extract the corresponding local features. In particular, since the [CLS] symbol is used to represent the semantics of the entire input text, the entire document image is used as the Image Embedding of this position to maintain modality alignment.

[0060] Masked Visual-Language Model (MVLM), a self-supervised pre-training task of the LayoutLM model. A large number of experiments have proven that the Masked Language Model (MLM) can effectively perform self-supervised learning in the pre-training stage. Modifications have been made on this basis: after masking the current word, the corresponding 2-D Position Embedding hint is retained, allowing the model to predict the corresponding word. In this method, the model predicts the masked word based on the existing context and the corresponding visual hint, allowing the model to better learn the modal alignment relationship between text position and text semantics.

[0061] Document intelligence mainly refers to the understanding, classification, extraction and information summarization of the text and layout information contained in the scanned documents through artificial intelligence technology. The structured data obtained by document intelligence technology can also be used for higher-level intelligent applications, such as information mining and intelligent decision-making.

[0062] In the application of relevant document extraction, document intelligence technology generally converts document images into text and symbols through technologies such as Optical Character Recognition (OCR), and then applies Natural Language Processing (NLP) technology to convert the text and symbols of document images into structured information.

[0063] At present, with the diversification of business demand scenarios and templates, document intelligence tasks involve multiple scenarios, such as: medical records, invoices, lists, physical examination reports, insurance policies, bank notes, etc. Different extraction solutions need to be developed for different scenarios, which will obviously greatly increase the work cost and make maintenance and optimization difficult. For general document intelligence solutions, a large amount of manual annotation is required during the application field migration process, and the cost of switching application fields each time is high. Therefore, how to provide a document information extraction method with a simple extraction process and convenient application field migration has become a technical problem that needs to be solved urgently.

[0064] Based on this, the embodiments of the present application provide a document information extraction method, device, electronic device and storage medium, which can simplify the process of document information extraction and improve the adaptability of document information extraction in different application fields.

[0065] A document information extraction method, device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the document information extraction method in the embodiments of the present application is described.

[0066] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0067] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0068] The document information extraction method provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the document information extraction method, etc., but is not limited to the above forms.

[0069] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0070] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a document information extraction method provided by an embodiment of the present application. Figure 1 As shown, the document information extraction method includes but is not limited to steps S110 to S150.

[0071] Step S110, performing recognition processing on the document to obtain multiple text blocks.

[0072] It is understandable that by performing recognition processing on a document through OCR technology, a plurality of text blocks in the document are obtained, wherein the text block includes information such as coordinate information of the text block in the document and character text.

[0073] Step S120: input the multiple text blocks into a pre-trained named entity recognition model to obtain semantic paragraph labels corresponding to each of the text blocks through the named entity recognition model.

[0074] It can be understood that, by using the pre-trained named entity recognition model, multi-label classification is performed on the characters in the text block to obtain the corresponding semantic paragraph labels in the text block.

[0075] Exemplarily, by predefining a semantic paragraph tag set of the document, such as Figure 6 As shown, Figure 6 A schematic diagram of an insurance policy provided by an embodiment of the present application is shown. Figure 6For the insurance policy in the example, the semantic paragraph label set can be defined as follows: {company, title, policy information, policyholder information, insured person information, beneficiary information, insurance period, insurance item}. The pre-trained named entity recognition model is used for sequence labeling tasks, and the labeled document data is fine-tuned. Multi-label classification is performed on the words in the text block to obtain the semantic paragraph labels corresponding to each text block.

[0076] In a specific embodiment, when different semantic paragraph labels appear in the text block, label smoothing is performed, that is, a label voting method is used to select the semantic paragraph label with the highest frequency in the text block, and the semantic paragraph label with the highest frequency is used as the semantic paragraph label of all words in the text block, that is, the corresponding semantic paragraph label in the text block.

[0077] In some embodiments, the named entity recognition model is a two-dimensional named entity recognition model based on the LayoutLM model;

[0078] Before inputting the plurality of text blocks into a pre-trained named entity recognition model to obtain semantic paragraph labels corresponding to each of the text blocks through the named entity recognition model, the method further includes:

[0079] Get the preset document training data set;

[0080] The two-dimensional named entity recognition model is augmented and pre-trained using the document training data set, wherein the augmented pre-training adopts a self-supervised pre-training method based on a masked visual language model.

[0081] It should be noted that the embodiments of the present application use Figure 5 The non-CNN part of the LayoutLm model, that is, the five-layer Embedding input layer and the LayouLm Embedding layer in the LayoutLm model. The Text Embedding layer represents the character-level text in the OCR recognition result. In Position Embedding, x0 and y0 represent the standardized coordinates of the upper left corner of the character in the OCR recognition result, and x1 and y1 represent the standardized coordinates of the center point of the text block where the character is located in the OCR recognition result. Coordinate standardization refers to mapping the x-coordinate and y-coordinate, that is, the width and height of the image, to numbers within a certain range, such as 0 to 1000.

[0082] It can be understood that by obtaining a preset document training data set and using the OCR recognition results of the document training data set, the two-dimensional named entity recognition model based on the LayoutLM model is augmented and pre-trained, wherein the training method adopts the MVLM method.

[0083] Exemplarily, when the document information extraction method provided in the embodiments of the present application is applied to insurance policy document information extraction, the named entity recognition model can be augmented and pre-trained by using the OCR recognition results of a large number of insurance policies to obtain a pre-trained named entity recognition model, so as to obtain the corresponding semantic paragraph labels in each text block in the insurance policy document through the pre-trained named entity recognition model.

[0084] It should be understood that according to the application field of the document information extraction method provided in the embodiment of the present application, it is only necessary to select the corresponding document training data set to pre-train the named entity recognition model to facilitate the migration of the application field.

[0085] Step S130: determining the semantic paragraphs of the document according to the semantic paragraph tags corresponding to the text blocks.

[0086] It can be understood that, according to the corresponding semantic paragraph tags in each text block, the text blocks belonging to the same semantic paragraph tag are merged to obtain the semantic paragraph of the document.

[0087] In a specific embodiment, according to the corresponding semantic paragraph tags in each text block, from left to right and from top to bottom, text blocks belonging to the same semantic paragraph tag are merged according to the position information of the text block and the corresponding semantic paragraph tags in the text block to obtain the semantic paragraphs of the document. It can be understood that the specific position of each semantic paragraph in the document can be determined by the position information of the first text block and the position information of the last text block in the semantic paragraph.

[0088] For example, see Figure 6 and Figure 7 , Figure 6 A schematic diagram of an insurance policy provided in an embodiment of the present application is shown. Figure 7 yes Figure 6 Schematic diagram of the semantic paragraph of the insurance policy in . Figure 6 The corresponding semantic paragraph tags in each text block in are determined as follows Figure 7 The semantic paragraph shown, i.e. Figure 7 The document includes the semantic paragraphs {company, title, policy information, policyholder information, insured person information, beneficiary information, insurance period, insurance items}; among them, for example, the semantic paragraph of the policyholder information includes information such as {policyholder's name, date of birth, certificate type and certificate number}.

[0089] Step S140, determining the layout type corresponding to each semantic paragraph in the document based on the text reading order, wherein the text reading order includes a left-right reading order and a top-bottom reading order, and the layout type includes a paragraph layout and a table layout.

[0090] It can be understood that the semantic paragraphs of a document can be divided into text reading orders such as left-right reading order and top-bottom reading order. According to different text reading orders, the layout type corresponding to each semantic paragraph in the document can be determined, where the layout type includes paragraph layout and table layout, etc.

[0091] In some embodiments, see Figure 2 , Figure 2 FIG. 1 is a flow chart of a document information extraction method provided by an embodiment of the present application. Figure 2 As shown, the determining of the layout type corresponding to each semantic paragraph in the document based on the text reading order includes:

[0092] Step S210: obtaining text feature information corresponding to the semantic paragraph.

[0093] Step S220: determining the text reading order of the semantic paragraphs according to the text feature information.

[0094] Step S230: When the text reading order of the semantic paragraph is the left-right reading order, determine that the layout type of the semantic paragraph is the paragraph layout.

[0095] Step S240: When the text reading order of the semantic paragraph is a top-down reading order, determining that the layout type of the semantic paragraph is a table layout.

[0096] It should be noted that the above text reading order is determined according to the reading order of field attributes and field values ​​in the document, including determining the text reading order of semantic paragraphs according to the reading order of field attributes first and then field values, as explained below:

[0097] For example, refer to Figure 7 , in the "Policyholder" column of the document in the figure, that is, in the "Policyholder" semantic paragraph, if you want to read the name of the policyholder, you can find the field attribute of "Name" in the "Policyholder" semantic paragraph, and then find the field value corresponding to the "Name" field attribute from left to right. Therefore, according to the reading order of field attributes first and then field values, it can be determined that the text reading order of the "Policyholder" semantic paragraph is left-right reading order;

[0098] For example, refer to Figure 7 In the "Insurance Items" column of the document in the figure, that is, in the "Insurance Items" semantic paragraph, if you want to read the specific insurance liability, you can find the field attribute of "Insurance Liability" in the "Insurance Items" semantic paragraph, and then find the field value corresponding to the "Insurance Items" field attribute from top to bottom. Therefore, according to the reading order of field attributes first and then field values, it can be determined that the text reading order of the "Insurance Items" semantic paragraph is the top-to-bottom reading order.

[0099] It is understandable that based on different text reading orders, the field attributes and field values ​​in the semantic paragraph have different arrangement orders, and the semantic paragraph has different text feature information. Therefore, the text reading order of the semantic paragraph can be determined according to the text feature information, thereby determining the layout type of the semantic paragraph.

[0100] For example, Figure 7 As shown in the figure, the document includes a semantic paragraph of "insured person", wherein the semantic paragraph includes the following information: {name of the insured person, date of birth of the insured person, type of ID of the insured person, number of ID of the insured person, relationship between the insured person and the policyholder, and age of the insured person}. By acquiring the text feature information corresponding to the semantic paragraph, the text reading order of the semantic paragraph is determined.

[0101] It should be noted that the above embodiment determines the text reading order of the semantic paragraph based on the text reading order of field attributes first and then field values. It can be understood that the text feature information corresponding to the semantic paragraph will not change due to different methods of determining the text reading order. In other words, the text reading order of the semantic paragraph can also be determined according to other methods such as the text reading order of field values ​​first and then field attributes.

[0102] In some embodiments, see Figure 3 , Figure 3 FIG. 1 is a flow chart of a document information extraction method provided by an embodiment of the present application. Figure 3 As shown, determining the text reading order of the semantic paragraph according to the text feature information includes:

[0103] Step S310, constructing a binary classification task for determining the text reading order of the semantic paragraph;

[0104] Step S320: input the text feature information into a classification model to perform the binary classification task through the classification model to obtain the text reading order of the semantic paragraph.

[0105] It can be understood that, on the basis of determining the text reading order of a semantic paragraph according to the reading order of field attributes first and then field values, when the semantic paragraph is in a top-down reading order or a left-right reading order, its text feature information has a large difference. By constructing a binary classification task for determining the text reading order of a semantic paragraph, and then inputting the text feature information into the classification model to perform the binary classification task through the classification model, the processing result of the binary classification task is obtained, that is, determining whether the text reading order of the semantic paragraph is a top-down reading order or a left-right reading order.

[0106] Exemplarily, when the text reading order of a semantic paragraph is a left-right reading order, the text of the semantic paragraph is first arranged from left to right and then from top to bottom at the field attribute (key) and field value (vault) level as {key-vault-key-vault-key-vault...}; and when the text reading order of a semantic paragraph is a top-bottom reading order, the text of the semantic paragraph is first arranged from left to right and then from top to bottom at the field attribute (key) and field value (vault) level as {key-key-key-...-vault-vault-vault-...}, that is, when the semantic paragraph is in a different text reading order, its text feature information has a large difference at the field attribute and field value (vault) level. Based on the arrangement feature of the key-value pair (key-vault) from left to right, a binary classification task for determining the text reading order of the semantic paragraph can be constructed. Based on the text feature information of the semantic paragraph, a classification model is used to perform a binary classification task to obtain the text reading order of the semantic paragraph.

[0107] It should be noted that the above embodiment first obtains the text feature information of the semantic paragraph from left to right and then from top to bottom. Under different text reading orders, the text feature information has significant differences. It is understandable that at the level of field attributes and field values, the text feature information will not change due to different acquisition methods, that is, the method of acquiring text feature information is not limited to the above embodiment. For example, the text feature information of the semantic paragraph can be obtained by first from top to bottom, then from left to right, and other methods, and a binary classification task for determining the text reading order of the semantic paragraph can still be constructed.

[0108] In some embodiments, the text feature information is a word vector matrix of the semantic paragraph, and the classification model is a TextCNN model;

[0109] The step of inputting the text feature information into a classification model to perform the binary classification task through the classification model to obtain the text reading order of the semantic paragraph includes:

[0110] The word vector matrix is ​​input into the TextCNN model to perform the binary classification task through the TextCNN model to obtain the text reading order of the semantic paragraph.

[0111] It can be understood that the text in the semantic paragraph is segmented and each word is mapped into a word vector, and then all the word vectors are concatenated to obtain the word vector matrix of the semantic paragraph, and the word vector matrix is ​​used as the input of the TextCNN model, and the binary classification task is performed through the TextCNN model, wherein the word vector matrix is ​​processed by the convolution layer, pooling layer and output layer of the TextCNN model to obtain the processing result of the binary classification task, that is, the text reading order of the semantic paragraph.

[0112] In some embodiments, the text feature information is a character feature matrix of the semantic paragraph, and the classification model is an Xgboost model;

[0113] The step of inputting the text feature information into a classification model to perform the binary classification task through the classification model to obtain the text reading order of the semantic paragraph includes:

[0114] Performing maximum pooling on each row of character vectors in the character feature matrix to obtain a row feature vector of the semantic paragraph;

[0115] The row feature vectors of the semantic paragraph are concatenated, and the concatenated row feature vectors are input into an Xgboost model, so as to perform the binary classification task through the Xgboost model and obtain the text reading order of the semantic paragraph.

[0116] For example, using Figure 5 The pre-trained Layout model in the LayoutLm model uses the five-layer Embedding input layer and the LayouLm Embedding layer in the LayoutLm model as the character feature extractor of the semantic paragraph, and outputs a 512-dimensional character vector, that is, a character feature matrix. In addition to the first dimension, each dimension in the character feature matrix represents the character vector in the semantic paragraph. Take out each row of the character vector in the character feature matrix Max-pooling is performed on each row of character vectors to obtain the row feature vector of the semantic paragraph N lines are intercepted, for example, a fixed 20 lines are intercepted, and then the line feature vectors are semantically concatenated, and the concatenated line feature vectors are input into the Xgboost model to perform a binary classification task through the Xgboost model to obtain the text reading order of the semantic paragraph.

[0117] It should be noted that the classification model described in the above embodiment refers to a model that can be used to perform binary classification tasks and determine the classification results at the text feature information level. The classification model can be a text classification model such as TextCNN, or a machine learning model such as Xgboost.

[0118] Step S150: extracting structured data of each semantic paragraph according to the layout type corresponding to the semantic paragraph.

[0119] It is understandable that, corresponding to different layout types of semantic paragraphs, different information extraction methods can be adopted to obtain information in the semantic paragraphs to form structured data.

[0120] By distinguishing different layout types in documents and applying different extraction methods to different layout types, the information extraction process can be simplified, the migration of application fields can be facilitated, and the adaptability of document information extraction methods to different document types and the accuracy of information extraction can be improved.

[0121] In some embodiments, see Figure 4 , Figure 4 FIG. 1 is a flow chart of a document information extraction method provided by an embodiment of the present application. Figure 4 As shown, extracting the structured data of each semantic paragraph according to the layout type includes:

[0122] When the layout type of the semantic paragraph is paragraph layout, the following processing is performed:

[0123] Step S410, obtaining field attributes and field values ​​in the semantic paragraph;

[0124] Step S420, standardizing the field attributes in the semantic paragraph by regular matching;

[0125] Step S430, mapping the field value to the standardized field attribute to obtain the structured data of the semantic paragraph.

[0126] For example, Figure 7 In the document shown, the layout structure of the semantic paragraph is a paragraph layout including {insured, policyholder, beneficiary, etc.}. For example, the field attributes {name, date of birth, certificate type, certificate number, relationship with policyholder, age} and field values ​​in the semantic paragraph of "policyholder" are obtained. Assuming that the field values ​​are {Zhang, January 1, 2000, ID card, 44000000000, father and son, 22 years old}, the field attributes in the semantic paragraph are standardized by regular matching. For the field attribute "name", it can be standardized to "name of policyholder" according to the specific semantic paragraph, and for the field attribute "age", it can be standardized to "age of policyholder", etc.

[0127] It is understandable that if the same field attributes exist in the document, different contents corresponding to the same field attributes may appear after the structure is generated. Figure 7As shown in the figure, there are the same field attributes in the document, such as the field attribute "ID number" in the semantic paragraph of "policyholder" and the field attribute "ID number" in the semantic paragraph of "insured person". After the structure is generated, the specific ID numbers of the policyholder and the insured person will correspond to the same field attribute. The corresponding field attributes are modified by standardization and other methods, and the field attribute "ID number" in the semantic paragraph of "policyholder" is standardized as "policyholder ID number", and the field attribute "ID number" in the semantic paragraph of "insured person" is standardized as "insured ID number" to distinguish the same field attributes in different semantic paragraphs, and reduce the problem of unclear corresponding field attributes of specific field values ​​due to the same field attributes in the generated structured data.

[0128] It should be noted that when the layout structure of the semantic paragraph is a table layout, other table information extraction methods can be used to extract information from the semantic paragraph to form structured data. The specific method is not limited in this embodiment of the present application.

[0129] It should also be noted that if Figure 7 As shown, the semantic paragraph also includes semantic paragraphs such as "company" and "title", and the semantic paragraph only has corresponding field values: {XX Insurance Company, XX Insurance Co., Ltd., Business Trip Personal Accident Insurance Application Form}. For the semantic paragraph, it can be directly mapped to the field value in the structured data, or the corresponding field attribute can be generated according to the semantic paragraph label of the semantic paragraph, and then the field value is mapped to the corresponding field attribute to obtain the structured data of the semantic paragraph. Exemplarily, for the field value: "XX Insurance Co., Ltd.", according to its semantic paragraph label: "Company", the field attribute is generated: "Company", and the field value is mapped to the corresponding field attribute to obtain the structured data: "Company: XX Insurance Co., Ltd.".

[0130] The document information extraction method provided by the embodiment of the present application is described below through a specific embodiment applied to the insurance policy field:

[0131] See also Figure 5 , Figure 6 and Figure 7 The document information extraction method obtains an insurance policy document training data set, adopts an MVLM method, and uses the document training data set to perform augmented pre-training on a two-dimensional named entity recognition model based on a LayoutLM model.

[0132] The insurance policy document is processed by OCR recognition to obtain multiple text blocks. By defining semantic paragraph labels {company, title, policy information, policyholder information, insured person information, beneficiary information, insurance period, insurance item}, the multiple text blocks are input into a pre-trained named entity recognition model to perform sequence labeling tasks through the named entity recognition model to obtain the corresponding semantic paragraph labels in each text block. According to the corresponding semantic paragraph labels in each text block, the semantic paragraphs in the document are further determined.

[0133] The text feature information of each semantic paragraph in the document is obtained. Based on the field attributes in the document and the reading order of the field values, a binary classification task is constructed to determine the text reading order of the semantic paragraph. The text feature information corresponding to the semantic paragraph is input into the classification model, such as inputting the word vector matrix corresponding to the semantic paragraph into the TextCNN model, or inputting the character feature matrix corresponding to the semantic paragraph into the Xgboost model, so as to perform the binary classification task through the above classification model to obtain the text reading order of the semantic paragraph, thereby determining the layout type of the semantic paragraph, wherein, when the text reading order of the semantic paragraph is a left-right reading order, the layout type of the semantic paragraph is determined to be a paragraph layout, and when the text reading order of the semantic paragraph is a top-down reading order, the layout type of the semantic paragraph is determined to be a table layout.

[0134] When the layout type of a semantic paragraph is paragraph layout, the structured data of the semantic paragraph is obtained by standardizing the field attributes in the semantic paragraph and mapping the field values ​​to the standardized field attributes, such as:

[0135]

[0136] When the layout type of the semantic paragraph is a table layout, the structured data of the semantic paragraph is obtained through other table information extraction methods, such as:

[0137]

[0138]

[0139] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present application, it should not be understood as requiring the operations to be performed in the specific order or serial order shown, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0140] The present application proposes a document information extraction method, which obtains multiple text blocks by performing recognition processing on the document, and inputs the multiple text blocks into a pre-trained named entity recognition model to obtain semantic paragraph labels corresponding to each text block through the named entity recognition model, and determines the semantic paragraphs of the document according to the semantic paragraph labels corresponding to each text block, thereby determining the layout type corresponding to each semantic paragraph in the document based on the text reading order, wherein the text reading order includes left-right reading order and top-bottom reading order, and the layout type includes paragraph layout and table layout, and extracts the structured data of each semantic paragraph according to the layout type corresponding to the semantic paragraph. The document information extraction method of the embodiment of the present application divides the document according to the layout type on the basis of determining the semantic paragraph corresponding to the document, and performs information extraction for semantic paragraphs of different layout types respectively, which can simplify the document information extraction process, facilitate the migration of application fields, and improve the adaptability to different document types.

[0141] See also Figure 8 The embodiment of the present application further provides a document information extraction device 100, and the document information extraction device 100 includes:

[0142] The text block acquisition module 110 is used to perform recognition processing on the document to obtain multiple text blocks;

[0143] A label classification module 120, configured to input the plurality of text blocks into a pre-trained named entity recognition model, so as to obtain a semantic paragraph label corresponding to each of the text blocks through the named entity recognition model;

[0144] A semantic paragraph extraction module 130, configured to determine the semantic paragraphs of the document according to the semantic paragraph tags corresponding to each of the text blocks;

[0145] A layout type classification module 140, configured to determine the layout type corresponding to each semantic paragraph in the document based on a text reading order, wherein the text reading order includes a left-right reading order and a top-bottom reading order, and the layout types include a paragraph layout and a table layout;

[0146] The document information extraction module 150 is used to extract the structured data of each semantic paragraph according to the layout type corresponding to the semantic paragraph.

[0147] The present application proposes a document information extraction device, which recognizes and processes a document through a text block acquisition module to obtain multiple text blocks, and a label classification module inputs the multiple text blocks into a pre-trained named entity recognition model to obtain semantic paragraph labels corresponding to each text block through the named entity recognition model. The semantic paragraph extraction module determines the semantic paragraphs of the document according to the semantic paragraph labels corresponding to each text block, thereby determining the layout type corresponding to each semantic paragraph in the document based on the text reading order, wherein the text reading order includes a left-right reading order and a top-bottom reading order, and the layout type includes a paragraph layout and a table layout. The document information extraction module extracts the structured data of each semantic paragraph according to the layout type corresponding to the semantic paragraph. The document information extraction device of the embodiment of the present application divides the document according to the layout type on the basis of determining the semantic paragraph corresponding to the document, and performs information extraction for semantic paragraphs of different layout types respectively, which can simplify the document information extraction process, facilitate the migration of application fields, and improve the adaptability to different document types.

[0148] It should be noted that the information interaction, execution process and other contents between the modules of the above-mentioned device are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0149] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device provided in an embodiment of the present application is shown, and the electronic device includes:

[0150] The processor 210 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant computer programs to implement the technical solutions provided in the embodiments of the present application;

[0151] The memory 220 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 220 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 220, and the processor 210 calls and executes the document information extraction method of the embodiment of this application;

[0152] Input / output interface 230, used to implement information input and output;

[0153] The communication interface 240 is used to realize the communication interaction between the device and other devices, and the communication can be realized by wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and the bus 250 transmits information between each component of the device (such as the processor 210, the memory 220, the input / output interface 230 and the communication interface 240);

[0154] The processor 210 , the memory 220 , the input / output interface 230 , and the communication interface 240 are connected to each other in communication within the device via the bus 250 .

[0155] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more computer programs, and the one or more computer programs can be executed by one or more processors to implement the above-mentioned document information extraction method.

[0156] The memory, as a computer-readable storage medium, can be used to store software programs and computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0157] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0158] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0159] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0160] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0161] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0162] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0163] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or can be integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0164] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0166] If the assembled units are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., and other media that can store programs.

[0167] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A document information extraction method, characterized in that: include: Recognize and process the document to obtain multiple text blocks; The plurality of text blocks are input into a pre-trained named entity recognition model, so as to perform multi-label classification on each of the text blocks through the named entity recognition model to obtain semantic paragraph labels corresponding to each of the text blocks; wherein, when different semantic paragraph labels appear in the text block, the semantic paragraph label with the highest frequency of appearance in the text block is selected through a label voting method, and the semantic paragraph label with the highest frequency of appearance is used as the semantic paragraph label corresponding to the text block; According to the semantic paragraph tags corresponding to the text blocks, the text blocks belonging to the same semantic paragraph tag are merged to obtain the semantic paragraphs of the document; Determining a layout type corresponding to each semantic paragraph in the document based on a text reading order, wherein the text reading order includes a left-right reading order and a top-bottom reading order, and the layout types include a paragraph layout and a table layout; According to the layout type corresponding to the semantic paragraph, the structured data of each semantic paragraph is extracted.

2. The document information extraction method according to claim 1, characterized in that: The determining, based on the text reading order, the layout type corresponding to each semantic paragraph in the document includes: Obtaining text feature information corresponding to the semantic paragraph; Determining a text reading order of the semantic paragraphs according to the text feature information; When the text reading order of the semantic paragraph is a left-right reading order, determining that the layout type of the semantic paragraph is a paragraph layout; When the text reading order of the semantic paragraph is a top-bottom reading order, it is determined that the layout type of the semantic paragraph is a table layout.

3. The document information extraction method according to claim 2, characterized in that: Determining the text reading order of the semantic paragraphs according to the text feature information includes: Constructing a binary classification task for determining the text reading order of the semantic paragraph; The text feature information is input into a classification model to perform the binary classification task through the classification model to obtain the text reading order of the semantic paragraph.

4. The document information extraction method according to claim 3, characterized in that: The text feature information is the word vector matrix of the semantic paragraph, and the classification model is a TextCNN model; The step of inputting the text feature information into a classification model to perform the binary classification task through the classification model to obtain the text reading order of the semantic paragraph includes: The word vector matrix is ​​input into the TextCNN model to perform the binary classification task through the TextCNN model to obtain the text reading order of the semantic paragraph.

5. The document information extraction method according to claim 3, characterized in that: The text feature information is a character feature matrix of the semantic paragraph, and the classification model is an Xgboost model; The step of inputting the text feature information into a classification model to perform the binary classification task through the classification model to obtain the text reading order of the semantic paragraph includes: Performing maximum pooling on each row of character vectors in the character feature matrix to obtain a row feature vector of the semantic paragraph; The row feature vectors of the semantic paragraph are concatenated, and the concatenated row feature vectors are input into an Xgboost model, so as to perform the binary classification task through the Xgboost model and obtain the text reading order of the semantic paragraph.

6. The document information extraction method according to claim 1, characterized in that: The step of extracting structured data of each semantic paragraph according to the layout type includes: When the layout type of the semantic paragraph is paragraph layout, the following processing is performed: Obtaining field attributes and field values ​​in the semantic paragraph; Standardizing the field attributes in the semantic paragraph by regular matching; The field value is mapped to the standardized field attribute to obtain the structured data of the semantic paragraph.

7. The document information extraction method according to claim 1, characterized in that: The named entity recognition model is a two-dimensional named entity recognition model based on the LayoutLM model; Before inputting the plurality of text blocks into a pre-trained named entity recognition model to obtain semantic paragraph labels corresponding to each of the text blocks through the named entity recognition model, the method further includes: Get the preset document training data set; The two-dimensional named entity recognition model is augmented and pre-trained using the document training data set, wherein the augmented pre-training adopts a self-supervised pre-training method based on a masked visual language model.

8. A document information extraction device, characterized in that: The device comprises: A text block acquisition module is used to identify and process the document to obtain multiple text blocks; A label classification module, used for inputting the plurality of text blocks into a pre-trained named entity recognition model, so as to perform multi-label classification on each of the text blocks through the named entity recognition model, and obtain semantic paragraph labels corresponding to each of the text blocks; wherein, when different semantic paragraph labels appear in the text block, the semantic paragraph label with the highest frequency of appearance in the text block is selected through a label voting method, and the semantic paragraph label with the highest frequency of appearance is used as the semantic paragraph label corresponding to the text block; A semantic paragraph extraction module, used for merging text blocks belonging to the same semantic paragraph tag according to the semantic paragraph tags corresponding to the text blocks, to obtain the semantic paragraphs of the document; A layout type classification module, used to determine the layout type corresponding to each semantic paragraph in the document based on a text reading order, wherein the text reading order includes a left-right reading order and a top-bottom reading order, and the layout types include a paragraph layout and a table layout; The document information extraction module is used to extract the structured data of each semantic paragraph according to the layout type corresponding to the semantic paragraph.

9. An electronic device, characterized in that: include: Memory, used to store programs; A processor, used to execute the program stored in the memory. When the processor executes the program stored in the memory, the processor is used to execute: the document information extraction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Computer executable instructions are stored, and the computer executable instructions are used to execute: the document information extraction method according to any one of claims 1 to 7.