Information extraction method, server, storage medium and program product

Abstract

The present application provides an information extraction method, server, storage medium and program product. The method converts the information extraction problem into a link relationship classification problem. For an input text sequence containing a given entity category and document text information, the link relationship between any two text tags is predicted by combining a multimodal model with the position coordinates of text tags in the document image and text information, making full use of the implicit relationship between entities and entity categories, so as to better understand the complex document structure and accurately predict the link relationship between text tags; according to the link relationship between any two text tags, the entity information contained in the document is parsed, which can improve the accuracy of information extraction; and the prediction results of various types of link relationships can be obtained through one reasoning, which improves the efficiency of information extraction; in addition, the entity category set to be extracted can be flexibly given according to the actual application scenario, which can adapt to emerging entity categories and changing document layouts.

Description

Technical Field

[0001] The present application relates to computer technology, and in particular to an information extraction method, a server, a storage medium and a program product. Background Art

[0002] In today's information age, enterprises and organizations often face the need to extract valuable data from a large number of unstructured documents. These documents may contain key data information, such as contract details, financial statements or customer data.

[0003] Traditional information extraction methods, such as rule-based or sequence-labeling-based information extraction methods, are limited to predefined patterns in specific fields. The key information extraction datasets used usually have relatively fixed layouts and formats, and the diversity of entity categories is insufficient, making it difficult to adapt to changing document layouts and emerging entity categories. Another method is to rely on machine learning models to implement information extraction functions, but existing information extraction models have low information utilization of original documents during the learning process and slow reasoning speed, resulting in low accuracy and efficiency of information extraction. Summary of the invention

[0004] The present application provides an information extraction method, a server, a storage medium and a program product to solve the problems of low accuracy and efficiency of information extraction and difficulty in adapting to changing document layouts and emerging entity categories.

[0005] In a first aspect, the present application provides an information extraction method, comprising: obtaining a document image, text information and text coordinates of a document to be processed, and at least one given entity category; generating an input text sequence containing the at least one entity category and the text information, and determining the position coordinates of each text tag in the text information according to the text coordinates; predicting, through a multimodal model, a link relationship between any two text tags in the input text sequence according to the document image of the document, the input text sequence and the position coordinates of each text tag in the text information; and parsing to obtain the entity information contained in the document according to the link relationship between any two text tags in the input text sequence.

[0006] In a second aspect, the present application provides an information extraction method, comprising: in response to an information extraction request, obtaining a document image, text information and text coordinates of a document to be processed, and at least one given entity category; generating an input text sequence containing the at least one entity category and the text information, and determining the position coordinates of each text tag in the text information according to the text coordinates; predicting, through a multimodal model, a link relationship between any two text tags in the input text sequence according to the document image of the document, the input text sequence and the position coordinates of each text tag in the text information; parsing the entity information contained in the document according to the link relationship between any two text tags in the input text sequence; and outputting the entity information contained in the document.

[0007] In a third aspect, the present application provides an information extraction method, comprising: in response to an information extraction request, obtaining a document image, text information and text coordinates of a given legal document, and at least one given entity category; generating an input text sequence containing the at least one entity category and the text information, and determining the position coordinates of each text tag in the text information according to the text coordinates; predicting, through a multimodal model, a link relationship between any two text tags in the input text sequence according to the document image of the legal document, the input text sequence and the position coordinates of each text tag in the text information; parsing the entity information contained in the legal document according to the link relationship between any two text tags in the input text sequence; and outputting the entity information contained in the legal document.

[0008] In a fourth aspect, the present application provides a server, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the server executes a method provided in any of the preceding aspects.

[0009] In a fifth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method provided in any of the foregoing aspects is implemented.

[0010] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method provided in any of the aforementioned aspects.

[0011] The information extraction method, server, storage medium and program product provided by the present application convert the information extraction problem into a link relationship classification problem (i.e., a pointer classification problem). For an input text sequence containing at least one given entity category and document text information, a multimodal model is used to combine the position coordinates of each text tag in the document image and the document text information to predict whether there are preset types of link relationships between any two text tags in the input text sequence, that is, to classify and predict the link relationship between any two text tags, and obtain the link relationship between any two text tags in the input text sequence. The multimodal model makes full use of the implicit relationship between entities and entity categories, can better understand the complex document structure, and accurately predict the link relationship between any two text tags in the input text sequence. Furthermore, according to the link relationship between any two text tags in the input text sequence, the entity information contained in the document can be obtained by parsing according to the rules, which can improve the accuracy of information extraction. In addition, the multimodal model can obtain the prediction results of multiple different types of link relationships after one reasoning, realize the parallel prediction of multiple types of link relationships, and improve the efficiency of information extraction. In addition, this solution can flexibly give a set of entity categories to be extracted according to the needs of actual application scenarios, and has no requirements on the layout of the document, and can adapt well to emerging entity categories and changing document layouts. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0013] Figure 1 A schematic diagram of an exemplary system architecture applicable to the present application;

[0014] Figure 2 A flowchart of an information extraction method provided for an exemplary embodiment of the present application;

[0015] Figure 3 A flowchart of link relationship classification provided for an exemplary embodiment of the present application;

[0016] Figure 4 A framework diagram of an information extraction method provided for an exemplary embodiment of the present application;

[0017] Figure 5 A flowchart of parsing entity information contained in a document provided by an exemplary embodiment of the present application;

[0018] Figure 6 A partial example of a document provided for an exemplary embodiment of the present application;

[0019] Figure 7An example diagram of link relationships between text tags provided for an exemplary embodiment of the present application;

[0020] Figure 8 A schematic diagram of a two-dimensional matrix representation of a link relationship provided by an exemplary embodiment of the present application;

[0021] Fig. 9 A flowchart of parsing key-value pair information contained in a document provided by an exemplary embodiment of the present application;

[0022] Fig.10 A flowchart of a method for multimodal model training provided for an exemplary embodiment of the present application;

[0023] Fig.11 A framework diagram of multimodal model training provided for an exemplary embodiment of the present application;

[0024] Fig.12 A flow chart of a method for information extraction provided for an exemplary embodiment of the present application;

[0025] Fig.13 A flowchart of a method for extracting information from a legal document provided for an exemplary embodiment of the present application;

[0026] Fig.14 A schematic diagram of the structure of a server provided in an embodiment of the present application.

[0027] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0028] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0029] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0030] First, the terms involved in this application are explained:

[0031] Information Extraction (IE): Also known as key information extraction, it is the process of extracting entity information with a specific structure from visually rich semi-structured documents. It is a key task in the field of document processing and is essential for automated and intelligent document understanding.

[0032] Optical Character Recognition (OCR) is a technology that recognizes and extracts text from image files. In the field of document processing and information extraction, OCR is used to convert text in scanned or photographed document images into editable and searchable text format.

[0033] Token Path Prediction: is a technique used in document understanding tasks that involves predicting the paths between words or tokens in a document to identify and extract information. Token path prediction can be used to identify entities in a document and their relationships with each other.

[0034] Visual question answering task: Given an input image and a question, determine the answer to the question from the visual information of the input image.

[0035] Image description task: Generate description text for an input image.

[0036] Visual entailment task: predict the semantic relevance of an input image and text, i.e., entailment, neutral, or contradiction.

[0037] Referential expression and comprehension task: locate the image area in the input image corresponding to the input text based on the input text.

[0038] Image generation task: Generate an image based on the input description text.

[0039] Text-based sentiment classification task: predict the sentiment classification information of the input text.

[0040] Text summarization task: Generate summary information of the input text.

[0041] Multimodal tasks: refers to downstream tasks whose input and output data involve multiple modal data such as images and text, such as visual question answering tasks, image description tasks, visual implication tasks, referential expression and understanding tasks, image generation tasks, etc.

[0042] Multimodal pre-trained model: refers to a pre-trained model whose input and output data involve multiple modal data such as images and text. After fine-tuning and training, it can be applied to multimodal task processing.

[0043] Pre-trained language model: A pre-trained model obtained by pre-training a large-scale language model (LLM).

[0044] A large model refers to a deep learning model with large-scale model parameters, usually containing hundreds of millions, tens of billions, or even hundreds of billions of model parameters. A large model can also be called a foundation model / foundation model (FM). It is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks and has good generalization capabilities, such as large-scale language models (LLMs) and multi-modal pre-training models.

[0045] When the big model is used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. The big model can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the big model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0046] The current information extraction methods mainly include:

[0047] 1) Rule-based information extraction uses predefined rules to guide information extraction, which is suitable for document processing with a high degree of standardization. However, it requires expert knowledge to formulate and maintain rules, and cannot meet the needs of document information extraction with diverse layouts and formats.

[0048] 2) Based on the sequence labeling method, a deep learning model is used to perform sequence labeling on each tag in the document to identify and classify entities. However, it may not be able to capture entity relationships across lines or paragraphs, and is sensitive to layout and format changes. When applied to documents with complex layouts or diverse format changes, the information extraction quality is poor.

[0049] 3) Dependency parsing-based methods treat documents as a graph and use dependency parsing techniques to identify entities and their relationships. However, for unseen entity types or relationships, the generalization ability is poor, and a large amount of computing resources are required, resulting in low information extraction efficiency.

[0050] 4) Methods based on graph neural networks: Build a graph structure between entities and use graph neural networks to learn complex relationships between entities. However, the construction of the graph structure between entities requires domain knowledge, and the construction of the graph neural network and the complexity of the graph structure may lead to increased computational costs and low efficiency in information extraction.

[0051] In summary, existing information extraction methods have the following technical defects: it is difficult to adapt to changing document layouts and emerging entity categories, and the accuracy and efficiency of information extraction are low.

[0052] The present application provides an information extraction method, comprising: obtaining a document image, text information and text coordinates of a document to be processed, and at least one given entity category; generating an input text sequence containing at least one entity category and text information, and determining the position coordinates of each text tag in the text information according to the text coordinates; predicting the link relationship between any two text tags in the input text sequence according to the document image, the input text sequence and the position coordinates of each text tag in the text information through a multimodal model; and parsing and obtaining the entity information contained in the document according to the link relationship between any two text tags in the input text sequence.

[0053] Among them, the possible link relationships between text tags include multiple types of link relationships between entities and key text tags in entity categories, which are pre-designed multiple types of link relationships. The link relationship between text tags can be understood as a directed edge (or pointer) from one text tag to another text tag, indicating the association relationship between text tags. Based on the preset link relationship, a piece of text can be represented as a directed word graph.

[0054] Possible link relationships between text tags include, but are not limited to: the link relationship between the head tag and the tail tag of the same entity, the link relationship between the head tag of an entity category and the head tag of an entity, and the link relationship between the tail tag of the previous sub-entity of a discontinuous entity and the head tag of the next sub-entity. Possible link relationships between text tags can be pre-defined and configured according to actual application requirements.

[0055] The method of the present application converts the information extraction problem into a link relationship classification problem (i.e., a pointer classification problem). For an input text sequence containing at least one given entity category and document text information, a multimodal model is used to combine the position coordinates of each text tag in the document image and the document text information to predict whether there are preset types of link relationships between any two text tags in the input text sequence, that is, the link relationship between any two text tags is classified and predicted to obtain the link relationship between any two text tags in the input text sequence. The multimodal model makes full use of the implicit relationship between entities and entity categories, can better understand the complex document structure, and accurately predict the link relationship between any two text tags in the input text sequence. Furthermore, according to the link relationship between any two text tags in the input text sequence, the entity information contained in the document can be obtained by parsing according to the rules, which can improve the accuracy of information extraction. In addition, the multimodal model can obtain the prediction results of multiple different types of link relationships after one reasoning, realize the parallel prediction of multiple types of link relationships, and improve the efficiency of information extraction. In addition, this solution can flexibly give a set of entity categories to be extracted according to the needs of actual application scenarios, and has no requirements on the layout of the document, and can adapt well to emerging entity categories and changing document layouts.

[0056] Figure 1 FIG. 1 is a schematic diagram of an exemplary system architecture applicable to the present application. Figure 1 As shown, the system architecture includes a server and a terminal device. There is a communication link between the server and the terminal device, which can realize the communication connection between the server and the terminal device.

[0057] The end-side device may be an electronic device running downstream applications, specifically a hardware device with network communication functions, computing functions, and information display functions, including but not limited to smart phones, tablet computers, desktop computers, local servers, cloud servers, etc. The downstream applications running on the end-side device may be natural language processing, data mining, semantic analysis, etc. When the end-side device needs to extract entity information from a document during the running of the downstream application, it submits the document to be extracted to the server.

[0058] The server is a device with computing power deployed in the cloud or locally, such as a cloud cluster. The server provides information extraction services and is responsible for obtaining the document image, text information and text coordinates of the document to be processed, as well as at least one given entity category based on the document of the information to be extracted; generating an input text sequence containing at least one entity category and text information, and determining the position coordinates of each text tag in the text information according to the text coordinates; predicting the link relationship between any two text tags in the input text sequence according to the document image, input text sequence and the position coordinates of each text tag in the text information through a multimodal model; parsing and obtaining the entity information contained in the document according to the link relationship between any two text tags in the input text sequence.

[0059] Furthermore, the server may return entity information contained in the document to the end-side device, and the end-side device executes subsequent processing logic based on the entity information contained in the document.

[0060] The method of the present application can be applied to extract key information from various documents, and can realize entity recognition, table information extraction, etc.

[0061] In actual application scenarios, as part of a document, a table is simple and easy to use, and is an effective way of expressing information commonly used in daily work. A table contains multiple cells, some of which contain the names of various information items, and these cells are called key cells. For example, "project name", "tenderer name", "contact name" and so on in the table header. The content of some cells is the specific information value of each information item, and these cells are called value cells. The value cell usually corresponds to at least one key cell, which is the specific information value of the information item in the corresponding key cell. For example, the specific content under the table header in the table is "Project A", "B Limited Liability Company", "Zhang XX", etc. In actual applications, the key cells in the table usually correspond to some fields in the database where the table content exists, and the content of the value cell corresponding to the key cell is the data value of the field corresponding to the key cell. The table contains rich key (key)-value (value) pair information (such as key = "project name", value = "Project A" to form a key-value pair information, that is, key-value pair information. The extraction of key-value pair information in the table essentially broadens the dimension of obtaining unstructured information, and has a huge promoting effect on natural language processing, data mining, semantic analysis, etc.

[0062] The solution of the present application can be applied to extract key-value pair information from documents containing tables or data similar to tables.

[0063] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0064] Figure 2 This is a flowchart of an information extraction method provided by an exemplary embodiment of the present application. The execution subject of this embodiment is the server in the aforementioned system architecture. Figure 2 As shown, the specific steps of this method are as follows:

[0065] Step S201: Obtain a document image, text information and text coordinates of a document to be processed, and at least one given entity category.

[0066] Among them, the document to be processed can be a portable document format (PDF for short), an editable text document (such as doc, docx format), a fillable form format (such as docxf, oform format), a spreadsheet format (such as xlsx, xls format), an image format (such as scanned images, photos, etc.), etc. This embodiment does not specifically limit the specific format of the document to be processed.

[0067] The document image of the document to be processed may be a file in the image format of the document, or may be a scanned image or a photo of the document, etc., which is not specifically limited here. The text information of the document refers to the information of the text contained in the document, that is, the text content information of the document. The text information of the document can be obtained by reading the content of the document (readable format) or by performing text recognition (such as OCR) on the document image. The text coordinates refer to the coordinates of each character in the text information of the document in the document image, which can be obtained by performing text recognition (such as OCR) on the document image.

[0068] In this embodiment, the given at least one entity category refers to the category of entity information to be extracted. Specifically, it includes which categories of entities need to be extracted from the document. The given at least one entity category constitutes an entity category set, which can be set by the customer according to the actual information extraction scenario. In different information extraction scenarios, the customer can set different entity category sets, that is, different entity categories can be given.

[0069] In an example scenario, when a client needs to extract information, the client sets at least one entity category to be extracted according to the requirement for information extraction of the document. Different entity categories to be extracted may be set for different documents. The client sends an information extraction request to the server through a terminal device, and the information extraction request carries a document image of the document and at least one given entity category.

[0070] In response to the information extraction request, the server obtains the document image of the document to be processed and at least one given entity category. Further, the server performs text recognition on the document image to obtain the text information and text coordinates of the document.

[0071] The text recognition is performed on the document image to obtain the text information and text coordinates of the document. Various commonly used OCR methods can be used for implementation, which are not specifically limited here.

[0072] In another example scenario, the end-side device may submit the document image, text information, and text coordinates of the document to be processed, as well as at least one given entity category, to the server. The server receives the document image, text information, and text coordinates of the document to be processed submitted by the end-side device, as well as at least one given entity category. Different entity categories to be extracted may be set for different documents.

[0073] Step S202: Generate an input text sequence including at least one entity category and text information, and determine the position coordinates of each text mark in the text information according to the text coordinates.

[0074] In this step, the server generates a text sequence containing the given entity category and the document text information according to at least one given entity category and the document text information as an input text sequence of the multimodal model.

[0075] Specifically, when generating an input text sequence containing at least one entity category and text information, at least one entity category is converted into a text tag sequence, and the text information is converted into a text tag sequence; the text tag sequence of at least one entity category and the text tag sequence of the text information are concatenated to obtain the input text sequence.

[0076] For example, the server may convert each entity category into a text tag sequence, and convert the text information into a text tag sequence; the input text sequence is obtained by concatenating the text tag sequences of each entity category and then concatenating them with the text tag sequence of the text information. Here, there is no requirement for the concatenation order of the text tag sequences of each entity category.

[0077] Optionally, in order to distinguish between the text tag sequence of the entity category and the text tag sequence of the text information, a separation tag (such as "[T]" or other segmentation tags) is inserted between the text tag sequences of different entity categories when splicing the text tag sequences of each entity category. In order to distinguish between the text tag sequence of the entity category and the text tag sequence of the text information, a specific tag is used to segment the text tag sequence of the entity category and the text tag sequence of the text information. Among them, the segmentation tags used to segment the text tag sequences of different entity categories, and the specific tags for segmenting the text tag sequence of the entity category and the text tag sequence of the text information, are all used as independent text tags. The specific tags used can be configured and adjusted according to the needs of the actual application scenario, and are not specifically limited here.

[0078] For example, let Q1, Q2, ... represent the text tag sequence of each entity category, and let C = [c1, c2, ..., c s ] represents the text tag sequence of the document text information, and the input text sequence can be expressed as: X = <s> Q1[T]Q2[T]…< / s> C. The text tag sequences of different entity categories are segmented using the segmentation tag “[T]”. <s>< / s> "To separate the text tag sequence of entity category and the text tag sequence of text information." <s> " indicates the beginning of a text tag sequence of an entity category, "< / s> " indicates the end of the text token sequence for an entity category.

[0079] Exemplarily, the server can concatenate each entity category, and then concatenate it with the text information to obtain a concatenated text; the concatenated text is converted into a text tag sequence as an input text sequence. Here, there is no requirement for the concatenation order of the entity categories. In order to distinguish different entity categories and text information, a separator is inserted between different entity categories when concatenating the entity categories, and a separator is inserted between the entity category and the text information. Among them, the separator between different entity categories can be different from the separator between the entity category and the text information. The specific separator to be used can be configured and adjusted according to the needs of the actual application scenario, and is not specifically limited here.

[0080] Furthermore, for a text tag belonging to text information in the input text sequence, the position coordinates of the text tag are determined according to the text coordinates of the text included in the text tag. The position coordinates of the text tag refer to the coordinates of the bounding box of the text tag, which may specifically include the coordinates of two vertices (such as the upper left vertex and the lower right vertex, or the lower left vertex and the upper right vertex) on a diagonal line of the bounding box of the text tag in the document image, or include the coordinates of four vertices of the bounding box.

[0081] For example, the position coordinates of the i-th text mark in the text information can be expressed as: in and Respectively represent the coordinates of the upper left corner vertex and the lower right corner vertex of the bounding box of the i-th text tag.

[0082] Step S203: predicting the link relationship between any two text tags in the input text sequence through the multimodal model according to the document image of the document, the input text sequence and the position coordinates of each text tag in the text information.

[0083] In this step, the document image of the document, the input text sequence containing the given entity category and the document text information, and the position coordinates of each text tag in the text information are input into the multimodal model. The multimodal model performs inference prediction based on the document image of the document, the input text sequence, and the position coordinates of each text tag in the text information to obtain the link relationship between any two text tags in the input text sequence. This is a classification problem of what kind of link relationship exists between any two text tags in the input text series.

[0084] Among them, the possible link relationships between text tags include multiple types of link relationships between entities and key text tags in entity categories, which are pre-designed multiple types of link relationships. The link relationship between text tags can be understood as a directed edge (or pointer) from one text tag to another text tag, indicating the association relationship between text tags. Based on the preset link relationship, a piece of text can be represented as a directed word graph.

[0085] Possible link relationships between text tags include, but are not limited to: the link relationship from the head tag to the tail tag of the same entity, the link relationship from the head tag of an entity category to the head tag of an entity, and the link relationship from the tail tag of the previous sub-entity to the head tag of the next sub-entity of a discontinuous entity. Different link relationships between text tags can be defined as different values ​​to distinguish different types of link relationships. Possible link relationships between text tags, i.e., the category values ​​of link relationships, can be pre-defined and configured according to actual application requirements.

[0086] For example, in the information extraction scenario of named entity recognition, possible link relationships between text tags may include:

[0087] The link relationship from the head tag to the tail tag of the same entity, the link relationship from the head tag of the entity category to the head tag of the entity, and the link relationship from the tail tag of the previous sub-entity of a discontinuous entity to the head tag of the next sub-entity.

[0088] Optionally, in a named entity recognition scenario, possible link relationships between text tags may also include: a link relationship from a tail tag to a head tag of the same entity, and a link relationship from a tail tag of an entity to a tail tag of an entity category.

[0089] For example, in a key-value pair information extraction scenario, possible link relationships between text tags may include:

[0090] The link relationship from the head tag to the tail tag of the same value entity, the link relationship from the head tag of the entity category to the head tag of the value entity, the link relationship from the tail tag of the previous child entity to the head tag of the next child entity of a discontinuous value entity, the link relationship from the head tag of the key entity to the head tag of the corresponding value entity, the link relationship from the head tag to the tail tag of the same key entity, the link relationship from the head tag of the entity category to the head tag of the key entity, and the link relationship from the tail tag of the previous child entity to the head tag of the next child entity of a discontinuous key entity.

[0091] Optionally, in the key-value pair information extraction scenario, possible link relationships between text tags may also include:

[0092] The link relationship from the tail tag to the head tag of the same value entity, the link relationship from the tail tag of the value entity to the tail tag of the entity category, the link relationship from the tail tag of the value entity to the tail tag of the key entity, and the link relationship from the tail tag of the key entity to the tail tag of the entity category.

[0093] In this embodiment, the information extraction problem is converted into a link relationship classification problem (i.e., a pointer classification problem). For an input text sequence containing at least one given entity category and document text information, a multimodal model is used to combine the position coordinates of each text tag in the document image and the document text information to predict whether there are preset types of link relationships between any two text tags in the input text sequence, that is, to classify and predict the link relationship between any two text tags, and obtain the link relationship between any two text tags in the input text sequence. The multimodal model makes full use of the implicit relationship between entities and entity categories, and the implicit relationship between key entities and value entities, and can better understand complex document structures and accurately predict the link relationship between any two text tags in the input text sequence.

[0094] Step S204: parse and obtain entity information contained in the document according to the link relationship between any two text tags in the input text sequence.

[0095] The scheme of this embodiment aims to extract entity information with a specific structure from a visually rich semi-structured document to achieve key information extraction of the document. Among them, entity information is the key information that only needs to be extracted. In practical applications, the key information of the current field is defined as an entity (i.e., a named entity) and the entity category corresponding to the entity. The user can flexibly specify which entity categories need to be extracted by specifying the entity category to be extracted. For example, define entity categories such as names of people and place names.

[0096] After obtaining the link relationship between any two text tags in the input text sequence, the parsing is performed according to the parsing rules to obtain which subsequences in the input text sequence correspond to entities and the entity categories corresponding to the entities. The subsequences corresponding to the entities are converted into text to obtain the entities contained in the document.

[0097] After obtaining the entity information contained in the document (including the entity contained in the document and the entity category corresponding to the entity), the server can return the entity information contained in the document to the end-side device. The end-side device executes subsequent processing logic based on the entity information contained in the document.

[0098] The method of this embodiment converts the information extraction problem into a link relationship classification problem (i.e., a pointer classification problem). For an input text sequence containing at least one given entity category and document text information, a multimodal model is used to combine the position coordinates of each text tag in the document image and the document text information to predict whether there are preset types of link relationships between any two text tags in the input text sequence, that is, to classify and predict the link relationship between any two text tags, and obtain the link relationship between any two text tags in the input text sequence. The multimodal model makes full use of the implicit relationship between entities and entity categories, can better understand the complex document structure, and accurately predict the link relationship between any two text tags in the input text sequence. Further, according to the link relationship between any two text tags in the input text sequence, the entity information contained in the document can be obtained by parsing according to the rules, which can improve the accuracy of information extraction. In addition, the multimodal model can obtain the prediction results of multiple different types of link relationships after one reasoning, realize the parallel prediction of multiple types of link relationships, and can quickly extract the required key information from complex documents without processing each entity category separately, thereby improving the efficiency of information extraction. In addition, this scheme can flexibly give the set of entity categories to be extracted according to the needs of the actual application scenarios, and the multimodal model does not use the layout information of the document, so there is no requirement for the layout of the document, and it can adapt well to emerging entity categories and changing document layouts.

[0099] Figure 3 A flowchart of link relationship classification provided for an exemplary embodiment of the present application. In an optional embodiment, Figure 3 As shown, in the aforementioned step S203, the link relationship between any two text tags in the input text sequence is predicted by the multimodal model according to the document image of the document, the input text sequence and the position coordinates of each text tag in the text information, which can be specifically implemented by the following steps:

[0100] Step S301: input the document image, input text sequence and position coordinates of each text mark in the text information of the document into a multimodal model for encoding to obtain a feature representation of the document.

[0101] In this embodiment, the document image of the document, the input text sequence containing a given entity category and document text information, and the position coordinates of each text tag in the text information are input into the multimodal model. The document image of the document, the input text sequence, and the position coordinates of each text tag in the text information are encoded by the encoding module of the multimodal model to obtain the feature representation of the document. The feature representation of the document refers to the feature vector or feature tensor of the document.

[0102] In this step, the document image, the input text sequence, and the position coordinates of each text tag in the input text sequence are input into the multimodal model. The document image is first mapped into a visual embedding (represented by A) and the input text sequence is mapped into a text tag embedding (represented by T) through the multimodal model; then the visual embedding A, the text tag embedding T, and the position coordinates B of each text tag in the document text information are encoded to obtain the feature representation of the document.

[0103] Specifically, the multimodal model maps the document image to a feature map, and then maps the feature map to a visual embedding A through a fully connected layer. The multimodal model maps the input text sequence to a text tag embedding T, and generates a position embedding (Position Embedding) based on the position of each text tag in the input text sequence, and assigns a segment embedding (Segment Embedding) to each text tag. The position embedding contains the order in which the text tags appear in the input text sequence, and the segment embedding indicates whether the text tag belongs to an entity category or text information. The multimodal model adds the text tag embedding T to the position embedding and segment embedding to obtain the text embedding T′.

[0104] Furthermore, the multimodal model encodes the text embedding T′, visual embedding A, and location coordinates B to obtain a feature representation of the document (denoted by H).

[0105] It should be noted that the encoding module (Encoder) of the multimodal model usually requires the use of the attention mask matrix M. The multimodal model encodes the text embedding T′, the visual embedding A, the position coordinates B and the attention mask matrix M to obtain the feature representation H of the document. The process can be expressed as: H = Encoder (T′, A, B, M). The encoding module of the multimodal model makes full use of the information of multiple modalities such as the text embedding, visual embedding, and position coordinates of the document to encode the feature representation of the document, which can improve the quality of the document feature representation and thus improve the quality of information extraction.

[0106] The attention mask matrix M is a matrix used to shield or mask some information in the attention mechanism. In multimodal models, especially when processing sequence data (such as text sequences), the attention mask matrix M is used to ensure that the model does not consider future information when generating the output at the current position. For text data, the attention mask matrix M is usually a lower triangular matrix, where the elements of the upper triangular part (including the diagonal) are 0 and the elements of the lower triangular part are negative infinity (or a very small number, such as -1×10 20 ) to ensure that when calculating the attention weight, the model can only focus on the sequence information before the current position.

[0107] Step S302: using a multimodal model, based on the feature representation of the document, a prediction score matrix corresponding to each type of link relationship of the input text sequence is predicted, wherein the prediction score matrix includes prediction scores of corresponding link relationships between any two text tags in the input text sequence.

[0108] After encoding the feature representation H of the document, the multimodal model is used to classify and predict the link relationship between any two text tags in the input text sequence according to the feature representation of the document, and the prediction score matrix corresponding to each type of link relationship of the input text sequence is obtained. The dimension of the prediction score matrix is ​​S×S×P. S represents the length of the input text sequence, that is, the number of text tags contained in the input text sequence. P represents the number of all possible link relationships.

[0109] The prediction score matrix of the input text sequence corresponding to any type of link relationship contains the prediction score of any two text tokens in the input text sequence having this type of link relationship. The higher the prediction score, the greater the possibility of the two text tokens having this type of link relationship.

[0110] Exemplarily, it is assumed that all possible link relationships include the following P different types of link relationships: p1, p2, ..., p P The input text sequence X includes the following text tokens: x1, x2, …, x S Then, for any type of link relationship p, the prediction score matrix of the input text sequence corresponding to the link relationship is a three-dimensional matrix Y of S×S×P, where each value Represents the i-th text token x in the input text sequence i and the jth text token x j The prediction score of the link relationship p between them.

[0111] For example, in practical applications, the encoder module (Encoder) of the multimodal model can be connected to two feed-forward neural networks (FFN for short), and the output results of the two feed-forward neural networks are multiplied to obtain the S×S×P prediction score matrix Y. The process can be expressed as: Y=FFN a (H)FFN b (H). FFN a and FFN b Represents two feedforward neural networks. The two feedforward neural networks do not share parameters.

[0112] In addition, the two feedforward neural networks may also be implemented using two multi-layer perceptrons (MLP) or other neural networks with similar capabilities, which is not specifically limited in this embodiment.

[0113] In an optional embodiment, in order to improve the information extraction performance and filter out impossible results, an information isolation mechanism is introduced. The information isolation mechanism provides more prior knowledge to the model through a masking method to enhance the model's recognition accuracy of the link relationship between entity categories and document text information, thereby improving the performance and robustness of the model.

[0114] Optionally, since the result of information extraction from a document is extracted from the document text information and cannot appear in the entity category, before determining the link relationship between any two text tags in the input text sequence based on the prediction score matrix corresponding to each type of link relationship of the input text sequence, a mask operation is performed on the prediction scores between the text tags included in the entity category in the prediction score matrix corresponding to each type of link relationship of the input text sequence.

[0115] By performing a masking operation on the part corresponding to the entity category, the corresponding part of the entity category is shielded, and the entity category and the document text information are isolated, thereby ensuring that the multimodal model can ignore the entity category part when extracting information, only focus on the text information of the document, and extract the entity information from the document text information.

[0116] In practical applications, an entity category can contain multiple text tags. Since the concatenation order of each entity category is determined by itself, the start and end positions of each entity category can be known before inputting into the multimodal model, that is, the head tag and tail tag of each entity category can be specified. By masking the relevant parts of the entity category head tag and / or entity category tail tag, the multimodal model is provided with additional prior knowledge related to the entity category head tag and / or entity category tail tag, which helps to improve the accuracy of link relationship recognition during the training and reasoning stages.

[0117] Optionally, in order to more accurately identify entity category header tags, before determining the link relationship between any two text tags in the input text sequence according to the prediction score matrix corresponding to each type of link relationship of the input text sequence, the server may also perform a mask operation on the prediction scores related to the text tags that belong to the entity category and are not the header tags of the entity category in the prediction score matrix corresponding to the link relationship related to the header tags of the entity category of the input text sequence. The prediction scores related to the text tags that are not the header tags of the entity category are shielded by the mask operation, so that the model can accurately identify the prediction score part related to the text tags of the header tags of the entity category, shield the interference of useless information, and thus improve the accuracy of identifying the header tags of the entity category.

[0118] Among them, the link relationship related to the header tag of the entity category includes but is not limited to: the link relationship from the header tag of the entity category to the header tag of the entity, the link relationship from the header tag of the entity category to the header tag of the value entity, and the link relationship from the header tag of the entity category to the header tag of the key entity.

[0119] Optionally, in order to more accurately identify the link operation related to the end tag of the entity category, before determining the link relationship between any two text tags in the input text sequence according to the prediction score matrix corresponding to each type of link relationship of the input text sequence, the server can also perform a mask operation on the prediction score related to the text tag that belongs to the entity category and is not the end tag of the entity category in the prediction score matrix of the input text sequence corresponding to the link relationship related to the end tag of the entity category. The mask operation shields the prediction score related to the text tag that is not the end tag of the entity category, so that the model can accurately identify the prediction score part related to the text tag of the end tag of the entity category, shielding the interference of useless information, thereby improving the accuracy of identifying the end tag of the entity category.

[0120] Among them, the link relationship related to the tail tag of the entity category includes but is not limited to: the link relationship from the tail tag of the entity to the tail tag of the entity category, the link relationship from the tail tag of the value entity to the tail tag of the entity category, and the link relationship from the tail tag of the key entity to the tail tag of the entity category.

[0121] For example, Represents any two text tokens x in the input text sequence i and x j The mask values ​​corresponding to any type of link relationship p form a mask matrix. After the information isolation mechanism is introduced, any two text tags x in the input text sequence i and x j The prediction score matrix with any type of link relationship p between them can be expressed as:

[0122] Step S303: Determine the link relationship between any two text tags in the input text sequence according to the prediction score matrix corresponding to each type of link relationship of the input text sequence.

[0123] In this step, the link relationship between any two text tags in the input text sequence is determined based on the prediction score matrix corresponding to each type of link relationship of the input text sequence, the prediction score of each type of link relationship between any two text tags in the input text sequence, and the configured score threshold. Among them, any two text tags may have one link relationship or no link relationship.

[0124] Specifically, for any type of link relationship p, for any two text tags x in the input text sequence i and x j , according to the prediction score matrix corresponding to this type of link relationship of the input text sequence, if these two text tags x i and x j If the predicted score of the link relationship p between the two text tags x is greater than or equal to the score threshold, then the two text tags x are determined to be i and x j There is a link relationship p of this type between them.

[0125] If these two text tags x i and x j If the predicted score of the link relationship p between the two text tags x is less than the score threshold, then the two text tags x are determined to be i and x j There is no link relationship p of this type between them.

[0126] By marking any two texts x i and x j The predicted score of the link relationship p between two text tags is compared with the score threshold to determine whether the two text tags have this type of link relationship p, and the recognition result of the link relationship between any two text tags in the input text sequence can be accurately obtained.

[0127] After determining the link relationship between any two text tags in the input text sequence, the link relationship between any two text tags in the input text sequence can be represented as a link relationship matrix. The link relationship matrix has the same dimension as the prediction score matrix, except that the value in the link relationship matrix can be 0 or 1, where a value of 0 indicates that there is no corresponding link relationship, and a value of 1 indicates that there is a corresponding link relationship.

[0128] Exemplarily, it is assumed that all possible link relationships include the following P different types of link relationships: p1, p2, ..., p PThe input text sequence X includes the following text tokens: x1, x2, …, x S Then, the link relationship matrix of the input text sequence is a three-dimensional matrix of S×S×P in Represents the i-th text token x in the input text sequence i and the jth text token x j There is a link relationship p between them. Represents the i-th text token x in the input text sequence i and the jth text token x j There is no link relationship p between them.

[0129] It should be noted that in practical applications, multimodal models have restrictions on input length. By introducing sliding window technology, the text information of the document to be extracted is divided into multiple text information segments (adjacent text segments may overlap partially), and information is extracted from one of the text information segments each time to obtain the information extraction result. Finally, the information extraction results of each text information segment are merged to obtain the final information extraction result. The length of the text information segment is determined by the configured sliding window size. The sliding window technology can dynamically adjust the input length to ensure that all information can be effectively processed by the model.

[0130] The method of this embodiment converts the information extraction problem into a link relationship classification problem (i.e., a pointer classification problem). For an input text sequence containing at least one given entity category and document text information, a multimodal model is used to combine the position coordinates of each text tag in the document image and the document text information to predict whether there are preset types of link relationships between any two text tags in the input text sequence, that is, to classify and predict the link relationship between any two text tags, and obtain the link relationship between any two text tags in the input text sequence. The multimodal model makes full use of the implicit relationship between entities and entity categories, can better understand complex document structures, and accurately predict the link relationship between any two text tags in the input text sequence.

[0131] Figure 4 A framework diagram of an information extraction method provided by an exemplary embodiment of the present application. Figure 4As shown, the overall process of the information extraction method is as follows: according to the document information (including document image, text information and text coordinates) and the given entity category to be extracted, the input text sequence X and the position coordinates B of the text mark contained in the text information are generated. The document image, X and B are input into the multimodal model. The document image, the input text sequence X and the position coordinates B of the text mark contained in the text information are encoded by the encoding module of the multimodal model to obtain the feature representation H of the document. The feature representation H of the document is processed by two feedforward neural networks FFN respectively, and the outputs of the two feedforward neural networks FFN are multiplied to obtain a prediction score matrix of S×S×P. In some optional embodiments, an information isolation mechanism is introduced, and the result of multiplying the outputs of the two feedforward neural networks FFN is multiplied element by element with a mask matrix determined based on the information isolation mechanism to obtain the final prediction score matrix. The prediction score matrix contains the prediction scores of the corresponding link relationship between any two text marks in the input text sequence. According to the prediction score matrix, the entity contained in the document and the entity category corresponding to the entity can be obtained by parsing according to the parsing rules.

[0132] In an exemplary embodiment, in the information extraction scenario of named entity recognition, it is necessary to extract entity information of a given entity category from a document. In this embodiment, possible link relationships between text tags include:

[0133] The first link relationship: the link relationship from the head tag to the tail tag of the same entity;

[0134] The second link relationship: the link relationship from the head tag of the entity category to the head tag of the entity;

[0135] The third link relationship: the link relationship from the tail tag of the previous sub-entity of the discontinuous entity to the head tag of the next sub-entity.

[0136] Among them, the first link relationship links the head tag of each entity to the tail tag, which can be used to detect the span of the entity. The second link relationship links the head tag of each entity category to the head tag of the corresponding entity (i.e., the entity with the entity category). In order to solve the problem of entity discontinuity (such as folded lines or breaks, which cause different parts belonging to the same entity to be split into multiple parts), the head tag and tail tag of different parts of the same entity are linked in sequence through the third link relationship, so that the model can recognize and reconstruct broken or discontinuous entities in the document.

[0137] Optionally, in the information extraction scenario of named entity recognition, possible link relationships between text tags may also include: a link relationship from the tail tag of the same entity to the head tag, and a link relationship from the tail tag of an entity to the tail tag of an entity category. The tail tag of each entity is linked to the head tag of the entity through the link relationship from the tail tag of the same entity to the head tag. The tail tag of each entity is linked to the tail tag of the corresponding entity category through the link relationship from the tail tag of the entity to the tail tag of the entity category. By enriching the link relationship between text tags, the accuracy of information extraction from multimodal models can be improved.

[0138] Figure 5 The flowchart of parsing entity information contained in a document provided in this embodiment. Figure 5 As shown, in the aforementioned step S204, according to the link relationship between any two text tags in the input text sequence, the entity information contained in the document is parsed and obtained, which can be specifically implemented by the following steps:

[0139] Step S501: taking any entity category as a target entity category, and determining the head tag of the target entity to which the head tag of the target entity category is linked based on the text tags having the second link relationship in the input text sequence.

[0140] In this embodiment, through the second link relationship, the head tag of each entity category is linked to the head tag of the corresponding entity. Based on the text tags with the second link relationship in the input text sequence, the connection from the head tag of one or more entity categories to the head tag of the corresponding entity can be determined. For the connection from the head tag of each entity category to the head tag of the corresponding entity, the entity therein is recorded as a target entity to be extracted, and the entity category therein is recorded as the entity category corresponding to the target entity. Thus, the head tag of the target entity to be extracted and the head tag of the entity category corresponding to the target entity are determined.

[0141] Step S502: Determine the target entity according to the text tags with the first link relationship and the text tags with the third link relationship in the input text sequence, and the head tag of the target entity.

[0142] In this step, for a continuous entity (i.e., an entity that has not been split into multiple different sub-entities), the head tag of the same entity is linked to the tail tag of the entity through the first link relationship. According to the head tag of the target entity, the tail tag of the target entity can be found based on the text tag with the first link relationship. Further, according to the head tag and tail tag of the target entity, the corresponding part of the target entity can be found from the input text sequence, that is, the target entity can be extracted.

[0143] For discontinuous entities (i.e. entities that are split into multiple different sub-entities), according to the head tag of the target entity (i.e. the head tag of the first sub-entity of the target entity), based on the text tag with the first link relationship, the tail tag of the first sub-entity of the target entity can be found, and the first sub-entity can be determined. Furthermore, according to the text tag with the third link relationship, the head tag of the next sub-entity to which the tail tag of the first sub-entity links can be found; and then based on the text tag with the first link relationship, the tail tag of the next sub-entity can be found, and the next sub-entity can be determined. By analogy, multiple sub-entities of the discontinuous target entity can be found in sequence, and the multiple sub-entities of the target entity can be spliced ​​in sequence to obtain the complete target entity.

[0144] Step S503: Use the target entity category as the entity category of the target entity.

[0145] In this embodiment, since different entity categories in the input text sequence are segmented using segmentation tags, the target entity category can be determined according to the head tag of the target entity category. The target entity category is used as the entity category corresponding to the target entity, and the entity category of each target entity can be obtained. In this way, the target entity contained in the document and the entity category of the target entity are extracted.

[0146] This embodiment converts the information extraction problem into a link relationship classification problem (i.e., a pointer classification problem). For an input text sequence containing at least one given entity category and document text information, a multimodal model is used to combine the position coordinates of each text tag in the document image and the document text information to predict whether there are preset types of link relationships between any two text tags in the input text sequence, that is, to classify and predict the link relationship between any two text tags, and obtain the link relationship between any two text tags in the input text sequence. The multimodal model makes full use of the implicit relationship between entities and entity categories, can better understand the complex document structure, and accurately predict the link relationship between any two text tags in the input text sequence. Furthermore, according to the link relationship between any two text tags in the input text sequence, the entity information contained in the document is parsed to realize named entity recognition, which can improve the accuracy of named entity information extraction. In addition, the multimodal model can obtain the prediction results of multiple different types of link relationships after one reasoning, realize the parallel prediction of multiple types of link relationships, and improve the efficiency of information extraction. In addition, this solution can flexibly give a set of entity categories to be extracted according to the needs of actual application scenarios, and has no requirements on the layout of the document, and can adapt well to emerging entity categories and changing document layouts.

[0147] In an optional embodiment, in a scenario of extracting information from a document containing a table or table-like data, that is, containing key-value pair information, possible link relationships between text tags include:

[0148] The fourth link relationship: the link relationship from the head tag to the tail tag of the same value entity;

[0149] The fifth link relationship: the link relationship from the head tag of the entity category to the head tag of the value entity;

[0150] The sixth link relationship: the link relationship from the tail tag of the previous sub-entity of the discontinuous value entity to the head tag of the next sub-entity;

[0151] The seventh link relationship: the link relationship from the header tag of the key entity to the header tag of the corresponding value entity;

[0152] The 8th link relationship: the link relationship from the head tag to the tail tag of the same key entity;

[0153] The 9th link relationship: the link relationship from the head tag of the entity class to the head tag of the key entity;

[0154] The 10th link relationship: the link relationship from the tail tag of the previous child entity of the discontinuous key entity to the head tag of the next child entity.

[0155] Among them, through the 4th link relationship, the head tag of each value entity is linked to the tail tag of the value entity. Through the 5th link relationship, the head tag of each entity category is linked to the head tag of the corresponding value entity (i.e., the value entity with the entity category). Through the 6th link relationship, the head tag and tail tag of different parts of the same value entity are linked in sequence, so that the model can recognize and reconstruct broken or discontinuous value entities in the document. Through the 7th link relationship, the head tag of each key entity is linked to the head tag of the corresponding value entity. Through the 8th link relationship, the head tag of each key entity is linked to the tail tag of the key entity. Through the 9th link relationship, the head tag of each entity category is linked to the head tag of the corresponding key entity (i.e., the key entity with the entity category). Through the 10th link relationship, the head tag and tail tag of different parts of the same key entity are linked in sequence, so that the model can recognize and reconstruct broken or discontinuous key entities in the document.

[0156] Optionally, in the scenario of extracting key-value pair information from a document, possible link relationships between text tags may also include:

[0157] The link relationship from the tail tag to the head tag of the same value entity, the link relationship from the tail tag of the value entity to the tail tag of the corresponding entity category, the link relationship from the tail tag to the head tag of the same key entity, the link relationship from the tail tag of the key entity to the tail tag of the corresponding entity category, and the link relationship from the tail tag of the value entity to the tail tag of the corresponding key entity.

[0158] Through the link relationship from the tail tag to the head tag of the same value entity, the tail tag of each value entity can be linked to the head tag of the value entity. Through the link relationship from the tail tag of the value entity to the tail tag of the corresponding entity category, the tail tag of each value entity can be linked to the tail tag of the corresponding entity category. Through the link relationship from the tail tag to the head tag of the same key entity, the tail tag of each key entity can be linked to the head tag of the key entity. Through the link relationship from the tail tag of the key entity to the tail tag of the corresponding entity category, the tail tag of each key entity can be linked to the tail tag of the corresponding entity category. Through the link relationship from the tail tag of the value entity to the tail tag of the corresponding key entity, the tail tag of each value entity can be linked to the tail tag of the corresponding key entity. By enriching the link relationship between text tags, the model's ability to understand complex document structures can be enhanced, and the accuracy of information extraction of multimodal models can be improved.

[0159] The method of this embodiment can be applied to perform named entity recognition on a document containing key-value pair information, or to extract key-value pair information.

[0160] In some example scenarios, when performing named entity recognition on a document containing table data (usually containing key-value pair information), the named entity information to be extracted usually comes from the value in the key-value pair. In the aforementioned step S204, according to the link relationship between any two text tags in the input text sequence, the entity information contained in the document is parsed and obtained, which can be specifically implemented in the following manner:

[0161] According to the text tags with the 4th link relationship, the text tags with the 5th link relationship, and the text tags with the 6th link relationship in the input text sequence, the target value entity contained in the document is determined, and the entity category of the target value entity is determined.

[0162] Specifically, for the value entities contained in the document, any entity category is used as the target entity category, and the head tag of the target value entity to which the head tag of the target entity category is linked can be determined according to the text tag with the 4th link relationship in the input text sequence; the target value entity is determined according to the text tag with the 5th link relationship and the text tag with the 6th link relationship in the input text sequence, as well as the head tag of the target value entity; the target entity category is used as the entity category of the target value entity. The specific implementation principle is referred to the aforementioned steps S501-S503, which will not be repeated here.

[0163] Exemplarily, possible link relationships between text tags that can be configured include:

[0164] ① The link relationship from the head tag to the tail tag of the same value entity. This type of link relationship links the head tag of each value entity to (points to) the tail tag of the value entity. The category value of this type of link relationship can be defined as 1.

[0165] ② The link relationship from the end tag of the same value entity to the head tag. This type of link relationship links the end tag of each value entity to (points to) the head tag of the value entity. The category value of this type of link relationship can be defined as 2.

[0166] ③ The link relationship from the header tag of the entity category to the header tag of the value entity. This type of link relationship links the header tag of each entity category to the header tag of the corresponding value entity (i.e., the value entity with the entity category). The category value of this type of link relationship can be defined as 3.

[0167] ④ The link relationship from the tail tag of the value entity to the head tag of the entity category. This type of link relationship links the tail tag of each value entity to the head tag of the corresponding entity category. The category value of this type of link relationship can be defined as 4.

[0168] ⑤ The link relationship from the tail tag of the previous sub-entity of the discontinuous value entity to the head tag of the next sub-entity. This type of link relationship links the head tag and tail tag of different parts of the same value entity in sequence, so that the model can identify and reconstruct the broken or discontinuous value entities in the document. The category value of this type of link relationship can be defined as 5.

[0169] ⑥ Link relationship from the head tag to the tail tag of the same key entity. This type of link relationship links the head tag of each key entity to (points to) the tail tag of the key entity. The category value of this type of link relationship can be defined as 6.

[0170] ⑦ A link relationship from the header tag of the key entity to the header tag of the corresponding value entity. This type of link relationship links the header tag of each key entity to the header tag of the corresponding value entity. The category value of this type of link relationship can be defined as 7.

[0171] ⑧ Link relationship from the end tag of the value entity to the end tag of the corresponding key entity. This type of link relationship links the end tag of each value entity to the end tag of the corresponding key entity. The category value of this type of link relationship can be defined as 8.

[0172] ⑨ A link relationship from the header tag of an entity class to the header tag of a key entity. This type of link relationship links (points to) the header tag of each entity class to the header tag of the corresponding key entity (the key entity with the entity class). The category value of this type of link relationship can be defined as 9.

[0173] ⑩ A link relationship from the end tag of the key entity to the end tag of the corresponding entity class. This type of link relationship links (points to) the end tag of each key entity to the end tag of the corresponding entity class. The category value of this type of link relationship can be defined as 10.

[0174] A link relationship from the tail tag of the previous child entity of a discontinuous key entity to the head tag of the next child entity. This type of link relationship links the head tag and tail tag of different parts of the same key entity in sequence, allowing the model to identify and reconstruct broken or discontinuous key entities in the document. The category value of this type of link relationship can be defined as 11.

[0175] Figure 6 A partial example of a document provided for this embodiment, for Figure 6 The document shown in the figure is identified by OCR, and the text information in the last row of the table is "work unit natural resources current Hunanprovince bureau address". Assume that the entity categories to be extracted include org (i.e. organization) and address. Then the input text sequence is: <s> org[T]address< / s> work unit natural resourcescurrent Hunan province bureau address. Based on the aforementioned link relationship The link relationship between each text token in the input text sequence is as follows Figure 7 shown.

[0176] Based on the defined link relationship, the input text sequence is represented as follows Figure 7 The arrows in the figure represent link relationships, and the category values ​​of the link relationships are marked on the arrows. The link relationship between any two text tags in the input text sequence can be expressed as follows: Figure 8 The two-dimensional relationship matrix shown in FIG. The dimension of the two-dimensional relationship matrix is ​​S×S, where S is the length of the input text sequence. The values ​​in the two-dimensional relationship matrix represent the category values ​​of the link relationship.

[0177] Furthermore, based on the link relationship between each text tag in the input text sequence, starting from the head tag of each entity category, the entity information contained in the text information can be found according to the link relationship between each text tag. For example, "natural resources bureau" corresponds to the entity category "org"; "Hunan province" corresponds to the entity category "address".

[0178] In other example scenarios, when extracting information from a document containing tabular data (usually containing key-value pair information), it is necessary to extract the key-value pair information of the document.

[0179] Fig. 9 The flowchart of parsing the key-value pair information contained in the document provided in this embodiment. Fig. 9 As shown, in the aforementioned step S204, according to the link relationship between any two text tags in the input text sequence, the entity information contained in the document is parsed and obtained, which can be specifically implemented in the following manner:

[0180] Step S901: Determine the target value entity contained in the document according to the text tag with the 4th link relationship, the text tag with the 5th link relationship, and the text tag with the 6th link relationship in the input text sequence.

[0181] In this step, for the value entities contained in the document, any entity category is used as the target entity category, and the head tag of the target value entity to which the head tag of the target entity category is linked can be determined according to the text tag with the 4th link relationship in the input text sequence; the target value entity is determined according to the text tag with the 5th link relationship and the text tag with the 6th link relationship in the input text sequence, as well as the head tag of the target value entity; the target entity category is used as the entity category of the target value entity. The specific implementation principle is referred to the aforementioned steps S501-S503, which will not be repeated here.

[0182] Step S902: Determine the target key entity contained in the document according to the text tag with the 8th link relationship, the text tag with the 9th link relationship, and the text tag with the 10th link relationship in the input text sequence.

[0183] In this step, for the key entities contained in the document, any entity category is used as the target entity category, and the head tag of the target key entity to which the head tag of the target entity category is linked can be determined according to the text tag with the 8th link relationship in the input text sequence; the target key entity is determined according to the text tag with the 9th link relationship and the text tag with the 10th link relationship in the input text sequence, as well as the head tag of the target key entity; the target entity category is used as the entity category of the target key entity. The specific implementation principle is referred to the aforementioned steps S501-S503, which will not be repeated here.

[0184] It should be noted that there is no data dependency between steps S901 and S902, and the two steps can no longer be executed in parallel, or can be executed successively in any order without affecting the quality of information extraction, which is not specifically limited here.

[0185] Thus, the document includes step S903, determining the corresponding relationship between the target key entity and the target value entity according to the text tag with the seventh link relationship in the input text sequence.

[0186] In this step, the correspondence between the target key entity and the target value entity is determined based on the text tag with the seventh link relationship in the input text sequence and the link relationship between the head tag of the target key entity and the head tag of the target value entity.

[0187] Step S904: Determine the key entity-value entity pair information contained in the document according to the corresponding relationship between the target key entity and the target value entity.

[0188] According to the extracted target key entity, target value entity, and the corresponding relationship between the target key entity and the target value entity, the key entity-value entity pair information contained in the document is determined, and the key-value pair information contained in the document is obtained.

[0189] In this embodiment, the key-value pair information extraction problem is converted into a link relationship classification problem (i.e., a pointer classification problem). For an input text sequence containing at least one given entity category and document text information, a multimodal model is used to combine the position coordinates of each text tag in the document image and the document text information to predict whether there are preset types of link relationships between any two text tags in the input text sequence, that is, to classify and predict the link relationship between any two text tags, and obtain the link relationship between any two text tags in the input text sequence. The multimodal model makes full use of the implicit relationship between entities and entity categories, as well as the implicit relationship between key entities and value entities, and can better understand the complex document structure and accurately predict the link relationship between any two text tags in the input text sequence. Furthermore, according to the link relationship between any two text tags in the input text sequence, the key-value pair information contained in the document is parsed, which can improve the accuracy of key-value pair information extraction. In addition, the multimodal model can obtain the prediction results of multiple different types of link relationships after one reasoning, realize the parallel prediction of multiple types of link relationships, and improve the efficiency of key-value pair information extraction. In addition, this solution can flexibly give a set of entity categories to be extracted according to the needs of actual application scenarios, and has no requirements on the layout of the document, and can adapt well to emerging entity categories and changing document layouts.

[0190] Fig.10 A flowchart of a method for multimodal model training provided by an exemplary embodiment of the present application, such as Fig.10 As shown, the training process of the multimodal model used in the above embodiments is as follows:

[0191] Step S1001, obtaining at least one entity category set and training samples corresponding to each entity category set, wherein the training samples include document data and annotation information, wherein the document data includes: a document image of the document sample, an input text sequence, and position coordinates of each text tag in the text information of the document sample, and the annotation information includes a link relationship between any two text tags in the input text sequence of the document sample.

[0192] In this embodiment, during training, the multimodal model can be trained for one or more different entity category sets. The trained multimodal model can be applied to other new entity category sets, thereby improving the model's generalization ability for new entity categories and being suitable for zero-sample / few-sample key information extraction tasks.

[0193] In practical applications, a training set for information extraction tasks can be obtained, which usually includes a document image of a document sample, a text tag sequence of document text information, and entity annotation information in the text tag sequence of document text information. The entity annotation information marks the entities (including key entities and value entities) in the text tag sequence of document text information and the entity categories of the entities.

[0194] Based on the training set of the information extraction task, according to the given entity category set and the text information of the document, an input text sequence can be generated. For the specific implementation principle, please refer to the relevant content of generating an input text sequence based on a document in the aforementioned embodiment, which will not be repeated here. By performing text recognition on the document image, the position coordinates of the text in the text information can be determined, and then the position coordinates of the text tags belonging to the document text information in the input text sequence can be determined. For the specific implementation principle, please refer to the relevant content of determining the position coordinates of each text tag in the text information in the aforementioned implementation, which will not be repeated here. In this way, the document data can be obtained.

[0195] Furthermore, according to the entity annotation information of the text tag sequence of the text information of the document sample, the link relationship between any two text tags in the input text sequence can be determined, thereby obtaining the annotation information of the document data.

[0196] Optionally, the link relationship between any two text tags in the input text sequence can be represented as a link relationship matrix. The dimension of the link relationship matrix is ​​S×S×P. The value in the link relationship matrix can be 0 or 1, where a value of 0 indicates that there is no corresponding link relationship, and a value of 1 indicates that there is a corresponding link relationship.

[0197] Exemplarily, it is assumed that all possible link relationships include the following P different types of link relationships: p1, p2, ..., p P The input text sequence X includes the following text tokens: x1, x2, …, x SThen, the link relationship matrix of the input text sequence is a three-dimensional matrix of S×S×P in Represents the i-th text token x in the input text sequence i and the jth text token x j There is a link relationship p between them. Represents the i-th text token x in the input text sequence i and the jth text token x j There is no link relationship p between them.

[0198] Step S1002: input the document data into the multimodal model for inference prediction, and obtain prediction scores for various types of link relationships between any two text tags in the input text sequence of the document data.

[0199] In this step, document data (including document images of document samples, position coordinates of each text tag in the input text sequence and text information of the document samples) is input into the multimodal model. The multimodal model predicts the link relationship between any two text tags in the input text sequence based on the document images of the document samples, the input text sequence and the position coordinates of each text tag in the text information of the document samples, and obtains the prediction scores of various types of link relationships between any two text tags in the input text sequence.

[0200] The prediction scores of various types of link relationships between any two text tags in the input text sequence of the document data can be expressed as a prediction score matrix corresponding to various types of link relationships of the input text sequence. The dimension of the prediction score matrix is ​​S×S×P. S represents the length of the input text sequence, that is, the number of text tags contained in the input text sequence. P represents the number of all possible link relationships.

[0201] The prediction score matrix of the input text sequence corresponding to any type of link relationship contains the prediction score of any two text tokens in the input text sequence having this type of link relationship. The higher the prediction score, the greater the possibility of the two text tokens having this type of link relationship.

[0202] Exemplarily, it is assumed that all possible link relationships include the following P different types of link relationships: p1, p2, ..., p P The input text sequence X includes the following text tokens: x1, x2, …, x S Then, for any type of link relationship p, the prediction score matrix of the input text sequence corresponding to the link relationship is a three-dimensional matrix Y of S×S×P, where each value Represents the i-th text token x in the input text sequence i and the jth text token x jThe prediction score of the link relationship p between them.

[0203] The implementation principle of this step refers to the relevant content of step S203 in the above embodiment, which will not be repeated here.

[0204] Step S1003: Calculate the loss function value according to the prediction scores of various types of link relationships between any two text tags in the input text sequence of the document data and the link relationship annotation information.

[0205] In this step, the link relationship between any two text tags in the input text sequence in the link relationship annotation information is represented as a link relationship matrix The prediction score matrix Y with various link relationships between any two text tags in the input text sequence is mapped into a one-dimensional vector to obtain the prediction score vector Z. Mapped into a one-dimensional vector, we get the link relationship vector

[0206] Further, according to the prediction score vector Z and the link relationship vector Calculate the ring loss function value: According to the ring loss function value, further, according to the calculated loss function value, the parameters of the multimodal model (including the parameters of the encoding module and the two feedforward neural networks) are updated through back propagation.

[0207] Optionally, the cross entropy loss function value can be calculated based on the predicted scores of various types of link relationships between any two text tags in the input text sequence of the document data, as well as the link relationship annotation information. Further, based on the calculated cross entropy loss function value, the parameters of the multimodal model (including the parameters of the encoding module and the two feedforward neural networks) are updated through back propagation.

[0208] Step S1004: adjust the parameters of the multimodal model according to the loss function value.

[0209] In this step, the parameters of the multimodal model (including the parameters of the encoding module and the two feedforward neural networks) are updated through back propagation according to the calculated loss function value.

[0210] It should be noted that, in this embodiment, the strategy used for domain fine-tuning of the base model (including the extended vocabulary) based on the domain data of the target domain, including but not limited to which training strategy to use (full parameter fine-tuning, or partial parameter fine-tuning), which optimization algorithm to use, learning rate, batch size, etc., can be configured and adjusted according to the needs of the actual application scenario, and no specific limitation is made here.

[0211] For example, Fig.11This is a framework diagram of the multimodal model training provided in this embodiment. Fig.11 As shown, the training process of the multimodal model for information extraction is as follows: the document information (including document image, text information and text coordinates) is combined with the given entity category to be extracted to generate the input text sequence X and the position coordinates B of the text tags contained in the text information to obtain the document data. The document data (including document image, X and B) is input into the multimodal model. The document data is encoded by the encoding module of the multimodal model to obtain the feature representation H of the document sample. The feature representation H of the document sample is processed by two feedforward neural networks FFN respectively, and the outputs of the two feedforward neural networks FFN are multiplied to obtain a prediction score matrix of S×S×P. In some optional embodiments, an information isolation mechanism is introduced, and the result of multiplying the outputs of the two feedforward neural networks FFN is multiplied element by element with the mask matrix determined based on the information isolation mechanism to obtain the final prediction score matrix. The prediction score matrix contains the prediction scores of corresponding link relationships between any two text tags in the input text sequence.

[0212] The link relationship between any two text tags in the input text sequence X is determined based on the entity annotation information of the document text information. The link relationship matrix of the input text sequence is determined based on the link relationship between any two text tags in the input text sequence X as the link relationship matrix annotation information. The loss function value is calculated based on the prediction score matrix and the link relationship matrix annotation information obtained by the multimodal model prediction, and the parameters of the multimodal model are adjusted based on the loss function value.

[0213] The method of this embodiment converts the key-value pair information extraction problem into a link relationship classification problem (i.e., a pointer classification problem). In the training process of the multimodal model, at least one entity category set and training samples corresponding to each entity category set are obtained. The training samples include document data and annotation information. The input text sequence in the document data contains the given entity category and the text information of the document sample. The multimodal model is combined with the position coordinates of each text tag in the document image and the text information to predict whether there are preset types of link relationships between any two text tags in the input text sequence, that is, the link relationship between any two text tags is classified and predicted to obtain the prediction score of any two text tags in the input text sequence having various types of link relationships. Further, according to the prediction score of any two text tags in the input text sequence of the document data having various types of link relationships and the link relationship annotation information, the loss function value is calculated, and the parameters of the multimodal model are adjusted according to the loss function value, so that the multimodal model can fully learn the implicit relationship between entities and entity categories, as well as the implicit relationship between key entities and value entities, and can better understand the complex document structure and accurately predict the link relationship between any two text tags in the input text sequence. Furthermore, according to the link relationship between any two text tags in the input text sequence, the key-value pair information contained in the document can be parsed to improve the accuracy of key-value pair information extraction. Moreover, the multimodal model can obtain the prediction results of multiple different types of link relationships after one reasoning, realize the parallel prediction of multiple types of link relationships, and improve the efficiency of key-value pair information extraction. In addition, this solution can flexibly give the set of entity categories to be extracted according to the needs of the actual application scenario, and has no requirements for the layout of the document, which can well adapt to emerging entity categories and changing document layouts.

[0214] Fig.12 A flowchart of a method for extracting information provided by an exemplary embodiment of the present application. Fig.12 As shown, the specific steps of this method are as follows:

[0215] Step S1201: In response to an information extraction request, a document image, text information and text coordinates of a document to be processed, and at least one given entity category are obtained.

[0216] The implementation principle of this step refers to the relevant content of step S201 in the above embodiment, and the steps are not repeated here.

[0217] Step S1202: Generate an input text sequence including at least one entity category and text information, and determine the position coordinates of each text mark in the text information according to the text coordinates.

[0218] The implementation principle of this step refers to the relevant content of step S202 in the above embodiment, and the steps are not repeated here.

[0219] Step S1203: predicting the link relationship between any two text tags in the input text sequence through the multimodal model according to the document image of the document, the input text sequence and the position coordinates of each text tag in the text information.

[0220] The implementation principle of this step refers to the relevant content of step S203 in the above embodiment, and the step is repeated here.

[0221] Step S1204: parse and obtain entity information contained in the document according to the link relationship between any two text tags in the input text sequence.

[0222] The implementation principle of this step refers to the relevant content of step S204 in the above embodiment, and the steps are not repeated here.

[0223] Step S1205: Output entity information contained in the document.

[0224] After obtaining the entity information contained in the document, the server may return the entity information contained in the document to the end-side device. The end-side device executes subsequent processing logic based on the entity information contained in the document.

[0225] The specific implementation principle and technical effects of this embodiment refer to the above embodiments and will not be repeated here.

[0226] Fig.13 A flowchart of a method for extracting information from a legal document provided by an exemplary embodiment of the present application. Fig.13 As shown, the specific steps of this method are as follows:

[0227] Step S1301: In response to an information extraction request, a document image, text information and text coordinates of a given legal document, and at least one given entity category are obtained.

[0228] The implementation principle of this step refers to the relevant content of step S201 in the above embodiment, and the steps are not repeated here.

[0229] Step S1302: Generate an input text sequence including at least one entity category and text information, and determine the position coordinates of each text mark in the text information according to the text coordinates.

[0230] The implementation principle of this step refers to the relevant content of step S202 in the above embodiment, and the steps are not repeated here.

[0231] Step S1303: predicting the link relationship between any two text tags in the input text sequence through the multimodal model according to the document image of the legal document, the input text sequence and the position coordinates of each text tag in the text information.

[0232] Specifically, the document image of the legal document, the input text sequence and the position coordinates of each text tag in the text information are input into the multimodal model for encoding to obtain the feature representation of the legal document. Through the multimodal model, based on the feature representation of the document, the prediction score matrix corresponding to each type of link relationship of the input text sequence is predicted, and the prediction score matrix contains the prediction scores of the corresponding link relationship between any two text tags in the input text sequence. Based on the prediction score matrix corresponding to each type of link relationship of the input text sequence, the link relationship between any two text tags in the input text sequence is determined.

[0233] The implementation principle of this step refers to the relevant content of step S203 in the above embodiment, and the step is repeated here.

[0234] Step S1304: parse and obtain entity information contained in the legal document according to the link relationship between any two text tags in the input text sequence.

[0235] The implementation principle of this step refers to the relevant content of step S204 in the above embodiment, and the steps are not repeated here.

[0236] Step S1305: Output entity information contained in the legal document.

[0237] After obtaining the entity information contained in the legal document, the server may return the entity information contained in the legal document to the end-side device. The end-side device executes subsequent processing logic based on the entity information contained in the legal document.

[0238] The specific implementation principle and technical effects of this embodiment refer to the above embodiments and will not be repeated here.

[0239] Fig.14 A schematic diagram of the structure of a server provided in an embodiment of the present application. Fig.14 As shown, the server includes: a memory 1401 and a processor 1402. The memory 1401 is used to store computer-executable instructions and can be configured to store various other data to support operations on the server. The processor 1402 is connected to the memory 1401 in communication and is used to execute the computer-executable instructions stored in the memory 1401 to implement the technical solution provided by any of the above method embodiments. The specific functions and technical effects that can be achieved are similar and will not be repeated here.

[0240] Optional, such as Fig.14 As shown, the server also includes: a firewall 1403, a load balancer 1404, a communication component 1405, a power supply component 1406 and other components. Fig.14 Only some components are shown schematically, which does not mean that the server only includes Fig.14 Components shown. Fig.14In the description, the server is only taken as a cloud server deployed in the cloud as an example for exemplary description. The server can also be deployed locally, and this embodiment is not specifically limited here.

[0241] An embodiment of the present application also provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the method of any of the aforementioned embodiments is implemented. The specific functions and technical effects that can be achieved are not repeated here.

[0242] The present application also provides a computer program product, including a computer program, which implements the method of any of the above embodiments when executed by a processor. The computer program is stored in a readable storage medium, and at least one processor of the server can read the computer program from the readable storage medium. At least one processor executes the computer program so that the server executes the technical solution provided by any of the above method embodiments. The specific functions and technical effects that can be achieved are not repeated here.

[0243] The embodiment of the present application provides a chip, including: a processing module and a communication interface, the processing module can execute the technical solution of the server in the aforementioned method embodiment. Optionally, the chip also includes a storage module (such as a memory), the storage module is used to store instructions, the processing module is used to execute the instructions stored in the storage module, and the execution of the instructions stored in the storage module enables the processing module to execute the technical solution provided by any of the aforementioned method embodiments.

[0244] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0245] It should be understood that the above processor can be a central processing unit (CPU), a graphics processing unit (GPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in at least one processor.

[0246] The memory may include high-speed random access memory (Random Access Memory, RAM for short), and may also include non-volatile storage, such as at least one disk storage, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc. The above-mentioned memory may be an object storage service (Object Storage Service, OSS for short). The above-mentioned memory may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, SRAM for short), electrically erasable programmable read-only memory (EEPROM for short), erasable programmable read-only memory (EPROM for short), programmable read-only memory (PROM for short), read-only memory (ROM for short), magnetic storage, flash memory, disk or optical disk.

[0247] The above-mentioned communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as a mobile hotspot (WiFi), a second-generation mobile communication system (2G), a third-generation mobile communication system (3G), a fourth-generation mobile communication system (4G) / Long Term Evolution (LongTerm Evolution, referred to as LTE), a fifth-generation mobile communication system (5G) and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (Near Field Communication, referred to as NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (Radio Frequency Identification, referred to as RFID) technology, infrared technology, ultra-wideband (Ultra Wide Band, referred to as UWB) technology, Bluetooth technology and other technologies.

[0248] The power supply assembly provides power to various components of the device where the power supply assembly is located. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0249] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0250] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in a dedicated integrated circuit. Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0251] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0252] The order of the above-mentioned embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. In addition, in some of the processes described in the above-mentioned embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed or executed in parallel in the order in which they appear in this article, and are only used to distinguish between different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or less operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit "first" and "second" to different types. The meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0253] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0254] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the art that are not disclosed in the present application.

[0255] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An information extraction method, characterized in that: include: Obtaining a document image, text information and text coordinates of a document to be processed, and at least one given entity category; generating an input text sequence including the at least one entity category and text information, and determining position coordinates of each text mark in the text information according to the text coordinates; Predicting, by a multimodal model, a link relationship between any two text tags in the input text sequence according to a document image of the document, the input text sequence, and position coordinates of each text tag in the text information; According to the link relationship between any two text tags in the input text sequence, the entity information contained in the document is parsed and obtained.

2. The method according to claim 1, characterized in that The generating of the input text sequence including the at least one entity category and text information comprises: Converting the at least one entity category into a text tag sequence, and converting the text information of the document into a text tag sequence; The text tag sequence of the at least one entity category and the text tag sequence of the text information are concatenated to obtain an input text sequence.

3. The method according to claim 1, characterized in that Predicting a link relationship between any two text tags in the input text sequence according to the document image of the document, the input text sequence and the position coordinates of each text tag in the text information by using a multimodal model, including: Inputting the document image of the document, the input text sequence and the position coordinates of each text mark in the text information into a multimodal model for encoding to obtain a feature representation of the document; By using the multimodal model, according to the feature representation of the document, a prediction score matrix corresponding to each type of link relationship of the input text sequence is predicted, wherein the prediction score matrix includes the prediction scores of corresponding link relationships between any two text tags in the input text sequence; The link relationship between any two text tags in the input text sequence is determined according to the prediction score matrix corresponding to each type of link relationship of the input text sequence.

4. The method according to claim 3, characterized in that The step of inputting the document image of the document, the input text sequence, and the position coordinates of each text mark in the text information into a multimodal model for encoding to obtain a feature representation of the document includes: The document image, the input text sequence and the position coordinates of each text tag in the input text sequence are input into a multimodal model, and the document image is mapped into a visual embedding, the input text sequence is mapped into a text tag embedding through the multimodal model, and the visual embedding, text tag embedding and the position coordinates of each text tag in the text information are encoded to obtain a feature representation of the document.

5. The method according to claim 3, characterized in that: Determining the link relationship between any two text tags in the input text sequence according to the prediction score matrix corresponding to each type of link relationship of the input text sequence includes: For any type of link relationship, for any two text tags in the input text sequence, a prediction score matrix corresponding to the type of link relationship according to the input text sequence; If the prediction score of having this type of link relationship between any two text tags is greater than or equal to the score threshold, it is determined that the any two text tags have this type of link relationship.

6. The method according to claim 3, characterized in that Before determining the link relationship between any two text tags in the input text sequence according to the prediction score matrix corresponding to each type of link relationship of the input text sequence, the method further includes: A mask operation is performed on the prediction scores between text tags included in the entity category in the prediction score matrix corresponding to each type of link relationship of the input text sequence.

7. The method according to claim 3, characterized in that Before determining the link relationship between any two text tags in the input text sequence according to the prediction score matrix corresponding to each type of link relationship of the input text sequence, the method further includes: Performing a mask operation on the prediction scores related to the text tags that belong to the entity category and are not the head tags of the entity category in the prediction score matrix of the input text sequence corresponding to the link relationship related to the head tag of the entity category; and / or, A mask operation is performed on the prediction scores related to the text tags that belong to the entity category and are not the tail tags of the entity category in the prediction score matrix of the input text sequence corresponding to the link relationship related to the tail tag of the entity category.

8. The method according to claim 1, characterized in that The link relationship includes: The first link relationship: the link relationship from the head tag to the tail tag of the same entity; The second link relationship: the link relationship from the head tag of the entity category to the head tag of the entity; The third link relationship: the link relationship from the tail tag of the previous sub-entity of the discontinuous entity to the head tag of the next sub-entity.

9. The method according to claim 8, characterized in that The step of parsing the entity information contained in the document according to the link relationship between any two text tags in the input text sequence includes: Taking any entity category as a target entity category, and determining the head tag of the target entity to which the head tag of the target entity category is linked according to the text tags having the second link relationship in the input text sequence; Determine the target entity according to the text tags having the first link relationship and the text tags having the third link relationship in the input text sequence, and the head tag of the target entity; The target entity category is used as the entity category of the target entity.

10. The method according to claim 1, characterized in that When applied to the scenario of extracting key-value pair information, the link relationship includes: The fourth link relationship: the link relationship from the head tag to the tail tag of the same value entity; The fifth link relationship: the link relationship from the head tag of the entity category to the head tag of the value entity; The sixth link relationship: the link relationship from the tail tag of the previous sub-entity of the discontinuous value entity to the head tag of the next sub-entity; The seventh link relationship: the link relationship from the header tag of the key entity to the header tag of the corresponding value entity; The 8th link relationship: the link relationship from the head tag to the tail tag of the same key entity; The 9th link relationship: the link relationship from the head tag of the entity class to the head tag of the key entity; The 10th link relationship: the link relationship from the tail tag of the previous child entity of the discontinuous key entity to the head tag of the next child entity.

11. The method according to claim 10, characterized in that The step of parsing the target entity contained in the document and the entity category of the target entity according to the link relationship between any two text tags in the input text sequence includes: Determining the target value entity contained in the document according to the text tags having the fourth link relationship, the text tags having the fifth link relationship, and the text tags having the sixth link relationship in the input text sequence; Determine the target key entity included in the document according to the text tag with the eighth link relationship, the text tag with the ninth link relationship, and the text tag with the tenth link relationship in the input text sequence; Determining the corresponding relationship between the target key entity and the target value entity according to the text tags having the seventh link relationship in the input text sequence; According to the corresponding relationship between the target key entity and the target value entity, the key entity-value entity pair information contained in the document is determined.

12. The method according to any one of claims 1 to 11, characterized in that The step of obtaining a document image, text information and text coordinates of a document to be processed, and at least one given entity category, includes: In response to an information extraction request, obtaining a document image of a document to be processed and at least one given entity category; Perform text recognition on the document image to obtain text information and text coordinates of the document.

13. The method according to any one of claims 1 to 11, characterized in that The training process of the multimodal model includes: Acquire at least one entity category set and training samples corresponding to each entity category set, wherein the training samples include document data and link relationship annotation information, wherein the document data includes: a document image of the document sample, an input text sequence, and position coordinates of each text tag in the text information of the document sample, and the link relationship annotation information includes a link relationship between any two text tags in the input text sequence of the document sample; Inputting the document data into a multimodal model for inference prediction, and obtaining prediction scores of various types of link relationships between any two text tags in an input text sequence of the document data; Calculating a loss function value according to the predicted scores of various types of link relationships between any two text tags in the input text sequence of the document data and the link relationship annotation information; According to the loss function value, the parameters of the multimodal model are adjusted.

14. An information extraction method, characterized in that: include: In response to the information extraction request, obtaining a document image, text information and text coordinates of a given legal document, and at least one given entity category; generating an input text sequence including the at least one entity category and the text information, and determining the position coordinates of each text mark in the text information according to the text coordinates; Predicting, by a multimodal model, a link relationship between any two text tags in the input text sequence according to the document image of the legal document, the input text sequence, and the position coordinates of each text tag in the text information; Parsing and obtaining entity information contained in the legal document according to a link relationship between any two text tags in the input text sequence; Output the entity information contained in the legal document.

15. An information extraction method, characterized in that: include: In response to the information extraction request, obtaining a document image, text information and text coordinates of a given legal document, and at least one given entity category; generating an input text sequence including the at least one entity category and the text information, and determining the position coordinates of each text mark in the text information according to the text coordinates; Predicting, by a multimodal model, a link relationship between any two text tags in the input text sequence according to the document image of the legal document, the input text sequence, and the position coordinates of each text tag in the text information; Parsing and obtaining entity information contained in the legal document according to a link relationship between any two text tags in the input text sequence; Output the entity information contained in the legal document.

16. The method according to claim 15, characterized in that Predicting a link relationship between any two text tags in the input text sequence according to the document image of the legal document, the input text sequence, and the position coordinates of each text tag in the text information through a multimodal model, including: Inputting the document image of the legal document, the input text sequence, and the position coordinates of each text mark in the text information into a multimodal model for encoding to obtain a feature representation of the legal document; By using the multimodal model, according to the feature representation of the document, a prediction score matrix corresponding to each type of link relationship of the input text sequence is predicted, wherein the prediction score matrix includes the prediction scores of corresponding link relationships between any two text tags in the input text sequence; The link relationship between any two text tags in the input text sequence is determined according to the prediction score matrix corresponding to each type of link relationship of the input text sequence.

17. A server, characterized in that: include: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the server to execute the method described in any one of claims 1-16.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 16 is implemented.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 16 is implemented.

Images