Information extraction model training method, information extraction method, device and electronic equipment
By training the information extraction model, using the encoder, graph convolution network and entity detection network, combined with graph structure information, the problem of low utilization rate of document information of the information extraction model is solved, and high-accuracy automatic information extraction and Cost reduction.
Patent Information
- Application Number
- CN202011430367.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-09
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2040-12-09
AI Technical Summary
The existing information extraction model has a low utilization rate of the original document during the learning process, resulting in low accuracy of the information extraction results, and high labor cost in formulating rules, and it is impossible to quickly transfer to documents in different fields.
By obtaining marked documents, using the information extraction model of encoder, graph convolution network and entity detection network for training, information extraction is automatically performed, labor costs are reduced, and text and layout information are fully utilized through graph structure information to improve information utilization rate .
Automatic information extraction is realized, labor costs are reduced, the accuracy of information extraction results is improved, and information extraction in documents in different fields is adapted to information extraction.
Smart Images

Figure CN114611499B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of computer technology, and in particular to an information extraction model training method, an information extraction method, a device and an electronic device. Background Art
[0002] Information Extraction (IE) refers to the extraction of entities and other information from unstructured data (such as text). The extraction results can generally be used by downstream tasks, such as building knowledge graphs.
[0003] In the prior art, rule-based extraction algorithms are usually used for information extraction, such as the TF-IDF (termfrequency–inverse document frequency) algorithm. However, the formulation of rules is labor-intensive and documents in different fields cannot be quickly migrated. Another existing information extraction method is to rely on information extraction models to implement information extraction functions. However, the existing information extraction models have a low utilization rate of the original document information during the learning process, resulting in low accuracy of the information extraction results. Summary of the invention
[0004] The embodiments of the present application propose an information extraction model training method, an information extraction method, a device and an electronic device to reduce labor costs and improve the accuracy of information extraction results.
[0005] In a first aspect, an embodiment of the present application provides an information extraction model training method, comprising: obtaining an annotated document, the annotated document comprising at least one text box, the text in each text box being labeled with an entity; encoding each text box through an encoder of an information extraction model to obtain text information in each text box, and obtaining graph structure information of the annotated document; processing the graph structure information through a graph convolutional network of the information extraction model to obtain layout information of the annotated document; performing entity extraction on the text in each text box based on the text information and the layout information through an entity extraction network of the information extraction model; and training parameters of the information extraction model based on the results of the entity extraction and the entity labeling.
[0006] In a second aspect, an embodiment of the present application provides an information extraction method, comprising: obtaining a target document, the target document comprising at least one text box; encoding each text box in the target document through an encoder of a pre-trained information extraction model to obtain text information in each text box, and obtaining graph structure information of the target document; processing the graph structure information through a graph convolutional network of the information extraction model to obtain layout information of the target document; performing entity extraction on the text in each text box based on the text information and the layout information through an entity extraction network of the information extraction model; and summarizing the extracted entities to obtain target information.
[0007] In the third aspect, an embodiment of the present application also provides an information extraction model training device, comprising: an acquisition unit, configured to acquire an annotated document, the annotated document comprising at least one text box, and the text in each text box is labeled with an entity; an encoding unit, configured to encode each text box through an encoder of the information extraction model to obtain the text information in each text box, and obtain the graph structure information of the annotated document; a processing unit, configured to process the graph structure information through a graph convolutional network of the information extraction model to obtain the layout information of the annotated document; an extraction unit, configured to perform entity extraction on the text in each text box according to the text information and the layout information through an entity extraction network of the information extraction model; and a training unit, configured to train the parameters of the information extraction model according to the result of the entity extraction and the entity labeling.
[0008] In a fourth aspect, an embodiment of the present application further provides an information extraction device, comprising: an acquisition unit, configured to acquire a target document, the target document comprising at least one text box; an encoding unit, configured to encode each text box in the target document through an encoder of a pre-trained information extraction model, obtain text information in each text box, and obtain graph structure information of the target document; a processing unit, configured to process the graph structure information through a graph convolutional network of the information extraction model, and obtain layout information of the target document; an extraction unit, configured to perform entity extraction on the text in each text box according to the text information and the layout information through an entity extraction network of the information extraction model; and a generation unit, configured to summarize the extracted entities to obtain target information.
[0009] In a fifth aspect, an embodiment of the present application further provides an electronic device, comprising: a processor; and a memory, on which executable code is stored, and when the executable code is executed, the processor executes the information extraction model training method as described in one or more of the embodiments of the present application.
[0010] In a sixth aspect, the embodiments of the present application further provide one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes the information extraction model training method as described in one or more of the embodiments of the present application.
[0011] Compared with the prior art, the embodiments of the present application have the following advantages:
[0012] In the embodiment of the present application, the information extraction model including the encoder, graph convolution network and entity detection network is trained through the annotated document set, so as to extract information from the document using the information extraction model, which can realize automatic information extraction without manually setting the extraction rules, thus greatly reducing the labor cost. At the same time, since the information extraction model includes the encoder and graph convolution network, the text information and the layout of the text box in the document, font and other visual information can be fully utilized in the model learning process, which improves the utilization rate of the information in the document in the model learning process. Therefore, the use of this information extraction model can improve the accuracy of the information extraction results. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:
[0014] Figure 1 This is a schematic diagram of the information extraction scenario of this application;
[0015] Figure 2 A flowchart of an embodiment of the information extraction model training method of the present application;
[0016] Figure 3 This is a schematic diagram of the structure of the information extraction model of this application;
[0017] Figure 4 A flowchart of another embodiment of the information extraction model training method of the present application;
[0018] Figure 5 A schematic diagram of adjacent text boxes in an unlabeled document of this application;
[0019] Figure 6 is a flow chart of an embodiment of the information extraction method of the present application;
[0020] Figure 7 It is a structural schematic diagram of an embodiment of the information extraction model training device of the present application;
[0021] Figure 8 is a schematic diagram of the structure of an embodiment of the information extraction device of the present application;
[0022] Fig. 9 It is a schematic diagram of the structure of a device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0023] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.
[0024] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0025] The information extraction method of the present application can be applied to digital-born information extraction scenarios, etc. Among them, digital documents may include visually rich documents (Visually Rich Documents), and visually rich documents are usually PDF (Portable Document Format) documents, which refer to documents with visual information such as pictures, fonts, and layouts in addition to text information. Documents rich in visual information may specifically include but are not limited to resume documents, bill documents, etc. A digital document may include multiple text boxes, each of which may contain text. Open source tools (such as PDF Miner tools) can usually be used to pre-divide the text boxes of digital documents so that the text therein is distributed in the first or multiple text boxes.
[0026] Figure 1 This is a schematic diagram of an information extraction scenario of the present application, which scenario can specifically be a resume information extraction scenario. The system architecture in this scenario may include terminal devices 101, 102, a network 103 and a server 104. The network 103 is used to provide a medium for a communication link between the terminal devices 101, 102 and the server 104. The network 103 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0027] The terminal devices 101 and 102 can be various electronic devices with display screens and supporting network communications, including but not limited to smart phones, tablet computers, e-book readers, laptop computers, desktop computers, wearable devices, etc., which are not specifically limited here.
[0028] The server 104 can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.
[0029] Users can use terminal devices 101 and 102 to interact with server 104 through network 103 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101 and 102, such as document editing applications, web browser applications, instant messaging tools, etc.
[0030] A pre-trained information extraction model may be deployed in the server 104, and the information extraction model can extract key information from the resume document. The terminal devices 101 and 102 can send a request to the server 104, and carry the resume documents that need to be extracted in the request, such as resume document A and resume document B. The server 104 can use the information extraction model running on it to extract information from the received resume document A and resume document B, and obtain the key information in the resume document A and resume document B, thereby returning the information extraction result containing the key information in the resume document A to the terminal device 101, and returning the information extraction result containing the key information in the resume document B to the terminal device 102.
[0031] It should be noted that the information extraction method provided in the embodiment of the present application is generally executed by the server 104 , and accordingly, the information extraction device is generally arranged in the server 104 .
[0032] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0033] It should be noted that the information extraction scenario is not limited to the resume information extraction scenario for resume documents, but can also be the bill information extraction scenario for bill documents, etc., which will not be described one by one here.
[0034] Continue to refer Figure 2 , showing a flowchart of an embodiment of the information extraction model training method of the present application.
[0035] The process of the information extraction model training method includes the following steps:
[0036] Step 201, obtaining annotated documents.
[0037] In this embodiment, the execution subject of the information extraction model training method (such as an electronic device such as a server) can obtain the labeled documents in a variety of ways. For example, the execution subject can obtain the labeled documents from another server (such as a database server) for storing samples through a wired connection or a wireless connection. In another example, samples can be collected on the Internet and labeled based on information such as the labels of the samples to obtain labeled documents. It should be noted that the above-mentioned wireless connection methods may include but are not limited to 3G / 4G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra wideband) connections, and other wireless connection methods currently known or to be developed in the future.
[0038] The annotated document may be a digital document that has been physically annotated, such as a document in PDF format with rich visual information, which may specifically include but is not limited to resume documents, bill documents, etc. The annotated document may include text information, as well as visual information such as pictures, fonts, and layouts.
[0039] The annotated document may include at least one text box. The text boxes in each annotated document may be pre-determined using an open source tool (such as a PDF Miner tool). Each text box may contain text. The text may be in various languages, such as Chinese text, English text, etc. The language type of the text is not limited here.
[0040] The text in each text box of the annotated document can be an entity annotation. The entity annotation here can specifically be a sequence annotation of the text in the text box, such as the BIO annotation. Among them, B indicates the beginning of the entity, I indicates the middle of the entity, and O indicates not an entity. For example, for the text "Invoice Number: 123456" in a text box, the text contains the word sequence "invoice number", ":", "123456". If "123456" is an entity, then "invoice number" can be marked as O, ":" can be marked as O, and "123456" can be marked as B.
[0041] Step 202, encode each text box through the encoder of the information extraction model to obtain the text information in each text box and obtain the graph structure information of the annotated document.
[0042] In this embodiment, a pre-built information extraction model may be deployed in the above-mentioned execution entity. The information extraction model may include an encoder, a graph convolutional network (GCN), and an entity detection network. Among them, the encoder can be connected to the graph convolutional network and the entity detection network respectively, and the graph convolutional network can also be connected to the entity detection network. The encoder, graph convolutional network, and entity detection network can all be networks that have not yet been trained or networks that have been pre-trained, and are not specifically limited here.
[0043] In this embodiment, the encoder can encode the words in the text to obtain a vector representation of the words in the text. Thus, the above-mentioned execution subject can encode each text box in the annotated document through the encoder to obtain the text information in each text box. The text information in each text box may include the encoding of each word in the text box, and the encoding can be represented in the form of a vector, which can also be called an encoding vector. In practice, the text in each text box in the annotated document can be taken as a word sequence and input into the encoder to obtain the encoding output by the encoder for each word in each word sequence.
[0044] In practice, a pre-trained language model can be directly used as an encoder, or a model obtained by retraining (fine-tuning) the pre-trained language model can be used as an encoder, or the encoder part of a trained encoder-decoder model can be directly used, etc., and is not limited to the above examples.
[0045] In some examples, the pre-trained language model may include but is not limited to a BERT (Bidirectional Encoder Representations from Transformer) model, an improved BERT model, etc. The BERT model may include but is not limited to a BERT-base model including a 12-layer transformer structure, a BERT-large model including a 24-layer transformer structure, etc. The improved BERT model may include but is not limited to a RoBERTa (A Robustly Optimized BERT Pre-training Approach) model.
[0046] It should be noted that the BERT model is an open source language model applied to natural language processing tasks. The BERT model has a multi-layer transformer structure. The transformer structure is a network structure based on the attention mechanism. This network structure can calculate the relationship between each word in a text and all the words in the text, and calculate the importance (weight) of each word based on the relationship between words, so as to obtain a new expression (such as a new vector) for each word. The above new expression not only involves the characteristics of the word itself, but also involves the relationship between other words and this word, so it is more global than the traditional word vector. Since the BERT model uses a multi-layer transformer structure (specifically the encoder in the transformer structure), the BERT model can learn the contextual relationship between words in the text. In addition, the transformer encoder uses a one-time reading method to read text, which supports the BERT model's two-way learning of text. Therefore, compared with language models that only support one-way learning of text, the BERT model can more accurately learn the contextual relationship between each word in the text, and has a deeper understanding of the context than a one-way language model, so it can accurately process various difficult-to-understand texts. Therefore, the BERT model and the improved BERT model have better task processing effects than other models for processing natural language processing tasks.
[0047] In addition, during the pre-training process of the BERT model, two unsupervised training tasks, MLM (Masked Language Model) and NSP (Next Sentence Prediction), are used. The MLM task randomly masks some words in the sequence and lets the model learn to restore these masked words. The NSP task is a 2-classification task used to determine whether one sentence appears after the other. The MLM and NSP tasks enable the BERT model to have better natural language processing performance, making it perform very well in various natural language processing tasks. Therefore, using the BERT model and the improved BERT model as a pre-training language model has better task processing effects.
[0048] In this embodiment, the above-mentioned execution subject can also obtain the graph structure information of the annotated document. Among them, the graph structure (graph) is a data format used to represent a network with a certain relationship. The graph structure may include nodes and edges between nodes. The edges between nodes are used to characterize the association relationship between nodes. The graph structure information is information that characterizes the graph structure, and specifically may include node information used to characterize nodes and edge information used to characterize the association relationship between nodes.
[0049] Here, the annotated document can be regarded as a text box network, each text box in the annotated document can be regarded as a node, and the relationship between text boxes can be regarded as the edge between nodes, so as to convert the annotated document into a graph structure, denoted as G(V, E), and obtain the graph structure information. Where V is the set of nodes v in the graph. If there are N text boxes in the document, there are N nodes in the graph structure. At this time, V = {v1, v2, v3, ..., v N}. E is the set of edges between nodes, where the edges are undirected. E∈M×V×V. M is the number of edge types.
[0050] In some implementations, for a text box (i.e., a node) in a marked document, its text information can be used to represent the text box itself, and the text information can be used as the node information of the node. In addition, the text information of the text box can be fused with other information, and the fused information can be used to represent the text box itself, and the fused information can be used as the node information of the node. In addition, other methods can be used to represent the text box, not limited to the above examples.
[0051] In some implementations, an adjacency matrix may be used to represent edge information. The adjacency matrix, which may also be referred to as a graph matrix, is a matrix used to represent the relationship between nodes. If the number of nodes is N, the adjacency matrix may be an n×n symmetric matrix. If two nodes are related, the values of the corresponding elements of the two nodes in the adjacency matrix may be set to 1; if two nodes are related, the values of the corresponding elements of the two nodes in the adjacency matrix may be set to 0. For example, text box A and text box B correspond to nodes v1 and v2, respectively. If text box A is related to text box B, it can be considered that v1 and v2 have an association relationship, and the elements of the first row and second column and the second row and first column of the adjacency matrix can be set to 1. Conversely, if text box A is not related to text box B, it can be considered that v1 and v2 have no association relationship, and the elements of the first row and second column and the second row and first column of the adjacency matrix can be set to 0. Thus, an adjacency matrix can be obtained. It should be noted that the criteria for judging whether two nodes are related can be pre-set as needed, and are not limited here. In addition, the values in the adjacency matrix may also be set to other values, not limited to 0 and 1 in the above example.
[0052] By obtaining graph structure information, the characteristics of the association relationship between text boxes can be obtained based on the text information, thereby improving the utilization rate of information in the document and making the document more accurately represented.
[0053] In some optional implementations of this embodiment, the execution subject can obtain the text information in each text box through the following steps: first, add a preset flag symbol, such as [CLS], before the first character of the text in each text box. Then, encode each text with the flag symbol added by the encoder of the information extraction model to obtain the global text information (i.e., global features) corresponding to the flag symbol in each text, and obtain the local text information (i.e., the encoding of each word) corresponding to each word in each text. Thus, text information containing global text information and local text information is obtained. Among them, the encoder can encode each item (including symbols and words) in the input text.
[0054] It should be noted that an identifier (such as [SEP]) can be added to the end of each word sequence to indicate the end. In addition, the word sequence corresponding to each text box can be set to a fixed length. If a word sequence is shorter than the length, the sequence can be padded. For example, the symbol [PAD] can be used for padding, which is convenient for model processing. The encoder can also encode the above identifiers and symbols at the same time to obtain local text information corresponding to the above identifiers and codes respectively.
[0055] In some optional implementations of this embodiment, the above-mentioned execution subject may also obtain the graph structure information of the annotated document in combination with the font information, as shown in the following sub-steps S11 to S13:
[0056] Sub-step S11, obtaining font information of each text box.
[0057] Since the visual information in digital documents is usually rich, the text in different areas usually contains different font features. Therefore, in order to improve the utilization of information, the text box can be characterized by combining font features. Font information can be information that characterizes font features, such as being represented in the form of a coding vector. The above-mentioned font features may include but are not limited to fonts, font sizes, etc.
[0058] In some examples, the font (such as Songti, Kaiti, Heiti, etc.) and font size of the text in each text box may be determined first. Then, the combination of the font and font size of the text in each text box may be regarded as a font type. After that, different font types are encoded and the encoding is used as font information, thereby obtaining the font information of each text box.
[0059] For example, the annotated document contains 5 text boxes, namely A, B, C, D, and E. The 5 text boxes involve 3 font types in total. Among them, text boxes A and B involve font type a, text boxes C and D involve font type b, and text box E involves font type c. Therefore, text boxes A and B have the same font information, text boxes C and D have the same font information, and text box E has separate font information.
[0060] In practical applications, for a text box i, the font type of the text box can be recorded as f i , the font type code is recorded as e(f i ). Among them, e() represents a word vector search method, which can obtain f by random initialization. i The word vector (i.e., encoding) of , and the word vector is used as the font information of text box i.
[0061] Sub-step S12, determining the adjacency relationship between text boxes in the annotated document.
[0062] For the annotated document, the execution subject can determine the adjacency relationship between each two text boxes therein, that is, determine whether the two text boxes are adjacent. The principle for determining whether the text boxes are adjacent can be preset as needed.
[0063] In some examples, each two text boxes in the annotated document may first be grouped into a bigram. If the bigram satisfies the first preset condition or the second preset condition, the adjacency relationship between the two text boxes in the bigram may be determined to be adjacent. If the bigram does not satisfy the first preset condition and does not satisfy the second preset condition, the adjacency relationship between the two text boxes is determined to be non-adjacent.
[0064] The first preset condition may include: the upper boundary coordinates and the lower boundary coordinates of the two text boxes are the same, and there is no other text box between the two text boxes. The second preset condition may include: the left boundary coordinates and the right boundary coordinates of the two text boxes are the same, and there is no other text box between the two text boxes.
[0065] For example, if text box A and text box B have the same left boundary coordinates and the same right boundary coordinates, and there is no other text box between text box A and text box B, the two can be considered adjacent. If text box C and text box D have the same upper boundary coordinates and the same lower boundary coordinates, and there is no other text box between text box C and text box D, the two can be considered adjacent.
[0066] Sub-step S13, obtaining graph structure information of the annotated document based on the font information and adjacency relationship.
[0067] In the first step, the execution entity may treat each text box as a node of a graph structure, and combine the font information of each node with its text information to obtain the node information of the node.
[0068] For example, both text information and node information can be represented by vector coding, so the text information and font information of each text box can be concatenated to obtain the node information of each node.
[0069] For another example, the text information of each text box may include global text information and local text information. The global text information of each text box may be concatenated with the font information to obtain the node information of each node, thereby reducing the amount of data. Specifically, for text box i, the font type of the text box is recorded as f i The font type is encoded as e(f i ), the node information of the text box is recorded as Then the node information can be expressed as:
[0070]
[0071] Among them, || represents the splicing operation, E i0 Represents the global text information of text box i.
[0072] In the second step, the adjacency relationship between the text boxes can be used as the association relationship between the nodes. Based on the adjacency relationship between the text boxes, the edge information of the edges between the nodes can be determined, such as establishing an adjacency matrix. For example, text box A and text box B correspond to nodes v1 and v2 respectively. If text box A is adjacent to text box B, it can be considered that v1 and v2 have an association relationship. At this time, the elements of the first row and second column and the second row and first column of the adjacency matrix can be set to 1. Conversely, if text box A is not adjacent to text box B, it can be considered that v1 and v2 have no association relationship. At this time, the elements of the first row and second column and the second row and first column of the adjacency matrix can be set to 0. In this way, the adjacency matrix can be obtained.
[0073] The third step is to obtain the graph structure information of the annotated documents based on the node information and edge information.
[0074] For example, the node information and edge information can be directly summarized to obtain the graph structure information.
[0075] Since the visual information in digital documents is very rich, if the model training relies only on text information, it will not be able to learn the font information, association relationship and other visual information of the text box. By obtaining the graph structure information, multiple factors such as font information, text information and association relationship can be integrated to improve the utilization rate of the information in the document, so that the annotated document can be represented more accurately.
[0076] Step 203: Process the graph structure information through the graph convolutional network of the information extraction model to obtain the layout information of the above-mentioned annotated document.
[0077] In this embodiment, the above-mentioned execution entity can input the graph structure information into the graph convolutional network of the information extraction model to obtain the layout information of the annotated document.
[0078] The graph structure information contains node information used to characterize each node (i.e., each text box in the annotated document), such as text information, or text information combined with other information. At the same time, it also contains edge information used to characterize node relationships, such as an adjacency matrix. After the graph structure information is input into the graph convolutional network, each layer of the graph convolutional network can use edge information (such as an adjacency matrix) to process the node information of each node input from the previous layer to this layer, thereby outputting new node information for each node. The node information output by the last layer of the graph convolutional network is the layout information that integrates the relationship information between nodes.
[0079] Specifically, the first layer of the graph convolutional network will convolve the node information of each node (i.e., text box) and the node information adjacent to the node. The other network layers of the graph convolutional network can further model more complex node relationships, and the last layer outputs the final node information of each node (i.e., text box). The node information output by the last layer integrates the association relationship between nodes, so the node information can reflect the layout of the text box in the document, which can be called layout information.
[0080] For node i (i.e. text box i), the node information of text box i output by the l+1th layer of the graph convolutional network for:
[0081]
[0082] Where N(i) represents all nodes connected to node i, including i itself. N is the size of the set N(i). is the feature of node i in layer l, that is, the representation of node i in layer l (i.e., node information). eLU represents the activation unit. W l , b l are the parameters of the lth layer of the graph convolutional network respectively.
[0083] Since the visual information in documents in different fields varies greatly, the layout of text boxes in different documents varies greatly. For example, the content in bill documents is usually presented in the form of a list, and the content in resume documents is usually presented in the form of chapters. If we only rely on the position coordinates of the text box to represent the layout features of the text in the text box, we cannot fully learn the unique layout features of the text box. Through the graph convolutional network, we can learn the unique structural features of the text boxes in a certain type of document, which can greatly improve the utilization of the information in the document.
[0084] Step 204: extract entities from the text in each text box according to the text information and layout information through the entity extraction network of the information extraction model.
[0085] In this embodiment, the execution entity may input the text information and layout information of each text box into the entity extraction network to obtain the result of the entity extraction network.
[0086] The entity detection network can be a multi-classification network, which is used to classify each word in the text box. The output types can include three categories, namely the beginning of the entity (which can be recorded as B), the middle of the entity (which can be recorded as I), and not an entity (which can be recorded as O). Based on the classification results output by the entity extraction network, the entities in the text box can be determined. For example, for the text "Invoice Number: 123456" in a text box, for the word sequence "Invoice Number", ":", "123456" corresponding to the text, the entity detection result can output "O, O, B-123456". The output result indicates that "123456" is an entity, and the remaining words are not entities.
[0087] In some optional implementations of this embodiment, the text information of the text box may include global text information (i.e., global features corresponding to the flag symbol) and local text information (i.e., the encoding of each word in the text). For each text box, the above-mentioned execution entity may first splice the layout information of the text box with the local text information of each word in the text box to obtain comprehensive information of each word in the text box. Then, the entity extraction network of the information extraction model is used to extract entities from the text in the text box based on the comprehensive information.
[0088] Specifically, for text box i, the layout information can be expressed as G i The local text information of each word in the text box can be represented as E i1 , E i2 , E i3 , and so on. G i With E i1 , E i2 , E i3Combine all the comprehensive information of text box i into a sequence and input it into the entity detection network to obtain the result of entity extraction.
[0089] Step 205: training the parameters of the information extraction model according to the entity extraction results and entity annotations.
[0090] In this embodiment, the above-mentioned execution entity can calculate the loss value based on the result of entity extraction and the entity annotation of each text box. The loss value can be used to characterize the difference between the result of entity extraction output by the entity detection network and the entity annotation. The larger the loss value, the greater the difference. The loss value can be calculated using a commonly used loss function, such as a cross entropy loss function. After obtaining the loss value corresponding to the selected annotated document, the parameters of the information extraction model (including the encoder, graph convolutional network, and entity detection network) can be updated based on the loss value.
[0091] In practice, a large number of annotated documents can be used to perform the training steps from step 201 to step 205. Each time the above process is performed, the parameters of the updated information extraction model (including the encoder, graph convolution network and entity detection network) can be trained once, so that the parameters of the information extraction model (including the encoder, graph convolution network and entity detection network) are updated once. The training can be iterated multiple times until the training is completed.
[0092] In practice, whether training is completed can be determined in a variety of ways. As an example, the accuracy of the output results of the entity detection model can be determined. If the accuracy is greater than a preset threshold (e.g., 98%), the training can be determined to be completed. As another example, if the number of iterative training times is equal to a preset number, the training can be determined to be completed.
[0093] Combine the following Figure 3 The structural diagram of the information extraction model shown in Figure 1 introduces the process of training the model using a labeled document and the input-output relationship of each component in the model. Figure 3 As shown in the figure, the model can use the BERT model as an encoder and include a graph convolutional network (GCN) and an output layer. The output layer is the entity detection network, which can be implemented by a multi-classification network.
[0094] The above-mentioned annotated document includes text boxes i and j, but is not limited to these two text boxes. The text in text box i is "Invoice Number:". The word sequence corresponding to text box i is "Invoice", "Number", ":".
[0095] When the annotated document is used for model training, the text in each text box can be preprocessed in advance, such as adding the [CLS] identifier before the first character, adding the [SEP] identifier after the last character, and completing the word sequence with the [PAD] identifier, etc., to obtain an input sequence, which is "[CLS]", "Invoice", "Number", ":", "[SEP]", "[PAD]", "[PAD]".
[0096] Then, after the input sequence is input into the encoder (i.e., the BERT model), the encoder can output vectors E corresponding to “[CLS]”, “Invoice”, “Number”, “:”, “[SEP]”, “[PAD]”, and “[PAD]” respectively. i0 、E i1 、E i2 、E i3 、E i4 、E i5 、E i6 Among them, E i0 is the global text information in text box i (i.e., the code corresponding to [CLS], which can be used as a global feature). i1 、E i2 、E i3 、E i4 、E i5 、E i6 are the local text information (i.e., the codes corresponding to each item) of other items (including words and logos) in text box i. Similarly, similar operations are performed on text box j and other text boxes.
[0097] After that, the font information F of the text in text box i can be obtained through the font and font size of text box i. i . F i With E i0 After splicing, the node information corresponding to text box i is obtained. Similarly, after performing similar operations on text box j and other text boxes, the node information corresponding to each text box can be obtained. In addition, an adjacency matrix (i.e., edge information) can be obtained based on the relationship between the text boxes in the annotated document (such as adjacency relationship). The adjacency matrix and the node information can constitute the graph structure information corresponding to the annotated document.
[0098] After that, the concatenated input of each text box can be fed into GCN, which can process it layer by layer and output the layout information of each text box through the last layer. The layout information of text box i can be recorded as G i . G i Can also be a vector.
[0099] Afterwards, the layout information G of text box i can be iRespectively with E i1 、E i2 、E i3 、E i4 、E i5 、E i6 Splice to get 6 spliced vectors. Input these 6 spliced vectors into the output layer (i.e., entity detection network) to get the entity detection results. Similarly, perform similar operations on text box j and other text boxes.
[0100] Finally, a loss value can be obtained based on the entity detection result of each text box and the entity label of the text box. The total loss value can be obtained by adding the loss values of each text box. Based on the total loss value, the gradient descent algorithm and the back propagation algorithm can be used to update the parameters of each component in the model to achieve a model training.
[0101] The above is the process of training a model using a labeled document in the labeled document set. By training the model using a large number of labeled documents, an information extraction model with excellent information extraction performance can be obtained.
[0102] The method provided by the above-mentioned embodiment of the present application trains an information extraction model including an encoder, a graph convolutional network, and an entity detection network through a set of annotated documents, thereby extracting information from the document using the information extraction model, which can realize automatic information extraction without the need to manually set extraction rules, greatly reducing labor costs. At the same time, since the information extraction model includes an encoder and a graph convolutional network, the text information and the layout of the text box in the document, font and other visual information can be fully utilized in the model learning process, thereby improving the utilization rate of the information in the document in the model learning process. Therefore, the use of this information extraction model can improve the accuracy of the information extraction results.
[0103] Further references Figure 4 , which shows a flowchart of another embodiment of the information extraction model training method.
[0104] The process of the information extraction model training method includes the following steps:
[0105] Step 401, initializing the encoder based on the parameters of the pre-trained language model.
[0106] In this embodiment, the execution subject (such as an electronic device such as a server) of the information extraction model training method can be deployed with a pre-built untrained encoder. The above execution subject can initialize the encoder based on the parameters of the pre-trained language model so that it has the parameters of the pre-trained language model.
[0107] In some examples, the BERT model can be used as a pre-trained language model. The BERT model is not limited to the BERT-base model with a 12-layer transformer structure and the BERT-large model with a 24-layer transformer structure. In the pre-training stage, the BERT model uses two unsupervised training tasks, MLM and NSP. Among them, the MLM task randomly masks some words in the sequence and lets the model learn to restore these masked words. The NSP task is a binary classification task used to determine whether one of the two sentences appears after the other. The MLM task and the NSP task can make the BERT model have better performance. Since the BERT model performs very well in various natural language processing tasks, the BERT model can be used as a pre-trained language model.
[0108] In other examples, an improved BERT model may be used as a pre-trained language model. The improved BERT model may include but is not limited to models such as RoBERTa. The improved BERT model is further optimized based on the BERT model, so it has better performance, and thus the improved BERT model may also be used as a pre-trained language model.
[0109] Step 402: Obtain unlabeled documents.
[0110] In this embodiment, the execution subject may obtain the unlabeled documents in a variety of ways. For example, the existing unlabeled documents stored in another server (such as a database server) for storing samples may be obtained through a wired connection or a wireless connection. For another example, unlabeled samples may be collected on the Internet.
[0111] Unlabeled documents can be digital documents that are not physically labeled, such as PDF documents with rich visual information, including but not limited to resume documents, bill documents, etc. Unlabeled documents can include text information, as well as visual information such as pictures, fonts, and layouts.
[0112] The unlabeled document may include at least one text box. The text boxes in the unlabeled document may be pre-determined using an open source tool (such as a PDF Miner tool). Each text box may contain text. The text may be in various languages, such as Chinese text, English text, etc. The language type of the text is not limited here. The text in each text box does not have entity annotations. The unlabeled document and the labeled document may be documents in the same field.
[0113] Step 403: train the encoder parameters according to the unlabeled documents.
[0114] In this embodiment, unlabeled documents can be used to perform unsupervised training tasks on the pre-trained language model. Since the acquisition cost of labeled documents is very high in information extraction tasks, but unlabeled documents are usually very rich, these rich unlabeled documents can be fully utilized to enhance the performance of the encoder. Compared with directly performing supervised training on the encoder, adding an unsupervised training link before supervised training can improve the accuracy of the encoding output by the encoder.
[0115] In some optional implementations of this embodiment, when using unlabeled documents to train the encoder, the MLM training task in the BERT model task set can be performed on the encoder to update the parameters of the encoder.
[0116] In some optional implementations of this embodiment, a SPRC (Sequences Positional Relationship Classification) training task may also be constructed, and the parameters of the encoder may be updated by executing the training task on the encoder. Specifically, the following sub-steps may be performed:
[0117] Sub-step S21, selecting adjacent text boxes in the unlabeled document.
[0118] Here, adjacent text boxes can be predefined as needed. As an example, Figure 5 FIG. 1 shows a schematic diagram of adjacent text boxes in an unlabeled document. The unlabeled document here may specifically be a bill document. Figure 5 As shown, if two text boxes have the same left boundary coordinates and the same right boundary coordinates, and there is no other text box between the two text boxes, the two text boxes can be considered to be adjacent text boxes; or, if two text boxes have the same upper boundary coordinates and the same lower boundary coordinates, and there is no other text box between the two text boxes, the two text boxes can be considered to be adjacent text boxes.
[0119] Optionally, every two text boxes in the unlabeled document can first be grouped into a tuple. If the tuple satisfies the first preset condition or the second preset condition, the adjacency relationship of the two text boxes in the tuple can be determined to be adjacent. If the tuple does not satisfy the first preset condition and does not satisfy the second preset condition, the adjacency relationship of the two text boxes is determined to be non-adjacent. Among them, the first preset condition may include: the upper boundary coordinates of the two text boxes are the same and the lower boundary coordinates are the same, and there are no other text boxes between the two text boxes. The second preset condition may include: the left boundary coordinates of the two text boxes are the same and the right boundary coordinates are the same, and there are no other text boxes between the two text boxes.
[0120] Sub-step S22, encoding the adjacent text boxes through an encoder to obtain text information of the adjacent text boxes.
[0121] Here, the execution subject may input the texts in the adjacent text boxes into the encoder at the same time to obtain the text information of the adjacent text boxes.
[0122] In some examples, the texts in adjacent text boxes can be connected by a preset segmentation symbol (such as [SEP]) to obtain a connected text. Then, a preset flag symbol (such as [CLS]) is added before the first character of the connected text. Afterwards, the connected text to which the flag symbol is added is encoded by an encoder to obtain an encoding of the flag symbol, which can be used as the global text information of the adjacent text box (which can be regarded as a global feature), and obtains local text information corresponding to other items in the connected text (i.e., the encoding of other items). The above-mentioned execution entity can use the global text information as the text information of the adjacent text box and execute the following sub-step S23. The global text information and the local text information can also be summarized as the text information of the adjacent text box, and the following sub-step S23 is executed.
[0123] Sub-step S23, based on the text information of the adjacent text boxes, predict the positional relationship type of the adjacent text boxes through a preset classification network to obtain a prediction result.
[0124] Here, the execution entity may input the text information of the adjacent text boxes into a preset classification network, and predict the positional relationship type of the adjacent text boxes through the classification network to obtain a prediction result. The classification network may be any network capable of realizing a multi-classification function, for example, a neural network composed of one or more fully connected layers. The positional relationship types may include "left-right", "right-left", "up-down" and "down-up".
[0125] Optionally, the global text information of adjacent text boxes (ie, the codes corresponding to the flag symbols) may be input into a preset classification network to obtain a prediction result of the positional relationship type of the adjacent text boxes to reduce the amount of data calculation.
[0126] Sub-step S24, training the parameters of the encoder based on the prediction results and the actual position relationship type of the adjacent text boxes.
[0127] Here, the actual position relationship type of each group of adjacent text boxes can be pre-stored. The above execution body can determine the loss value based on the prediction result and the actual position relationship type of the adjacent text boxes. Then update the parameters of the encoder based on the loss value.
[0128] By adjusting the encoder parameters through the above-mentioned SPRC training task, the encoder can fully learn the contextual relationship of the text in the unlabeled documents, make full use of a large number of unlabeled documents, and improve the accuracy of the encoder in text encoding.
[0129] In some optional implementations of this embodiment, the above-mentioned MLM training task and the above-mentioned SPRC training task may be performed on the pre-trained language model to obtain an encoder. In addition, the pre-trained language model may be retrained using other training methods as needed, which are not limited to the above examples.
[0130] It should be noted that the execution subject may use multiple unlabeled documents to iteratively execute the step of training the encoder parameters. When the performance of the encoder reaches the expected index or the number of training times reaches the preset number, the training is considered to be completed.
[0131] Step 404: Obtain annotated documents.
[0132] In this embodiment, the annotated document may include at least one text box, and the text in each text box may be annotated with entities. The annotated documents in the annotated document set and the unannotated documents in the unannotated document set are documents in the same field. For example, they may all be bill documents or resume documents.
[0133] Step 404 in this embodiment can be found in Figure 2 The step 201 of the corresponding embodiment will not be described in detail here.
[0134] Step 405 , encode each text box through the encoder of the information extraction model to obtain the text information in each text box and obtain the graph structure information of the annotated document.
[0135] Step 405 in this embodiment can be found in Figure 2 The step 202 of the corresponding embodiment will not be described in detail here.
[0136] Step 406: Process the graph structure information through the graph convolutional network of the information extraction model to obtain the layout information of the annotated document.
[0137] Step 406 in this embodiment can be found in Figure 2 The step 203 of the corresponding embodiment will not be described in detail here.
[0138] Step 407: extract entities from the text in each text box according to the text information and layout information through the entity extraction network of the information extraction model.
[0139] Step 407 in this embodiment can be found in Figure 2 The step 204 of the corresponding embodiment will not be described in detail here.
[0140] Step 408: training parameters of the information extraction model based on the entity extraction results and entity annotations.
[0141] Step 408 in this embodiment can be found in Figure 2The step 205 of the corresponding embodiment will not be described in detail here.
[0142] from Figure 4 It can be seen that Figure 2 Compared with the corresponding embodiment, the process of the information extraction model training method in this embodiment highlights the step of training the encoder with an unlabeled document set before using the labeled documents to train the information extraction model. Therefore, the scheme described in this embodiment can make full use of these rich unlabeled documents to enhance the performance of the encoder, and can improve the accuracy of the encoding output by the encoder compared to directly conducting supervised training on the encoder.
[0143] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.
[0144] Continue to refer Figure 6 , shows a flow chart of an embodiment of the information extraction method of the present application. The process of the information extraction method includes the following steps:
[0145] Step 601, obtaining a target document.
[0146] In this embodiment, the execution subject of the information extraction method (such as an electronic device such as a server) can obtain a target document to be extracted, such as a resume document, a bill document, etc. The target document may include at least one text box, each of which contains text. In practice, the text boxes in the target document can be determined by an open source tool (such as a PDF Miner tool).
[0147] Step 602, encode each text box in the target document through the encoder of the pre-trained information extraction model to obtain the text information in each text box and obtain the graph structure information of the target document.
[0148] In this embodiment, the above-mentioned execution entity can input the text in each text box into the encoder in the pre-trained information extraction model to obtain the text information in each text box. In addition, the graph structure information of the marked document can also be obtained. Here, the information extraction model can be trained using the methods described in the above embodiments. The specific training process can be found in the relevant description of the above embodiments, which will not be repeated here. The steps of obtaining the text information and graph structure information of the target text are basically the same as the steps of obtaining the text information and graph structure information of the marked text in the above embodiments, which will not be repeated here.
[0149] Step 603: Process the graph structure information through the graph convolutional network of the information extraction model to obtain the layout information of the target document.
[0150] In this embodiment, the execution subject can input the graph structure information into the graph convolution network of the information extraction model to obtain the layout information of the target document. The graph convolution network and the steps of obtaining the layout information of the target document can be found in the description of the above embodiment and will not be repeated here.
[0151] Step 604: extract entities from the text in each text box according to the text information and layout information through the entity extraction network of the information extraction model.
[0152] In this embodiment, the execution subject can splice the layout information of each text box and the text information and input them into the entity detection network in the information extraction model. The entity detection network can output the type of each word in each text box, such as entity start, entity middle, and not entity. The execution subject can determine the entity in each text box based on the output result of the information extraction model. The operation of the entity detection network and entity extraction can refer to the description in the above embodiment, which will not be repeated here.
[0153] Step 605, summarize the extracted entities to obtain target information.
[0154] In this embodiment, the execution entity may aggregate the entities to obtain target information in the target document.
[0155] The method provided by the above-mentioned embodiment of the present application can realize automatic information extraction by extracting information from documents using an information extraction model including an encoder, a graph convolutional network, and an entity detection network, without the need to manually set extraction rules, thereby greatly reducing labor costs. At the same time, since the information extraction model includes an encoder and a graph convolutional network, the text information and the layout of the text box in the document, font and other visual information can be fully utilized during the model learning process, thereby improving the utilization rate of the information in the document during the model learning process. Therefore, the use of this information extraction model can improve the accuracy of the information extraction results.
[0156] Further references Figure 7 Based on the above embodiments, the present application provides an embodiment of an information extraction model training device, which can be specifically applied to various electronic devices.
[0157] like Figure 7As shown, the information extraction model training device 700 of this embodiment includes: an acquisition unit 701, configured to acquire an annotated document, the annotated document includes at least one text box, and the text in each text box is labeled with an entity; an encoding unit 702, configured to encode each of the text boxes through an encoder of the information extraction model to obtain the text information in each of the text boxes, and obtain the graph structure information of the annotated document; a processing unit 703, configured to process the graph structure information through a graph convolutional network of the information extraction model to obtain the layout information of the annotated document; an extraction unit 704, configured to perform entity extraction on the text in each of the text boxes according to the text information and the layout information through an entity extraction network of the information extraction model; a training unit 705, configured to train the parameters of the information extraction model according to the result of the entity extraction and the entity labeling.
[0158] In some optional implementations of this embodiment, the encoding unit 702 is further configured to: obtain font information of each text box; determine the adjacency relationship between the text boxes in the annotated document; and obtain graph structure information of the annotated document based on the font information and the adjacency relationship.
[0159] In some optional implementations of the present embodiment, the encoding unit 702 is further configured to: determine the font and font size of the text in each of the text boxes; take the combination of the font and font size of the text in each of the text boxes as a font type, encode different font types, and obtain the font information of each of the text boxes.
[0160] In some optional implementations of the present embodiment, the encoding unit 702 is further configured to: group every two text boxes in the annotated document into a tuple; if the tuple satisfies the first preset condition or the second preset condition, determine that the adjacency relationship between the two text boxes in the tuple is adjacent; if the tuple does not satisfy the first preset condition and does not satisfy the second preset condition, determine that the adjacency relationship between the two text boxes is non-adjacent; wherein the first preset condition includes: the upper boundary coordinates of the two text boxes are the same and the lower boundary coordinates are the same, and there are no other text boxes between the two text boxes; the second preset condition includes: the left boundary coordinates of the two text boxes are the same and the right boundary coordinates are the same, and there are no other text boxes between the two text boxes.
[0161] In some optional implementations of the present embodiment, the encoding unit 702 is further configured to: take each of the text boxes as a node of the graph structure, respectively concatenate the text information and font information of each of the text boxes to obtain the node information of each node; based on the adjacency relationship, determine the edge information of the edges between the nodes; based on the node information and the edge information, obtain the graph structure information of the annotated document.
[0162] In some optional implementations of this embodiment, the text information of each of the above text boxes includes global text information and local text information; and the above encoding unit 702 is further configured to: respectively splice the global text information of each of the above text boxes with the font information to obtain the node information of each of the above nodes.
[0163] In some optional implementations of the present embodiment, the encoder is obtained through the following steps: initializing the encoder based on the parameters of a pre-trained language model; obtaining an unlabeled document, wherein the unlabeled document includes at least one text box and the text in each text box does not have an entity annotation, and the unlabeled document and the unlabeled document are documents in the same field; and training the parameters of the encoder based on the unlabeled document.
[0164] In some optional implementations of this embodiment, the above-mentioned pre-trained language model includes any of the following: a transformer-based bidirectional encoder representation BERT model, an improved BERT model.
[0165] In some optional implementations of this embodiment, the parameters of the encoder are trained based on the unlabeled documents, including: using the unlabeled documents to perform the language model mask training task in the BERT model task set on the pre-trained language model to obtain the encoder.
[0166] In some optional implementations of the present embodiment, the above-mentioned training of the parameters of the encoder based on the above-mentioned unlabeled document includes: selecting adjacent text boxes in the above-mentioned unlabeled document; encoding the above-mentioned adjacent text boxes through the above-mentioned encoder to obtain text information of the above-mentioned adjacent text boxes; based on the text information of the above-mentioned adjacent text boxes, predicting the positional relationship type of the above-mentioned adjacent text boxes through a preset classification network to obtain a prediction result; based on the above-mentioned prediction result and the actual positional relationship type of the above-mentioned adjacent text boxes, training the parameters of the encoder.
[0167] In some optional implementations of the present embodiment, the text information of the above-mentioned adjacent text boxes includes global text information and local text information; and the above-mentioned encoding of the above-mentioned adjacent text boxes by the above-mentioned encoder to obtain the text information of the above-mentioned adjacent text boxes includes: connecting the texts in the above-mentioned adjacent text boxes by a preset segmentation symbol to obtain a connected text; adding a preset flag symbol before the first character of the above-mentioned connected text; encoding the above-mentioned connected text to which the above-mentioned flag symbol is added by the above-mentioned encoder to obtain the global text information corresponding to the above-mentioned flag symbol, and obtaining the local text information corresponding to other items in the above-mentioned connected text.
[0168] In some optional implementations of the present embodiment, based on the text information of the above-mentioned adjacent text boxes, the positional relationship types of the above-mentioned adjacent text boxes are predicted through a preset classification network to obtain prediction results, including: inputting the global text information of the above-mentioned adjacent text boxes into a preset classification network to obtain the prediction results of the positional relationship types of the above-mentioned adjacent text boxes.
[0169] In some optional implementations of the present embodiment, the above-mentioned text information includes global text information and local text information; and the above-mentioned encoding unit 702 is further configured to: add a preset flag symbol before the first character of the text in each of the above-mentioned text boxes; encode each text with the above-mentioned flag symbol added through the encoder of the above-mentioned information extraction model to obtain the global text information corresponding to the flag symbol in each of the above-mentioned texts, and obtain the local text information corresponding to each word in each of the above-mentioned texts.
[0170] In some optional implementations of the present embodiment, the extraction unit 704 is configured to: for each text box, concatenate the layout information of the text box with the local text information of each word in the text box to obtain comprehensive information of each word in the text box; and perform entity extraction on the text in the text box according to the comprehensive information through the entity extraction network of the information extraction model.
[0171] In some optional implementations of this embodiment, the above-mentioned annotated document includes at least one of the following: a bill document, a resume document.
[0172] The device provided by the above-mentioned embodiment of the present application trains the information extraction model including the encoder, graph convolution network and entity detection network through the annotated document set, thereby extracting information from the document using the information extraction model, and can realize automatic information extraction without manually setting the extraction rules, which greatly reduces the labor cost. At the same time, since the information extraction model includes the encoder and the graph convolution network, the text information and the layout of the text box in the document, the font and other visual information can be fully utilized in the model learning process, which improves the utilization rate of the information in the document in the model learning process. Therefore, the use of this information extraction model can improve the accuracy of the information extraction results.
[0173] Further references Figure 8 Based on the above embodiments, the present application provides an embodiment of an information extraction device, which can be specifically applied to various electronic devices.
[0174] like Figure 8 As shown, the information extraction device 800 of this embodiment includes: an acquisition unit 801, configured to acquire a target document, wherein the target document includes at least one text box; an encoding unit 802, configured to encode each text box in the target document through an encoder of a pre-trained information extraction model, obtain text information in each text box, and obtain graph structure information of the target document; a processing unit 803, configured to process the graph structure information through a graph convolutional network of the information extraction model, and obtain layout information of the target document; an extraction unit 804, configured to perform entity extraction on the text in each text box according to the text information and the layout information through an entity extraction network of the information extraction model; and a generation unit 805, configured to summarize the extracted entities to obtain target information.
[0175] In some optional implementations of this embodiment, the target document includes at least one of the following: a bill document, a resume document.
[0176] The device provided by the above-mentioned embodiment of the present application can realize automatic information extraction by extracting information from documents using an information extraction model including an encoder, a graph convolutional network, and an entity detection network, without the need to manually set extraction rules, thereby greatly reducing labor costs. At the same time, since the information extraction model includes an encoder and a graph convolutional network, the text information and the layout of the text box in the document, font and other visual information can be fully utilized during the model learning process, thereby improving the utilization rate of the information in the document during the model learning process. Therefore, the use of this information extraction model can improve the accuracy of the information extraction results.
[0177] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.
[0178] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and servers (clusters).
[0179] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as terminal devices and servers (clusters). Fig. 9 An exemplary apparatus 900 that can be used to implement various embodiments described in this application is schematically illustrated.
[0180] For one embodiment, Fig. 9 An exemplary apparatus 900 is shown having one or more processors 902, a control module (chip set) 904 coupled to at least one of the processor(s) 902, a memory 906 coupled to the control module 904, a non-volatile memory (NVM) / storage device 908 coupled to the control module 904, one or more input / output devices 910 coupled to the control module 904, and a network interface 912 coupled to the control module 904.
[0181] The processor 902 may include one or more single-core or multi-core processors, and the processor 902 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 900 can be used as a terminal device, server (cluster), etc. described in the embodiments of the present application.
[0182] In some embodiments, the apparatus 900 may include one or more computer-readable media (e.g., memory 906 or NVM / storage device 908) having instructions 914 and one or more processors 902 combined with the one or more computer-readable media and configured to execute the instructions 914 to implement a module to perform the actions described in the present disclosure.
[0183] For one embodiment, the control module 904 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 902 and / or any suitable device or component in communication with the control module 904 .
[0184] The control module 904 may include a memory controller module to provide an interface to the memory 906. The memory controller module may be a hardware module, a software module, and / or a firmware module.
[0185] The memory 906 may be used, for example, to load and store data and / or instructions 914 for the device 900. For one embodiment, the memory 906 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 906 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).
[0186] For one embodiment, control module 904 may include one or more input / output controllers to provide an interface to NVM / storage device 908 and input / output device(s) 910 .
[0187] For example, NVM / storage 908 may be used to store data and / or instructions 914. NVM / storage 908 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).
[0188] NVM / storage device 908 may include storage resources that are physically part of the device on which apparatus 900 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 908 may be accessed via (one or more) input / output devices 910 over a network.
[0189] (One or more) input / output devices 910 may provide an interface for the apparatus 900 to communicate with any other appropriate device, and the input / output device 910 may include a communication component, an audio component, a sensor component, etc. The network interface 912 may provide an interface for the apparatus 900 to communicate through one or more networks, and the apparatus 900 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.
[0190] For one embodiment, at least one of the processor(s) 902 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 904. For one embodiment, at least one of the processor(s) 902 may be packaged together with the logic of one or more controllers of the control module 904 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 902 may be integrated on the same die with the logic of one or more controllers of the control module 904. For one embodiment, at least one of the processor(s) 902 may be integrated on the same die with the logic of one or more controllers of the control module 904 to form a system-on-chip (SoC).
[0191] In various embodiments, the device 900 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the device 900 may have more or fewer components and / or a different architecture. For example, in some embodiments, the device 900 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.
[0192] Among them, the main control chip can be used as a processor or control module in the device, sensor data, location information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.
[0193] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0194] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0195] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0196] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0197] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0198] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.
[0199] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one information or operation from another information or operation, and do not necessarily require or imply any such actual relationship or order between these information or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the presence of other identical elements in the process, method, article or terminal device including the elements.
[0200] The above is a detailed introduction to the information extraction model training method, information extraction method, device and electronic device provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A method for training an information extraction model, characterized in that: The method comprises: Acquire an annotated document, wherein the annotated document includes at least one text box, and text in each text box is annotated with an entity; Encoding each text box through an encoder of an information extraction model to obtain text information in each text box and obtain graph structure information of the annotated document; Processing the graph structure information through the graph convolutional network of the information extraction model to obtain layout information of the annotated document; Performing entity extraction on the text in each text box according to the text information and the layout information through the entity extraction network of the information extraction model; Training the parameters of the information extraction model according to the entity extraction results and the entity annotations; Wherein, obtaining the graph structure information of the annotated document includes: Get the font information of each text box; Determining adjacency relationships between text boxes in the annotated document; Based on the font information and the adjacency relationship, graph structure information of the annotated document is obtained.
2. The method according to claim 1, characterized in that: The obtaining of the font information of each text box includes: Determine the font and size of the text in each text box; The combination of the font and font size of the text in each text box is regarded as a font type, and different font types are encoded to obtain the font information of each text box.
3. The method according to claim 1, characterized in that: The determining the adjacency relationship between the text boxes in the annotated document includes: Combining every two text boxes in the annotated document into a tuple; If the two-tuple satisfies the first preset condition or the second preset condition, determining that the adjacency relationship between the two text boxes in the two-tuple is adjacent; If the two-tuple does not satisfy the first preset condition and does not satisfy the second preset condition, determining that the adjacency relationship between the two text boxes is non-adjacent; Wherein, the first preset condition includes: the upper boundary coordinates and the lower boundary coordinates of the two text boxes are the same, and there is no other text box between the two text boxes; The second preset condition includes: the left boundary coordinates of the two text boxes are the same and the right boundary coordinates are the same, and there is no other text box between the two text boxes.
4. The method according to claim 1, characterized in that: The obtaining of the graph structure information of the annotated document based on the font information and the adjacency relationship includes: Taking each of the text boxes as a node of the graph structure, respectively concatenating the text information and the font information of each of the text boxes to obtain node information of each node; Based on the adjacency relationship, determining edge information of edges between nodes; Based on the node information and the edge information, graph structure information of the annotated document is obtained.
5. The method according to claim 4, characterized in that: The text information of each text box includes global text information and local text information; and, The step of respectively splicing the text information and the font information of each text box to obtain the node information of each node includes: The global text information and the font information of each text box are respectively spliced to obtain the node information of each node.
6. The method according to claim 1, characterized in that: The encoder is obtained by the following steps: Initializing the encoder based on parameters of a pre-trained language model; Acquire an unlabeled document, wherein the unlabeled document includes at least one text box and the text in each text box does not have an entity annotation, and the unlabeled document and the unlabeled document are documents in the same field; The parameters of the encoder are trained according to the unlabeled document.
7. The method according to claim 6, characterized in that: The pre-trained language model includes any of the following: a transformer-based bidirectional encoder representation BERT model, an improved BERT model.
8. The method according to claim 6, characterized in that: The step of training the parameters of the encoder according to the unlabeled document comprises: The unlabeled document is used to perform a language model mask training task in a BERT model task set on the pre-trained language model to obtain an encoder.
9. The method according to claim 6, characterized in that: The step of training the parameters of the encoder according to the unlabeled document comprises: Selecting adjacent text boxes in the unlabeled document; Encoding the adjacent text box by the encoder to obtain text information of the adjacent text box; Based on the text information of the adjacent text boxes, predicting the positional relationship type of the adjacent text boxes through a preset classification network to obtain a prediction result; Based on the prediction result and the actual position relationship type of the adjacent text boxes, the parameters of the encoder are trained.
10. The method according to claim 9, characterized in that: The text information of the adjacent text box includes global text information and local text information; and, The encoding of the adjacent text box by the encoder to obtain text information of the adjacent text box includes: Connecting the texts in the adjacent text boxes by using a preset segmentation symbol to obtain connected text; Adding a preset flag symbol before the first character of the connection text; The encoder encodes the connection text to which the flag symbol is added, obtains global text information corresponding to the flag symbol, and obtains local text information corresponding to other items in the connection text.
11. The method according to claim 10, characterized in that The predicting of the positional relationship type of the adjacent text boxes based on the text information of the adjacent text boxes by a preset classification network to obtain a prediction result includes: The global text information of the adjacent text boxes is input into a preset classification network to obtain a prediction result of the positional relationship type of the adjacent text boxes.
12. The method according to claim 1, characterized in that The text information includes global text information and local text information; as well as, The encoder of the information extraction model encodes each text box to obtain text information in each text box, including: Adding a preset flag symbol before the first character of the text in each text box; Each text to which the flag symbol is added is encoded by the encoder of the information extraction model to obtain global text information corresponding to the flag symbol in each text, and local text information corresponding to each word in each text is obtained.
13. The method according to claim 12, characterized in that The entity extraction network of the information extraction model extracts entities from the text in each text box according to the text information and the layout information, including: For each text box, the layout information of the text box is spliced with the local text information of each word in the text box to obtain comprehensive information of each word in the text box; and entity extraction is performed on the text in the text box according to the comprehensive information through the entity extraction network of the information extraction model.
14. The method according to claim 1, characterized in that The annotated document includes at least one of the following: a bill document and a resume document.
15. An information extraction method, characterized in that: The method comprises: Acquire a target document, wherein the target document includes at least one text box; Encoding each text box in the target document through an encoder of a pre-trained information extraction model to obtain text information in each text box and obtain graph structure information of the target document; Processing the graph structure information through the graph convolutional network of the information extraction model to obtain layout information of the target document; Performing entity extraction on the text in each text box according to the text information and the layout information through the entity extraction network of the information extraction model; Summarize the extracted entities to obtain the target information; Wherein, obtaining the graph structure information of the target document includes: Get the font information of each text box; Determining adjacency relationships between text boxes in the target document; Based on the font information and the adjacency relationship, graph structure information of the target document is obtained.
16. The method according to claim 15, characterized in that The target document includes at least one of the following: a bill document and a resume document.
17. An information extraction model training device, characterized in that: The device comprises: An acquisition unit is configured to acquire a marked document, wherein the marked document includes at least one text box, and the text in each text box is marked with an entity; An encoding unit, configured to encode each text box through an encoder of an information extraction model to obtain text information in each text box and obtain graph structure information of the annotated document; A processing unit, configured to process the graph structure information through a graph convolutional network of the information extraction model to obtain layout information of the annotated document; An extraction unit, configured to perform entity extraction on the text in each text box according to the text information and the layout information through an entity extraction network of the information extraction model; A training unit, configured to train parameters of the information extraction model according to the entity extraction result and the entity annotation; Wherein, the encoding unit is further used for: Get the font information of each text box; Determining adjacency relationships between text boxes in the annotated document; Based on the font information and the adjacency relationship, graph structure information of the annotated document is obtained.
18. An information extraction device, characterized in that: The device comprises: An acquisition unit, configured to acquire a target document, wherein the target document includes at least one text box; An encoding unit, configured to encode each text box in the target document through an encoder of a pre-trained information extraction model, obtain text information in each text box, and obtain graph structure information of the target document; A processing unit, configured to process the graph structure information through a graph convolutional network of the information extraction model to obtain layout information of the target document; An extraction unit, configured to perform entity extraction on the text in each text box according to the text information and the layout information through an entity extraction network of the information extraction model; A generation unit is configured to summarize the extracted entities to obtain target information; Wherein, the encoding unit is further used for: Get the font information of each text box; Determining adjacency relationships between text boxes in the target document; Based on the font information and the adjacency relationship, graph structure information of the target document is obtained.
19. An electronic device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed, cause the processor to perform the method according to one or more of claims 1-16.
20. One or more machine-readable media having executable codes stored thereon, which, when executed, cause a processor to perform the method of one or more of claims 1-16.
Citation Information
Patent Citations
Method and system for recognizing certificates based on graph neural network
CN112016438A
