Business document information extraction method based on variant attention and hierarchy

Abstract

The application relates to a business document information extraction method based on variant attention and hierarchy, and belongs to the field of natural language processing. First, the document is classified, the data is processed into supervised training data required by a model, and is input into a variant attention and hierarchy network model for training, and a hierarchical network structure is used to extract features of word-level, sentence-level and paragraph-level document information; the document required to be extracted and the label of the content of the document required to be extracted are input into the trained hierarchical variant attention network model for information extraction; and finally, the extracted information is statistically analyzed. The application realizes accurate business document information extraction technology, and finally, the document information is extracted through the technology, and then the document information is statistically analyzed to realize special business related information extraction.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of natural language processing, and particularly relates to a business document information extraction method based on variant attention and hierarchical structure. BACKGROUND

[0002] How to extract and refine key information from a large amount of unstructured data is the core of information extraction. The current information extraction of documents adopts a template matching method, which needs to first develop relevant templates, and then match the document content according to the templates. This method needs a large amount of manpower to manually customize templates, and the efficiency is relatively low.

[0003] The application adopts a neural network method for document information extraction. For the neural network method, the current extraction technology is mainly divided into: extractive summary and generative summary. The information extraction technology has the following problems: 1. Redundancy - the extracted sentences are not refined enough, the key words are repeated, and it is difficult to determine their importance in the context; 2. Unnecessary, there are sentences that are not enough to express the core content; 3. Harshness, the extracted core information is directly spliced, although each word is a core key word, but combined together it is meaningless, and has no readability. In order to solve this drawback, a hierarchical mode semantic information feature extraction structure is proposed, which is associated through word level, sentence level and paragraph level, so that the neural network can calculate the correlation weight between each token in the document; and the AdaptiveAttention structure of the variant Attention is introduced, which expands the context correlation distance of the token without increasing the calculation resources, so as to realize accurate business document information extraction. The known technology involved in the application includes the following:

[0004] a) Hierarchical structure: first extract the features of each token at the word level, then combine them together to form the sentence level features, and then combine the sentence level features into the paragraph level features. The hierarchical structure constructed in this way calculates the correlation weight between each token, sentence and segment through the neural network, so that the relevant information in the document can be extracted more accurately.

[0005] b) Attention (attention mechanism): the traditional neural network framework is Encoder-semantic encoding-Decoder, such as Figure 1As shown, the text information is mapped to a high-dimensional space by the Encoder, the features are extracted, the extracted features are saved in the semantic information, and then the corresponding information is obtained by decoding the Decoder. For shorter texts, this can be used, but the defect is that the length of the intermediate semantic encoding vector is fixed, and once the sentence is too long, some information will be lost, so it is impossible to pay attention to the semantic information of the whole text. The principle of Attention mechanism is that there are multiple semantic information, such as Figure 2 As shown, each time the Encoder is performed, the information is saved in the semantic information, and different weights are assigned to each encoding information, so that longer text sentences can be processed, and the associated distance between tokens is longer.

[0006] c) Multi-Head Attention (Multi-Head Attention): The core idea of Multi-Head Attention is to use multiple Attention, and the weight coefficients of each Attention are different, so the focus points of the same sentence are also different. By associating multiple Attention, the function of wider attention range can be realized.

[0007] d) Adaptive Attention (Adaptive Attention): The purpose of Adaptive Attention is to obtain a longer distance correlation weight. For Attention, the correlation distance is indeed improved, but this distance is generally only 2000 tokens, and a longer distance cannot be obtained. Adaptive Attention observes that, through the multi-Head mechanism in Transformer, the starting attention mechanism is not the same as the focus point. For example, HeadA focuses on the last few dozen tokens, and the weight is low for the previous tokens. HeadB focuses on all tokens, so Adaptive Attention uses a Mask function, as shown in equation (1), to calculate the Attention weight, using the distance between the text and the current position as input, and using a learnable parameter Z to assign weights to each Attention, as shown in equation (2), so as to realize the attention to each Attention, that is, the weight is larger. In this way, it can avoid focusing on small values of attention weight, and allocate computing resources, so as to realize the expansion of context correlation distance with unchanged computing resources.

[0008]

[0009] Mask function: map the distance between context and token to [0,1]

[0010] Z: parameter to learn, for each head, Z is different R: hyper-parameter, manually set, used for smoothing

[0011] X: distance of text context from current token

[0012]

[0013] Attention function: when calculating attention weight, use the distance of context and current position as the input of Mask function, and z is the parameter to learn in the calculation process, for each head, z is different.

[0014] t: current token position

[0015] t-r: relative position

[0016] S: attention span

[0017] q=t-S: distance from current token under different attention span. SUMMARY

[0018] (I) Technical problems to be solved

[0019] The technical problem to be solved by the present application is how to provide a business document information extraction method based on variant attention and hierarchical structure, to solve the problem that the current information extraction of documents uses a template matching method, which needs to first develop relevant templates, and then matches the document content according to the templates. This requires a large amount of manpower to manually customize templates, and the efficiency is low.

[0020] (ii) Technical solution

[0021] In order to solve the above technical problems, the present application provides a business document information extraction method based on variant attention and hierarchical structure, which comprises the following steps:

[0022] Step 1: training, first classify a large number of documents, process the classified documents into supervised training format, input the processed data into a hierarchical variant attention neural network model for training, and extract features of word-level, sentence-level and paragraph-level text respectively;

[0023] Step 2: prediction, input the document to be extracted as input data into the trained hierarchical variant attention neural network model, and input the information label to be extracted as input, so that the trained model can extract information and find the most relevant words, sentences or paragraphs in the document.

[0024] Step 3: statistics, statistical analysis of the extracted information.

[0025] Further, the step one of classifying a large number of documents includes: receiving document training data, the data being a large number of document data, first classifying the received data through document names, and then preprocessing the document to be extracted to obtain candidate words corresponding to the text to be extracted, and inputting the document to be extracted and each candidate word into a neural network for training.

[0026] Further, the hierarchical variant attention neural network model in step one includes a word-level module, a sentence-level module, a paragraph-level module, and a softmax layer module.

[0027] Further, the word-level module includes:

[0028] Embedding: the text data is converted into a word vector Vector through an Embedding layer;

[0029] Word Encoder: the word vector received by the Embedding layer is encoded through a bidirectional LSTM to obtain the bidirectional semantic features of each word as a feature vector;

[0030] Word Adaptive Attention: the feature vector is input into a variant attention model, the context relationship of a word is connected through a variant attention mechanism, and the distance between associated words is increased through Adaptive Attention to calculate the inter-word relationship weight of each word.

[0031] Further, the sentence-level module includes:

[0032] Sentence Encoder: the feature vector of each word is connected through matrix addition to construct the word-level features into sentence-level features S1, S2, …, Sn, n being the number of sentences in the document, and the semantic information of each sentence is obtained through a bidirectional LSTM network model, i.e., a word vector is spliced to obtain a sentence-level feature vector;

[0033] Sentence Adaptive Attention: The context relationship of a sentence is connected through the attention mechanism, and the connection between the sentence-level feature vectors is obtained through the Adaptive Attention mechanism, and the relationship weight of each sentence is calculated.

[0034] Further, the paragraph-level module comprises:

[0035] Segment Encoder: The sentence-level relationship is constructed through the Sentence Encoder and the Sentence Adaptive Attention to obtain the sentence-level feature vector. The sentence-level feature vectors of each sentence in each paragraph are obtained through matrix splicing to obtain the segment-level features P1, P2, …, Pm of each paragraph, m being the number of segments. The semantic information of each paragraph is obtained as a segment-level feature vector by using bidirectional LSTM to encode the segment-level features again.

[0036] Segment Adaptive Attention: The connection between the segment-level feature vectors is obtained through the Adaptive Attention mechanism, and the relationship weight of each paragraph is calculated.

[0037] Further, the softmax layer module is used to classify the relationship weight between words, sentences, and paragraphs through the softmax layer. First, the paragraph with the highest probability is found, then the sentence with the highest probability in the paragraph is found, and finally the word with the highest probability in the sentence is found. The sentence is calculated from the word level, the sentence level, and the paragraph level, and then the probability is calculated through the softmax normalization. Through the above calculation, the most relevant paragraph in the full text can be matched, the sentence with the highest probability in the paragraph is found, and finally the most relevant token, i.e., the most relevant word, is calculated.

[0038] Further, the softmax layer module also adds a constraint rule through the conditional random field CRF. The input is the probability of each word predicting each label, and the output is each possible annotation sequence, and the sequence with the highest score is selected as the final result.

[0039] Further, in the conditional random field CRF processing process, the invalid prediction sequence is removed after removing the B-per error label combination.

[0040] Further, the step two specifically comprises: receiving a single or multiple documents to be processed, predicting and outputting the extraction result; first inputting the document to be extracted, and then inputting the keyword label to be extracted, and the label is formatted into a json template, that is, the same label is extracted from all documents, and the extracted information is output in accordance with the special business requirements.

[0041] (III) Beneficial Effects

[0042] The present application provides a business document information extraction method based on variant attention and hierarchical structure, which can expand the context correlation degree of information extraction under the premise of keeping the performance unchanged. The technology is applied to the information extraction of documents, replaces inefficient manual review, and accurately identifies the key information required to be extracted in the document. Compared with the prior art, the improvement of the technology is that the context correlation is performed from three levels: word level, sentence level and paragraph level. And the Adaptive-Attention structure is added, so that the correlation distance of the context is greatly increased without affecting the model performance. The technology has the characteristics of high accuracy, high availability, high speed and multi-scene application of equipment. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 The basic network framework is provided;

[0044] Figure 2 The Attention mechanism is provided;

[0045] Figure 3 The hierarchical variant Attention neural network module architecture is provided;

[0046] Figure 4 The system flowchart of the present application is provided. DETAILED DESCRIPTION

[0047] In order to make the purpose, content and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in combination with the drawings and examples.

[0048] The technical problem to be solved by the present application is: based on the neural network framework, the network structure is optimized, and the original template matching document information extraction technology is replaced, such as Figure 3 as shown.

[0049] The application extracts features from the words level, combines into the sentences level, continues to extract features from the sentence level to obtain the final segments level features, and adds Adaptive-Attention between the levels to keep the performance unchanged in the case of context correlation distance expansion, and finally uses a softmax layer for normalization to output the paragraph with the maximum probability. Through the method, the context can be maximally associated, and the key information in the business document can be extracted without affecting the performance of the model.

[0050] The application relates to a business document information extraction method based on variant attention and hierarchical structure, which comprises the following steps: 1, training, first, classifying documents, processing data into supervised training data required by a model, inputting the data into a variant attention and hierarchical structure network model for training, and respectively extracting features of word-level, sentence-level and paragraph-level document information by using a hierarchical network structure; 2, prediction, taking a document to be extracted and a label of content of the document to be extracted as input, feeding the input into the trained hierarchical variant attention network model to perform information extraction; and 3, statistics, finally, performing statistical analysis on the extracted information. The final output layer of the model is a SoftMax layer, the output of the layer is a probability value of a predicted sentence, therefore, a sentence with the highest probability value is obtained as the output of the model, so that the accurate business document information extraction technology is realized, and finally, the document information is extracted by the technology, and statistical analysis is performed on the document information, so that special business related information extraction is realized.

[0051] The technical scheme of the application is as follows: a business document information extraction method based on variant attention and hierarchical structure, the method comprising the following steps:

[0052] The step one: training, first, classifying a large number of documents, and dividing document types into categories required by businesses. The classified documents are processed into a supervised training format, data is input into a hierarchical variant attention neural network model for training, and features of word-level, sentence-level and paragraph-level texts are respectively extracted. Figure 3 As shown in the figure.

[0053] The step two: prediction, taking a document to be extracted as input data and feeding the input data into the trained hierarchical variant attention neural network model, and simultaneously taking an information label to be extracted as input; and letting the trained model perform information extraction.

[0054] The step three: statistics, performing statistical analysis on the extracted information.

[0055] The design module and implementation steps of the application are as follows:

[0056] The present application is divided into three processes, which are: 1. training; 2. prediction, 3. statistics.

[0057] 1 training

[0058] The training process uses a hierarchical variant attention neural network model to train the document data. The modules included in this process are:

[0059] 1.1 training input module

[0060] Receive document training data, which is a large amount of document data. The purpose is to first classify the received data, and classify the input large amount of text data by document name. Then pre-process the document to be extracted keywords to obtain the candidate words corresponding to the text to be extracted keywords. The document to be extracted keywords and each candidate word are input into the neural network for training.

[0061] 1.2 hierarchical variant attention network training module

[0062] Receive the pre-processed document data, use the hierarchical variant attention neural network model proposed in this paper to train the data, learn the hidden vectors between tokens, extract the features of keywords, and realize the recognition and extraction of core information.

[0063] 1.2.1 word level module:

[0064] (1) Embedding: First, the text data is converted into a word vector Vector by the Embedding layer.

[0065] (2) Word Encoder: The word vector received by the encoding layer from the Embedding layer is encoded by the bidirectional LSTM to obtain the bidirectional semantic features of each word as the feature vector.

[0066] (1.3) WordAdaptiveAttention: Input the feature vector into the variant attention model, connect the context relationship of a word through the variant attention mechanism, and increase the distance between the associated words through AdaptiveAttention. Calculate the inter-word relationship weight of each word, even if it exceeds 8000 words (the context association distance of the normal Attention mechanism is about 512 tokens, and it is difficult to associate between tokens when the distance exceeds this distance). The relationship between words can be connected without consuming computing power.

[0067] 1.2.2 sentence level module:

[0068] (1) Sentence Encoder: The feature vector of each word is obtained through word-level Embedding, Word Encoder and WordAdaptiveAttention. The feature vectors of each word in each sentence are connected by matrix addition, so as to construct the sentence-level features S1, S2, …, Sn of each word in each sentence, n being the number of sentences in the document. The semantic information of each sentence is obtained by encoding through a bidirectional LSTM network model, i.e. by concatenating the word vectors to obtain the sentence-level feature vector.

[0069] (2) SentenceAdaptiveAttention: The context relationship of a sentence is connected through the attention mechanism, and the relationship between the sentence-level feature vectors is obtained through the AdaptiveAttention attention mechanism, and the relationship weight of each sentence is calculated.

[0070] 1.2.3 Paragraph-level module:

[0071] (1) Segment Encoder: The sentence-level relationship is constructed by Sentence Encoder and SentenceAdaptiveAttention to obtain the sentence-level feature vector. The sentence-level feature vectors of each sentence in each paragraph are also connected by matrix concatenation to obtain the paragraph-level features P1, P2, …, Pm of each paragraph, m being the number of paragraphs. The semantic information of each paragraph is obtained by encoding the paragraph-level features using a bidirectional LSTM again as a paragraph-level feature vector.

[0072] (2) SegmentAdaptiveAttention: The relationship between the paragraph-level feature vectors is obtained through the AdaptiveAttention attention mechanism, and the relationship weight of each paragraph is calculated.

[0073] At this point, through the three-layer coding and concatenation of word-level, sentence-level and paragraph-level, and through the attention of the three-layer variant attention mechanism (WordAdaptiveAttention, SentenceAdaptiveAttention, SegmentAdaptiveAttention), the relationship between each word and other words, each sentence and other sentences, each paragraph and other paragraphs, and the relationship between words, sentences and paragraphs can be effectively learned. The relationship is represented by weight, and the higher the weight, the closer the relationship. When performing information extraction, the keyword is first selected from the paragraph with higher weight, then from the sentence with higher weight, and finally from the word with higher weight. Thus, more accurate information extraction is achieved. In order to make the results more intuitive, the weight needs to be converted into probability.

[0074] (3.1) softmax layer module: through the above calculation, the relationship weight between words, sentences and paragraphs can be obtained, and classification is performed through the softmax layer. First, the paragraph with the highest probability is found, then the sentence with the highest probability in the paragraph is found, and finally the word with the highest probability in the sentence is found. The sentence is calculated from the word level, the sentence level and the paragraph level, and then normalized by softmax to calculate the probability. If you need to find "some information", you can match the paragraph related to "some information" in the full text through the above calculation, find the sentence with the highest correlation to "some information" in the paragraph, and finally calculate the token most relevant to "some information", i.e. the most associated word. Finally, CRF (Conditional Random Field) is added to add constraint rules. The input is the probability of each word predicting each label, and the output is each possible annotation sequence. The highest scoring sequence is selected as the final result, and the B-per and other obvious error label combination problems are removed. Invalid prediction sequences are removed.

[0075] 2 prediction

[0076] The prediction process uses the trained hierarchical variant attention neural network model to extract information from the required processed documents.

[0077] 2.1 Prediction input module

[0078] Receive a single or multiple documents that need to be processed for information extraction. The prediction input module does not require a large amount of data, unlike the training input module.

[0079] 2.2 Prediction output module

[0080] The extracted results are output after processing the received documents. First, input the document that needs to be extracted, and then input the keyword label that needs to be extracted. The label here can also be formatted into a json template, i.e. extract the same label for all documents, i.e. output the extracted information according to the special business requirements. This can avoid the need to manually input information every time.

[0081] 3 Statistics

[0082] Finally, the extracted information is statistically analyzed.

[0083] The application provides an information extraction technology capable of expanding context correlation under the premise of keeping performance unchanged, which is applied to information extraction of documents, replaces inefficient manual review, and accurately identifies key information required to be extracted in the documents. Compared with the prior art, the improvement of the technology is that context correlation is carried out from three levels, respectively, that is, word level, sentence level and paragraph level. And the Adaptive-Attention structure is added, so that the model performance is not affected under the premise that the context correlation distance is greatly increased. The technology has the characteristics of high accuracy, high availability, high speed and improved multi-scene applicability of equipment.

[0084] The above only describes the preferred embodiments of the present application, and it should be noted that those skilled in the art can make several improvements and modifications without departing from the technical principles of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.

Claims

1. A method for business document information extraction based on variant attention and hierarchy, characterized in that, The method comprises the following steps: Step one: training, first, a large number of documents are classified and processed, the classified documents are processed into a supervised training format, and the processed data is input into a hierarchical variant attention neural network model for training, and features of word level, sentence level and paragraph level text are extracted respectively; Step two: prediction, the document to be extracted is input into the trained hierarchical variant attention neural network model as input data, and the information label to be extracted is input as input, so that the trained model performs information extraction to find the most relevant words, sentences or paragraphs in the document; Step three: statistics, statistical analysis is performed on the extracted information; Wherein, The hierarchical variant attention neural network model in step one comprises a word level module, a sentence level module, a paragraph level module and a softmax layer module; The word level module comprises: Embedding: the text data is converted into a word vector Vector through the Embedding layer; Word Encoder: the word vector transmitted from the Embedding layer is encoded by the bidirectional LSTM to obtain the bidirectional semantic features of each word as a feature vector; WordAdaptiveAttention: the feature vector is input into the variant attention model, the context relationship of a word is connected through the variant attention mechanism, and the distance between the associated words is increased through the AdaptiveAttention, and the inter-word relationship weight of each word is calculated; The sentence level module comprises: Sentence Encoder: the feature vector of each word is connected by matrix addition in each sentence to construct the word level features into sentence level features S1, S2, …, Sn, n is the number of sentences in the document, and the semantic information of each sentence is obtained by encoding through the bidirectional LSTM network, that is, the word vector is spliced to obtain the sentence level feature vector; SentenceAdaptiveAttention: the context relationship of a sentence is connected through the attention mechanism, and the connection between the sentence level feature vectors is obtained through the AdaptiveAttention attention mechanism, and the relationship weight of each sentence is calculated; The paragraph level module comprises: Segment Encoder: the sentence-level feature vectors are obtained by constructing the sentence-level relationship through the Sentence Encoder and the SentenceAdaptiveAttention, the sentence-level feature vectors of each sentence in each segment are also obtained by matrix splicing to obtain the segment-level features P1, P2, …, Pm of each paragraph, m is the number of segments; the semantic information of each paragraph is obtained by using the bidirectional LSTM to encode the segment-level features again as the segment-level feature vectors; SegmentAdaptiveAttention: the relationship between the segment-level feature vectors is obtained by the AdaptiveAttention attention mechanism, and the relationship weight of each sentence is calculated; The softmax layer module is used to classify the relationship weight between words, sentences and segments through the softmax layer, first find the paragraph with the highest probability, then find the sentence with the highest probability in the paragraph, and finally find the most relevant token, i.e. the most relevant word, in the sentence; the probability is calculated by normalizing the calculation from the word level, sentence level and segment level, and through the above calculation, the most relevant paragraph in the full text can be matched, the sentence with the highest probability in the paragraph is found, and finally the most relevant token, i.e. the most relevant word, is calculated; The softmax layer module also adds constraint rules through conditional random field CRF, the input is the probability of each label corresponding to each word, and the output is each possible annotation sequence, and the highest score sequence is selected as the final result. 2.The business document information extraction method based on variant attention and hierarchy structure according to claim 1, wherein, The classified documents in step one are processed into supervised training formats, which specifically include: receiving document training data, which is a large amount of document data, first classifying the received data by document name, and then preprocessing the documents to be extracted keywords to obtain the candidate words corresponding to the text to be extracted keywords, and inputting the documents to be extracted keywords and each candidate word into the neural network for training. 3.The business document information extraction method based on variant attention and hierarchy structure according to claim 1, wherein, In the conditional random field CRF processing process, the invalid prediction sequence is removed after removing the B-per error label combination.

4. The business document information extraction method based on variant attention and hierarchy according to any one of claims 1-3, characterized in that, The step two specifically includes: receiving a single or multiple documents that need to be processed, predicting the extraction result, and outputting the extraction result; first input the document that needs to be extracted, then input the keyword label that needs to be extracted, and format the label into a json template, that is, extract the same label from all documents, and output the extracted information according to the special business requirements.

Images