A general information extraction method

Abstract

The present invention discloses a general information extraction method. The present invention reformulates the information extraction task into span extraction and span-span relationship extraction, thereby unifying the information extraction task into a target box detection task on a two-dimensional matrix. The present invention proposes a lightweight attention module, which combines the advantages of the Softmax attention mechanism and the linear attention mechanism, and reduces the space and time complexity from O(L 2 ) is reduced to O(L 1.5 ), while achieving better performance. In addition, the present invention designs and uses a lightweight transformer model that integrates a lightweight attention module with a convolutional neural network to capture long-range and local dependencies in a two-dimensional feature matrix. A comprehensive evaluation of six information extraction tasks on eight datasets shows that the proposed method achieves very advanced performance. In addition, the present invention also exhibits excellent knowledge transfer capabilities and has significant efficiency.

Description

Technical Field

[0001] The present invention relates to an information extraction method in the field of information extraction, and specifically relates to a universal information extraction method that can uniformly and efficiently process different information extraction tasks in a single lightweight architecture. Background Art

[0002] Information extraction refers to the process of automatically extracting structured data from unstructured or semi-structured machine-readable documents. Information extraction includes various subtasks, the main subtasks include named entity recognition, relation extraction, and event extraction. Current methods in the field of information extraction are usually task-independent, resulting in resource consumption and hindering knowledge sharing between tasks.

[0003] For named entity recognition, many traditional named entity recognition methods model it as a sequence labeling task. In order to solve the problem of nested entities, some methods introduce stacking or cascading labeling strategies. One popular method is the span-based method, which completes the named entity recognition by completing the classification task of potential spans; another popular method is the generative method based on the pre-trained language model. At the same time, there are also some innovative methods that use reading comprehension models, prompt learning or diffusion models to solve the named entity recognition problem.

[0004] For relation extraction, pipeline methods and joint methods are two main methods. The pipeline method first uses the entity recognition model to mark the entities in the sentence, and then uses the relation extraction model to predict the relationship between the entities. Although this method is flexible, it faces the problem of error propagation, that is, the prediction error of entity recognition will affect the relation extraction in the next step. In addition, this method lacks interaction between subtasks and ignores the connection and dependency between subtasks. Recent studies have begun to focus on the interaction modeling between subtasks and proposed several joint extraction methods. In addition, some other methods, such as sequence generation-based methods and set generation-based methods, have also been used to solve this task.

[0005] For event extraction, traditional methods define it as a sequence labeling problem or adopt pipeline-based methods. However, the former may ignore non-traditional cases such as overlapping and nested spans; while the latter may encounter problems with error propagation and lack of interaction between subtasks. Some recent studies explore the use of structured sequence generation methods or graph-based methods to solve the event extraction problem.

[0006] In addition to modeling information extraction tasks separately, building joint models for general information extraction has also attracted increasing interest. Some previous methods have formulated general information extraction as a text-to-graph generation task, while further considering interactions at the task level and instance level. In addition to graph-based methods, a class of sequence generation-based methods defines output formats for different information tasks, converting different information extraction tasks into structured sequence generation tasks. In addition, some recent methods have attempted to apply innovative models to handle general information extraction tasks, such as modeling different information extraction tasks as a semantic matching task. However, these previous general information extraction methods face challenges such as high model complexity, excessively time-consuming training or decoding, and resource consumption. Summary of the invention

[0007] The purpose of the present invention is to propose a general information extraction method that can uniformly and efficiently process different information extraction tasks in a single lightweight architecture in response to the shortcomings of the prior art. The present invention unifies the information extraction task into a target box detection task on a two-dimensional matrix by reformulating the information extraction task into span extraction and span-span relationship extraction. The core of the present invention is a lightweight attention module, which combines the advantages of the Softmax attention mechanism and the linear attention mechanism, and can achieve better performance while reducing the spatial and temporal complexity. In addition, a lightweight transformer model is designed and used in the present invention, which integrates the lightweight attention module with a convolutional neural network to capture long-range and local dependencies in a two-dimensional feature matrix. A comprehensive evaluation of six information extraction tasks on eight data sets shows that the method proposed in the present invention achieves very advanced performance. In addition, the method proposed in the present invention exhibits excellent knowledge transfer capabilities and remarkable efficiency.

[0008] The technical solution adopted by the present invention is:

[0009] 1. A general information extraction method

[0010] 1) Constructing a general information extraction model, and then using the training data set to train the general information extraction model to obtain a trained general information extraction model;

[0011] 2) Input the input sequence to be extracted into the trained general information extraction model, and the model outputs the predicted second-order probability matrix Y'', where the predicted second-order probability matrix Y'' consists of N×N cells; then use the decoder to decode the predicted second-order probability matrix Y'' to obtain the span set S and span-span relationship set R of the input sequence to be extracted, and complete the information extraction.

[0012] In 1), the training data set includes training samples of different task types, each training sample consists of an input sequence including N words and a corresponding label, and the label is a binary label matrix including span and span-span relationship.

[0013] For each input sequence containing N words, its label is specifically:

[0014] Each label is a binary label matrix containing span and span-span relationship. For the i-th span (l i ,r i ,t i s ), where l i ,r i Represent the left and right boundaries of the span, t i s Represents the span type, the i-th span (l i ,r i ,t i s ) are the four span corners of the i-th span (l i ,r i ,t i s ) corresponds to the four corners of the quadrilateral in the current binary label matrix (l i ,l i ), (l i ,r i ), (r i ,l i ), (r i ,r i ), and the i-th span (l i ,r i ,t i s )’s four span corners are marked as 1 at their corresponding positions in the binary label matrix;

[0015] For the jth span-span relation (s j h ,s j t ,t j r ), s j h Represents the head span, s j h =(l j h ,r j h ,t j h ), l j h,r j h ,t j h Represents the left boundary of the header span, the right boundary of the header span, and the type of the header span, s j t represents the tail span, s j t =(l j t ,r j t ,t j t ), l j t ,r j t ,t j t Represent the left boundary of the tail span, the right boundary of the tail span and the type of the tail span, t j r represents the type of the jth span-span relationship; the jth span-span relationship (s j h ,s j t ,t j r ) is the jth span-span relationship (s j h ,s j t ,t j r ) corresponds to the four corners of the quadrilateral in the current binary label matrix (l j h ,l j t ), (l j h ,r j t ) , (r j h ,r j t ), (r j h ,l j t ), and mark the corresponding positions of the four span-span relationship corners in the binary label matrix as 1.

[0016] In the above 1), the general information extraction model includes a connected text vector representation extraction module and a feature encoding module, the input sequence is used as the input of the text vector representation extraction module, and the feature encoding module outputs a second-order probability matrix Y' as the output of the general information extraction model;

[0017] The feature encoding module includes a dual affine model, a lightweight transformer model and a label space mapping module. The input of the feature encoding module is used as the input of the dual affine model, the output of the dual affine model is used as the input of the lightweight transformer model, the output of the lightweight transformer model is fused with the output of the dual affine model and then input into the label space mapping module. The output of the label space mapping module is a second-order probability matrix Y' and is used as the output of the feature encoding module.

[0018] The lightweight transformer model includes N sequentially connected Transformer layers, the output of the dual affine model is used as the input of the first Transformer layer, and the output of the last Transformer layer is used as the output of the lightweight transformer model; each Transformer layer includes a residual CNN module, a lightweight attention module and two layer normalization layers, the input of each Transformer layer is used as the input of the residual CNN module, the residual CNN module is connected to the lightweight attention module after the first layer normalization layer, the output of the lightweight attention module is fused with the input of the lightweight attention module and then used as the input of the second layer normalization layer, and the output of the second layer normalization layer is used as the output of each Transformer layer.

[0019] The lightweight attention module is specifically:

[0020] S1: After implementing the Softmax attention mechanism on each row in the input of the lightweight attention module, the two-dimensional feature matrix A after row processing is obtained row , and after implementing the Softmax attention mechanism on each column in the input of the lightweight attention module, the two-dimensional feature matrix A after column processing is obtained column ;

[0021] S2: After processing the input of the lightweight attention module using the linear attention layer, a first linear attention processing matrix is ​​obtained, and the input of the lightweight attention module is processed again using the linear attention layer to obtain a second linear attention processing matrix; then, the two-dimensional feature matrix A after row processing is row After adding it to the first linear attention processing matrix, we get the row feature matrix A h , the two-dimensional feature matrix A after column processing column After adding it to the second linear attention processing matrix, we get the column feature matrix A v ;

[0022] S3: Use a gated linear unit to process the input of the current lightweight attention module to obtain the gated value G;

[0023] S4: The row feature matrix A h and the column feature matrix Av The connected matrix is ​​recorded as the complete feature matrix. The complete feature matrix is ​​then element-wise multiplied with the gate value G and then multiplied with the attention weight, and the result is taken as the output A of the lightweight attention module, where the attention weight is a learnable weight.

[0024] The linear attention layer is obtained by using a kernel function instead of the Softmax in the Softmax attention mechanism.

[0025] In the label space mapping module, a linear transformation is used to map the feature vector corresponding to each cell in the result of fusing the output of the lightweight transformer model with the output of the bi-affine model to the label space to obtain a second-order probability matrix Y'.

[0026] After the decoder is used to decode the predicted second-order probability matrix Y'', the span set S and span-span relationship set R of the input sequence to be extracted are obtained, which are specifically:

[0027] First, find all t in the upper triangular matrix of the predicted second-order probability matrix Y'' i s The probability value of the dimension is greater than the span corner threshold Th s t, so as to obtain the initial span set; for each cell in the initial span set, determine the span corners of each cell corresponding to t i s Are the probability values ​​of the dimensions greater than the span corner threshold Th? s , if both are greater than the span angle threshold Th s , then the cell is a span, otherwise the cell is deleted from the initial span set; after traversing and judging all cells in the initial span set, the final span set S is obtained;

[0028] Next, for each span pair (l p ,r p ,t p s )、(l q ,r q ,t q s ), if p(l p ,l q , t j r )>Th r 、p(l p ,r q ,t j r )>Th r 、p(r p ,l q ,tj r )>Th r and p(r p ,r q ,t j r )>Th r , where p(l p ,l q , t j r )、p(l p ,r q ,t j r )、p(r p ,l q ,t j r ) and p(r p ,r q ,t j r ) are the t corresponding to the four span-span relationship corners of the current span-span relationship. j r The probability value of the dimension, Th r is the span-span relationship corner threshold, then the current span pair (l p ,r p ,t p s )、(l q ,r q ,t q s ) is a span-span relationship and is added to the span-span relationship set R. All span pairs in the span set S are traversed to obtain the final span-span relationship set R.

[0029] The text vector representation extraction module specifically uses a pre-trained language model to extract the text vector representation of the input sequence.

[0030] During the training process of the general information extraction model, the general information extraction model is trained by calculating the cross entropy loss function between the second-order probability matrix Y' output by the general information extraction model and the corresponding label Y in the training data set until the training is completed.

[0031] 2. A General Information Extraction System

[0032] A training data set construction unit, used for labeling each input sequence with a label, wherein the label is a binary label matrix including span and span-span relationship, thereby generating a training data set;

[0033] A general information extraction unit, used to train the general information extraction model using the training data set, obtain the trained general information extraction model and store it;

[0034] The information extraction output unit is used to input the input sequence to be extracted into the trained general information extraction model, and decode the predicted second-order probability matrix Y'' output by the model to obtain the span set S and the span-span relationship set R.

[0035] 3. A computer device

[0036] The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the general information extraction method when executing the computer program.

[0037] 4. A computer-readable storage medium

[0038] The storage medium stores a computer program, which implements the steps of the general information extraction method when executed by the processor.

[0039] V. A computer program product

[0040] The computer program product comprises a computer program / instruction, which implements the steps of the general information extraction method when executed by a processor.

[0041] Compared with the prior art, the method of the present invention has the following beneficial effects:

[0042] 1. The present invention divides common information extraction tasks into two subtasks: span extraction and span-span relationship extraction. Therefore, different information extraction tasks can be modeled uniformly, and entities, relationships, event trigger words, event arguments, attributes, etc. in the input can be extracted at one time.

[0043] 2. The present invention designs a lightweight framework for general information extraction. In the model part of the framework, the present invention designs a lightweight attention module, which combines the advantages of the Softmax attention mechanism and the linear attention mechanism. Experiments have shown that it reduces the spatiotemporal complexity of the Softmax attention mechanism while improving the actual effect. At the same time, the model part of the present invention also designs a lightweight transformer model, which is a Transformer-like model composed of several stacked identical Transformer layers, used to model the interaction between cell latent vectors, where each Transformer layer consists of a CNN module and a lightweight attention module. Experiments have shown that the lightweight transformer model significantly improves the performance of the model in various tasks.

[0044] 3. The present invention designs a lightweight framework for general information extraction. In the decoding part of the framework, the present invention designs a two-step decoding method, which can efficiently decode all spans and span-span relationships in the input and can handle nested entities.

[0045] 4. The method proposed in this invention has good extraction effect. It has achieved results that exceed or are comparable to the most advanced models in the industry on 6 IE tasks and 8 data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is the overall flow chart of the method of the present invention.

[0047] Figure 2 It is a structural diagram of a general information extraction model in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] The method of the present invention is applied to the ACE2005-E+ dataset, which is a commonly used information extraction dataset. The ACE2005-E+ dataset is annotated with 6 entity types, 7 relationship types, 33 event types and 22 event argument types.

[0050] In order to make a fair comparison, this embodiment uses the partitioning method of Lin et al. to maintain the same data set partitioning as previous work, where the training set contains 19,244 data items, the validation set contains 902 data items, and the test set contains 676 data items.

[0051] like Figure 1 and Figure 2 As shown, the present invention provides an efficient general information extraction method capable of uniformly processing different information extraction tasks in a single lightweight architecture, the method comprising the following steps:

[0052] 1) constructing a general information extraction model, and then training the general information extraction model with a training data set to obtain a trained general information extraction model; wherein the training data set includes training samples of different task types, each training sample consists of an input sequence containing N words and a corresponding label, and the label is a binary label matrix containing a span and a span-span relationship.

[0053] The purpose of label construction is to decompose different information extraction tasks into two subtasks: span extraction and span-span relation extraction, respectively obtaining a span set and a span-span relation set, and unifying the information extraction task into a target box detection task on a two-dimensional matrix. The label is a binary label matrix Y∈R N×N×T , T is the label length, which represents the total number of spans + the total number of span-span relationships. N×N×T is a set of real numbers of dimension N×N×T. Among them, the span represents a continuous character segment in a text, which can be formalized as a triple of the form (left boundary, right boundary, span type), and the span-span relationship represents the relationship between two spans, which can be formalized as a triple of the form (head span, tail span, span relationship). For example, in named entity recognition, the span includes the entity; in attribute extraction, the span includes the attribute; in relation extraction, the span includes the entity, and the span-span relationship includes the relationship; in event extraction, the span includes the event trigger word and the event argument, and the span-span relationship includes the argument role, which is essentially the relationship between the event trigger word and the argument. For each input sequence containing N words (i.e., input sentence), its label is specifically:

[0054] Each label is a binary label matrix containing span and span-span relationship. For the i-th span (l i ,r i ,t i s ), where l i ,r i Represent the left and right boundaries of the span, t i s Represents the span type, the i-th span (l i ,r i ,t i s ) are the four span corners of the i-th span (l i ,r i ,t i s ) corresponds to the four corners of the quadrilateral in the current binary label matrix (l i ,l i ), (l i ,r i ), (r i ,l i ), (r i ,r i ), and the i-th span (l i ,r i ,t i s) are marked as 1 at the corresponding positions in the binary label matrix; in order to facilitate decoding and avoid confusion during decoding, the nested entities can be correctly decoded, and the span types of the upper left and lower right corners are marked as t i s' .

[0055] For the jth span-span relation (s j h ,s j t ,t j r ), s j h Represents the head span, s j h =(l j h ,r j h ,t j h ), l j h ,r j h ,t j h Represents the left boundary of the header span, the right boundary of the header span, and the type of the header span, s j t represents the tail span, s j t =(l j t ,r j t ,t j t ), l j t ,r j t ,t j t Represent the left boundary of the tail span, the right boundary of the tail span and the type of the tail span, t j r represents the type of the jth span-span relationship; the jth span-span relationship (s j h ,s j t ,t j r ) is the jth span-span relationship (s j h ,s j t ,t j r) corresponds to the four corners of the quadrilateral in the current binary label matrix (l j h ,l j t ), (l j h ,r j t ) , (r j h ,r j t ), (r j h ,l j t ), and mark the corresponding positions of the four span-span relationship corners in the binary label matrix as 1.

[0056] The general information extraction model includes a connected text vector representation extraction module and a feature encoding module. The input sequence is used as the input of the text vector representation extraction module, and the feature encoding module outputs a second-order probability matrix Y' as the output of the general information extraction model.

[0057] The text vector representation extraction module specifically uses the pre-trained language model to extract the text vector representation H of the input sequence containing N words, which includes the context vector representation of each token, satisfying H=[h1, …,h k ,…, h N ],H∈R N×e , where h k is the vector representation of the kth word, e represents the hidden dimension of the pre-trained language model, the value of e is 1024, R N×e It is a real number set of dimension N×e. The pre-trained language model is specifically the DeBERTa-large model.

[0058] The feature encoding module includes a dual affine model, a lightweight transformer model, and a label space mapping module. The input of the feature encoding module is used as the input of the dual affine model, and the output of the dual affine model is used as the input of the lightweight transformer model. The output of the lightweight transformer model is fused with the output of the dual affine model and then input into the label space mapping module. The output of the label space mapping module is a second-order probability matrix Y' and is used as the output of the feature encoding module. Among them, the dual affine model is used to encode the one-dimensional sequence features into a two-dimensional table. The two-dimensional table is an N×N×d matrix. Each cell is a d-dimensional feature vector, representing the representation of a word pair. The dual affine model is used to better encode the direction information of the word pair. The value of d is 200.

[0059] The lightweight transformer model includes three identical transformer layers connected in sequence, and the UniIEFormer model is used to model the interaction between cell latent vectors. Among them, the output of the dual affine model is used as the input of the first transformer layer, and the output of the last transformer layer is used as the output of the lightweight transformer model; each transformer layer contains a residual CNN module for modeling local dependencies, a lightweight attention module for capturing long-distance dependencies between cells, and two layer normalization layers. The input of each transformer layer is used as the input of the residual CNN module. The residual CNN module is connected to the lightweight attention module after the first layer normalization layer. The output of the lightweight attention module is fused with the input of the lightweight attention module and then used as the input of the second layer normalization layer to prevent problems such as gradient disappearance and gradient explosion. The output of the second layer normalization layer is used as the output of each transformer layer. The residual CNN module consists of two 3×3 convolutional blocks and the first activation layer. The input of the residual CNN module is used as the input of the first convolutional block. The first convolutional block is connected to the second convolutional layer after the first activation layer. The output of the second convolutional layer is fused with the input of the first convolutional block and then input into the first normalization layer.

[0060] The lightweight attention module is specifically:

[0061] S1: After implementing the Softmax attention mechanism on each row of the input of the lightweight attention module (i.e., the output of the first normalization layer), the two-dimensional feature matrix A after row processing is obtained. row , and after implementing the Softmax attention mechanism on each column of the input of the lightweight attention module (i.e., the output of the first normalization layer), the two-dimensional feature matrix A after column processing is obtained column ;

[0062] S2: After processing the input of the lightweight attention module using the linear attention layer, a first linear attention processing matrix is ​​obtained, and the input of the lightweight attention module is processed again using the linear attention layer to obtain a second linear attention processing matrix; then, the two-dimensional feature matrix A after row processing is row After adding it to the first linear attention processing matrix, we get the row feature matrix A h , the two-dimensional feature matrix A after column processing column After adding it to the second linear attention processing matrix, we get the column feature matrix A v In S1 and S2, the linear cross-row attention mechanism is combined to enhance the local quadratic Softmax attention mechanism, which reduces the computational complexity while ensuring the global receptive field. The linear attention layer is obtained by using the kernel function to replace the Softmax in the Softmax attention mechanism.

[0063] S3: After processing the input of the current lightweight attention module using a gated linear unit, the gate value G is obtained to control how much attention output each cell should retain. It is calculated by linear projection and Sigmoid function. The formula is as follows:

[0064] G = Sigmoid(H'W g )

[0065] Among them, H' is the input of the current lightweight attention module, W g is the control parameter of the gate value G, which is a learnable parameter, W g ∈R 2d×2d , d is the row feature matrix A h and the column feature matrix A v Dimension.

[0066] S4: The row feature matrix A h and the column feature matrix A v The concatenated matrix is ​​recorded as the complete feature matrix, and the complete feature matrix is ​​element-wise multiplied by the gate value G and then multiplied by the attention weight, and the result is used as the output A of the lightweight attention module, where the attention weight is a learnable weight. The specific formula is as follows:

[0067] A=[(A h ⊕A v )⊙G] W a

[0068] Among them, ⊕ represents the connection operation, ⊙ represents the element multiplication, and W a is the attention weight. The complexity of the LightAttention module is O(L 1.5 ), which is lower than Softmax Attention’s O(L 2 ) complexity, which reflects the lightweight nature of the lightweight attention module. L represents the number of all cells in the two-dimensional feature matrix, L=N×N.

[0069] The label space mapping module is specifically a perceptron model. In the perceptron model, the output of the lightweight transformer model is fused with the output of the bi-affine model using linear transformation to map the feature vector corresponding to each cell to the label space, and obtain a second-order probability matrix Y', satisfying Y'∈R N×N×T , the formula is as follows:

[0070] Y'=Sigmoid((H+H')W)

[0071] Among them, H is the two-dimensional feature matrix output by the bi-affine model, H' is the two-dimensional feature matrix output by the lightweight transformer model, W is the mapping weight, which is a learnable weight matrix, W∈R d×T , R d×T is a set of real numbers of dimension d×T.

[0072] During the training process of the general information extraction model, the general information extraction model is trained by calculating the cross entropy loss function between the second-order probability matrix Y' output by the general information extraction model and the corresponding label Y in the training data set until the training is completed.

[0073] The formula of the cross entropy loss function Loss(Y,Y') is as follows:

[0074] Loss(Y,Y')= -∑ m,n N ∑ t T [Y mnt log Y mnt '+(1-Y mnt )log(1- Y mnt ')]

[0075] Where m, n represent the row and column indexes, t represents the type index; N is the length of the input sentence, that is, the number of words in the input sentence; Y mnt Y is the value (0 or 1) corresponding to the index (m,n,t) in the label matrix. mnt ' is the probability value (a number between 0 and 1) corresponding to the index (m,n,t) in the predicted probability matrix.

[0076] In this example, the optimizer AdamW with a linear warm-up decay learning rate schedule is used, and the peak learning rate of the model is set to 1e -5 The batch size is set to 24, and the model is trained for 100 epochs. The network model is built using the PyTorch framework, and the model is trained using the data in the training set on an NVIDIA GeForce RTX3090 graphics card.

[0077] 2) Input the test set data into the trained general information extraction model. This part of the data did not appear in the previous training process. The model outputs the predicted second-order probability matrix Y'', satisfying Y''∈R N×N×T , that is, the predicted second-order probability matrix Y'' is composed of N×N cells; after the decoder is used to decode the predicted second-order probability matrix Y'', the span set S and span-span relationship set R of the input sequence to be extracted are obtained to complete the extraction of information.

[0078] Among them, after the decoder is used to decode the predicted second-order probability matrix Y'', the span set S and span-span relationship set R of the input sequence to be extracted are obtained, which are specifically:

[0079] First, find all t in the upper triangular matrix (i.e. the span corner in the upper left corner) of the predicted second-order probability matrix Y'' i s The probability value of the dimension is greater than the span corner threshold Th s The cells, that is, determine the span angle t corresponding to the upper left corner of each cell i s The probability value p(l i ,r i ,t i s ) satisfies p(l i ,r i ,t i s )>Th s , span corner threshold Th s is 0.5, thereby obtaining the initial span set; for each cell in the initial span set, determine the span angles of each cell (i.e., the other three span angles) corresponding to t i s The probability value p(r i ,l i ,t i s )、p(l i ,l i ,t i s' )、p(r i ,r i ,t i s' ) are both greater than the span angle threshold Th s , that is, whether p(r i ,l i ,t i s )>Th s , p(l i ,l i ,t i s' )>Th s , p(r i ,r i ,t i s' )>Th s , if both are greater than the span angle threshold Th s , then the cell is of type t i s , the left and right boundaries are li and r i span, otherwise the cell is deleted from the initial span set; after traversing and judging all cells in the initial span set, the final span set S is obtained, satisfying S={(l i ,r i ,t i s )} i KS , KS is the number of spans in the span set S;

[0080] Next, for each span pair (l p ,r p ,t p s )、(l q ,r q ,t q s ), if p(l p ,l q ,t j r )>Th r 、p(l p ,r q ,t j r )>Th r 、p(r p ,l q ,t j r )>Th r and p(r p ,r q ,t j r )>Th r , where p(l p ,l q ,t j r )、p(l p ,r q ,t j r )、p(r p ,l q ,t j r ) and p(r p ,r q ,t j r ) are the t corresponding to the four span-span relationship corners of the current span-span relationship. r The probability value of the dimension, Th r is the span-span relationship corner threshold, the span-span relationship corner threshold Th ris 0.5, then the current span is (l p ,r p ,t p s )、(l q ,r q ,t q s ) is a span-span relationship and is added to the span-span relationship set R, otherwise it is not added to the span-span relationship set R, and all span pairs in the span set S are traversed to obtain the final span-span relationship set R, satisfying R={(s j h ,s j t ,t j r )} j Kr , Kr is the number of span-span relations in the span-span relation set R.

[0081] In this embodiment, the F1 value is used as the evaluation parameter for entity extraction, relationship extraction and event extraction. Strict evaluation criteria are adopted in this embodiment. Specifically, for the predicted entity, it is considered a correct predicted entity only when and only when the boundary and type of the predicted entity completely match the boundary and type of a certain annotated entity; for the predicted relationship, it is considered a correct predicted relationship only when and only when the type of the predicted relationship, the type and boundary of the corresponding subject entity and object entity, and the relationship type of a certain annotated relationship, the type and boundary of the corresponding subject entity and object entity all match, it is considered a correct predicted relationship; for the predicted event trigger word, it is considered a correct predicted event trigger word only when and only when the boundary and corresponding event type of the predicted event trigger word completely match the boundary and event type of a certain annotated event trigger word; for the predicted event argument, it is considered a correct event argument only when and only when the boundary, argument type, and corresponding event type of the predicted argument completely match the boundary, argument type, and corresponding event type of a certain annotated event argument. The test results are shown in Table 1, where OneIE, FourIE, and UTC-IE are all proposed general information extraction methods, and the F1 performance parameters of the method proposed in the present invention are consistently better than these methods in the four subtasks.

[0082] Table 1 shows the F1 values ​​of the method proposed in this invention and other methods

[0083] entity relation Event trigger words Event Argument The method proposed by the present invention 91.59 67.97 75.73 60.09 OneIE 89.60 58.60 72.80 54.80 FourIE 91.10 63.60 73.30 57.50 UTC-IE 91.48 65.54 73.36 57.62

[0084] The present invention also proposes a general information extraction system, comprising:

[0085] A training data set construction unit is used to label each input sequence, where the label is a binary label matrix containing spans and span-span relationships, thereby generating a training data set;

[0086] A general information extraction unit, used to train the general information extraction model using the training data set, obtain the trained general information extraction model and store it;

[0087] The information extraction output unit is used to input the input sequence to be extracted into the trained general information extraction model, and decode the predicted second-order probability matrix Y'' output by the model to obtain the span set S and the span-span relationship set R.

[0088] The present invention also proposes a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the steps of a general information extraction method when executing the computer program.

[0089] The present invention also proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a general information extraction method are implemented.

[0090] The present invention also proposes a computer program product, which includes a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of a general information extraction method are implemented.

[0091] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A general information extraction method, characterized in that: The following steps are involved: 1) Constructing a general information extraction model, and then using the training data set to train the general information extraction model to obtain a trained general information extraction model; 2) Input the input sequence to be extracted into the trained general information extraction model, and the model outputs the predicted second-order probability matrix Y'', where the predicted second-order probability matrix Y'' consists of N×N cells; then use the decoder to decode the predicted second-order probability matrix Y'' to obtain the span set S and span-span relationship set R of the input sequence to be extracted, and complete the information extraction; In 1), the training data set includes training samples of different task types, each training sample consists of an input sequence including N words and a corresponding label, and the label is a binary label matrix including span and span-span relationship; For each input sequence containing N words, its label is specifically: Each label is a binary label matrix containing span and span-span relationship. For the i-th span (l i ,r i ,t i s ), where l i ,r i Represent the left and right boundaries of the span, t i s Represents the span type, the i-th span (l i ,r i ,t i s ) are the four span corners of the i-th span (l i ,r i ,t i s ) corresponds to the four corners of the quadrilateral in the current binary label matrix (l i ,l i ),(l i ,r i ), (r i ,l i ), (r i ,r i ), and the i-th span (l i ,r i ,t i s )’s four span corners are marked as 1 at their corresponding positions in the binary label matrix; For the jth span-span relation (s j h ,s j t ,t j r ), s j h Represents the head span, s j h =(l j h ,r j h ,t j h ), l j h ,r j h ,t j h Represents the left boundary of the header span, the right boundary of the header span, and the type of the header span, s j t represents the tail span, s j t =(l j t ,r j t ,t j t ), l j t ,r j t ,t j t Represent the left boundary of the tail span, the right boundary of the tail span and the type of the tail span, t j r represents the type of the jth span-span relationship; the jth span-span relationship (s j h ,s j t ,t j r ) is the jth span-span relationship (s j h ,s j t ,t j r ) corresponds to the four corners of the quadrilateral in the current binary label matrix (l j h ,l j t ),(l j h ,r j t ) , (r j h ,r j t ), (r j h ,l j t ), and mark the corresponding positions of the four span-span relationship corners in the binary label matrix as 1; In the above 1), the general information extraction model includes a connected text vector representation extraction module and a feature encoding module, the input sequence is used as the input of the text vector representation extraction module, and the feature encoding module outputs a second-order probability matrix Y' as the output of the general information extraction model; The feature encoding module includes a dual affine model, a lightweight transformer model and a label space mapping module. The input of the feature encoding module is used as the input of the dual affine model, the output of the dual affine model is used as the input of the lightweight transformer model, the output of the lightweight transformer model is fused with the output of the dual affine model and then input into the label space mapping module. The output of the label space mapping module is a second-order probability matrix Y' and is used as the output of the feature encoding module.

2. A general information extraction method according to claim 1, characterized in that: The lightweight transformer model includes N sequentially connected Transformer layers, the output of the dual affine model is used as the input of the first Transformer layer, and the output of the last Transformer layer is used as the output of the lightweight transformer model; each Transformer layer includes a residual CNN module, a lightweight attention module and two layer normalization layers, the input of each Transformer layer is used as the input of the residual CNN module, the residual CNN module is connected to the lightweight attention module after the first layer normalization layer, the output of the lightweight attention module is fused with the input of the lightweight attention module and then used as the input of the second layer normalization layer, and the output of the second layer normalization layer is used as the output of each Transformer layer.

3. A general information extraction method according to claim 2, characterized in that: The lightweight attention module is specifically: S1: After implementing the Softmax attention mechanism on each row in the input of the lightweight attention module, the two-dimensional feature matrix A after row processing is obtained row , and after implementing the Softmax attention mechanism on each column in the input of the lightweight attention module, the two-dimensional feature matrix A after column processing is obtained column ; S2: After processing the input of the lightweight attention module using the linear attention layer, a first linear attention processing matrix is ​​obtained, and the input of the lightweight attention module is processed again using the linear attention layer to obtain a second linear attention processing matrix; then, the two-dimensional feature matrix A after row processing is row After adding it to the first linear attention processing matrix, we get the row feature matrix A h , the two-dimensional feature matrix A after column processing column After adding it to the second linear attention processing matrix, we get the column feature matrix A v ; S3: Use a gated linear unit to process the input of the current lightweight attention module to obtain the gated value G; S4: The row feature matrix A h and the column feature matrix A v The connected matrix is ​​recorded as the complete feature matrix. The complete feature matrix is ​​then element-wise multiplied with the gate value G and then multiplied with the attention weight, and the result is taken as the output A of the lightweight attention module, where the attention weight is a learnable weight.

4. A general information extraction method according to claim 3, characterized in that: The linear attention layer is obtained by using a kernel function instead of the Softmax in the Softmax attention mechanism.

5. A general information extraction method according to claim 1, characterized in that: In the label space mapping module, a linear transformation is used to map the feature vector corresponding to each cell in the result of fusing the output of the lightweight transformer model with the output of the bi-affine model to the label space to obtain a second-order probability matrix Y'.

6. A general information extraction method according to claim 1, characterized in that: After the decoder is used to decode the predicted second-order probability matrix Y'', the span set S and span-span relationship set R of the input sequence to be extracted are obtained, which are specifically: First, find all t in the upper triangular matrix of the predicted second-order probability matrix Y'' i s The probability value of the dimension is greater than the span corner threshold Th s 's cells to obtain the initial span set; For each cell in the initial span set, determine the span corners of each cell corresponding to t i s Are the probability values ​​of the dimensions greater than the span corner threshold Th? s , if both are greater than the span angle threshold Th s , then the cell is a span, otherwise the cell is deleted from the initial span set; after traversing and judging all cells in the initial span set, the final span set S is obtained; Next, for each span pair (l p ,r p ,t p s )、(l q ,r q ,t q s ), if p(l p ,l q ,t j r )>Th r 、p(l p ,r q ,t j r )>Th r 、p(r p ,l q ,t j r )>Th r and p(r p ,r q ,t j r )>Th r , where p(l p ,l q ,t j r )、p(l p ,r q ,t j r )、p(r p ,l q ,t j r ) and p(r p ,r q ,t j r ) are the t corresponding to the four span-span relationship corners of the current span-span relationship. j r The probability value of the dimension, Th r is the span-span relationship corner threshold, then the current span pair (l p ,r p ,t p s )、(l q ,r q ,t q s ) is a span-span relationship and is added to the span-span relationship set R. All span pairs in the span set S are traversed to obtain the final span-span relationship set R.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the processor implements the steps of a general information extraction method described in any one of claims 1 to 6.

Images