A document-level financial relation extraction method fusing entity and window attention

Abstract

The application designs a document-level financial relationship extraction method fusing entity and window attention, and comprises the following steps: data is preprocessed in a sliding window mode and is input to a pre-training model; the expression of each entity is obtained through the output of the pre-training model, the expression of the head entity-tail entity and the distance of the head entity-tail entity are further combined, and then one convolutional neural network is used to realize the interaction of global entity information; the document-level attention output by the pre-training model is extracted for local information by using a fixed window; the two kinds of features are fused and input to a U-shaped neural network for relationship reasoning; and finally, relationship prediction is performed through a bilinear function. The method can extract document-level financial relationships and achieves good extraction effect.

Description

Technical Field

[0001] This invention relates to the field of natural language processing, and in particular to a document-level financial relationship extraction method that integrates entity and window attention. Background Technology

[0002] As knowledge graphs are increasingly used in fields such as finance, their construction has become more significant for vertical industries, making core technologies crucial. In recent years, financial data has exploded, with numerous financial relationships existing in announcements related to changes in listed company shareholdings. Extracting these relationships to build structured information and financial knowledge graphs is an effective means for governments and markets to manage and predict risks. From the perspective of data sources, relation extraction is even more important for building knowledge graphs in this field. Relation extraction, which involves extracting semantic relationships between entities in text data, is one of the core tasks of information extraction and a crucial part of question-answering systems and large-scale knowledge graph construction. The results of relation extraction are generally represented as triples such as (head entity, relation, tail entity).

[0003] Existing sentence-level relation extraction methods focus on issues such as triple entity overlap (different triples sharing the same entity). However, for real-world text data, there are numerous instances where entities span across sentences (the first and last entities of a triple are in different sentences), which sentence-level relation extraction cannot address. Some document-level relation extraction methods aim to extend the solution to this problem; these methods not only solve the cross-sentence problem but also perform relation reasoning. Document-level relation extraction methods primarily employ two models: one is a reasoning-focused model based on graph neural networks to build attention and dependency structures, and the other is a Transformer architecture model that emphasizes contextual semantic relationships. Regarding encoding modules, there are traditional encoding modules such as CNN, LSTM, and BILSTM, and pre-trained models such as BERT. While traditional encoding modules can handle long texts, their performance is poor. Pre-trained models perform significantly better than traditional encoding modules, but their maximum input character length is limited to 512. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a document-level financial relationship extraction method that integrates entity and window attention, implemented using the following technical solution:

[0005] A document-level financial relationship extraction method that integrates entity and window attention includes the following steps:

[0006] Step 1: For each document in the dataset, preprocess it using a sliding window method, divide the document into several paragraphs and input them into the BERT pre-trained model to obtain the vector representation of each paragraph. Then, perform max pooling on the vector representations of each paragraph to obtain the vector representation of each character in each document and the attention of the document to each character.

[0007] Step 2: Based on the vector representation of each document, obtain the vector representation of each entity within the document. For different entity combinations, concatenate the representation of the head entity-tail entity combination and the distance representation between the head entity and tail entity. Input the concatenation result into a linear layer for global entity information interaction to obtain global entity information.

[0008] Step 3: Based on the attention of each character in the acquired document, obtain the local attention of each entity through a fixed window;

[0009] Step 4: Concatenate the global information of the acquired entities and the local attention of each entity to obtain the fused entity features, and input the fused entity features into the U-shaped neural network for relation reasoning;

[0010] Step 5: Use the output of the U-shaped neural network to predict relationships through a bilinear function to obtain multi-classification results of relationships between each entity.

[0011] Furthermore, in step one, the vector representation of each character in the document is as follows:

[0012] H = [h1, h2, ..., h L =MaxPool[S1,S2,...,S] K ]

[0013] Where h1, h2, ..., h L S represents the vector representation of a single character within the document. i These are vector representations of each paragraph. w si These are the characters in the paragraph, MaxPool represents max pooling, and H is the final vector representation of the document.

[0014] Furthermore, in step two, the global information of the entity obtained is as follows:

[0015]

[0016] Where W1 is a learnable parameter, e s e o These are the vector representations of the head entity and the tail entity, respectively. It is the distance expression between the head entity and the tail entity, F e (e s ,eo ) is the global information of the entity.

[0017] Furthermore, in step three, the local attention for each entity is obtained as follows:

[0018] F c (e s ,e o =W2window p (Ha (s,o) )

[0019] Where W2 is a learnable parameter, a (s,o) It is the document's attention to a set of head entity-tail entity combinations, window p A window of size p represents the number of characters p that can receive attention from a character up to p characters to its left and up to p characters to its right.

[0020] Furthermore, in step four, the fused entity features are as follows:

[0021] F = σ(W) m [F e (e s ,e o );F c (e s ,e o )]+b m )

[0022] Among them, W m b m σ is a learnable parameter, σ is the activation function, and F is a fusion feature matrix that stores all head-tail entity combinations.

[0023] Furthermore, in step five, the U-shaped neural network includes two downsampling modules and two upsampling modules with skip connections. Each downsampling module has two max pooling and independent convolutional modules, and the number of channels doubles after passing through each downsampling module. The upsampling module has a deconvolutional neural network and two independent convolutional modules, and the number of channels is halved after passing through each upsampling module.

[0024] Furthermore, in step five, the output of the U-shaped neural network is:

[0025] G = U(W3F)

[0026] Where U and G∈R N×N×D These are a U-shaped neural network and an entity-level relation matrix, respectively, and W3 is a learnable weight matrix.

[0027] Furthermore, in step five, the multi-classification results of the relationships between each entity are as follows:

[0028] P(r|e s ,e o )=σ(z s W r z o +b r )

[0029] Among them, z s =tanh(W s e s +G s,o ), z o =tanh(W o e o +G s,o ), G s,o It represents the entity pair relation at position (s,o) in the entity-level relation matrix G. W s ∈R (d+D)×d and W o ∈R (d+D)×d These are learnable parameters, where d is the default dimension of the pre-trained model. r It represents the total number of relationships.

[0030] The present invention also relates to an electronic device comprising: a processor and a memory for storing a computer program capable of running on the processor; wherein, when the processor runs the computer program, it performs the steps of the above-described method.

[0031] The present invention also relates to a computer storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0032] The beneficial technical effects of the present invention are as follows:

[0033] The method of this invention first addresses the limitation of the maximum input length of the pre-trained BERT model by employing a long text preprocessing optimization strategy based on a sliding window. During preprocessing, long texts are segmented using a fixed-size window and input into the pre-trained model, then the output is concatenated using max pooling. Secondly, two feature extraction strategies are proposed, extracting global entity information and local contextual information respectively. On one hand, by combining the representations and distances between head-to-tail entities, a single convolutional neural network layer is used to achieve interaction of global entity information. On the other hand, contextual features based on window attention are used to obtain the attention of the context to entities within the window size, reducing noise from irrelevant text. Finally, a U-shaped neural network is used for entity relationship inference, and a bilinear function is used for relationship classification.

[0034] Therefore, the present invention has the following advantages:

[0035] (1) The problem of the maximum input length of the pre-trained model BERT was solved by the long text preprocessing optimization strategy based on sliding window, which is better than using traditional encoding modules;

[0036] (2) A head-to-tail entity feature extraction strategy was designed to obtain global feature information of the entities;

[0037] (3) A local attention feature extraction strategy was designed to obtain the local feature information of the entity;

[0038] (4) Using a U-shaped neural network for relation reasoning and finally using a bilinear function for relation prediction, the problem of head-to-tail entity cross-sentence and entity overlap in document-level relation extraction is solved.

[0039] (5) The application of document-level relation extraction technology in the financial field provides a way to solve the problem of upstream knowledge extraction in the knowledge graph of the financial field, and has achieved good results. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the method of an embodiment of the present invention.

[0041] Figure 2 This is a diagram of the overall network architecture in an embodiment of the present invention.

[0042] Figure 3 This is a diagram illustrating the preprocessing steps of segmenting long text using a sliding window in an embodiment of the present invention.

[0043] Figure 4 This is the process of extracting head-to-tail entity features and context features in this embodiment of the invention.

[0044] Figure 5 This is a case study of MEIWAN and the benchmark model in the embodiments of the present invention. Detailed Implementation

[0045] To facilitate a better understanding of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The following are merely exemplary and do not limit the scope of protection of the present invention.

[0046] Terminology Explanation:

[0047] 1. BERT: It stands for Bidirectional Encoder Representation from Transformers. It is a pre-trained model for natural language processing that converts text into corresponding vectors.

[0048] 2. Attention mechanism: Represents a modified linear unit, which is an activation function.

[0049] 3. concatenate: indicates the concatenation of features.

[0050] 4. Sigmoid: Represents the activation function of a convolutional neural network, which maps variables to the range of 0 and 1.

[0051] 5. tanh: Represents the hyperbolic activation function of a convolutional neural network, which maps variables to the range of 0 and 1.

[0052] 6. Entity: A type of individual defined in the data.

[0053] 7. Referential: Different representations of the same entity in data.

[0054] like Figure 1 As shown in the figure, the document-level financial relationship extraction method that integrates entity and window attention described in this embodiment specifically includes the following steps:

[0055] Step 1, such as Figure 3 As shown, each document in the dataset is preprocessed using a sliding window approach and then input into a pre-trained model. The output is then concatenated using max pooling to obtain the vector representation of each document. Specifically, this includes:

[0056] First in the document In the middle, first insert a special symbol "" at the beginning and end of each reference. <e> "and"< / e> "Used to mark the reference position, and then the document with the inserted special characters is segmented using a fixed-size (510) window and input into the pre-trained model, as shown in equation (1), where Encoder is the pre-trained model, w si It is the character of each paragraph, S i It is a vector representation of a paragraph.

[0057]

[0058] Then, the output results are concatenated using max pooling to obtain the vector representation of the document, as shown in equation (2), where MaxPool is the max pooling method and H is the final vector representation of the document:

[0059] H = [h1, h2, ..., h L =MaxPool[S1,S2,...,S] K (2)

[0060] Step 2, as follows Figure 4 As shown, the representation of each entity is obtained through the output of the pre-trained model. This is further combined with the head-to-tail entity representation and the head-to-tail entity distance. A convolutional neural network layer is then used to realize the interaction of global entity information, specifically including:

[0061] Based on the specific location of the entities labeled in the dataset within the document, find the reference to each entity from the document's vector representation. <e>The vector representation of this symbol represents the entire reference. Simultaneously, different references belonging to the same entity are further processed through LSE pooling (Log-Sum-Exp pooling) to obtain the comprehensive vector representation of each entity, as shown in equation (3), where m j Then it is the expression of each reference, e i That is, the expression of each entity.

[0062]

[0063] As in equation (4), The learnable weight matrix uses the first reference of each entity as the computational basis to obtain the distance between every two entities. After splicing together the head entity and tail entity, we obtain the head entity e. s Combined tail entity e o Combinatorial expression F e (e s ,e o ):

[0064]

[0065] Step 3, as follows Figure 4 As shown, a fixed window is used to extract local information from the document-level attention output by the pre-trained model, specifically including:

[0066] For each entity, attention is obtained by using a fixed window p to the left and p characters to the right of the entity. The context features F based on window attention are obtained by using the entity context features output by the pre-trained model. c (e s ,e o ):

[0067] F c (e s ,e o )=W2Ha (s,o) (5)

[0068]

[0069] In equations (5) and (6), F e (e s ,e o )∈R N×N×D ,F c (e s ,e o )∈R N×N×D ,a (s,o) It represents the attention weights of a document's expression for a set of head and tail entities, window p (X) represents the local attention of the i-th entity, with a window size of p. It is the document's attention to the i-th header entity. H represents the document's attention to the i-th tail entity, and H is the document's embedded representation. It is a learnable weight matrix, D e =2×emb_size+20,D c =emb_size, D=256, this article sets emb_size=768 as the default dimension of the pre-trained model. K is the number of heads in the multi-head attention, the default is 12.

[0070] Step 4: Concatenate the global and local entity features obtained from the feature mapping module to obtain the fused feature F∈R. N×N×D’ In equation (7), Wm∈R 2×D×D‘ and b m ∈R D‘ It is a learnable parameter, and this paper sets D'=3 to achieve dimensionality reduction by using D', which is much smaller than D:

[0071] F = σ(W) m [F e (e s ,e o );F c (e s ,e o )]+b m (7)

[0072] Furthermore, the fused features are input into a U-shaped neural network for semantic segmentation to perform relation reasoning. This network includes two downsampling modules and two upsampling modules with skip connections. Each downsampling module has two max-pooling modules and independent convolutional modules, and the number of channels doubles with each downsampling module. By increasing the receptive field of each relation through the CNN and downsampling modules, each relation can learn information from other relations, thus achieving relation reasoning. The upsampling module of the model has a deconvolutional neural network and two independent convolutional modules. Conversely, with downsampling, the number of channels is halved with each upsampling module, allowing for information compression for each relation. Finally, a matrix G containing both local and global information is obtained.

[0073] G=U(W3F) (8)

[0074] In equation (8), U and G ∈ R N×N×D These are a U-shaped neural network and an entity-level relation matrix, respectively. W3 is a learnable weight matrix. Starting from the input of 3 channels, the dimensions of the entire U-shaped network change to [3, 256, 512, 256, 128, 256], with 256 being the dimension used for relation classification.

[0075] Step 5: Predict the relationship using a bilinear function, specifically including:

[0076] Using the entity-level relation matrix G obtained in step four, combined with the entity pair expression e obtained in step two... s and e o Relationship probability prediction is performed using a bilinear network:

[0077] z s =tanh(W s e s +G s,o (9)

[0078] z o =tanh(W o e o +G s,o (10)

[0079] P(r|e s ,e o )=σ(z s W r z o +b r (11)

[0080] In equations (9), (10), and (11), G s,o It represents the entity pair relation at position (s,o) in the entity-level relation matrix G. W s ∈R (d+D)×d and W o ∈R (d+D)×d These are learnable parameters; d = 768 is the default dimension of the pre-trained model. r It represents the total number of relationships.

[0081] Furthermore, since most entity pairs are unrelated (NA), the model uses a balanced max pooling method for training. By introducing an additional class 0, the target class score is larger than s0, while the non-target class score is smaller than s0. The generalized loss function and the loss function after simplifying s0 to 0 are shown in equations (12) and (13):

[0082]

[0083]

[0084] The above description only outlines the basic principles and preferred embodiments of the present invention. Those skilled in the art can make many changes and modifications based on the above description, and these changes and modifications should fall within the protection scope of the present invention.< / e>

Claims

1. A document-level financial relationship extraction method that integrates entity and window attention, comprising the following steps: Step 1: For each document in the dataset, preprocess it using a sliding window method, divide the document into several paragraphs and input them into the BERT pre-trained model to obtain the vector representation of each paragraph. Then, perform max pooling on the vector representations of each paragraph to obtain the vector representation of each character in each document and the attention of the document to each character. Step 2: Based on the vector representation of each document, obtain the vector representation of each entity within the document. For different entity combinations, concatenate the representation of the head entity-tail entity combination and the distance representation between the head entity and tail entity. Input the concatenation result into a linear layer for global entity information interaction to obtain global entity information. Step 3: Based on the attention of each character in the acquired document, obtain the local attention of each entity through a fixed window; Step 4: Concatenate the global information of the acquired entities and the local attention of each entity to obtain the fused entity features, and input the fused entity features into the U-shaped neural network for relation reasoning; Step 5: Use the output of the U-shaped neural network to predict relationships through a bilinear function to obtain multi-classification results of relationships between each entity.

2. The document-level financial relation extraction method of claim 1, wherein, In step one, the vector representation of each character in the document is as follows: H = [h1, h2,..., h L ] = MaxPool [S1, S2,..., S K ] where h1, h2,..., h L are the vector representations of individual words within the document, S i is the vector representation of each paragraph, w si is the character in the paragraph, MaxPool denotes max-pooling, and H is the final vector representation of the document.

3. The document-level financial relation extraction method of claim 2, wherein, In step two, the global information of the entity obtained is as follows: where W1is a learnable parameter, e s , e o are vector representations of head and tail entities, respectively, is a distance representation between head and tail entities, F e (e s , e o ) is global information of entities.

4. The document-level financial relation extraction method of claim 3, wherein, In step three, the local attention of each entity is obtained as follows: F c (e s ,e o )=W2window p (Ha (s,o) ) Where W2 is a learnable parameter, a (s,o) It is the document's attention to a set of head entity-tail entity combinations, window p A window of size p represents the number of characters p that can receive attention from a character up to p characters to its left and up to p characters to its right.

5. The document-level financial relation extraction method of claim 4, wherein, In step four, the fused entity features are as follows: F = σ(W m [F e (e s ,e o ) ; F c (e s ,e o ) + b m ) where W m , b m are learnable parameters, σ is an activation function, and F is a fusion feature matrix that holds all head entity-tail entity combinations.

6. The document-level financial relation extraction method of claim 5, wherein, In step five, the U-shaped neural network includes two downsampling modules and two upsampling modules with skip connections. Each downsampling module has two max pooling and independent convolutional modules, and the number of channels doubles after passing through each downsampling module. The upsampling module has a deconvolutional neural network and two independent convolutional modules, and the number of channels is halved after passing through each upsampling module.

7. The document-level financial relation extraction method of claim 6, wherein, In step five, the output of the U-shaped neural network is: G = U(W3F) where U and G ∈ R N×N×D are the U-shaped neural network and the entity-level relation matrix, respectively, and W3is a learnable weight matrix.

8. The document-level financial relation extraction method of claim 7, wherein, In step five, the multi-classification results of the relationships between each entity are as follows: P(r|e s ,e o )=σ(z s W r z o +b r ) Among them, z s =tanh(W s e s +G s,o ), z o =tanh(W o e o +G s,o ), G s,o It represents the entity pair relation at position (s,o) in the entity-level relation matrix G. W s ∈R (d+D)×d and W o ∈R (d+D)×d These are learnable parameters, where d is the default dimension of the pre-trained model. r It represents the total number of relationships.

9. An electronic device, comprising: The electronic device includes: a processor and a memory for storing a computer program capable of running on the processor; wherein, when the processor runs the computer program, it performs the steps of the method according to any one of claims 1 to 8.

10. A computer storage medium storing a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

Images