Financial entity and emotion joint extraction method

Abstract

The invention belongs to the field of financial text sentiment analysis, and particularly relates to a financial entity and sentiment joint extraction method. In order to identify entities and emotional polarities thereof, the method comprises the following steps: firstly, carrying out initial representation on sentences by utilizing a pre-training model BERT-wwm-ext, then, obtaining a plurality of score matrixes of the sentences through a multi-head self-attention mechanism, and respectively constructing a graph convolutional network (GCN) for the plurality of score matrixes of the entities so as to obtain deep semantic representation of the entities, so that the deep semantic representation of the entities is obtained on the basis of the deep semantic representation. According to the method, a multi-head attention mechanism with entities as queries is utilized, interaction information representation between the entities and sentences is obtained, sentence information representation fusing the entities is established, and joint extraction of financial entities and emotional polarities of the financial entities is achieved through a Transform decoding layer.

Description

Technical Field

[0001] The present invention belongs to the field of financial text sentiment analysis, and specifically relates to a method for jointly extracting financial entities and sentiments. Background Art

[0002] In the real scenario of financial text application, the expression of financial entities is diverse, including names of people, companies, stocks and brands, etc. These entities are difficult to be given in advance. In addition, a sentence may contain multiple entities, and the length of financial entities is relatively long. There is also a certain semantic relationship between entities, and the emotional expression of different financial entities may be different, and even the emotion is implicitly expressed. For example, "Nowadays, there are many online lending platforms. JD Finance is deeply loved by users, while Qianbao.com, Yatang Finance, Tang Xiaoseng and Lianbi Finance are known as the four high-return platforms in the industry. Before Tang Xiaoseng, Qianbao.com and Yatang Finance had already exploded." In this example, multiple financial entities "JD Finance, Qianbao.com, Yatang Finance, Tang Xiaoseng, Lianbi Finance" are involved. The emotional opinions expressed in the sentences they are in are different, and their emotional polarity is also different. For example, the emotional opinion of "JD Finance" is "deeply loved by users", while the emotional opinions of "Qianbao.com" and "Yatang Finance" metaphorically express "explosion". Therefore, the identification of entities and their emotional opinion expressions is the key to entity-level sentiment analysis. Therefore, sentiment analysis of financial text entities can provide ordinary users with reasonable and reliable reference opinions for investment and consumption. At the same time, it can provide important technical support for the financial industry's intelligent customer service, intelligent monitoring, and the establishment of intelligent early warning and prevention systems for financial risks. It is also beneficial for businesses to innovate intelligent financial products and services and develop new financial formats.

[0003] In the financial field, there are relatively few studies on entity-level sentiment analysis, and there are many problems. The expression of proper nouns of financial entities is diverse, making it difficult for some word segmentation tools to accurately segment them. For example, "Qianbao.com, Yatang Finance, Tang Xiaoseng and Lianbi Finance are known as the four high-return platforms in the industry", and there are multiple entities; the emotional expression in the financial field is implicit, making it difficult to ensure the accuracy of its recognition, for example, "Before Tang Xiaoseng, Qianbao.com and Yatang Finance had already exploded". Most of the existing studies give financial entities and discriminate their sentiment polarity on this basis. However, in real scenarios in the financial field, there are many financial entities involved, which are generally impossible to give. Now the learning mechanism of the pre-trained language model and the joint learning framework provides an important theoretical basis for solving the extraction of financial entities and the discrimination of sentiment polarity. Therefore, how to establish the semantic relationship between financial entities, the relationship between entities and opinion expressions, and the interactive information representation between entities and the sentences in which they are located is an urgent problem to be solved in the joint extraction of financial entities and sentiment polarity. The present invention provides a joint extraction method of financial entities and sentiments, which can serve smart finance and financial public opinion analysis. Summary of the invention

[0004] The purpose of the present invention is to identify entities and their sentiment polarity, and to propose a multi-task-based joint extraction method for financial entities and sentiment. Aiming at the characteristics of long length of financial entities and semantic relationships between financial entities, the method designs sentence embedding representation, deep semantic representation, interaction between entity extraction sequence and sentiment representation sequence, and entity sentiment joint extraction, thereby improving the performance of joint extraction of financial entity and sentiment polarity tuples. On the entity-level sentiment analysis dataset, the P value and F1 value of the joint extraction of tuples reached 66.55% and 67.27% respectively, both higher than the best method currently available.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] The idea of ​​the present invention is: first, the pre-trained model BERT-wwm-ext is used to initially represent the sentence, and then multiple score matrices of the sentence are obtained through a multi-head self-attention mechanism, and multiple graph convolutional networks are constructed respectively with the multiple score matrices of the entity to obtain the deep semantic representation of the entity. On this basis, by using the multi-head attention mechanism with the entity as the query, the interactive information representation between the entity and the sentence is obtained, the sentence information representation of the fused entity is established, and the discrimination of the financial entity and its sentiment polarity is realized through the Transformer decoding layer.

[0007] A method for jointly extracting financial entities and sentiments comprises the following steps:

[0008] Step 1: Use the BERT-wwm-ext model to embed the given financial sentence into a representation H;

[0009] Step 2: Take the embedding representation H obtained in step 1 as input and build a joint extraction model of financial entities and sentiments, including:

[0010] 1) Entity representation: Using the self-attention mechanism, we obtain the deep semantic representation of m different graph convolutional networks for financial sentences. That is, we get the deep semantics of the financial sentence H GCN , transform it linearly, and then establish the deep semantic representation H of the entity ete ;

[0011] 2) Entity extraction: To obtain the deep semantic representation H of the entity ete , and then use the softmax function to obtain the label of whether each word is an entity;

[0012] 3) Interaction between entity sequence and sentiment representation sequence: Multi-head attention mechanism is used to analyze the deep semantic representation of entity ete Interact with the sentence embedding representation H to obtain the interactive sentence representation H s;

[0013] 4) Entity sentiment polarity discrimination: Using the sentence representation H after interaction s , and then through the Transformer decoding layer, using the full connection operation, the sentiment discrimination sentence representation H is obtained esc , and then use the softmax function to obtain the entity’s sentiment polarity discrimination result, and finally realize the joint extraction of financial entity and sentiment polarity binary groups.

[0014] Furthermore, in step 1, the BERT-wwm-ext model is used to embed the given financial sentence into a representation H, specifically:

[0015] Represent the given financial sentence as “[CLS] sentence [SEP]” to obtain the word-level embedding representation H of the sentence [X] ∈R (n+2)×d , the formula is as follows:

[0016] H [1:X] =BERT(S)

[0017] Where S represents a given sentence, X represents the maximum number of encoding layers of BERT-wwm-ext, and H [X] The layer is used to complete the entity extraction task.

[0018] Furthermore, the BERT-wwm-ext model uses shared BERT-wwm-ext to jointly complete the entity extraction task and the financial entity sentiment polarity discrimination task. The low layer, middle layer and high layer, the low layer obtains phrase-level information, the middle layer obtains language information, and the high layer obtains task-related information.

[0019] Furthermore, in step 2, the deep semantic representation H of the entity is established. ete , specifically:

[0020] (1) Word attention score matrix: Using the self-attention mechanism, the words in the sentence are regarded as queries and keys to obtain the attention score matrix of each word in the sentence. The formula is as follows:

[0021]

[0022] Among them, W Q ∈R d×d and W K ∈R d×d represents the learnable parameter matrix, d represents the input node dimension, Q and K represent the Query and Key in the self-attention mechanism respectively, and T represents the transpose operation of the matrix;

[0023] (2) Node representation in graph convolutional networks: In a graph convolutional network, the hidden vector representation h of the i-th word node in the l-th layer of the network is i (l) , the formula is as follows:

[0024]

[0025] Among them, A ij W represents the degree of association between the i-th word node and the j-th word node of its neighbor on the graph convolutional network. l represents the weight matrix of the lth layer in the graph convolutional network, represents the hidden vector representation of the i-th word node in the l-1th layer of the network, b l represents the bias term of the lth layer, σ represents the ReLU activation function, and n represents the total number of layers in the graph convolutional network;

[0026] (3) Deep semantic representation of sentences based on graph convolutional networks: Using graph convolutional networks to obtain representations of all nodes in a sentence Finally, we get the deep semantic representation H of sentence S GCN (S), the formula is as follows:

[0027]

[0028] in, Represents the i-th word node of the sentence in the graph convolutional network, i is 1, 2, ..., n, m;

[0029] (4) Sentence fusion representation under multi-head attention mechanism: Based on the multiple word score matrices A of the sentence obtained by the multiple self-attention mechanisms in (1) above, k graph convolutional networks are established accordingly, and then the k graph networks are fused using linear transformation to obtain the deep semantic representation H of the entity. ete .

[0030] Furthermore, the sentence representation H obtained after the interaction in step 2 is s , specifically:

[0031] The deep semantic representation of the entity H ete As the query of the multi-head attention mechanism, the sentence is the key and value of the multi-head attention mechanism. The sentence of the i-th head attention is expressed as follows:

[0032]

[0033] in, and represents the learnable parameter matrix, m represents the number of attention heads, and d k =d v= d / m, Q, K and V represent Query, Key and Value in the attention mechanism respectively;

[0034] Sentence representation H through m-head attention s as follows:

[0035] H s =MultiHead(Q,K,V)=Concat(head1,...,head m )W O .

[0036] Furthermore, the training method of the financial entity and sentiment joint extraction model in step 2 is as follows:

[0037] S1, for entity extraction, the deep semantic representation H of k graph convolutional neural networks is transformed through linear transformation i GCN Perform linear fusion to obtain the deep semantic representation H of the entity ete , entity boundary discrimination probability p ete as follows:

[0038]

[0039] p ete =softmax(H ete )

[0040]

[0041] Among them, W i ete and b ete Represent the trainable weight matrix and bias parameters respectively;

[0042] Entity boundary discrimination probability p ete The cross entropy objective function is used for joint training to obtain the loss L for entity extraction. ate , the formula is as follows:

[0043]

[0044] Among them, if the i-th position is the boundary of the entity, then otherwise

[0045] When the sentiment polarity discrimination task is regarded as a polarity sequence label during model training, there is a problem of inconsistent entity sequence labels generated. For example, the sentiment polarity of the financial entity "Jingdong Baitiao" may be positive, neutral, or negative. Therefore, before entity sentiment polarity discrimination, p ete Get the boundary of the entity [b ind :eind ], the sentiment polarity discrimination of the joint entity, the process is as follows:

[0046] H esc =H ete W esc +b esc

[0047] o i =max(H esc [b ind :e ind ])

[0048]

[0049] Among them, H esc [b ind :e ind ] indicates from b ind To e ind Fragment of H esc , max represents the maximum pooling operator of the sequence dimension, W esc represents the trainable weight matrix, b esc represents the trainable bias parameter, represents the sentiment probability of the i-th entity boundary, W h Trainable parameter weight matrix;

[0050] The sentiment discrimination loss L of the joint entity is obtained by the sentiment polarity discrimination probability of the entity asc , the formula is as follows:

[0051]

[0052] Among them, if the i-th word is the boundary of the entity, then otherwise

[0053] S2, in order to make the model robust, virtual adversarial training is used to perturb the word-level embedding representation E of the input sentence, and the resulting adversarial loss L vat as follows:

[0054]

[0055] E * =E+r

[0056] r=αq / ||q||2

[0057]

[0058] E′=E+ξd

[0059] Among them, D KL(·||·) represents the KL divergence, p(·|·) represents the model conditional probability, θ represents the current parameter constant, α and ξ represent hyperparameters, and d represents the parameter sampled from the normal distribution N(0,1). represents the gradient operation;

[0060] S3, the final loss is the entity extraction loss L ate , sentiment polarity discrimination loss L asc , against the loss L vat The final losses are jointly determined as follows:

[0061] L=L ate +L asc +L vat .

[0062] Compared with the prior art, the present invention has the following advantages:

[0063] 1. To measure the relationship between words in a sentence, based on the BERT-wwm-ext model representing the sentence, a multi-head self-attention mechanism is used to obtain matrices of different relationships between words, and representation fusion of different layers in the network is used to make the semantic representation of the sentence more comprehensive.

[0064] 2. For the deep representation of sentences based on inter-word relations, a multi-head self-attention mechanism is used to establish a relationship matrix based on inter-word relations, thereby establishing a word node representation of sentences in the GCN network to achieve deep semantic representation of sentences.

[0065] 3. For the fusion representation of entities and sentences, the obtained entities are used as query items, and a sentence representation of the fused entity information with a multi-head attention mechanism is established, so that the present invention is more targeted at the sentiment polarity in the sentence.

[0066] 4. A joint discrimination model of entities and emotions is designed, so that multiple entities and their viewpoints in a sentence can be effectively modeled, thereby improving the recognition performance of the present invention for entities and their emotional polarity.

[0067] 5. The present invention adopts a multi-task learning mechanism to train the model and adjust the shared parameters. It achieves good results in the deep representation of sentences, the interactive information representation of entities and emotions in sentences, and the prediction of entities and their emotional polarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is the overall hierarchical framework diagram of the financial entity and sentiment joint extraction method of the present invention (entity and sentiment joint extraction model based on multi-graph convolutional network). DETAILED DESCRIPTION

[0069] In order to further illustrate the technical solution of the present invention, the present invention is further described below through embodiments.

[0070] like Figure 1 As shown, a method for jointly extracting financial entities and emotions in this embodiment includes the following steps:

[0071] Step 1: Use the BERT-wwm-ext model to embed the given financial sentence into a representation H, as follows:

[0072] Represent the given financial sentence as “[CLS] sentence [SEP]” to obtain the word-level embedding representation H of the sentence [X] ∈R (n+2)×d , the formula is as follows:

[0073] H [1:X] =BERT(S)

[0074] Where S represents a given sentence, X represents the maximum number of encoding layers of BERT-wwm-ext, and H [X] The layer is used to complete the entity extraction task.

[0075] The BERT-wwm-ext model of this embodiment adopts shared BERT-wwm-ext, including a low layer, a middle layer and a high layer, wherein the low layer obtains phrase-level information, the middle layer obtains language information, and the high layer obtains task-related information.

[0076] Step 2: Take the embedding representation H obtained in step 1 as input and build a joint extraction model of financial entities and sentiments, including:

[0077] 1) Entity recognition: Using the self-attention mechanism, we can obtain the deep semantic representation of m different graph convolutional networks for financial sentences. That is, we get the deep semantics of the financial sentence H GCN , transform it linearly, and then establish the deep semantic representation H of the entity ete , specifically:

[0078] (1) Word attention score matrix: Using the self-attention mechanism, the words in the sentence are regarded as queries and keys to obtain the attention score matrix of each word in the sentence. The formula is as follows:

[0079]

[0080] Among them, W Q ∈R d×d and W K ∈R d×d represents the learnable parameter matrix, d represents the input node dimension, Q and K represent the Query and Key in the self-attention mechanism respectively, and T represents the transpose operation of the matrix;

[0081] (2) Node representation in graph convolutional networks: In graph convolutional networks, the hidden vector representation of the i-th word node in the l-th layer of the network is The formula is as follows:

[0082]

[0083] Among them, A ij W represents the degree of association between the i-th word node and the j-th word node of its neighbor on the graph convolutional network. l represents the weight matrix of the lth layer in the graph convolutional network, represents the hidden vector representation of the i-th word node in the l-1th layer of the network, b l represents the bias term of the lth layer, σ represents the ReLU activation function, and n represents the total number of layers in the graph convolutional network;

[0084] (3) Deep semantic representation of sentences based on graph convolutional networks: Using graph convolutional networks to obtain representations of all nodes in a sentence Finally, we get the deep semantic representation H of sentence S GCN (S), the formula is as follows:

[0085]

[0086] in, Represents the i-th word node of the sentence in the graph convolutional network, i is 1, 2, ..., n, m;

[0087] (4) Sentence fusion representation under multi-head attention mechanism: Based on the multiple word score matrices A of the sentence obtained by the multiple self-attention mechanisms in (1) above, k graph convolutional networks are established accordingly, and then the k graph networks are fused using linear transformation to obtain the deep semantic representation H of the entity. ete .

[0088] 2) Entity extraction: To obtain the deep semantic representation H of the entity ete , and then use the softmax function to obtain the label of whether each word is an entity;

[0089] 3) Interaction between entity sequence and sentiment representation sequence: Multi-head attention mechanism is used to analyze the deep semantic representation of entity ete Interact with the sentence embedding representation H to obtain the interactive sentence representation H s , specifically:

[0090] The deep semantic representation of the entity H ete As the query of the multi-head attention mechanism, the sentence is the key and value of the multi-head attention mechanism. The sentence of the i-th head attention is expressed as follows:

[0091]

[0092] in, and represents the learnable parameter matrix, m represents the number of attention heads, and d k =d v = d / m, Q, K and V represent Query, Key and Value in the attention mechanism respectively;

[0093] Sentence representation H through m-head attention s as follows:

[0094] H s =MultiHead(Q,K,V)=Concat(head1,...,head m )W O .

[0095] 4) Entity sentiment polarity discrimination: Use the sentence after interaction to represent H s , and then through the Transformer decoding layer, using the full connection operation, the sentiment discrimination sentence representation H is obtained esc , and then use the softmax function to obtain the entity’s sentiment polarity discrimination result, and finally realize the joint extraction of financial entity and sentiment polarity binary groups.

[0096] The training method of the financial entity and sentiment joint extraction model of this embodiment is as follows:

[0097] S1, for entity extraction, the deep semantic representation of k graph convolutional neural networks is transformed through linear transformation Perform linear fusion to obtain the deep semantic representation H of the entity ete , entity boundary discrimination probability p ete as follows:

[0098]

[0099] p ete =softmax(H ete )

[0100]

[0101] Among them, W i ete and b ete Represent the trainable weight matrix and bias parameters respectively;

[0102] The probability p of entity boundary discrimination ete , using the cross entropy objective function for joint training, we get the loss L for entity extraction ate , the formula is as follows:

[0103]

[0104] Among them, if the i-th position is the boundary of the entity, then otherwise

[0105] Before determining the entity sentiment polarity, use p ete Get the boundary of the entity [b ind :e ind ], the sentiment polarity discrimination of the joint entity, the process is as follows:

[0106] H esc =H ete W esc +b esc

[0107] o i =max(H esc [b ind :e ind ])

[0108]

[0109] Among them, H esc [b ind :e ind ] indicates from b ind To e ind Fragment of H esc , max represents the maximum pooling operator of the sequence dimension, W esc represents the trainable weight matrix, b esc represents the trainable bias parameter, represents the sentiment probability of the i-th entity boundary, w h Represents a trainable parameter weight matrix;

[0110] The sentiment discrimination loss L of the joint entity is obtained by the sentiment polarity discrimination probability of the entity asc , the formula is as follows:

[0111]

[0112] Among them, if the i-th word is the boundary of the entity, then otherwise

[0113] S2, using virtual adversarial training, input sentence word-level embedding representation E for perturbation, resulting in adversarial loss L vat as follows:

[0114]

[0115] E * =E+r

[0116] r=αq / ||q||2

[0117]

[0118] E′=E+ξd

[0119] Among them, D KL (·||·) represents the KL divergence, p(·|·) represents the model conditional probability, θ represents the current parameter constant, α and ξ represent hyperparameters, and d represents the parameter sampled from the normal distribution N(0,1). represents the gradient operation;

[0120] S3, the final loss is the entity extraction loss L ate , sentiment polarity discrimination loss L asc , against the loss L vat The final losses are jointly determined as follows:

[0121] L=L ate +L asc +L vat .

[0122] This embodiment provides some parameter settings and detailed descriptions during the model learning process.

[0123] (1) The model of this embodiment uses BERT-wwm-ext to encode sentences, the hidden layer vector dimension is 768, and the maximum sentence length is 500.

[0124] (2) The Adam optimization model was used with an initial learning rate of 0.000003 and a warm-up learning rate of 0.1.

[0125] (3) During the model training process, the batch size is set to 8 and the Epoch value is 15.

[0126] (4) In order to ensure stable results of the model under the same parameter configuration, the seed value of the parameter environment of the model in this embodiment is set to 1000.

[0127] (5) This embodiment uses a multi-head attention mechanism to interactively represent the generated entity sequence and sentence embedding representation, and the number of attention heads is 2.

[0128] (6) This embodiment uses a graph convolutional network to obtain the network representation between financial sentences, and the number of graph convolution layers is 2.

[0129] Example 2

[0130] This embodiment constructs a data set for sentiment analysis of entities in the financial field. Since the 19,148 annotated data are based on the entity level, and in the financial field there is often a situation where one financial statement corresponds to multiple financial entities, this embodiment sorts out the 19,148 entity-level data to obtain 12,434 financial statements. When studying the joint extraction task of <Financial Entity, Sentiment Polarity Discrimination>, in order to balance the proportion of positive, neutral, and negative instances among the training set, validation set, and test set, the data set tested in this embodiment is divided into a training set, a validation set, and a test set, and the ratio is 8:1:1. The statistics are shown in Table 1.

[0131] Table 1 Statistical results of entity sentiment analysis dataset in the financial field

[0132]

[0133] In order to test the performance of the present invention, comparative experiments were conducted with multiple models, as follows:

[0134] E2E-TBSA: This method designs two stacked LSTMs. The upper LSTM is used to predict a unified label to obtain the sentiment polarity of the target; the lower LSTM is used to assist in target boundary prediction to improve the performance of aspect item extraction.

[0135] DOER: This method uses two recurrent neural networks to handle the aspect item extraction and sentiment polarity prediction tasks respectively, and adopts a shared module to interact the hidden layer representations of the two parts to improve the performance of joint extraction of aspect items and sentiment polarity.

[0136] SPAN: This method uses a heuristic encoding algorithm to extract aspect items and then uses an attention mechanism to weight the data to determine the sentiment polarity of the aspects.

[0137] BERT-E2E-TBSA: This method uses the pre-trained model BERT for encoding representation and performs end-to-end joint extraction of aspect items and sentiment polarity.

[0138] GRACE: This method uses the cascade labeling algorithm to obtain the mutual representation between labels, and then solves the label imbalance problem through the gradient coordination algorithm.

[0139] This embodiment uses common evaluation indicators, namely precision P, recall R and F1 value. The method of the present invention and the above five comparison methods are compared in a self-built data set, and the experimental results are shown in Table 2.

[0140] Table 2 Comparative experimental results among six different methods

[0141]

[0142]

[0143] As can be seen from Table 2, the P-value and F1-value of the joint extraction of financial entities and sentiment polarity reached 66.55% and 67.27% respectively, which shows that the graph convolutional network is conducive to capturing the semantic relationship between financial entities and can more accurately determine the boundaries of financial entities, thereby improving the performance of the joint extraction of <financial entity, sentiment polarity>.

[0144] The method proposed in this invention outperforms all the comparison models in terms of precision P and F1 value. This advantage is mainly due to the following reasons:

[0145] (1) The entity and sentiment binary extraction method of the graph convolutional network proposed in the present invention is more suitable for the joint extraction of <financial entity and sentiment polarity> in the financial entity sentiment analysis dataset with long entity length and semantic relationship between entities.

[0146] (2) The graph convolutional network of the present invention is not only conducive to capturing the semantic relationship between financial entities, but also conducive to enhancing the accurate extraction of financial entities, and plays a guiding role in sentiment polarity, thereby improving the overall <financial entity, sentiment polarity> joint extraction performance.

Claims

1. A method for jointly extracting financial entities and sentiments, characterized in that: The following steps are involved: Step 1: Use the BERT-wwm-ext model to embed the given financial sentence into a representation H; Step 2: Take the embedding representation H obtained in step 1 as input and build a joint extraction model of financial entities and sentiments, including: 1) Entity representation: Using the self-attention mechanism, we obtain the deep semantic representation H of m different graph convolutional networks of financial sentences i GCN , i = 1, 2, ..., m, that is, the deep semantics H of the financial sentence is obtained GCN , transform it linearly, and then establish the deep semantic representation H of the entity ete ; 2) Entity extraction: To obtain the deep semantic representation H of the entity ete , and then use the softmax function to obtain the label of whether each word is an entity; 3) Interaction between entity sequence and sentiment representation sequence: Multi-head attention mechanism is used to analyze the deep semantic representation of entity ete Interact with the sentence embedding representation H to obtain the interactive sentence representation H s ; 4) Entity sentiment polarity discrimination: Using the sentence representation H after interaction s , and then through the Transformer decoding layer, using the full connection operation, the sentiment discrimination sentence representation H is obtained esc , and then use the softmax function to obtain the entity’s sentiment polarity discrimination result, and finally realize the joint extraction of financial entity and sentiment polarity binary groups.

2. A method for jointly extracting financial entities and emotions according to claim 1, characterized in that: The step 1 is specifically as follows: Represent the given financial sentence as "[CLS] sentence [SEP]" to obtain the word-level embedding representation H of the sentence [X] ∈R (n +2)×d , the formula is as follows: H [1:X] =BERT(S) Where S represents a given sentence, X represents the maximum number of encoding layers of BERT-wwm-ext, and H [X] The layer is used to complete the entity extraction task.

3. A method for jointly extracting financial entities and emotions according to claim 2, characterized in that: The BERT-wwm-ext model adopts shared BERT-wwm-ext, including a low layer, a middle layer and a high layer, wherein the low layer obtains phrase-level information, the middle layer obtains language information, and the high layer obtains task-related information.

4. A method for jointly extracting financial entities and emotions according to claim 1, characterized in that: In step 2, the deep semantic representation H of the entity is established. ete , specifically: (1) Word attention score matrix: Using the self-attention mechanism, the words in the sentence are regarded as queries and keys to obtain the attention score matrix of each word in the sentence. The formula is as follows: Among them, W Q ∈R d×d and W K ∈R d×d represents the learnable parameter matrix, d represents the input node dimension, Q and K represent the Query and Key in the self-attention mechanism respectively, and T represents the transpose operation of the matrix; (2) Node representation in graph convolutional networks: In graph convolutional networks, the hidden vector representation of the i-th word node in the l-th layer of the network is The formula is as follows: Among them, A ij W represents the degree of association between the i-th word node and the j-th word node of its neighbor on the graph convolutional network. l represents the weight matrix of the lth layer in the graph convolutional network, represents the hidden vector representation of the i-th word node in the l-1th layer of the network, b l represents the bias term of the lth layer, σ represents the ReLU activation function, and n represents the total number of layers in the graph convolutional network; (3) Deep semantic representation of sentences based on graph convolutional networks: Using graph convolutional networks to obtain representations of all nodes in a sentence Finally, we get the deep semantic representation H of sentence S GCN (S), the formula is as follows: in, Represents the i-th word node of the sentence in the graph convolutional network, i is 1, 2, ..., n; (4) Sentence fusion representation under multi-head attention mechanism: Based on the multiple word score matrices A of the sentence obtained by the k self-attention mechanisms in (1) above, k graph convolutional networks are established accordingly, and then the k graph networks are fused using linear transformation to obtain the deep semantic representation H of the entity. ete .

5. A method for jointly extracting financial entities and emotions according to claim 1, characterized in that: The sentence representation H obtained after interaction in step 2 s , specifically: The deep semantic representation of the entity H ete As the query of the multi-head attention mechanism, the sentence is the key and value of the multi-head attention mechanism. The sentence of the i-th head attention is expressed as follows: in, and represents the learnable parameter matrix, m represents the number of attention heads, and d k =d v = d / m, Q, K and V represent Query, Key and Value in the attention mechanism respectively; Sentence representation H through m-head attention s as follows: H s =MultiHead(Q,K,V)=Concat(head1,...,head m )W O 。 6. A method for jointly extracting financial entities and emotions according to claim 4, characterized in that: The training method of the financial entity and sentiment joint extraction model in step 2 is as follows: S1, for entity extraction, the deep semantic representation of k graph convolutional neural networks is transformed through linear transformation Perform linear fusion to obtain the deep semantic representation H of the entity ete , entity boundary discrimination probability p ete as follows: p ete =softmax(H ete ) in, and b ete Represent the trainable weight matrix and bias parameters respectively. Entity boundary discrimination probability p ete The cross entropy objective function is used for joint training to obtain the loss L for entity extraction. ate , the formula is as follows: Among them, if the i-th position is the boundary of the entity, then otherwise Before determining the entity sentiment polarity, use p ete Get the boundary of the entity [b ind :e ind ], the sentiment polarity discrimination of the joint entity, the process is as follows: H esc =H ete W esc +b esc o i =max(H esc [b ind :e ind ]) Among them, H esc [b ind :e ind ] indicates from b ind To e ind Fragment of H esc , max represents the maximum pooling operator of the sequence dimension, W esc represents the trainable weight matrix, b esc represents the trainable bias parameter, represents the sentiment probability of the i-th entity boundary, W h represents a trainable weight matrix; The sentiment discrimination loss L of the joint entity is obtained by the sentiment polarity discrimination probability of the entity asc , the formula is as follows: Among them, if the i-th word is the boundary of the entity, then otherwise S2, using virtual adversarial training, input sentence word-level embedding representation E for perturbation, resulting in adversarial loss L vat as follows: AND * =E+r r=αq / ||q||2 E′=E+ξd Among them, α and ξ represent hyperparameters, d represents the parameter sampled from the normal distribution N(0,1), θ represents the current parameter constant, and D KL (·||·) represents KL divergence, p(·|·) represents the model conditional probability, represents the gradient operation; S3, the final loss is the entity extraction loss L ate , sentiment polarity discrimination loss L asc , against the loss L vat The final losses are jointly determined as follows: L=L ate +L asc +L vat .

Images