Multi-feature financial domain named entity recognition method

Abstract

The present application is directed to a multi-feature financial domain named entity recognition method for some long-length, difficult-to-bound and various forms of expression named entities in the Chinese financial field. Firstly, the method divides entity boundary and entity classification as two independent tasks, introduces a multi-channel attention mechanism based on gate, divides the financial entity boundary on the basis of the pre-trained language model, and learns the enhanced Chinese character features. Then, the boundary division result is input into the pre-trained language model in the form of MASK for data enhancement. Subsequently, an enhanced feature based on the document-level entity is introduced to construct a financial entity classification model. Through experiments, the best-performing Chinese pre-trained language model is determined from several state-of-the-art models, which is then embedded into the method of the present application and compared with other benchmark models. The experimental results show that the proposed model is superior to other benchmark models in the named entity recognition task in the financial field.

Description

Technical Field

[0001] This invention relates to the field of named entity recognition technology, and more specifically to a multi-feature named entity recognition method in the financial field. Background Technology

[0002] Named Entity Recognition (NER) is an important task in information extraction, aiming to locate and classify named entities in text into predefined categories, such as people, organizations, locations, time expressions, quantities, currency values, percentages, etc. Text from different domains has different characteristics. In the financial field, in addition to recognizing names, locations, and times, text also includes financial entities belonging to specific professional fields, such as company names, company abbreviations, product names, project names, and financial terminology. In general, the new challenges of Chinese NER in the financial field mainly focus on the following two points: 1) The forms of expression for financial entities are diverse. For example, the same institution may be expressed in various ways, extracting individual characters from its full name to form different abbreviations, and even using online nicknames, such as "Tencent Technology (Shenzhen) Co., Ltd." which is abbreviated as "Tencent," and also has nicknames like "Goose Factory," "Goose," or "Nanshan Invincible." 2) Entities have strong domain characteristics, such as product type entities like "hedge fund" and "white knight." Furthermore, financial texts often contain entity names composed of a mixture of Chinese, English, and numbers, making it difficult for word segmentation tools to accurately segment the data.

[0003] Existing Named Entity Recognition (NER) methods mainly include rule-based and dictionary-based methods, traditional machine learning-based methods, and deep learning-based methods. Rule-based and dictionary-based methods rely on manually defined rules. When the dictionary size is limited, rule-based NER systems can achieve good results. However, due to domain-specific rules and incomplete dictionaries, these NER systems are characterized by high precision but low recall. Furthermore, similar systems are difficult to transfer to other domains; for new domains, rules need to be redefined, and different domains have different dictionaries, requiring significant human effort and making it difficult to scale to other entity types or datasets. Machine learning-based methods also heavily rely on corpora, but large-scale general-purpose corpora suitable for building and evaluating named entity recognition systems are relatively few. Deep learning-based methods are susceptible to the impact of excessively large training models, leading to reduced efficiency. Summary of the Invention

[0004] This invention provides a multi-feature named entity recognition method for the financial field, in order to solve the technical problems of low efficiency and poor accuracy of existing named entity recognition methods for text recognition in the financial field.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] Design a multi-feature named entity recognition method in the financial field, including:

[0007] This method consists of two parts: the first part is a financial entity boundary partitioning model, and the second part is a financial entity classification model. The reason for dividing the named entity recognition task into two sub-tasks is primarily because: 1) some financial entities are quite long, such as company names, and using a separate boundary partitioning model helps to partition the boundaries of long entities; 2) for Chinese, word-based MASK (Mask Unit) learning yields better feature representations than single-character MASK. Furthermore, to better address the issue of diverse representations of financial entities, this invention incorporates word-level feature representations into the entity classification model. Similarly, to reduce computational complexity, this invention does not learn features from all words in the document, but instead, through preprocessing, selects only a few sentences with high relevance to the current word for learning. Specifically, as follows: Assume there is a document... ,in, Indicates the first in the document One sentence. Indicates the composition of sentences The method proposed in this invention requires first predicting the characters in the sentence. Entities in ,

[0008] (1)

[0009] Next, put the sentence Using a single entity as a mask unit, the data is fed back into the pre-trained language model to predict the label sequence using CRF. ,

[0010] (2)

[0011] Where MA() represents the multichannel attention module, and S represents the... The set of n-grams appearing in the dictionary; GATE() represents a gating mechanism. This indicates that the global characteristics of the entity in the document are retrieved; more details will be provided below.

[0012] (1) Financial entity boundary delineation model

[0013] This invention proposes an entity boundary partitioning model that can be adapted to the financial field. This model introduces a gating mechanism that can flexibly adjust the samples entering the multi-channel attention module.

[0014] Will As the input sequence for the BERT model, assuming the current character The feature representation learned through the Bert model is as follows: It is passed to the CRF model to calculate about Conditional distribution probability of joint labels Through a normalized entropy That is, to judge To determine whether the output meets expectations, the following formula is used:

[0015] (3)

[0016] Here, Y represents the total number of sequence labels.

[0017] Definition 1: Do we need to change the characters? The threshold assigned to the multichannel attention model.

[0018] if If the result obtained by the CRF model is correct, then the result is the expected result; otherwise, it will be incorrect. Feature representation obtained through BERT The process is handled by a multi-channel attention mechanism. For some complex financial entities, their length far exceeds 6, while others are relatively short. Therefore, this invention improves the matching method, first by using the target character... In the dictionary, the unit is The set of word groups is obtained through n-gram matching. Not every word group All words contain words; therefore, this invention sets n to the maximum length of words in the dictionary, for empty word groups. The corresponding attention channels are masked in this invention and not involved in the computation. Therefore, the multi-channel attention mechanism can be simplified as follows:

[0019] (5)

[0020] (6)

[0021] (7)

[0022] here, For each non-empty word group The words contained therein It is a word The characteristics are represented. Word groups The number of words contained in it.

[0023] Then get information about the characters New feature representation ,

[0024] (8)

[0025] This represents the training weights. Then, the resulting new feature representations are fed into the CRF model to predict the labels.

[0026] (2) Financial entity classification model

[0027] Next, the word segmentation results are used as input to a pre-trained language model. Here, "pre-trained language model" refers to various pre-trained language models based on the transformer, such as BERT and ERNIE. At this point, the original sentence... Composed of entities and general words Composition (hereinafter referred to as words). Meanwhile, to better address the diversity of entity representations in the financial field, simply using the context of the current sentence is insufficient; it may be necessary to consider the global features of the current word at the level of the entire document. Given the superior performance of BiLSTM and attention in learning long-text representations, this invention will... While inputting words as masks into the pre-trained language model, it also... The document in question The BiLSTM layer takes words as input and uses an attention layer to calculate the current word. The relevance of the words to other words in the document is used to calculate the relevance. Global feature representation .

[0028] However, if all words in document D are included in the calculation at this point, it will undoubtedly introduce more noise and may lead to the loss of important features. Therefore, this invention preprocesses the text, retaining only the words in the document that are relevant to the current sentence. More relevant sentences are included in the calculation. To reduce reliance on training data, this invention chooses an unsupervised method to automatically extract relevant sentences from the document. Related sentences. TextRank is a method frequently used in unsupervised extraction of key sentences from documents. It treats each sentence in the text as a node, and if two sentences are similar, then an undirected weighted edge is considered to exist between the nodes corresponding to these two sentences. The sentence is then evaluated using a weighted edge algorithm. The formula for weighting in text is as follows:

[0029] (9)

[0030] Here, d represents the damping coefficient. and It represents the set of sentences that point to the given sentence, as well as the set of sentences that lead from that sentence to other sentences. It represents a sentence and The similarity weight.

[0031] Because the purpose of this invention is to find text that is related to the current sentence. Instead of extracting summaries from sentences with high relevance, this invention improves upon the original method by using the Pearson correlation coefficient instead of similarity. If two sentences are related, an edge is considered to exist between their corresponding nodes, with the edge weight equal to the Pearson correlation coefficient values ​​of the two sentences. After the relevant graph is constructed, the sentences in the graph are calculated. With the current sentence Relevant scores .

[0032] (10)

[0033] (11)

[0034] Sentence The characteristic representation of , where cov represents covariance. The standard deviation is represented. The sentences in the graph are sorted by score, and the top k sentences are selected as... Closely related sentences form a set These sentences are divided into word units. As input to the BiLSTM. Let the current word be the sentence. words in Then we can obtain information about Feature representation :

[0035] (12)

[0036] It is used as input to the Attention layer, which is mainly used to calculate the current word. With other words in set S The relevance of the Attention weight value It can be represented as:

[0037] (13)

[0038] (14)

[0039] At this point, the word can be derived. A global feature representation based on the document level ,

[0040] (15)

[0041] word Output obtained from the Attention layer It can be represented as:

[0042] (17)

[0043] This feature and Feature representation obtained through a pre-trained language model Combined together, we get information about words. New feature representation

[0044] (18)

[0045] Will The CRF is passed to the upper layer as input, and the final classification result is output.

[0046] Compared with the prior art, the beneficial technical effects of the present invention are as follows:

[0047] 1. This paper proposes a multi-feature named entity recognition method for the financial domain based on a pre-trained language model. Addressing existing problems in financial domain named entity recognition, this method treats the delineation of financial entity boundaries as an independent task, introduces domain expert knowledge to learn domain-enhanced feature representations of characters, and then uses the output as a mask unit to re-input into the pre-trained language model. Furthermore, document-level global features are incorporated as feature representations for the financial domain named entity recognition task. Experiments demonstrate that this method achieves better performance compared to other benchmark models.

[0048] 2. This paper proposes a model for entity boundary partitioning that can be adapted to the financial field. The model introduces a gating mechanism, flexibly adjusts the samples entering the multi-channel attention module, and simplifies and improves the multi-channel attention module, making it flexible to adapt to different situations, avoiding redundant calculations, and improving the efficiency of the model.

[0049] 3. A text-level global word feature extraction method is proposed for long texts to solve the problems of feature loss and noise introduction caused by excessive text length. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the recognition model in this invention.

[0051] Figure 2 This is a schematic diagram of N-gram matching in this invention.

[0052] Figure 3 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0053] The specific embodiments of the present invention will be described below with reference to the accompanying drawings and examples. However, the following examples are only used to illustrate the present invention in detail and do not limit the scope of the present invention in any way.

[0054] Example 1: A multi-feature named entity recognition method in the financial field, see [link to example]. Figures 1 to 3 The dataset used is divided into two parts. The first part is mainly used to compare the running time and performance of the proposed word segmentation model on the dataset. The dataset used here is the public dataset CTB8.0, referred to as Dataset 1. The second part is mainly used to compare the performance of the model in this invention with other benchmark models on the named entity recognition task in the financial field. This part of the dataset mainly comes from the financial field, including news from various financial websites, company financial reports, financial short messages on social media, and financial article summaries. 7521 sentences were extracted from them (including 50 summaries, 30 financial news articles, 40 financial short messages, and 40 company financial reports). Six entity types were manually labeled: person_name (person name, 5141), product_name (product name, 7222), company_name (company name, 5475), location (location, 6597), org_name (organization name, 5689), and time (time, 4250), referred to as Dataset 2.

[0055] Take a text from the dataset As the input sequence for the BERT model, text Contains multiple characters In this embodiment To establish the "China Foreign Exchange Trading Center," the final step is to identify the organization name "China Foreign Exchange Trading Center." Assuming the current character... The feature representation learned through the Bert model is as follows: It is passed to the CRF model to calculate about Conditional distribution probability of joint labels Through a normalized entropy That is, to judge To determine whether the output meets expectations, the following formula is used:

[0056] (1)

[0057] Where Y represents the total number of sequence tags.

[0058] definition Do we need to change the characters? The threshold assigned to the multi-channel attention model. If If the result obtained by the CRF model is correct, then the result is the expected result; otherwise, it will be incorrect. Feature representation obtained through BERT The solution is handled by a multi-channel attention mechanism using N-grams. This invention simplifies existing multi-channel attention mechanisms, which require all characters in a sentence to be viewed in a dictionary. In this process, n-gram matching is performed, and then all matched words are grouped according to n, and then fed into the corresponding attention channel to calculate the weights.

[0059] (2)

[0060] It represents a sentence Each character in the dictionary Word groups matched by n-gram Each word group Contained words Feature representation, Used to represent the current character Is it a word? Part of it, if so =1, otherwise, =0. This means that if... Not by Phrases formed from these phrases, then in learning Contextual features are ineffective in this approach, and when sentences are long, this matching method consumes a significant amount of time on character matching, with the matched words potentially being of little use. Furthermore, existing attention mechanisms fix n to a constant value, but in reality, the length of some complex financial entities far exceeds this set value, while others are relatively short. Therefore, this invention improves the matching method by first using the target character... In the dictionary, the unit is The set of word groups is obtained through n-gram matching. ,like Figure 2 As shown, not every word group All words contain words; therefore, this invention sets n to the maximum length of words in the dictionary, for empty word groups. The corresponding attention channels are masked and excluded from computation. Therefore, the multi-channel attention mechanism can be simplified as follows:

[0061] (3)

[0062] (4)

[0063] (5)

[0064] here, For each non-empty word group The words contained therein It is a word The characteristics are represented. Word groups The number of words contained in it.

[0065] Then get information about the characters New feature representation ,

[0066] (6)

[0067] The training weights are then used to generate the new feature representations, which are then fed into the CRF model to predict the labels.

[0068] Next, the word segmentation results are used as input to the pre-trained language model. At this point, the original sentence... Composed of entities and general words Composition. Furthermore, to better address the diversity of entity representations in the financial field, simply using the context of the current sentence is insufficient; it's necessary to consider the global features of the current word from the perspective of the entire document. Therefore, this invention will... While inputting words into the pre-trained language model, it also... The document in question The BiLSTM layer takes words as input and uses an attention layer to calculate the current word. The relevance of the words to other words in the document is used to calculate the relevance. Global feature representation .

[0069] However, if all words in document D are included in the calculation at this point, it will undoubtedly introduce more noise and may lead to the loss of important features. Therefore, this invention preprocesses the text, retaining only the words in the document that are relevant to the current sentence. More relevant sentences are included in the calculation. To reduce reliance on training data, an unsupervised method is chosen to automatically extract relevant sentences from the document. Related sentences TextRank is a method frequently used in unsupervised extraction of key sentences from documents. It treats each sentence in the text as a node, and if two sentences are similar, then an undirected weighted edge is considered to exist between the nodes corresponding to these two sentences. The text then calculates the sentence's similarity. The formula for weighting in text is as follows:

[0070] (7)

[0071] Here, d represents the damping coefficient. and It represents the set of sentences that point to the given sentence, as well as the set of sentences that lead from that sentence to other sentences. It represents a sentence and The similarity weight.

[0072] The purpose here is to find the text that is related to the current sentence. Instead of extracting summaries from sentences with high relevance, this method is improved upon by using the Pearson correlation coefficient instead of the original similarity metric. If two sentences are related, an edge is considered to exist between their corresponding nodes, with the edge weight equal to the Pearson correlation coefficient values ​​of the two sentences. After the relevant graph is constructed, the sentences in the graph are calculated. With the current sentence Relevant scores .

[0073] (8)

[0074] (9)

[0075] Sentence The characteristic representation of , where cov represents covariance. The standard deviation is represented. The sentences in the graph are sorted by score, and the top k sentences are selected as... Closely related sentences form a set These sentences are divided into word units. As input to the BiLSTM. Let the current word be the sentence. words in Then we can obtain information about Feature representation :

[0076] (10)

[0077] It is used as input to the Attention layer in the BiLSTM model, where the Attention layer is mainly used to calculate the current word. With other words in set S The relevance of the Attention weight value It can be represented as:

[0078] (11)

[0079] (12)

[0080] At this point, the word can be derived. A global feature representation based on the document level ,

[0081] (13)

[0082] word Output obtained from the Attention layer It can be represented as:

[0083] (14)

[0084] This feature and Feature representation obtained through a pre-trained language model Combined together, we get information about words. New feature representation

[0085] (15)

[0086] Will The CRF (Confirmation Rendering Framework) is passed to the upper layer as input, and the final classification result is output. Taking the model diagram as an example, the output is the words "China Foreign Exchange Trading Center" and "established." Ultimately, "China Foreign Exchange Trading Center" is identified as the entity, which is the organization name. "Established" is not an entity and does not need to be identified.

[0087] The aforementioned named entity recognition method consists of two parts: a financial entity boundary partitioning model and a financial entity classification model. A neural network model with n-gram multi-channel attention enhancement was proposed, based on the BERT model, for Chinese word segmentation tasks. However, in reality, for ordinary Chinese word segmentation, BERT's accuracy can reach approximately 96%, meaning that not all words need to pass through the multi-channel attention enhancement module. Therefore, this invention proposes an entity boundary partitioning model adaptable to the financial field. This model introduces a gating mechanism, which can flexibly adjust the samples entering the multi-channel attention module. Next, the word segmentation results are used as input to a pre-trained language model. At this point, the original sentence... Composed of entities and general words Composition (hereinafter referred to as words). Meanwhile, to better address the diversity of entity representations in the financial field, simply using the context of the current sentence is insufficient; it may be necessary to consider the global features of the current word at the level of the entire document. Given the superior performance of BiLSTM and attention in learning long-text representations, this invention will... While inputting words as masks into the pre-trained language model, it also... The document in question The BiLSTM layer takes words as input and uses an attention layer to calculate the current word. The relevance of the words to other words in the document is used to calculate the relevance. Global feature representation Combine this feature with Feature representation obtained through a pre-trained language model Combined together, we get information about words. New feature representation Finally The CRF is passed to the upper layer as input, and the final classification result is output. This approach is not only more suitable for the financial field, but also improves model efficiency.

[0088] The present invention has been described in detail above with reference to the accompanying drawings and embodiments. However, those skilled in the art will understand that, without departing from the spirit of the present invention, various specific parameters in the above embodiments can be changed to form multiple specific embodiments, all of which are common variations of the present invention, and will not be described in detail here.

Claims

1. A multi-feature named entity recognition method in the financial field, characterized in that, This method consists of two parts: the first part is the financial entity boundary partitioning model, and the second part is the financial entity classification model: (1) Financial entity boundary partitioning model: This model introduces a gating mechanism, which can flexibly adjust the samples entering the multi-channel attention module; As the input sequence for the BERT model, assuming the current character The feature representation learned through the Bert model is as follows: It is passed to the CRF model to calculate about Conditional distribution probability of joint labels Through a normalized entropy That is, to judge To determine whether the output meets expectations, the following formula is used: ， Here, Y represents the total number of sequence tags; The multichannel attention mechanism is defined as follows: To determine whether characters need to be... The threshold assigned to the multi-channel attention model, if If the result obtained by the CRF model is correct, then the result is the expected result; otherwise, it will be incorrect. Feature representation obtained through BERT Entrust the task to a multi-channel attention mechanism; for some complex financial entities, first, focus on the target character. In the dictionary, the unit is The set of word groups is obtained through n-gram matching. Set n to the maximum length of words in the dictionary; for empty word groups... If the corresponding attention channels are masked and not involved in the computation, the multi-channel attention mechanism can be simplified as follows: here, For each non-empty word group The words contained therein It is a word Feature representation, Word groups The number of words contained in it, and then get information about the characters. New feature representation , The training weights are then used to generate the new feature representations, which are then fed into the CRF model to predict the labels. (2) Financial entity classification model: The word segmentation result is used as the input of the pre-trained language model. At this time, the original sentence Composed of entities and general words In addition, to better address the diversity of entity representation in the financial sector, [the following will be implemented / constructed]. While inputting the word "Mask" into the pre-trained language model, it also... The document in question The BiLSTM layer takes words as input and uses an attention layer to calculate the current word. The relevance of the words to other words in the document is used to calculate the relevance. Global feature representation .

2. The multi-feature named entity recognition method in the financial field according to claim 1, characterized in that, The financial entity classification model first preprocesses the text, retaining only the parts of the document that are relevant to the current sentence. More relevant sentences are involved in the calculation, and then a BiLSTM layer and an attention layer are introduced to... While inputting words into the BERT model, it also... The document in question The BiLSTM layer takes words as input and uses an attention layer to calculate the current word. The relevance of the words to other words in the document is used to calculate the relevance. Global feature representation .

3. The multi-feature named entity recognition method in the financial field according to claim 2, characterized in that, Text preprocessing uses an unsupervised method to automatically extract text from the document containing... Related sentences, using TextRank, treat each sentence in the text as a node. If two sentences are similar, then an undirected weighted edge exists between the nodes corresponding to these two sentences. The sentence is then calculated... The formula for weighting in text is as follows: Here, d represents the damping coefficient. and It represents the set of sentences that point to the given sentence, as well as the set of sentences that lead from that sentence to other sentences. It represents a sentence and Similarity weights Next, the Pearson correlation coefficient is used to replace the similarity in the original method. If there is a correlation between two sentences, then it is assumed that there is an edge between the nodes corresponding to the two sentences, and the weight of the edge is the Pearson coefficient value of the two sentences. After the relevant graph is constructed, the sentences in the graph are calculated. With the current sentence Relevant scores , Sentence The characteristic representation of , where cov represents covariance. To represent the standard deviation, the sentences in the correlation graph are sorted by score, and the top k sentences are selected as... Closely related sentences form a set These sentences are divided into word units. As input to BiLSTM, let the current word be a sentence. words in Then we can obtain information about Feature representation : 。 4. The multi-feature named entity recognition method in the financial field according to claim 3, characterized in that, Features obtained in financial entity classification models As input to the Attention layer, the Attention layer is used to calculate the current word. With other words in set S The relevance of the Attention weight value It can be represented as: At this point, the word can be derived. A global feature representation based on the document level , ， word Output obtained from the Attention layer It can be represented as: ， This feature and Feature representation obtained through a pre-trained language model Combined together, we get information about words. New feature representation ， Will The CRF is passed to the upper layer as input, and the final classification result is output.

Images