A named entity recognition method for a financial product intelligent recommendation system

Abstract

A named entity recognition method for intelligent recommendation systems of financial products is proposed. Based on existing datasets, a BERT pre-trained model is fine-tuned to predict other unlabeled data within the domain, and a confidence screening mechanism is used to select reliable samples. More data is then fed into subsequent training. Word vector features are generated for each sample using the BERT pre-trained model. Based on the word vector features, multi-scale convolution is used to obtain the sample sentence features. Furthermore, a FLAT-BERT model is used to generate word features for each sample. The sentence features and word features are concatenated with equal weights to obtain the overall features. A linear network is then used to identify named entities in the text based on the overall features. This invention can fully utilize product information text to accurately extract named entities from product information. It achieves excellent results on publicly available named entity recognition data and has good practicality in intelligent recommendation systems with high accuracy requirements.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and artificial intelligence, specifically to a named entity recognition method for intelligent recommendation systems for financial products. Background Technology

[0002] Named entity recognition (NER) refers to algorithms that extract predefined entities from given text. How to enable computers to accurately encode the semantic information of a sentence is a key problem that intelligent recommendation systems need to solve. In intelligent recommendation systems for financial products, basic elements of the financial products can be extracted using NER technology, such as organization names, dates and times, personal names, place names, and proper nouns. Extracting entities from product information is of great significance for financial product recommendations in the era of intelligent recommendation, enabling more accurate content mining and effective content supervision. NER belongs to the field of information extraction, and as an important branch of information extraction, its task is to label entities in sentences, which is of crucial significance. As the basic unit of text, it contains a large amount of semantic information and plays an important role in encoding the semantic information of product information.

[0003] Traditional supervised learning methods based on feature engineering require manual feature extraction by humans when solving named entity recognition, which wastes a lot of manpower. Furthermore, human experience may lead to errors in the feature extraction process, resulting in the propagation of errors and limiting the effectiveness of the method.

[0004] With the development of deep learning in the field of natural language processing, more and more named entity recognition (NAME) methods using deep learning have been proposed, such as the named entity recognition method for the auditing domain based on adversarial training disclosed in CN114462409A, and the method and apparatus for named entity recognition disclosed in CN111291565A. These methods require converting sentences into word vectors and performing some calculations through deep learning models. The named entity recognition task can be transformed into a token classification problem. The existing financial field suffers from limited data volume, and deep learning models are highly dependent on reliable data. Limited data makes it difficult for deep learning models to converge, resulting in insufficient accuracy. Recent advances in natural language representation have made it possible to transfer the internal state of upstream trained models to downstream tasks. To encode each token with more accurate semantics, incorporating more accurate semantic information into the tokens has received widespread attention. Simultaneously, in the case of limited existing data, introducing more reliable data to improve model performance is of profound significance. However, traditional methods using sequence labeling to solve named entity recognition tasks do not incorporate more dimensions of information into the tokens, leading to insufficient accuracy in named entity recognition, and this problem is more pronounced in the domain of limited data. Summary of the Invention

[0005] To overcome the problems of insufficient accuracy and underutilization of pre-trained model performance in existing named entity recognition methods, this paper proposes a named entity recognition method for intelligent recommendation systems of financial products. This method can accurately identify entities in product information sentences in financial product recommendation systems, achieve excellent results on publicly available named entity recognition data, and has good practicality in intelligent recommendation systems of financial products where high accuracy is required.

[0006] Technical Solution: To achieve the above objectives, this invention provides a named entity recognition method for intelligent recommendation systems of financial products, comprising the following steps:

[0007] S1: Fine-tune the BERT pre-trained model based on the existing dataset, and use the trained model to predict other unlabeled data collected in the domain. For each unlabeled sample, obtain the feature matrix E. pre ;

[0008] S2: According to E pre The confidence level of the samples is obtained by using a confidence level screening mechanism; and samples with a confidence level greater than the threshold are used as reliable samples and are put into subsequent training along with the original samples.

[0009] S3: Construct a lattice structure for each sample to generate samples with lattice structures; obtain the original sample vector s1 and the lattice sample s2;

[0010] S4: The vector s2 is fed into the FLAT-BERT model as input. After word embedding, position embedding, and weight calculation, the feature representation matrix E incorporating word features is obtained. flat ;

[0011] S5: Input vector s1 into the BERT pre-trained model, and after word embedding, position embedding, and weight calculation operations, obtain the word vector feature representation matrix E. bert ;

[0012] S6: Transform matrix E bert We obtain vector l by performing convolution and pooling operations respectively, and then expand l to be the vector E. flat With the same shape, the feature representation matrix E of the sample sentence is obtained. sentence E flat E sentence The sample population feature representation matrix E is obtained by concatenating the components horizontally in an equally weighted manner. context ;

[0013] S7: Using the linear transformation matrix T predict For E contextPerform a linear transformation to obtain the named entity to which each token belongs; the extracted entity is the recognition result.

[0014] Furthermore, the word embedding, position embedding, and weight calculation in step S1 require the following operations:

[0015] E embedding =T embedding s1

[0016]

[0017]

[0018] E input =E embedding +PE

[0019] E pre =Self-Attention(E) input )

[0020] Where s1 is the sample vector, T embedding It is a word vector matrix, using T embedding A linear transformation of s1 yields the word vector matrix E. embedding Next, linear transformations of the sin and cos functions are used for position encoding, where PE (pos，2i) and PE (pos，2i+1) The positional encodings represent the odd and even positions in the word vectors, respectively, resulting in the positional encoding matrix PE; PE and E are then compared... embedding The final model input E is obtained by adding the corresponding positions. input ; For E input The sample features E are obtained using the Self-Attention algorithm in BERT. pre .

[0021] Furthermore, the confidence screening mechanism in step S2 is based on E pre Perform the following calculations in sequence:

[0022] E confidence =T confidence E pre

[0023] P confidence =Softmax(E confidence )

[0024] I confidence =max(P confidence )

[0025]

[0026] P sample =min(P entity1 P entity2 , ..., P entityn )

[0027] You need to go through T first confidence For E pre Perform a linear transformation to obtain E confidence E confidence T represents the score by which each token is assigned to each named entity. confidence For E confidence The matrix to be linearly transformed; in order to obtain the confidence matrix E confidence Represented as probability, E is analyzed using the Softmax method. confidence Perform the operation, where Softmax is applied to the feature matrix E. confidence The feature vector of each token is normalized to obtain the confidence level I of each token. confidence , among which, I confidence This represents the score for each token category; based on the confidence level of each token i. Then calculate the confidence level P for each predicted named entity. entity A named entity may contain multiple tokens. The confidence level P of the named entity is calculated based on the tokens it contains. entity ;P entityj Let represent the confidence level of the j-th named entity. Finally, the confidence level P of the sample is calculated based on the confidence levels of multiple named entities. sample .

[0028] Furthermore, in step S3, the method for generating samples containing lexical features is to treat a word as a token and use a head pointer and a tail pointer to identify the starting position of the word in the text.

[0029] Furthermore, for a single character, its head pointer and tail pointer are the same.

[0030] Furthermore, the specific calculation process for word embedding, position embedding, and weight calculation operations in step S4 is as follows:

[0031] E fembedding =T fembedding s2

[0032]

[0033]

[0034]

[0035]

[0036]

[0037]

[0038]

[0039] E finput =E fembedding +R

[0040] E flat =Attention(E finput )

[0041] The lattice structure consists of spans of different lengths. For two spans x in the lattice... i and x j There are three relationships between them: intersection, containment, and separation, and the relationship between them is determined by their head and tail; let head[i] and tail[i] represent x. i The index positions of the head and tail are represented by head[j] and tail[j], respectively. j The index positions of the head and tail; x represents i Header index position and x j The distance from the header index position; x represents i Head index position and x j The distance from the tail index position; x represents i The tail index position and x j The distance from the header index position; x represents i The tail index position and x j The distance to the tail index position; the relative position of the span is encoded as a simple linear transformation of four distances, where W r It is a learnable parameter. P represents the connection operator. d The calculation method also uses sin and cos. R and E fembedding The final model input E is obtained by adding the corresponding positions. finput ; For E finput The sample features E are obtained using the self-attention algorithm in FLATBERT. flat .

[0042] Furthermore, in step S6, matrix E bertAfter obtaining vector l through convolution and pooling operations at multiple scales, matrix E is then obtained. sentence With E context The specific calculation process is as follows:

[0043] A = Conv(E) bert )

[0044] A = Relu(A)

[0045] l = MaxPooling(A)

[0046] E sentence =expand(l)

[0047] E context =concat(E flat E sentence )

[0048]

[0049] First, E bert Matrix A is obtained by applying three 2D convolution kernels of different scales, where the lengths of the kernels are 2, 3, and 4, and the width is E. bert The width of the vector is determined, and after convolution by the three kernels, ReLU activation and MaxPooling operations are performed respectively. Finally, the vectors obtained from the three scales are horizontally concatenated to obtain vector l. Then, l is expanded to the width of E. flat The same shape yields the sentence feature representation matrix E. sentence Two feature representation matrices E are concatenated using a horizontal concatenation method. flat E sentence Obtain the context feature representation matrix E context concat is a horizontal concatenation operation.

[0050] Furthermore, in step S7, a linear transformation matrix T is used. predict For E context The specific calculation process for performing a linear transformation is as follows:

[0051] I pred =softmax(E context T predict )

[0052]

[0053] First, use matrix T. predict With the sample population feature representation matrix E context Multiply the results and perform a softmax operation to obtain the probability matrix I of all tokens belonging to each entity category.pred , where softmax is a normalization operation on the vector; for I pred Use argmax to get the result of each token belonging to each entity category.

[0054] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0055] 1. This invention addresses the problem of insufficient training data in specific fields, which leads to difficulty in model convergence. Based on the confidence screening method, more reliable samples are added to the original data to be used in subsequent model training, thereby improving the accuracy of named entity recognition.

[0056] 2. This invention utilizes multi-scale convolutional operations to encode the BERT output matrix, obtaining the feature representation of the entire sentence, and using it as the global feature of each token. Furthermore, it employs a horizontally concatenated method with equal weights to fuse the sentence features into the word features, ensuring that each token, during subsequent classification, simultaneously considers the character, word, and sentence information of the text, guaranteeing both efficiency and accuracy in recognition.

[0057] 3. This invention fully considers the information of words, phrases, and sentences in product information text and obtains more reliable external data through confidence level filtering. It can achieve excellent results on publicly available data for named entity recognition and has good practicality in intelligent recommendation systems for financial products with high accuracy requirements. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the identification method described in an embodiment of the present invention. Detailed Implementation

[0059] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.

[0060] like Figure 1 As shown, this invention provides a named entity recognition method for an intelligent recommendation system for financial products, with the following specific steps:

[0061] S1: Fine-tune the BERT pre-trained model based on the existing dataset. Train the BERT model through word embedding, position embedding, and weight calculation operations. Use the trained model to predict other unlabeled data collected in the domain. Obtain the feature matrix E for each unlabeled sample. pre Here, we take an unlabeled product information text from a financial product information recommendation system: m1 = "Price lower than the lowest price in the research report rating" as an example. The goal here is to predict m1. We input m1 into the BERT model and use T... embeddingA linear transformation is performed on the vector s1 encoded by m1 to obtain the word vector matrix E. embedding Next, linear transformations of the sin and cos functions are used for position encoding, where PE... (pos,2i) and PE (pos,2i+1) Let E represent the positional encodings of odd and even positions in the word vector, respectively, to obtain the positional encoding matrix PE. Then, PE and E... embedding The final model input E is obtained by adding the corresponding positions. input For E input The sample features E are obtained using the Self-Attention algorithm in BERT. pre :

[0062] [[-5.2552e-01,8.9774e+00,…,-1.5621e+00,-1.3391e+00,8.3995e-02],

[0063] [-1.6414e+00, 7.0795e-02,..., -2.6681e+00, -2.4268e+00, 8.4805e+00],

[0064] …,

[0065] [-1.6657e+00, -1.0633e-02,..., -1.3841e+00, -2.6671e+00, 8.4881e+00]]

[0066] S2: According to E in step S1 pre A confidence level screening mechanism is used to obtain the confidence level of the samples. Samples with a confidence level greater than a threshold are considered reliable samples and are used in subsequent training along with the original samples. This requires first passing a T... confidence For E pre Perform a linear transformation to obtain E confidence E confidence T represents the score by which each token is assigned to each named entity. confidence For E confidence The matrix to be linearly transformed. To obtain the confidence matrix E... confidence Represented as probability, E is analyzed using the Softmax method. cinfidence Perform the operation, where Softmax is applied to the feature matrix E. confidence The feature vector of each token is normalized to obtain the confidence level I of each token. confidence , among which, I confidence This represents the score for each token category; based on the confidence level of each token i. Then calculate the confidence level P for each predicted named entity. entity A named entity may contain multiple tokens. The confidence level P of the named entity is calculated based on the tokens it contains. entity ;P entityj Let represent the confidence level of the j-th named entity. Finally, the confidence level P of the sample is calculated based on the confidence levels of multiple named entities. sample : [0.93312]

[0068] E confidence =T confidence E pre

[0069] P confidence =Softmax(E confidence )

[0070] I confidence =max(P confidence )

[0071]

[0072] P sample =min(P entity1 P entity2 , ..., P entityn )

[0073] 0.93312 indicates that the sample is likely to be used for the next stage of training. With a threshold of 0.8, the confidence level of the sample is greater than the threshold, so it can be used as a reliable sample and added to the original labeled data.

[0074] S3: Construct a lattice structure for each sample to generate samples with a lattice structure. Taking m2 = "The meaning of net asset value per share" as an example, the goal is to extract named entities from m2. This yields the original sample vector s1 and the lattice sample vector s2.

[0075] s1:[101,3680,5500,…,2692,721]

[0076] s2:[101,3680,5500,…,6598,772]

[0077] S4: Pass vector s2 as input to the FLAT-BERT model, and then use head[i] and tail[i] to represent x. i The index positions of the head and tail are represented by head[j] and tail[j], respectively. j The index positions of the head and tail. x represents i Head index position and xj The distance from the header index position; x represents i Head index position and x j The distance from the tail index position; x represents i The tail index position and x j The distance from the header index position; x represents i The tail index position and x j The distance to the tail index position. The relative position of the span is encoded as a simple linear transformation of four distances, where W r It is a learnable parameter. P represents the connection operator. d The calculation method also uses sin and cos. R and E fembedding The final model input E is obtained by adding the corresponding positions. finput For E finput The sample features E are obtained using the self-attention algorithm in FLATBERT. flat :

[0078] [[0.2721,0.0282,0.2155,...,0.5183,0.2182,-0.4214],

[0079] [0.0047, -0.0124, -0.4007, ..., 0.1974, -0.0406, 0.2414],

[0080] [-0.6375,-0.5862,-0.4815,...,0.3525,-0.8189,-0.8074],

[0081] ...,

[0082] [0.2151,0.4289,1.0611,...,0.7970,0.1746,0.7077],

[0083] [0.1662,0.0137,0.7564,...,0.8919,0.7399,0.5714],

[0084] [-0.0132,0.4917,-0.0409,...,-0.0583,-0.8535,-0.4296]]

[0085] S5: Input vector s1 to the BERT pre-trained model and perform word embedding, position embedding, and weight calculation operations to obtain the word vector feature representation matrix E.bert :

[0086] [[-0.0155,0.4506,0.2322,...,-0.1506,1.0870,-0.3780],

[0087] [0.0037, -0.0996, 0.1890, ..., -0.0720, 0.1073, 0.2848],

[0088] [-0.3881,-0.3162,-0.7579,...,-0.6887,-0.5073,-0.7561],

[0089] ...,

[0090] [-0.1934,0.6990,0.5317,...,-0.1723,0.1414,0.0322],

[0091] [0.0246, -0.0866, -0.1527, ..., 0.2598, -0.3809, 0.8235],

[0092] [-0.0755,-0.8101,-0.6499,...,-1.1519,-0.3522,-0.6536]]

[0093] S6: Transform matrix E bert The vector l is obtained by performing convolution, regularization, and pooling operations respectively, and then l is expanded to be the vector E. flat With the same shape, we obtain matrix E. sentence First, put E bert Matrix A is obtained by applying three 2D convolution kernels of different scales, where the lengths of the kernels are 2, 3, and 4, and the width is E. bert The width of E is determined, and activation and pooling operations are performed on each of the three convolutional kernels after convolution. Finally, the vectors obtained from the three scales are horizontally concatenated to obtain vector l. Then, l is expanded to the width of E using the expand method. flat The same shape yields the global feature representation matrix E. sentence Two feature representation matrices E are horizontally concatenated using equal weights. bert E sentence The overall characteristic representation matrix E is obtained. context :

[0094] [[0.0557,-0.0238,-0.0348,...,-0.4540,0.1469,1.4565],

[0095] [-0.4259, 0.6513, -0.1968, ..., -0.4540, 0.1469, 1.4565],

[0096] [-0.1279,-0.0533,-0.7103,...,-0.4540,0.1469,1.4565],

[0097] ...,

[0098] [0.1707,0.0250,-0.4435,...,-0.4540,0.1469,1.4565],

[0099] [0.7466,0.1553,-1.1365,...,-0.4540,0.1469,1.4565],

[0100] [0.1363, -0.0771, -0.3147, ..., -0.4540, 0.1469, 1.4565]]

[0101] S7: Using matrix T predict With the sample population feature representation matrix E context Multiply the results and perform a softmax operation to obtain the probability matrix I of all tokens belonging to each entity category. pred . to I pred Use argmax to get the result of each token belonging to each entity category. The probability of obtaining the named entity to which each token belongs.

[0102] In this embodiment, a probability matrix is ​​obtained, and a unique named entity "net asset per share" is obtained based on the probability matrix.

[0103] [[6.2718e-05,2.1599e-04,6.1586e-05,…,3.0437e-03,1.0200e-02],

[0104] [8.0598e-05,5.2731e-03,9.9853e-01,…,1.7135e-02,2.2274e-01],

[0105] [2.6652e-05,1.5772e-04,3.6598e-05,…,1.5841e-04,9.3904e-04],

[0106] …,

[0107] [6.9465e-05,4.0517e-04,1.2220e-05,..,,4.5803e-03,2.1012e-04]]

[0108] The above description is only a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. Any equivalent modifications or changes made by those skilled in the art based on the content disclosed in the present invention should be included within the scope of protection set forth in the claims.

Claims

1. A named entity recognition method for an intelligent recommendation system for financial products, characterized in that: Includes the following steps: S1: Fine-tune the BERT pre-trained model based on the existing dataset. After word embedding, position embedding, and weight calculation, use the trained model to predict other unlabeled data collected in the domain. Obtain the feature matrix for each unlabeled sample. ; S2: According to The confidence level of the sample is obtained using a confidence level screening mechanism. Samples with a confidence level greater than a threshold are considered reliable samples and are used together with the original samples in subsequent training. S3: Construct a lattice structure for each sample to generate samples with a lattice structure; The original sample vector s1 and the grid sample s2 are obtained; in step S3, the sample containing lexical features is generated by treating a word as a token and using head and tail pointers to identify the starting position of the word in the text; the grid structure has spans of different lengths, and for two spans in the grid... and There are three relationships between them: intersection, containment, and separation. The relationship between them is determined by their head and tail; we use head[i] and tail[i] to represent these relationships. The index positions of the head and tail are represented by head[j] and tail[j]. The index positions of the head and tail; express Header index position and The distance from the header index position; express Header index position and The distance from the tail index position; express The tail index position and The distance from the header index position; express The tail index position and The distance from the tail index position; S4: The vector s2 is fed into the FLAT-BERT model as input. After word embedding, position embedding, and weight calculation, a feature representation matrix incorporating word features is obtained. ; S5: Input vector s1 into the BERT pre-trained model, and obtain the word vector feature representation matrix after word embedding, position embedding, and weight calculation operations. ; S6: Matrix We obtain vector l by performing convolution and pooling operations respectively, and then expand l into a form that is analogous to the vector vector. The same shape yields the feature representation matrix of the sample sentences. ;Will The overall feature representation matrix of the samples is obtained by concatenating the components horizontally in an equally weighted manner. ; S7: Using linear transformation matrices right Perform a linear transformation to obtain the named entity to which each token belongs; the extracted entity is the recognition result.

2. The named entity recognition method for an intelligent recommendation system for financial products according to claim 1, characterized in that: The specific calculation process for word embedding, position embedding, and weight calculation in step S1 is as follows: Where s1 is the sample vector, It is a linear transformation matrix of word vectors, using A linear transformation of s1 yields the word vector matrix. Next, linear transformations of the sin and cos functions are used for position encoding, where... and The positional encoding matrix PE is obtained by representing the positional codes of odd and even positions in the word vector, respectively; PE is then compared with... The final model input is obtained by adding the corresponding positions. ;right Sample features are obtained using the Self-Attention algorithm in BERT. .

3. The named entity recognition method for an intelligent recommendation system for financial products according to claim 1, characterized in that: The confidence screening mechanism in step S2 is based on Perform the following calculations in sequence: It needs to be passed first right Perform a linear transformation to obtain ,in This represents the score by which each token is assigned to each named entity. Yes The matrix to be linearly transformed; in order to obtain the confidence matrix Represented as probability, using Methods Perform the calculation, where For the characteristic matrix The feature vector of each token is normalized to obtain the confidence level of each token. ,in, This represents the score for each token category; based on the confidence level of each token i. Then calculate the confidence score for each predicted named entity. A named entity contains multiple tokens, and the confidence level of the named entity is calculated based on the tokens it contains. ; This represents the confidence score of the j-th named entity; the final confidence score of the sample is calculated based on the confidence scores of multiple named entities. .

4. The named entity recognition method for an intelligent recommendation system for financial products according to claim 1, characterized in that: For a single character, its head pointer and tail pointer are the same.

5. The named entity recognition method for an intelligent recommendation system for financial products according to claim 2, characterized in that: The specific calculation process for word embedding, position embedding, and weight calculation in step S4 is as follows: in, It is a learnable parameter. Indicates the connect operator. The calculation method uses sin and cos; R is then compared with... The final model input is obtained by adding the corresponding positions. ;right Sample features are obtained using the attention algorithm from FLAT BERT. .

6. The named entity recognition method for an intelligent recommendation system for financial products according to claim 1, characterized in that: In step S6, the matrix will be... After obtaining vector l through convolution and pooling operations at multiple scales, a matrix is ​​then obtained. and The specific calculation process is as follows: First of all Matrix A is obtained by using three 2D convolution kernels of different scales, where the lengths of the kernels are 2, 3, and 4, and the width is... The width of the vector is determined by the convolution of the three kernels, which then undergo ReLU activation and MaxPooling operations to obtain the vector l. The vector l is then expanded using the expand method to the width of the vector. The same shape yields the global feature representation matrix. Two feature representation matrices are concatenated using a horizontal concatenation method. , Obtain the context feature representation matrix concat is a horizontal concatenation operation.

7. The named entity recognition method for an intelligent recommendation system for financial products according to claim 1, characterized in that: In step S7, a linear transformation matrix is ​​used. right The linear transformation is performed, and the specific calculation process is as follows: First, use a matrix. With the sample population feature representation matrix Multiply, and pass through The calculation yields a probability matrix of all tokens belonging to each entity category. ,in This is an operation that normalizes a vector; for use The result is obtained that each token belongs to each entity category. .

Images