A financial entity relation extraction system and method combined with prior knowledge

Abstract

The application relates to a financial entity relation extraction system and method combined with prior knowledge, and the system at least comprises: a deep pre-training module used for training a deep neural network based on a corpus related to finance and generating a deep pre-training model capable of identifying financial entity attributes; a keyword analysis module used for extracting and outputting position information and importance vectors of keywords in Chinese financial text according to Chinese financial prior knowledge; an attention mechanism module used for encoding the position information of the keywords to obtain an attention mask, and inputting the attention mask and entity information of the Chinese financial text into the deep pre-training model to obtain a text feature vector; and an optimal interval distribution model module used for predicting a financial entity relation based on the input text feature vector and importance vector. In view of the defect that the existing model has low applicability to a Chinese specific field, the application is more accurate in the relation extraction result of entities and related features in the Chinese financial text.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing relation extraction model of computer, and particularly relates to a financial entity relation extraction system and method combined with prior knowledge. BACKGROUND

[0002] Relation extraction is a common task in natural language processing. For two entities that have a relationship, they can be called subject and object respectively. Relation extraction is to find the relationship between the subject and the object in unstructured or semi-structured data and represent it as an entity relation triple, i.e. (subject, relation, object). Existing relation extraction models are usually general and mostly analyze English text. They often perform poorly on Chinese and specific domain relation extraction tasks. With the development of economy, various economic and financial activity data have exploded. By analyzing such data, a large amount of hidden knowledge can be obtained to better serve the financial industry.

[0003] In recent years, relation extraction technology represented by deep learning has extracted text features and mapped specific semantics of entities and relations in the text to low-dimensional continuous space vectors through large pre-training models and massive corpus. Through vector calculation and processing, the corresponding complex semantic information between entities is predicted. However, most models only focus on the extraction process of text features, and there are still some problems in the process of predicting modeling using text feature vectors, such as poor generalization performance and poor semantic interpretability. Specific domain prior professional knowledge and better classification model framework need to be introduced to predict the modeling process. The use of Chinese analysis tools to construct a Chinese financial domain knowledge analysis module can reflect the key information of the relationship between financial entities. After fusion with text feature vectors, the superior multi-class optimal interval distribution model can greatly improve the effect of relation extraction tasks, but there are still some problems: (1) the current financial domain knowledge analysis only uses word segmentation and statistical methods, ignoring key word order and deeper semantic information; (2) existing work only does entity pair and single relation extraction, and cannot be extended to multi-entity and multi-relation extraction tasks.

[0004] For example, the Chinese patent with publication number CN114417846A discloses an entity relationship extraction method based on attention contribution degree. The method performs word segmentation on original sentences in a data set by spaCy to obtain a word list, and stores the word list and labels in the input data set D in the form of a dictionary. Then, the input data set D is sampled to obtain entity sample set and relationship sample set of each sentence in the input data set D. A BERT model pre-trained on a large-scale biomedical corpus, a judicial database, and a tourism database is selected, and an attention contribution degree algorithm is used to calculate the interaction information between entities, which is then passed to the downstream entity extraction and relationship extraction tasks to form a span-based entity relationship extraction model. Finally, the entity sample set and the relationship sample set are put into the span-based entity relationship extraction model for training, which greatly improves the F1 value of the entity extraction task and the F1 value of the relationship extraction task. However, the BioBERT model used by the invention is mainly used in the field of biomedical text mining. On the other hand, when predicting relationships, the vector representation of entities and the context relationship of maximum pooling are used, which may lose some of the associated information between entities and sentences, and the prior knowledge in the field is not used to explore the potential connection between entities and relationships.

[0005] The Chinese patent with publication number CN112800774A discloses an entity relationship extraction method and device based on attention mechanism, medium and equipment. The method includes the following steps: preprocessing the content of the original text and inputting it into a BERT pre-training model to obtain the text vector representation of each word in the text sentence; inputting the text vector into a convolutional neural network to obtain a sentence convolutional feature vector; obtaining a semantic context vector containing entity information through an attention weight calculation unit; then connecting into a feature vector, and then passing through a fully connected neural network and a SoftMax function to calculate the probability of different categories, wherein the maximum probability corresponds to the entity relationship type of the entity pair. However, the invention only considers the sentence context information and does not utilize entity attribute information and entity position information. Moreover, the invention does not have an optimization process for specific domain relationship extraction, and cannot further optimize the model using prior knowledge in the field.

[0006] How to introduce Chinese financial domain knowledge into text feature vector representation and how to optimize the relationship extraction model are the core problems that the invention wants to solve. Based on this problem, the invention is proposed. On the one hand, the model is trained specifically by collecting financial related Chinese corpus to improve the performance of the model. On the other hand, a Chinese financial domain text analysis module is constructed, which can provide more domain prior knowledge and combine it with a large pre-training model, which can significantly enrich the information contained in the text feature vector and optimize the feature extraction process of the model, and improve the explainability of the model.

[0007] In addition, on the one hand, due to the difference in understanding of those skilled in the art; on the other hand, due to the fact that the applicant studied a large number of literatures and patents when making the present application, but limited by the size and did not list all the details and contents in detail, but this does not mean that the present application does not have these prior art characteristics, on the contrary, the present application has all the characteristics of the prior art, and the applicant reserves the right to add relevant prior art in the background art. SUMMARY

[0008] In view of the defects of the prior art, the present application provides a financial entity relationship extraction system combined with prior knowledge, which comprises at least: a deep pre-training module for training a deep neural network based on a corpus related to finance and generating a deep pre-training model capable of identifying financial entity attributes, a keyword analysis module for extracting and outputting position information and importance vectors of keywords in Chinese financial text according to Chinese financial prior knowledge; an attention mechanism module for encoding the position information of the keywords in the Chinese financial text to obtain an attention mask, and inputting the attention mask and entity information of the Chinese financial text into the deep pre-training model to obtain a text feature vector; and an optimal interval distribution model module for predicting financial entity relationships based on the input text feature vector and importance vector, wherein the attention mechanism module is connected to the deep pre-training module, the keyword analysis module and the optimal interval distribution model module, respectively.

[0009] Compared with the defects of the prior art, the present application provides a financial entity relationship extraction system and method combined with prior knowledge, which can improve the text feature extraction capability according to the prior knowledge in the Chinese financial field, improve the semantic expression capability of the model, and use a multi-class optimal interval distribution model to train the text feature vector. Compared with the traditional relationship extraction model, it has better performance in the relationship extraction task in the Chinese financial field.

[0010] Preferably, the keyword analysis module is configured to analyze the Chinese financial text and collect related corpus, divide the corpus set based on the collected vocabulary and / or financial entity relationship related to the Chinese financial field; perform word segmentation processing on the corpus set corresponding to the Chinese financial relationship, and sort the words according to the frequency of occurrence; normalize the frequency of occurrence of the words and their synonyms before a preset number of the frequency of occurrence to an importance vector, and output the keyword set of the financial entity relationship and the corresponding importance vector.

[0011] Preferably, the attention mechanism module is configured to extract a structural representation vector of the financial entity.

[0012] The specified name entity is vector-optimized based on the self-attention feature in the sentence, so that the structural representation vectors of the same entity in different sentences have certain distinguishability, and the attention mask is obtained based on the position information of the keyword.

[0013] Preferably, the optimal interval distribution model module is configured to splice the received text feature vector and importance vector into a sentence representation vector and input the optimal interval distribution model, and the optimal interval distribution model outputs prediction information of the financial entity relationship.

[0014] The present application analyzes the characteristics of financial texts, extracts keywords in the financial field and the relevance of financial entity relationships and keywords, combines the Bert model and attention mechanism to further extract text features, performs sentence-level feature enhancement representation, and combines the multi-class optimal interval distribution theory to learn the financial entity relationship in the text, thereby better utilizing the entities and related features in the Chinese financial text and obtaining more accurate relationship extraction results.

[0015] Preferably, the deep pre-training module is further configured to construct an entity attribute prediction model based on the first Bert deep neural network model to perform an entity attribute prediction task, construct an entity relationship extraction model based on the second Bert deep neural network model to perform an entity relationship extraction task, and perform an entity labeling task based on a Bert-LSTM+CRF model.

[0016] Preferably, the keyword analysis module is further configured to collect the glossary related to the Chinese financial field in the following manner: extracting common words from the Chinese financial text as a supplement to the general word segmentation dictionary to form a glossary for the financial field.

[0017] Preferably, the keyword analysis module normalizes the appearance frequency of the keywords and their synonyms whose ranking is before the preset ranking to an importance vector in the following manner:

[0018] Wherein, p i represents the importance value, a i represents the appearance frequency, n represents the preset number, and i represents the index of the first n keywords.

[0019] Preferably, the attention mechanism module obtains the attention mask in the following manner:

[0020] The attention mask m is set according to the position information of the keyword in the Chinese financial text, wherein when the keyword exists in the Chinese financial text, m=1, otherwise m=0.

[0021] Preferably, the execution manner in which the deep pre-training module obtains the text feature vector comprises:

[0022] The Chinese financial text part with attention mask m=1 is given a greater weight to reflect the information of the keyword in the text feature vector.

[0023] The application also provides a priori knowledge combined financial entity relationship extraction method executed by the priori knowledge combined financial entity relationship extraction system, which comprises at least the following steps: training a deep neural network based on a corpus related to finance and generating a deep pre-training model capable of identifying financial entity attributes, extracting and outputting position information and importance vectors of keywords in Chinese financial text for Chinese financial priori knowledge; encoding the position information of the keywords in the Chinese financial text to obtain an attention mask, and inputting the attention mask and entity information of the Chinese financial text into the deep pre-training model to obtain a text feature vector; and predicting a financial entity relationship based on the input text feature vector and importance vector. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is a schematic diagram of the text processing framework of the financial entity relationship extraction system provided by the application;

[0025] Figure 2 is a logical diagram of the relationship extraction model of the financial entity relationship extraction system provided by the application. DETAILED DESCRIPTION

[0026] The application will be described in detail below with reference to the accompanying drawings.

[0027] In view of the defects of the prior art, the application provides a priori knowledge combined financial entity relationship extraction system and method, which can improve the text feature extraction capability according to Chinese financial field priori knowledge, improve the semantic expression capability of the model, and use a multi-class optimal interval distribution model to train the text feature vector. Compared with the traditional relationship extraction model, the application has better performance in the relationship extraction task in the Chinese financial field.

[0028] Bert model: a large pre-training deep learning model. In the application, a large amount of financial related Chinese corpus is used for pre-training, so that the Bert model is suitable for the text mining problem in the financial field. At the same time, the Bert model is fine-tuned to obtain better word vector expression in the relationship extraction task.

[0029] Bert-LSTM+CRF model: an open source model for entity recognition. Among them, the Bert model is used for word embedding to obtain sentence vector, the LSTM model learns the context content of the sentence to give the labeling information of each word, and the CRF layer is used to limit the syntactic requirements to improve the labeling effect. In the present application, this model is used for preprocessing the corpus to obtain the entity information in the sentence. The financial entity relationship extraction system combining prior knowledge of the present application comprises at least one processor. The processor comprises at least a deep pre-training module, a keyword analysis module, an attention mechanism module and an optimal interval distribution model module. The deep pre-training module and the keyword analysis module are connected with the attention mechanism module in a wired or wireless manner. The optimal interval distribution model module is connected with the attention mechanism module in a wired or wireless manner. The wired connection line can be a data bus.

[0030] The deep pre-training module, the keyword analysis module, the attention mechanism module and the optimal interval distribution model module can all be special integrated chips, processors, servers, etc. capable of performing corresponding tasks. For example, the deep pre-training module can execute the running program of the deep neural network model, and the deep neural network model can be trained according to the input Chinese financial field corpus to form a deep pre-training model.

[0031] The keyword analysis module can execute the encoding program corresponding to the step of extracting and outputting the position information and importance vector of the key word in the Chinese financial text according to the Chinese financial prior knowledge.

[0032] The attention mechanism module can execute the encoding program corresponding to the step of encoding the position information of the key word in the Chinese financial text to obtain the attention mask.

[0033] The optimal interval distribution model module can execute the encoding program corresponding to the step of predicting the financial entity relationship based on the input text feature vector and importance vector.

[0034] As shown in Figure 1 The text processing framework of the first Bert model of the present application includes an input layer, a word representation layer, an Encoder layer and an output layer. In the input layer, the input component or input port is used for inputting sentence information. In the word representation layer, the word vector is obtained based on the vocabulary table. In the Encoder layer, the word-level features and sentence-level features are obtained based on the Bert model. In the output layer, the output component or output port is used for outputting the encoded text feature vector, which contains word-level and word-level sentence features.

[0035] As shown in Figure 2 The execution steps of the financial entity relationship extraction system combining prior knowledge of the present application basically include:

[0036] The deep pre-training module is used to train a deep neural network based on a corpus related to finance and generate a deep pre-training model capable of identifying financial entity attributes. Compared with a general pre-training model, the deep pre-training model trained using a corpus related to finance in the Chinese field is more accurate in expression and can obtain word embedding vectors that are more suitable for the use scenario.

[0037] The keyword analysis module is used to extract and output the position information and importance vector of the keywords in the Chinese financial text based on Chinese financial prior knowledge. Prior knowledge in the financial field is very important information, and a part of the keywords can reflect the true meaning of the sentence. By introducing the keyword position and importance vector, the text features can be enriched, and the relationship extraction task can be better performed.

[0038] The attention mechanism module is used to encode the position information of the keywords in the Chinese financial text to obtain an attention mask, and input the attention mask and the entity information of the Chinese financial text into the deep pre-training model to obtain a text feature vector. The attention mask can assist the deep pre-training model to know which positions in the text need to be focused on, so as to eliminate the interference of other irrelevant texts on the inference of the context information of the sentence.

[0039] The optimal interval distribution model module is used to predict the financial entity relationship based on the input text feature vector and importance vector. The optimal interval distribution model is based on the latest research on statistical learning theory, and optimizes the mean and variance of the interval of the sample to the interface to obtain better classification effect and better generalization performance.

[0040] The execution steps of the keyword analysis module are configured in the following manner.

[0041] S11: The deep pre-training module can run a deep neural network model. The deep pre-training model is preferably a Bert deep neural network. The deep pre-training module uses the relevant corpus extracted from the entity relationship in the Chinese financial field as the input information of the deep neural network model to realize model fine-tuning and obtain a deep pre-training model.

[0042] Among them, model fine-tuning is a kind of pre-operation of the deep pre-training model for actual tasks, which makes the model suitable for the current task, called "fine-tune". That is, a softmax layer is added to the last output part of the Bert deep neural network model to perform classification tasks, and the previously mentioned financial related corpus is used for pre-training of the relationship extraction task.

[0043] S12: The deep pre-training model receives and trains the entity annotation and entity attribute as a pre-training model feature, and outputs a text feature vector. The text feature vector is represented as v1.

[0044] The text feature vector refers to the sentence vector output by the Bert deep neural network model after the training sample is input into the Bert deep neural network model. For example, inputting the sentence "CC Shanghai headquarters invests 1 billion in XXX technology, and the technology director is A" into the Bert deep neural network model, the Bert deep neural network model will output the encoded text feature vector [0.75, 0.23, 0.87, …, 0.88] of the sentence vector. The vector can be used for various tasks.

[0045] The deep pre-training module performs the entity annotation task through the Bert deep neural network model. Similarly, the deep pre-training module uses the collected Chinese financial field corpus as training data, annotates the entities in the corpus with BIO, and trains and predicts the model using the existing Bert-LSTM+CRF model. The obtained model can complete the entity annotation task and mark the entities in the input sentence.

[0046] The deep pre-training model in the deep pre-training module processes the collected financial field corpus and extracts entity attribute information, such as company name, person name, and place name. For example, "CC Shanghai headquarters invests 1 billion in XXX technology, and the technology director is A" can extract that CC is a company, the attribute of CC is a company, the attribute of A is a person name, and the attribute of Shanghai is a place name. The deep pre-training module runs another Bert deep neural network model, adds a softmax layer at the last layer of the text, and performs a multi-classification prediction task. After obtaining the labeled entities, the deep pre-training model inputs the entity information into another Bert deep neural network model to obtain the attributes of the entity and determine whether the entity attribute is a company name or a person name.

[0047] The keyword analysis module is used to extract and output the position information and importance vector of the keywords in the Chinese financial text based on the Chinese financial prior knowledge.

[0048] In the present application, prior knowledge refers to some sentences related to the Chinese financial field. For example, A company invests in B company, then the word "investment" is a special word in the financial field, and the relationship of investment is likely to be cooperation. For another example, A manages the fund disbursement in the financial department of XXX company, then the financial department is a word in the financial field, and the relationship between A and XXX company is likely to be an in-service relationship. Such associated information obtained through human experience is prior knowledge.

[0049] Specifically, the execution steps of the keyword analysis module are configured as:

[0050] S21: Analyze the Chinese financial text and collect related corpus, and divide the corpus set based on the collected Chinese financial field related vocabulary and / or financial entity relationship.

[0051] The keyword analysis module extracts common words from the Chinese financial text as a supplement to the general word segmentation dictionary, forming a vocabulary table for the financial field.

[0052] The keyword analysis module divides the corpus set according to the financial entity relationship. The corpus refers to the text used for training. The corpus in the present application refers to the text related to the financial technology field, such as the text "A company and B company reach a partnership in the Internet financial field". Taking "A company and B company reach a partnership in the Internet financial field" as an example, A company and B company are entities, and their relationship is "cooperation".

[0053] Common words in the Chinese financial field, such as investment, joint venture, stock, purchase, appointment, general manager, etc. are common in financial field-related texts. Words such as "monitoring, capturing, and education" are not common in financial field-related texts. Whether it is common or not is directly based on the collected financial corpus. The words in the collected corpus information are extracted, and the frequency is directly counted to obtain the common and uncommon.

[0054] S22: Perform word segmentation processing on the corpus set corresponding to the Chinese financial relationship, and sort according to the frequency of the words.

[0055] The keyword analysis module can run a Chinese word segmentation model. The Chinese word segmentation model is, for example, the Chinese processing toolkit jieba of python, or other processing applications with the same function.

[0056] The Chinese word segmentation model directly inputs and processes the collected financial field corpus based on a specific dictionary and automatically divides the Chinese words.

[0057] S23: Normalize the frequency of the words and their near-synonyms before the preset number of frequency sorting to an importance vector.

[0058] The keyword analysis module counts the frequency a of each word based on the Chinese word segmentation model, and selects the top n as the keywords according to the number from high to low. That is, for the entity relationship r, its keyword dictionary is d r =[d1,d2,...,d n ].

[0059] The keyword analysis module extracts common words from the Chinese financial text as a supplement to the general word segmentation dictionary, forming a specific dictionary for the financial field.

[0060] S24: Output the keyword set of the financial entity relationship and the corresponding importance vector.

[0061] The keyword analysis module normalizes the appearance frequencies of the words and their synonyms whose appearance frequencies are ranked before the preset ranking into an importance vector in the following manner:

[0062]

[0063] wherein p i represents the importance vector, a i represents the appearance frequency, n represents the preset number, and i represents the index of the first n words. The importance vector refers to a vector composed of the importance values corresponding to the plurality of keywords.

[0064] For example, for the relationship of “cooperation”, the following 5 high-frequency words [“strategic partner”, “investment”, “stock purchase”, “joint venture”, “joint development”] are collected based on the Chinese financial field corpus, and the appearance frequencies of the keywords are 90, 80, 70, 60, and 50 times, respectively.

[0065] For “strategic partner”, the appearance frequency a1=90, and the importance value p1=90 / (90+80+…+50).

[0066] For “investment”, the appearance frequency a2=80, and the importance value p2=80 / (90+80+…+50).

[0067] For “stock purchase”, the appearance frequency a3=70, and the importance value p3=70 / (90+80+…+50).

[0068] For “joint venture”, the appearance frequency a4=60, and the importance value p4=60 / (90+80+…+50).

[0069] For “joint development”, the appearance frequency a5=50, and the importance value p5=50 / (90+80+…+50).

[0070] The attention mechanism module of the present application is configured in the following manner.

[0071] S31: Extract the structural representation vector of the financial entity.

[0072] S32: Optimize the vector of the specified name entity based on the self-attention feature in the sentence, so that the structural representation vectors of the same entity in different sentences have a certain degree of distinction.

[0073] The self-attention feature refers to different words or characters, which should have different vector representations in different sentences.

[0074] S33: Encode the position information of the keyword to obtain the attention mask.

[0075] Specifically, the attention mechanism module encodes the keywords in the text according to the attention mechanism. The attention mask m is set according to the position information of the keywords in the Chinese financial text, wherein m = 1 when there is a keyword in the Chinese financial text, otherwise m = 0. The Chinese financial text part with attention mask m = 1 is given greater weight to reflect the information of the keywords in the text feature vector.

[0076] The attention mechanism is a way to get the key information of the input sample. For example, for the sentence "To develop the Chinese market, some products of ALBO company have realized online self-operation or offline cooperation operation in A, B, C, etc. in some supermarkets in Shanghai, Beijing, Guangzhou, etc. and entered some supermarkets in Shanghai, Beijing, Guangzhou, etc. to improve the popularity of ALBO products in China", the entities "ALBO company", "A, B, C", "online self-operation or offline cooperation operation" and the keywords "Chinese market" and the like should be focused on. Therefore, when the attention mechanism module inputs the text into the deep pre-training model, an attention mask about these key words in the sentence is also input, that is, 1 is marked at the position of these words and 0 is marked at other positions, so that the deep pre-training model pays attention to these words in the training process.

[0077] The optimal interval distribution model module of the application is configured in the following manner.

[0078] S41: splice the received text feature vector and importance vector into a sentence representation vector and input the optimal interval distribution model.

[0079] Specifically, the importance vector of the k entity relationships and the corresponding first n keywords can form an importance matrix P with a size of k x n. For each row of P, if there is a keyword in the text, the importance value in P is unchanged. If it does not appear, set the value of P to 0, then update the importance matrix P' to an importance vector v2 with a length of k x n. The importance vector v2 and the text feature vector v1 are spliced to obtain a sentence representation vector x = [v1, v2].

[0080] For example, assume that the Shenzhen pre-training module only analyzes two entity relationships, "cooperation" and "employment", and analyzes the collected Chinese financial field corpus after word segmentation. In the "cooperation" relationship corpus, the words appearing in the order of frequency are "strategic partner", "investment", "stock", "joint venture", "joint development", "operation", and "offline". At this time, the key word analysis module extracts only the top 4 words in terms of frequency, i.e., forms a key word dictionary ["strategic partner", "investment", "stock", "joint venture"]. The entities most likely to be "cooperation" corresponding to the key word dictionary are calculated according to the frequency of the key words, and the importance value p i . Then, for "cooperation", the optimal interval distribution model generates an importance vector [p 11 , p 12 , p 13 , p 14 ], and for "employment", the optimal interval distribution model also generates a 4-dimensional importance vector denoted as [p 21 , p 22 , p 23 , p 24 ]. If there are k relationships, there can be k*4 importance vectors, i.e., a k*4 importance matrix can be obtained.

[0081] The optimal interval distribution model expands it to obtain a 4k*1 sentence representation vector. With the above two relationship importance vectors, the importance vector is [p 11 , p 12 , p 13 , p 14 , p 21 , p 22 , p 23 , p 24 ]. When the input text feature vector appears "investment" and "management", the importance vector v2 corresponding to the text feature vector v1 is [0, p 12 , 0, 0, 0, 0, 0, p 24 ]. The importance vector v2 and the text feature vector v1 are spliced to form a sentence representation vector and used in the subsequent multi-class optimal interval distribution model.

[0082] S42: The optimal interval distribution model outputs the prediction information of the financial entity relationship.

[0083] The optimal interval distribution model is trained according to the input sentence representation vector, and the expression of the optimal interval distribution model is as follows, and the final financial relationship extraction task is completed.

[0084]

[0085]

[0086]

[0087] i represents a sample index;mu represents the weight of interval variance;m represents the total number of samples;theta represents a parameter for controlling the number of support vectors;omega l represents the classifier parameter of the lth class;k represents the total number of classes;lambda represents a balance parameter;y represents the real class corresponding to the sample;phi(·) represents the feature mapping based on a certain positive kernel.

[0088] represents the prediction value of the ith sample on the corresponding real class; represents the prediction value of the ith sample on the lth class.

[0089] The application illustrates the financial entity relation extraction method combined with prior knowledge based on an application scenario.

[0090] The financial entity relation extraction method combined with prior knowledge of the application at least includes the following steps.

[0091] S51: Corresponding models are constructed in each module in advance.

[0092] Two different Bert deep neural network models and a Bert-LSTM+CRF model are set in the deep pre-training module. The first Bert deep neural network model is used for entity attribute prediction task, the second Bert deep neural network model is used for entity relation extraction task, and the Bert-LSTM+CRF model is used for entity labeling task.

[0093] The construction of entity attribute prediction is carried out. The entity in the financial corpus is taken as a sample, and the entity attribute is taken as a label, which is sent to the first Bert deep neural network model. The first Bert deep neural network model adds a softmax layer at the last layer for multi-classification, so that the model of entity attribute prediction can be obtained.

[0094] The construction of entity relation extraction model is carried out. The second Bert deep neural network model with a softmax layer is used for the training of relation extraction task. The second Bert deep neural network model is fine-tuned using the softmax layer, so that the second Bert deep neural network model is suitable for the relation extraction task, and a more suitable sentence vector can be obtained to facilitate the use of the following multi-classification optimal interval distribution model.

[0095] The optimal interval distribution model for multi-classification is constructed. After the fine-tuning of the second Bert deep neural network model is completed, the softmax layer of the last layer of the second Bert deep neural network model is removed to form the optimal interval distribution model for multi-classification. The optimal interval distribution model for multi-classification outputs the corresponding sentence feature vector according to the attention mask of the input text and the keyword position. After the sentence representation vector x is finally obtained, the optimal interval distribution model for multi-classification makes relationship prediction based on x.

[0096] The entity annotation model is constructed. The collected corpus is subjected to BIO annotation, and then is sent into the Bert-LSTM+CRF model for training to obtain the entity annotation model.

[0097] The entities, entity relationships and entity attributes in the collected corpus related to the Chinese financial field are marked and used as pre-training samples.

[0098] S52: The deep neural network is trained based on the corpus related to finance to generate a deep pre-training model capable of identifying financial entity attributes.

[0099] After the construction of each model is completed, the existing training set is used for model training. After the input text, the entity annotation model performs entity annotation on the input text and obtains the entity. The entity annotation model sends each entity to the entity attribute prediction model. The entity attribute prediction model outputs the attribute of the entity, and the attribute of the entity can be a company, a person's name or other.

[0100] The entity relationship extraction module selects entity pairs and performs training of the entity relationship extraction model. For example, the sentence "In order to develop the Chinese market, some products of ALBO company have realized online self-operation or offline cooperation operation in Tmall, Jingdong, Pinduoduo, etc. and have entered some supermarkets in Shanghai, Beijing, Guangzhou, etc. to improve the popularity of ALBO products in China" includes multiple entities, so the relationship extraction task training can be performed on the "ALBO-Tmall" entity pair and the "ALBO-Jingdong" entity pair.

[0101] The entity relationship extraction module will also concatenate the attribute of the entity after the input of the entity relationship extraction model, for example, for the above sentence, when the "ALBO-Tmall" entity pair is input, "company-company" is added after it.

[0102] S53: The position information and importance vector of the keyword in the Chinese financial text are extracted and output according to the Chinese financial prior knowledge.

[0103] The keyword analysis module constructs an importance vector according to the keywords appearing in the sentence. The keyword analysis module performs word segmentation analysis on the corresponding corpus according to each known k entity relationship to obtain a corresponding dictionary of n high-frequency appearing words. The importance is calculated to obtain a k*n matrix, such as the 2*4 importance matrix constructed under 2 relationships mentioned above. The importance matrix is prior knowledge analyzed according to the financial corpus.

[0104] For example, the k*n matrix is unfolded, and if a keyword appears in the sentence, the importance value corresponding to the word is retained, otherwise it is set to 0, and then the importance vector [0, 0, 0, 0.29, 0, 0.14, 0, …, 0.33] can be obtained. S54: Encode according to the position information of the keywords in the Chinese financial text to obtain an attention mask, and input the attention mask and the entity information of the Chinese financial text into the deep pre-training model to obtain a text feature vector.

[0105] The attention mechanism module calculates the attention mask according to the obtained dictionary. For example, if “open up the Chinese market”, “product”, “online self-operation” and the like appear in the sentence, the corresponding position value is 1, that is, the attention mask [0, 1, 1, 1, 1, 1, 1, …, 0, 0] is obtained. The attention mask is used to tell the keyword analysis model during training that the words at these positions may contain important information.

[0106] The attention mechanism module inputs the attention mask and the entity information of the Chinese financial text into the deep pre-training model. The deep pre-training model outputs a text feature vector. For example, the text feature vector is [0.021, 0.24, 0.33, …, 0.01].

[0107] S55: Predict the financial entity relationship based on the input text feature vector and the importance vector. The optimal interval distribution model module concatenates the text feature vector and the importance vector and sends them together as the sentence representation vector of the input text to the multi-class optimal interval distribution learning machine for training.

[0108] It should be noted that the above specific embodiments are exemplary, and those skilled in the art can think of various solutions under the inspiration of the disclosure of the present application, and these solutions also belong to the disclosed range of the present application and fall within the protection scope of the present application. Those skilled in the art should understand that the specification and drawings of the present application are illustrative and do not constitute a limitation on the claims. The protection scope of the present application is defined by the claims and their equivalents. The specification of the present application contains multiple inventive concepts, such as “preferably”, “according to a preferred embodiment” or “optionally”, which all indicate that the corresponding paragraph discloses an independent concept, and the applicant reserves the right to file a divisional application according to each inventive concept.

Claims

1. A system for extracting financial entity relationships incorporating prior knowledge, characterized in that, The system at least comprises: a deep pre-training module for training a deep neural network based on a corpus related to finance and generating a deep pre-training model capable of identifying financial entity attributes, a keyword analysis module for extracting and outputting position information and importance vectors of keywords in Chinese financial text with respect to Chinese financial prior knowledge; an attention mechanism module for encoding the position information of the keywords in the Chinese financial text to obtain an attention mask, and inputting the attention mask and entity information of the Chinese financial text into the deep pre-training model to obtain a text feature vector; an optimal interval distribution model module for predicting financial entity relationships based on the input text feature vector and importance vector, wherein the attention mechanism module is connected to the deep pre-training module, the keyword analysis module, and the optimal interval distribution model module, respectively. 2.The financial entity relationship extraction system incorporating prior knowledge of claim 1, wherein, The keyword analysis module is configured to: analyze Chinese financial text and collect relevant corpus, divide the corpus set based on the collected vocabulary and / or financial entity relationship related to the Chinese financial field; perform word segmentation processing on the corpus set corresponding to the Chinese financial relationship, and sort according to the frequency of the words; normalize the frequency of the keywords and their synonyms before the preset number of the frequency sorting as an importance vector, output the keyword set of the financial entity relationship and the corresponding importance vector. 3.The system for extracting financial entity relationships incorporating prior knowledge according to claim 1 or 2, characterized in that, The attention mechanism module is configured to: extract the structural representation vector of the financial entity; vector optimize the specified name entity based on the self-attention feature in the sentence, so that the structural representation vectors of the same entity in different sentences have certain distinguishability; encode the position information of the keywords to obtain an attention mask.

4. The system for financial entity relation extraction incorporating prior knowledge according to claim 3, wherein, The optimal interval distribution model module is configured to: concatenate the received text feature vector and importance vector into a sentence representation vector and input it into the optimal interval distribution model, the optimal interval distribution model outputs the prediction information of the financial entity relationship.

5. The system for financial entity relation extraction incorporating prior knowledge according to claim 3, wherein, The deep pre-training module is further configured to: construct an entity attribute prediction model based on a first Bert deep neural network model to perform an entity attribute prediction task, construct an entity relationship extraction model based on a second Bert deep neural network model to perform an entity relationship extraction task, perform an entity labeling task based on a Bert-LSTM+CRF model.

6. The system for financial entity relation extraction incorporating prior knowledge according to claim 3, wherein, The keyword analysis module is further configured to: acquire the vocabulary related to the Chinese financial field in the following way: extract common words from Chinese financial text as a supplement to the general word segmentation dictionary to form a vocabulary for the financial field.

7. The system for financial entity relation extraction incorporating prior knowledge according to claim 6, wherein, The keyword analysis module normalizes the frequency of the keywords and their synonyms before the preset ranking of the frequency sorting as an importance vector in the following way: ， wherein, represents an importance value, represents a frequency of occurrence, represents a preset number of selection, represents an index of the top n keywords counted.

8. The system for financial entity relation extraction incorporating prior knowledge according to claim 6, wherein, The attention mechanism module obtains the attention mask in the following way: Setting an attention mask according to position information of a keyword in a Chinese financial text , when the keyword exists in the Chinese financial text, , otherwise .

9. The system for financial entity relation extraction incorporating prior knowledge according to claim 8, wherein, The deep pre-training module obtains the text feature vector in the following way: Attention mask for chinese financial text greater weight to the Chinese financial text portion to reflect the information of the keywords in the text feature vector.

10. A method for extracting financial entity relationships in combination with prior knowledge, executed by the system for extracting financial entity relationships in combination with prior knowledge according to any one of claims 1 to 9, characterized in that: The method at least comprises: training a deep neural network based on a corpus related to finance and generating a deep pre-training model capable of identifying financial entity attributes, The position information and importance vector of the keywords in the Chinese financial text are extracted and output according to the Chinese financial prior knowledge; The attention mask is obtained according to the position information of the keywords in the Chinese financial text, and the text feature vector is obtained by inputting the attention mask and the entity information of the Chinese financial text into the deep pre-training model; The financial entity relationship is predicted based on the input text feature vector and the importance vector.

Images