Design and optimization method of relation extraction model based on remote supervision in finance field

Abstract

The application discloses a financial field remote supervision-based relation extraction model design and optimization method, which can improve the ability of entity relation mining from text and obtain rich relation information, thereby supplementing the knowledge base in the financial field, and comprises the following steps: 1) using the financial field KG, pre-processing and relation marking of the corpus on the financial news are carried out, and the remote supervision process is realized; 2) ner identifies the entities in the news corpus, and after sampling, the entities in the sentence are combined two by two according to the given type requirement and input into the relation extraction model for prediction; the prediction result is mounted to the input address of the brat marking through the cloud disk, and the format is aligned to the.ann and.txt formats used by the brat; the corpus is manually marked; the marked corpus is read, parsed back to the json format, and stored as the fine-marking corpus; the number of markings of each label is counted, and the relation type with less marking is supplemented in combination with the remote supervision; and the pretrain bert model is fine-tuned by using the artificially marked data.

Description

Technical Field

[0001] This invention relates to fields such as information processing technology, and more specifically, to a method for designing and optimizing a relation extraction model based on remote supervision in the financial field. Background Technology

[0002] Relation extraction, a fundamental task in information extraction, has garnered significant attention in recent years. Its purpose is to identify target relationships between entities in text. Relation extraction is crucial for knowledge base construction and text understanding, particularly beneficial for natural language processing tasks such as question answering and text understanding. Furthermore, as a core component of knowledge graph construction, relation extraction is an indispensable step. The relation extraction task can be defined as follows: given a sentence with noun pairs e1 and e2, our goal is to identify the relationship between e1 and e2. After reviewing a large amount of international literature on relation extraction, it is found that this task remains challenging in both academia and industry. In practical applications, models proposed for relation extraction in specific domains are particularly scarce.

[0003] In existing technologies, most relation extraction models are trained on public knowledge bases, and there are no relation extraction models trained for specific domains. However, in practical applications, relation extraction is often required for specific domains, and general relation extraction models cannot be directly applied. For example, in the financial domain, many terms are unique, and directly applying general relation extraction models yields poor results. Therefore, designing and optimizing relation mining models for the financial domain is urgently needed. Summary of the Invention

[0004] The purpose of this invention is to provide a method for designing and optimizing a relation extraction model based on remote supervision in the financial field. This method can improve the ability to mine relationships between entities from text and obtain rich relation information, thereby supplementing the knowledge base in the financial field.

[0005] This invention is achieved through the following technical solution: a method for designing and optimizing a relation extraction model based on remote supervision in the financial field, comprising the following steps: 1) Use a knowledge graph (KG) in the financial field to preprocess and annotate the relationships in financial news corpora to enable remote monitoring of the process; 2) Generate manually annotated corpus (since it is used for pre-training the model, the annotated corpus does not need to be too large at first); 3) Fine-tune the pretrained BERT model using manually labeled data.

[0006] To further improve the design and optimization method for a relation extraction model based on remote supervision in the financial field as described in this invention, the following settings are specifically adopted: Step 1) specifically includes the following steps: 1.1) Corpus preprocessing: 1.1.1) Use web crawlers to collect relevant financial news corpora from relevant websites; 1.1.2) Perform data preprocessing on the crawled results, including removing HTML tags and segmenting the data into sentences; 1.1.3) Perform data filtering, removing meaningless short phrases or short sentences with a single line that resembles a title, and removing those with a length of less than 20 characters; 1.1.4) Perform innovative corrective processing, including unique dirty data removal and noisy data removal; 1.2) KG (Knowledge Graph) Data Processing: Since KG (Knowledge Graph) data is stored using the platform-independent data communication protocol Google Protocol Buffers, processing requires parsing the KG data according to a specific configuration file (.proto file). After parsing, the following two types of information files are created: name2type.txt --> Mapping file, specifically recording information about the mapping from entity names to entity types (quantity: 4,612,723): a) Ignore duplicate names (the constructed dictionary is 1-to-1). b) e.g. (Huawei, 205); and triplets.txt --> the basic relationship file, specifically recording one triplet from a KG (Knowledge Graph) per line, with each line formatted as Entity 1 Name Relationship Entity 2 Name (Quantity: 2,160,372). e.g. (Huawei Technologies Co., Ltd., Founder, Ren Zhengfei); 1.3) The remote supervision module is used to annotate the existing data to complete the remote supervision.

[0007] To further improve the design and optimization method for a relation extraction model based on remote supervision in the financial field as described in this invention, the following settings are specifically adopted: The unique dirty data removal includes the following steps: 1.1.4.a.1) First, a high-quality entity set is constructed by using filtering methods at the coarse-grained level through specific constraints; a preferred specific solution is to regard entities whose confidence scores and lengths meet certain constraints as high-quality entities. Simply put, the shorter the entity length, the higher the threshold required for its confidence score. 1.1.4.a.2) Combine specific frequency information to filter meaningless high-frequency words at a fine-grained level; a preferred specific solution is to combine the frequency of words obtained from statistics in the constructed dataset to build a blacklist of high-frequency meaningless entity words appearing in the article at a fine-grained level. 1.1.4.a.3) Based on the above two steps, the data of all entities in the corrected KG (Knowledge Graph) will be obtained, and useful instance relationship information will be extracted with the help of AC automata technology.

[0008] The key feature of this approach is its ability to extract truly important relational information from large-scale data samples with high accuracy, while removing various negative data that may affect subsequent tasks.

[0009] To further improve the design and optimization method for a relation extraction model based on remote supervision in the financial field described in this invention, the collected data generally contains a certain amount of noise. For example, news articles often contain numerous enumeration sentences, such as "The companies attending this event are: Company A, Company B, Company C, Company D," etc. Such sentence structures can negatively impact the labeled data due to mislabeling. To remove this negative noise, the following settings are specifically adopted: The noise data removal includes the following steps: 1.1.4.b.1) Limit the number of entities identified in a sentence; 1.1.4.b.2) Calculate the proportion of the length of all entities in a sentence to the total length of the sentence. This is called entity purity. If the entity purity is too high, it means that the sentence may be a coordinate sentence and should be discarded.

[0010] To further improve the design and optimization method for a relation extraction model based on remote supervision in the financial field as described in this invention, the following setup is specifically adopted: Step 1.3) uses a remote supervision module to annotate existing data to complete remote supervision, including the following specific steps: 1.3.1) Constructing the AC automaton; a preferred approach is to use the target string set (i.e., read the name2type.txt file and extract all the keys to form the target string set) to construct an automaton with relevant information; 1.3.2) For the input sentence, the AC automaton is used to perform the longest match of the target entity in the sentence through its internal Trie tree structure; 1.3.3) Read triplets.txt and construct a dictionary mapping (Entity 1 name, Entity 2 name) to relations; 1.3.4) Combine multiple entities matched in the sentence into pairs to form a key and look it up in the dictionary. If the dictionary returns the relationship between the entities, mark it and output a marked corpus.

[0011] To further improve the design and optimization method for a relation extraction model based on remote supervision in the financial field as described in this invention, the following settings are specifically adopted when implementing step 1.3.4): 1.3.4.1) When there are multiple relationships between some entities, we can count the number of each relationship, define the priority of relationship matching according to the principle of balancing the number of each type as much as possible, and then select the relationship with the highest priority as the label for processing. 1.3.4.2) For special cases where some sentences do not express the expected relationship between entities, specific data annotation and denoising techniques are used, e.g.: a) Elon Musk answers reporters' questions on a SpaceX pressconference; b) SpaceX was founded in 2002 by Elon Musk; A preferred approach is to address relationships that are prone to introducing a lot of noise (such as CEO, founder, subsidiary, shareholder relationships, etc.) by adding keyword restrictions. Only when the required keywords appear in a sentence will it be considered as the corresponding relationship type.

[0012] 1.3.4.3) To achieve effective annotation when KG (Knowledge Graph) relations are lacking, candidate sentences are discovered by using knowledge about specific relation entity pairs in the KG (Knowledge Graph), and then corresponding annotations are performed using keywords; the annotation quality of the results produced by this method meets the requirements after sampling.

[0013] 1.3.4.4) For some relations, while meeting business requirements, adding type restrictions can further standardize the types of entities in the relations; because as mentioned earlier, entity information helps with relation extraction, and the most important entity information is the entity type information, which helps to assist the training model in capturing entity type information.

[0014] 1.3.4.5) To resolve matching ambiguity issues, the following techniques are employed: a) Using named entity recognition and entity linking tools, the results identified by named entity recognition are linked to the corresponding entities in the knowledge graph, and the linking process resolves ambiguity. b) Matching-based method. The advantage of this method is that it is faster. When tested on seven million sentences, the NER-Link method (named entity recognition and entity linking method) takes 43 hours in a single process, but matching only takes 3 hours. Matching also generates more labeled data, and the labeled data results meet expectations. After using various noise reduction techniques, the data cleaning ratio reached 0.7986, and the quality of the sampled evaluation data annotation was significantly improved.

[0015] To further improve the design and optimization method for a relation extraction model based on remote supervision in the financial field as described in this invention, the following configuration is specifically adopted: Step 2) includes the following steps: 2.1) NER (Named Entity Recognition) identifies entities in news corpora, and after sampling, it combines entities in sentences pairwise according to given type requirements to extract relationships and make predictions. 2.2) Mount the prediction results to the input address marked by brat via cloud disk, and align its format to the .ann and .txt formats used by brat; 2.3) Manually annotate the corpus; 2.4) Read the labeled corpus, parse it back into JSON format, and store it as finely labeled corpus; 2.5) Count the number of labels for each type of relationship and supplement them in a targeted manner by combining remote supervision (as a labeling reference) for relationship types with fewer labels.

[0016] Part of the manually labeled corpus will be used to fine-tune the pre-trained model, and the other part will serve as a new evaluation set. In the implementation, all predicted relations and NER (named entity recognition) results for each sentence will be labeled on the BART platform. This way, while labeling relations, the NER (named entity recognition) results can also be corrected, and NER (named entity recognition) can also use this data to optimize the model.

[0017] To further improve the design and optimization method of the relation extraction model based on remote supervision in the financial field described in this invention, the following settings are specifically adopted: When performing step 2.3) of manually annotating the corpus, the following situation will occur. 2.3.1) If an entity recognition error is found, correct the entity; 2.3.2) If the recalled relation is incorrect, correct the relation type; 2.3.3) For relationships not yet recalled, please supplement; 2.3.4) The results of identifying unrecalled NERs and the resulting relationships are supplemented; 2.3.5) Hard cases (difficult sample) that are considered to be easily misrecalled by the relation extraction model are labeled as unrelated. Among them, negative cases (i.e. strong negative cases) that are considered to be necessary to be added to the fine-tune set are labeled as particularly unrelated. Cases (samples) with particularly unrelated labels are guaranteed to be added to the training corpus as strong negative cases. The remaining negative cases are randomly sampled from the remaining entity pairs that have not been labeled with relations.

[0018] To further improve the design and optimization method of the relation extraction model based on remote supervision in the financial field described in this invention, and to further enhance the model's predictive performance, a new type feature technology is introduced, specifically adopting the following setting: Step 3) during fine-tuning includes: Compared to directly using the embeddings (a set of tensors used to store object information, also called embedding vectors) corresponding to the BERT-encoded classification label vectors ([CLS]) to perform softmax (probability normalization function) in a fully connected layer for relation prediction, considering that entity information is also important for relation prediction, and the most important entity information is the entity type information, the type label embeddings of entity 1 and entity 2 are concat (embedding vectors concatenated) with the embeddings corresponding to the classification label vectors ([CLS]) and then input into the fully connected layer for infer (prediction). Previously, this technique would label multiple locations of an entity at the same time, but now it labels only one location at a time (because with the addition of type embeddings, the type embeddings of different locations of the same entity are also different, containing semantic information of its surroundings, so they are processed separately).

[0019] Based on this, the modified strong negative samples were added at the same time. It was observed that the strong negative samples did not penalize the model much, so the upper limit of the number of strong negative samples was increased. The preferred method is to increase the upper limit of the number of strong negative samples from 2,000 to 5,000.

[0020] Meanwhile, the remote supervision end adds a subsidiary label restriction: previously, only the presence of subsidiary keywords in the sentence was required, regardless of their specific position. However, this revealed a bad case (referring to a judgment error) where parallel subsidiaries were mistakenly recalled. For example: The company's controlling subsidiary, Zhonglai Minsheng, and its wholly-owned subsidiary, Zhonglai Zhilian, signed an "EPC general contracting agreement for household photovoltaic power generation project" with Shanghai Yuanye New Energy Co., Ltd.

[0021] Among them, "Zhonglai Zhilian" and "Shanghai Yuanye New Energy Co., Ltd." will be mistakenly recalled as subsidiaries.

[0022] Therefore, a restriction is added: only when the word 'subsidiary' appears between the two entities in the sentence is it marked as a positive example; otherwise, it is marked as unrelated. This marking as unrelated is also deliberate. If it were not marked as unrelated and simply discarded, the trained model would still predict some bad cases in the subsidiary relationship where the word 'subsidiary' does not appear between the two entities, such as: As the younger brother of Zhang Rongming, the actual controller of Aimer Group, Zhang Ronglong's Suzhou Rongshi not only shares the same business name as Aimer Group's subsidiaries Suzhou Aimer and Suzhou Meishanzi, but Zhang Ronglong also serves as a director and general manager of Suzhou Aimer and Suzhou Meishanzi.

[0023] This indicates that directly discarding the samples does not penalize the model enough. Instead, the samples should be labeled as unrelated and then solved.

[0024] In addition, following the implemented fine-calibration process, 738 data points were labeled (431 positive examples, including 221 related products), and then 500 randomly sampled negative examples were added from the remote supervision end, for a total of 1200 data points to fine-tune the model.

[0025] Overall, fine-tuning has a significant positive effect on the model, with most metrics showing positive results. The model showed the greatest performance improvement in related product relationships after fine-tuning, thanks to the concentrated labeling of many related product relationships in the fine-tuned dataset, which accounted for half of the total positive examples. Shareholder relationships and subsidiary relationships were also labeled, and the model was corrected accordingly, leading to performance improvements.

[0026] To further improve the design and optimization method of the relationship extraction model based on remote supervision in the financial field described in this invention, due to the incompleteness of the KG (Key Knowledge Group), many companies are not in the KG. Therefore, it is necessary to supplement the aliases of these companies. Specifically, the following setting method is adopted: When executing step 1.1.1), an alias supplementation process is also included: 1. For a given company, perform alias mining from both the model and rule dimensions; 2. Use a search engine to search for a given company and crawl the top 10 articles that contain the query (the query itself) as valid. The criterion for finding a reasonable and valid article is whether it contains the search query. Alternatively, use the group's internal financial news database as the corpus source for the model. In other words, the model has two corpus sources: one is to use Toutiao search to search for companies like TYC (related financial companies), crawl the top 10 articles returned by the search, and consider articles containing the query as valid; the second is the group's internal financial news database. 3. To reduce the workload of manual review, a confidence function was designed to score the confidence of the results mined by the model. The top results with high scores can be directly entered into the database, while the bottom results need to be manually reviewed before being entered into the database. 4. The candidate aliases generated by the rules will be further processed by NER (Named Entity Recognition). Only candidate words that are recognized as company types by NER will be retained, and the rest will be discarded. 5. Sample the alias results generated by the rule-based method. After manual evaluation, the accuracy rate is above 95%, and they can be directly added to the database.

[0027] Compared with the prior art, the present invention has the following advantages and beneficial effects: This invention can enhance the ability to mine relationships between entities from text and obtain rich relationship information, thereby supplementing the knowledge base in the financial field, such as performing downstream tasks of knowledge graph completion.

[0028] This invention can extract truly important relational information from large-scale data samples with high accuracy, while removing various negative data that may affect subsequent tasks.

[0029] After employing various noise reduction techniques, the data cleaning ratio of this invention reaches 0.7986, and the quality of the sampled evaluation data annotation is significantly improved.

[0030] This invention designs and optimizes a relation extraction model for a specific domain, namely the financial field, and further enhances the ability to mine relationships between entities from text. In this way, rich relational information can be obtained to supplement the knowledge base in the financial field, and it can be applied to various important industrial scenarios, such as relevance services, search, and recommendation. It provides specific design and optimization ideas for practical applications in industry, especially in securities NLP (Natural Language Processing). Attached Figure Description

[0031] Figure 1 This is a strong negative example comparison experiment.

[0032] Figure 2 This is a fine-tuned type feature fusion model for a precisely labeled corpus.

[0033] Figure 3 Example of breaking down the full name of a company.

[0034] Figure 4 This is an example of the result of rule recall.

[0035] Figure 5 This is an example image of news text related to the financial field. Detailed Implementation

[0036] The present invention will be further described in detail below with reference to embodiments, but the implementation of the present invention is not limited thereto.

[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] To achieve the present invention, the following technical difficulties exist: 1. Firstly, the data quality generated by remote supervision is low. How can we improve the quality of data annotation as much as possible and reduce the interference of noisy data? 2. Financial corpora are dense with entity information and have complex relational patterns, placing high demands on the model's semantic understanding capabilities. Consider the following examples of bad cases encountered in practice: Figure 5 : a) The relationship between 'CITIC Limited' and 'Zhejiang Taifu Seamless Steel Pipe Co., Ltd.' in the first sentence cannot be determined even by human observation; b) It is also impossible to determine whether there is a shareholding relationship or a parent-subsidiary relationship between the three companies 'Xin'ao Holdings Investment Co., Ltd.' and 'Guofeng Culture, Tibet Naming and Yueqing Yicheng' in the second sentence; c) The third sentence first mentions the relationship between wholly-owned subsidiaries and subsidiaries, the acquisition relationship of subsidiaries, the relationship between parties acting in concert, the relationship between the controlling shareholder of the company, and finally the relationship between the person and the actual controller. It is very difficult to distinguish such a complex and overlapping relationship. How to optimize models to improve prediction accuracy for complex sentences is an urgent issue to be addressed.

[0039] 3. How to generate more effective high-quality labeled corpora to maximize annotation efficiency and reduce annotation manpower is an urgent issue to be addressed.

[0040] 4. When the model is implemented in actual engineering projects, how can we ensure that the prediction results can be directly entered into the database without manual review? This is an urgent issue to be addressed.

[0041] All of the above technical challenges need to be addressed. To this end, a method for designing and optimizing a relation extraction model based on remote supervision in the financial field is proposed. This method effectively solves the above technical challenges. To explain this technology in detail, the following embodiments are introduced for further explanation.

[0042] Definitions:

[0043] KG: Knowledge Graph is a structured semantic knowledge base used to describe concepts and their relationships in the physical world in symbolic form. Its basic building blocks are the "entity-relationship-entity" triple, as well as entities and their associated attribute-value pairs. Entities are interconnected through relations, forming a network-like knowledge structure.

[0044] BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained model for Natural Language Processing (NLP) proposed by Google. It's a transformer-based bidirectional encoder representation technology. Jacob Devlin and colleagues created and released BERT in 2018. Google is using BERT to better understand the semantics of user search queries.

[0045] eg (Eg): For example.

[0046] Label: A label is a description of the attributes of sample data in a dataset during supervised training tasks, and it is also the information that the final model aims to predict.

[0047] Named entity recognition (NER) is an abbreviation for named entity recognition.

[0048] Link: to connect or link different pieces of information.

[0049] brat is a web-based text annotation tool; that is, it is used to add annotations to existing text documents. brat is designed for structured annotation, where annotations are not free-form text, but have a fixed form that can be automatically processed and interpreted by computers.

[0050] embedding concat: tensor concatenation.

[0051] infer: to deduce, to predict.

[0052] bad case: a bad situation.

[0053] query: to search.

[0054] Knowledge graph completion (KGC) can predict missing links and is crucial for real-world knowledge graphs, which are often incomplete.

[0055] Example 1: This invention designs and optimizes a relation extraction model based on remote supervision in the financial field. This method enhances the ability to mine relationships between entities from text and obtains rich relational information, thereby supplementing the knowledge base in the financial field. Figure 2 As shown, it includes the following steps: 1) Using a financial data genealogy tool (KG) to preprocess and annotate the corpus of financial news (preferably financial news from Toutiao) to enable remote monitoring of the process. This process consists of three parts: corpus preprocessing, KG data processing, and remote monitoring. Baseline + Pretrain, that is, the base version based on pretraining, includes: 2) Generate (produce) manually annotated corpora based on remote supervision (since it is used for pre-training models, the annotated corpora do not need to be too large at first); 3) Fine-tune the pretrained BERT model using manually labeled data.

[0056] Example 2: This embodiment is a further optimization based on the above embodiment. The parts that are the same as those in the aforementioned technical solutions will not be repeated here. Furthermore, to better realize the design and optimization method for a relationship extraction model based on remote supervision in the financial field described in this invention, the following settings are specifically adopted: Step 1) specifically includes the following steps: 1.1) Corpus preprocessing: 1.1.1) Use web crawlers to collect relevant financial news corpora from relevant websites; 1.1.2) Perform data preprocessing on the crawled results, including removing HTML tags and segmenting the data into sentences; 1.1.3) Perform data filtering. Remove meaningless short phrases or short sentences with a single line that are title-like. If the length is less than 20 characters, remove them. 1.1.4) Perform innovative corrective processing, including unique dirty data removal and noisy data removal; 1.2) KG data processing: Since KG data is stored using the platform-independent data communication protocol Google Protocol Buffers, processing requires parsing the KG data according to a specific configuration file (.proto file). After parsing, the following two types of information files are created: name2type.txt --> Mapping file, specifically recording information about the mapping from entity names to entity types (quantity: 4,612,723): a) No duplicate names are considered (the constructed dict is one-to-one). b) e.g. (Huawei, 205); and triplets.txt --> the basic relational file, specifically recording each triplet in a KG on a single line, with each line formatted as: Entity 1 Name Relationship Entity 2 Name (Quantity: 2160372). e.g. (Huawei Technologies Co., Ltd., Founder, Ren Zhengfei); 1.3) The remote supervision module is used to annotate the existing data to complete the remote supervision.

[0057] Example 3: This embodiment is a further optimization based on any of the above embodiments. The parts identical to the aforementioned technical solutions will not be repeated here. Furthermore, to better realize the design and optimization method for a relationship extraction model based on remote supervision in the financial field described in this invention, the following settings are specifically adopted: The unique dirty data removal includes the following steps: 1.1.4.a.1) First, a high-quality entity set is constructed by using filtering methods at the coarse-grained level through specific constraints; a preferred specific solution is to regard entities whose confidence scores and lengths meet certain constraints as high-quality entities. Simply put, the shorter the entity length, the higher the threshold required for its confidence score. 1.1.4.a.2) Combine specific frequency information to filter meaningless high-frequency words at a fine-grained level; a preferred specific solution is to combine the frequency of occurrence of mentions obtained from statistics in the constructed dataset to build a blacklist of high-frequency meaningless entity words appearing in the article at a fine-grained level. 1.1.4.a.3) Based on the above two steps, the data of all entities in the corrected KG will be obtained. Using the AC automaton technique, useful instance relationship information will be extracted.

[0058] The key feature of this approach is its ability to extract truly important relational information from large-scale data samples with high accuracy, while removing various negative data that may affect subsequent tasks.

[0059] Example 4: This embodiment is a further optimization based on any of the above embodiments. The parts identical to the aforementioned technical solutions will not be repeated here. Furthermore, to better implement the relationship extraction model design and optimization method based on remote supervision in the financial field described in this invention, the collected data generally contains a certain amount of noise. For example, when faced with a large number of enumeration sentences in news reports, such as "The companies attending this event are: Company A, Company B, Company C, Company D," etc., the annotated corpus generated by this sentence structure will have a negative impact on our data due to mislabeling. To remove this negative noise, the following settings are specifically adopted: The noise data removal includes the following steps: 1.1.4.b.1) Limit the number of entities identified in a sentence; 1.1.4.b.2) Calculate the proportion of the length of all entities in a sentence to the total length of the sentence. This is called entity purity. If the entity purity is too high, it means that the sentence may be a coordinate sentence and should be discarded.

[0060] Example 5: This embodiment is a further optimization based on any of the above embodiments. The parts identical to the aforementioned technical solutions will not be repeated here. Furthermore, to better realize the design and optimization method for a relationship extraction model based on remote supervision in the financial field described in this invention, the following setup is specifically adopted: Step 1.3) uses a remote supervision module to annotate existing data to complete remote supervision, including the following specific steps: 1.3.1) Constructing the AC automaton; a preferred approach is to use the target string set (i.e., read the name2type.txt file and extract all the keys to form the target string set) to construct an automaton with relevant information; 1.3.2) For the input sentence, the AC automaton is used to perform the longest match of the target entity in the sentence through its internal tire tree structure; 1.3.3) Read triplets.txt and construct a dictionary mapping (Entity 1 name, Entity 2 name) to relations; 1.3.4) Combine multiple entities matched in the sentence into pairs to form a key and look it up in the dictionary. If the dictionary returns the relationship between the entities, mark it and output a marked corpus.

[0061] Example 6: This embodiment is a further optimization based on any of the above embodiments. The parts identical to the aforementioned technical solutions will not be repeated here. Furthermore, to better realize the design and optimization method for a relationship extraction model based on remote supervision in the financial field described in this invention, the following settings are specifically adopted: When implementing step 1.3.4): 1.3.4.1) When there are multiple relationships between entities, we can count the number of each relationship, define the priority of relationship matching according to the principle of balancing the number of each type as much as possible, and then select the relationship with the highest priority as the label for processing. 1.3.4.2) For special cases where some sentences do not express the expected relationship between entities, specific data annotation and denoising techniques are used, e.g.: a) Elon Musk answers reporters' questions on a SpaceX pressconference; b) SpaceX was founded in 2002 by Elon Musk; A preferred approach is to address relationships that are prone to introducing a lot of noise (such as CEO, founder, subsidiary, shareholder relationships, etc.) by adding keyword restrictions. Only when the required keywords appear in a sentence will it be considered as the corresponding relationship type.

[0062] 1.3.4.3) To achieve effective annotation when KG relations are lacking, candidate sentences are discovered by using knowledge about specific relation entity pairs in the KG, and then corresponding annotations are performed using keywords; the annotation quality of the results produced by this method meets the requirements after sampling.

[0063] 1.3.4.4) For some relations, while meeting business requirements, adding type restrictions can further standardize the types of entities in the relations; because as mentioned earlier, entity information helps with relation extraction, and the most important entity information is the entity type information, which helps to assist the training model in capturing entity type information.

[0064] 1.3.4.5) To resolve matching ambiguity issues, the following techniques are employed: a) Using the ner-link tool, the results identified by ner are linked to the corresponding entities in KG. The linking process will resolve ambiguities. b) Matching-based method. The advantage of this method is that it is faster. When tested on seven million sentences, the ner-link method takes 43 hours in a single process, but the matching method only takes 3 hours. Matching also generates more labeled data, and the labeled data results meet expectations. After using various noise reduction techniques, the data cleaning ratio reached 0.7986, and the quality of the sampled evaluation data annotation was significantly improved.

[0065] Example 7: This embodiment is a further optimization based on any of the above embodiments. The parts identical to the aforementioned technical solutions will not be repeated here. Furthermore, to better realize the design and optimization method for a relationship extraction model based on remote supervision in the financial field described in this invention, the following settings are specifically adopted: Step 2) includes the following steps: 2.1) NER identifies entities in the news corpus, and after sampling, combines entities in sentences pairwise according to given type requirements to extract the relationship model for prediction; 2.2) Mount the prediction results to the input address marked by brat via cloud disk, and align its format to the .ann and .txt formats used by brat; 2.3) Manually annotate the corpus; 2.4) Read the labeled corpus, parse it back into JSON format, and store it as finely labeled corpus; 2.5) Count the number of labels for each type of relationship and supplement them in a targeted manner by combining remote supervision (as a labeling reference) for relationship types with fewer labels.

[0066] Part of the manually labeled corpus will be used to fine-tune the pre-trained model, and the other part will serve as a new evaluation set. In the implementation, the brat platform will label all predicted relations and NER results for each sentence. This allows the NER results to be corrected while labeling relations, and the NER can also use this data to optimize the model.

[0067] Example 8: This embodiment is a further optimization based on any of the above embodiments. The parts identical to the aforementioned technical solutions will not be repeated here. Furthermore, to better realize the design and optimization method for a relation extraction model based on remote supervision in the financial field described in this invention, the following settings are specifically adopted: When executing step 2.3) of manually annotating the corpus, the following situation will occur: 2.3.1) If an entity recognition error is found, correct the entity; 2.3.2) If the recalled relation is incorrect, correct the relation type; 2.3.3) For relationships not yet recalled, please supplement; 2.3.4) The results of identifying unrecalled NERs and the resulting relationships are supplemented; 2.3.5) Hard cases that are considered to be easily misrecalled by the relation extraction model are labeled as unrelated. Negative cases that are considered to be necessary to be added to the fine-tune set (i.e., strong negative cases) are labeled as particularly unrelated. Cases with particularly unrelated labels are guaranteed to be added to the training corpus as strong negative cases. The remaining negative cases are randomly sampled from the remaining entity pairs that have not been labeled with relations.

[0068] Because the relation extraction model may falsely recall some bad cases caused by interference from other companies, such as the following: For example: The company mistakenly recalled a 'shareholder': The company's advertising business has two important operating entities, namely the controlling shareholder... <e1>Falcon Networks <e1>and <e2>SPE <e2>However, their performance last year was not ideal.

[0069] The negative samples in the training set are of low quality because they were randomly sampled. However, the total number of negative samples generated by remote supervision far exceeds the number of positive samples, making the random negative sampling less effective against the positive examples in the training set. They can be easily distinguished from positive samples through simple learning, resulting in insufficient ability of the relation extraction model to recognize sentences from multiple companies.

[0070] Therefore, to solve this problem, we tried to add negative samples (called strong negative samples) generated by different entities in the sentences corresponding to positive examples to the training set.

[0071] The results of the retrained model with added strong negative samples and the baseline model on the TN dataset (public test dataset) are as follows: Figure 1 As shown in the figure, the bolded areas indicate the relationship types that improve accuracy.

[0072] As can be seen, adding hard negative instances improved the model's prediction accuracy for some difficult relationships. However, because these increased hard negative instances are more likely to be mislabeled, the model's recall decreased. In industrial relationship prediction tasks, improving accuracy is more important than decreasing recall, as it reduces the time and manpower costs of reviewing the reasonableness of prediction results. Furthermore, the optimized model showed a decrease in both accuracy and recall for upstream and downstream relationships within products. Sampling and examining the generated strong negative samples revealed the following specific reasons: 1. The number of negative samples related to product relationships accounts for almost half of the total number of strong negative samples, which does not meet the preset ratio of 2000 strong negative samples added for each relationship type. The reason for the imbalance is that the method for generating strong negative samples does not limit the number of strong negative samples generated for each relationship type. Instead, it directly shuffles all generated strong negative samples and takes the total number. Since multiple products often appear in sentences, the distribution of strong negative samples among products is even larger. This causes the method of randomly shuffling and taking the total number to retrieve too many strong negative samples among products.

[0073] 2. At the same time, KG is not comprehensive enough in terms of product terms and the relationships between products. Many strong negative samples between products are actually positive samples. These false negatives affect the model's performance and cause the model's predictive performance on product relationships to decline.

[0074] Example 9: This embodiment is a further optimization based on any of the above embodiments. The parts identical to the aforementioned technical solutions will not be repeated here. Furthermore, to better realize the relationship extraction model design and optimization method based on remote supervision in the financial field described in this invention, and to further improve the model's predictive performance, a new type feature technology is introduced, specifically adopting the following setting: Step 3) during fine-tuning includes: Compared to directly using the embedding (a set of tensors used to store object information) corresponding to [CLS] after BERT encoding and performing softmax (probability normalization function) through a fully connected layer for relation prediction, considering that entity information is also important for relation prediction, and the most important information among entity information is the entity type information, the type label embedding of the entity (including entity 1 and entity 2) and the embedding corresponding to [CLS] are concatenated and then input into the fully connected layer for inference. Previously, this technique would label multiple positions of an entity at the same time, but now it labels only one position at a time (because with the addition of type embedding, the type embedding of different positions of the same entity is also different, containing its surrounding semantic information, so it is processed separately).

[0075] Based on this, the modified strong negative samples were added at the same time. It was observed that the strong negative samples did not penalize the model much, so the upper limit of the number of strong negative samples was increased. The preferred method is to increase the upper limit of the number of strong negative samples from 2,000 to 5,000.

[0076] Meanwhile, the remote supervision end added a subsidiary label restriction: previously, only the presence of the subsidiary keyword in the sentence was required to meet the requirement, regardless of its specific position. However, this revealed a bad case (a judgment error) where parallel subsidiaries were mistakenly recalled. For example: The company's controlling subsidiary, Zhonglai Minsheng, and its wholly-owned subsidiary, Zhonglai Zhilian, signed an "EPC general contracting agreement for household photovoltaic power generation project" with Shanghai Yuanye New Energy Co., Ltd.

[0077] Among them, "Zhonglai Zhilian" and "Shanghai Yuanye New Energy Co., Ltd." will be mistakenly recalled as subsidiaries.

[0078] Therefore, a restriction is added: only when the word 'subsidiary' appears between the two entities in the sentence is it marked as a positive example; otherwise, it is marked as unrelated. This marking as unrelated is also deliberate. If it were not marked as unrelated and simply discarded, the trained model would still predict some bad cases in the subsidiary relationship where the word 'subsidiary' does not appear between the two entities, such as: As the younger brother of Zhang Rongming, the actual controller of Aimer Group, Zhang Ronglong's Suzhou Rongshi not only shares the same business name as Aimer Group's subsidiaries Suzhou Aimer and Suzhou Meishanzi, but Zhang Ronglong also serves as a director and general manager of Suzhou Aimer and Suzhou Meishanzi.

[0079] This indicates that directly discarding the case does not penalize the model enough; instead, labeling them as unrelated and then solving these two types of cases is necessary.

[0080] In addition, following the implemented fine-calibration process, 738 data points were labeled (431 positive examples, including 221 related products), and then 500 randomly sampled negative examples were added from the remote supervision end, for a total of 1200 data points to fine-tune the model.

[0081] Overall, fine-tuning has a significant positive effect on the model, with most metrics showing positive results. The model showed the greatest performance improvement in related product relationships after fine-tuning, thanks to the concentrated labeling of many related product relationships in the fine-tuned dataset, which accounted for half of the total positive examples. Shareholder relationships and subsidiary relationships were also labeled, and the model was corrected accordingly, leading to performance improvements.

[0082] Example 10: This embodiment is a further optimization based on any of the above embodiments. The parts identical to the aforementioned technical solutions will not be repeated here. Furthermore, to better implement the design and optimization method for a relationship extraction model based on remote supervision in the financial field described in this invention, due to the incompleteness of the KG (Key Knowledge Group), many companies are not in the KG, so it is necessary to supplement the aliases of these companies. Specifically, the following setting method is adopted: When using a web crawler to crawl relevant financial news corpora from relevant websites, an alias supplementation process is also included: 1. For a given company, perform alias mining from both the model and rule dimensions; 2. Use a search engine to search for a given company and crawl the top 10 articles containing the query, considering them valid, or use the group's internal financial news database as the corpus source for the model. That is, the model's corpus source has two sources: one is to use Toutiao search to search for TYC (related financial companies) and crawl the top 10 articles returned by the search, considering those containing the query as valid; the second is the group's internal financial news database. 3. To reduce the workload of manual review, a confidence function was designed to score the confidence of the results mined by the model. The top results with high scores can be directly entered into the database, while the bottom results need to be manually reviewed before being entered into the database. The design idea for the confidence function is as follows: First, for a predicted alias candidate, there may be a series of sentences that are predicted to express the relation, i.e., expressed as formula (1). .

[0083] Since the more entities a sentence contains, the more interference it may bring to the model, the lower its contribution to the confidence of the alias candidate. Therefore, the score calculated for a single sentence is divided by the total number of company-type entities in the sentence, and the square root controls the degree of penalty, which is expressed as formula (2).

[0084]

[0085] Meanwhile, the more sentences mention these two entities, the higher the overall confidence of the alias candidate will be. Therefore, the confidence score of the candidate will be obtained by summing all the sentences in the set, which is expressed as formula (3).

[0086] ; The full name of the company is: Company alias: ,sentence: The total number of company entities in the sentence: The model predicts the score by which the sentence expresses the alias relation: score(sent) , yes The confidence function for calculating the true alias: ; 4. The candidate aliases generated by the rules will be further processed by NER. Only candidate words that are identified as company types by NER will be retained, and the rest will be discarded. 5. The alias results generated by the rule-based method were sampled and manually evaluated. The accuracy rate was over 95%, and they can be directly added to the database. To verify this, 12,502,504 additional alias relationships were added for more than 5 million companies.

[0087] For most companies' full names, such as Figure 3 As shown, it can be broken down into four parts: Location (Loc), Company Keywords (CN), Field, and Suffix.

[0088] During the process of generating alias candidates using rules, suffixes are removed, followed by location information, resulting in company keywords. Keywords can be combined with domains or place names to form candidates. Finally, a blacklist filters out non-compliant combinations, yielding the results generated by the rules. This is then processed by a NER (Network Provider) to recall candidates identified as company types, which can then be used as aliases. The final recall results are as follows: Figure 4 As shown.

[0089] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications or equivalent changes made to the above embodiments based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for designing and optimizing a relation extraction model based on remote supervision in the financial field, characterized by the following steps: 1) Using a knowledge graph in the financial field, preprocess and annotate the relationships in financial news corpora to achieve remote monitoring processes; including the following steps: 1.1) Corpus preprocessing: 1.1.1) Use web crawlers to collect relevant financial news corpora from relevant websites; 1.1.2) Perform data preprocessing on the crawled results, including removing HTML tags and segmenting the data into sentences; 1.1.3) Perform data filtering; 1.1.4) Perform specific dirty data removal and noise data removal; The specific dirty data removal includes the following steps: 1.1.4.a.1) High-quality entity sets are constructed by using filtering methods at the coarse-grained level; 1.1.4.a.2) Combine specific frequency information to filter meaningless high-frequency words at a fine-grained level; 1.1.4.a.3) Using the AC automaton technique, extract useful instance relationship information from the data of all entities in the corrected knowledge graph; The noise data removal includes the following steps: 1.1.4.b.1) Limit the number of entities identified in a sentence; 1.1.4.b.2) Calculate the ratio of the length of all entities in a sentence to the total length of the sentence. This is called entity purity. If the entity purity is too high, it indicates that the sentence may be a coordinate sentence and should be discarded. 1.2) Knowledge graph data processing: Parse the knowledge graph data according to the specific configuration file .proto file, and create name2type.txt --> mapping file and triplets.txt --> basic relationship file; 1.3) Complete remote supervision by annotating existing data; including the following specific steps: 1.3.1) Construct an AC automaton; 1.3.2) For the input sentence, the AC automaton is used to perform the longest match of the target entity in the sentence through its internal Trie tree structure; 1.3.3) Read triplets.txt and construct a dictionary mapping the relationships; 1.3.4) For multiple entities matched in the sentence, combine them pairwise into a key to look up the dictionary. If the dictionary returns a relationship between the entities, mark it and output a marked corpus; when implementing step 1.3.4): 1.3.4.1) When there are multiple relationships between entities, we can count the number of each relationship, define the priority of relationship matching according to the principle of balancing the number of each type as much as possible, and then select the relationship with the highest priority as the label for processing. 1.3.4.2) For special cases where some sentences do not express the expected relationship between entities, data annotation and denoising techniques are used; 1.3.4.3) To achieve effective annotation when knowledge graph relationships are lacking, candidate sentences are discovered by using knowledge about specific relation entity pairs in the knowledge graph, and then corresponding annotations are performed using keywords; 1.3.4.4) For some relationships, while meeting business requirements, type restrictions are added to standardize the types of entities in the relationships; 1.3.4.5) To resolve matching ambiguity issues, the following techniques are employed: a) Using named entity recognition and entity linking tools, the results identified by named entity recognition are linked to the corresponding entities in the knowledge graph, and the linking process resolves ambiguity. b) Matching-based methods; 2) Generate manually annotated corpora; 3) Fine-tune the pre-trained BERT model using manually labeled data.

2. The method for designing and optimizing a relation extraction model based on remote supervision in the financial field according to claim 1, characterized in that: Step 2) includes the following steps: 2.1) Named entity recognition identifies entities in the news corpus, and after sampling, it combines entities in sentences pairwise according to given type requirements to extract the relationship model for prediction. 2.2) Mount the prediction results to the input address marked by brat via cloud disk, and align its format to the .ann and .txt formats used by brat; 2.3) Manually annotate the corpus; 2.4) Read the labeled corpus, parse it back into JSON format, and store it as finely labeled corpus; 2.5) Count the number of labels for each type, and supplement the labels for relationship types with fewer labels by combining remote supervision.

3. The method for designing and optimizing a relation extraction model based on remote supervision in the financial field according to claim 2, characterized in that: When performing step 2.3) of manually annotating the corpus, 2.3.1) If an entity recognition error is found, correct the entity; 2.3.2) If the recalled relation is incorrect, correct the relation type; 2.3.3) For relationships not yet recalled, please supplement; 2.3.4) The identification results of unrecalled named entities and the resulting relationships are supplemented; 2.3.5) Samples that are easily misrecalled by the relation extraction model are labeled as unrelated; among them, negative samples that need to be added to the fine-tuning training set are labeled as particularly unrelated, and samples with particularly unrelated labels are added to the training corpus as strong negative samples; the remaining negative samples are randomly sampled from the remaining entity pairs that have never been labeled with relations.

4. The method for designing and optimizing a relation extraction model based on remote supervision in the financial field according to claim 1, characterized in that: Step 3) includes the following when making fine adjustments: Instead of directly using the embedding vectors corresponding to the BERT-encoded classification label vectors to perform softmax in a fully connected layer for relation prediction, we concatenate the embedding vectors of the entity type label and the embedding vectors corresponding to the classification label vectors, and then input them into the fully connected layer for prediction. On this basis, corrected strong negative samples are added, and the upper limit of the number of strong negative samples is increased. Meanwhile, remote monitoring will increase restrictions on subsidiary labels.

5. The method for designing and optimizing a relation extraction model based on remote supervision in the financial field according to claim 1, characterized in that: When performing step 1.1.1), an alias completion process is also included: 1.1.1.1 For a given company, perform alias mining from both the model and rule dimensions; 1.1.1.

2. Use a search engine to search for a given company and crawl the top 10 returned articles that the query considers valid, or use the financial news database within the group as the corpus source for the model. 1.1.1.

3. For the results mined by the model, design a confidence function to score the confidence of the results. The top results with high scores are directly entered into the database, while the bottom results need to be manually reviewed before entering the database. 1.1.1.4 The candidate aliases generated by the rules will be further processed by named entity recognition. Only candidate words that are identified as company types by named entity recognition will be retained, and the rest will be discarded. 1.1.1.

5. Sample the alias results generated by the rule-based method. After manual evaluation, the accuracy rate is above 95%, and they can be directly added to the database.

Images