Method for establishing large language correlation judgment large model for financial information retrieval

Abstract

The invention provides a method for establishing a big language correlation judgment model for financial information retrieval, which comprises the following steps of: screening out difficult financial terms by calculating ppl numerical values of financial term pairs in total financial terms, weighting logits hitting financial term tokens by adopting a weighting formula to obtain a composite financial question set, inputting the composite financial question set into an expert question-answering system in the RAG financial field, and establishing a big language correlation judgment model for the financial information retrieval. Data pairs of user questions and search texts are obtained to serve as a financial question and answer data set, correlation is judged according to the financial question and answer data set to obtain a useful data set, the useful data set is combined with the financial question and answer data set, then the ppl value is calculated again, difficulty classification is carried out, and sequential superposition is carried out to obtain a fine tuning model; and performing correlation judgment on the fine-tuning large model by using the useful data set to obtain a positive sample training data set and a negative sample training data set, and performing enhanced feedback training on the fine-tuning large model to obtain a final correlation judgment large model, thereby solving the core problems of financial terminology confusion, financial entity disambiguation failure, financial composite logic analysis missing and the like.

Description

Technical Field

[0001] This invention relates to the technical field of financial information retrieval, specifically to a method for establishing a large-scale model for determining the relevance of large languages ​​in financial information retrieval. Background Technology

[0002] Currently, search systems based on general-purpose large language models are widely used in the field of financial information retrieval. Their core capability lies in the semantic matching of user natural language questions with target documents, i.e., relevance assessment. Existing technical solutions for financial search systems based on general-purpose large language models typically employ a large training corpus provided by the general-purpose model for further pre-training. This pre-training relies heavily on the general-purpose model's inherent world knowledge and analytical capabilities. Fine-tuning is performed using publicly available financial texts such as research reports and announcements, allowing the general-purpose model to gradually improve its financial knowledge and analytical capabilities, thus achieving relevance assessment. However, existing relevance assessment techniques suffer from two major systemic flaws:

[0003] 1. Existing general-purpose large-scale models are mostly trained using incremental pre-training, which does not consider the gradual transfer of financial knowledge and skills, i.e., the course learning method. Using course learning can effectively maintain the general capabilities of large-scale models and improve their specialized capabilities in financial verticals compared to not using it. For correlation judgment models, the business training sets constructed by current large-scale model fine-tuning methods do not specify the difficulty stratification of samples required by the course learning method. Large-scale models are prone to a "seesaw" phenomenon in performance evaluation after training. For example, the specialized capabilities of the large-scale model may improve, but the general capabilities may decrease, which may affect the overall performance of the large-scale model in correlation judgment.

[0004] 2. Text relevance assessment in financial information retrieval involves the ability to identify and understand financial indicators, mine and match financial entities, and analyze financial business processes between user queries and search results. Existing large-scale models and their applications can understand the meaning of common financial terms, but they are very prone to misinterpreting the meaning and role of financial entities in different scenarios, causing entity disambiguation to fail. They also have significant shortcomings in the analysis of complex financial indicators, resulting in a bottleneck in domain adaptability. Summary of the Invention

[0005] The technical problem that this invention aims to solve is to address the core issues of existing correlation judgment techniques, such as confusion of financial terminology, failure of financial entity disambiguation, and lack of analysis of financial complex logic, caused by the lack of professional and detailed financial training data and fine-tuning methods. This is to avoid seriously affecting the effectiveness and accuracy of correlation judgment.

[0006] The present invention provides a method for establishing a large-scale model for determining the relevance of large languages ​​in financial information retrieval, comprising the following steps:

[0007] By utilizing the large model prompt project and employing a multi-dimensional financial semantic feature extraction strategy, and by repeatedly sampling neighboring financial terms, a full set of financial terms with financial term pairs is obtained from the financial knowledge corpus of massive Internet content.

[0008] Calculate the PPL values ​​of financial term pairs in the full set of financial terms. If the PPL values ​​differ by less than 20%, they are classified as difficult financial terms.

[0009] Based on complex financial terms, generate compound financial questions, and finally generate a set of compound financial questions;

[0010] When generating complex financial questions based on complex financial terms, the logits of the matching financial term tokens are weighted, and the weighting coefficient alpha is calculated using the following formula:

[0011] alpha = max(1, 0.5 * log10(ppl) i ))

[0012] The large model outputs the relationship between PPL and loss, and classifies the input data into easy and difficult categories based on the PPL value, so as to generate a training dataset with different levels of difficulty required for course learning as a set of complex financial questions containing features such as market indicators, financial terms, and stock market data.

[0013] A set of complex financial questions is input into the RAG financial expert question-and-answer system to obtain data pairs of user questions and search texts as financial question-and-answer datasets. Few-shots of prompts to judge relevance are designed according to business rules to obtain the usefulness judgment results of the large model. Based on multiple usefulness judgment results, the relevance results of the output of the large model are determined by the majority vote, resulting in useful and useless datasets.

[0014] After combining the useful dataset with the financial question-and-answer dataset, the PPL value is calculated again and classified into easy and difficult categories and stacked sequentially to obtain the training dataset. After fine-tuning, the large model is trained as the fine-tuned large model.

[0015] The large model is fine-tuned using a useful dataset to determine relevance, resulting in a training dataset of positive and negative samples. The fine-tuned large model is then reinforced with feedback training to obtain the final large model for relevance determination.

[0016] Preferably, the step of repeatedly sampling adjacent financial terms includes: for financial terms with the same number of words or with a difference of less than 10% in word count, combining the financial terms in pairs to obtain a full set of financial terms with financial term pairs.

[0017] Preferably, the full range of financial terms includes definitions of financial indicators, financial entities, and basic financial services.

[0018] Preferably, the composite financial problem set includes definitions of financial terms, calculation problems containing financial terms, or enterprise application cases containing financial terms.

[0019] Preferably, when generating complex financial questions based on complex financial terms, the process also includes the relationship between the logits and loss of the large model output token, and the formula for the relationship between the logits and loss is as follows:

[0020] pi = exp(z i ) / Σ j exp(z j loss=-Σ i (yi*log(pi))

[0021] Among them, z i Let pi be the predicted token of the i-th token, and z be the token of the i-th token. i logits(z) i The probability distribution transformed by the softmax function.

[0022] Preferably, the relationship between the large model output PPL and the loss is as follows:

[0023]

[0024] In the formula, PPL represents the ppl value of the financial term, which is calculated only from the logits of the financial term token pairs. The loss is calculated by exp, where loss is the average cross-entropy loss of each word in the entire sequence.

[0025] This invention proposes a method for establishing a large-scale model for determining the relevance of financial information retrieval. It filters out difficult financial terms by calculating the PPL values ​​of financial term pairs from a full set of financial terms. A weighted formula is used to weight the logits of the matched financial term tokens to obtain a composite financial question set. This set is input into the RAG financial expert question-and-answer system to obtain data pairs of user questions and search text as a financial question-and-answer dataset. Relevance is determined based on the financial question-and-answer dataset to obtain a useful dataset. The useful dataset is combined with the financial question-and-answer dataset, and the PPL values ​​are calculated again, categorized by difficulty, and sequentially stacked to obtain a fine-tuned large-scale model. The useful dataset is used to determine the relevance of the fine-tuned large-scale model, resulting in positive and negative sample training datasets. The fine-tuned large-scale model is then reinforced and trained to obtain the final relevance determination model. This method solves the core problems of existing relevance determination techniques, such as confusion of financial professional terms, failure of financial entity disambiguation, and lack of analysis of complex financial logic. Attached Figure Description

[0026] Figure 1This is a flowchart of a method for establishing a large-scale model for determining the relevance of large languages ​​in financial information retrieval, provided as an embodiment of the present invention. Detailed Implementation

[0027] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0028] like Figure 1 As shown, this embodiment of the invention provides a method for determining the relevance of a large language model for financial information retrieval. By introducing a compound question generation strategy and an expert system, a high-quality corpus of financial knowledge and analytical capabilities is constructed. Through a three-stage training architecture and a hybrid course learning method, the relevance judgment performance of a general large model in financial scenarios is improved. The method includes the following steps:

[0029] By leveraging the large model prompt project and employing a multi-dimensional financial semantic feature extraction strategy, this approach involves repeatedly sampling neighboring financial terms. Specifically, for financial terms with the same number of words or a difference of less than 10%, these terms are paired up. This method extracts a full set of financial terms with financial term pairs from a massive corpus of financial knowledge content on the internet, such as financial indicators, financial entities, and definitions of basic financial businesses.

[0030] The PPL (Programmable Logic Product) values ​​for all financial term pairs are calculated. If the PPL values ​​differ by less than 20%, the financial term is considered relatively difficult for the large model to distinguish. Financial term pairs meeting this condition are categorized as difficult financial terms. Based on these difficult financial terms, complex financial questions are generated, ultimately producing a set of complex financial questions. Examples of complex financial questions include: explaining the definition of a financial term, calculation problems containing financial terms, or enterprise application cases containing financial terms. These three aspects—financial term definitions, calculation logic, and application cases—constitute the financial domain knowledge and capabilities that the relevant large model needs to learn.

[0031] In the process of generating complex financial questions based on difficult financial terms, the large model outputs the relationship between the logits and loss of the token, as well as the relationship between the PPL and loss of the large model, to generate a training dataset of varying difficulty levels required for course learning as a set of complex financial questions.

[0032] Specifically, the relationship between the logits of the large model's output token and the loss is as follows:

[0033] pi = exp(z i ) / Σ j exp(zj ) loss = -Σ i (yi * log(pi))

[0034] Where, the predicted token for the i-th token: z i The logits(z i ) are converted into a probability distribution pi through the softmax function, and the cross-entropy between the predicted probability distribution p and the one-hot encoding distribution y of the true "next word" is calculated.

[0035] When calculating the loss of the predicted token, the logits that hit the financial term token are weighted, and the calculation formula for the weighting coefficient alpha is:

[0036] alpha = max(1, 0.5 * log10(ppl_i))

[0037] The relationship between the large model PPL and loss is as follows:

[0038]

[0039] Where, PPL represents the ppl value of the financial term, and this value is calculated only through the logits of the financial term token pair. Perform the exp operation on Loss, and Loss is the average value of the cross-entropy loss of each word on the entire sequence.

[0040] Classify the input data according to the ppl value. If 0 < ppl <= 15, it indicates that the large model's prediction of the input data is relatively simple, and label this type of dataset as [Simple]; if 15 < ppl <= 30, it indicates that the large model's prediction of the input data is relatively medium, and label this type of dataset as [Medium]; if 30 < ppl, it indicates that the large model's prediction of the input data is relatively difficult, and label this type of dataset as [Difficult], and obtain a composite financial problem set containing features such as market indicators, financial terms, and stock quotes.

[0041] Input the composite financial problem set into the RAG (Retrieval-Augmented Generation system) financial domain expert Q&A system, which is used as an interface for dynamically updating domain knowledge, to obtain a data pair of user questions and search texts as a financial Q&A dataset. Design few-shots for judging relevance according to business rules, and classify the judgment results of the usefulness of the large model into [Completely Useful], [Partially Useful], [Useless but Relevant], and [Completely Useless].

[0042] Based on multiple (for example, 7) usefulness judgment results, determine the relevance result of the large model output according to the majority voting result. Consider [Completely Useful], [Partially Useful], and [Useless but Relevant] as the final useful dataset, and consider [Completely Useless] as the final useless dataset.

[0043] After combining the useful dataset with the financial question-and-answer dataset, the PPL value is calculated again and classified into easy and difficult categories. The PPL values ​​are stacked in the order of easy-medium-difficult to obtain the training dataset. After fine-tuning, the large model is trained to obtain the fine-tuned large model.

[0044] The reinforcement learning process involves using a useful dataset to fine-tune the large model for relevance judgment, obtaining positive and negative sample training datasets, and then using reinforcement feedback training on the fine-tuned large model to obtain the final large model for relevance judgment.

[0045] This invention integrates a multi-stage training framework of progressive knowledge transfer and course learning, constructing a three-stage training architecture: incremental domain knowledge post-training, supervised fine-tuning training, and reinforcement feedback training. It designs a hybrid course learning strategy, gradually increasing task difficulty and knowledge depth to improve the model's relevance judgment ability in search scenarios. Ultimately, it solves the core problems of existing relevance judgment technologies, such as confusion of financial terminology, failure of financial entity disambiguation, and lack of analysis of financial complex logic, due to the lack of professional and detailed financial training data and fine-tuning methods. This avoids seriously affecting the effectiveness and accuracy of relevance judgment.

[0046] The embodiments of this invention, by constructing a comprehensive and diverse corpus of financial characteristic questions and professional and effective expert answers, eliminate the inherent financial cognitive defects of large models, improve the accuracy and analytical depth of large models in handling financial issues in relational judgment, and significantly alleviate the problems of confusion of professional terms and failure of entity disambiguation.

[0047] Through multi-stage knowledge transfer training, the large-scale correlation judgment model of this invention gradually masters complex financial analysis capabilities, and its analysis dimensions and depth are on par with those of financial experts. This improves the large-scale correlation judgment model's analytical and judgment capabilities when dealing with complex financial problems, thereby enhancing its ability to conduct complex financial analysis.

[0048] This invention, by constructing a dataset of question-and-answer pairs of varying difficulty and a dataset of complex financial terms, has obtained a comprehensive and professional dataset of financial knowledge and analytical capabilities, effectively improving the financial understanding capabilities of the large-scale relevance judgment model.

Claims

1. A method for establishing a large language correlation determination large model for financial information retrieval, characterized in that, Comprise the following steps: By using the large model prompt engineering, the full-amount financial term pairs with financial term pairs are obtained from the financial knowledge corpus in massive Internet content through repeated sampling of adjacent financial terms by using a multi-dimensional financial semantic feature extraction strategy; The ppl values of the financial term pairs in the full-amount financial term pairs are calculated, and if the ppl values differ by 20%, the financial term pairs are classified as difficult financial terms; Compound financial question sets are generated according to the difficult financial terms, and finally the compound financial question sets are generated; When generating compound financial questions according to the difficult financial terms, the logits of the hit financial term tokens are weighted, and the weighting coefficient alpha is calculated as follows: alpha = max(1, 0.5*log10(ppl i )) The relationship between the PPL and the loss output by the large model is determined, the input data is classified according to the ppl value, and the difficulty classification is achieved to generate a difficulty hierarchical training data set required for course learning as a compound financial question set containing market indicators, financial terms, stock market trends and other features; The compound financial question set is input into the RAG financial field expert question answering system to obtain a data pair of user question and search text as a financial question answering data set, the relatedness of the few-shots is determined according to the business rule design prompt, the usefulness judgment result of the large model is obtained, the relatedness result output by the large model is determined according to the majority voting result of multiple usefulness judgment results, and the useful data set and the useless data set are obtained; The useful data set is combined with the financial question answering data set, and the ppl value is calculated again and classified according to the difficulty, and the training data set is stacked in sequence to obtain a fine-tuned large model as a fine-tuned large model; The useful data set is used to judge the relatedness of the fine-tuned large model, and the positive sample and negative sample training data set is obtained, the fine-tuned large model is trained by reinforcement feedback, and the final relatedness determination large model is obtained.

2. The method of claim 1, wherein the method further comprises: determining a relevance of the financial information based on the first information and the second information; and determining whether the relevance of the financial information is greater than a predetermined threshold. The repeated sampling of adjacent financial terms comprises: for financial terms with the same number of characters or a difference of 10% or less in the number of characters, the financial terms are combined two by two to obtain full-amount financial term pairs.

3. The method of claim 1, wherein the method further comprises: determining a large language model for financial information retrieval based on the large language model for general information retrieval; and training the large language model for financial information retrieval based on the large language model for general information retrieval. The full-amount financial term pairs comprise financial indicators, financial entities and basic financial business definitions.

4. The method of claim 1, wherein the method further comprises: determining a large language model for financial information retrieval based on the large language model for general information retrieval; and training the large language model for financial information retrieval based on the large language model for general information retrieval. The compound financial question set comprises financial term definition explanation, calculation problems containing financial terms or enterprise application cases containing financial terms.

5. The method of claim 1, wherein the method further comprises: determining a relevance of the financial information based on the first information and the second information; and determining a relevance of the financial information based on the first information and the second information. When generating compound financial questions according to the difficult financial terms, the logits and loss relationship of the token output by the large model is also included, and the logits and loss relationship formula is as follows: pi = exp(z i ) / ∑ j exp(z j ) loss = -∑ i (yi*log(pi)) where z i is the predicted token for the i-th token, p i is the probability distribution converted from logits(z i ) by the softmax function.

6. The method of claim 1, wherein the method further comprises: determining a relevance of the financial information based on the first information and the second information; and determining a relevance of the financial information based on the first information and the second information. The PPL and loss relationship formula output by the large model is as follows: In the formula, PPL represents the ppl value of the financial term, which is calculated only by the logits of the financial term token pair, and the Loss is subjected to exp operation, and the Loss is the average value of the cross-entropy loss of each word in the entire sequence.

Images