Method and device for evaluating large financial model generation text

Abstract

The invention discloses a method and a device for evaluating a text generated by a large financial model. The method and the device are used for solving the technical problem of low evaluation accuracy caused by an existing evaluation technology for the text generated by the large financial model. The method comprises the following steps: taking an obtained financial text data test set as input of a preset training converter, and outputting identity data of a plurality of financial expert reviews; determining a plurality of machine learning models according to the financial text data test set and the identity data of each financial expert reviewer; generating an enhanced financial text data set according to the obtained financial text data training set; carrying out model training on the plurality of machine learning models by adopting the enhanced financial text data set, and determining a plurality of target language evaluation models; inputting each piece of financial text test data into a preset to-be-evaluated financial large model, and outputting a plurality of to-be-evaluated financial large model generation texts; and evaluating the to-be-evaluated financial large model generation text by adopting each target language evaluation model to generate a text evaluation result.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and device for evaluating text generated by a large financial model. Background Art

[0002] With the rapid development of generative artificial intelligence, the capabilities of large language models have gradually improved and have been widely used in various natural language generation tasks.

[0003] The recent trend is to introduce big models into vertical fields, such as finance, to complete tasks such as risk assessment and financial text analysis. As more vertical big models are launched, there is an urgent need for corresponding accurate and reliable evaluation solutions to evaluate the capabilities of the models in the corresponding fields.

[0004] Most of the existing evaluation technologies for text generated by financial big models use the generalization ability of a single evaluation big model, allowing the single evaluation big model to follow the written instructions or context learning methods to score the text content generated by the financial big model. However, this process is too dependent on the generalization ability of the single evaluation big model itself, and the single evaluation big model will have preferences during evaluation, which makes the evaluation results not objective enough and leads to low evaluation accuracy. Summary of the invention

[0005] The present invention provides a method and device for evaluating text generated by a large financial model, which are used to solve the technical problem that the evaluation technology for generating text by a large financial model has low evaluation accuracy.

[0006] The first aspect of the present invention provides a method for evaluating text generated by a large financial model, comprising:

[0007] Acquire a financial text data set, wherein the financial text data set includes a financial text data training set and a financial text data test set;

[0008] Using the financial text data test set as input to a pre-set training converter, and outputting identity data of multiple financial expert reviewers;

[0009] Using a preset similarity function to perform model assignment based on multiple financial text test data in the financial text data test set and the identity data of each financial expert reviewer, to determine multiple machine learning models corresponding to each financial problem text data;

[0010] Performing data enhancement on the financial text data training set based on a preset evolutionary instruction method to generate an enhanced financial text data set;

[0011] Based on a preset loss function, the enhanced financial text data set is used to perform model training on multiple machine learning models corresponding to each of the financial problem text data, and multiple target language evaluation models are determined;

[0012] Input each of the financial text test data into a preset financial big model to be evaluated, and output a text generated by the financial big model to be evaluated corresponding to each of the financial text test data;

[0013] The target language evaluation models are used to evaluate the texts generated by the plurality of large financial models to be evaluated, and text evaluation results corresponding to the texts generated by the large financial models to be evaluated are generated.

[0014] Furthermore, the method of using a preset similarity function to perform model allocation according to the plurality of financial text test data in the financial text data test set and the identity data of each financial expert reviewer to determine a plurality of machine learning models corresponding to each of the financial problem text data includes:

[0015] Performing feature mapping on each of the financial text test data and each of the financial expert reviewer identity data respectively, generating a text vector representation corresponding to each of the financial text test data and an identity vector representation corresponding to each of the financial expert reviewer identity data;

[0016] Using a preset similarity function, the similarity between each financial text test data and the plurality of financial expert reviewer identity data is calculated based on the text vector representation corresponding to each financial text test data and the identity vector representation corresponding to each financial expert reviewer identity data;

[0017] Sorting the multiple similarities corresponding to each of the financial text test data in descending order, and selecting the financial expert reviewer identity data corresponding to a preset number of similarities as the target financial expert reviewer identity data;

[0018] According to the preset model allocation quantity, model allocation is performed on the multiple target financial expert reviewer identity data corresponding to each of the financial text test data, and multiple machine learning models corresponding to each of the financial problem text data are determined.

[0019] Furthermore, the financial text data training set includes a plurality of financial text training data; the method of performing data enhancement on the financial text data training set based on the preset evolutionary instruction method to generate an enhanced financial text data set includes:

[0020] Reversing and expanding and rewriting each of the financial text training data using a preset evolutionary instruction method to generate a plurality of enhanced financial text data;

[0021] An enhanced financial text dataset is constructed based on the plurality of enhanced financial text data.

[0022] Furthermore, based on the preset loss function, the enhanced financial text data set is used to perform model training on multiple machine learning models corresponding to each of the financial problem text data to determine multiple target language evaluation models, including:

[0023] Initializing multiple machine learning models corresponding to each of the financial problem text data to determine multiple initial language evaluation models;

[0024] Using each of the initial language evaluation models to output a first sample total score according to the enhanced financial text dataset;

[0025] Substituting the total score of the first sample into a preset loss function and performing derivation to determine the model gradient;

[0026] Using the model gradient to fine-tune the model parameters of each of the initial language evaluation models to determine a plurality of intermediate language evaluation models;

[0027] Using each of the intermediate language evaluation models to output a second sample total score according to the enhanced financial text dataset;

[0028] Calculate a target loss value according to the total score of the second sample using the preset loss function;

[0029] Determining whether the target loss value converges;

[0030] If converged, each of the intermediate language evaluation models is used as the corresponding target language evaluation model.

[0031] Furthermore, the similarity calculation process is specifically as follows:

[0032] ;

[0033] in, is the jth similarity corresponding to the financial text test data; is the embedding model; is the i-th financial text test data; is the identity data of the jth financial expert reviewer; l is the total number of identity data of financial expert reviewers.

[0034] Furthermore, the preset loss function is specifically:

[0035] ;

[0036] in, are the model parameters corresponding to the first language evaluation model; The model parameters corresponding to the second language evaluation model; is the model parameter corresponding to the nth language evaluation model; is the preset loss function, which represents the loss function of the real score and the model score; For real score; Give the total score for the j-th sample of enhanced financial text data; is the conditional probability prediction function of the large model; is the j-th enhanced financial text data; prompt is the scoring instruction; is the model parameter corresponding to the i-th language evaluation model; Score the samples output by the i-th language evaluation model.

[0037] A second aspect of the present invention provides an evaluation device for generating texts from a large financial model, comprising:

[0038] An acquisition module, used to acquire a financial text data set, wherein the financial text data set includes a financial text data training set and a financial text data test set;

[0039] An input module, used to use the financial text data test set as input to a preset training converter, and output identity data of multiple financial expert reviewers;

[0040] A model allocation module, configured to use a preset similarity function to perform model allocation according to the plurality of financial text test data in the financial text data test set and the identity data of each financial expert reviewer, and determine a plurality of machine learning models corresponding to each of the financial problem text data;

[0041] A data enhancement module, used for performing data enhancement on the financial text data training set based on a preset evolutionary instruction method to generate an enhanced financial text data set;

[0042] A model training module, used to perform model training on multiple machine learning models corresponding to each of the financial problem text data using the enhanced financial text data set based on a preset loss function, and determine multiple target language evaluation models;

[0043] An output module, used to input each of the financial text test data into a preset financial big model to be evaluated, and output a text generated by the financial big model to be evaluated corresponding to each of the financial text test data;

[0044] The evaluation module is used to use each of the target language evaluation models to evaluate the texts generated by the multiple financial large models to be evaluated, and generate text evaluation results corresponding to the texts generated by each of the financial large models to be evaluated.

[0045] A third aspect of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method for evaluating the generation of text from a large financial model as described in any one of the above items.

[0046] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the steps of the method for evaluating text generated by a large financial model as described in any one of the above items.

[0047] A fifth aspect of the present invention provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions, wherein when the program instructions are executed by a computer, the computer executes the steps of the method for evaluating the generation of text from a large financial model as described in any one of the above items.

[0048] It can be seen from the above technical solutions that the present invention has the following advantages:

[0049] The above technical scheme of the present invention provides an evaluation method for text generated by a large financial model. First, a financial text data set is obtained, and the financial text data set includes a financial text data training set and a financial text data test set; then, the financial text data test set is used as the input of a preset training converter to output multiple financial expert reviewer identity data; a preset similarity function is used to perform model allocation according to multiple financial text test data and the identity data of each financial expert reviewer in the financial text data test set, and multiple machine learning models corresponding to each financial problem text data are determined; based on a preset evolutionary instruction method, data enhancement is performed on the financial text data training set to generate an enhanced financial text data set; based on a preset loss function, the enhanced financial text data set is used to perform model training on multiple machine learning models corresponding to each financial problem text data, and multiple target language evaluation models are determined; finally, each financial text test data is assigned to a target language evaluation model. The data are respectively input into the preset financial big model to be evaluated, and the text generated by the financial big model to be evaluated corresponding to each financial text test data is output; each target language evaluation model is used to evaluate the texts generated by multiple financial big models to be evaluated, and the text evaluation results corresponding to the texts generated by each financial big model to be evaluated are generated; based on the above scheme, according to the preset similarity function and the preset training converter, the model is assigned to the acquired financial text data test set to obtain multiple machine learning models, and then the enhanced financial text data set obtained by data enhancement is used to train the multiple machine learning models, and multiple target language evaluation models are obtained. The generated text generated by the financial big model to be evaluated is evaluated, and the process of generating text evaluation results reduces the dependence on the generalization ability of a single evaluation big model, and can ensure that the evaluation process fully considers the evaluation results from different identities, thereby improving the accuracy of the evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0051] Figure 1 A flowchart of the steps of a method for evaluating text generated by a large financial model provided in the first embodiment of the present invention;

[0052] Figure 2 The overall framework diagram of the evaluation method for generating text from a financial large model provided in the first embodiment of the present invention;

[0053] Figure 3 This is a structural block diagram of an evaluation device for generating text from a large financial model provided in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0054] The embodiment of the present invention provides a method and device for evaluating text generated by a large financial model, which are used to solve the technical problem that the evaluation technology of the existing text generated by a large financial model leads to low evaluation accuracy.

[0055] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] See also Figure 1 , Figure 1 A flowchart of the steps of a method for evaluating text generated by a large financial model provided in Example 1 of the present invention.

[0057] The present invention provides a method for evaluating text generated by a large financial model, comprising:

[0058] Step 101: Obtain a financial text data set, where the financial text data set includes a financial text data training set and a financial text data test set.

[0059] It should be noted that the financial text dataset is a dataset that contains expert knowledge questions and answers in the financial field. Each financial text data contains questions, correct answers (positive samples) and incorrect samples (negative samples). The financial text dataset includes a financial text data training set and a financial text data test set. The financial text data training set includes multiple financial text test data. The financial text test data includes questions, correct answers (positive samples) and incorrect samples (negative samples). The financial text data test set includes multiple financial text training data. The financial text training data includes questions, correct answers (positive samples) and incorrect samples (negative samples).

[0060] Step 102: Use the financial text data test set as input to a preset training converter to output identity data of multiple financial expert reviewers.

[0061] The identity data of multiple financial expert reviewers include bank risk control experts, senior fund managers, financial technology experts, stock analysts, bond market experts, insurance actuaries, hedge fund analysts, quantitative trading experts, private equity investment experts, financial legal advisors, macroeconomists, foreign exchange traders, real estate investment advisors, credit rating experts, corporate financial advisors, commodity market analysts, financial data analysts, start-up company financing experts, wealth management advisors, financial market regulatory officials, wealth management clients, personal banking clients, etc.

[0062] It should be noted that several related identities are generated based on the relevant background in the financial field. Specifically, the GPT-4 model (Generative Pre-trained Transformer 4) is used to design prompts (instructions) for a given task, that is, a test set of financial text data, so that the GPT-4 model returns relevant role identities (financial expert reviewer identity data); the instruction format is {task description}\nGenerate {number} different financial expert reviewer identities based on this task. The generated identity information should be brief so that the generated identity information is as comprehensive as possible.

[0063] Step 103: Use a preset similarity function to perform model allocation based on multiple financial text test data in the financial text data test set and the identity data of each financial expert reviewer to determine multiple machine learning models corresponding to each financial problem text data.

[0064] Specifically, step 103 includes the following sub-steps S31-S34:

[0065] Step S31, feature mapping is performed on each financial text test data and each financial expert reviewer identity data to generate a text vector representation corresponding to each financial text test data and an identity vector representation corresponding to each financial expert reviewer identity data;

[0066] Step S32, using a preset similarity function to calculate the similarity between each financial text test data and the identity data of multiple financial expert reviewers according to the text vector representation corresponding to each financial text test data and the identity vector representation corresponding to each financial expert reviewer identity data;

[0067] Step S33, sorting the multiple similarities corresponding to each financial text test data in descending order, and selecting the financial expert reviewer identity data corresponding to a preset number of similarities as the target financial expert reviewer identity data;

[0068] Step S34: According to the preset model allocation quantity, model allocation is performed on multiple target financial expert reviewer identity data corresponding to each financial text test data, and multiple machine learning models corresponding to each financial problem text data are determined.

[0069] It should be noted that the embedding model is used to retrieve the n identities most relevant to these samples (financial text test data), and then they are assigned to the corresponding n models in turn. Specifically, based on the gte-Qwen2-7B-instruct model (embedding model), after obtaining the text vector representation corresponding to each financial text test data and the identity vector representation corresponding to each financial expert reviewer identity data, for each financial text test data x i The corresponding text vector representation is calculated in turn, and the text vector representation and the identity data of each financial expert reviewer c1, c2, ..., c l The similarity of the corresponding identity vectors s1,s2,...,s l Then, all the similarities corresponding to each financial text test data are sorted in descending order, and a preset number of financial expert reviewer identity data that are most relevant to the financial text test data are selected. For example, if the preset number is 3, the financial expert reviewer identity data corresponding to the first three similarities are selected as the target financial expert reviewer identity data of the financial text test data, which are used for subsequent evaluation; wherein the total number of similarities corresponding to the financial text test data is equal to the total number of financial expert reviewer identity data.

[0070] Furthermore, the similarity calculation process is specifically as follows:

[0071] ;

[0072] in, is the jth similarity corresponding to the financial text test data; is the embedding model; is the i-th financial text test data; is the identity data of the jth financial expert reviewer; l is the total number of identity data of financial expert reviewers.

[0073] Furthermore, for each financial text test data corresponding to multiple target financial expert reviewer identity data, a preset model allocation number of machine learning models will be assigned. For example, if the preset model allocation number is 3, then 3 machine learning models will be assigned to each target financial expert reviewer identity data corresponding to the financial text test data, and prompts will be written separately to allow these 3 models to evaluate the data in terms of fluency, text accuracy and text conciseness.

[0074] Step 104: perform data enhancement on the financial text data training set based on the preset evolutionary instruction method to generate an enhanced financial text data set.

[0075] The financial text data training set includes multiple financial text training data.

[0076] Specifically, step 104 may include the following sub-steps S41-S42:

[0077] Step S41, using a preset evolutionary instruction method to reverse and expand and rewrite each financial text training data to generate a plurality of enhanced financial text data;

[0078] Step S42: construct an enhanced financial text data set according to the plurality of enhanced financial text data.

[0079] It should be noted that for the professional financial data set (financial text data training set), the evol-instruct method (preset evolutionary instruction method) is used to rewrite the financial text data training set to obtain an enhanced data set (enhanced financial text data set). The evol-instruct method rewrites the data from the following two perspectives: (1) In-depth Evolving. By introducing restrictions, deepening content details, refining the analysis process, and adding complex input methods, the generated data is made more complex and difficult than the original text. (2) In-breadth Evolving. By replacing the instructions with words, changing the word order, etc., a more diverse but semantically consistent text is generated.

[0080] Furthermore, the Evol-instruct method can be used to generate positive sample data from methods such as complexity and diversity. On this basis, the negative samples and positive samples in the financial text training data are reversed to obtain data with opposite semantics, and then the negative samples and positive samples in the financial text training data are expanded and rewritten to obtain new positive samples and new negative samples. Based on the above processing principle, after processing each financial text training data, multiple enhanced financial text data can be obtained. The enhanced financial text data can be expressed as x1, x2,..., x l .

[0081] Step 105: Based on a preset loss function, the enhanced financial text data set is used to perform model training on multiple machine learning models corresponding to each financial problem text data, and multiple target language evaluation models are determined.

[0082] Specifically, step 105 may include the following sub-steps S51-S58:

[0083] Step S51, initializing multiple machine learning models corresponding to each financial question text data, and determining multiple initial language evaluation models;

[0084] Step S52: using each initial language evaluation model to output a first sample total score according to the enhanced financial text dataset;

[0085] Step S53: Substitute the total score of the first sample into the preset loss function and perform derivation to determine the model gradient;

[0086] Step S54: fine-tune the model parameters of each initial language evaluation model using model gradient to determine multiple intermediate language evaluation models;

[0087] Step S55: using each intermediate language evaluation model to output a second sample total score according to the enhanced financial text data set;

[0088] Step S56, using a preset loss function to calculate a target loss value according to the total score of the second sample;

[0089] Step S57, judging whether the target loss value converges;

[0090] Step S58: If converged, each intermediate language evaluation model is used as the corresponding target language evaluation model.

[0091] It should be noted that the n evaluation models (machine learning models) obtained, that is, the sum of multiple machine learning models corresponding to all financial problem text data, are denoted as m1, m2, ..., m n , initialize the scoring weights (model parameters) of each model in turn, set to w1,w2,...,w n =1 / n, and then according to the obtained enhanced financial text data x1,x2,...,x l , the true score (true score) of each enhanced financial text data is , where the positive samples in the enhanced financial text data are given full marks, and their true scores are , the true score of negative samples in the augmented financial text data is .

[0092] Furthermore, all the initial language evaluation models are used to evaluate a certain enhanced financial text data x in the enhanced financial text data set. j Scoring is performed to obtain multiple sample scores p1, p2, ..., p n , multiply the sample score by the score weights w1,w2,...,w corresponding to the initial language evaluation model n , and sum to get the multi-evaluation model for the enhanced financial text data x j Total score (total sample score) .

[0093] Further, according to the total score of the first sample, the model gradient is determined; the model gradient is used to fine-tune the model parameters of all the initial language evaluation models to determine multiple intermediate language evaluation models; each intermediate language evaluation model is used to output the total score of the second sample according to the new enhanced financial text data (i.e., another enhanced financial text data of the enhanced financial text data set); a preset loss function is used to calculate the target loss value according to the total score of the second sample; it is determined whether the target loss value converges; if it converges, each intermediate language evaluation model is used as the corresponding target language evaluation model; if it does not converge, each intermediate language evaluation model is used as the corresponding new initial language evaluation model, and a new model gradient is determined according to the total score of the second sample, and the execution of step S54 is jumped until the target loss value converges, and each intermediate language evaluation model determined when the target loss value converges is used as the corresponding target language evaluation model.

[0094] Furthermore, the preset loss function is:

[0095] ;

[0096] in, are the model parameters corresponding to the first language evaluation model; The model parameters corresponding to the second language evaluation model; is the model parameter corresponding to the nth language evaluation model; is the preset loss function, which represents the loss function of the real score and the model score; For real score; Give the total score for the j-th sample of enhanced financial text data; is the conditional probability prediction function of the large model; is the j-th enhanced financial text data; prompt is the scoring instruction; is the model parameter corresponding to the i-th language evaluation model; Score the samples output by the i-th language evaluation model.

[0097] It is worth mentioning that each enhanced financial text data in the enhanced financial text dataset is continuously scored and evaluated, and the weight w i Dynamic fine-tuning is performed to make the weighted score close to the true score of the sample .

[0098] Step 106: Input each financial text test data into the preset financial big model to be evaluated, and output the generated text of the financial big model to be evaluated corresponding to each financial text test data.

[0099] It should be noted that the text generated by the financial big model to be evaluated is the text answer output by the preset financial big model to be evaluated based on the questions in the financial text test data. The preset financial big model to be evaluated can be any existing big model in the financial field, and the present invention is not limited to this.

[0100] Step 107: Use each target language evaluation model to evaluate the texts generated by the multiple large financial models to be evaluated, and generate text evaluation results corresponding to the texts generated by the large financial models to be evaluated.

[0101] It should be noted that after the optimization of the scoring weights of the multiple models is completed, that is, after obtaining multiple target language evaluation models, for each given to-be-evaluated data x (text generated by the large financial model to be evaluated), the comprehensive score of the multiple evaluation models is , which is the evaluation result of the data (text evaluation result), where Score the text output by the i-th target language evaluation model.

[0102] For example, see Figure 2 The overall framework of text evaluation in the present invention can be divided into five parts. First, according to the relevant background in the financial field, several related identities are generated, that is, based on the given downstream tasks, multiple identity data are generated. Then, different model identities and evaluation angles are assigned, and the data set is enhanced. The enhanced data set is used to fine-tune the scoring weight of each model in the evaluation model group to obtain a multi-evaluation model. Finally, text evaluation is performed based on the multi-evaluation model and the score is output.

[0103] In this embodiment, after obtaining multiple trained target language evaluation models based on the above steps, multiple target language evaluation models are used to evaluate the text answers output by the existing large financial model, which can enhance the evaluation ability of the existing large language model in NLG tasks in the financial field and reduce the deviations that may occur in the evaluation process.

[0104] As a comparison of technical effects, we can refer to existing technologies. Some evaluation schemes include: (1) matching degree, such as BLEU, etc., calculating the text similarity between the generated content of the large model and the reference answer; (2) constructing special evaluation sets, such as BigBench, LongBench and MMLU; (3) manual evaluation, inviting professionals to evaluate the generated content of the large model, which has the highest accuracy; (4) large model evaluation, which utilizes the generalization ability of the large model and allows the large model to follow the written prompt or context learning method to score the text content.

[0105] The shortcomings of the above-mentioned evaluation schemes are: (1) matching degree, which only focuses on the text similarity of the model generation results, but does not take into account the deeper text information; (2) building special evaluation sets, which may not fully cover the needs of vertical fields, cost a lot of resources to build, and there is a risk of leakage; (3) manual evaluation, which requires a lot of human resources; (4) large model evaluation, which is extremely dependent on the generalization ability of the evaluation model itself. If the model is weaker, it may not be able to be accurately evaluated. At the same time, the model may have bias during evaluation, resulting in objective evaluation results.

[0106] In view of the above problems, the present invention proposes an evaluation method for text generated by a large financial model. Based on a preset model, multiple financial expert identity data are generated, and a suitable model is assigned to each financial expert identity data according to similarity, and then different evaluation dimensions are assigned; data rewriting technology is used to rewrite each data item in the original data set to generate training data with similar or opposite semantics, which is used as preference data; the preference data is respectively input into the scoring model that has been assigned with a specific role identity and evaluation dimension for scoring, and the weight of each scoring model is dynamically adjusted according to the data label. After the weight adjustment, the multi-model can be used to further evaluate the quality of the generated financial text.

[0107] Furthermore, the present invention performs linear weighted processing on each evaluation result according to the weights adjusted by the models of different identities and different angles, and finally obtains the evaluation score. Each model obtains different weights according to different evaluation identities and angles, and participates in the calculation of the final score, ensuring that the evaluation process can fully consider the evaluation results from different professions and different perspectives, and improve the breadth and accuracy of the evaluation. The experimental results show that the results of multi-model scoring are higher in similarity than the direct scoring of a single model and compared with human scoring. At the same time, the present invention can significantly enhance the evaluation ability of the large language model in NLG tasks in the financial field by evaluating the multi-role and multi-dimensional large language model, reduce the deviations that may occur in the evaluation process, and improve the consistency of human preferences in the evaluation results.

[0108] In an embodiment of the present invention, the present invention provides an evaluation method for text generated by a large financial model. First, a financial text data set is obtained, and the financial text data set includes a financial text data training set and a financial text data test set; then, the financial text data test set is used as an input of a preset training converter to output multiple financial expert reviewer identity data; a preset similarity function is used to perform model allocation according to multiple financial text test data and identity data of each financial expert reviewer in the financial text data test set, and multiple machine learning models corresponding to each financial problem text data are determined; based on a preset evolutionary instruction method, data enhancement is performed on the financial text data training set to generate an enhanced financial text data set; based on a preset loss function, the enhanced financial text data set is used to perform model training on multiple machine learning models corresponding to each financial problem text data, and multiple target language evaluation models are determined; finally, each financial text test set is assigned a model. The data are respectively input into the preset financial big model to be evaluated, and the text generated by the financial big model to be evaluated corresponding to each financial text test data is output; each target language evaluation model is used to evaluate the texts generated by multiple financial big models to be evaluated, and the text evaluation results corresponding to the texts generated by each financial big model to be evaluated are generated; based on the above scheme, according to the preset similarity function and the preset training converter, the model is assigned to the acquired financial text data test set to obtain multiple machine learning models, and then the enhanced financial text data set obtained by data enhancement is combined to train the multiple machine learning models, and multiple target language evaluation models are obtained. The generated text generated by the financial big model to be evaluated is subjected to text evaluation, and the process of generating text evaluation results reduces the dependence on the generalization ability of a single evaluation big model, and can ensure that the evaluation process fully considers the evaluation results from different identities, thereby improving the accuracy of the evaluation.

[0109] See also Figure 3 , Figure 3 This is a structural block diagram of an evaluation device for generating text from a large financial model provided in Embodiment 2 of the present invention.

[0110] The present invention provides a device for evaluating text generated by a large financial model, comprising:

[0111] An acquisition module 301 is used to acquire a financial text data set, where the financial text data set includes a financial text data training set and a financial text data test set;

[0112] An input module 302, used to use the financial text data test set as input to a preset training converter, and output identity data of multiple financial expert reviewers;

[0113] A model allocation module 303 is used to use a preset similarity function to perform model allocation based on multiple financial text test data in the financial text data test set and the identity data of each financial expert reviewer, and determine multiple machine learning models corresponding to each financial problem text data;

[0114] A data enhancement module 304 is used to perform data enhancement on the financial text data training set based on a preset evolutionary instruction method to generate an enhanced financial text data set;

[0115] A model training module 305 is used to perform model training on multiple machine learning models corresponding to each financial question text data using an enhanced financial text data set based on a preset loss function, and determine multiple target language evaluation models;

[0116] Output module 306, used to input each financial text test data into the preset financial big model to be evaluated, and output the generated text of the financial big model to be evaluated corresponding to each financial text test data;

[0117] The evaluation module 307 is used to use each target language evaluation model to evaluate the texts generated by the multiple financial large models to be evaluated, and generate text evaluation results corresponding to the texts generated by each financial large model to be evaluated.

[0118] Furthermore, the model allocation module 303 is specifically used for:

[0119] Perform feature mapping on each financial text test data and each financial expert reviewer identity data respectively, and generate a text vector representation corresponding to each financial text test data and an identity vector representation corresponding to each financial expert reviewer identity data;

[0120] Using a preset similarity function, the similarity between each financial text test data and the identity data of multiple financial expert reviewers is calculated according to the text vector representation corresponding to each financial text test data and the identity vector representation corresponding to each financial expert reviewer identity data;

[0121] Sort the multiple similarities corresponding to each financial text test data in descending order, and select the financial expert reviewer identity data corresponding to a preset number of similarities as the target financial expert reviewer identity data;

[0122] According to the preset model allocation quantity, model allocation is performed on multiple target financial expert reviewer identity data corresponding to each financial text test data, and multiple machine learning models corresponding to each financial problem text data are determined.

[0123] Furthermore, the financial text data training set includes a plurality of financial text training data; the data enhancement module 304 is specifically used for:

[0124] The preset evolutionary instruction method is used to reverse, expand and rewrite each financial text training data to generate multiple enhanced financial text data;

[0125] An enhanced financial text dataset is constructed based on multiple enhanced financial text data.

[0126] Furthermore, the model training module 305 is specifically used for:

[0127] Initialize multiple machine learning models corresponding to the text data of each financial question and determine multiple initial language evaluation models;

[0128] Use each initial language evaluation model to output the total score of the first sample based on the enhanced financial text dataset;

[0129] Substitute the total score of the first sample into the preset loss function and perform derivation to determine the model gradient;

[0130] The model parameters of each initial language evaluation model are fine-tuned using model gradients to determine multiple intermediate language evaluation models;

[0131] Using each intermediate language evaluation model to enhance the financial text data set, output the second sample total score;

[0132] Use the preset loss function to calculate the target loss value according to the total score of the second sample;

[0133] Determine whether the target loss value converges;

[0134] If convergence occurs, each intermediate language evaluation model is used as the corresponding target language evaluation model.

[0135] Furthermore, the similarity calculation process is specifically as follows:

[0136] ;

[0137] in, is the jth similarity corresponding to the financial text test data; is the embedding model; is the i-th financial text test data; is the identity data of the jth financial expert reviewer; l is the total number of identity data of financial expert reviewers.

[0138] Furthermore, the preset loss function is:

[0139] ;

[0140] in, are the model parameters corresponding to the first language evaluation model; The model parameters corresponding to the second language evaluation model; is the model parameter corresponding to the nth language evaluation model; is the preset loss function, which represents the loss function of the real score and the model score; For real score; Give the total score for the j-th sample of enhanced financial text data; is the conditional probability prediction function of the large model; is the j-th enhanced financial text data; prompt is the scoring instruction; is the model parameter corresponding to the i-th language evaluation model; Score the samples output by the i-th language evaluation model.

[0141] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0142] An embodiment of the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory; when the computer program is executed by the processor, the processor executes the steps of the method for evaluating the generation of text by a large financial model as described in any of the above embodiments.

[0143] An embodiment of the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the method for evaluating the text generated by a large financial model as in any of the above embodiments are implemented.

[0144] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the method for evaluating text generated by a large financial model as in any of the above embodiments.

[0145] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0146] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0147] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for evaluating text generated by a large financial model, characterized in that: include: Acquire a financial text data set, wherein the financial text data set includes a financial text data training set and a financial text data test set; Using the financial text data test set as input to a pre-set training converter, and outputting identity data of multiple financial expert reviewers; Using a preset similarity function to perform model assignment based on multiple financial text test data in the financial text data test set and the identity data of each financial expert reviewer, to determine multiple machine learning models corresponding to each financial problem text data; Performing data enhancement on the financial text data training set based on a preset evolutionary instruction method to generate an enhanced financial text data set; Based on a preset loss function, the enhanced financial text data set is used to perform model training on multiple machine learning models corresponding to each of the financial problem text data, and multiple target language evaluation models are determined; Input each of the financial text test data into a preset financial big model to be evaluated, and output a text generated by the financial big model to be evaluated corresponding to each of the financial text test data; The target language evaluation models are used to evaluate the texts generated by the plurality of large financial models to be evaluated, and text evaluation results corresponding to the texts generated by the large financial models to be evaluated are generated.

2. The evaluation method for text generated by a financial large model according to claim 1 is characterized in that: The method of using a preset similarity function to perform model allocation according to the plurality of financial text test data in the financial text data test set and the identity data of each financial expert reviewer to determine a plurality of machine learning models corresponding to each of the financial problem text data includes: Performing feature mapping on each of the financial text test data and each of the financial expert reviewer identity data respectively, generating a text vector representation corresponding to each of the financial text test data and an identity vector representation corresponding to each of the financial expert reviewer identity data; Using a preset similarity function, the similarity between each financial text test data and the plurality of financial expert reviewer identity data is calculated based on the text vector representation corresponding to each financial text test data and the identity vector representation corresponding to each financial expert reviewer identity data; Sorting the multiple similarities corresponding to each of the financial text test data in descending order, and selecting the financial expert reviewer identity data corresponding to a preset number of similarities as the target financial expert reviewer identity data; According to the preset model allocation quantity, model allocation is performed on the multiple target financial expert reviewer identity data corresponding to each of the financial text test data, and multiple machine learning models corresponding to each of the financial problem text data are determined.

3. The evaluation method for text generated by a financial large model according to claim 1 is characterized in that: The financial text data training set includes a plurality of financial text training data; the method of performing data enhancement on the financial text data training set based on the preset evolutionary instruction method to generate an enhanced financial text data set includes: Reversing and expanding and rewriting each of the financial text training data using a preset evolutionary instruction method to generate a plurality of enhanced financial text data; An enhanced financial text dataset is constructed based on the plurality of enhanced financial text data.

4. The method for evaluating text generated by a financial large model according to claim 1, characterized in that: The method of performing model training on multiple machine learning models corresponding to each of the financial problem text data using the enhanced financial text data set based on the preset loss function to determine multiple target language evaluation models includes: Initializing multiple machine learning models corresponding to each of the financial problem text data to determine multiple initial language evaluation models; Using each of the initial language evaluation models to output a first sample total score according to the enhanced financial text dataset; Substituting the total score of the first sample into a preset loss function and performing derivation to determine the model gradient; Using the model gradient to fine-tune the model parameters of each of the initial language evaluation models to determine a plurality of intermediate language evaluation models; Using each of the intermediate language evaluation models to output a second sample total score according to the enhanced financial text dataset; Calculate a target loss value according to the total score of the second sample using the preset loss function; Determining whether the target loss value converges; If converged, each of the intermediate language evaluation models is used as the corresponding target language evaluation model.

5. The method for evaluating text generated by a financial large model according to claim 2, characterized in that: The similarity calculation process is specifically as follows: ； in, is the j-th similarity corresponding to the financial text test data; is the embedding model; is the i-th financial text test data; is the identity data of the jth financial expert reviewer; l is the total number of identity data of financial expert reviewers.

6. The method for evaluating text generated by a financial large model according to claim 1, characterized in that: The preset loss function is specifically: ； in, are the model parameters corresponding to the first language evaluation model; Model parameters corresponding to the second language evaluation model; is the model parameter corresponding to the nth language evaluation model; is the preset loss function, which represents the loss function of the real score and the model score; For real score; Give the total score for the j-th sample of enhanced financial text data; is the conditional probability prediction function of the large model; is the j-th enhanced financial text data; prompt is the scoring instruction; is the model parameter corresponding to the i-th language evaluation model; Score the samples output by the i-th language evaluation model.

7. An evaluation device for generating texts from a large financial model, characterized in that: include: An acquisition module, used to acquire a financial text data set, wherein the financial text data set includes a financial text data training set and a financial text data test set; An input module, used to use the financial text data test set as input to a preset training converter, and output identity data of multiple financial expert reviewers; A model allocation module, configured to use a preset similarity function to perform model allocation according to the plurality of financial text test data in the financial text data test set and the identity data of each financial expert reviewer, and determine a plurality of machine learning models corresponding to each of the financial problem text data; A data enhancement module, used for performing data enhancement on the financial text data training set based on a preset evolutionary instruction method to generate an enhanced financial text data set; A model training module, used to perform model training on multiple machine learning models corresponding to each of the financial problem text data using the enhanced financial text data set based on a preset loss function, and determine multiple target language evaluation models; An output module, used to input each of the financial text test data into a preset financial big model to be evaluated, and output a text generated by the financial big model to be evaluated corresponding to each of the financial text test data; The evaluation module is used to use each of the target language evaluation models to evaluate the texts generated by the multiple financial large models to be evaluated, and generate text evaluation results corresponding to the texts generated by each of the financial large models to be evaluated.

8. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method for evaluating the text generated by a large financial model as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the method for evaluating the text generated by the financial large model as described in any one of claims 1-6 is implemented.

10. A computer program product, characterized in that The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions, wherein when the program instructions are executed by a computer, the computer is caused to execute the evaluation method for generating texts from a large financial model as described in any one of claims 1 to 6.

Images