Financial large model capability evaluation method and device, storage medium and electronic equipment

Abstract

The invention discloses a method and a device for evaluating the capability of a financial large model, a storage medium and electronic equipment, relates to the technical field of model capability evaluation, and mainly aims to solve the problems of incomplete evaluation standards and data sample pollution in the performance evaluation of the financial large model. The method comprises the following steps: selecting basic knowledge in the financial field and related academic texts in the industry as a basic data source to generate a basic corpus; performing decontamination processing on the basic corpus, and constructing an evaluation data set based on the decontaminated basic corpus; creating a plurality of evaluation tasks, and configuring a corresponding prompt instruction for each evaluation task; obtaining a to-be-evaluated financial large model, and evaluating the to-be-evaluated financial large model by adopting the evaluation data set, so that the to-be-evaluated financial large model sequentially executes each evaluation task corresponding to the prompt instruction to obtain an execution result; and calculating an evaluation index based on the execution result corresponding to each evaluation task, and determining the comprehensive capability of the financial large model to be evaluated based on the evaluation index.

Description

Evaluation methods, devices, storage media, and electronic equipment for financial large-scale models Technical Field

[0001] This invention relates to the field of model capability assessment technology, and in particular to a method, apparatus, storage medium, and electronic device for assessing the capability of large financial models. Background Technology

[0002] In the financial sector, general-purpose large-scale models are often ill-suited to the highly specialized and risky demands of finance. Evaluating large-scale financial models can clarify their professional capabilities and limitations, helping financial institutions make accurate model selections. It can also identify risks such as model illusions and data biases, ensuring the safety and compliance of transactions and decision-making. Furthermore, it can provide a basis for regulatory authorities to develop governance frameworks, guide institutions in optimizing models, and promote a healthy competitive ecosystem within the industry, facilitating the reliable implementation of financial AI and the industry's digital transformation.

[0003] Currently, evaluation methods for large-scale financial models focus more on their contextual understanding and text generation capabilities, leaving insufficient room for assessing their professional competence in the financial field. Furthermore, because these models are trained using large datasets scraped from the internet, these training datasets may contain samples highly similar to those used in benchmark tests, leading to data contamination issues in existing evaluation benchmarks and methods. Therefore, there is an urgent need to design a comprehensive evaluation benchmark to systematically assess the performance of large-scale models in complex financial scenarios across various tasks. Summary of the Invention

[0004] In view of this, the present invention provides a method and apparatus, storage medium and electronic device for evaluating the capabilities of large financial models. The main purpose is to solve the problems of incomplete evaluation standards and contaminated data samples in the existing performance evaluation benchmarks for large financial models.

[0005] According to one aspect of the present invention, a method for evaluating the capabilities of a large financial model is provided, comprising: selecting basic financial knowledge and relevant academic texts within the industry as basic data sources to obtain a basic corpus based on two application scenarios: financial domain knowledge and financial domain academic texts; constructing a three-level defense mechanism based on a large model decontamination framework using deep semantic screening to decontaminate the basic corpus; constructing an evaluation dataset based on the decontaminated basic corpus; creating multiple evaluation tasks for the financial domain knowledge and the financial domain academic texts respectively; configuring corresponding prompts for each evaluation task; obtaining a large financial model to be evaluated, and using the evaluation dataset to evaluate the large financial model to be evaluated, so that the large financial model to be evaluated sequentially executes each evaluation task corresponding to the prompts, obtaining execution results corresponding to each evaluation task; calculating corresponding evaluation indicators based on the execution results corresponding to each evaluation task, and determining the comprehensive capabilities of the large financial model to be evaluated based on the evaluation indicators.

[0006] Furthermore, the large-scale model decontamination framework based on deep semantic screening constructs a three-level defense mechanism to decontaminate the basic corpus, including: using a combination of vector space retrieval and similarity calculation to perform coarse-grained screening on the selected basic corpus to identify potentially contaminated samples; using a pre-trained discriminant model to perform deep verification processing on the potentially contaminated samples to identify real contaminated samples; using a dynamic reconstruction engine to perform syntactic deconstruction on the real contaminated samples and generate semantically equivalent but syntactically different alternative samples; and using the alternative samples to update the corresponding basic corpus.

[0007] Furthermore, the construction of the evaluation dataset based on the decontaminated basic corpus includes: performing data structuring processing on the decontaminated basic corpus to transform the basic corpus into a form suitable for each evaluation task; based on the construction logic of domain knowledge graphs, dividing the processed structured data into four major subject modules according to knowledge attributes, including a finance module, an economics module, an accounting module, and an industry qualification certification module; adjusting the data ratio of each subject module so that the knowledge density ratio of the finance module, the economics module, the accounting module, and the industry qualification certification module reaches a preset ratio, thereby obtaining the evaluation dataset that has both professional depth and cross-disciplinary breadth.

[0008] Furthermore, the step of creating multiple evaluation tasks for both financial domain knowledge and academic texts in the financial domain, and configuring corresponding prompts for each evaluation task, includes: creating multiple question-answering tasks and terminology explanation tasks for the generation scenario of financial domain knowledge; and configuring corresponding prompts for each of the multiple question-answering tasks and the terminology explanation tasks; creating paper text classification tasks, paper abstract generation tasks, and academic text translation tasks for the model generation scenario of academic texts in the financial domain; and configuring corresponding prompts for each of the paper text classification tasks, the paper abstract generation tasks, and the academic text translation tasks.

[0009] Furthermore, after configuring corresponding prompts for each of the evaluation tasks, the method further includes: constructing multiple contextual example templates for each of the multiple question-answering tasks, the terminology explanation task, the text generation task, the text classification task, and the text translation task; conducting graded prompt tests on the financial big model by increasing the number of contextual example templates in a stepwise manner, and recording the generation results of the financial big model at each gradient; using perplexity and ROUGE-L value as dual-dimensional evaluation indicators to quantify the semantic coherence and task suitability of the generation results; and determining the number of target contextual example templates corresponding to each evaluation task based on the quantification results.

[0010] Furthermore, the calculation of corresponding evaluation metrics based on the execution results of each of the evaluation tasks includes: calculating the accuracy rate for the execution results of multiple question-answering tasks; calculating the MAUVE and ROUGE-L metrics for the execution results of terminology explanation tasks; calculating the F1 score for the execution results of paper text classification tasks; calculating the ROUGE-N metric for the execution results of paper abstract generation tasks; and calculating the BLEU and chrF metrics for the execution results of academic text translation tasks.

[0011] Furthermore, determining the comprehensive capability of the financial big data model to be evaluated based on the evaluation indicators includes: normalizing each of the evaluation indicators to obtain normalized evaluation indicator values; and performing weighted summation on the normalized evaluation indicator values ​​to obtain the comprehensive score of the financial big data model to be evaluated.

[0012] According to another aspect of the present invention, a device for evaluating the capabilities of a large financial model is provided, comprising: a data acquisition module, used to select basic financial knowledge and relevant academic texts in the industry as basic data sources to acquire basic corpus based on two application scenarios: financial knowledge and financial academic texts, and generate a basic corpus; a desaturation and construction module, used to construct a three-level defense mechanism based on a large model desaturation framework using deep semantic screening to desaturate the basic corpus; and to construct an evaluation dataset based on the desaturated basic corpus; a task creation module, used to create multiple evaluation tasks for the financial knowledge and the financial academic texts respectively; and to configure corresponding prompts for each evaluation task; an evaluation processing module, used to acquire the large financial model to be evaluated, and to use the evaluation dataset to evaluate the large financial model to be evaluated, so that the large financial model to be evaluated sequentially executes each evaluation task corresponding to the prompts, and obtains execution results corresponding to each evaluation task; and an evaluation analysis module, used to calculate corresponding evaluation indicators based on the execution results corresponding to each evaluation task, and to determine the comprehensive capability of the large financial model to be evaluated based on the evaluation indicators.

[0013] Furthermore, the decontamination and construction module includes a decontamination unit, which is used to: perform coarse-grained screening on the selected basic corpus using a combination of vector space retrieval and similarity calculation to identify potential contaminated samples; perform deep verification processing on the potential contaminated samples using a pre-trained discriminant model to identify real contaminated samples; perform syntactic deconstruction on the real contaminated samples using a dynamic reconstruction engine and generate semantically equivalent but syntactically different alternative samples; and update the corresponding basic corpus using the alternative samples.

[0014] Furthermore, the decontamination and construction module includes a construction unit, which is used to: perform data structuring processing on the decontamination-processed basic corpus, transforming the basic corpus into a form suitable for each of the evaluation tasks; based on the construction logic of the domain knowledge graph, divide the processed structured data into four major subject modules according to knowledge attributes, including a finance module, an economics module, an accounting module, and an industry qualification certification module; adjust the data ratio of each subject module so that the knowledge density ratio of the finance module, the economics module, the accounting module, and the industry qualification certification module reaches a preset ratio, thereby obtaining the evaluation dataset that has both professional depth and cross-disciplinary breadth.

[0015] Furthermore, the task creation module is also used to: create multiple question-and-answer tasks and terminology explanation tasks for the generation scenario of financial knowledge; and configure corresponding prompts for the multiple question-and-answer tasks and the terminology explanation tasks respectively; and create paper text classification tasks, paper abstract generation tasks and academic text translation tasks for the model generation scenario of academic text in the financial field; and configure corresponding prompts for the paper text classification tasks, the paper abstract generation tasks and the academic text translation tasks respectively.

[0016] Furthermore, the device also includes a tiered testing module, which is used to: construct multiple contextual example templates for the multiple question-answering tasks, the terminology explanation task, the text generation task, the text classification task, and the text translation task respectively; perform graded prompting tests on the financial big model by increasing the number of contextual example templates in a tiered manner, and record the generation results of the financial big model at each gradient; use perplexity and ROUGE-L value as dual-dimensional evaluation indicators to quantify the semantic coherence and task suitability of the generation results; and determine the number of target contextual example templates corresponding to each evaluation task based on the quantification results.

[0017] Furthermore, the evaluation and analysis module is also used to: calculate the accuracy rate for the execution results corresponding to multiple question-answering tasks; calculate the MAUVE index and ROUGE-L index for the execution results corresponding to the terminology explanation task; calculate the F1 score for the execution results corresponding to the paper text classification task; calculate the ROUGE-N index for the execution results corresponding to the paper abstract generation task; and calculate the BLEU index and chrF index for the execution results corresponding to the academic text translation task.

[0018] Furthermore, the evaluation and analysis module is also used to: normalize each of the evaluation indicators to obtain normalized evaluation indicator values; and perform weighted summation on the normalized evaluation indicator values ​​to obtain the comprehensive score of the financial large-scale model to be evaluated. According to another aspect of the present invention, a storage medium is provided, wherein at least one executable instruction is stored in the storage medium, the executable instruction causing a processor to perform operations corresponding to the evaluation method for the capabilities of the aforementioned financial large-scale model.

[0019] According to another aspect of the present invention, an electronic device is provided, including a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, which causes the processor to perform an operation corresponding to the above-described evaluation method for the capabilities of large financial models.

[0020] By employing the above technical solutions, the technical solutions provided by the embodiments of the present invention have at least the following advantages: The present invention provides a method and apparatus, storage medium, and electronic device for evaluating the capabilities of large financial models. Compared with the prior art, the present invention selects basic financial knowledge and relevant academic texts within the industry as basic data sources to obtain basic corpus based on two major application scenarios: financial domain knowledge and financial domain academic texts, thereby generating a basic corpus and ensuring the comprehensiveness and professionalism of the sample data. Furthermore, a three-level defense mechanism is constructed based on a large model decontamination framework using deep semantic screening to decontaminate the basic corpus; and an evaluation dataset is constructed based on the decontaminated basic corpus, eliminating data pollution in the dataset and ensuring the accuracy of the evaluation results. The present invention also creates multiple evaluation tasks for the financial domain knowledge and the financial domain academic texts respectively; and configures corresponding prompts for each evaluation task, simulating different industry scenarios through specific roles to enhance the modeling capabilities of domain knowledge. The present invention also calculates corresponding evaluation indicators based on the execution results of each evaluation task, and determines the comprehensive capabilities of the large financial model to be evaluated based on the evaluation indicators, realizing a comprehensive evaluation of the generation quality and question-answering accuracy of the large model from different perspectives.

[0021] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0022] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 shows a flowchart illustrating a method for evaluating the capabilities of a large financial model according to an embodiment of the present invention; Figure 2 shows a technical framework diagram illustrating a method for evaluating the capabilities of a large financial model according to an embodiment of the present invention; Figure 3 shows a flowchart illustrating the construction of an evaluation dataset according to an embodiment of the present invention; Figure 4 shows a flowchart illustrating a step-by-step testing method according to an embodiment of the present invention; Figure 5 shows a structural diagram illustrating a device for evaluating the capabilities of a large financial model according to an embodiment of the present invention; Figure 6 shows a structural diagram illustrating an electronic device according to an embodiment of the present invention. Detailed Implementation

[0023] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0024] This invention provides a method for evaluating the capabilities of a large financial model, as shown in Figure 1. The method includes: 101. Based on two application scenarios—financial knowledge and academic texts in the financial field—basic financial knowledge and relevant academic texts within the industry are selected as basic data sources to obtain basic corpus, generating a basic corpus. In this embodiment, for the financial knowledge scenario, the current execution end selects materials related to basic financial knowledge, covering core content from disciplines such as finance, economics, and accounting, as well as authoritative textbooks and assessment outlines for various financial industry certifications (such as professional qualification examinations), as basic data sources to obtain basic corpus, ensuring the practical adaptability of the data. For the academic text scenario, financial research literature is selected as core materials, including journal articles, dissertations, and other academic achievements, as basic data sources to obtain basic corpus, ensuring the academic rigor of the data.

[0025] By combining data sources from two different scenarios, we can achieve comprehensive coverage of professional knowledge and academic research content in the financial field, laying the foundation for the professionalism of the dataset.

[0026] 102. A three-tiered defense mechanism based on deep semantic screening is constructed to decontaminate the basic corpus. An evaluation dataset is then built based on the decontaminated corpus. Since the large model uses a large amount of data scraped from the internet for training, the training dataset may contain samples highly similar to the questions in the benchmark tests. This contamination may lead to incorrect assessments of the model's reasoning ability—they may simply have guessed the correct answer during training. For example, evaluating BaiChuan-13B and ChatGPT4 using a contaminated dataset yields similar scores, indicating that in the presence of data contamination, the evaluation results cannot accurately reflect the large model's capabilities in the financial domain.

[0027] In this embodiment of the invention, the current execution end addresses the cross-source data contamination problem of existing evaluation benchmarks by constructing a three-level defense mechanism based on the large model decontamination framework (LLM-Decontaminator v2.1) using deep semantic screening to decontaminate the basic corpus. This three-level defense mechanism represents a three-tiered processing flow comprising "coarse-grained screening," "precise verification," and "sample reconstruction," achieving efficient dataset decontamination and effectively preventing the phenomenon of large models "guessing the right answer" during training.

[0028] 103. Multiple evaluation tasks are created for the financial domain knowledge and the financial domain academic text, respectively; and corresponding prompts are configured for each evaluation task. In this embodiment of the invention, the current execution end creates multiple evaluation tasks for the financial domain knowledge and the financial domain academic text, as shown in Figure 2. For the generation scenario of the financial domain knowledge, multiple question-answering tasks and terminology explanation tasks are created; and corresponding prompts are configured for each of the multiple question-answering tasks and the terminology explanation tasks. As shown in Figure 2, for the model generation scenario of the financial domain academic text, paper text classification tasks, paper abstract generation tasks, and academic text translation tasks are created; and corresponding prompts are configured for each of the paper text classification tasks, the paper abstract generation tasks, and the academic text translation tasks. Each task performs specific processing and analysis on the data in the financial domain, which can systematically and comprehensively evaluate the capabilities of the large model in different financial scenarios and demonstrate its performance in managing complex financial decision-making scenarios.

[0029] The system includes multiple question-and-answer tasks to assess the financial model's grasp of relevant knowledge and its ability to answer questions in points. These tasks include multiple-choice and open-ended questions, covering topics such as finance, economics, accounting, and professional qualification examinations. Multiple-choice questions primarily test basic financial knowledge, while open-ended questions focus on essay questions related to finance, including explaining professional knowledge and listing financial examples. This embodiment of the invention does not impose specific limitations.

[0030] The terminology explanation task is used to evaluate the ability of the large financial model to understand, explain, and generate professional terms in the field. This task requires the model not only to accurately describe the definition of a term, but also to provide multi-dimensional explanations in conjunction with context, application scenarios, and related knowledge. The data for terminology explanation is selected from domestic research results on financial terminology, and financial-related terminology texts and their explanations are selected to comprehensively evaluate the performance of the Chinese large model in the financial terminology explanation task. Its essence is to certify the credibility of the large financial model in the professional field.

[0031] The text classification task in this example includes sentence step recognition and keyword classification. A sentence step is a basic unit in a discourse that has relatively independent function and semantics. It usually consists of one or more sentences and revolves around a specific theme or function, such as posing a question, providing information, stating a viewpoint, or making an argument. In this example, sentence step recognition automatically identifies sentences with semantic types such as background, purpose, method, and conclusion from financial academic text summaries. Combined with keyword classification, this can test the text recognition and classification capabilities of a large-scale financial model.

[0032] The paper abstracting task aims to leverage powerful language models to extract key information from the full text of academic papers and generate concise and accurate summaries. Technically, large models are typically pre-trained on massive amounts of text data to learn rich linguistic knowledge and semantic representations. These models are then fine-tuned or adapted to specific abstracting algorithms to suit the paper abstracting task.

[0033] In the academic text translation task, given the strong bilingual capabilities of current large-scale Chinese models, this embodiment of the invention uses a Chinese-English bidirectional translation task to conduct a more comprehensive evaluation of the model.

[0034] In this embodiment, the current execution end also designs dedicated prompt instructions for different evaluation tasks. The construction of prompt words not only takes into account the requirements of different task scenarios, but also ensures the accuracy of the generated results from large models. Based on the special requirements of financial text processing, the construction of prompt words is adapted to different industry scenarios through specific role simulation, thereby strengthening the modeling capability of domain knowledge. In the verification phase, we select a small-scale sample dataset for iterative testing of multi-model architectures. By optimizing the dialogue strategy, we ensure that mainstream large models on the market can achieve the expected generation quality. For details of the specific instruction configuration scheme, please refer to the architecture comparison table in Table 1 below.

[0035] Table 1. Set of prompts for each evaluation task

[0036] 104. Obtain the large financial model to be evaluated, and use the evaluation dataset to evaluate the large financial model to be evaluated, so that the large financial model to be evaluated sequentially executes each of the evaluation tasks corresponding to the prompt instructions, and obtains the execution results corresponding to each evaluation task. In this embodiment of the invention, as shown in Figure 2, the large financial model to be evaluated is accessed and deployed in the current execution terminal, and then the evaluation dataset obtained in step 102 is used to evaluate the accessed large financial model to be evaluated. During the evaluation process, the large financial model to be evaluated needs to sequentially execute each of the evaluation tasks corresponding to the prompt instructions set in step 103 above, including multiple question-answering tasks, terminology explanation tasks, paper text classification tasks, paper abstract generation tasks, and academic text translation tasks, etc., and obtain the execution results corresponding to each evaluation task.

[0037] 105. Calculate the corresponding evaluation indicators based on the execution results of each evaluation task, and determine the comprehensive capability of the financial big data model to be evaluated based on the evaluation indicators.

[0038] In this embodiment of the invention, since each evaluation task cannot intuitively reflect the comprehensive capabilities of the financial big data model to be evaluated, the current execution end also needs to calculate the corresponding evaluation indicators based on the execution results of each evaluation task, as follows: (1) Calculate the accuracy rate for the execution results corresponding to the multiple question-and-answer tasks; Accuracy is usually used to measure the proportion of the number of samples correctly predicted by the classification model to the total number of samples. For example, in a classification task with 100 samples, if the model correctly predicts 80, then the accuracy rate is 80%. In this embodiment of the invention, the current execution end scores the multiple question-and-answer tasks of the financial big data model to be evaluated by statistically analyzing the percentage of accuracy of the financial big data model in the multiple-choice test.

[0039] (2) Calculate the MAUVE and ROUGE-L indices for the execution results corresponding to the terminology explanation task. The MAUVE (Measuring the Gap Between Neural Text and Human Text using DivergenceFrontiers) index is a text generation effect evaluation method based on KL divergence, aiming to solve the problem of the difference between existing indices and human evaluation. In this embodiment of the invention, this index is used to calculate the scores of the financial large model in open question answering and terminology explanation tasks. The ROUGE-L (Longest Common Subsequence) index essentially measures the degree of overlap of the longest common subsequence (LCS), so it is particularly suitable for evaluating whether the output is close to the reference answer in terms of overall content structure.

[0040] In terminology explanation tasks, a term may have a definition, features, examples, etc. Different models may have flexible wording in their outputs, but the core content (key points, order of key information) is often similar. ROUGE-L is based on LCS, so it can effectively evaluate whether the explanations generated by large models cover the same key points as the reference explanation, whether the order of key points is reasonable, and whether there are any omissions of key content.

[0041] It should be noted that, in order to apply LCS to the term explanation task, we treat the term explanation sentences as word sequences. Generally speaking, the longer the LCS of two sentences, the more similar we consider the two sentences to be. This invention uses an LCS-based F-measure to estimate the similarity between the large model-generated sentence and the reference term explanation sentence. Assuming X is the reference term explanation sentence and Y is the term explanation sentence output by the large model, the formula is as follows:

[0042]

[0043]

[0044] in, Recall, also known as coverage, indicates how much of the reference content the model's output covers. Precision indicates how much of the model output is reference content; This represents the harmonic mean.

[0045] (3) Calculate the F1 score for the corresponding execution results of the paper text classification task; the F1 score is often used to evaluate the classification effect. The F1 score is an important indicator for comprehensively evaluating the model performance in classification algorithms. It is the harmonic mean of precision (P) and recall (R). The calculation formula is as follows:

[0046] The F1 score ranges from 0 to 1, with higher values ​​indicating better model performance. A high F1 score is only achieved when both precision and recall are high, avoiding the limitations of focusing on a single metric and providing a more comprehensive reflection of the model's performance in classification tasks. This is particularly suitable for evaluating classification models with imbalanced data, effectively measuring the model's overall performance. Furthermore, in this embodiment, a weighted averaging strategy is employed to calculate the model's score in large-scale text classification tasks. First, weights are determined according to the sample proportion of each category. Then, these weights are used to calculate the weighted average of each metric, ensuring that the importance of each category is reflected as much as possible.

[0047] (4) Calculate the ROUGE-N metric for the execution results corresponding to the paper abstract generation task; ROUGE-N evaluates the recall of the model's generated abstract by calculating the co-occurrence of N-grams. In this embodiment of the invention, the ROUGE-N metric is used to evaluate the score of the paper abstract generation task, where N refers to the length of the N-gram (N consecutive words), and the calculation formula is as follows:

[0048] Where S represents the generated summary, Represents N consecutive words, Indicates a reference summary, Count match This represents the maximum number of N-grams that appear together in the generated and reference summaries.

[0049] (5) Calculate the BLEU index and chrF index for the execution results corresponding to the academic text translation task.

[0050] In this embodiment of the invention, the BLEU (Bilingual Evaluation Understudy) metric primarily focuses on measuring the similarity between the machine translation output and the reference translation, that is, evaluating the accuracy of machine translation by calculating the matching degree of N-grams (n consecutive words). This invention uses the BLEU-N metric to evaluate the score of the text translation task, where N refers to the length of the N-gram (N consecutive words), and its calculation formula is as follows:

[0051] in, 'S' represents the reference text, 'S' represents the generated text, and 'Count' represents the reference text. match This represents the maximum number of N-grams that appear together in the generated text and the reference text. The denominator represents the total number of N-grams in the generated text.

[0052] In addition to the BLEU metric, we also used the chrF metric to measure the accuracy of the text translation task. Compared to the BLEU metric, the chrF metric focuses more on character-level calculations, and its calculation formula is as follows:

[0053] Here, chrP is precision, referring to the proportion of character-level n-grams that match the reference sentence in the translated sentence generated by the large model. chrR is recall, referring to the proportion of character-level n-grams that match the reference sentence in the translated sentence. β indicates that recall is β times more important than precision; it is used here to control the importance of both recall and precision.

[0054] In this embodiment of the invention, the current execution end further determines the comprehensive capability of the financial model to be evaluated based on the multiple evaluation indicators calculated above. Specifically, this includes: normalizing each of the evaluation indicators to obtain normalized evaluation indicator values; and performing a weighted summation of the normalized evaluation indicator values ​​to obtain the comprehensive score of the financial model to be evaluated, as shown in the following formula:

[0055] Among them, P i P represents the actual score of the i-th indicator. min,i and P max,i These represent the minimum and maximum values ​​of the indicator, w. i The weights are used to represent the indicator weights. This invention uses normalization and weighted summation methods to ultimately derive the overall score of the model, ensuring fair comparison and comprehensive evaluation among different indicators.

[0056] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, in order to effectively eliminate data pollution in the training dataset and ensure that the evaluation results truly reflect the capabilities of the large model in the financial field, rather than relying on the correct answers that are accidentally matched during training, thereby improving the accuracy and credibility of the evaluation, another evaluation method for the capabilities of large financial models is provided, as shown in Figure 3. The steps involve constructing a three-level defense mechanism based on a large model decontamination framework using deep semantic screening to decontaminate the basic corpus, and constructing an evaluation dataset based on the decontaminated basic corpus, including: 201. Using a combination of vector space retrieval and similarity calculation to perform coarse-grained screening on the selected basic corpus to identify potentially contaminated samples; In this embodiment of the invention, the current execution end uses vector space retrieval technology (FAISS Index) as the first line of defense for data decontamination, utilizing the efficient similarity retrieval capability of this technology to quickly screen potentially contaminated data in candidate samples. Candidate samples from the evaluation dataset are vector-transformed with samples from the known large model training dataset. A preliminary contamination assessment is then performed by calculating the cosine similarity between the two samples, with a similarity threshold of ≥0.82. Samples with a cosine similarity of ≥0.82 are considered potentially contaminated. Highly similar samples selected after cosine similarity calculation are individually labeled to form a candidate pool of potentially contaminated samples, which then proceed to the next level of defense for further verification. Samples that do not meet the similarity threshold are directly classified as clean samples and retained in the final evaluation dataset.

[0057] 202. A pre-trained discriminant model is used to perform deep verification processing on the potentially contaminated samples to determine the real contaminated samples. In this embodiment of the invention, the current execution end uses a pre-trained discriminant model to perform deep verification processing on the potentially contaminated samples. For example, a locally trained DeepSeek-R1-Large discriminant model is deployed. This model is built on a 32-layer Transformer architecture with a hidden layer dimension (hidden_size) of 4096. It has powerful semantic understanding and feature extraction capabilities and is specifically used for the accurate identification of potentially contaminated samples. A semantic fingerprint matching algorithm is used to perform deep verification on the potentially contaminated candidate samples screened by the first-level defense. By extracting the semantic fingerprint features of the samples, a precise comparison is made with the semantic fingerprint database of the training dataset samples to determine whether the samples are substantially contaminated from a semantic perspective. A confidence threshold p < 0.01 is set. That is, when the confidence of the discriminant model in determining that a sample is a contaminated sample meets the threshold condition, the sample is confirmed as a real contaminated sample. If the confidence does not reach the threshold, the sample is returned to the clean sample pool to avoid data loss caused by false screening.

[0058] 203. A dynamic reconstruction engine is used to perform syntactic deconstruction on the real contaminated samples and generate semantically equivalent but syntactically different alternative samples; and the alternative samples are used to update the corresponding basic corpus.

[0059] In this embodiment of the invention, the current execution end does not directly delete genuine contaminated samples confirmed by secondary defense (to avoid insufficient dataset coverage). Instead, it performs semantically equivalent reconstruction through a dynamic reconstruction engine to ensure data integrity and usability. The dynamic reconstruction engine performs syntactic deconstruction on the contaminated samples, optimizing and adjusting sentence structure, expression, and vocabulary selection while fully preserving the core semantics and key information of the original samples. This generates semantically equivalent but syntactically different alternative samples, avoiding similarity associations with large model training data. The generated alternative samples are quality-verified using BERTScore to ensure that the reconstructed samples meet the semantic consistency standard with the original samples, and a BERTScore of 0.91 ± 0.03 is required to ensure that the alternative samples meet the requirements of the evaluation task.

[0060] It should be noted that the effectiveness of the entire decontamination scheme was also verified through CoQA benchmark testing in this embodiment. The results show that the scheme can reduce the evaluation bias by 41.7% (ΔEM=+12.3), control the false positive rate below 3.2% (95% confidence interval: 2.8-3.6%), and is significantly better than the traditional N-gram filtering method (t=5.34, p<0.001), proving the effectiveness and superiority of the decontamination effect. Through the layer-by-layer screening and processing of the three-level defense mechanism, an anti-leakage protocol is established for the model capability assessment to ensure that the final evaluation dataset is free from data contamination, has verifiable statistical validity, and provides reliable data support for subsequent evaluation tasks.

[0061] 204. Perform data structuring processing on the decontaminated basic corpus to transform the basic corpus into a form suitable for each of the evaluation tasks; In this embodiment of the invention, the current execution terminal performs data structuring processing on the corpus in the decontaminated basic corpus from steps 201 to 203 above, transforming the basic corpus into a form suitable for each of the evaluation tasks. For example, based on the core information of the text, according to the form of "question-answer", "terminology-explanation", "text-key information" and other forms suitable for evaluation tasks, the decontaminated text is transformed into structured question-answer pairs or standardized data units, improving the adaptability of the data to subsequent evaluation tasks and providing directly usable materials for model evaluation.

[0062] 205. Based on the construction logic of domain knowledge graph, the processed structured data is divided into four major discipline modules according to knowledge attributes, including finance module, economics module, accounting module and industry qualification certification module; In this embodiment of the invention, the current execution end is based on the construction logic of domain knowledge graph, and the processed structured data is divided into four core modules according to knowledge attributes, so as to achieve the dual guarantee of professional coverage and interdisciplinary nature: (1) Finance module: 11 core knowledge units are constructed, covering key areas such as monetary banking, financial market, and corporate finance, and finally 1538 sets of structured question and answer pairs are formed to achieve in-depth coverage of financial knowledge. Cover; (2) Economics module: Deeply analyzes 7 theoretical dimensions such as macroeconomics, microeconomics, and econometrics, and precipitates 1623 sets of structured question and answer pairs, comprehensively covering the core theories and applications of economics; (3) Accounting module: Decomposed into 10 practical fields such as financial accounting, management accounting, and auditing, and constructs 1180 structured question and answer pairs, which are in line with accounting practice operations and professional knowledge application scenarios; (4) Industry qualification certification module: Based on 7 core assessment subjects (such as securities practitioners, fund practitioners and other qualification examination subjects), 1342 sets of scenario-based assessment question and answer pairs are generated to match the assessment needs of financial industry practitioners' practical skills.

[0063] 206. Adjust the data ratio of each subject module so that the knowledge density ratio of the finance module, the economics module, the accounting module and the industry qualification certification module reaches the preset ratio, so as to obtain the evaluation dataset that has both professional depth and cross-border breadth.

[0064] In this embodiment of the invention, the current execution end optimizes and verifies the data distribution of the four major modules to ensure the statistical significance and balance of the dataset, including: (1) Distribution feature verification: Through statistical analysis, it is confirmed that the overall distribution of the dataset conforms to the characteristics of Benford's Law (χ²=1.84, p>0.05), which has good statistical rationality; (2) Knowledge density balance: The data ratio of each subject module is adjusted so that the knowledge density ratio of the four major modules of finance, economics, accounting and industry qualification certification reaches 1:1.25:0.92:1.08, which not only ensures the professional depth of each module, but also achieves a balanced configuration of cross-disciplinary breadth; (3) Final verification: Through the above optimization, the dataset forms an evaluation base with both professional depth and cross-disciplinary breadth, which has statistical significance and can provide comprehensive and objective test material support for subsequent multi-dimensional evaluation tasks.

[0065] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, in order to enhance the instruction following accuracy of the model, another evaluation method for the capabilities of large financial models is provided, as shown in Figure 4. After configuring corresponding prompt instructions for each of the evaluation tasks, based on the hierarchical prompting engineering framework and few-shot learning strategy, through a closed-loop process of "baseline establishment - progressive example injection - dual-dimensional quantitative evaluation - optimal value determination", the optimal number of prompt examples required for each of the five evaluation tasks is accurately determined. The specific process is as follows: 301. Construct multiple contextual example templates for the multi-question answering task, the terminology explanation task, the text generation task, the text classification task, and the text translation task respectively; In this embodiment of the invention, the current execution end constructs contextual example templates for the core requirements of the five evaluation tasks of multi-question answering, terminology explanation, text generation, text classification, and text translation respectively. The template may contain core elements such as task scenario description, input format examples, and output standard examples. This embodiment of the invention does not make specific limitations, ensuring the adaptability of examples to tasks and providing clear instruction following references for the model.

[0066] 302. A graded prompting test is conducted on the large financial model by progressively increasing the number of context example templates, and the generation results of the large financial model at each gradient are recorded. In this embodiment of the invention, the current execution end adopts a graded prompting engineering framework, clarifying the implementation path of few-shot learning, that is, gradually transitioning from no-example injection (0-shot) to multiple-example injection (1-shot to 3-shot). By progressively increasing the number of examples, the model's understanding of the task depth is systematically verified. Specifically, in the no-example injection (0-shot) scenario, for each evaluation task, only a basic prompt instruction is input to the model, without any task-related examples (0-shot), allowing the model to complete the task output without reference cases. The task execution results of the model in the zero-shot scenario are recorded, focusing on collecting the core data required for subsequent evaluation, and establishing a benchmark for performance comparison under different numbers of examples.

[0067] During tiered prompt testing, contextual example templates are added to the prompt instructions for each task sequentially, following a gradient of 1-shot (injecting 1 task example), 2-shot (injecting 2 task examples), and 3-shot (injecting 3 task examples). Example selection follows the principles of "typicality and diversity," ensuring that examples cover the core scenarios and key requirements of the tasks. For each gradient of example quantity (1-shot, 2-shot, 3-shot) for each task, the same test sample set is used for model testing to ensure consistency in testing conditions. The model's generation results are recorded at each gradient to ensure data comparability.

[0068] 303. Using perplexity and ROUGE-L value as dual-dimensional evaluation indicators, the semantic coherence and task suitability of the generated results are quantified; and the number of target context example templates corresponding to each evaluation task is determined based on the quantification results.

[0069] In this embodiment of the invention, the current execution end uses perplexity (PPL) and ROUGE-L value as two-dimensional evaluation indicators to comprehensively quantify the quality of the generated results: (1) Perplexity (PPL): used to evaluate the semantic coherence of the generated results. The lower the PPL value, the stronger the fluency and logic of the generated text and the more accurate the understanding of the instructions; (2) ROUGE-L value: used to evaluate the task adaptability of the generated results to the reference standard. The higher the ROUGE-L value, the higher the fit between the generated content and the task requirements and the better the instruction following effect.

[0070] In this embodiment, the current execution end performs comparative analysis on the multi-gradient indicator data for each evaluation task, focusing on the decreasing trend of perplexity (PPL) and the increasing trend of ROUGE-L value as the number of examples increases. When the number of examples increases to a certain value, if both dimensions of the indicators reach the optimal level (lowest PPL and highest ROUGE-L value), and the indicators do not show significant optimization or even decline after further increasing the number of examples, then this value is the optimal number of prompt examples for the task, i.e., the number of target context example templates.

[0071] This invention provides a method for evaluating the capabilities of a large financial model. Compared with existing technologies, this invention selects basic financial knowledge and relevant academic texts as basic data sources to obtain a basic corpus, ensuring the comprehensiveness and professionalism of the sample data. It also constructs a three-tiered defense mechanism based on a deep semantic screening framework to decontaminate the basic corpus; and builds an evaluation dataset based on the decontaminated corpus, eliminating data pollution and ensuring the accuracy of the evaluation results. Furthermore, this invention creates multiple evaluation tasks for both the financial knowledge and academic texts, and configures corresponding prompts for each task, simulating different industry scenarios through specific roles to enhance the modeling capabilities of the domain knowledge. Finally, this invention calculates corresponding evaluation indicators based on the execution results of each evaluation task, and determines the comprehensive capabilities of the large financial model under evaluation based on these indicators, achieving a comprehensive assessment of the model's generation quality and question-answering accuracy from different perspectives.

[0072] As an implementation of the method shown in Figure 1 above, this embodiment of the invention provides a device for evaluating the capabilities of a large financial model, as shown in Figure 5. The device includes: a data acquisition module 41, used to select basic financial knowledge and relevant academic texts within the industry as basic data sources to acquire basic corpus based on two application scenarios: financial domain knowledge and financial domain academic texts, generating a basic corpus; a decontamination and construction module 42, used to construct a three-level defense mechanism based on a large model decontamination framework using deep semantic screening to decontaminate the basic corpus; and to construct an evaluation dataset based on the decontamination-treated basic corpus; and a task creation module 43, used to create evaluation datasets for the financial domain... Multiple evaluation tasks are created using domain knowledge and academic texts in the financial field; and corresponding prompts are configured for each evaluation task; the evaluation processing module 44 is used to obtain the large financial model to be evaluated, and to evaluate the large financial model to be evaluated using the evaluation dataset, so that the large financial model to be evaluated sequentially executes each evaluation task corresponding to the prompts, and obtains the execution results corresponding to each evaluation task; the evaluation analysis module 45 is used to calculate the corresponding evaluation indicators based on the execution results corresponding to each evaluation task, and to determine the comprehensive capability of the large financial model to be evaluated based on the evaluation indicators.

[0073] Furthermore, the decontamination and construction module 42 includes a decontamination unit, which is used to: perform coarse-grained screening on the selected basic corpus using a combination of vector space retrieval and similarity calculation to identify potential contaminated samples; perform deep verification processing on the potential contaminated samples using a pre-trained discriminant model to identify real contaminated samples; perform syntactic deconstruction on the real contaminated samples using a dynamic reconstruction engine and generate semantically equivalent but syntactically different alternative samples; and update the corresponding basic corpus using the alternative samples.

[0074] Furthermore, the decontamination and construction module 42 includes a construction unit, which is used to: perform data structuring processing on the decontamination basic corpus, transforming the basic corpus into a form suitable for each of the evaluation tasks; based on the construction logic of the domain knowledge graph, divide the processed structured data into four major subject modules according to knowledge attributes, including the finance module, economics module, accounting module, and industry qualification certification module; adjust the data ratio of each subject module so that the knowledge density ratio of the finance module, the economics module, the accounting module, and the industry qualification certification module reaches a preset ratio, thereby obtaining the evaluation dataset that has both professional depth and cross-border breadth.

[0075] Furthermore, the task creation module 43 is also used to: create multiple question-and-answer tasks and terminology explanation tasks for the generation scenario of financial knowledge; and configure corresponding prompts for the multiple question-and-answer tasks and the terminology explanation tasks respectively; and create paper text classification tasks, paper abstract generation tasks and academic text translation tasks for the model generation scenario of academic text in the financial field; and configure corresponding prompts for the paper text classification tasks, the paper abstract generation tasks and the academic text translation tasks respectively.

[0076] Furthermore, the device also includes a tiered testing module, which is used to: construct multiple contextual example templates for the multiple question-answering tasks, the terminology explanation task, the text generation task, the text classification task, and the text translation task respectively; perform graded prompting tests on the financial big model by increasing the number of contextual example templates in a tiered manner, and record the generation results of the financial big model at each gradient; use perplexity and ROUGE-L value as dual-dimensional evaluation indicators to quantify the semantic coherence and task suitability of the generation results; and determine the number of target contextual example templates corresponding to each evaluation task based on the quantification results.

[0077] Furthermore, the evaluation and analysis module 45 is also used to: calculate the accuracy rate of the execution results corresponding to multiple question-answering tasks; calculate the MAUVE index and ROUGE-L index of the execution results corresponding to the terminology explanation task; calculate the F1 score of the execution results corresponding to the paper text classification task; calculate the ROUGE-N index of the execution results corresponding to the paper abstract generation task; and calculate the BLEU index and chrF index of the execution results corresponding to the academic text translation task.

[0078] Furthermore, the evaluation and analysis module 45 is also used to: normalize each of the evaluation indicators to obtain normalized evaluation indicator values; and perform weighted summation on the normalized evaluation indicator values ​​to obtain the comprehensive score of the financial big model to be evaluated.

[0079] This invention provides an evaluation device for the capabilities of large financial models. Compared with existing technologies, this invention selects basic financial knowledge and relevant academic texts within the industry as basic data sources to obtain a basic corpus, ensuring the comprehensiveness and professionalism of the sample data. Furthermore, a three-tiered defense mechanism based on a deep semantic screening framework is constructed to decontaminate the basic corpus. An evaluation dataset is then built based on the decontaminated basic corpus, eliminating data pollution and ensuring the accuracy of the evaluation results. This invention also creates multiple evaluation tasks for the financial knowledge and academic texts, respectively, and configures corresponding prompts for each task. By simulating different industry scenarios through specific roles, it enhances the modeling capabilities of the domain knowledge. Finally, this invention calculates corresponding evaluation indicators based on the execution results of each evaluation task and determines the comprehensive capabilities of the large financial model under evaluation based on these indicators, achieving a comprehensive assessment of the model's generation quality and question-answering accuracy from different perspectives.

[0080] According to one embodiment of the present invention, a storage medium is provided, the storage medium storing at least one executable instruction, the computer-executable instruction being capable of executing the evaluation method for the financial large model capability in any of the above method embodiments.

[0081] Figure 6 shows a schematic diagram of an electronic device according to an embodiment of the present invention. The specific implementation of the present invention does not limit the specific implementation of the electronic device.

[0082] As shown in Figure 6, the electronic device may include: a processor 502, a communications interface 504, a memory 506, and a communications bus 508.

[0083] The processor 502, communication interface 504, and memory 506 communicate with each other via communication bus 508.

[0084] Communication interface 504 is used to communicate with other network elements such as clients or other servers.

[0085] Processor 502 is used to execute program 510, which can specifically perform the relevant steps of the above-mentioned evaluation method for the capabilities of large financial models.

[0086] Specifically, program 510 may include program code that includes computer operation instructions.

[0087] Processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The electronic device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0088] Memory 506 is used to store program 510. Memory 506 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0089] Specifically, program 510 can be used to enable processor 502 to perform the following operations: based on two major application scenarios—financial knowledge and academic texts in the financial field—it selects basic financial knowledge and relevant academic texts in the industry as basic data sources to obtain basic corpus and generate a basic corpus; it constructs a three-level defense mechanism based on a large-scale model decontamination framework using deep semantic screening to decontaminate the basic corpus; it constructs an evaluation dataset based on the decontaminated basic corpus; it creates multiple evaluation tasks for the financial knowledge and academic texts in the financial field respectively; and it configures corresponding prompts for each evaluation task; it obtains a large-scale financial model to be evaluated and uses the evaluation dataset to evaluate the large-scale financial model to be evaluated, so that the large-scale financial model to be evaluated sequentially executes each evaluation task corresponding to the prompts, and obtains the execution results corresponding to each evaluation task; it calculates corresponding evaluation indicators based on the execution results corresponding to each evaluation task, and determines the comprehensive capability of the large-scale financial model to be evaluated based on the evaluation indicators.

[0090] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0091] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for evaluating the capabilities of a large financial model, characterized in that, include: Based on two major application scenarios—financial knowledge and academic texts in the financial field—basic financial knowledge and relevant academic texts within the industry are selected as basic data sources to obtain basic corpus and generate a basic corpus. A three-tiered defense mechanism is constructed based on a large-scale model decontamination framework using deep semantic screening to decontaminate the aforementioned basic corpus. An evaluation dataset was constructed based on the decontaminated basic corpus; Multiple evaluation tasks were created for the aforementioned financial knowledge and the aforementioned academic texts in the financial field, respectively. The system configures corresponding prompts for each evaluation task; obtains the financial model to be evaluated, and uses the evaluation dataset to evaluate the financial model, so that the financial model to be evaluated sequentially executes each evaluation task corresponding to the prompts, and obtains the execution results corresponding to each evaluation task; calculates the corresponding evaluation indicators based on the execution results corresponding to each evaluation task, and determines the comprehensive capability of the financial model to be evaluated based on the evaluation indicators.

2. The method according to claim 1, characterized in that, The large-scale decontamination framework based on deep semantic screening constructs a three-level defense mechanism to decontaminate the basic corpus, including: using a combination of vector space retrieval and similarity calculation to perform coarse-grained screening on the selected basic corpus to identify potentially contaminated samples; using a pre-trained discriminant model to perform deep verification on the potentially contaminated samples to identify real contaminated samples; using a dynamic reconstruction engine to perform syntactic deconstruction on the real contaminated samples and generate semantically equivalent but syntactically different alternative samples; and using the alternative samples to update the corresponding basic corpus.

3. The method according to claim 1, characterized in that, The construction of the evaluation dataset based on the decontaminated basic corpus includes: performing data structuring processing on the decontaminated basic corpus to transform the basic corpus into a form suitable for each evaluation task; based on the construction logic of domain knowledge graphs, dividing the processed structured data into four major subject modules according to knowledge attributes, including finance, economics, accounting, and industry qualification certification; adjusting the data ratio of each subject module to achieve a preset ratio of knowledge density for the finance, economics, accounting, and industry qualification certification modules, thereby obtaining the evaluation dataset that combines professional depth and cross-disciplinary breadth.

4. The method according to claim 1, characterized in that, The process involves creating multiple evaluation tasks for both financial domain knowledge and academic texts in the financial domain, and configuring corresponding prompts for each evaluation task. This includes: creating multiple question-and-answer tasks and terminology explanation tasks for the generation scenario of financial domain knowledge; and configuring corresponding prompts for each of the multiple question-and-answer tasks and the terminology explanation tasks; creating paper text classification tasks, paper abstract generation tasks, and academic text translation tasks for the model generation scenario of academic texts in the financial domain; and configuring corresponding prompts for each of the paper text classification tasks, the paper abstract generation tasks, and the academic text translation tasks.

5. The method according to claim 4, characterized in that, After configuring corresponding prompts for each of the evaluation tasks, the method further includes: constructing multiple contextual example templates for each of the multiple question-answering tasks, the terminology explanation task, the text generation task, the text classification task, and the text translation task; conducting graded prompt tests on the financial big model by increasing the number of contextual example templates in a stepwise manner, and recording the generation results of the financial big model at each gradient; using perplexity and ROUGE-L value as dual-dimensional evaluation indicators to quantify the semantic coherence and task suitability of the generation results; and determining the number of target contextual example templates corresponding to each evaluation task based on the quantification results.

6. The method according to claim 1, characterized in that, The calculation of multiple evaluation metrics based on the execution results corresponding to each of the evaluation tasks includes: calculating the accuracy rate for the execution results corresponding to multiple question-answering tasks; calculating the MAUVE and ROUGE-L metrics for the execution results corresponding to the terminology explanation task; calculating the F1 score for the execution results corresponding to the paper text classification task; calculating the ROUGE-N metric for the execution results corresponding to the paper abstract generation task; and calculating the BLEU and chrF metrics for the execution results corresponding to the academic text translation task.

7. The method according to any one of claims 1 to 6, characterized in that, The step of determining the comprehensive capability of the financial model to be evaluated based on the evaluation indicators includes: normalizing each of the evaluation indicators to obtain normalized evaluation indicator values; and performing weighted summation on the normalized evaluation indicator values ​​to obtain the comprehensive score of the financial model to be evaluated.

8. A device for evaluating the capabilities of a large financial model, characterized in that, include: The data acquisition module is used to select basic financial knowledge and relevant academic texts in the industry as basic data sources to acquire basic corpus based on two major application scenarios: financial knowledge and financial academic texts, and generate a basic corpus. The decontamination and construction module is used to build a three-level defense mechanism based on a large model decontamination framework using deep semantic screening to decontaminate the basic corpus. An evaluation dataset was constructed based on the decontaminated basic corpus; The task creation module is used to create multiple evaluation tasks for the financial knowledge and the academic texts in the financial field, respectively. The evaluation processing module is configured with corresponding prompts for each evaluation task; the evaluation processing module is used to obtain the financial large model to be evaluated and to evaluate the financial large model to be evaluated using the evaluation dataset, so that the financial large model to be evaluated executes each evaluation task corresponding to the prompts in sequence, and obtains the execution results corresponding to each evaluation task. The evaluation and analysis module is used to calculate the corresponding evaluation indicators based on the execution results of each evaluation task, and to determine the comprehensive capability of the financial big data model to be evaluated based on the evaluation indicators.

9. A storage medium, characterized in that, The storage medium stores at least one executable instruction, which performs the operation corresponding to the evaluation method for the financial large model capability as described in any one of claims 1-7.

10. An electronic device, characterized in that, It includes a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface communicate with each other through the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the evaluation method of the financial large model capability as described in any one of claims 1-7.

Images