A multi-dimensional index fusion-based large model comprehensive evaluation method

Abstract

This invention discloses a comprehensive evaluation method for large-scale models based on multi-dimensional index fusion. The comprehensive evaluation method includes: Step S1: constructing a hierarchical evaluation system; Step S2: constructing the evaluation set and managing data; Step S3: constructing an automated evaluation process; Step S4: performing result analysis and feedback. A systematic and standardized evaluation method for large-scale models is established to quantify the comprehensive capabilities of models in general, financial, and specific financial scenarios.

Description

Technical Field

[0001] This invention relates to the field of large language models, and in particular to a comprehensive evaluation method for large models based on the fusion of multi-dimensional indicators. Background Technology

[0002] In recent years, with the widespread application of Large Language Models (LLMs) in natural language processing, intelligent question answering, text generation, and knowledge retrieval, the financial industry has begun to explore the use of large models for various tasks such as investment and financing analysis, intelligent investment advisory, risk identification, marketing copy generation, and insurance clause parsing. The application of large models in the financial field is generally divided into three levels: General scenarios: Basic capabilities unrelated to finance, such as language understanding, logical reasoning, and text generation; General financial scenarios: covering common skills such as knowledge of financial laws and regulations, chart and data calculation, and financial translation; Specific financial scenarios: Tasks that directly serve specific business scenarios, such as fund recommendations, financial statement analysis, and compliance reviews.

[0003] However, large models exhibit significant performance differences across different task levels. Due to variations in training data, fine-tuning methods, parameter sizes, computational resources, and model structure, an imbalance often exists between a model's general reasoning ability and its domain knowledge capability. For example, models fine-tuned for financial data often perform exceptionally well on domain-specific tasks, but their general capabilities diminish, making it difficult to establish a unified evaluation benchmark.

[0004] Currently, the industry mainly uses the following methods to evaluate large models: (1) General ability assessment based on public evaluation sets: Use general Chinese evaluation sets such as C-Eval, CMMLU, and SuperCLUE to evaluate the model's language comprehension, logical reasoning and knowledge mastery abilities.

[0005] (2) Professional competence assessment based on domain evaluation sets: In the financial field, open-source financial evaluation sets such as CFLUE, Fin-Eval, and FinanceIQ are often used to test the financial expertise, compliance analysis and document generation capabilities of the model.

[0006] (3) Based on subjective human evaluation: experts are invited to score the content generated by the model and conduct subjective evaluation from dimensions such as correctness, logic and readability.

[0007] (4) Quantitative evaluation based on automated indicators: such as accuracy, precision, recall, F1 score, ROUGE, etc.

[0008] Existing technologies lack a unified evaluation logic and indicator system, and the evaluation dimensions of various solutions are scattered, making it difficult to simultaneously measure general and domain capabilities.

[0009] 1. Lack of a unified, multi-level evaluation system: Existing assessment methods are mostly targeted at a single task or a single dimension, failing to form an assessment system with "general capabilities - general financial capabilities - specific financial capabilities" levels, making it difficult to compare assessment results horizontally.

[0010] 2. Evaluation results cannot be quantified for comparison: Human evaluation is highly subjective, while automated evaluation has only one dimension and cannot achieve quantitative comparisons across models, versions, and stages.

[0011] 3. Domain data lacks representativeness and dynamism: Some open-source financial evaluation datasets suffer from problems such as limited samples, monotonous question types, and outdated updates, making it difficult to reflect real-world business scenarios.

[0012] 4. Low evaluation efficiency and high cost: Traditional evaluation relies on extensive manual annotation and expert scoring, resulting in long evaluation cycles that are unsuitable for rapid iteration environments of large models. Summary of the Invention

[0013] In view of the above problems, the present invention is proposed to provide a comprehensive evaluation method for large models based on multi-dimensional index fusion to overcome or at least partially solve the above problems.

[0014] According to one aspect of the present invention, a comprehensive evaluation method for large models based on multi-dimensional index fusion is provided, the comprehensive evaluation method comprising: Step S1: Construct a hierarchical evaluation system; Step S2: Evaluation set construction and data management; Step S3: Build an automated evaluation process; Step S4: Analyze and provide feedback on the results.

[0015] Optionally, step S1: constructing a hierarchical evaluation system specifically includes: Construct a hierarchical capability model and establish a pyramid-shaped capability system; Dimension mapping and task classification: each level corresponds to a specific task type and indicator dimension, and multi-task alignment is achieved through a unified definition.

[0016] Optionally, the construction of the capability hierarchy model and the establishment of a pyramid-shaped capability system specifically includes: General competency layer: The evaluation model assesses general language comprehension, logical reasoning, and multidisciplinary knowledge; General Financial Competency Layer: Evaluates the model's performance in general financial tasks such as understanding financial concepts, laws and regulations, chart analysis, and translation generation; Financial Scenario Capability Layer: Evaluates the actual output capability of the model in specific financial tasks.

[0017] Optionally, the financial tasks may specifically include: insurance interpretation, risk analysis, and investment recommendation generation.

[0018] Optionally, step S2: evaluation set construction and data management specifically includes: Selection of public datasets: Incorporating mainstream general evaluation sets and datasets from the financial field; Self-built closed-source evaluation set: Data quality and update mechanism: Establish data quality control standards; regularly update the evaluation set to reflect the latest financial dynamics.

[0019] Optionally, the self-built closed-source evaluation set specifically includes: Build your own evaluation set for specific business scenarios within the enterprise, including task types such as question answering, classification, generation, retrieval, and reasoning; The self-built process includes: target definition → data collection → data cleaning → data anonymization → data labeling → data aggregation.

[0020] Optionally, step S3: Automated evaluation process construction specifically includes: Based on the selected public and self-built evaluation sets, a unified index is calculated for the model output results; Design a standardized Prompt to enable the evaluation model to score the output of the tested model from multiple dimensions, including correctness, logic, syntax, and intelligence, and generate a reasoning report. Domain experts were brought in to review key samples, resulting in a comprehensive score that combines human and automated evaluation results. It supports horizontal and vertical comparative analysis and automatically generates radar charts and trend curves of model capabilities.

[0021] Optionally, step S4: performing result analysis and feedback specifically includes: Results aggregation and weight calculation: A comprehensive model performance score is generated by fusing multi-source scoring results through a weighted average mechanism. The weight parameters can be dynamically adjusted according to the task type and importance; Performance report generation: Automatically generates evaluation reports, including scores, rankings, and trends of three categories of indicators: model general capability, financial general capability, and scenario capability; Optimize the feedback loop: Return the evaluation results to the training phase to provide a basis for decision-making on model fine-tuning and parameter selection, forming a self-circulating mechanism of "training-evaluation-feedback".

[0022] This invention provides a comprehensive evaluation method for large models based on multi-dimensional index fusion. The comprehensive evaluation method includes: step S1: constructing a hierarchical evaluation system; step S2: constructing the evaluation set and managing the data. Step S3: Build an automated evaluation process; Step S4: Analyze and provide feedback on the results. Establish a systematic and standardized large-scale model evaluation method to quantify the comprehensive capabilities of the model in general, financial, and specific financial scenarios.

[0023] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart of a large-scale model comprehensive evaluation method based on multi-dimensional index fusion provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the capability hierarchy model design provided for an embodiment of the present invention. Detailed Implementation

[0026] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0027] The terms "comprising" and "having," and any variations thereof, in the specification, embodiments, claims, and drawings of this invention are intended to cover non-exclusive inclusion, such as including a series of steps or units.

[0028] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0029] This invention provides a method for evaluating large models. By establishing a hierarchical evaluation system, integrating multi-source datasets, and adopting an automated and model-driven evaluation mechanism, it achieves a comprehensive assessment of the general capabilities, financial domain capabilities, and scenario adaptability of large models.

[0030] like Figure 1 As shown, a comprehensive evaluation method for large-scale models based on multi-dimensional index fusion includes: Step 1: Construct a hierarchical evaluation system 1. Capability Hierarchy Model Design like Figure 2 As shown, a pyramid-shaped capability system is established, including: General competency layer: The evaluation model assesses general language comprehension, logical reasoning, and multidisciplinary knowledge; General Financial Competency Layer: Evaluates the model's performance in general financial tasks such as understanding financial concepts, laws and regulations, chart analysis, and translation generation; Financial Scenario Capability Layer: Evaluates the model's actual output capability in specific financial tasks (such as insurance interpretation, risk analysis, and investment advice generation).

[0031] 2. Dimension Mapping and Task Classification Each level corresponds to a specific task type and indicator dimension, and multi-task alignment is achieved through unified definition.

[0032] Step Two: Evaluation Set Construction and Data Management 1. Selection of Public Datasets We introduce mainstream general evaluation sets (such as C-Eval, SuperCLUE, CMMLU, GSM8K, MATH) and financial domain datasets (such as CFLUE, Fin-Eval, FinanceIQ, FinBen, FinDABench).

[0033] 2. Self-built closed-source evaluation set Build your own evaluation set for specific business scenarios within the enterprise, including task types such as question answering, classification, generation, retrieval, and reasoning.

[0034] The self-built process includes: target definition → data collection → data cleaning → data anonymization → data labeling → data aggregation.

[0035] 3. Data Quality and Update Mechanism Establish data quality control standards to ensure the accuracy, representativeness, and balance of the evaluation set; regularly update the evaluation set to reflect the latest financial dynamics.

[0036] Step 3: Design of Automated Evaluation Process 1. Benchmark Evaluation Module Based on the selected public and self-built evaluation sets, unified indicators such as accuracy, F1 score, and ROUGE are calculated for the model output results.

[0037] 2. Large Model Self-Assessment Module The standardized Prompt is designed to enable the evaluation model to score the output of the tested model from multiple dimensions, including correctness, logic, syntax, and intelligence, and to generate a reasoning report.

[0038] 3. Manual Evaluation Module Domain experts were brought in to review key samples, resulting in a comprehensive score that combines human and automated evaluation results.

[0039] 4. Multi-model comparison module It supports horizontal (different models with the same parameter scale) and vertical (different versions of the same model) comparative analysis, and automatically generates model capability radar charts and trend curves.

[0040] Step 4: Results Analysis and Feedback 1. Result aggregation and weight calculation A comprehensive score for the model's overall capability is generated by fusing multi-source scoring results through a weighted average mechanism.

[0041] The weight parameters can be dynamically adjusted according to the task type and importance.

[0042] 2. Performance report generation The system automatically generates evaluation reports, including scores, rankings, and trends for three categories of indicators: general model capabilities, general financial capabilities, and scenario capabilities.

[0043] 3. Optimize the feedback loop The evaluation results are fed back into the training phase to provide a basis for decision-making on model fine-tuning and parameter selection, forming a self-circulating mechanism of "training-evaluation-feedback".

[0044] Beneficial effects: Establish a hierarchical evaluation system to systematically assess the model's capabilities in general, financial, and specific financial scenarios; Comparative analysis between multiple models and versions can be achieved by using unified evaluation indicators and quantitative standards; Improve evaluation efficiency and reduce human intervention through automated and intelligent evaluation mechanisms; Ensure the representativeness, balance, and up-to-dateness of the evaluation dataset to improve the objectivity and reproducibility of the results.

[0045] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A comprehensive evaluation method for large-scale models based on multi-dimensional index fusion, characterized in that, The comprehensive evaluation method includes: Step S1: Construct a hierarchical evaluation system; Step S2: Evaluation set construction and data management; Step S3: Build an automated evaluation process; Step S4: Analyze and provide feedback on the results.

2. The comprehensive evaluation method for a large model based on multi-dimensional index fusion according to claim 1, characterized in that, Step S1: Constructing a hierarchical evaluation system specifically includes: Construct a hierarchical capability model and establish a pyramid-shaped capability system; Dimension mapping and task classification: each level corresponds to a specific task type and indicator dimension, and multi-task alignment is achieved through a unified definition.

3. The comprehensive evaluation method for a large model based on multi-dimensional index fusion according to claim 2, characterized in that, The construction of the capability hierarchy model and the establishment of a pyramid-shaped capability system specifically include: General competency layer: The evaluation model assesses general language comprehension, logical reasoning, and multidisciplinary knowledge; General Financial Competency Layer: Evaluates the model's performance in general financial tasks such as understanding financial concepts, laws and regulations, chart analysis, and translation generation; Financial Scenario Capability Layer: Evaluates the actual output capability of the model in specific financial tasks.

4. The comprehensive evaluation method for a large model based on multi-dimensional index fusion according to claim 3, characterized in that, The financial tasks specifically include: insurance interpretation, risk analysis, and investment recommendation generation.

5. The comprehensive evaluation method for a large model based on multi-dimensional index fusion according to claim 1, characterized in that, Step S2: Evaluation set construction and data management specifically includes: Selection of public datasets: Incorporating mainstream general evaluation sets and datasets from the financial field; Self-built closed-source evaluation set: Data quality and update mechanism: Establish data quality control standards; regularly update the evaluation set to reflect the latest financial dynamics.

6. The comprehensive evaluation method for a large model based on multi-dimensional index fusion according to claim 5, characterized in that, The self-built closed-source evaluation set specifically includes: Build your own evaluation set for specific business scenarios within the enterprise, including task types such as question answering, classification, generation, retrieval, and reasoning; The self-built process includes: target definition → data collection → data cleaning → data anonymization → data labeling → data aggregation.

7. The comprehensive evaluation method for a large model based on multi-dimensional index fusion according to claim 1, characterized in that, Step S3: Construction of the automated evaluation process specifically includes: Based on the selected public and self-built evaluation sets, a unified index is calculated for the model output results; Design a standardized Prompt to enable the evaluation model to score the output of the tested model from multiple dimensions, including correctness, logic, syntax, and intelligence, and generate a reasoning report. Domain experts were brought in to review key samples, resulting in a comprehensive score that combines human and automated evaluation results. It supports horizontal and vertical comparative analysis and automatically generates radar charts and trend curves of model capabilities.

8. The comprehensive evaluation method for a large model based on multi-dimensional index fusion according to claim 1, characterized in that, Step S4: Result analysis and feedback specifically includes: Results aggregation and weight calculation: A comprehensive model performance score is generated by fusing multi-source scoring results through a weighted average mechanism. The weight parameters can be dynamically adjusted according to the task type and importance; Performance report generation: Automatically generates evaluation reports, including scores, rankings, and trends of three categories of indicators: model general capability, financial general capability, and scenario capability; Optimize the feedback loop: Return the evaluation results to the training phase to provide a basis for decision-making on model fine-tuning and parameter selection, forming a self-circulating mechanism of "training-evaluation-feedback".

Images