A method for evaluating the entire application process of a large language model for complex decision-making scenarios

Abstract

This paper presents a comprehensive evaluation method for large language models in complex decision-making scenarios, relating to the field of artificial intelligence evaluation technology. The method includes: constructing a domain knowledge base and test case set; calling the large language model to be evaluated, executing test cases, and obtaining its output results; performing quantitative evaluations in parallel or sequentially according to multiple defined evaluation dimensions based on the output results and standard answers; and aggregating the quantitative index calculation results of each dimension to generate a structured comprehensive evaluation result. This method decomposes large model evaluation into five closely interconnected dimensions: data, knowledge, reasoning, generation, and evolution. It proposes a series of dedicated quantitative indicators for complex decision-making challenges in vertical domains, enabling a systematic, comprehensive, and in-depth evaluation of the comprehensive capabilities of large language models in complex decision-making scenarios, effectively guiding model selection and application.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence evaluation technology, and in particular to a method for evaluating the entire application process of a large language model (LLM) for complex decision-making scenarios. Background Technology

[0002] As large language models are increasingly applied to various complex decision-making tasks, the reliability, security, and effectiveness of their outputs have become crucial. Currently, model evaluation in the general domain focuses primarily on text generation quality, commonsense question answering, or general reasoning capabilities, lacking a systematic evaluation framework tailored to the complex decision-making needs of specific vertical domains.

[0003] The existing evaluation system has the following shortcomings: 1. Poor domain adaptability: General evaluation metrics cannot effectively measure the model's understanding and adherence to domain-specific knowledge, terminology, data formats, and rules.

[0004] 2. Single-dimensional approach: Focusing too much on the accuracy of the final result while neglecting comprehensive consideration of multiple dimensions such as data compatibility, logical consistency, generation standardization, and dynamic evolution capabilities.

[0005] 3. Insufficient dynamism: There is a lack of means to evaluate the performance evolution efficiency of the model in dynamic scenarios such as continuous learning, adversarial environments, and human-computer interaction.

[0006] 4. Weak interpretability: It is difficult to assess the transparency, logical completeness, and compliance with domain rules (such as industry regulations and operating procedures) of the model's decision-making process.

[0007] Therefore, there is an urgent need for an evaluation system that can comprehensively and deeply assess the practical application capabilities of large models in complex decision-making scenarios. Summary of the Invention

[0008] This disclosure provides a comprehensive performance evaluation method for Large Language Models (LLMs) across the entire application process in complex decision-making scenarios characterized by multi-source heterogeneous data, multiple constraints, and dynamic uncertainty (such as smart city scheduling, emergency management, industrial control, and financial risk control). The aim is to: construct a multi-level evaluation framework covering five core capabilities: data acquisition, knowledge understanding, inference and prediction, result generation, and learning evolution; and design a series of quantifiable indicators to accurately measure the model's performance in handling multi-source heterogeneous data, complex constraint planning, uncertain decision-making, and dynamic environmental adaptation. This evaluation system helps to select and optimize large models more suitable for specific complex decision-making scenarios, promoting their safe, reliable, and efficient application in real-world business.

[0009] The method for evaluating the entire application process of a large language model for complex decision-making scenarios disclosed herein mainly includes the following steps: S1, build a domain knowledge base and test case set to establish the domain knowledge system and test scenarios required for evaluation; S2, Multi-dimensional Ability Assessment: The large language model (LLM) to be assessed is invoked, test cases are executed, and its output results are obtained. Based on the output results and the standard answers, quantitative assessments are performed in parallel or sequentially according to multiple defined assessment dimensions. S3, Comprehensive Evaluation: Aggregates the calculation results of all quantitative indicators from various dimensions to generate a structured comprehensive evaluation result.

[0010] Compared with existing technologies, the beneficial effects of this disclosure are: ① It decomposes large model evaluation into five closely connected dimensions: data, knowledge, reasoning, generation, and evolution, providing greater breadth and depth of evaluation and fully reflecting the model's performance in practical applications; ② It proposes a series of dedicated quantitative indicators for complex decision-making challenges in vertical domains, which can not only be used for model comparison and ranking, but also effectively diagnose model capability shortcomings, providing clear directional guidance for model optimization, training data construction, and application scenario selection; ③ This methodology does not depend on specific system implementations and can be widely applied to various complex decision-making scenarios that require in-depth evaluation of large models. Attached Figure Description

[0011] The above and other objects, features and advantages of this disclosure will become more apparent from the more detailed description of exemplary embodiments of this disclosure taken in conjunction with the accompanying drawings, in which the same reference numerals generally represent the same components.

[0012] Figure 1 Here is an overall flowchart of the large language model full application process evaluation method in an exemplary embodiment; Figure 2 This provides an example of how test cases and knowledge base building modules are constructed. Detailed Implementation

[0013] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0014] This disclosure provides a method for evaluating the entire application process of a large language model for complex decision-making scenarios. The process is shown in the attached figure. Figure 1 As shown, the main steps include: building a domain knowledge base and test cases, multi-dimensional capability assessment, and comprehensive evaluation.

[0015] 1. Build a domain knowledge base and test case set Based on the complex decision-making scenarios of target evaluation, a standard knowledge base is constructed, which includes domain terminology, business rules, constraints, and historical cases. A series of test cases covering different task types and difficulties were designed. Each test case includes multimodal input instructions / data and corresponding expert annotations or recognized ideal outputs as standard answers.

[0016] 2. Multi-dimensional ability assessment Input the test cases into the large language model to be evaluated and obtain its output results; based on the output results and the standard answer, perform quantitative evaluation in parallel or sequentially according to the following five dimensions: Dimension 1: Data Acquisition Capability Assessment Objective: To evaluate the model's compatibility and preprocessing capabilities for multi-source heterogeneous data.

[0017] Methods: Calculate indicators such as domain database coverage, success rate of heterogeneous file format parsing, accuracy of multimodal data alignment, and F1 score of false / sensitive information alarms.

[0018] Dimension Two: Knowledge Comprehension Ability Assessment Objective: To evaluate the model's ability to deeply understand domain knowledge, instruction intent, and contextual logic.

[0019] Methods: Metrics include Kappa coefficient for computational domain terminology understanding, multilingual instruction alignment accuracy, task context association coverage, and multimodal data fusion gain.

[0020] Dimension Three: Assessment of Reasoning and Prediction Ability Objective: To evaluate the model's ability to perform logical reasoning and situation prediction under complex constraints and uncertain environments.

[0021] Methods: The F1 score of commander intent reasoning, accuracy of spatiotemporal relationship modeling, Brier score of enemy intent prediction, completeness of causal chain inference, and proportion of Pareto frontier in multi-constraint task planning were calculated.

[0022] Dimension Four: Assessment of Result Generation Capability Objective: To evaluate the usability, standardization, and interpretability of the model's output results.

[0023] Methods: Calculate indicators such as the degree of requirement satisfaction of generated solutions, the compliance rate of generated instructions, the fluency and confusion of generated semantics, the comprehensive score of the interpretability of generated solutions, and the coefficient of difference between solutions.

[0024] Dimension Five: Assessment of Learning and Evolutionary Ability Objective: To evaluate the model's adaptability and efficiency in continuous improvement within a dynamic environment.

[0025] Methods: The following indicators were calculated: knowledge absorption rate of incremental learning for new tasks, number of convergence iterations in human-machine collaboration, cross-domain transfer accuracy, slope of small sample adaptation learning curve, and historical task forgetting control rate.

[0026] 3. Comprehensive evaluation By aggregating all quantitative indicators from five dimensions, a structured comprehensive evaluation result is generated. The model's performance in each dimension is visualized through radar charts and bar charts, and strengths analysis and improvement suggestions are provided based on the scores.

[0027] In one exemplary implementation, taking the complex decision-making scenario of "smart city traffic scheduling" as an example, the evaluation method of the full application process of the large language model for complex decision-making scenarios is further explained.

[0028] This embodiment mainly includes: a test case and knowledge base construction module, a multi-dimensional capability evaluation execution module, and a comprehensive evaluation module.

[0029] 1. The test case and knowledge base construction module is used to build the domain knowledge system and test scenarios required for evaluation. In this embodiment, this module is built on three aspects: domain knowledge, task type, and data format, and then refined layer by layer into operable and quantifiable test elements. Its core components are as follows: Figure 2 As shown.

[0030] Domain knowledge base: contains specialized terms in the field of "intelligent transportation" (such as "green wave", "tidal lane", "saturation"), business rules (such as relevant clauses in the "Implementation Specifications for Traffic Signal Timing of Urban Roads"), geographic information (such as the coordinates of key intersections and the number of lanes), equipment parameters (such as the signal light cycle range and detector accuracy), and historical typical cases (such as holiday traffic management plans for a certain intersection).

[0031] Test Case Set: Multiple test tasks of varying complexity are designed. For example, a high-level task is: "During the evening rush hour, subway construction reduces the northbound traffic capacity of Renmin Road by 40%, and the probability of rainfall in the next two hours is 60%. Please generate a regional traffic management plan, prioritizing the main roads and ensuring unobstructed routes for emergency vehicles." Each test case includes: standard input (the aforementioned natural language instructions, simulated sensor data streams, and a DOCX file of the intersection structure diagram) and standard answers developed by domain experts (including specific signal timing adjustment schemes, information displayed on traffic guidance screens, and expected performance metrics).

[0032] 2. The multi-dimensional capability assessment execution module is the core of the method. It is responsible for calling the large language model (LLM) to be evaluated, executing test cases, and calculating various indicators according to predefined algorithms.

[0033] In this embodiment, an open-source large model (such as LLaMA 3) and a commercial large model (such as GPT-4) are evaluated and compared. The evaluation dimensions and process are as follows: (1) Data acquisition capability assessment: Input a test case into the model containing real-time traffic flow data in JSON format, a screenshot of a road construction notice in PDF format (including an OCR challenge), and a voice command describing the weather. Evaluate its success rate in parsing heterogeneous file formats (whether it can correctly extract the construction end time from the PDF), multimodal data alignment accuracy (whether it can correctly associate the construction location with the road segment ID in the JSON), and sensitive information alarm F1 score (whether the solution generated by the model includes suggestions for de-identifying or encrypting the sensitive information of "emergency vehicle routes").

[0034] (2) Assessment of knowledge comprehension ability: Using another test case, the instruction contains the domain terminology "enable 'green wave' coordination control". The depth of knowledge understanding is assessed by calculating the domain terminology understanding Kappa coefficient (whether the model-generated scheme correctly understands "green wave" and applies it to the specified trunk line, rather than misinterpreting it as something else).

[0035] (3) Evaluation of reasoning and prediction ability: For the above-mentioned evening peak use cases, evaluate the reasoning ability of the model.

[0036] Calculating the Pareto Front Ratio (PFP) for Multi-Constraint Task Programming: Experts provide a reference set of Pareto optimal solutions that balance the objectives of "minimum average delay" and "maximum throughput." If the model generates 10 solutions, and 3 of them fall within the reference set, then the PFP is 30%.

[0037] At the same time, the accuracy of its uncertainty decision-making is calculated (whether the model considers a 60% probability of rainfall and generates response measures such as "turn on street lighting" and "issue a slippery warning").

[0038] (4) Evaluation of result generation capability: assess the quality of model output.

[0039] Calculate the demand satisfaction of the generated scheme (AS): compare the model output scheme with the expert scheme, and calculate the weighted similarity of key elements such as "traffic capacity of construction points", "average speed of main roads" and "unimpeded emergency routes".

[0040] At the same time, assess the standardization of its generated instructions (whether the output scheme conforms to the standard instruction format of the traffic control center, such as including fields such as executing unit, effective time, and issuer).

[0041] (5) Learning evolution ability assessment: simulate a dynamic scenario.

[0042] First, the model is given a simple task and its output (baseline performance) is obtained. Then, a new traffic regulation ("Speed ​​limits on roads around schools have been reduced to 30 km / h") and three related sample cases are injected for fine-tuning with a small number of samples. Subsequently, similar tasks are tested again, and the slope of the small-sample adaptation learning curve (the ratio of performance improvement to the required sample size) and the historical task forgetting control rate (RA) are calculated (to detect the degree to which the model retains previously acquired knowledge, such as "highway control," after adapting to the new traffic regulations).

[0043] 3. The comprehensive evaluation module aggregates the results of all quantitative indicators from the above five dimensions. In this embodiment, a visual radar chart is generated, and the radar chart visualization code is as follows.

[0044] import matplotlib.pyplot as plt import numpy as np # Set Chinese font plt.rcParams['font.sans-serif'] = ['SimHei'] plt.rcParams['axes.unicode_minus'] = False # Define five dimensions labels = np.array(['Data Acquisition Ability', 'Knowledge Understanding Ability', 'Reasoning and Prediction Ability', 'Result Generation Ability', 'Learning and Evolutionary Ability']) # Define the data for the two models model1_scores = np.array([0.85, 0.78, 0.92, 0.95, 0.88]) # Commercial model model2_scores = np.array([0.82, 0.75, 0.79, 0.82, 0.93]) # Open source model # Set Angle angles = np.linspace(0, 2*np.pi, len(labels), endpoint=False).tolist() angles += angles[:1] # Closed radar image model1_scores = np.concatenate((model1_scores, [model1_scores[0]])) model2_scores = np.concatenate((model2_scores, [model2_scores[0]])) # Create a chart fig, ax = plt.subplots(figsize=(8, 8), subplot_kw=dict(polar=True)) # Draw commercial model data lines ax.plot(angles, model1_scores, 'o-', linewidth=2, label='Commercial Model (GPT-4)', color='royalblue') ax.fill(angles, model1_scores, alpha=0.25, color='royalblue') # Draw open source model data lines ax.plot(angles, model2_scores, 'o-', linewidth=2, label='Open Source Model (LLaMA 3)', color='orange') ax.fill(angles, model2_scores, alpha=0.25, color='orange') # Set tags ax.set_thetagrids(np.degrees(angles[:-1]), labels) # Set y-axis tick marks ax.set_ylim(0, 1) ax.set_yticks([0.2, 0.4, 0.6, 0.8, 1.0]) # Add title and legend plt.title('Smart Transportation Scenarios Large Model Capability Comparison Radar Chart', size=16, pad=20) plt.legend(loc='upper right', bbox_to_anchor=(1.3, 1.0)) # Add grid ax.grid(True) plt.tight_layout() plt.show() In this embodiment, the commercial model scored higher than the open-source model in both the "Inference Prediction" and "Result Generation" dimensions, but were nearly identical in the "Data Acquisition" dimension (parsing complex PDFs). The open-source model exhibited higher efficiency (steeper slope) in the "Learning and Evolution" dimension. The radar chart clearly illustrates the different capabilities of the two models, providing an intuitive basis for selection: if high-reliability solution generation is required, the commercial model can be chosen; if rapid adaptation to new regulations is required and cost is a concern, the open-source model is more advantageous.

[0045] This embodiment verifies that the method described in this disclosure can systematically, comprehensively, and deeply evaluate the overall capabilities of large language models in complex decision-making scenarios, and effectively guide the selection and application of models.

[0046] The above technical solutions are merely exemplary embodiments of the present invention. For those skilled in the art, based on the application methods and principles disclosed in the present invention, it is easy to make various types of improvements or modifications, and not limited to the methods described in the specific embodiments of the present invention. Therefore, the methods described above are merely preferred and not restrictive.

Claims

1. A method for evaluating the entire application process of a large language model for complex decision-making scenarios, characterized in that, Includes the following steps: S1, build the domain knowledge base and test case set; S2, Multi-dimensional capability assessment: Call the large language model to be evaluated, execute test cases, and obtain its output results; Based on the output results and standard answers, quantitative evaluation is performed in parallel or sequentially according to multiple set evaluation dimensions; S3, Comprehensive Evaluation: Aggregates the calculation results of all quantitative indicators from various dimensions to generate a structured comprehensive evaluation result.

2. The method according to claim 1, characterized in that, Step S1 specifically includes: Based on the complex decision-making scenarios of target evaluation, a standard knowledge base is constructed, which includes domain terminology, business rules, constraints, and historical cases. Several test cases covering different task types and difficulties were designed. Each test case includes: multimodal input instructions and / or data, and corresponding expert annotations or recognized ideal output as the standard answer.

3. The method according to claim 1, characterized in that, In step S2, the evaluation dimensions include: Data acquisition capability evaluation is used to assess the model's compatibility and preprocessing capabilities with multi-source heterogeneous data. Knowledge comprehension assessment is used to evaluate the model's deep understanding of domain knowledge, instruction intent, and contextual logic. The reasoning and prediction capability assessment is used to evaluate the model's ability to perform logical reasoning and situation prediction under complex constraints and uncertain environments. The results generation capability assessment is used to evaluate the usability, standardization, and interpretability of the model's output results. Learning evolution capability assessment is used to evaluate the model's adaptability and continuous improvement efficiency in dynamic environments.

4. The method according to claim 3, characterized in that, In step S2, the evaluation metrics for each evaluation dimension include one or more of the following metrics: Data acquisition capability evaluation: domain database coverage, success rate of heterogeneous file format parsing, accuracy of multimodal data alignment, and F1 score for false / sensitive information alarms; Knowledge comprehension ability assessment: Kappa coefficient for domain terminology comprehension, accuracy of multilingual instruction alignment, task context coverage, and multimodal data fusion gain; Evaluation of reasoning and prediction capabilities: Commander's intention reasoning F1 score, accuracy of spatiotemporal relationship modeling, enemy intention prediction Brier score, completeness of causal chain inference, and proportion of Pareto front in multi-constraint task planning. Evaluation of generation capability: Degree of satisfaction of generation solution requirements, compliance rate of generation instructions, fluency and confusion of generation semantics, comprehensive score of interpretability of generation solution, and coefficient of difference between solutions; Learning evolution ability assessment: knowledge absorption rate of incremental learning for new tasks, number of convergence iterations in human-machine collaboration, cross-domain transfer accuracy, slope of small sample adaptive learning curve, and historical task forgetting control rate.

5. The method according to any one of claims 1-4, characterized in that, Step S3 includes: visually displaying the model's performance in each dimension through radar charts or bar charts, and providing strength analysis and improvement suggestions based on the scores.

Images