Large language model evaluation system and method

Abstract

The invention discloses a big language model evaluation system and method, and belongs to the technical field of big language model evaluation. The system comprises a data preparation module, a basic performance evaluation module, an efficiency evaluation module, a robustness and multi-mode evaluation module, a distributed task scheduling module and a feedback optimization module. Through the system, the technical effect of multi-dimensionally, efficiently and automatically evaluating the large language model is achieved.

Description

Technical Field

[0001] The present application relates to the technical field of large language model evaluation, and in particular to a large language model evaluation system and method. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models are increasingly used in the field of natural language processing. Their performance and efficiency are directly related to the overall performance of the artificial intelligence system, so it is necessary to evaluate large language models.

[0003] However, traditional large language model evaluation methods often focus on single-dimensional performance indicators, such as accuracy and recall. Although these indicators can reflect the performance of the model on specific tasks, they cannot comprehensively evaluate the multi-dimensional performance of the model in practical applications. In terms of evaluation efficiency, traditional methods often adopt a serial evaluation method, that is, completing basic performance testing, efficiency testing, robustness testing and other tasks one by one. This method is not only time-consuming, but also fails to fully utilize computing resources, resulting in low evaluation efficiency. Especially when faced with large-scale data sets and complex models, the limitations of this evaluation method are particularly obvious.

[0004] In addition, traditional large language model evaluation methods require manual processing, which not only increases the complexity of the evaluation, but also easily introduces human errors. Therefore, how to evaluate large language models in a multi-dimensional, efficient and automated manner has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The embodiments of the present application provide a large language model evaluation system and method to solve the following technical problem: how to evaluate a large language model in a multi-dimensional, efficient and automated manner.

[0006] In a first aspect, an embodiment of the present application provides a large language model evaluation system, the system comprising: a data preparation module, for obtaining a standard data set, a custom data set, and an adversarial sample data set related to a target large language model, and determining an evaluation task objective; the evaluation task objective comprises: a basic performance test, an efficiency test, a robustness test, and a multi-domain adaptability test; a basic performance evaluation module, for testing the performance of the target large language model based on a standard data set and / or a custom data set; wherein the performance indicators comprise accuracy, recall, and F1 score; an efficiency evaluation module, for testing the performance of the target large language model based on a standard data set and / or a custom data set and The preset hardware configuration tests the processing speed, memory usage and energy efficiency of the target large language model; the robustness and multimodal evaluation module is used to test the robustness and stability of the target large language model, as well as the ability to process multimodal data based on an adversarial sample data set and a preset multimodal task verification model; the distributed task scheduling module is used to adjust task allocation based on the evaluation task objectives and the operating status of the evaluation system; the feedback optimization module is used to generate an evaluation report based on the test results of the basic performance evaluation module, the efficiency evaluation module and the robustness and multimodal evaluation module, and provide optimization suggestions for the target large language model.

[0007] In one implementation of the present application, the basic performance evaluation module includes: running the target large language model based on the standard data set to obtain model output data; comparing the model output data with the standard answer in the standard data set to calculate the accuracy, recall rate and F1 score of the target large language model; processing the accuracy, recall rate and F1 score of the target large language model based on a preset weight ratio to generate a basic performance evaluation report.

[0008] In one implementation of the present application, the efficiency evaluation module includes a performance testing tool; determining a hardware configuration based on a preset first number of CPUs, a preset second number of GPUs, and a preset maximum memory number; running the target large language model based on the hardware configuration to process the standard data set; testing the inference time of the target large language model in the hardware configuration based on the performance testing tool; obtaining the energy consumption of the hardware configuration based on an interface in the hardware configuration; and recording the inference time and energy consumption under the hardware configuration to generate an efficiency evaluation report.

[0009] In one implementation of the present application, the robustness and multimodal evaluation module includes: running the target large language model based on the adversarial sample data set to obtain an adversarial output; comparing the adversarial output with a standard answer in the adversarial sample data set to calculate the error rate of the target large language model; running the target large language model based on the multimodal task to obtain a multimodal output; wherein the multimodal task includes at least image text data and file text data; comparing the multimodal output with the standard answer in the multimodal output to calculate the cross-modal capability of the target large language model; and outputting the error rate and cross-modal capability as a robustness and multimodal evaluation report.

[0010] In one implementation of the present application, the distributed task scheduling module includes: a task distribution submodule, which is used to distribute the evaluation task objectives to the basic performance evaluation module, the efficiency evaluation module, and the robustness and multimodal evaluation module; a task priority scheduling submodule, which is used to determine the task priorities of basic performance testing, efficiency testing, robustness testing, and multi-domain adaptability testing according to the priorities in the evaluation task objectives; a fault-tolerant submodule, which is used to restart the task distribution submodule when an error occurs in the basic performance evaluation module and / or the efficiency evaluation module, the robustness and / or the multimodal evaluation module; and a resource elastic scaling submodule, which is used to determine the allocation of resources according to the operating status of the evaluation system and the task priority.

[0011] In one implementation of the present application, the resource elastic scaling submodule includes: real-time monitoring of the resource usage status of the evaluation system; wherein the resource usage status includes the usage status of computing resources, the usage status of storage resources, and the usage status of network resources; adjusting resource allocation based on the resource usage status; wherein the resource allocation includes increasing computing resources, reducing computing resources, increasing storage resources, reducing storage resources, increasing network resources, and reducing network resources.

[0012] In one implementation of the present application, the feedback optimization module includes: a result summary submodule, which is used to associate the output results of the basic performance evaluation module, the efficiency evaluation module, and the robustness and multimodal evaluation module to generate an evaluation report; a task monitoring submodule, which is used to monitor the status of the evaluation system in real time; and an optimization suggestion generation submodule, which is used to generate optimization suggestions based on the evaluation report.

[0013] In the second aspect, an embodiment of the present application also provides a large language model evaluation method, which is applied to the above-mentioned large language model evaluation system, and the method includes: obtaining a standard data set, a custom data set, and an adversarial sample data set related to the target large language model, and determining the evaluation task objectives; the evaluation task objectives include: basic performance test, efficiency test, robustness test, and multi-domain adaptability test; allocating the evaluation task objectives to the basic performance evaluation module, efficiency evaluation module, and robustness and multimodal evaluation module; determining the task priorities of the basic performance test, efficiency test, robustness test, and multi-domain adaptability test according to the priorities in the evaluation task objectives; When an error occurs in the basic performance evaluation module and / or the efficiency evaluation module, the robustness and / or the multimodal evaluation module, restart the task distribution submodule; determine the allocation of resources according to the operating status of the evaluation system and the task priority; the basic performance evaluation module, the efficiency evaluation module, and the robustness and multimodal evaluation module execute the evaluation task objectives to output a basic performance evaluation report, an efficiency evaluation report, and a robustness and multimodal evaluation report; associate the output basic performance evaluation report, the efficiency evaluation report, and the robustness and multimodal evaluation report to generate an evaluation report; generate optimization suggestions based on the evaluation report; and output a comprehensive evaluation report based on the evaluation report and the optimization suggestions.

[0014] In one implementation of the present application, the basic performance evaluation module, the efficiency evaluation module, and the robustness and multimodal evaluation module execute the evaluation task objectives to output a basic performance evaluation report, an efficiency evaluation report, and a robustness and multimodal evaluation report, specifically including: testing the performance of the target large language model based on the standard data set and / or the custom data set to output a basic performance evaluation report; wherein the performance indicators include at least accuracy, recall rate, and F1 score; testing the processing speed, memory occupancy, and energy efficiency of the target large language model based on a preset hardware configuration, a standard data set, and / or a custom data set to evaluate the operating performance of the target large language model in different environments to output an efficiency evaluation report; testing the robustness and stability of the target large language model based on an adversarial sample data set and a preset multimodal task verification model, as well as the ability to process multimodal data, to output a robustness and multimodal evaluation report.

[0015] In one implementation of the present application, generating optimization suggestions based on the evaluation report specifically includes: retrieving the evaluation report and comparing the evaluated report with a preset optimization threshold to determine the data to be optimized; and comparing the suggestion database based on the data to be optimized to generate optimization suggestions.

[0016] The large language model evaluation system and method provided in the embodiments of the present application have at least the following technical effects: By integrating the data preparation module, basic performance evaluation module, efficiency evaluation module, robustness and multimodal evaluation module, a relatively comprehensive evaluation of the large language model is achieved. The evaluation content covers the accuracy, recall rate, F1 score, processing speed, memory usage, energy efficiency, robustness, and stability of the target large language model based on multimodal processing capabilities. It can evaluate the large language model in multiple dimensions, and provide a comprehensive evaluation result to a certain extent.

[0017] Through the distributed task scheduling module, the task allocation can be dynamically adjusted according to the evaluation task objectives and the operating status of the evaluation system to ensure the efficient execution of the evaluation tasks to a certain extent. At the same time, the evaluation modules work together to improve the evaluation efficiency to a certain extent.

[0018] The resource elastic scaling submodule can dynamically adjust resource allocation according to the operating status of the evaluation system and the task priority, to ensure that the evaluation task is executed efficiently to a certain extent when resources are sufficient. The setting of this submodule not only improves resource utilization, but also reduces evaluation costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A schematic diagram of a large language model evaluation system provided in an embodiment of the present application; Figure 2 A flow chart of a large language model evaluation method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0021] The embodiments of the present application provide a large language model evaluation system and method to solve the following technical problem: how to evaluate a large language model in a multi-dimensional, efficient and automated manner.

[0022] The technical solution proposed in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0023] Figure 1 A schematic diagram of a large language model evaluation system provided in an embodiment of the present application.

[0024] like Figure 1 As shown, a large language model evaluation system provided in an embodiment of the present application includes: a data preparation module, a basic performance evaluation module, an efficiency evaluation module, a robustness and multimodal evaluation module, a distributed task scheduling module and a feedback optimization module.

[0025] The data preparation module is used to obtain standard data sets, custom data sets, and adversarial sample data sets related to the target large language model, and determine the evaluation task objectives; the evaluation task objectives include: basic performance testing, efficiency testing, robustness testing, and multi-domain adaptability testing.

[0026] First, we need to determine the target large language model that needs to be evaluated.

[0027] Furthermore, a standard dataset, a custom dataset, and an adversarial sample dataset are determined according to the type of the target large language model.

[0028] Furthermore, the evaluation task objectives are determined according to the type of the target large language model; the evaluation task objectives include: basic performance testing, efficiency testing, robustness testing, and multi-domain adaptability testing.

[0029] The basic performance evaluation module is used to test the performance of the target large language model based on a standard dataset; performance indicators include accuracy, recall, and F1 score.

[0030] First, run the target large language model based on the standard dataset to obtain model output data.

[0031] Furthermore, the model output data is compared with the standard answers in the standard dataset to calculate the accuracy, recall and F1 score of the target large language model.

[0032] Furthermore, the accuracy, recall, and F1 score of the target large language model are processed based on a preset weight ratio to generate a basic performance evaluation report.

[0033] The efficiency evaluation module is used to test the processing speed, memory usage and energy efficiency of the target large language model based on a standard data set and multiple preset hardware configurations, so as to evaluate the operating performance of the target large language model in different environments; among them, the efficiency evaluation module includes a performance testing tool.

[0034] First, any hardware configuration is selected from a variety of hardware configurations, and a target large language model is run based on any hardware configuration to process a standard dataset.

[0035] Furthermore, the inference time of the target large language model in any hardware configuration is tested based on the performance testing tool.

[0036] Furthermore, the energy consumption of any hardware configuration is obtained based on the interface in any hardware configuration.

[0037] Further, the above steps are repeated until various hardware configurations are tested.

[0038] Furthermore, the inference time and energy consumption under various hardware configurations are compared to generate an efficiency evaluation report.

[0039] The robustness and multimodal evaluation module is used to test the robustness and stability of the target large language model and its ability to process multimodal data based on adversarial sample datasets and preset multimodal task verification models.

[0040] First, the target large language model is run on the adversarial sample dataset to obtain adversarial output.

[0041] Furthermore, the adversarial output is compared with the standard answer in the adversarial sample dataset to calculate the error rate of the target large language model.

[0042] Furthermore, the target large language model is run based on the multimodal task to obtain a multimodal output; wherein the multimodal task includes at least image text data and file text data.

[0043] Furthermore, the multimodal output is compared with the standard answer in the multimodal output to calculate the cross-modal capability of the target large language model.

[0044] Furthermore, the error rate and cross-modal capability are output as robustness and multimodal evaluation reports.

[0045] The distributed task scheduling module is used to adjust the task allocation based on the evaluation task objectives and the operating status of the evaluation system. The distributed task scheduling module includes a task distribution submodule, a task priority scheduling submodule, a fault tolerance submodule, and a resource elastic scaling submodule.

[0046] The task distribution submodule is used to distribute the evaluation task objectives to the basic performance evaluation module, the efficiency evaluation module, and the robustness and multimodal evaluation module.

[0047] The task priority scheduling submodule is used to determine the task priorities of basic performance testing, efficiency testing, robustness testing, and multi-domain adaptability testing according to the priorities in the evaluation task objectives.

[0048] The fault-tolerant submodule is used to restart the task distribution submodule when an error occurs in the basic performance evaluation module and / or the robustness of the efficiency evaluation module and / or the multimodal evaluation module.

[0049] The resource elastic scaling submodule is used to determine the allocation of resources based on the operating status and task priority of the evaluation system.

[0050] First, the resource usage status of the evaluation system is monitored in real time; the resource usage status includes the usage status of computing resources, storage resources and network resources.

[0051] Furthermore, based on the resource usage status of each module and combined with the Kubernetes (K8s) elastic resource scaling mechanism, the basic performance evaluation module, efficiency evaluation module, and robustness and multimodal evaluation module can be automatically adjusted and scaled to dynamically optimize the resource allocation of the evaluation system. Furthermore, resource allocation is adjusted based on resource usage and resource demand; wherein resource allocation includes increasing computing resources, decreasing computing resources, increasing storage resources, decreasing storage resources, increasing network resources, and decreasing network resources.

[0052] The feedback optimization module is used to generate an evaluation report based on the test results of the basic performance evaluation module, the efficiency evaluation module, and the robustness and multimodal evaluation module, and provide optimization suggestions for the target large language model. The feedback optimization module includes a result summary submodule, a task monitoring submodule, and an optimization suggestion generation submodule.

[0053] The result summary submodule is used to associate the output results of the basic performance evaluation module, the efficiency evaluation module, and the robustness and multimodal evaluation module to generate an evaluation report.

[0054] The task monitoring submodule is used to monitor the status of the evaluation system in real time.

[0055] The optimization suggestion generation submodule is used to generate optimization suggestions based on the evaluation report.

[0056] The above is an embodiment of the method proposed in this application. Based on the same inventive concept, this application embodiment also provides a large language model evaluation method, the method flow chart is as follows Figure 2 shown.

[0057] Figure 2 A flow chart of a large language model evaluation method provided in an embodiment of the present application. Figure 2 As shown, the method includes: Step 1: Obtain standard data sets, custom data sets, and adversarial sample data sets for the target large language model, and determine the evaluation task objectives; the evaluation task objectives include: basic performance testing, efficiency testing, robustness testing, and multi-domain adaptability testing.

[0058] Step 11: Determine the target large language model to be evaluated.

[0059] First, we need to determine the large language model that needs to be evaluated, that is, the target large language model.

[0060] For example, if the reasoning ability of a large medical model needs to be evaluated, the large medical model is determined as the target large language model.

[0061] Step 12: Determine the standard dataset, custom dataset, and adversarial sample dataset based on the type of the target large language model.

[0062] Standard datasets: datasets with labels. For example, standard datasets for disciplines such as C-Eval, MMLU, and ARC; standard datasets for mathematics such as GSM8K; standard datasets for reasoning such as HellaSwag and BBH; all of the above standard datasets can be downloaded from the Internet.

[0063] Custom dataset: A dataset built according to user needs. For example, if a user wants to know the reasoning ability of the target large language model for financial data, a set of financial data can be selected as the standard dataset.

[0064] Adversarial sample dataset: a dataset used to test the robustness of a target large language model. In the embodiment of the present application, adversarial samples with spelling errors, malicious input or semantic deviation are generated by the data generation tool TextAttack.

[0065] In the embodiment of the present application, the types of medical big models are reasoning, disciplines, and mathematics. Therefore, C-Eval, GSM8K, and BBH are selected as standard data sets; Generate adversarial samples with spelling errors, malicious input, or semantic deviation through the data generation tool TextAttack; The user wants to know the level of medication dispensing of the medical model for patients of different weights, so he chooses the past medication dispensing data as the standard data set, and uses the data generation tool TextAttack to generate misspelled medication dispensing data as the adversarial sample data set. It can be understood that a custom data set can be used as both a standard data set and an adversarial sample data set, and the specific situation is determined by the data provider.

[0066] Step 13: Determine the evaluation task objectives according to the type of the target large language model; the evaluation task objectives include: basic performance testing, efficiency testing, robustness testing, and multi-domain adaptability testing.

[0067] Basic performance test: Evaluate the performance of the target large language model on benchmark datasets or custom datasets, such as the accuracy, recall, and F1 score on C-Eval and GSM8K.

[0068] Efficiency test: Evaluate the efficiency of the target large language model in terms of computing resources and time costs, such as inference speed and memory usage.

[0069] Robustness testing: Evaluate the stability and performance of the target large language model in the face of changes, noise, or attacks in the input data.

[0070] Multi-domain adaptability testing: Evaluate the performance of the target large language model in different domains or specific domains to determine how well it has mastered domain-specific knowledge.

[0071] In the present application example, the types of the medical big model are medicine (discipline), mathematics, and reasoning, and the medical big model can only process text data, so the evaluation task objectives of the medical big model include basic performance testing and efficiency testing.

[0072] It is understandable that the target large language model usually requires basic performance testing, efficiency testing, robustness testing, and multi-domain adaptability testing.

[0073] Step 2: Assign the evaluation task objectives to the basic performance evaluation module, efficiency evaluation module, and robustness and multimodal evaluation module.

[0074] Basic performance evaluation module: used to test the performance of the target large language model against standard datasets and / or custom datasets; such as accuracy, recall rate, and F1 score.

[0075] Efficiency evaluation module: used to evaluate the efficiency of the target large language model in terms of computing resources and time costs, such as inference speed and memory usage.

[0076] Robustness and multimodal evaluation module: used to evaluate the stability and performance of the target large language model in the face of changes, noise or attacks in the input data; and to evaluate the performance of the target large language model in different fields or specific fields to determine its mastery of specific field knowledge.

[0077] In the present application example, by determining the evaluation task objectives, the corresponding task objectives are allocated to the basic performance evaluation module, the efficiency evaluation module, and the robustness and multimodal evaluation module.

[0078] Step 3: Determine the task priorities of basic performance testing, efficiency testing, robustness testing, and multi-domain adaptability testing based on the priorities in the assessment task objectives.

[0079] This priority is set manually and will not be described in detail here.

[0080] Step 4: When an error occurs in the basic performance evaluation module and / or the efficiency evaluation module robustness and / or the multimodal evaluation module, restart the task distribution submodule.

[0081] When evaluating the target large language model, it is crucial to ensure the stable operation of each evaluation module. When errors occur in the basic performance evaluation module, efficiency evaluation module, robustness and multimodal evaluation module, the final generated comprehensive evaluation report will be inaccurate, so it is necessary to restart the task distribution submodule to ensure the continuity and accuracy of the evaluation task.

[0082] First, the evaluation system tracks the operating status of the basic performance evaluation module, efficiency evaluation module, robustness and multimodal evaluation module in real time through log records.

[0083] Furthermore, when an abnormal condition occurs in any module, it is determined that an error occurs in the module; the abnormal condition includes but is not limited to: no response for a long time, crash.

[0084] Furthermore, the current evaluation task is stopped, and the evaluation data of the basic performance evaluation module, efficiency evaluation module, robustness and multimodal evaluation module are cleaned up.

[0085] Furthermore, tasks are redistributed through the task distribution submodule.

[0086] Step 5: Determine resource allocation based on the operating status of the evaluation system and task priority.

[0087] Step 51: monitor and evaluate the resource usage of the system in real time; wherein the resource usage includes the computing resource usage, storage resource usage and network resource usage.

[0088] Through the API provided by the operating system (the device running the evaluation system), the utilization and load of the operating system's CPU and GPU computing resources are monitored in real time.

[0089] Monitor the operating system disk space usage and read and write speeds through the database management system.

[0090] Step 52: Adjust resource allocation based on resource usage; In the embodiment of the present application, the containerized deployment and automatic scaling mechanism of Kubernetes is adopted to automatically adjust computing resources according to actual task requirements.

[0091] Among them, resource allocation includes increasing computing resources, reducing computing resources, increasing storage resources, reducing storage resources, increasing network resources, and reducing network resources.

[0092] For example, Kubernetes automatically monitors resources and collects indicators. The collected indicators (i.e., resource usage) mainly include CPU usage, memory usage, disk usage, and network traffic. If the CPU load of the Pod of the evaluation task is too high, a new Pod copy will be automatically created to share the load. If the memory demand of the Pod exceeds the predetermined resources, the resource request of the Pod will be automatically adjusted.

[0093] Step 6: The basic performance evaluation module, the efficiency evaluation module, and the robustness and multimodal evaluation module execute the evaluation task objectives to output a basic performance evaluation report, an efficiency evaluation report, and a robustness and multimodal evaluation report.

[0094] Step 61: Test the performance of the target large language model based on a standard dataset and / or a custom dataset; wherein the performance indicators include at least accuracy, recall, and F1 score.

[0095] Step 611: Run the target large language model based on the standard dataset and / or the custom dataset to obtain model output data.

[0096] The standard dataset is input into the target large language model, and the target large language model processes the standard dataset to generate model output data.

[0097] Step 612: Compare the model output data with the standard answers in the standard data set to calculate the precision, recall and F1 score of the target large language model.

[0098] Accuracy: measures the ratio of the number of samples correctly predicted by the target large language model to the total number of samples.

[0099] Recall rate: measures the ratio of the number of samples correctly predicted as positive by the target large language model to the number of samples that are actually positive.

[0100] F1 score: The harmonic mean of precision and recall, used to comprehensively evaluate the performance of the target large language model.

[0101] By comparing the model output data with the standard answers in the standard dataset, the accuracy, recall and F1 score can be determined.

[0102] Step 613: Process the accuracy, recall, and F1 score of the target large language model based on a preset weight ratio to generate a basic performance evaluation report.

[0103] The weight ratio is set manually, and different large language models have different requirements. For example, in a large medical model, accuracy is crucial, and its weight ratio should be set higher.

[0104] In this application example, the MNLI text implication recognition task in the GLUE basic test set is selected as a custom dataset. This dataset contains a training set, a validation set, and a test set. The standard answers of the test set are not made public and are only used to submit to the GLUE official website for performance evaluation.

[0105] Use the target large language model to process the validation set in the GLUE basic test set. The target large language model will process these data and generate corresponding model output data.

[0106] Download the standard answers of the validation set from the GLUE official website and align them with the model output data. Use scikit-learn to calculate the accuracy, recall, and F1 score of the target large language model.

[0107] According to the specific application scenario and requirements, set appropriate weight ratios for accuracy, recall, and F1 score (such as 0.5 for accuracy, 0.3 for recall, and 0.2 for F1 score). The default weight ratios are: 0.33 for accuracy, 0.33 for recall, and 0.33 for F1 score. Use these weights to weight the model performance indicators and calculate the weighted performance indicators.

[0108] Based on the weighted performance indicators, a basic performance evaluation report is generated, which lists the original value, weighted value and weighted performance indicator of each performance indicator.

[0109] Furthermore, the weight ratios of precision, recall, and F1 score are initialized as follows: Weight ratio setting: accuracy, recall, F1 score; Image recognition: 0.3, 0.5, 0.2; Recommender system, 0.3, 0.5, 0.2; Sentiment analysis, 0.5, 0.3, 0.2; medical diagnosis, 0.3, 0.5, 0.2; Default value, 0.33, 0.33, 0.33; The weight ratio needs to be adjusted according to specific goals and application risks. For example, in the medical and security fields, the recall rate is usually more important because missed diagnosis or missed detection may have serious consequences; while in fields such as recommendation systems and sentiment analysis, accuracy is usually more important. The F1 score plays an important role in scenarios where precision and recall need to be balanced.

[0110] This application improves the model evaluation efficiency and reliability by presetting the weight ratios of performance indicators according to different scenarios.

[0111] Step 62: Test the processing speed, memory usage, and energy efficiency of the target large language model based on a preset hardware configuration, a standard data set, and / or a custom data set to evaluate the operating performance of the target large language model in different environments; wherein the efficiency evaluation module includes a performance testing tool.

[0112] Hardware configuration refers to the specific configuration of a computer system, including processor type, memory size, and storage device type.

[0113] The performance testing tool can detect the response time of the target large language model, and the performance testing tool used in the embodiment of the present application is PerfTools. It is understandable that the performance testing tool is not limited.

[0114] Step 621: Determine a hardware configuration based on a preset first number of CPUs, a preset second number of GPUs, and a preset maximum memory amount.

[0115] Preset first number of CPUs: refers to the number of central processing units (CPUs) preset before testing.

[0116] Preset second number of GPUs: refers to the number of graphics processors (GPUs) that is preset before testing.

[0117] Preset maximum memory number: refers to the maximum memory capacity that the test system can provide.

[0118] In another specific example, a 4-core CPU, 2 GPUs, and 64GB of maximum memory are selected as the hardware configuration of the test environment. These parameters are selected based on the configuration level of current mainstream servers and are intended to simulate typical situations in actual deployment environments.

[0119] Step 622: Run the target large language model based on the hardware configuration to process the standard data set.

[0120] Load the target large language model on the hardware configuration and prepare standard datasets and / or custom datasets as input.

[0121] Further, the target large language model is started to process the standard data set and / or the custom data set, and the running status and resource usage of the target large language model are monitored.

[0122] Step 623: Test the inference time of the target large language model in the hardware configuration based on the performance testing tool.

[0123] Inference time refers to the time required for the target large language model to receive input data and produce output results.

[0124] First, configure the performance testing tool to the hardware configuration.

[0125] Furthermore, when the model processes the standard data set, a performance testing tool is used to record the inference time it takes for the model to complete each task or batch of data.

[0126] Step 624: Obtain energy consumption of the hardware configuration based on the interface in the hardware configuration.

[0127] Energy consumption refers to the electrical energy consumed by a computer system during operation.

[0128] In the embodiment of the present application, the energy consumption can be obtained through NVIDIA's nvidia-smi interface and Intel's RAPL interface.

[0129] Step 625: Record the inference time and energy consumption under the hardware configuration to generate an efficiency evaluation report.

[0130] According to the inference time and energy consumption, write an efficiency evaluation report, which should include the test purpose, test method, test results, performance comparison and conclusion.

[0131] Step 63: Test the robustness and stability of the target large language model and its ability to process multimodal data based on the adversarial sample dataset and / or custom dataset, and the preset multimodal task verification model.

[0132] Step 631: Run the target large language model based on the adversarial sample dataset to obtain adversarial output.

[0133] The adversarial sample dataset is input into the target large language model, and the target large language model processes the adversarial sample dataset to generate adversarial output.

[0134] Step 632: Compare the adversarial output with the standard answer in the adversarial sample dataset to calculate the error rate of the target large language model.

[0135] The error rate is the ratio of the number of samples with incorrect output to the total number of samples.

[0136] The adversarial output obtained in step 631 is compared one by one with the standard answers in the adversarial sample dataset to calculate the error rate of the target large language model.

[0137] Step 633: Run the target large language model based on the multimodal task to obtain a multimodal output; wherein the multimodal task includes at least image text data and file text data.

[0138] Multimodal tasks refer to processing tasks that involve multiple types of data (image-text, text-file) at the same time.

[0139] The multimodal tasks are input into the target large language model, and the target large language model processes the multimodal tasks to generate multimodal outputs.

[0140] Step 634: Compare the multimodal output with the standard answer in the multimodal output to calculate the cross-modal capability of the target large language model.

[0141] The multimodal output obtained in step 633 is compared with the standard answers in the multimodal tasks one by one to evaluate the performance of the target large language model on various multimodal tasks.

[0142] The performance of the target large language model on various multimodal tasks is evaluated in the same manner as steps 612 to 613, and is reflected by accuracy, recall, and F1 score, which will not be elaborated here.

[0143] Step 635: Output the error rate and cross-modal capability as a robustness and multimodal evaluation report.

[0144] The error rate calculated in step 632 and the performance on various multimodal tasks in step 634 are summarized to generate a robustness and multimodal evaluation report.

[0145] Step 7: Associate and output the basic performance evaluation report, the efficiency evaluation report, and the robustness and multimodal evaluation report to generate an evaluation report.

[0146] Step 8: Generate optimization suggestions based on the evaluation report.

[0147] Step 81: retrieve the evaluation report, and compare the evaluation report with a preset optimization threshold to determine the data to be optimized.

[0148] The evaluation report includes basic performance evaluation report, efficiency evaluation report, and robustness and multimodal evaluation report; The basic performance evaluation report includes precision, recall and F1 score.

[0149] The efficiency evaluation report includes inference time, memory usage, and energy consumption.

[0150] Robustness and multimodal evaluation reports include adversarial example error rate and cross-modal capabilities.

[0151] The optimization thresholds include accuracy threshold, recall threshold, F1 score threshold, inference time threshold, memory usage threshold, energy consumption threshold, error rate threshold, and cross-modal capability threshold.

[0152] For example, the accuracy threshold is set to 0.5, and the accuracy of the target large language model is 0.32, which is lower than the preset accuracy threshold. In this case, the accuracy is data to be optimized.

[0153] Step 82: Compare the data to be optimized with the suggestion database to generate optimization suggestions.

[0154] The suggestion database is a database set with data to be optimized and optimization suggestions associated with the data to be optimized.

[0155] In the embodiments of the present application: For accuracy, recall, and F1 score, if one of the indicators performs poorly, optimization suggestions include: hyperparameter optimization, model architecture adjustment, or data enhancement. For example, if the recall rate is low, it is recommended to increase the diversity of training data or apply enhancement techniques (such as data augmentation or transfer learning) to improve the recall ability of the model.

[0156] If the memory usage or inference speed is slow, it is recommended to use model compression methods to reduce the computational complexity of the model.

[0157] If the model performs poorly on adversarial samples, it is recommended to use techniques such as adversarial training and adversarial sample enhancement to improve the robustness of the model.

[0158] If the performance in multimodal tasks is poor, it is recommended to enhance the multimodal features of the dataset or apply a cross-modal learning strategy.

[0159] Step 9: Output a comprehensive evaluation report based on the evaluation report and optimization suggestions.

[0160] Integrate the assessment report with optimization suggestions to generate a comprehensive assessment report.

Claims

1. A large language model evaluation system, characterized in that: The system comprises: A data preparation module is used to obtain standard data sets, custom data sets, and adversarial sample data sets related to the target large language model, and determine the evaluation task objectives; the evaluation task objectives include: basic performance testing, efficiency testing, robustness testing, and multi-domain adaptability testing; A basic performance evaluation module, used to test the performance of the target large language model based on a standard data set and / or a custom data set; wherein the performance indicators include accuracy, recall rate and F1 score; An efficiency evaluation module, used to test the processing speed, memory usage and energy efficiency of the target large language model based on a standard data set and / or a custom data set and a preset hardware configuration; A robustness and multimodal evaluation module, used to test the robustness and stability of the target large language model and its ability to process multimodal data based on an adversarial sample dataset and a preset multimodal task verification model; A distributed task scheduling module, used to adjust task allocation based on the evaluation task objectives and the operating status of the evaluation system; The feedback optimization module is used to generate an evaluation report based on the test results of the basic performance evaluation module, the efficiency evaluation module, and the robustness and multimodal evaluation module, and provide optimization suggestions for the target large language model.

2. A large language model evaluation system according to claim 1, characterized in that: The basic performance evaluation module includes: Running the target large language model based on the standard data set to obtain model output data; Comparing the model output data with the standard answers in the standard data set to calculate the accuracy, recall and F1 score of the target large language model; The accuracy, recall and F1 score of the target large language model are processed based on a preset weight ratio to generate a basic performance evaluation report.

3. A large language model evaluation system according to claim 1, characterized in that: The efficiency assessment module includes a performance testing tool; Determine the hardware configuration based on a preset first number of CPUs, a preset second number of GPUs, and a preset maximum memory amount; Running the target large language model based on the hardware configuration to process the standard data set; Testing the inference time of the target large language model in the hardware configuration based on the performance testing tool; Acquiring energy consumption of the hardware configuration based on an interface in the hardware configuration; The inference time and energy consumption under the hardware configuration are recorded to generate an efficiency evaluation report.

4. A large language model evaluation system according to claim 1, characterized in that: The robustness and multimodal evaluation module includes: Running the target large language model based on the adversarial sample dataset to obtain an adversarial output; Comparing the adversarial output with a standard answer in an adversarial sample dataset to calculate an error rate of the target large language model; Running the target large language model based on the multimodal task to obtain a multimodal output; wherein the multimodal task includes at least image text data and file text data; Comparing the multimodal output with a standard answer in the multimodal output to calculate the cross-modal capability of the target large language model; The error rate and cross-modal capability are output as a robustness and multimodal evaluation report.

5. A large language model evaluation system according to claim 1, characterized in that: The distributed task scheduling module includes: A task distribution submodule, used for distributing the evaluation task objectives to the basic performance evaluation module, the efficiency evaluation module and the robustness and multimodal evaluation module; A task priority scheduling submodule is used to determine the task priorities of basic performance testing, efficiency testing, robustness testing, and multi-domain adaptability testing according to the priorities in the evaluation task objectives; A fault-tolerant submodule, used for restarting the task distribution submodule when an error occurs in the basic performance evaluation module and / or the efficiency evaluation module robustness and / or the multimodal evaluation module; The resource elastic scaling submodule is used to determine the allocation of resources according to the operating status of the evaluation system and the task priority.

6. A large language model evaluation system according to claim 5, characterized in that: The resource elastic scaling submodule includes: Real-time monitoring of resource usage of the evaluation system; wherein the resource usage includes computing resource usage, storage resource usage, and network resource usage; Adjust resource allocation based on the resource usage status; wherein the resource allocation includes increasing computing resources, decreasing computing resources, increasing storage resources, decreasing storage resources, increasing network resources, and decreasing network resources.

7. A large language model evaluation system according to claim 1, characterized in that: The feedback optimization module comprises: A result summary submodule, used to associate the output results of the basic performance evaluation module, the efficiency evaluation module, and the robustness and multimodal evaluation module to generate an evaluation report; A task monitoring submodule, used for monitoring the status of the evaluation system in real time; The optimization suggestion generation submodule is used to generate optimization suggestions based on the evaluation report.

8. A large language model evaluation method, applied to a large language model evaluation system as claimed in claims 1-7, characterized in that: The method comprises: Obtain standard data sets, custom data sets, and adversarial sample data sets for the target large language model, and determine evaluation task objectives; the evaluation task objectives include: basic performance testing, efficiency testing, robustness testing, and multi-domain adaptability testing; Allocating the evaluation task objectives to the basic performance evaluation module, the efficiency evaluation module, and the robustness and multimodal evaluation module; Determine the task priorities of basic performance testing, efficiency testing, robustness testing, and multi-domain adaptability testing according to the priorities in the evaluation task objectives; When an error occurs in the basic performance evaluation module and / or the efficiency evaluation module robustness and / or the multimodal evaluation module, restart the task distribution submodule; Determining resource allocation according to the operating status of the evaluation system and the task priority; The basic performance evaluation module, the efficiency evaluation module, and the robustness and multimodal evaluation module execute the evaluation task objectives to output a basic performance evaluation report, an efficiency evaluation report, and a robustness and multimodal evaluation report; Associating the output basic performance evaluation report, the efficiency evaluation report, and the robustness and multimodal evaluation report to generate an evaluation report; generating optimization suggestions based on the evaluation report; A comprehensive evaluation report is output based on the evaluation report and the optimization suggestions.

9. A large language model evaluation method according to claim 8, characterized in that: The basic performance evaluation module, the efficiency evaluation module, and the robustness and multimodal evaluation module execute the evaluation task objectives to output a basic performance evaluation report, an efficiency evaluation report, and a robustness and multimodal evaluation report, specifically including: Testing the performance of the target large language model based on the standard dataset and / or the custom dataset to output a basic performance evaluation report; wherein the performance indicators include at least accuracy, recall, and F1 score; Test the processing speed, memory usage, and energy efficiency of the target large language model based on preset hardware configurations, standard data sets, and / or custom data sets to evaluate the performance of the target large language model in different environments and output an efficiency evaluation report; Based on the adversarial sample dataset and the preset multimodal task verification model, the robustness and stability of the target large language model and its ability to process multimodal data are tested to output a robustness and multimodal evaluation report.

10. A large language model evaluation method according to claim 8, characterized in that: Generate optimization suggestions based on the evaluation report, including: Retrieving the evaluation report, and comparing the evaluation report with a preset optimization threshold to determine the data to be optimized; The data to be optimized is compared with a suggestion database to generate optimization suggestions.

Images