Assessment system and method based on artificial intelligence large model

Abstract

The invention discloses an evaluation system and method based on an artificial intelligence large model, the system comprises a data set management module, a my model module and a model evaluation module, and the method comprises the following steps: S1, model management; s2, performing model evaluation; s3, performing online evaluation; and S4, evaluating the report. Compared with a static index, the method is more flexible, covers more complex scenes, aims at comprehensive evaluation of the professional field, and comprehensively evaluates a large model in the aspects of safety, functionality, accuracy and confusion; more edge scenes are covered, a large model often encounters various extreme or unseen inputs in practical application, automatic rules can effectively identify and perform necessary intervention, and referees can perform evaluation from a more humanized perspective, so that the defects of traditional evaluation are overcome; according to the method, the evaluation efficiency and expansibility can be improved, large-scale evaluation tasks are automatically processed according to rules, the evaluation process after model deployment is accelerated, and problems can be quickly executed and found in mass data.

Description

Technical Field

[0001] The present invention relates to the field of large model technology, and in particular to an evaluation system and method based on an artificial intelligence large model. Background Art

[0002] Model evaluation is a crucial step in the development of machine learning and artificial intelligence (AI) models. It aims to verify the model's performance across different datasets, scenarios, or tasks, and ensure that the model's output meets expectations. As large models such as GPT, BERT, and LLaMA become mainstream technologies, effectively and accurately evaluating their performance has become a key challenge.

[0003] Traditional evaluation methods often rely on static metrics, but these metrics are difficult to handle large-scale generation tasks or complex natural language processing scenarios. The following problems exist: 1. Performance issues in complex scenarios: Traditional static metrics such as accuracy and F1 score are often difficult to evaluate in complex scenarios. 2. Edge Case Problems: Large models often encounter extreme or unseen inputs in real-world applications. Traditional methods make it difficult to identify and intervene in these situations. 3. Evaluation efficiency and subjective bias issues: Manual evaluation is difficult to ensure consistency of evaluation standards and has low evaluation efficiency. The purpose of the present invention is to propose an evaluation system and method based on artificial intelligence large models to address the above-mentioned technical problems. Based on cutting-edge deep learning methods, it evaluates the ability of customized large models through complete evaluation data sets, evaluation indicators, and evaluation tasks, providing a basis for continued optimization and model application.

[0004] An evaluation system based on artificial intelligence large models, including a data set management module, a my model module, and a model evaluation module; The data set management module defines data types, stores data in a versioned manner, imports data incrementally, exports data, and publishes data sets; The My Model module manages the domain model generated by fine-tuning; The model evaluation module provides visual model evaluation and manual online evaluation functions, evaluates different domain models according to needs, selects different dataset versions, sets evaluation resource usage, and generates evaluation reports.

[0005] Furthermore, an evaluation system based on an artificial intelligence large model, the data types include Prompt+Response, Prompt+Multiple Responses, plain text, Prompt set, and Prompt-chosen-rejected.

[0006] Furthermore, an evaluation system based on an artificial intelligence large model, wherein the model evaluation module includes a data set selection module, a basic model selection module, a parameter setting module, a resource setting module, an evaluation method module, and a model evaluation report module; The data set selection module selects the data set defined by the data set management module; The basic model selection module selects the domain model defined by my model module; The parameter setting module adjusts the parameters of the model and checks the current GPU usage to cope with various different evaluations; The resource setting module sets the storage space used by the model to ensure that resources are controllable; The storage space includes CPU, memory, number of GPUs, and GPU memory; The evaluation mode module provides two evaluation modes: manual and automatic; The manual evaluation provides manual evaluation indicator settings and online evaluation functions, integrating the subjective opinions and experience of human experts to score the model responses from different evaluation dimensions to evaluate the effectiveness of the model responses; The automatic evaluation method includes automatic rules and referees; The model evaluation report graphically displays the evaluation results.

[0007] Furthermore, an evaluation system based on a large artificial intelligence model, wherein the evaluation method module includes a LlamaFactory model; The LlamaFactory model is a framework or tool set for evaluating and fine-tuning large language models, providing an evaluation and tuning mechanism; The LlamaFactory model includes multi-dimensional evaluation indicators, specific field evaluation, referee mechanism, and automatic rule system; The multi-dimensional evaluation indicators include standard static evaluation indicators and high-dimensional dynamic evaluation indicators; The static evaluation indicators include accuracy and F1 score; The dynamic evaluation indicators are evaluated in generation tasks and complex natural language processing scenarios, with dimensions such as contextual relevance, semantic consistency, and logic, to help developers understand the performance of the model in various tasks. The domain-specific evaluation allows users to define domain-specific evaluation criteria to ensure that the model generates accurate text content in a specific domain; The referee mechanism introduces human referees or expert knowledge-based systems to conduct manual evaluation, capturing the performance of the model in the generation task and judging the quality and consistency of the output; The automatic rule system monitors the output of the model in real time through predefined rules, and automatically intervenes according to set standards during the model generation process to ensure that the output meets the expected grammatical, logical and ethical standards.

[0008] An evaluation method based on an artificial intelligence large model includes the following sub-steps: S1: Model management; S11: Generate domain model after model fine-tuning; S12: Upload the fine-tuned domain model and select the corresponding base model; S13: Generate model after online code development; S2: Model evaluation; S3: Online assessment; S31: Manual evaluation mode, scoring the evaluation rules online based on the model's answers; S4: Evaluation report.

[0009] Furthermore, an evaluation method based on an artificial intelligence large model, said S2 comprises the following sub-steps: S21: Get the dataset version; S22: Get domain model; S23: Select the evaluation method; S24: Set evaluation parameters, model type, and evaluation method; S25: Set evaluation resources and storage space used by the model; S26: Define manual scoring evaluation rules; S27: Record the evaluation process log; S28: Generate evaluation data and store it in a shared file system for sharing in the distributed system.

[0010] Furthermore, in an evaluation method based on an artificial intelligence large model, S4 includes the following sub-steps: S41: Check the authenticity, satisfaction, average score, GoodCase ratio, and subjective evaluation indicators of manual evaluation; S42: View the accuracy, F1 score, recall rate after unigram splitting, recall rate after bigram splitting, recall rate calculated by measuring the longest common subsequence, and the weighted average of unigram, bigram, trigram, and 4-games of the automatic rule evaluation; view the referee score, referee score reason, score standard deviation, score average, and score median of the referee evaluation; S43: Export the evaluation results.

[0011] The beneficial effects of the present invention are as follows: through an evaluation system and method based on a large artificial intelligence model, it is more flexible than static indicators, covers more complex scenarios for all-round evaluation of professional fields, and comprehensively evaluates large models in terms of security, functionality, accuracy, and perplexity; covers more edge scenarios, and large models often encounter various extreme or unprecedented inputs in actual applications. Automatic rules can effectively identify and make necessary interventions, and referees can evaluate from a more humane perspective, thereby making up for the shortcomings of traditional evaluations; it can improve evaluation efficiency and scalability, and automatic rules can handle large-scale evaluation tasks, speed up the evaluation process after model deployment, and can quickly execute and discover problems in massive data. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a structural diagram of an evaluation system based on a large artificial intelligence model.

[0013] Figure 2 It is a flow chart of an evaluation method based on a large artificial intelligence model. DETAILED DESCRIPTION

[0014] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described with reference to the accompanying drawings.

[0015] As attached Figure 1 As shown, an evaluation system based on artificial intelligence large models includes a data set management module, a my model module, and a model evaluation module; The data set management module defines data types, stores data in a versioned manner, imports data incrementally, exports data, and publishes data sets; The My Model module manages the domain model generated by fine-tuning; The model evaluation module provides visual model evaluation and manual online evaluation functions, evaluates different domain models according to needs, selects different dataset versions, sets evaluation resource usage, and generates evaluation reports.

[0016] Furthermore, an evaluation system based on an artificial intelligence large model, the data types include Prompt+Response, Prompt+Multiple Responses, plain text, Prompt set, and Prompt-chosen-rejected.

[0017] Furthermore, an evaluation system based on an artificial intelligence large model, wherein the model evaluation module includes a data set selection module, a basic model selection module, a parameter setting module, a resource setting module, an evaluation method module, and a model evaluation report module; The data set selection module selects the data set defined by the data set management module; The basic model selection module selects the domain model defined by my model module; The parameter setting module adjusts the parameters of the model and checks the current GPU usage to cope with various different evaluations; The resource setting module sets the storage space used by the model to ensure that resources are controllable; The storage space includes CPU, memory, number of GPUs, and GPU memory; The evaluation mode module provides two evaluation modes: manual and automatic; The manual evaluation provides manual evaluation indicator settings and online evaluation functions, integrating the subjective opinions and experience of human experts to score the model responses from different evaluation dimensions to evaluate the effectiveness of the model responses; The automatic evaluation method includes automatic rules and referees; The model evaluation report graphically displays the evaluation results.

[0018] Furthermore, an evaluation system based on a large artificial intelligence model, wherein the evaluation method module includes a LlamaFactory model; The LlamaFactory model is a framework or tool set for evaluating and fine-tuning large language models, providing an evaluation and tuning mechanism; The LlamaFactory model includes multi-dimensional evaluation indicators, specific field evaluation, referee mechanism, and automatic rule system; The multi-dimensional evaluation indicators include standard static evaluation indicators and high-dimensional dynamic evaluation indicators; The static evaluation indicators include accuracy and F1 score; The dynamic evaluation indicators are evaluated in generation tasks and complex natural language processing scenarios, with dimensions such as contextual relevance, semantic consistency, and logic, to help developers understand the performance of the model in various tasks. The domain-specific evaluation allows users to define domain-specific evaluation criteria to ensure that the model generates accurate text content in a specific domain; The referee mechanism introduces human referees or expert knowledge-based systems to conduct manual evaluation, capturing the performance of the model in the generation task and judging the quality and consistency of the output; The automatic rule system monitors the output of the model in real time through predefined rules, and automatically intervenes according to set standards during the model generation process to ensure that the output meets the expected grammatical, logical and ethical standards.

[0019] As attached Figure 2 As shown, an evaluation method based on an artificial intelligence large model includes the following sub-steps: S1: Model management; S11: Generate domain model after model fine-tuning; S12: Upload the fine-tuned domain model and select the corresponding base model; S13: Generate model after online code development; S2: Model evaluation; S3: Online assessment; S31: Manual evaluation mode, scoring the evaluation rules online based on the model's answers; S4: Evaluation report.

[0020] Furthermore, an evaluation method based on an artificial intelligence large model, said S2 comprises the following sub-steps: S21: Get the dataset version; S22: Get domain model; S23: Select the evaluation method; S24: Set evaluation parameters, model type, and evaluation method; S25: Set evaluation resources and storage space used by the model; S26: Define manual scoring evaluation rules; S27: Record the evaluation process log; S28: Generate evaluation data and store it in a shared file system for sharing in the distributed system.

[0021] Furthermore, in an evaluation method based on an artificial intelligence large model, S4 includes the following sub-steps: S41: Check the authenticity, satisfaction, average score, GoodCase ratio, and subjective evaluation indicators of manual evaluation; S42: View the accuracy, F1 score, recall rate after unigram splitting, recall rate after bigram splitting, recall rate calculated by measuring the longest common subsequence, and the weighted average of unigram, bigram, trigram, and 4-games of the automatic rule evaluation; view the referee score, referee score reason, score standard deviation, score average, and score median of the referee evaluation; S43: Export the evaluation results.

[0022] Specific embodiment 1 The HAMI database that the data set management module relies on: HAMi, formerly known as k8s-vGPU-scheduler, is a middleware for managing heterogeneous devices in Kubernetes. It manages different types of heterogeneous devices, such as GPUs and NPUs, shares heterogeneous devices between Pods, and makes better scheduling decisions based on the device's topology information and scheduling policies. It aims to eliminate the gap between different heterogeneous devices and provide users with a unified management interface without changing applications.

[0023] The HAMI database includes device reuse capabilities and device resource isolation capabilities; The device reuse capability specifies the video memory to apply for computing power equipment or specifies the computing power usage ratio to apply for computing power equipment; The device resource isolation capability supports hard isolation of device resources.

[0024] Specific embodiment 2 The MongoDB database that the dataset management module relies on: MongoDB is an open source, document-based NoSQL database that uses a JSON-like format called BSON (Binary JSON) to store data. MongoDB is a non-relational NoSQL database and is particularly suitable for processing large amounts of unstructured or semi-structured data.

[0025] Here are some key features of MongoDB: Document storage: MongoDB stores data in the form of documents. Each document is a collection of key-value pairs, similar to a JSON object. This model is very suitable for storing hierarchical data and allows the flexibility of adding new fields without pre-defining the entire structure. Dynamic schema: Unlike traditional relational databases, MongoDB does not require a fixed table structure to be defined in advance. This means that different documents in the same collection (equivalent to a table in a relational database) can have completely different structures. High scalability: MongoDB supports horizontal scalability and can distribute data across multiple servers through sharding technology, thereby improving read and write performance and storage capacity; Rich query language: MongoDB provides powerful query capabilities, including complex aggregation operations and full-text search; Built-in replication and high availability: Data redundancy is achieved through replica sets, ensuring data security and high system availability; Support for multiple storage engines: Users can choose different storage engines according to their needs, such as WiredTiger, MMAPv1, etc.

[0026] Index support: To speed up queries, MongoDB allows you to create indexes for any field in a document, and even composite indexes for arrays and nested documents. Transaction support: Starting with version 4.0, MongoDB introduced multi-document transaction support, allowing atomic operations to be performed across multiple documents, collections, and even databases. Community and support: As an open source project, MongoDB has an active community that provides a wealth of resources and tools. There are also commercial versions that provide professional technical support and services.

[0027] Specific embodiment 3 The dataset management module relies on the MinIO object storage system: MinIO is a high-performance object storage system that is compatible with the Amazon S3 API, which means it can be seamlessly integrated with widely used cloud storage services. MinIO is designed for private cloud and edge computing environments, can handle large-scale data sets, and supports multiple data protection mechanisms such as Erasure Coding to ensure high availability and durability of data.

[0028] The main features are: High performance: MinIO can achieve very high throughput and low latency on standard hardware, making it particularly suitable for large-scale data applications that require fast read and write speeds, such as machine learning and big data analysis. Compatible: Fully compatible with the Amazon S3 API, which means that all tools, applications, and services that use S3 can be directly connected to MinIO. This compatibility makes it easy to migrate existing S3-based workloads.

[0029] Distributed architecture: MinIO can be deployed as a single node or a distributed multi-node cluster. In distributed mode, data redundancy is provided by erasure coding, ensuring that data is not lost even if multiple nodes fail.

[0030] Security: Supports TLS / SSL encrypted transmission to ensure data security during transmission, provides authentication and authorization mechanisms, including policy-based access control. Supports server-side encryption, which can encrypt data stored in MinIO.

[0031] Easy to deploy and manage: MinIO can be easily deployed in containerized environments such as Kubernetes. It provides command-line tools and a graphical interface for management and monitoring.

[0032] Cross-platform support: It supports multiple operating systems such as Linux, Windows, and macOS. It can also run on the ARM architecture and is suitable for edge computing scenarios.

[0033] Open Source: MinIO is open source software, released under the GNU Affero General Public License v3.0 (AGPLv3).

[0034] The open source community is active, with a large number of contributors and support resources.

[0035] Rich plug-ins and tools: The community provides many tools and plug-ins for operations such as backup, synchronization, and migration. It supports monitoring tools such as Prometheus and Grafana, facilitating performance monitoring and alarm settings.

[0036] Specific embodiment 4 The model evaluation module relies on Qwen1.5-14B-Chat: Qwen-1.5-14B-Chat is a version in the Qwen series of large-scale language models launched by Alibaba Cloud. It is optimized for conversation and chat scenarios. Based on the Transformer architecture, this model has 14 billion parameters and is designed to provide high-quality natural language understanding and generation capabilities, suitable for a variety of application scenarios.

[0037] It has the following features: Large-scale parameters: With 14 billion parameters, Qwen-1.5-14B-Chat can capture complex language patterns and subtle semantic differences. A large number of parameters helps improve the model's ability to understand and generate long texts.

[0038] Conversational optimization: The model is specially trained to handle multi-turn conversations and excels at maintaining contextual coherence, enabling it to understand and respond to user questions or statements while maintaining conversational fluency and relevance.

[0039] Breadth and depth of knowledge: The training data covers a wide range of subject areas, from general knowledge to professional knowledge. Through extensive pre-training and fine-tuning, the model is able to demonstrate a deep knowledge background on different topics.

[0040] Multi-language support: Qwen series models generally support multiple languages. Although they are mainly optimized for Chinese, they can also handle input and output of English and some other languages, which makes them more flexible and practical in international applications.

[0041] Security and Compliance: We adhere to strict data privacy protection principles, take security and compliance requirements into consideration during design, take measures to avoid generating harmful, false, or biased content, and strive to ensure the security and reliability of output.

[0042] This solution uses an evaluation system and method based on artificial intelligence large models. It is more flexible than static indicators, covers more complex scenarios for all-round evaluation of professional fields, and comprehensively evaluates large models in terms of security, functionality, accuracy, and perplexity; covers more edge scenarios. Large models often encounter various extreme or unprecedented inputs in actual applications. Automatic rules can effectively identify and make necessary interventions, and referees can evaluate from a more humane perspective, thus making up for the shortcomings of traditional evaluations; it can improve evaluation efficiency and scalability. Automatic rules handle large-scale evaluation tasks, speed up the evaluation process after model deployment, and can quickly execute and discover problems in massive data.

[0043] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. An evaluation system based on an artificial intelligence large model, characterized in that: Including dataset management module, my model module, and model evaluation module; The data set management module defines data types, stores data in a versioned manner, imports data incrementally, exports data, and publishes data sets; The My Model module manages the domain model generated by fine-tuning; The model evaluation module provides visual model evaluation and manual online evaluation functions, evaluates different domain models according to needs, selects different data set versions, sets evaluation resource usage, and generates evaluation reports.

2. The evaluation system based on artificial intelligence big model according to claim 1 is characterized in that: The data types include Prompt+Response, Prompt+Multiple Responses, Plain Text, Promp Set, and Prompt-chosen-rejected.

3. The evaluation system based on artificial intelligence big model according to claim 1 is characterized in that: The model evaluation module includes a data set selection module, a basic model selection module, a parameter setting module, a resource setting module, an evaluation method module, and a model evaluation report module; The data set selection module selects the data set defined by the data set management module; The basic model selection module selects the domain model defined by my model module; The parameter setting module adjusts the parameters of the model and checks the current GPU usage to cope with a variety of different evaluations; The resource setting module sets the storage space used by the model to ensure that the resources are controllable; The storage space includes CPU, memory, number of GPUs, and GPU memory; The evaluation mode module provides two evaluation modes: manual and automatic; The manual evaluation provides manual evaluation indicator settings and online evaluation functions, and scores the model responses from different evaluation dimensions based on the subjective opinions and experience of human experts to evaluate the effectiveness of the model responses; The automatic evaluation method includes automatic rules and referees; The model evaluation report graphically displays the evaluation results.

4. The evaluation system based on artificial intelligence big model according to claim 3 is characterized in that: The evaluation method module includes a LlamaFactory model; The LlamaFactory model is a framework or tool set for evaluating and fine-tuning large language models, providing an evaluation and tuning mechanism; The LlamaFactory model includes multi-dimensional evaluation indicators, specific field evaluation, referee mechanism, and automatic rule system; The multi-dimensional evaluation indicators include standard static evaluation indicators and high-dimensional dynamic evaluation indicators; The static evaluation indicators include accuracy and F1 score; The dynamic evaluation indicators are evaluations in generation tasks and complex natural language processing scenarios, with dimensions of context relevance, semantic consistency, and logic, to help developers understand the performance of the model in various tasks; The domain-specific evaluation allows users to define domain-specific evaluation criteria to ensure that the model generates accurate text content in a specific domain; The referee mechanism introduces human referees or expert knowledge-based systems for manual evaluation to capture the performance of the model in the generation task and judge the quality and consistency of the output; The automatic rule system monitors the output of the model in real time through predefined rules, and automatically intervenes in the model generation process according to the set standards to ensure that the output meets the expected grammatical, logical and ethical standards.

5. An evaluation method based on an artificial intelligence big model, implemented by an evaluation system based on an artificial intelligence big model according to any one of claims 1 to 4, characterized in that: It includes the following sub-steps: S1: Model management; S11: Generate domain model after fine-tuning the model; S12: Upload the fine-tuned domain model and select the corresponding basic model; S13: Generate model after online code development; S2: Model evaluation; S3: Online evaluation; S31: Manual evaluation mode, scoring the evaluation rules online based on the model’s answers; S4: Evaluation report.

6. The evaluation method based on artificial intelligence large model according to claim 5 is characterized in that: The S2 comprises the following sub-steps: S21: Get the dataset version; S22: Get domain model; S23: Select the evaluation method; S24: Set evaluation parameters, model type, and evaluation method; S25: Setting evaluation resources and setting the storage space used by the model; S26: define manual scoring evaluation rules; S27: Record the evaluation process log; S28: Generate evaluation data and store it in a shared file system for sharing in a distributed system.

7. The evaluation method based on artificial intelligence large model according to claim 5 is characterized in that: The S4 comprises the following sub-steps: S41: Check the authenticity, satisfaction, average score, GoodCase ratio, and subjective evaluation indicators of manual evaluation; S42: Check the accuracy of automatic rule evaluation, F1 score, recall rate after unigram splitting, recall rate after bigram splitting, recall rate calculated by measuring the longest common subsequence, and the weighted average of unigram, bigram, trigram, and 4-games; check the referee score, reason for the referee score, score standard deviation, score average, and score median of the referee evaluation; S43: Export the evaluation results.

Images