Model evaluation system, evaluation method, and related devices

Abstract

The model evaluation system, the evaluation method and the related equipment provided in the embodiments of the present application, wherein the model evaluation system comprises a plug-in layer, a module layer and a core layer connected in sequence; the plug-in layer is provided with corresponding evaluation model plug-ins based on different architectures; the core layer is provided with a plurality of executors, each of which corresponds to an evaluation type; when the module layer receives a target evaluation task and target configuration information, the module layer is used to acquire a target evaluation model plug-in corresponding to the target evaluation task from a plurality of evaluation model plug-ins of the plug-in layer, and send the target evaluation model plug-in to a target executor corresponding to the target evaluation task among a plurality of executors of the core layer; the core layer is used to call the target executor to perform task evaluation based on the target evaluation model plug-in and the target configuration information, and output a target evaluation result of the target evaluation task, so as to effectively improve the universality and compatibility of the model evaluation system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of model processing, in particular to a model evaluation system, an evaluation method and related equipment. BACKGROUND

[0002] With the release of ChatGPT, research related to large models and the development of downstream applications based on large models have become a global hotspot, and the evaluation of the capabilities of large models is one of them. In recent years, well-known large model evaluation datasets such as logical ability evaluation gsm8k, mathematical ability evaluation math, English general ability mmlu, Chinese general ability cmmlu, and code ability humaneval have appeared. When evaluating a target model, the evaluation system downloads the evaluation dataset corresponding to the target model to perform model evaluation. However, due to the architectural limitations of existing evaluation systems, they can only evaluate large models that support the hardware architecture and software architecture, and cannot evaluate large models of other architectures, which limits the model evaluation of the evaluation system and reduces its versatility. SUMMARY

[0003] The embodiments of the present application provide a model evaluation system, an evaluation method and related equipment, which can improve the versatility and compatibility of the model evaluation system.

[0004] To achieve the above-mentioned purpose, the first aspect of the embodiments of the present application proposes a model evaluation system, which comprises a plug-in layer, a module layer and a core layer connected in sequence;

[0005] The plug-in layer is provided with corresponding evaluation model plug-ins based on different architectures;

[0006] The core layer is provided with a plurality of executors, each of which corresponds to an evaluation type;

[0007] When the module layer receives a target evaluation task and target configuration information, the module layer is configured to acquire a target evaluation model plug-in corresponding to the target evaluation task from a plurality of evaluation model plug-ins of the plug-in layer, and send the target evaluation model plug-in to a target executor corresponding to the target evaluation task in a plurality of executors of the core layer;

[0008] The core layer is configured to call the target executor to perform task evaluation based on the target evaluation model plug-in and the target configuration information, and output a target evaluation result of the target evaluation task.

[0009] In some embodiments, the architecture includes a hardware architecture and a software architecture, and the evaluation model plug-in includes a plurality of model plug-ins, wherein the generation process of the plurality of model plug-ins includes the following steps:

[0010] generate at least one target hardware evaluation model plug-in applied to the target hardware architecture and a target hardware calling interface associated with the target hardware evaluation model plug-in, by taking each of the hardware architectures as a target hardware architecture one by one;

[0011] generate at least one target software evaluation model plug-in applied to the target software architecture and a target software calling interface associated with the target software evaluation model plug-in, by taking each of the software architectures as a target software architecture one by one;

[0012] obtain the plurality of model plug-ins based on all the target hardware evaluation model plug-ins and corresponding target hardware calling interfaces and all the target software evaluation model plug-ins and corresponding target software calling interfaces.

[0013] In some embodiments, the evaluation model plug-in includes a plurality of evaluation data sets and a plurality of evaluation methods.

[0014] Among at least two of the evaluation data sets, the data types are different.

[0015] The evaluation methods are used to convert the result data output by the evaluation model plug-in into corresponding evaluation scores, and the conversion processes of at least two of the evaluation methods are different.

[0016] In some embodiments, the generation process of the executor includes the following steps:

[0017] Take each of the evaluation types as a target evaluation type one by one.

[0018] Determine the number of threads corresponding to the target evaluation type, and generate at least one of the executors corresponding to the number of threads and the target evaluation type.

[0019] In some embodiments, the module layer includes a configuration management component, a plug-in management component, and a storage management component.

[0020] The configuration management component is configured to read command line information of the target evaluation task to determine a data set identifier, a model identifier, an evaluation method identifier, and a target task type, and send the data set identifier, the model identifier, the evaluation method identifier, and the target task type to the plug-in management component.

[0021] The plug-in management component is configured to obtain and load the target evaluation model plug-in from the plug-in layer according to the data set identifier, the model identifier, and the evaluation method identifier, and is further configured to call the target executor from the core layer according to the target task type.

[0022] The storage management component is configured to store evaluation data required for the evaluation task.

[0023] To achieve the above object, a second aspect of the embodiment of the application provides an evaluation method of a model evaluation system, the method comprising:

[0024] In response to an evaluation request of the target evaluation task, target configuration information of the target evaluation task is obtained;

[0025] A target evaluation model plug-in and a target executor corresponding to the target evaluation task are determined from a plurality of evaluation model plug-ins and a plurality of executors, at least two of the evaluation model plug-ins corresponding to different architectures, and each of the executors corresponding to an evaluation type;

[0026] The target evaluation model plug-in is configured based on the target configuration information, and the target evaluation model plug-in is evaluated based on the target executor, to obtain a target evaluation result of the target evaluation task.

[0027] In some embodiments, the target evaluation model plug-in comprises a target data set, a target model plug-in, and a target evaluation method, the target evaluation model plug-in and the target executor corresponding to the target evaluation task are determined by:

[0028] Command line information and a target task type corresponding to the target evaluation task are obtained;

[0029] A target data set is determined from a plurality of evaluation data sets based on a data set identifier of the command line information, a target model plug-in is determined from a plurality of evaluation model plug-ins based on a model identifier of the command line information, and a target evaluation method is determined from a plurality of evaluation methods based on an evaluation method identifier of the command line information;

[0030] The target executor is determined from a plurality of executors based on the target task type.

[0031] In some embodiments, before the target evaluation model plug-in and the target executor corresponding to the target evaluation task are determined, the method further comprises:

[0032] An output location and a storage mode of the target evaluation task are obtained;

[0033] The target evaluation model plug-in is evaluated based on the target executor, to obtain a target evaluation result of the target evaluation task, comprising:

[0034] Each piece of evaluation data in the target data set is input into the target evaluation model plug-in for task evaluation based on the target executor one by one, to obtain an evaluation data result, and the evaluation data result is stored in a storage location corresponding to the storage mode.

[0035] When the evaluation data results corresponding to all the evaluation data pairs in the target data set are obtained, the target evaluation result is obtained based on all the evaluation data results, and the target evaluation result is output to the output position.

[0036] To achieve the above object, a third aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the evaluation method of the model evaluation system according to the second aspect when executing the computer program.

[0037] To achieve the above object, a fourth aspect of the embodiments of the present application provides a storage medium, which is a computer readable storage medium, the storage medium stores a computer program, and the computer program is executed by a processor to implement the evaluation method of the model evaluation system according to the second aspect.

[0038] The model evaluation system, the evaluation method and the related device provided by the embodiments of the present application, wherein the model evaluation system comprises a plug-in layer, a module layer and a core layer connected in sequence; the plug-in layer is provided with corresponding evaluation model plug-ins based on different architectures; the core layer is provided with a plurality of executors, each executor corresponding to an evaluation type; when the module layer receives a target evaluation task and target configuration information, the module layer is used to acquire a target evaluation model plug-in corresponding to the target evaluation task from a plurality of evaluation model plug-ins of the plug-in layer, and send the target evaluation model plug-in to a target executor corresponding to the target evaluation task in a plurality of executors of the core layer; the core layer is used to call the target executor to perform task evaluation based on the target evaluation model plug-in and the target configuration information, and output a target evaluation result of the target evaluation task. In the model evaluation system, the embodiments of the present application pre-set corresponding evaluation model plug-ins according to a plurality of different architectures, and pre-set executors corresponding to a plurality of different evaluation types, so that when the evaluation operation of the target evaluation task is actually performed, the target evaluation model plug-in corresponding to the software architecture and the hardware architecture of the target evaluation task can be accessed by the module layer, and the target executor corresponding to the task type of the target evaluation task can be used for model evaluation, so that the model evaluation of different architectures and task types can be flexibly performed in one model evaluation system, thereby effectively improving the versatility and compatibility of the model evaluation system.

[0039] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the following description, or can be learned by practice of the present application. The objects and other advantages of the present application can be achieved and obtained by the structure particularly pointed out in the specification, claims and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is a structural schematic diagram of an existing evaluation system provided by an embodiment of the present application.

[0041] Figure 2 is a structural schematic diagram of a model evaluation system provided by another embodiment of the present application.

[0042] Figure 3 is a structural schematic diagram of a plug-in layer provided with a plurality of evaluation model plug-ins provided by another embodiment of the present application.

[0043] Figure 4 is a relationship structural schematic diagram of a model evaluation system performing model evaluation provided by another embodiment of the present application.

[0044] Figure 5 is an implementation structural schematic diagram of a model evaluation system provided by another embodiment of the present application.

[0045] Figure 6 is a flowchart of an evaluation method of a model evaluation system provided by another embodiment of the present application.

[0046] Figure 7 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0048] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0050] With the release of ChatGPT, research related to large models and the development of downstream applications based on large models have become a global hotspot, and the evaluation of the capabilities of large models is one of them. In recent years, well-known large model evaluation datasets have emerged, including the logical capability evaluation gsm8k, the mathematical capability evaluation math, the English general capability mmlu, the Chinese general capability cmmlu, and the code capability humaneval. When evaluating a target model, the evaluation system downloads the evaluation dataset corresponding to the target model for model evaluation.

[0051] Referring to Figure 1 is a structural schematic diagram of an existing evaluation system provided by an embodiment of the present application. As Figure 1 indicated, taking an existing evaluation system opencompass as an example, the evaluation system is configured to perform model evaluation on large models supporting two types of task executors (including Transformer Executor and vLLMExecutor) with a hardware architecture of GPU and a software architecture of PyTorch.

[0052] However, in order to completely evaluate a model, 5-20 different datasets are needed to obtain a more comprehensive model performance evaluation, and the operation of evaluating each dataset is too cumbersome and is not conducive to large-scale parallel processing. However, due to the architecture limitation of the existing evaluation system, it can only evaluate large models supporting the hardware architecture and the software architecture, and cannot evaluate large models of other architectures, which greatly limits the model evaluation of the evaluation system and reduces the universality.

[0053] In order to improve the universality and compatibility of the model evaluation system, in the model evaluation system, corresponding evaluation model plugins are set according to a plurality of different architectures in advance, and corresponding executors are set according to a plurality of different evaluation types in advance, so that when actually evaluating a target evaluation task, the target evaluation model plugin corresponding to the software architecture and the hardware architecture of the target evaluation task can be accessed by using the module layer, and the target executor corresponding to the task type of the target evaluation task can be used for model evaluation, so that corresponding model evaluation can be flexibly performed on different architecture and task type evaluation tasks in one model evaluation system, thereby effectively improving the universality and compatibility of the model evaluation system.

[0054] The model evaluation system, the evaluation method, and the related equipment provided by the embodiments of the present application will be further described below. Referring to Figure 2 is a structural schematic diagram of a model evaluation system provided by an embodiment of the present application. As Figure 2As shown in the model evaluation system, the module layer, the core layer and the plugin layer are provided, and the module layer is connected with the core layer and the plugin layer.

[0055] Referring to Figure 3 FIG. 1 is a structural schematic diagram of a plugin layer provided with multiple evaluation model plugins according to an embodiment of the present application. As shown in FIG. 1, the plugin layer (module) is provided with multiple evaluation model plugins, which include three types of plugins, i.e., model plugins, evaluation data sets and evaluation methods. Each type of plugin defines an API interface and a function to be implemented, and can support model evaluation processing of a large model of a corresponding architecture by implementing plugins of different software architectures and hardware architectures. Figure 3

[0056] The model plugin (Model) sets up corresponding model plugins for each different software architecture (such as Pytorch, Mindspore, Tensorflow, etc.) and hardware architecture (such as GPU, NPU, etc.), such as GLM4, Llama3, etc. Each model plugin supports completing inference functions in an independent environment.

[0057] The evaluation data set (Dataset) sets up multiple data sets corresponding to evaluation tasks of different scene types, such as commonly used CEval, MMLU, CMMLU, etc. Users can select different evaluation data set plugins to implement different types of model evaluation.

[0058] The evaluation method (Eval) sets up multiple evaluation methods, such as Acc, Rouge, etc. These evaluation methods define the implementation process of converting the results generated by the model evaluation system in the model evaluation process into corresponding evaluation scores of different types.

[0059] When using the model evaluation system, users can select corresponding evaluation model plugins to execute model evaluation tasks according to their evaluation task requirements, and can connect the evaluation data sets, models and evaluation methods they need into the model evaluation system by implementing different evaluation model plugins, so as to adaptively select appropriate architectures, data sets and evaluation methods for model evaluation in the same model evaluation system, thereby avoiding the limitation that only model evaluation of a corresponding single architecture type can be performed in one model evaluation system, and further improving the evaluation universality and flexibility of the model evaluation system.

[0060] ​It can be understood that the generation steps of different types of model plug-ins include: taking each hardware architecture as a target hardware architecture one by one, generating at least one target hardware evaluation model plug-in applied to the target hardware architecture, and generating a target hardware calling interface associated with the target hardware evaluation model plug-in; and taking each software architecture as a target software architecture one by one, generating at least one target software evaluation model plug-in applied to the target software architecture, and generating a target software calling interface associated with the target software evaluation model plug-in; finally, obtaining a plurality of model plug-ins based on all target hardware evaluation model plug-ins and corresponding target hardware calling interfaces and all target software evaluation model plug-ins and corresponding target software calling interfaces.

[0061] For example, taking NVIDIA GPU and domestic NPU as target hardware architectures respectively. For NVIDIA GPU, a target hardware evaluation model plug-in named NVIDIA_GPU_Plugin is generated, which contains optimized inference code for NVIDIA GPU and can call CUDA library for efficient parallel computing; then, a target hardware calling interface named CUDA_Interface is generated, which encapsulates the calling logic of CUDA library and allows the model evaluation system to call GPU for inference through a unified interface. For domestic NPU, a target hardware evaluation model plug-in named NPU_Plugin is generated, which contains optimized code for the specific instruction set of NPU and can call the special API of NPU for inference; then, a target hardware calling interface named NPU_Interface is generated, which encapsulates the special API calling logic of NPU and allows the model evaluation system to call NPU for inference through a unified interface.

[0062] For example, take PyTorch and MindSpore as the target software framework respectively. For PyTorch, a target software evaluation model plug-in named PyTorch_Plugin is generated, which contains model loading and inference code for the PyTorch framework and can call the API of PyTorch for model inference. Then, a target software calling interface named PyTorch_Interface is generated, which encapsulates the API calling logic of PyTorch and allows the model evaluation system to call PyTorch for inference through a unified interface. For MindSpore, a target software evaluation model plug-in named MindSpore_Plugin is generated, which contains model loading and inference code for the MindSpore framework and can call the API of MindSpore for model inference. Then, a target software calling interface named MindSpore_Interface is generated, which encapsulates the API calling logic of MindSpore and allows the model evaluation system to call MindSpore for inference through a unified interface.

[0063] In the model evaluation system, corresponding plug-ins and calling interfaces are generated for different architectures in advance, so that in actual application, corresponding plug-ins can be dynamically loaded through corresponding calling interfaces according to user's needs, to realize the evaluation task requirements of different architectures in the same model evaluation system.

[0064] In addition, the module layer (module) sets a variety of module components, including a configuration management component (ConfigManager), a plug-in management component (PluginManager), and a storage management component (StorageManager), each of which is instantiated in the form of a singleton (Singleton) in the model evaluation system.

[0065] The configuration management component (ConfigManager) is used to read global information, command line information, including the names of the evaluation model plug-ins of each plug-in layer (plugin), and the like.

[0066] The plug-in management component (PluginManager) is used to manage the life cycle of the evaluation model plug-ins and global information used in the plug-in layer (plugin) and is responsible for the loading and initialization of the evaluation model plug-ins. The global information is obtained from another configuration management component (ConfigManager).

[0067] The storage management component (StorageManager) is used to manage data storage and IO operations during running, and is also used to manage data interaction operations between evaluation model plug-ins.

[0068] A plurality of different types of executors Executor are arranged in the core, which are responsible for completing specific evaluation tasks. Each executor can meet different evaluation needs by selecting different plug-ins. Each executor Executor corresponds to a type of evaluation function, such as large language model evaluation, multi-modal model evaluation, etc. In addition, it can also be divided into single-threaded executors Executor and multi-threaded executors Executor, so as to select the corresponding executor Executor according to the needs of the evaluation task.

[0069] In addition, the executor Executor is also responsible for loading instances of each component, and realizing the evaluation function corresponding to the entire evaluation task by calling the API interface provided by each component.

[0070] It can be understood that the generation step of the plurality of executors includes taking each evaluation type as a target evaluation type one by one; then, determining the number of threads corresponding to the target evaluation type, and generating at least one executor corresponding to the number of threads and the target evaluation type.

[0071] Next, take the executors corresponding to the language model evaluation type (mainly evaluating the performance of the model in natural language processing tasks such as text generation, question and answer, etc.) and the multi-modal model evaluation type (mainly evaluating the performance of the model in multi-modal tasks such as image-text generation, video understanding, etc.) as examples.

[0072] For language model evaluation, since the processing of text data is usually a CPU-intensive task, and each evaluation task is relatively independent, a large number of threads can be allocated to parallel processing of multiple evaluation tasks. Assuming that we allocate 8 threads for language model evaluation. It generates an executor named LanguageModelExecutor, which supports 8 threads to process evaluation tasks in parallel. The executor will load the required datasets, models and evaluation methods for language model evaluation, and process multiple evaluation tasks in parallel through multi-threading. Each thread is responsible for processing an evaluation task, and the executor coordinates the work of multiple threads to ensure efficient completion of the evaluation task.

[0073] For multimodal model evaluation, due to the processing of large data such as images and videos, it is usually an I / O intensive task, and each evaluation task may require more memory and computing resources, so the number of threads allocated is less. Suppose we allocate 4 threads for multimodal model evaluation. It generates an executor named MultimodalModelExecutor, which supports 4 threads to process evaluation tasks in parallel. The executor loads the dataset, model and evaluation method required for multimodal model evaluation, and processes multiple evaluation tasks in parallel through multi-threading. Due to the high demand for computing resources of multimodal tasks, the executor limits the number of threads to ensure that each task has enough resources for processing.

[0074] In the model evaluation system, corresponding plugins and call interfaces are generated in advance for different architectures, so that in actual application, according to the user's demand task type, the corresponding executor is used to flexibly and accurately realize the user's evaluation task, in addition, the model evaluation system can be easily extended to support more evaluation types through the configuration file, and appropriate number of threads is allocated for each evaluation type, to effectively improve the flexibility and versatility of the model evaluation system; and in the actual evaluation task execution process, for different task types, the corresponding multi-threaded executor is used, which can utilize parallel processing capability to improve the efficiency of model evaluation.

[0075] Reference Figure 4 is a relationship structure diagram of a model evaluation system improved by embodiments of the present application to perform model evaluation. As shown in Figure 4 When the model evaluation system receives the command line information and target configuration information (i.e. the global configuration information for this target evaluation task) input by the user for the target evaluation task, first read and parse the command line information and target configuration information through the configuration management component ConfigManager of the module layer, and determine the task information required by the user according to the parsed command line information, including dataset identifier, model identifier, evaluation method identifier, target task type (such as language model evaluation, multimodal model evaluation), user's output location and storage mode (such as remote call, log output, etc.) and the like, then send these task information to the plugin management component PluginManager.

[0076] The plugin management component PluginManager determines and loads the corresponding target evaluation model plugin (including target dataset, target model plugin and target evaluation method) from the multiple evaluation model plugins of the plugin layer according to the dataset identifier, model identifier and evaluation method identifier therein; and also calls the target executor from the multiple executors of the core layer according to the target task type.

[0077] The storage management component StoreageManager initializes its memory pool according to the user output location and the storage mode in the task information.

[0078] The target executor Executor acquires the corresponding target evaluation model plug-in (including the target data set, the target model plug-in, and the target evaluation method) through the plug-in management component PluginManager, and acquires the corresponding data storage space and the output location through the storage management component StoreageManager. Then, the target executor Executor starts to execute the model evaluation process corresponding to the evaluation task, which specifically includes the following steps.

[0079] (1) The target executor Executor acquires each piece of evaluation data question in the target data set D (denoted as Q={q1, q2, …, qn}).

[0080] (2) The target executor Executor calls the inference API interface of the target model plug-in M to acquire the model inference result predict (denoted as P={p1, p2, …, pn}) from the evaluation data Q.

[0081] (3) The target executor Executor stores the entire inference result P into the data storage space S.

[0082] (4) The target executor Executor acquires the standard answer answer of each piece of evaluation data in the evaluation data set D (denoted as A={a1, a2, …, an}) and stores it into the data storage space S.

[0083] (5) The target executor Executor adopts the target evaluation method E to calculate the final evaluation score according to the inference result P and the standard answer A stored in the data storage space S, and records the final evaluation score as the target evaluation result of the target evaluation task.

[0084] (6) The target executor Executor outputs the final evaluation score and the intermediate result to the user through the output location O.

[0085] Referring to Figure 5 is an implementation structure schematic diagram of a model evaluation system provided by an embodiment of the present application. As shown in Figure 5As shown in the middle, in the model evaluation system, it is mainly divided into 5 subdirectories user, core, module, plugin and utils. Among them, the user directory provides user-oriented APIs, including command line tools, launch tools, etc.; the core directory is the core running component, and different executors are used to realize different computing patterns for language models and multi-modal model evaluation; the module directory provides runtime management modules, including memory management, configuration management and plug-in management modules; the plugin directory encapsulates different platform model supports in the form of components, and can complete cross-platform running by replacing the corresponding Model; the utils directory provides basic functions such as log recording and parameter configuration, which come from the more general large model tool set MindUtils.

[0086] The API interfaces that need to be provided in the module directory include API interfaces related to the configuration management component ConfigManager, API interfaces related to the plug-in management component PluginManager, and API interfaces related to the storage management component.

[0087] Among them, the API interfaces related to the configuration management component ConfigManager include the ReadArgs interface for reading related configuration information in the command line, the ReadParam interface for reading related configuration information in the param, the LoadConfig interface for reading global yaml information, and the Get(Dataset / Model / Eval)Name interface for obtaining each plugin information and providing it to other singletons, etc.

[0088] The API interfaces related to the plug-in management component PluginManager include the Init interface for initializing each plugin by obtaining the plugin name from the ConfigManager, the LoadPlugin interface for calling the Load function of each plugin(Dataset / Model / Eval) to load the Plugin, and the Get(Dataset / Model / Eval) interface for obtaining each plugin instance, which can call the APU of each plugin.

[0089] The API interfaces related to the storage management component include the StoreData / LoadData interface for obtaining and storing data from the Dataset, the StorePredict / LoadPredict interface for obtaining and storing data predicted by the Model, the StoreEval / LoadEva interface for obtaining and storing evaluation data, etc.

[0090] The API interfaces mainly provided in the Core directory include: Init interface, used to call the API of PluginManager to obtain the instances of Dataset / Model / Eval; Run interface, used to complete the entire evaluation calculation pattern.

[0091] The API interfaces provided in the plugin directory include: (1) Dataset: cfg=LoadConfig(path), import configuration file from path; Init(cfg), initialization of dataset; GetExample, obtain a data sample; GetQuestion, obtain the question of a sample; GetAnswer, obtain the answer of a sample; (2) Model: LoadConfig, import configuration file; Init, model initialization; a=Predict(q), generate model prediction result; (3) Retrieval: LoadConfig, import configuration file; Init, initialize how to splice prompt words, such as few shot, zero shot, etc.; (4) Eval: LoadConfig, import configuration file; Init, initialization, set evaluation method; Eval(Model.Predict, Dataset.GetAnswer), score conversion according to model prediction result and dataset.

[0092] The model evaluation system performs evaluation in the following mode:

[0093] e=dataset.GetExample();

[0094] p=retrieval.SetPrompt(e);

[0095] StorageManager.StoreQuestion(dataset.GetQuestion());

[0096] for d in StorgeManager.LoadNextQuestion();

[0097] StorageManager.StorePredict(model.predict(p,d));

[0098] StorageManager.StoreAnswer(dataset.GetAnswer());

[0099] StorageManger.StoreEval(eval.eval(StorageManager.LoadEval()))#Answerand Predict.

[0100] The configuration file contains the configuration of the dataset, model, and running hardware and software. Different executors are selected according to different hardware and software configurations. The dataset configuration also includes the inference method to be used and how to evaluate the model.

[0101] In some embodiments, the model evaluation system provided in this application can further support large-scale parallel model evaluation tasks through multi-threaded executors and distributed storage management, and users can specify the number of parallel tasks and resource allocation through configuration files, thereby significantly improving evaluation efficiency.

[0102] This application provides a model evaluation system that utilizes a plug-in design to generate plug-ins and corresponding calling interfaces for different hardware and software architectures. This enables the model evaluation system to flexibly support the evaluation of large models across multiple architectures, thereby improving its flexibility and versatility. Furthermore, by encapsulating datasets, models, and evaluation methods as independent plug-ins, the system can be easily extended to support new datasets, models, and evaluation methods, thus enhancing its scalability. Additionally, it generates executors with appropriate threads for different evaluation types, allowing for the utilization of suitable parallel processing capabilities to flexibly and efficiently implement evaluation tasks of various types. Moreover, by providing a unified evaluation process, users only need to select the dataset, model, and evaluation method through a configuration file, and the platform automatically completes the evaluation task and outputs the results through a unified interface. Users do not need to concern themselves with the underlying implementation details, thereby reducing the operational complexity of the model evaluation system and improving process standardization.

[0103] This application also provides an evaluation method for a model evaluation system, which can be applied to any smart terminal equipped with the model evaluation system. The evaluation method for the model evaluation system in this application will be described in detail below. (Refer to...) Figure 6 This is an optional flowchart of the evaluation method for the model evaluation system provided in the embodiments of this application. Figure 6 The method may include, but is not limited to, steps 601 to 603. It is also understood that this embodiment... Figure 6 The order of steps 601 to 603 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0104] Step 601: In response to the evaluation request of the target evaluation task, obtain the target configuration information of the target evaluation task.

[0105] Step 602: determining a target evaluation model plug-in and a target executor corresponding to the target evaluation task from a plurality of evaluation model plug-ins and a plurality of executors, at least two evaluation model plug-ins corresponding to different architectures, and each executor corresponding to an evaluation type.

[0106] Step 603: configuring the target evaluation model plug-in based on the target configuration information, and performing task evaluation on the target evaluation model plug-in based on the target executor to obtain a target evaluation result of the target evaluation task.

[0107] The steps 601 to 603 are described in detail as follows.

[0108] In some embodiments, similar to the implementation process of the model evaluation system described above, when the model evaluation system responds to the evaluation request of the target evaluation task input by the user, the target configuration information, the command line information, and the target task type of the target evaluation task are obtained, and the output position and the storage mode of the target evaluation task are also obtained. Then, the dataset identifier, the model identifier, and the evaluation method identifier are parsed from the command line information, the target dataset is determined from a plurality of evaluation datasets based on the dataset identifier, the target model plug-in is determined from a plurality of evaluation model plug-ins based on the model identifier, the target evaluation method is determined from a plurality of evaluation methods based on the evaluation method identifier, and the target executor is determined from a plurality of executors based on the target task type.

[0109] Next, each piece of evaluation data in the target dataset is input into the target evaluation model plug-in for task evaluation based on the target executor, the evaluation data result is obtained, and the evaluation data result is stored in the storage position corresponding to the storage mode; then, when the evaluation data results corresponding to all the evaluation data in the target dataset are obtained, the target evaluation result is obtained based on all the evaluation data results, and the target evaluation result is output to the output position. Thus, in the model evaluation system, a plurality of evaluation model plug-ins corresponding to a plurality of different architectures set in advance and a plurality of executors corresponding to a plurality of different evaluation types are set, so that in the process of actually processing the target evaluation task, the target evaluation model plug-in corresponding to the software architecture and the hardware architecture of the target evaluation task is selected for access, and the target executor corresponding to the task type of the target evaluation task is selected for model evaluation. Therefore, the corresponding model evaluation of the evaluation task of different architectures and task types can be flexibly performed in one model evaluation system, so as to effectively improve the universality and compatibility of the evaluation method.

[0110] The embodiment of the present application also provides an electronic device, comprising:

[0111] at least one memory;

[0112] at least one processor;

[0113] at least one program;

[0114] The program is stored in the memory, and the processor executes the at least one program to implement the model evaluation method of the model evaluation system. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a vehicle-mounted computer, etc.

[0115] Referring to Figure 7 , Figure 7 The hardware structure of the electronic device of another embodiment is illustrated, and the electronic device includes:

[0116] The processor 701 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0117] The memory 702 can be implemented in the form of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), etc. The memory 702 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 702 and are called and executed by the processor 701 to implement the model evaluation method of the model evaluation system.

[0118] The input / output interface 703 is used to realize information input and output.

[0119] The communication interface 704 is used to realize the communication interaction between the device and other devices, and can realize communication through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.).

[0120] The bus 705 transmits information between various components (such as the processor 701, the memory 702, the input / output interface 703, and the communication interface 704) of the device.

[0121] The processor 701, the memory 702, the input / output interface 703, and the communication interface 704 are connected to each other through the bus 705 to realize communication connection between them inside the device.

[0122] The embodiment of the present application further provides a storage medium, which is a computer readable storage medium, and stores a computer program. The computer program is executed by a processor to implement the evaluation method of the model evaluation system.

[0123] The memory is a non-transitory computer readable storage medium, and can be used to store a non-transitory software program and a non-transitory computer executable program. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0124] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0125] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than those shown in the figures, or combine certain steps, or different steps.

[0126] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the embodiments of the present application.

[0127] Those skilled in the art can understand that all or some steps in the above disclosed method, the function modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0128] The terms "first", "second", "third", "fourth", and the like in the description of this application and in the claims hereof, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed herein is solely for the convenience of the reader and does not limit the scope of the application. It is also to be understood that the description and examples in this application are intended to cover all possible combinations where any of the several elements can represent one or more elements.

[0129] It should be understood that, in this application, "at least one" means one or more, "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c, can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0130] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above-mentioned units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. The coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0131] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment of the present application.

[0132] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0133] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media that can store programs.

[0134] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not intended to limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A model evaluation system, characterized in that, The system comprises a plug-in layer, a module layer, and a core layer connected in sequence. The plugin layer has corresponding evaluation model plugins based on different architectures; The core layer is equipped with multiple actuators, each of which corresponds to a type of evaluation. When the module layer receives the target evaluation task and target configuration information, the module layer is used to obtain the target evaluation model plugin corresponding to the target evaluation task from the multiple evaluation model plugins in the plugin layer, and send the target evaluation model plugin to the target executor corresponding to the target evaluation task from the multiple executors in the core layer; The core layer is used to call the target executor to perform task evaluation based on the target evaluation model plugin and the target configuration information, and output the target evaluation result of the target evaluation task; The actuator generation process includes the following steps: Each of the aforementioned evaluation types is then used as the target evaluation type. Determine the number of threads corresponding to the target evaluation type, and generate at least one executor corresponding to the number of threads and the target evaluation type.

2. The model evaluation system according to claim 1, characterized in that, The architecture includes hardware architecture and software architecture, and the evaluation model plugin includes multiple model plugins. The generation process of the multiple model plugins includes the following steps: Each of the aforementioned hardware architectures is taken as a target hardware architecture, at least one target hardware evaluation model plugin is generated for the target hardware architecture, and a target hardware calling interface associated with the target hardware evaluation model plugin is generated. Each of the aforementioned software architectures is taken as a target software architecture, at least one target software evaluation model plugin is generated for the target software architecture, and a target software calling interface associated with the target software evaluation model plugin is generated. The various model plugins are obtained based on all the target hardware evaluation model plugins and their corresponding target hardware calling interfaces, as well as all the target software evaluation model plugins and their corresponding target software calling interfaces.

3. The model evaluation system according to claim 2, characterized in that, The evaluation model plugin includes multiple evaluation datasets and multiple evaluation methods; Among them, at least two of the evaluation datasets have different data types; The evaluation method is used to convert the output data of the evaluation model plugin into corresponding evaluation scores, and there are at least two evaluation methods with different conversion processes.

4. The model evaluation system according to claim 1, characterized in that, The module layer includes a configuration management component, a plugin management component, and a storage management component; The configuration management component is used to read the command line information of the target evaluation task to determine the dataset identifier, model identifier, evaluation method identifier and target task type, and send the dataset identifier, model identifier, evaluation method identifier and target task type to the plugin management component; The plugin management component is used to obtain and load the target evaluation model plugin from the plugin layer according to the dataset identifier, the model identifier and the evaluation method identifier. The plugin management component is also used to call the target executor from the core layer according to the target task type. The storage management component is used to store and manage the evaluation data required for the evaluation task.

5. An evaluation method for the model evaluation system as described in claim 1, characterized in that, The method includes: In response to the evaluation request of the target evaluation task, obtain the target configuration information of the target evaluation task; The target evaluation model plugin and target executor corresponding to the target evaluation task are determined from multiple evaluation model plugins and multiple executors, wherein at least two of the evaluation model plugins have different architectures, and each executor corresponds to an evaluation type; Configure the target evaluation model plugin based on the target configuration information, and perform task evaluation on the target evaluation model plugin based on the target executor to obtain the target evaluation result of the target evaluation task.

6. The evaluation method of the model evaluation system according to claim 5, characterized in that, The target evaluation model plugin includes a target dataset, a target model plugin, and a target evaluation method. Determining the target evaluation model plugin and target executor corresponding to the target evaluation task includes: Obtain the command line information and target task type corresponding to the target evaluation task; The target dataset is determined from multiple evaluation datasets based on the dataset identifier of the command line information, the target model plugin is determined from multiple evaluation model plugins based on the model identifier of the command line information, and the target evaluation method is determined from multiple evaluation methods based on the evaluation method identifier of the command line information. The target executor is determined from multiple executors based on the target task type.

7. The evaluation method for the model evaluation system according to claim 6, characterized in that, Before determining the target evaluation model plugin and target executor corresponding to the target evaluation task, the method further includes: Obtain the output location and storage mode of the target evaluation task; The step of evaluating the target evaluation model plugin based on the target executor to obtain the target evaluation result of the target evaluation task includes: Based on the target executor, each evaluation data in the target dataset is input into the target evaluation model plugin for task evaluation, and the evaluation data results are obtained and stored in the storage location corresponding to the storage mode. When the evaluation data results corresponding to all the evaluation data in the target dataset are obtained, the target evaluation result is obtained based on all the evaluation data results, and the target evaluation result is output to the output position.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the evaluation method of the model evaluation system according to any one of claims 5 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the evaluation method of the model evaluation system as described in any one of claims 5 to 7.

Images