An evaluation method, system, device, medium, product, and chip.

Abstract

This application relates to the field of large model technology, and more particularly to an evaluation method, system, device, medium, product, and chip. In this method, an inference environment corresponding to each model is pre-built. When evaluating a model, the evaluation environment and the corresponding inference environment are run. Based on the inference environment, the inference service interface corresponding to the model is started, and an inference request is initiated by calling the inference service interface. Based on the evaluation environment, the inference results from the inference service interface are received, and the evaluation task for the model is executed based on the evaluation environment and the inference results. In this way, the inference process and the evaluation process of the model are decoupled, allowing the evaluation environment to adapt to the evaluation of various models.

Description

Technical Field

[0001] This application relates to the field of large model technology, and more particularly to an evaluation method, system, device, medium, product, and chip. Background Technology

[0002] A large model is a machine learning model with a large number of parameters and a complex computational structure. It can assist users in completing various tasks, thereby changing users' production and lifestyles and improving productivity.

[0003] When evaluating a model, electronic devices first load the model in the inference environment to generate inference results, and then evaluate the model's prediction accuracy based on the base evaluation environment and the inference results. However, when configuring the virtual inference environment and the base runtime environment on electronic devices, the model's evaluation environment is not deployed independently, but is nested within the model's inference environment, resulting in a high degree of coupling between the inference and evaluation environments. Under this coupled architecture, conflicts may arise due to shared hardware resources and software dependencies. In the event of conflicts between the inference and evaluation environments, the model evaluation process will malfunction. Summary of the Invention

[0004] To address the issue of the model evaluation process failing to function properly, embodiments of this application provide an evaluation method, system, device, medium, product, and chip, including:

[0005] In a first aspect, embodiments of this application provide an evaluation method applied to an electronic device, comprising: running an evaluation environment and a first inference environment corresponding to a first model; starting a first inference service interface corresponding to the first model based on the first inference environment; initiating a first inference request by calling the first inference service interface based on the evaluation environment; receiving a first inference result from the first inference service interface based on the evaluation environment, wherein the first inference result is determined by the first model based on the first inference environment; and performing a first evaluation task on the first model based on the evaluation environment and the first inference result.

[0006] In this embodiment of the application, by pre-constructing the inference environment corresponding to each model, the inference environment corresponding to the model is isolated from the evaluation environment. Thus, when evaluating the model, the inference result can be determined based on the inference environment corresponding to the model, and the model can be evaluated based on the evaluation environment. This decouples the inference process and the evaluation process of the model, so that the evaluation environment can be adapted to the evaluation of various models.

[0007] Furthermore, by pre-building inference service interfaces corresponding to each model to receive inference requests or obtain inference results, it is only necessary to call the inference service interface corresponding to the model to receive inference requests or obtain inference results. There is no need to check whether the process is the main process or whether the child process has completed inference each time the inference result is written to the storage device. In this way, the complexity of the model's inference process can be reduced.

[0008] In some implementations of the first aspect, an evaluation environment is run; if the first inference service interface is not started, the first inference environment corresponding to the first model is run.

[0009] In some implementations of the evaluation environment and the first inference environment corresponding to the first model, when the evaluation process of the first model is detected to be started, the electronic device can run the evaluation environment and determine whether the first inference service interface corresponding to the first model is started. If the first inference service interface corresponding to the first model is not started, the electronic device can run the first inference environment corresponding to the first model; if the first inference service interface corresponding to the first model is started, the electronic device can invoke the first inference service interface based on the evaluation environment to initiate a first inference request.

[0010] In some implementations of the first aspect, initiating a first inference request by calling the first inference service interface based on the evaluation environment includes: initiating a first inference request to the target inference service among the multiple inference services corresponding to the first model by calling the first inference service interface based on the evaluation environment.

[0011] For example, electronic devices can pre-build inference service interfaces corresponding to each model based on the FastAPI framework, so that each model's inference service interface can serve as a unified scheduling entry point for that model, receive inference requests and send them to the target inference service among the multiple inference services corresponding to the model, and receive inference results from the target inference service.

[0012] In this way, by pre-building inference service interfaces corresponding to each model to receive inference requests or obtain inference results, it is only necessary to call the inference service interface corresponding to the model to receive inference requests or obtain inference results. There is no need to check whether the process is the main process or whether the child process has completed inference each time the inference result is written to the storage device. This reduces the complexity of the model's inference process.

[0013] In some implementations of the first aspect, initiating a first inference request to a target inference service among multiple inference services corresponding to the first model includes: obtaining queue information for each inference service among the multiple inference services; identifying the inference service whose queue information content is less than the information content threshold among the multiple inference services as the target inference service; and initiating a first inference request to the target inference service.

[0014] For example, an electronic device can obtain queue information for each of multiple inference services and identify the inference service with the lowest information content threshold among the multiple inference services as the target inference service. The electronic device can then initiate a first inference request to the target inference service. The first model loaded by the target inference service can respond to the first inference request and determine the inference result.

[0015] For example, during the inference process of the Qwen model, prompts are input into the Qwen concurrent inference interface, and the Qwen model generates inference results. These results can be used to evaluate the Qwen model and obtain evaluation results. Similarly, during the inference process of the InternVL model, prompts are input into the InternVL concurrent inference interface, and the InternVL model generates inference results. These results can be used to evaluate the InternVL model and obtain evaluation results.

[0016] In some implementations of the first aspect, before initiating the first inference request by calling the first inference service interface based on the evaluation environment, the method further includes: obtaining the first dataset corresponding to the first model based on the evaluation environment; and constructing the prompt words carried in the first inference request according to the data category and data content of the first dataset.

[0017] In some implementations of the first aspect, the first inference result is determined in the following ways: if the first model supports loading based on the vision larger language model (vLLM) library, the target inference service loads the first model based on the vision larger language model library; if the first model does not support loading based on the vision larger language model library, the target inference service loads the first model based on the Transformers library (a basic large model loading core library); the first model responds to the first inference request and determines the first inference result based on the first inference environment and the prompt words in the first inference request.

[0018] In some implementations of the first aspect, the prompt words carried in the first inference request are constructed according to the data category and data content of the first dataset, including: if the first dataset belongs to a vision language model (VLM) dataset or a large language model (LLM) dataset, the prompt words carried in the first inference request are constructed based on the data content in the first dataset; if the first dataset belongs to a graphical user interface (GUI) dataset or an agent dataset under the Task dimension, the first dataset is divided into multiple data subsets, and hierarchical prompt words carried in the first inference request are constructed based on the data content in each data subset.

[0019] In some implementations of the first aspect, the first dataset is divided into multiple data subsets, and hierarchical prompts carried in the first inference request are constructed based on the data content of each data subset. This includes: obtaining the task requirements labeled in the first dataset; determining the target evaluation method that matches the task requirements from preset evaluation methods according to the task requirements; wherein the preset evaluation methods include parallel evaluation method, serial evaluation method, plan evaluation method, and tool call evaluation method; dividing the first dataset into multiple data subsets corresponding to multiple steps in the task according to the task execution logic corresponding to the target evaluation method; and constructing hierarchical prompts carried in the first inference request based on the data content of each data subset.

[0020] In some implementations of the first aspect, the multiple steps in the task include a preceding step and a current step. The data content in each data subset includes input data and truth data. Based on the data content in each data subset, a hierarchical prompt word carried in the first inference request is constructed, including: when the target evaluation method is a serial evaluation method, the truth data in the data subset corresponding to the preceding step is used as historical information; and the hierarchical prompt word corresponding to the current step is constructed based on the historical information and the input data in the data subset corresponding to the current step.

[0021] In some implementations of the first aspect, the multiple steps in the task include a preceding step and a current step. The data content in each data subset includes input data and truth data. Based on the data content in each data subset, a hierarchical prompt word carried in the first inference request is constructed, including: when the target evaluation method is a parallel evaluation method, the inference result generated by the preceding step based on the input data in the corresponding data subset of the preceding step is used as historical information; and the hierarchical prompt word corresponding to the current step is constructed based on the historical information and the input data in the corresponding data subset of the current step.

[0022] In some implementations of the first aspect, the task includes multiple steps, including the current step, and each data subset includes input data and truth data. Based on the data content in each data subset, a hierarchical prompt word carried in the first inference request is constructed, including: when the target evaluation method is a plan evaluation method or a tool call evaluation method, the inference result generated by the current step based on the data content in the data subset corresponding to the current step is used as guidance information; and the hierarchical prompt word corresponding to the current step is constructed according to the guidance information and the data content in the data subset corresponding to the current step.

[0023] Secondly, embodiments of this application provide an evaluation system, which includes an evaluation environment and at least one inference environment. The evaluation environment is used to call a first inference service interface corresponding to a first model launched by a first inference environment in the at least one inference environment to initiate a first inference request. The first inference environment in the at least one inference environment is used to perform model inference and send a first inference result to the evaluation environment through the first inference service interface. The evaluation environment is used to receive the inference result from the first inference service interface and perform an evaluation task on the first model based on the first inference result.

[0024] Thirdly, embodiments of this application provide an electronic device, including: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the one or more processors of the electronic device, for executing the evaluation method provided in the first aspect and any possible implementation of the first aspect.

[0025] Fourthly, embodiments of this application provide a readable storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the first aspect and any possible implementation of the evaluation method provided in the first aspect.

[0026] Fifthly, embodiments of this application provide a computer program product, the computer program product including computer instructions, which, when executed by an electronic device, enable the electronic device to perform the evaluation method provided as in the first aspect and any possible implementation of the first aspect.

[0027] In a sixth aspect, embodiments of this application provide a chip including a processor coupled to a memory for executing computer programs or instructions stored in the memory, such that the chip implements the first aspect and any possible implementation of the evaluation method provided in the first aspect. Attached Figure Description

[0028] Figure 1 According to some embodiments of this application, a flowchart of a model evaluation process is shown;

[0029] Figure 2 According to some embodiments of this application, a comparative schematic diagram of an evaluation process is shown in the case where there is a conflict between the inference environment and the evaluation environment.

[0030] Figure 3 According to some embodiments of this application, a flowchart of an evaluation method is shown;

[0031] Figure 4A According to some embodiments of this application, a flowchart of model loading and inference is shown;

[0032] Figure 4B According to some embodiments of this application, an evaluation schematic diagram of a model is shown;

[0033] Figure 5 According to some embodiments of this application, a flowchart of an evaluation method is shown;

[0034] Figure 6 According to some embodiments of this application, a flowchart of another evaluation method is shown;

[0035] Figure 7A According to some embodiments of this application, a schematic diagram of the reasoning process of an existing model is shown;

[0036] Figure 7B According to some embodiments of this application, a reasoning flow for the model in this application is shown;

[0037] Figure 8 According to some embodiments of this application, a flowchart illustrating a first method for constructing prompt words is shown;

[0038] Figure 9 According to some embodiments of this application, a flowchart illustrating a second method for constructing prompt words is shown;

[0039] Figure 10 According to some embodiments of this application, a flowchart illustrating a third method for constructing prompt words is shown;

[0040] Figure 11 According to some embodiments of this application, a flowchart illustrating a fourth type of constructing prompt words is shown;

[0041] Figure 12 According to some embodiments of this application, a flowchart of a model reasoning evaluation process is shown;

[0042] Figure 13 According to some embodiments of this application, a structural block diagram of an electronic device 100 to which the evaluation method provided in this application is applicable is shown. Detailed Implementation

[0043] The embodiments of this application include, but are not limited to, evaluation methods, systems, devices, media, products, and chips.

[0044] It is understood that the evaluation methods mentioned in the embodiments of this application can be applied to electronic devices. The electronic device can be referred to as a terminal, user equipment (UE), mobile station (MS), or mobile terminal (MT), etc. Exemplarily, the electronic device can be a desktop computer, laptop computer, personal computer (PC), server, or other electronic device; the embodiments of this application do not specifically limit it.

[0045] It is understandable that model evaluation is necessary for a deeper understanding of the model. For ease of explanation, the following text will use the performance evaluation of the model's prediction accuracy as an example.

[0046] It is understandable that different code execution tasks require electronic devices to be configured with different virtual runtime environments. For example, the virtual runtime environment required for the model's inference task can be called the inference environment, and the virtual runtime environment required for the model's evaluation task can be called the base evaluation environment.

[0047] When evaluating the prediction accuracy of a model, the electronic device first loads the model based on the inference environment to generate inference results, and then evaluates the model's prediction accuracy based on the base evaluation environment and the inference results. For example, such as... Figure 1 As shown, the electronic device can initiate Model 1 evaluation, that is, initiate the evaluation process of Model 1, activate the base evaluation environment and activate the Model 1 inference environment, start model inference based on the Model 1 inference environment, run the evaluation process of Model 1 based on the base evaluation environment and the inference results, and complete the evaluation.

[0048] It is understandable that when configuring virtual inference and base runtime environments on electronic devices, the model's evaluation environment is not deployed independently but is nested within the model's inference environment, resulting in a high degree of coupling between the model's inference and evaluation environments. Under this coupled architecture, the two may conflict due to shared hardware resources and software dependencies, or they may maintain stable operation and not conflict when resources are allocated reasonably.

[0049] like Figure 2 As shown, there is no conflict between the Model 1 inference environment and the evaluation environment (the Model 1 inference environment includes the base evaluation environment as an illustration of a conflict-free scenario). The electronic device can initiate the evaluation process of Model 1, and then the electronic device can activate the Model 1 inference environment, run the Model 1 inference process and evaluation process, and complete the evaluation.

[0050] Similarly, if there is no conflict between the Model 2 inference environment and the evaluation environment (assuming that the Model 2 inference environment includes the base evaluation environment as an example of a conflict-free scenario), the electronic device can also start the Model 2 evaluation process. Then, the electronic device can activate the Model 2 inference environment, run the Model 2 inference process and evaluation process, and complete the evaluation.

[0051] Therefore, the model evaluation process can function normally when there is no conflict between the inference environment and the evaluation environment.

[0052] Continue as Figure 2 As shown, a conflict occurs between the Model 3 inference environment and the evaluation environment (the intersection of the Model 3 inference environment and the base evaluation environment is used as an illustration of the conflict scenario). The electronic device can initiate the evaluation process of Model 3 and activate the Model 3 inference environment. However, although the Model 3 inference environment can be activated, the Model 3 evaluation environment cannot be used normally due to the conflict between the Model 3 inference environment and the evaluation environment, which will cause the Model 3 evaluation process to fail.

[0053] Therefore, when conflicts occur between the inference and evaluation environments, such as when the version of the library required by the evaluation environment does not match the version of the corresponding library in the inference environment, or when the evaluation and inference environments compete for the same hardware resources, the model evaluation process will not function properly.

[0054] To address the aforementioned issues, this application provides an evaluation method. In this method, an inference environment corresponding to each model is pre-built. When evaluating a model, the evaluation environment and the inference environment corresponding to the model are run. Based on the inference environment, the inference service interface corresponding to the model is started, and an inference request is initiated by calling the inference service interface. Based on the evaluation environment, the inference result from the inference service interface is received, and the evaluation task for the model is executed based on the evaluation environment and the inference result.

[0055] In this embodiment, by pre-constructing the inference environment corresponding to each model, the inference environment corresponding to the model is isolated from the evaluation environment. Thus, when evaluating the model, the inference result can be determined based on the inference environment corresponding to the model, and the model can be evaluated based on the evaluation environment. This approach can decouple the model's inference process and evaluation process, thereby avoiding the problem that the model's evaluation process cannot run normally due to the conflict between the inference environment and the evaluation environment. Since the evaluation environment is relatively independent after decoupling, it can save memory and make the evaluation environment adaptable to the evaluation of various models, thereby improving the user experience.

[0056] The evaluation methods mentioned in the embodiments of this application are described below. For example... Figure 3The diagram illustrates a flowchart of an evaluation method. This evaluation method can be performed by an electronic device. For example, the evaluation method may include:

[0057] S301: Run the evaluation environment and the first inference environment corresponding to the first model.

[0058] It is understood that the evaluation environment can also be called the base evaluation environment, and this application does not make a specific limitation.

[0059] It is understandable that the first model can be any one of multiple models. For example, model 1, model 2, model 3, ... or model n. In some implementations, the first model can be a large language model (LLM), a graphical user interface evaluation (GUI) model, a visual language model (VLM), an agent, or an audio model, etc.

[0060] It is understandable that the first inference environment corresponding to the first model can also be called the model inference environment. For example, the inference environment corresponding to model 1 can be called the model 1 inference environment, the inference environment corresponding to model 2 can be called the model 2 inference environment, the inference environment corresponding to model 3 can be called the model 3 inference environment, and so on. The inference environment corresponding to model n can be called the model n inference environment.

[0061] In some implementations of the evaluation environment and the first inference environment corresponding to the first model, when the evaluation process of the first model is detected to be started, the electronic device can run the evaluation environment and determine whether the first inference service interface corresponding to the first model is started. If the first inference service interface corresponding to the first model is not started, the electronic device can run the first inference environment corresponding to the first model; if the first inference service interface corresponding to the first model is started, the electronic device can execute S303, that is, initiate a first inference request by calling the first inference service interface based on the evaluation environment.

[0062] It is understandable that in some implementations, after running the first inference environment corresponding to the first model, if the loading instruction of the first model is detected, multiple graphics processing units (GPUs) in the electronic device can independently start a process. Then, the electronic device can build an inference service corresponding to the first model based on each process, and integrate the output of multiple processes to provide a concurrent inference interface, namely the first inference service interface, to complete the deployment of the multi-process inference service of the first model.

[0063] Furthermore, the inference service among the multiple inference services corresponding to the first model can determine whether the first model supports the loading method based on the vLLM library, and complete the loading of the first model according to the loading method supported by the first model. Then, the inference service among the multiple inference services corresponding to the first model can enter the ready state, continuously waiting for inference requests sent based on the evaluation environment, so as to respond in a timely manner and generate inference results.

[0064] If the first model supports loading via the vLLM library, the inference services among the multiple inference services corresponding to the first model can load the first model based on the vLLM library to generate the corresponding first model instance. If the first model does not support loading via the vLLM library, the inference services among the multiple inference services corresponding to the first model can load the first model based on the Transformers library to generate the corresponding first model instance.

[0065] For ease of understanding, the following text uses the first model as the Tongyi Qianwen (Qwen) model or the Intern Vision and Language (InternVL) model, with the inference environment corresponding to both the Qwen model and the InternVL model being the conda environment, to illustrate the model loading method mentioned in the embodiments of this application.

[0066] like Figure 4A As shown, during the loading of the Qwen model, the electronic device can run the inference environment corresponding to the Qwen model, i.e., the conda environment. At this time, the current conda environment is only responsible for the inference process of the Qwen model. Then, upon detecting a loading instruction (e.g., `load`) for loading the Qwen model, the electronic device can allocate resources to the four graphics processors: GPU1, GPU2, GPU3, and GPU4. GPU1 can start process 1, GPU2 can start process 2, GPU3 can start process 3, and GPU4 can start process 4. Then, the electronic device can build inference service 1 based on process 1, inference service 2 based on process 2, inference service 3 based on process 3, and inference service 4 based on process 4. Furthermore, the electronic device can integrate the outputs of inference service 1, inference service 2, inference service 3, and inference service 4 to provide a Qwen concurrent inference interface.

[0067] Similarly, continue as Figure 4AAs shown, during the loading of the InternVL model, the electronic device can run the inference environment corresponding to the InternVL model, i.e., the conda environment. At this time, the current conda environment is only responsible for the inference process of the InternVL model. Then, upon detecting a loading instruction (e.g., `load`) for loading the InternVL model, the electronic device can allocate resources to the four graphics processors: GPU1, GPU2, GPU3, and GPU4. GPU1 can start process 1, GPU2 can start process 2, GPU3 can start process 3, and GPU4 can start process 4. Then, the electronic device can build inference service 1 based on process 1, inference service 2 based on process 2, inference service 3 based on process 3, and inference service 4 based on process 4. Furthermore, the electronic device can integrate the outputs of inference service 1, inference service 2, inference service 3, and inference service 4 to provide an internvl concurrent inference interface.

[0068] S302: Start the first inference service interface corresponding to the first model based on the first inference environment.

[0069] It is understood that the first inference service interface corresponding to the first model can also be called the concurrent inference interface corresponding to the first model, the inference service application programming interface (API) corresponding to the first model, the concurrent inference interface of the first model, or the inference service application programming interface of the first model. This application does not make any specific limitation.

[0070] In some implementations, electronic devices can pre-build inference service interfaces corresponding to each model. For example, inference service interfaces for model 1, model 2, model 3, ... and model n.

[0071] For example, electronic devices can pre-build inference service interfaces corresponding to each model based on the FastAPI framework, so that each model's inference service interface can serve as a unified scheduling entry point for that model, receive inference requests and send them to the target inference service among the multiple inference services corresponding to the model, and receive inference results from the target inference service.

[0072] S303: Initiate the first inference request by calling the first inference service interface based on the evaluation environment.

[0073] It can be understood that the first inference request can be a request instructing the first model to perform inference, and the first inference request can carry a prompt word. The prompt word can be used to provide constraints for the inference process of the first model, enabling the first model to complete the corresponding inference based on the prompt word.

[0074] In some implementations, the electronic device can invoke the first inference service interface based on the evaluation environment to initiate a first inference request to the target inference service among the multiple inference services corresponding to the first model.

[0075] In some methods of initiating a first inference request to a target inference service among multiple inference services corresponding to a first model, the electronic device can obtain queue information for each of the multiple inference services and identify the inference service whose queue information is less than a threshold as the target inference service. Then, the electronic device can initiate a first inference request to the target inference service. Simply put, multiple processes are launched in FastAPI to complete a model deployment. When an inference request arrives, each process competes for the request based on its idle time and performs model inference.

[0076] It is understandable that before initiating the first inference request by calling the first inference service interface based on the evaluation environment, the electronic device can obtain the first dataset corresponding to the first model based on the evaluation environment, and construct the prompt word carried in the first inference request according to the data category and data content of the first dataset. The data category can include categories such as VLM dataset, LLM dataset, GUI dataset, or Agent dataset, and the data content can include input data and ground truth data. The specific implementation method for constructing the prompt word carried in the first inference request will be detailed below.

[0077] In some methods of obtaining the first dataset corresponding to the first model, the target inference service can send a first data loading request to the storage device and obtain the first dataset corresponding to the first model based on the evaluation environment. It can be understood that, in the storage device, in response to the first data loading request, the storage device can determine the data that satisfies the first data loading request from multiple datasets.

[0078] For example, when the first data loading request is for "testing tasks in the natural sciences category", the storage device can obtain the first dataset as shown in Table 3 from dataset 1 shown in Table 1 and dataset 2 shown in Table 2, and return it to the target inference service.

[0079] Table 1

[0080]

[0081] Table 2

[0082]

[0083] Table 3

[0084]

[0085] Thus, by starting with an open-source dataset and reconstructing it into a completely new dataset based on the dataset modality and the vertical domain task to be tested, the vertical domain evaluation can obtain more intuitive results.

[0086] S304: Receive the first inference result from the first inference service interface based on the evaluation environment. The first inference result is determined by the first model based on the first inference environment.

[0087] It is understandable that the first inference result can be the prediction data determined by the first model based on the first dataset corresponding to the first model.

[0088] In some implementations that determine the inference result, the first model loaded by the target inference service can respond to the first inference request and determine the inference result. Continuing... Figure 4A As shown, during the inference process of the Qwen model, prompt words are input into the Qwen concurrent inference interface, and the Qwen model generates inference results. These inference results can be used to evaluate the Qwen model and obtain evaluation results. Similarly, during the inference process of the InternVL model, prompt words are input into the InternVL concurrent inference interface, and the InternVL model generates inference results. These inference results can be used to evaluate the InternVL model and obtain evaluation results.

[0089] For example, if the first reasoning request is "How many planets are there in the solar system?", the first model can determine the true answer as "9" based on the first dataset. The first model can determine the first reasoning result as "9", and the first model can also determine the first reasoning result as "8". This application embodiment does not make specific limitations.

[0090] In some implementations, after the first model loaded by the target inference service determines the first inference result, the electronic device can receive the first inference result based on the evaluation environment.

[0091] S305: Perform the first evaluation task on the first model based on the evaluation environment and the first inference result.

[0092] It is understandable that the first evaluation task of the first model can be the task of evaluating the prediction accuracy of the first model.

[0093] In some implementations, after receiving the first inference result from the first inference service interface based on the evaluation environment, the electronic device can evaluate the prediction accuracy of the first model based on the evaluation environment and the first inference result. For example, the electronic device can determine the prediction accuracy of the first model based on the ground truth data and predicted data in the first dataset corresponding to the first model.

[0094] In the case where the first model is a large language model, for example, such as Figure 4B As shown, in the process of evaluating the large language model, the truth data (i.e., the real answer) corresponding to the large language model can be "I think the person in your question is Bruce Lee, and he is practicing martial arts." The first inference result determined by Model 1 can be "This is Bruce Lee, and he is practicing martial arts," and the prompt can be "Known answer: xxx, the model output answer is: xxx, please output a score of 0-100 for Model 1's answer according to the xx standard." Thus, the electronic device can output the score "90".

[0095] Model 2's first inference result could be "The figures in the picture are fighting; he is Bruce Lee," with the prompt "Known answer: xxx, model output answer: xxx, please output a score of 0-100 for Model 2's answer according to xx standard." Thus, the electronic device could output the score "88".

[0096] Model 3's first reasoning result could be "I believe the person in your question is Jackie Chan, and he is practicing martial arts," with the prompt "Known answer: xxx, model output answer: xxx, please output a score of 0-100 for Model 3's answer according to xx standard." Thus, the electronic device could output the score "50".

[0097] In this embodiment of the application, by pre-constructing the inference environment corresponding to each model, the inference environment corresponding to the model is isolated from the evaluation environment. Thus, when evaluating the model, the inference result can be determined based on the inference environment corresponding to the model, and the model can be evaluated based on the evaluation environment. This decouples the inference process and the evaluation process of the model, so that the evaluation environment can be adapted to the evaluation of various models.

[0098] The following section uses an application scenario that evaluates model n out of multiple models as an example to provide a detailed description of the evaluation method mentioned in the embodiments of this application.

[0099] like Figure 5 The diagram shows a flowchart of an evaluation method that can be performed by an electronic device. Exemplarily, the evaluation method may include:

[0100] S501: Initiate model n evaluation.

[0101] In some implementations, the electronic device can initiate the evaluation process for any one of multiple models. For example, the electronic device can initiate the evaluation process for model n.

[0102] S502: Activate the base evaluation environment.

[0103] In some implementations, upon detecting the start of the evaluation process for model n, the electronic device can activate the base evaluation environment, i.e., run the evaluation environment.

[0104] S503: Activate the inference environment of model n.

[0105] In some implementations, after activating the base evaluation environment, the electronic device can determine whether the inference service interface of model n is started. If the inference service interface of model n is not started, the electronic device can activate the inference environment of model n, that is, run the inference environment corresponding to model n.

[0106] S504: Start the inference service API for model n.

[0107] In some implementations, after activating the inference environment of model n, the electronic device can launch the inference service API of model n based on the inference environment of model n. That is, the inference service interface corresponding to model n is launched based on the inference environment corresponding to model n.

[0108] S505: The inference service API for model n has been successfully started.

[0109] In some implementations, after the inference service API of model n is started, the electronic device can wait for the inference service API of model n to finish starting.

[0110] S506: Enable the target inference service among the multiple inference services corresponding to model n.

[0111] In some implementations, once it is determined that the inference service API of model n has been started, the electronic device can call the inference service API corresponding to model n based on the base evaluation environment, and send an activation request to the target inference service among the multiple inference services corresponding to model n, so as to activate the target inference service among the multiple inference services corresponding to model n.

[0112] S507: Waiting for the target inference service among the multiple inference services corresponding to model n to start.

[0113] In some implementations, after sending an enable request to the target inference service among the multiple inference services corresponding to model n, it is possible to wait for the target inference service among the multiple inference services corresponding to model n to start.

[0114] S508: Call model n to complete the inference and evaluate.

[0115] In some implementations, once the target inference service among the multiple inference services corresponding to model n has started, the target inference service can load model n. Model n can then respond to inference requests and determine the inference result based on its inference environment. Furthermore, the electronic device can receive the inference result from the inference service API of model n based on the base evaluation environment, and evaluate the prediction accuracy of model n based on the base evaluation environment and the inference result.

[0116] S509: Assessment completed.

[0117] In some implementations, after evaluating the prediction accuracy of model n, the electronic device can wait for the evaluation of the prediction accuracy of model n to be completed.

[0118] S510: Close the target inference service among the multiple inference services corresponding to model n.

[0119] In some implementations, after completing the prediction accuracy evaluation of model n, the electronic device can call the inference service API corresponding to model n based on the base evaluation environment, and send a shutdown instruction to the target inference service among the multiple inference services corresponding to model n, so as to shut down the target inference service among the multiple inference services corresponding to model n.

[0120] The evaluation environment and inference environment involved in the embodiments of this application are used to evaluate the model, and the evaluation result can be determined. This avoids the problem that the model evaluation process cannot run normally due to the conflict between the inference environment and the evaluation environment. Since the evaluation environment is relatively independent after decoupling, it can save memory and make the evaluation environment adaptable to the evaluation of various models, thereby improving the user experience.

[0121] It's understandable that the vLLM-based loading method is a high-performance inference engine specifically designed for large models, and the Transformers-based loading method is typically deployed in the model's inference environment. Therefore, after activating the inference environment of model n, the electronic device can determine whether model n supports the vLLM-based loading method. If model n supports the vLLM-based loading method, it can be loaded using the vLLM library to improve its inference efficiency. If model n does not support the vLLM-based loading method, it can be loaded using the Transformers library to ensure that model n can be loaded.

[0122] like Figure 6 As shown, a flowchart of another evaluation method is illustrated. Figure 6 S601 to S603, S607 to S613 in the illustrated embodiment and Figure 5S501 to S510 in the illustrated embodiment are similar, except that:

[0123] S604: Determine whether model n supports loading based on the vLLM library.

[0124] In some implementations, after the multi-process inference service for the first model is deployed, the inference service among the multiple inference services corresponding to the first model can determine whether the first model supports loading based on the vLLM library, and load the first model according to the loading method supported by the first model. Then, the inference service among the multiple inference services corresponding to the first model can enter a ready state, continuously waiting for inference requests sent based on the evaluation environment, so as to respond in a timely manner and generate inference results.

[0125] S605: Loading model n based on vLLM library.

[0126] In some implementations, if model n supports loading based on the vLLM library, the electronic device can determine that model n can be loaded based on the vLLM library and an instance of model n can be generated.

[0127] S605: Load model n based on the Transformers library.

[0128] In some implementations, if model n does not support loading based on the vLLM library, the electronic device can determine that model n can be loaded based on the Transformers library and generate an instance of model n accordingly.

[0129] In this embodiment of the application, by integrating a loading method based on the vLLM library on top of supporting the loading method based on the transformers library, the performance of fast model loading based on the vLLM library can be utilized to improve the inference efficiency of model n.

[0130] The following section compares the inference process of existing models with that of the model in this application to illustrate the effect of pre-constructing the inference service interfaces corresponding to each model in the embodiments of this application.

[0131] In some existing model inference processes, electronic devices can employ a collaborative mechanism of multi-threaded request distribution and multi-process model instance loading to achieve high concurrency in the model inference process. Multi-threaded request distribution refers to sending inference requests to multiple processes via multiple threads. Multi-process model instance loading refers to loading a model instance in each of the multiple processes.

[0132] In one possible implementation, after the model is deployed and an executable model instance is generated, the main process in the electronic device can start multiple child processes and control each child process to independently load the model instance. Furthermore, the main process can simultaneously create multiple threads to receive external inference requests and distribute these requests to multiple processes in a multi-threaded manner. Meanwhile, the main process continuously queries the running status of all child processes, such as whether the model instance has been loaded successfully and whether inference has been completed.

[0133] For example, such as Figure 7A As shown, after the model is deployed and an executable model instance is generated, the electronic device can start child processes process2, process3 and process4 through the main process process1. At the same time, it can create threads thread1, thread2, thread3 and thread4. Multiple threads can be used to send inference requests to multiple processes through multiple model inference services, thereby realizing the sending of inference requests to the main process and child processes in a multi-threaded manner.

[0134] For example, thread1 sends an inference request to inference service 1 corresponding to the model, and inference service 1 sends the inference request to the main process process1; thread2 sends an inference request to inference service 2 corresponding to the model, and inference service 2 sends the inference request to the child process process2; thread3 sends an inference request to inference service 3 corresponding to the model, and inference service 3 sends the inference request to the child process process3; thread4 sends an inference request to inference service 4 corresponding to the model, and inference service 4 sends the inference request to the child process process4.

[0135] Then, the model instance loaded by each process (including the main process process1, child process process2, child process process3 and child process4) can process the received inference requests and summarize the inference results to the main process process1. The main process process1 can then write the inference results into the storage device for subsequent model evaluation.

[0136] During this process, it is necessary to determine whether the process is the main process each time the inference result is written to the storage device. Furthermore, the main process needs to poll the status of child processes to determine whether the child processes have completed inference. This increases the complexity of model inference.

[0137] However, in the embodiments of this application, such as Figure 7BAs shown, after the model is deployed and an executable model instance is generated, the electronic device can create threads thread1, thread2, thread3, and thread4 to send inference requests to the server. The server processes the received inference requests from each thread and generates inference results res1, res2, res3, and res4 respectively. These inference results are then fed back to the inference result module, completing the closed loop of multi-threaded request and result return.

[0138] Thus, by pre-building inference service interfaces corresponding to each model to receive inference requests or obtain inference results, it is only necessary to call the inference service interface corresponding to the model to receive inference requests or obtain inference results, without needing to... Figure 7A The method shown determines whether the process is the main process each time the inference result is written to the storage device, without needing to check whether the child process has completed inference. This reduces the complexity of the model's inference process. Furthermore, by pre-constructing inference environments for each model and isolating them from the evaluation environment, the inference result can be determined based on the model's inference environment, and the model can be evaluated based on the evaluation environment. This decouples the inference and evaluation processes, allowing the evaluation environment to adapt to various model evaluations.

[0139] The following section describes the specific implementation method for generating the prompt words carried in the first inference request.

[0140] It is understandable that, if the category of the first dataset is the first category, for example, if the first dataset belongs to the VLM dataset or the LLM dataset, the electronic device can directly generate the prompt words carried in the first inference request based on the first dataset.

[0141] When the category of the first dataset is the second category, for example, when the first dataset belongs to the GUI dataset or Agent dataset under the Task dimension, the electronic device can divide the first dataset into multiple data subsets and construct hierarchical prompt words carried in the first inference request based on the data content of each data subset.

[0142] In some implementations, the electronic device can acquire the task requirements labeled in the first dataset and, based on these requirements, determine a target evaluation method that matches the task requirements from a set of preset evaluation methods. These preset evaluation methods may include parallel evaluation, serial evaluation, plan evaluation, and tool call evaluation. Then, the electronic device can divide the first dataset into multiple data subsets corresponding to multiple steps in the task, based on the task execution logic corresponding to the target evaluation method, and construct hierarchical prompts carried in the first inference request based on the data content of each subset.

[0143] For example, based on the data category and data content of the first dataset, the prompt word carried in the first inference request is constructed using any of the following methods:

[0144] In Method 1, when a task involves multiple steps, including preceding steps and the current step, and each data subset contains both input data and truth data, if the target evaluation method that matches the task requirements is determined to be a serial evaluation method, the electronic device can use the truth data in the data subset corresponding to the preceding steps as historical information, and construct the hierarchical prompt words corresponding to the current step based on the historical information and the input data in the data subset corresponding to the current step.

[0145] For example, if the target evaluation method that matches the task requirements is determined to be a sequential evaluation method, this sequential evaluation method includes task1. Figure 8 As shown, task1 can include a first step, a second step, and a third step. The data content in the data subset corresponding to the first step, step1, includes a first image-text pair and a first truth value. The data content in the data subset corresponding to the second step, step2, includes a second image-text pair and a second truth value. The data content in the data subset corresponding to the third step, step3, includes a third image-text pair and a third truth value.

[0146] When the evaluation begins in the first step (step1), the first model can provide a model response based on the first input image-text pair, thus obtaining the reasoning result corresponding to the first step (step1).

[0147] When evaluating at the start of step 2, step 1 can be a preceding step, and step 2 can be the current step. The first model can use the first truth data in the data subset corresponding to step 1 as historical information, and construct new prompt words based on the historical information and the second image-text pair in the data subset corresponding to step 2. That is, the historical information is used to construct new prompt words. Then, the first model can perform model responses based on the input new prompt words to obtain the inference result corresponding to step 2.

[0148] When evaluating at the start of step 3, step 2 can be a preceding step, and step 3 can be the current step. The first model can use the second ground truth data in the data subset corresponding to step 2 as historical information, and construct new prompt words based on the historical information and the third image-text pair in the data subset corresponding to step 3. That is, historical information is used to construct new prompt words. Then, the first model can perform model responses based on the input new prompt words to obtain the inference result corresponding to step 3.

[0149] Method 2: In a task with multiple steps including preceding steps and the current step, where each data subset contains both input data and truth data, if the target evaluation method that matches the task requirements is determined to be a parallel evaluation method, the electronic device can use the reasoning results generated by the preceding steps based on the input data in the corresponding data subset as historical information, and generate hierarchical prompt words for the current step based on the historical information and the input data in the corresponding data subset.

[0150] For example, if the target evaluation method that matches the task requirements is determined to be a parallel evaluation method, then the serial evaluation method includes task1. For example... Figure 9 As shown, task1 can include a first step, a second step, and a third step. The data content in the data subset corresponding to the first step, step1, includes a first image-text pair and a first truth value. The data content in the data subset corresponding to the second step, step2, includes a second image-text pair and a second truth value. The data content in the data subset corresponding to the third step, step3, includes a third image-text pair and a third truth value.

[0151] When the evaluation begins in the first step (step1), the first model can provide a model response based on the first input image-text pair, thus obtaining the reasoning result corresponding to the first step (step1).

[0152] When evaluating at the start of step 2, step 1 can be a preceding step, and step 2 can be the current step. The first model can use the inference result corresponding to step 1 as historical information, and construct new prompt words based on the historical information and the second image-text pair in the data subset corresponding to step 2. That is, historical information is used to construct new prompt words. Then, the first model can perform model responses based on the input new prompt words to obtain the inference result corresponding to step 2.

[0153] When evaluating at the start of step 3, step 2 can be a preceding step, and step 3 can be the current step. The first model can use the inference result corresponding to step 2 as historical information, and construct new prompt words based on the historical information and the third image-text pair in the data subset corresponding to step 3. That is, historical information is used to construct new prompt words. Then, the first model can perform model responses based on the input new prompt words to obtain the inference result corresponding to step 3.

[0154] Method 3: In a task with multiple steps, including the current step, and each data subset containing both input data and truth data, if the target evaluation method matching the task requirements is determined to be either plan evaluation or tool call evaluation, the electronic device can use the inference result generated by the current step based on the input data in the corresponding data subset as guidance information, and generate hierarchical prompts for the current step based on the guidance information and the input data in the corresponding data subset.

[0155] For example, such as Figure 10 As shown, the first model can acquire input information containing GUI (Graphical User Interface) information, which may include questions and images. Then, the first model can perform a model-based response based on the GUI information to obtain a reasoning result. This reasoning result may include the thought process (a), the answer (b), and a summary (c).

[0156] For example, such as Figure 11 As shown, the first model can obtain input information containing GUI graphic information, which may include the question "Order me a coffee" and an image. Then, the inference service 1 corresponding to the first model can generate low-level instructions based on the GUI graphic information, such as "Click on XX on the screen". The low-level instruction is then used as a prompt word, and a new prompt word is formed based on the prompt word and the image to instruct the inference service 2 corresponding to the first model to perform inference and output the corresponding execution behavior, such as "click(x,y)".

[0157] It is understandable that during the process of S305 performing the first evaluation task on the first model based on the evaluation environment and the first inference result, the electronic device can determine the evaluation result based on the true value data and prompt words in the dataset corresponding to the model.

[0158] The following is combined Figure 12 The flowchart shown here illustrates the model reasoning and evaluation process, and introduces the evaluation method mentioned in the embodiments of this application.

[0159] like Figure 12As shown, the model inference and evaluation process can include the model service deployment stage, the data loading and classification stage, and the model inference and evaluation stage.

[0160] (1) During the model service deployment phase

[0161] After detecting the start of the model evaluation process, the electronic device can determine the loading methods supported by the model and whether the model supports the loading method based on the vLLM library.

[0162] When the model supports loading based on the vLLM library, multiple inference services corresponding to the model are started through the openai third-party library, and the output of multiple inference services is encapsulated through FastAPI to obtain the concurrent inference interface and expose the concurrent inference interface to the outside. Thus, the inference service is deployed and multiple inference services corresponding to the model are obtained.

[0163] If the model does not support loading via the vLLM library, the electronic device can deploy multiple inference services corresponding to the model locally through multiple processes. The output of these services is then encapsulated using FastAPI to obtain concurrent inference interfaces, which are then exposed externally. This completes the deployment of the inference services, resulting in multiple inference services corresponding to the model. The inference services can then enter a ready state, continuously waiting for inference requests sent based on the evaluation environment, in order to respond promptly and generate inference results.

[0164] (2) Data loading and classification stage

[0165] After the inference service has been deployed, the target inference service can send a data loading request to the storage device (e.g., memory) and obtain the dataset corresponding to the model based on the evaluation environment, thus completing the dataset loading. Then, the target inference service can determine the category of the dataset and perform differentiated prompt word construction for different categories of datasets.

[0166] For example, for the VLM dataset, image-text combined prompts are constructed based on the features of image-text pairs to adapt to the VLM model. For another example, for the LLM dataset, text prompts are constructed based on plain text content to adapt to the LLM model. For yet another example, for the GUI / Agent dataset, the evaluation method (parallel evaluation, serial evaluation, plan / tool ​​call evaluation) is first selected, then the MagicEval code box is invoked to load the corresponding task execution logic, dividing the first dataset into multiple data subsets corresponding to multiple steps in the task, and constructing hierarchical prompts carried in the first inference request based on the data content in each data subset.

[0167] It is understandable that the specific implementation of constructing the prompt words can be referred to the above-described implementation of the prompt words carried in the first inference request, and will not be repeated here.

[0168] (3) Model reasoning and evaluation stage.

[0169] Input the prompt words into the inference service, perform inference calculations, and generate an inference result (pred). Store the inference result in an entity file to form a traceable inference record for easy review and analysis later.

[0170] Next, the evaluation metric type of the inference result is determined to see if subjective evaluation metrics (such as semantic coherence, content rationality, etc., which cannot be directly quantified by the algorithm) are involved. If no subjective evaluation metrics are involved, the similarity between the inference result (pred) and the ground truth data (gt) is directly calculated using the algorithm to obtain objective evaluation metrics such as accuracy, recall, and F1 score. If subjective evaluation metrics are involved, the referee model deployment process is initiated: first, a pre-trained referee model is deployed; then, the referee model interface is encapsulated through FastAPI and the referee service is started. After the referee service deployment is completed, the inference result (pred) and the ground truth data (gt) are integrated into standardized prompt words and input into the referee model, which then outputs objective subjective evaluation metrics.

[0171] Figure 13 According to some embodiments of this application, a structural block diagram of an electronic device 100 to which the evaluation method provided in this application is applicable is shown.

[0172] Specifically, such as Figure 13 As shown, the electronic device 100 includes one or more processors 104, system control logic 108 connected to at least one of the processors 104, system memory 112 connected to the system control logic 108, non-volatile memory (NVM) 116 connected to the system control logic 108, and network interface 120 connected to the system control logic 108.

[0173] In some embodiments, processor 104 may include one or more single-core or multi-core processors. In some embodiments, processor 104 may include any combination of general-purpose processors and special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In embodiments where electronic device 100 employs an Evolved Node B (ENB) or Radio Access Network (RAN) controller, processor 104 may be configured to perform various corresponding embodiments.

[0174] In some embodiments, the processor 104 may pre-build inference environments corresponding to each model. When evaluating a model, it runs the evaluation environment and the inference environment corresponding to the model, starts the inference service interface corresponding to the model based on the inference environment, calls the inference service interface to initiate an inference request, receives the inference results from the inference service interface based on the evaluation environment, and performs the evaluation task of the model based on the evaluation environment and the inference results.

[0175] In this way, by pre-constructing the inference environment corresponding to each model and isolating the inference environment from the evaluation environment, the inference result can be determined based on the inference environment when evaluating the model, and the model can be evaluated based on the evaluation environment. This approach decouples the model's inference process from the evaluation process, thereby avoiding the problem of the model's evaluation process failing to operate normally due to conflicts between the inference environment and the evaluation environment. Since the evaluation environment is relatively independent after decoupling, it can save memory and make the evaluation environment adaptable to the evaluation of various models, thereby improving the user experience.

[0176] In some embodiments, system control logic 108 may include any suitable interface controller to provide any suitable interface to at least one of the processors 104 and / or any suitable device or component communicating with system control logic 108.

[0177] In some embodiments, system control logic 108 may include one or more memory controllers to provide an interface to system memory 112. System memory 112 may be used to load and store data and / or instructions. In some embodiments, system memory 112 of electronic device 100 may include any suitable volatile memory, such as suitable dynamic random access memory (DRAM). In some embodiments, system memory 112 may be used to load or store instructions for implementing the evaluation methods described above.

[0178] NVM / memory 116 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, NVM / memory 116 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as at least one of a hard disk drive (HDD), a compact disc (CD) drive, and a digital versatile disc (DVD) drive. NVM / memory 116 may also be used to store the inference results and evaluation results from the evaluation methods described above.

[0179] NVM / Memory 116 may include a portion of the storage resources on the device on which Electronic Device 100 is installed, or it may be accessible by the device, but is not necessarily part of the device. For example, NVM / Memory 116 may be accessed over a network via Network Interface 120.

[0180] Specifically, system memory 112 and NVM / memory 116 may each include a temporary copy and a permanent copy of instruction 124. Instruction 124 may include, when executed by at least one of processors 104, causing electronic device 100 to perform, as Figure 2 The instructions for the evaluation method are shown. In some embodiments, instructions 124, hardware, firmware and / or their software components may additionally / alternatively be located in system control logic 108, network interface 120 and / or processor 104.

[0181] Network interface 120 may include a transceiver for providing a radio interface to electronic device 100, thereby enabling communication with any other suitable device (such as a front-end module, antenna, etc.) via one or more networks. In some embodiments, network interface 120 may be integrated into other components of electronic device 100. For example, network interface 120 may be integrated into at least one of processor 104, system memory 112, NVM / memory 116, and firmware device (not shown) with instructions that, when at least one of processor 104 executes the instructions, electronic device 100 implements as follows: Figure 2 The evaluation method shown.

[0182] The network interface 120 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, the network interface 120 may be a network adapter, a wireless network adapter, a telephone modem, and / or a wireless modem.

[0183] In some embodiments, at least one of the processors 104 may be packaged together with the logic of one or more controllers for system control logic 108 to form a system-in-package (SIP). In some embodiments, at least one of the processors 104 may be integrated on the same die with the logic of one or more controllers for system control logic 108 to form a system-on-a-chip (SOC).

[0184] The electronic device 100 may further include an input / output (I / O) device 132. The I / O device 132 may include a user interface that enables a user to interact with the electronic device 100; the peripheral component interface is designed so that peripheral components can also interact with the electronic device 100.

[0185] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0186] This application provides an evaluation system, which includes an evaluation environment and at least one inference environment. The evaluation environment is used to invoke a first inference service interface corresponding to a first model launched by a first inference environment within the at least one inference environment to initiate a first inference request. The first inference environment within the at least one inference environment is used to perform model inference and send a first inference result to the evaluation environment through the first inference service interface. The evaluation environment is used to receive the inference result from the first inference service interface and perform an evaluation task on the first model based on the first inference result. Its implementation principle and technical effects are similar to those of the related embodiments of the above method, and will not be repeated here.

[0187] This application provides a computer program product that, when run on a device, causes the device to execute the technical solutions described in the above embodiments. Its implementation principle and technical effects are similar to those of the related embodiments described above, and will not be repeated here.

[0188] This application provides a readable storage medium containing instructions that, when executed on a device, cause the device to perform the technical solutions described in the above embodiments. The implementation principle and technical effects are similar and will not be repeated here.

[0189] This application provides a chip for executing instructions. When the chip is running, it executes the technical solutions described in the above embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0190] It is understood that the various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0191] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0192] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. The mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0193] The above describes the possible hardware structures of electronic devices. It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of both.

[0194] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0195] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the aforementioned element.

[0196] Although this application has been illustrated and described with reference to certain embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made thereto without departing from the scope of this application.

Claims

1. An evaluation method, characterized in that, Applied to electronic devices, including: The evaluation environment and the first inference environment corresponding to the first model are run; The first inference service interface corresponding to the first model is started based on the first inference environment; Based on the evaluation environment, the first inference service interface is invoked to initiate a first inference request; The evaluation environment receives a first inference result from the first inference service interface, wherein the first inference result is determined by the first model based on the first inference environment. Based on the evaluation environment and the first inference result, perform a first evaluation task on the first model.

2. The method according to claim 1, characterized in that, The operational evaluation environment and the first inference environment corresponding to the first model include: Run the evaluation environment; If the first inference service interface is not started, the first inference environment corresponding to the first model is run.

3. The method according to claim 1 or 2, characterized in that, The step of initiating a first inference request by calling the first inference service interface based on the evaluation environment includes: Based on the evaluation environment, the first inference service interface is invoked to initiate the first inference request to the target inference service among the multiple inference services corresponding to the first model.

4. The method according to claim 3, characterized in that, The step of initiating the first inference request to the target inference service among the multiple inference services corresponding to the first model includes: Obtain the queue information for each of the multiple inference services; The inference service whose queue information content is less than the information content threshold among the multiple inference services is identified as the target inference service. Initiate the first inference request to the target inference service.

5. The method according to claim 4, characterized in that, Before initiating the first inference request by calling the first inference service interface based on the evaluation environment, the method further includes: Based on the evaluation environment, obtain the first dataset corresponding to the first model; Based on the data categories and content of the first dataset, construct the prompt words carried in the first inference request.

6. The method according to claim 5, characterized in that, The first inference result was determined in the following manner: If the first model supports loading based on the visual large language model library, the target inference service loads the first model based on the visual large language model library; If the first model does not support the loading method based on the visual large language model library, the target inference service loads the first model based on the Transformers library; The first model responds to the first inference request and determines the first inference result based on the first inference environment and the prompt words in the first inference request.

7. The method according to claim 5, characterized in that, The step of constructing the prompt words carried in the first inference request based on the data category and data content of the first dataset includes: If the first dataset belongs to a visual language model dataset or a large language model dataset, construct the prompt words carried in the first inference request based on the data content in the first dataset; If the first dataset belongs to a graphical user interface dataset or an agent dataset under the task dimension, the first dataset is divided into multiple data subsets, and based on the data content in each data subset, hierarchical prompt words carried in the first inference request are constructed.

8. The method according to claim 7, characterized in that, The step of dividing the first dataset into multiple data subsets and constructing hierarchical prompt words carried in the first inference request based on the data content of each data subset includes: The task requirements for annotating the first dataset; Based on the task requirements, a target evaluation method matching the task requirements is determined from the preset evaluation methods; wherein, the preset evaluation methods include parallel evaluation method, serial evaluation method, planning evaluation method, and tool invocation evaluation method; Based on the task execution logic corresponding to the target evaluation method, the first dataset is divided into multiple data subsets corresponding to multiple steps in the task; Based on the data content of each data subset, the hierarchical prompt words carried in the first inference request are constructed.

9. The method according to claim 8, characterized in that, The task comprises multiple steps, including preceding steps and the current step. The data content in each data subset includes input data and truth data. The step of constructing hierarchical prompt words carried in the first inference request based on the data content of each data subset includes: When the target evaluation method is a serial evaluation method, the truth data in the data subset corresponding to the preceding steps is used as historical information; Based on the historical information and the input data in the data subset corresponding to the current step, construct the hierarchical prompt words corresponding to the current step.

10. The method according to claim 8, characterized in that, The task comprises multiple steps, including preceding steps and the current step. The data content in each data subset includes input data and truth data. The step of constructing hierarchical prompt words carried in the first inference request based on the data content of each data subset includes: When the target evaluation method is a parallel evaluation method, the reasoning result generated by the preceding step based on the input data in the data subset corresponding to the preceding step is used as historical information; Based on the historical information and the input data in the data subset corresponding to the current step, construct the hierarchical prompt words corresponding to the current step.

11. The method according to claim 8, characterized in that, The task comprises multiple steps, including the current step, and each subset of data includes input data and truth data. The step of constructing hierarchical prompt words carried in the first inference request based on the data content of each data subset includes: When the target evaluation method is the planning evaluation method or the tool invocation evaluation method, the reasoning result generated by the current step based on the data content in the data subset corresponding to the current step is used as guiding information; Based on the guidance information and the data content in the data subset corresponding to the current step, construct the hierarchical prompt words corresponding to the current step.

12. An evaluation system, characterized in that, The evaluation system includes an evaluation environment and at least one reasoning environment. The evaluation environment is used to call the first inference service interface corresponding to the first model launched based on the first inference environment in the at least one inference environment to initiate a first inference request. The first inference environment in the at least one inference environment is used to perform model inference and send the first inference result to the evaluation environment through the first inference service interface; The evaluation environment is used to receive inference results from the first inference service interface and perform an evaluation task on the first model based on the first inference results.

13. An electronic device, characterized in that, Includes: a memory for storing instructions executed by one or more processors of the electronic device. And a processor, one of one or more processors of the electronic device, for performing the evaluation method according to any one of claims 1 to 11.

14. A readable storage medium, characterized in that, The readable storage medium stores instructions that, when executed on an electronic device, cause the electronic device to perform the evaluation method according to any one of claims 1 to 11.

15. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed by an electronic device, enable the electronic device to perform the evaluation method as described in any one of claims 1 to 11.

16. A chip, characterized in that, The chip includes a processor coupled to a memory for executing a computer program or instructions stored in the memory, such that the chip implements the evaluation method of any one of claims 1 to 11.

Images