Method, device, electronic device and computer program product for large model evaluation

Abstract

Embodiments of the present disclosure relate to a method, apparatus, electronic device, and computer program product for large model evaluation, which are used in the field of large models. The method includes sending a request for executing the selected evaluation strategy based on user input indicating the selection of an evaluation strategy for evaluating a target model from a plurality of evaluation strategies. In addition, the method also includes obtaining an execution result of the request, the execution result including at least an evaluation result of the target model. Through the embodiments of the present disclosure, model evaluation can be completed efficiently, evaluation efficiency can be improved, the model development cycle can be shortened, model iteration can be accelerated, and the difficulty of developing model evaluation tasks can be reduced, enabling more users to participate in the model evaluation process.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more particularly to a method, device, electronic device, and computer program product for large model evaluation. Background Art

[0002] Large models, trained on massive amounts of data, possess powerful generalization capabilities and exceptional performance, enabling them to handle complex tasks and diverse application scenarios. Furthermore, they have demonstrated tremendous potential and application prospects in fields such as medicine, finance, and autonomous driving, driving the comprehensive development and widespread application of AI technology and becoming a key driver of current scientific and technological innovation.

[0003] Large models are increasingly being used in various fields, and their performance and reliability directly impact their practical application outcomes. Model evaluation can reveal performance differences among large models in different application scenarios and provide a basis for model optimization. Therefore, evaluating large models has become particularly important. Summary of the Invention

[0004] Embodiments of the present disclosure provide a method, apparatus, electronic device, computer program product, and medium for large model evaluation.

[0005] According to a first aspect of the present disclosure, a method for large model evaluation is provided. The method includes sending a request to execute the selected evaluation strategy based on user input indicating selection of an evaluation strategy for evaluating a target model from a plurality of evaluation strategies, wherein the plurality of evaluation strategies are published on a policy system, and the policy system is configured to: create a policy file for the evaluation strategy, wherein the policy file is stored in a database; set dependencies required for executing the evaluation strategy; and publish the evaluation strategy to a policy service in the policy system. In addition, the method also includes obtaining an execution result of the request, wherein the execution result includes at least an evaluation result of the target model.

[0006] According to the second aspect of the present disclosure, a device for large model evaluation is provided. The device includes a request sending module, which is configured to send a request for executing the selected evaluation strategy based on user input indicating that an evaluation strategy for evaluating a target model is selected from multiple evaluation strategies, wherein the multiple evaluation strategies are published on a policy system, and the policy system includes: a policy creation module, which is configured to create a policy file for the evaluation strategy, and the policy file is stored in a database; a dependency setting module, which is configured to set the dependencies required for executing the evaluation strategy; and a policy publishing module, which is configured to publish the evaluation strategy to the policy service in the policy system. In addition, the device also includes a result acquisition module, which is configured to obtain the execution result of the request, and the execution result at least includes the evaluation result of the target model.

[0007] According to a third aspect of the present disclosure, an electronic device is provided, comprising a processor and a memory coupled to the processor, wherein the memory has instructions stored therein, and when the instructions are executed by the processor, the electronic device executes the method according to the first aspect.

[0008] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided. The computer program product is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions that, when executed, cause a computer to perform the steps of the method of the first aspect of the present disclosure.

[0009] In a fifth aspect of the present disclosure, a computer-readable storage medium is provided, wherein one or more computer instructions are stored on the computer-readable storage medium, wherein the one or more computer instructions are executed by a processor to implement the method according to the first aspect.

[0010] This summary is intended to introduce a selection of concepts in a simplified form that are further described below in the detailed description. It is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0012] Figure 1 A schematic diagram illustrating an example environment in which devices and / or methods according to embodiments of the present disclosure may be implemented;

[0013] Figure 2A flowchart illustrating a method for large model evaluation according to an embodiment of the present disclosure is shown;

[0014] Figure 3 A schematic diagram showing the architecture of a model evaluation and strategy system according to an embodiment of the present disclosure;

[0015] Figure 4A A flowchart illustrating a process of issuing an evaluation policy according to an embodiment of the present disclosure is shown;

[0016] Figure 4B A schematic diagram showing a user interface for creating a function according to an embodiment of the present disclosure is shown;

[0017] Figure 4C A schematic diagram showing an object relationship of a context object according to an embodiment of the present disclosure;

[0018] Figure 5A A flowchart illustrating a process for performing model evaluation according to an embodiment of the present disclosure is shown;

[0019] Figure 5B A schematic diagram showing a user interface for creating an assessment task according to an embodiment of the present disclosure

[0020] Figure 5C A schematic diagram showing an execution result of a policy request according to an embodiment of the present disclosure;

[0021] Figure 6 A block diagram illustrating an apparatus for large model evaluation according to an embodiment of the present disclosure; and

[0022] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0023] Throughout the drawings, the same or similar reference numbers denote the same or similar elements. DETAILED DESCRIPTION

[0024] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information (such as voice) involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0025] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0026] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. can refer to different or the same objects, unless explicitly stated otherwise. Other explicit and implicit definitions may also be included below.

[0027] As mentioned earlier, the development of large models is accelerating, and the demand for their evaluation is also increasing. Technologies for large model evaluation require users to develop an evaluation strategy for each model to be evaluated and deploy it after development is complete, making this approach inefficient. Furthermore, large models typically require multiple rounds of optimization and continuous updates, which must be repeated after each optimization and update, significantly impacting the efficiency of large model development and optimization.

[0028] To this end, embodiments of the present disclosure propose a large-scale model evaluation solution. This solution evaluates the target model by selecting an evaluation strategy from multiple evaluation strategies. The evaluation system can provide many evaluation strategies to choose from, meeting the model evaluation requirements in different situations. In this way, only a request to execute the evaluation strategy is required to obtain the corresponding evaluation results, thereby completing the evaluation of the target model. Therefore, the model evaluation solution proposed by the embodiments of the present disclosure can efficiently complete model evaluation, improve evaluation efficiency, shorten the model development cycle, accelerate model iteration, and reduce the difficulty of developing model evaluation tasks, enabling more people to participate in the model evaluation process.

[0029] Figure 1 Schematic diagram of an example environment in which devices and / or methods according to embodiments of the present disclosure may be implemented. Figure 1 As shown, example environment 100 may include an evaluation and policy system 110, which may include an evaluation system 120 and a policy system 130. Evaluation system 120 may receive user input 140, which may indicate an evaluation policy 122 selected from a plurality of evaluation policies for evaluating a target model. For example, evaluation policy 122 may include a target model to be evaluated, an evaluation dataset, and a process for evaluating the target model. Evaluation system 120 may send a policy request 124 to policy system 130 for executing evaluation policy 122. For example, policy request 124 may be a Hypertext Transfer Protocol (HTTP) request.

[0030] The policy system 130 can deploy a policy service, which can have multiple evaluation policies. In some embodiments, the policy service can be a Function as a Service (FaaS). Upon receiving the policy request 124, the policy system 130 can trigger the FaaS service to execute the corresponding evaluation policy. After executing the evaluation policy 122, the policy system 130 can transmit the execution result 126 to the evaluation system 120. The execution result 126 includes at least the evaluation result for the target model. For example, when evaluating the accuracy of the target model, the execution result 126 may include a measure of the accuracy of the target model. In some embodiments, the execution result 126 may also include the model input, model output, ground truth, and execution status for each evaluation data. In some embodiments, the content of the execution result 126 may be specified in the evaluation policy 122.

[0031] It should be understood that the architecture and functions in the example environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure. The embodiments of the present disclosure may also be applied to other environments with different structures and / or functions.

[0032] The following will be combined Figures 2 to 7 The process according to the embodiment of the present disclosure is described in detail. For ease of understanding, the specific data mentioned in the following description are exemplary and are not intended to limit the scope of protection of the present disclosure. It is understood that the embodiments described below may also include additional actions not shown and / or may omit the actions shown, and the scope of the present disclosure is not limited in this respect.

[0033] Figure 2 A flow chart of a method 200 for large model evaluation according to an embodiment of the present disclosure is shown. At block 202, based on user input indicating that an evaluation strategy for evaluating a target model is selected from a plurality of evaluation strategies, a request for executing the selected evaluation strategy may be sent. For example, referring to Figure 1 The evaluation system 120 may receive user input 140 indicating a selection of an evaluation strategy 122 from a plurality of evaluation strategies for evaluating a target model, and then the evaluation system 120 may send a request 124 for executing the selected evaluation strategy 122. The policy system 130 may be configured to create a policy file for the evaluation strategy 122 and store the policy file in a database, set dependencies required for executing the evaluation strategy 122, and publish the evaluation strategy 122 to a policy service in the policy system 130.

[0034] At block 204, the execution result of the request may be obtained, the execution result including at least the evaluation result of the target model. Figure 1The evaluation system 120 may obtain an execution result 126 of the request 124 , where the execution result 126 at least includes an evaluation result of the target model.

[0035] Therefore, according to the method 200 of the embodiment of the present disclosure, model evaluation can be completed efficiently, evaluation efficiency can be improved, the model development cycle can be shortened, the model iteration can be accelerated, and the difficulty of developing model evaluation tasks can be reduced, so that more people can participate in the model evaluation process.

[0036] Figure 3 FIG. 3 is a schematic diagram showing an architecture 300 of a model evaluation and strategy system according to an embodiment of the present disclosure. Figure 3 As shown, the model evaluation and strategy system 302 may include an evaluation system 304 and a strategy system 306. The evaluation system 304 may manage and execute model evaluation tasks, and the strategy system 306 may manage, deploy and execute evaluation strategies. User 308 may be a user with a large model evaluation requirement, which may include an administrator and an ordinary user. The data set module 310 may receive evaluation data uploaded by user 308. For example, the evaluation data may include questions for evaluating the large model. In addition, the data set module 310 may be used to annotate the evaluation data, such as the correct answer, type, source, and other information for each question in the evaluation data. In some embodiments, the original questions may be used to evaluate the large model, and the original questions may be updated during the evaluation process. In some embodiments, different evaluation tasks may use the same original questions. In some embodiments, different evaluation tasks may use different original questions.

[0037] The evaluation creation module 312 can receive user input from the user 308 to select one or more evaluation strategies from a plurality of evaluation strategies. In some embodiments, the evaluation strategy can include an execution strategy (also known as an API function) and a scoring strategy (also known as a scoring function). For example, when performing a large model evaluation, the API function can be executed first to complete each question in the evaluation data, and then the scoring function can be executed to generate the evaluation results for the large model. In some embodiments, the evaluation creation module 312 can receive the user 308's selection of an execution strategy and a scoring strategy. For example, the user can specify both the API function and the scoring function. Furthermore, the evaluation creation module 312 can receive additional user selections. For example, the user 308 can select one or more large models to be evaluated, one or more evaluation datasets, and so on. The evaluation creation module 312 creates an evaluation task based on the user input and then saves the evaluation task to the scheduling module 314. By separating the evaluation strategy into an execution strategy and a scoring strategy, the execution strategy can be executed once to obtain the output of the large model, and then different scoring strategies can be used to obtain different evaluation results. This avoids repeated execution of the large model, thereby improving evaluation efficiency.

[0038] In some embodiments, the scheduling module 314 can save one or more evaluation tasks. For example, when multiple users create multiple evaluation tasks, the scheduling module 314 can save the multiple evaluation tasks and then schedule them when conditions are met. After the scheduling module 314 schedules to the created evaluation task, the evaluation execution module 316 can execute the evaluation task. When the evaluation execution module 316 can execute the evaluation task, the operation policy request module 318 can send a request for executing an API function (for example, an HTTP request). For example, the API function (i.e., the operation policy) can include obtaining questions in the evaluation data, obtaining prompt content, setting answers to questions, obtaining standard answers to questions, and obtaining other front-end parameters, etc. The operation policy request module 318 can send a request for executing an API function to the policy service module 344 in the policy system 306 to access the selected API function.

[0039] In some embodiments, the policy service module 344 can be a FaaS service. FaaS service is a cloud computing service that allows users to write code in the form of functions and deploy it on a cloud platform. The platform is responsible for managing the execution environment, resource scheduling, and expansion of the function, while users do not need to manage the underlying infrastructure and only need to focus on the implementation of business logic. The embodiments of the present disclosure combine large model evaluation with FaaS services to achieve an efficient and scalable evaluation process, reduce costs and maintenance difficulties, provide flexible and fast iteration capabilities, and ensure the isolation and security of evaluation tasks, thereby improving overall evaluation efficiency and accuracy.

[0040] The evaluation data update module 320 can receive the result of executing the API function from the policy service module 344. For example, after the API function is executed, the execution result of the large model can be generated. The evaluation data update module 320 can update the evaluation data set based on the execution result of the large model and store it in the database 322. For example, the execution results of different large models can be written into the evaluation data set, and the large model name can be used as the field name of the corresponding execution result, so that the differences between the execution results of different large models can be compared. The scoring strategy request module 324 can send a request for executing the scoring function (i.e., the scoring strategy) to the policy service module 344, and the evaluation result processing module 326 can receive the result of executing the scoring function from the policy service module 344, thereby obtaining the evaluation result of the large model. The evaluation result processing module 326 can write the evaluation result of the large model into the database 322 and display the evaluation result on the user interface.

[0041] The policy management module 330 in the policy system 306 can receive a request from a user 328 to create an evaluation policy. User 328 can be the same user as user 308 or a different user. For example, an evaluation policy created by a user can be subsequently used by that user or another user (e.g., with authorization). This reuse of evaluation policies can significantly improve the evaluation efficiency of large models. The policy management module 330 can receive user input to create an evaluation policy 332, for example, by creating an API function or a Python file for the policy. The policy management module 330 can save file content 334, such as the content of the evaluation policy file uploaded by the user. The policy file can also utilize a toolkit 336 provided by the policy system 306. Toolkit 336 supports dataset management, environment variable calls, metadata retrieval, and other functions, improving the efficiency of user function file creation. The evaluation policy file can then be stored in a database 338, such as a MySQL database. For example, the evaluation policy file can be stored in a data table in the database 338, and its identifier key (i.e., ID key) in the data table can be used as an identifier for external access.

[0042] Policy management module 330 can be used to configure dependencies 340. For example, when writing API functions and scoring functions, multiple external dependencies are often required. Through policy management module 330, user 328 can easily add dependencies. In some embodiments, required dependencies can be added to a dependency file, thereby achieving space-level dependency isolation. For example, a dependency file called requirements.txt can be used to add required dependency packages (e.g., specifying the name and version number of the dependency package), eliminating the need for users to separately install and configure the dependency packages, thereby facilitating user use.

[0043] The policy management module 330 can be used to execute the release evaluation policy 342, for example, to the policy service module 344. As previously mentioned, the policy service module 344 can be a FaaS service, and it can also be other types of cloud computing services or event-driven services, which are not limited by the present disclosure. In some implementations, the release file 346 can include an evaluation file 348, a toolkit 350, an entry function 352, a service configuration file 354, and a dependency file 356. For example, the evaluation file 348 can be a file of the evaluation policy obtained after executing the save file content 334, which can be an API function file or a scoring function file; the toolkit 350 can be all or part of the toolkit 336; the entry function 352 can be used to route the entry of the evaluation file 348; the configuration file 354 can be a configuration file required by the policy service module 344; and the dependency file 356 can be a dependency file generated after executing the configuration dependency 340.

[0044] In addition, the policy management module 330 can publish function functions to the function plug-in market 358. The function plug-in market 358 can save commonly used function functions and encapsulate them into function plug-ins. Function functions are function functions that may be used when writing API functions or scoring functions. For example, user 328 can encapsulate the function of comparing whether two SQL queries are consistent into a function plug-in. The function plug-in market 358 can publish the function plug-in. Then, when writing API functions or scoring functions, it is only necessary to call the function plug-in. This process is similar to calling a local function, which makes it convenient for users to write functions. For example, when a function function is published to the function plug-in market 358, the policy system 306 can provide a list of function plug-ins and provide a template code for calling the function plug-in. In the process of writing the evaluation policy, by copying the template code, the function plug-in can be called just like calling a local function.

[0045] Figure 4A A flowchart of a process 400A for publishing an evaluation strategy according to an embodiment of the present disclosure is shown. Figure 3 The process of publishing an evaluation strategy can be done in Figure 3 The model evaluation and policy system 302 is executed on the policy system 306. As shown in Figure 4, at block 402, a policy file for the evaluation policy can be created. As previously mentioned, the evaluation policy, also known as the evaluation function, can include an execution policy (also known as an API function) and a scoring policy (also known as a scoring function). When performing a large model evaluation, the execution policy can be executed first to complete each question in the evaluation data, and then the scoring policy can be executed to generate the evaluation results of the large model. It should be understood that the creation of the evaluation policy described here includes the creation of the execution policy and the creation of the scoring policy.

[0046] For example, after receiving user input, the policy system can create a function file in the user space where the user is located. In some embodiments, the user input includes basic information related to creating the evaluation function, such as the function name, file name, path, input parameters, and output parameters. In some embodiments, the created function file can be saved in a database (e.g., Figure 3 The database 338). For example, the created function file can be saved in a data table of a MySQL database, and the ID key serves as an identifier for external access. Figure 4B FIG. 4 is a schematic diagram showing a user interface 400B for creating a function according to an embodiment of the present disclosure, as shown in FIG. Figure 4B As shown, users can specify the basic information of the created function, such as function name, file name, path, function description, input parameters, output parameters, etc.

[0047] Return Reference Figure 4AAt block 404, the file contents of the policy file may be saved. For example, the policy file may include an API function file and a scoring function file. In some embodiments, the file contents of the API function file may include an import section, a registration section, an information acquisition section, and a setting section, and the scoring function file may include a score setting section. For example, the import section may include, but is not limited to, built-in libraries, dependency files (e.g., Figure 3 The registration section includes dependencies, toolkits, and user-written functions (supporting importing functions from other files) included in the dependency file 356 shown. The registration section can include decorators that can register functions as functions to be published. The information acquisition section can be used to obtain relevant information when executing a function (for example, the context object ctx), receive additional parameters defined by the function (defined in the incoming parameters), obtain a row of data from the current dataset, obtain the current evaluation task, and so on.

[0048] The setting part can be used to perform one or more of the following: modify the data set, such as modifying the field name of the data set; set the answer, that is, set the variable to save the execution result of the large model (i.e., the prediction result), which can be saved in the database (e.g., Figure 3 The database 322 shown in the figure) can be saved in JSON format for subsequent parsing and processing; set logs and comments, which can be used to record information during the execution of the API function and the basis for comment users to record scores; set the layout for display; set the score of the large model, for example, you can set the score name, which can be specified synchronously in the output parameter, and you can also set it as a group name. When multiple scores are set, the same group name will be displayed together, and you can set a specific value, which is set according to the requirements of the evaluation strategy. Figure 4C A schematic diagram of an object relationship 400C of a context object according to an embodiment of the present disclosure is shown. When the above setting process is performed, one or more properties or methods in the context object ctx object may be operated.

[0049] At block 406, the dependencies required for the policy file may be configured. This process may be, for example, Figure 3 The policy management module 330 in the policy system 306 performs the dependency configuration 340 process. For example, a dependency file can be retrieved from the project's outermost directory, and all the dependency packages required by the policy file can be configured in that dependency file. This allows only the required policy file dependencies to be configured in the dependency file. When the system deploys or executes the policy function, it automatically reads the dependency file and installs all the dependency packages listed therein, eliminating the need for users to manually install these dependency packages. This makes it easier for users to evaluate large models.

[0050] At block 408, the evaluation policy may be published. Figure 3 As described above, the evaluation file 348, toolkit 350, entry point function 352, configuration file 354, and dependency file 356 can be packaged as a release file, and the evaluation policy can be published to the policy service module 344 (e.g., a FaaS service). For example, the policy service module can provide an interface for accessing the FaaS service (e.g., an HTTP interface) and an entry point (e.g., an entry point function) for the FaaS service, for routing and invoking user-written evaluation functions. In this way, the evaluation policy can be published to the FaaS service and invoked through the service interface, allowing the evaluation policy to run in an isolated, secure, and scalable environment.

[0051] Figure 5A FIG. 5 shows a flow chart of a process 500A for performing model evaluation according to an embodiment of the present disclosure. Figure 3 The process of performing model evaluation can be done as described in Figure 3 The evaluation system 304 of the model evaluation and strategy system 302 is executed. At block 502, an evaluation task for a model evaluation may be created. For example, user input may be received to create the evaluation task, and the user input may specify an evaluation strategy to be executed for performing the model evaluation. In some embodiments, the user input may specify a task name, an evaluation dataset, an execution strategy (i.e., an API function), a scoring strategy (i.e., a scoring function), etc. After the evaluation task is created, the scheduling module may call to execute the evaluation task. Figure 5B A schematic diagram of a user interface 500B for creating an assessment task according to an embodiment of the present disclosure is shown. Figure 5B As shown in Figure 1, the user specifies the basic information for creating an evaluation task, such as the task name, task description, dataset, scoring function, and API function.

[0052] At block 504, a request may be sent to execute the running strategy and the scoring strategy. This process may be executed in a loop for each data in the evaluation data set. For example, Figure 3 As described above, the operation policy request module 318 can send a request to the policy service module 344 to execute the operation policy, and the scoring policy request module 324 can send a request to the policy service module 344 to execute the scoring policy. In some embodiments, an HTTP request can be sent to obtain the file path of the API function, and the module where the API function is located can be obtained through the file path. The context object (ctx object) required by the API function can be encapsulated, and the ctx object can be serialized to access the FaaS service by sending an HTTP request. Then, when the policy service module can distribute the request, for example, when the policy service module receives the request, the entry function (for example, Figure 3The policy service module dispatches the request to the specified API function using the entry point function 352 described above. For example, when dispatching a request, the policy service module can parse the object in the request to obtain the ctx object and module path, deserialize the ctx object, and then use Python reflection to retrieve the method containing the decorator (e.g., the registration portion described in FIG4 ) by loading the module path. The deserialized context object is then passed to the function and executed.

[0053] At block 506, the execution results of the running strategy and the scoring strategy can be obtained. For example, the ctx object can be deserialized, and then the dataset fields, scores, comments, etc. that need to be updated can be obtained from the deserialized ctx object. Figure 3 As mentioned above, the evaluation data update module 320 can obtain the execution results of the operation strategy, which can include the output of the large model; the evaluation result processing module 326 can obtain the execution results of the scoring strategy, which can include the scores, parameters, comments, etc. of the large model. For example, Figure 5C A schematic diagram of the execution result 500C of a policy request according to an embodiment of the present disclosure is shown. Execution result 500C displays information for each evaluation data point. The model output is the output of the large model running each evaluation data point, and the evaluation score is the score for each evaluation data point. The scores for each evaluation data point are aggregated to generate the evaluation score for the large model. The execution result also includes the ground truth value for each evaluation data point and the execution status.

[0054] Figure 6 FIG. 6 is a block diagram of an apparatus 600 for large model evaluation according to some embodiments of the present disclosure. Figure 6 As shown, the device 600 includes a request sending module 602, which is configured to send a request for executing the selected evaluation strategy based on user input indicating that an evaluation strategy for evaluating a target model is selected from multiple evaluation strategies, wherein the multiple evaluation strategies are published on a policy system, and the policy system includes: a policy creation module, which is configured to create a policy file for the evaluation strategy, and the policy file is stored in a database; a dependency setting module, which is configured to set the dependencies required for executing the evaluation strategy; and a policy publishing module, which is configured to publish the evaluation strategy to the policy service in the policy system. In addition, the device also includes a result acquisition module 604, which is configured to obtain the execution result of the request, and the execution result at least includes the evaluation result of the target model.

[0055] Figure 7 A block diagram of an electronic device 700 is shown, in accordance with certain embodiments of the present disclosure. Figure 7FIG2 shows a block diagram of an electronic device 700 according to some embodiments of the present disclosure. The device 700 may be a device or apparatus described in the embodiments of the present disclosure. Figure 7 As shown, the device 700 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 701, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 702 or loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the device 700 can also be stored in the RAM 703. The CPU / GPU 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704. Although not shown in FIG. Figure 7 As shown in FIG, device 700 may further include a co-processor.

[0056] Various components in device 700 are connected to I / O interface 705, including an input unit 706, such as a keyboard, mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, optical disk, etc.; and a communication unit 709, such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0057] The various methods or processes described above may be performed by the CPU / GPU 701. For example, in some embodiments, the methods may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed onto the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the CPU / GPU 701, one or more steps or actions in the methods or processes described above may be performed.

[0058] In some embodiments, the methods and processes described above may be implemented as a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0059] A computer-readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure within a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0060] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0061] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​and conventional procedural programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, the state information of the computer-readable program instructions is used to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), so that the electronic circuit can execute the computer-readable program instructions, thereby implementing various aspects of the present disclosure.

[0062] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0063] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0064] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart, can be implemented by a special hardware-based system that performs the prescribed function or action, or can be implemented by a combination of special hardware and computer instructions.

[0065] The embodiments of the present disclosure have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, practical applications, or technical improvements to existing technologies, or to enable other persons skilled in the art to understand the embodiments disclosed herein.

[0066] Some example implementations of the present disclosure are listed below.

[0067] Example 1. A method for evaluating a large model, comprising:

[0068] sending a request for executing the selected evaluation strategy based on a user input indicating selection of an evaluation strategy for evaluating a target model from a plurality of evaluation strategies, the plurality of evaluation strategies being published on a policy system, the policy system being configured to: create a policy file for the evaluation strategy, the policy file being stored in a database; set dependencies required for executing the evaluation strategy; and publish the evaluation strategy to a policy service in the policy system; and

[0069] Obtaining an execution result of the request, where the execution result at least includes an evaluation result of the target model.

[0070] Example 2. The method according to Example 1, further comprising:

[0071] The execution result is displayed, wherein the execution result further includes at least one of the following: model input, model output, correct answer, and execution status.

[0072] Example 3. The method of any one of Examples 1-2, wherein the evaluation strategy includes an execution strategy and a scoring strategy, and receiving the user input includes:

[0073] receiving a first user input indicating a selection of the operation strategy from a plurality of operation strategies; and

[0074] A second user input is received, the second user input indicating selection of the scoring strategy from a plurality of scoring strategies.

[0075] Example 4. The method of any of Examples 1-3, wherein sending a request to execute the selected evaluation strategy comprises:

[0076] sending a first request for executing the selected execution strategy, the execution strategy being configured to generate a model output based on the target model; and

[0077] A second request is sent for executing the selected scoring strategy, the scoring strategy being configured to generate the evaluation result based on the model output.

[0078] Example 5. The method of any one of Examples 1-4, wherein obtaining the execution result of the request includes obtaining a first execution result of the first request, and the method further comprises:

[0079] Based on the first execution result, an evaluation dataset used to evaluate the target model and a model output of the target model are updated.

[0080] Example 6. The method according to any one of Examples 1-5, wherein obtaining the execution result of the request includes obtaining a second execution result of the second request, and the method further includes: updating an evaluation dataset for evaluating the target model and an evaluation score for the target model based on the second execution result.

[0081] Example 7. The method according to any one of Examples 1-6, wherein the identifier of the evaluation policy is a key value of the policy file in a data table of the database.

[0082] Example 8. The method according to any one of Examples 1-7, further comprising:

[0083] In response to detecting a request for executing the evaluation policy, the policy system obtains a context object for executing the evaluation policy;

[0084] The policy system executes the evaluation policy by transferring the context object to the evaluation policy; and

[0085] The policy system generates the execution result by executing the evaluation policy.

[0086] Example 9. The method of any one of Examples 1-8, wherein dependencies required to execute the evaluation policy are specified in a dependency file, and publishing the evaluation policy to the policy service in the policy system comprises:

[0087] The policy file, the toolkit for the policy file, the entry function of the policy service, the configuration file of the policy service, and the dependent files are merged and published.

[0088] Example 10. The method of any one of Examples 1-9, wherein the evaluation strategy includes a published functionality.

[0089] Example 11 A device for large model evaluation, comprising:

[0090] a request sending module configured to send a request for executing an evaluation strategy selected from a plurality of evaluation strategies for evaluating a target model based on user input indicating the selection of the evaluation strategy, the plurality of evaluation strategies being published on a policy system, the policy system comprising: a policy creating module configured to create a policy file for the evaluation strategy, the policy file being stored in a database; a dependency setting module configured to set dependencies required for executing the evaluation strategy; and a policy publishing module configured to publish the evaluation strategy to a policy service in the policy system; and

[0091] The result acquisition module is configured to obtain the execution result of the request, where the execution result at least includes the evaluation result of the target model.

[0092] Example 12. The apparatus of Example 11, further comprising:

[0093] The result display module is configured to display the execution result, wherein the execution result further includes at least one of the following: model input, model output, correct answer, and execution status.

[0094] Example 13. The apparatus of any of Examples 11-12, wherein the evaluation strategy includes an execution strategy and a scoring strategy, and the input receiving module includes:

[0095] A first input receiving module is configured to receive a first user input, wherein the first user input indicates a selection of the operation strategy from a plurality of operation strategies; and

[0096] The second input receiving module is configured to receive a second user input, where the second user input indicates selecting the scoring strategy from a plurality of scoring strategies.

[0097] Example 14. The apparatus of any one of Examples 11-13, wherein the request sending module comprises:

[0098] a first request sending module configured to send a first request for executing the selected operation strategy, wherein the operation strategy is configured to generate a model output based on the target model; and

[0099] The second request sending module is configured to send a second request for executing the selected scoring strategy, where the scoring strategy is configured to generate the evaluation result based on the model output.

[0100] Example 15. The apparatus according to any one of Examples 11-14, wherein the result acquisition module includes a first result acquisition module configured to acquire a first execution result of the first request, and the apparatus further includes:

[0101] An evaluation data updating module is configured to update an evaluation data set used to evaluate the target model and a model output of the target model based on the first execution result.

[0102] Example 16. The apparatus according to any one of Examples 11-15, wherein the result acquisition module includes a second result acquisition module configured to acquire a second execution result of the second request, and the apparatus further includes:

[0103] The evaluation data second updating module is configured to update the evaluation data set used to evaluate the target model and the evaluation score for the target model based on the second execution result.

[0104] Example 17. The apparatus of any one of Examples 11-16, wherein the identifier of the evaluation policy is a key value of the policy file in a data table of the database.

[0105] Example 18. The apparatus of any one of Examples 11-17, further comprising:

[0106] a context object acquisition module configured to, in response to detecting a request for executing the evaluation policy, cause the policy system to acquire a context object for executing the evaluation policy;

[0107] an evaluation policy execution module configured to execute the evaluation policy by transmitting the context object to the evaluation policy through the policy system; and

[0108] The evaluation result generating module is configured to generate the execution result by executing the evaluation strategy in the strategy system.

[0109] Example 19. The apparatus of any of Examples 11-18, wherein dependencies required to execute the evaluation policy are specified in a dependency file, and publishing the evaluation policy to the policy service in the policy system comprises:

[0110] The policy file, the toolkit for the policy file, the entry function of the policy service, the configuration file of the policy service, and the dependent files are merged and published.

[0111] Example 20. The apparatus of any of Examples 11-19, wherein the evaluation policy comprises a published functionality.

[0112] Example 21. An electronic device comprising:

[0113] processor; and

[0114] A memory coupled to the processor, the memory having instructions stored therein, wherein when the instructions are executed by the processor, the electronic device performs actions, the actions comprising:

[0115] Based on user input indicating selection of an evaluation strategy for evaluating a target model from a plurality of evaluation strategies, sending a request for executing the selected evaluation strategy, the plurality of evaluation strategies being published on a policy system, the policy system being configured to:

[0116] Creating a policy file for the evaluation policy, wherein the policy file is stored in a database;

[0117] Setting up the dependencies required to execute the evaluation strategy; and

[0118] Publishing the evaluation policy to a policy service in the policy system; and

[0119] Obtaining an execution result of the request, where the execution result at least includes an evaluation result of the target model.

[0120] Example 22. The electronic device according to Example 21, further comprising:

[0121] The execution result is displayed, wherein the execution result further includes at least one of the following: model input, model output, correct answer, and execution status.

[0122] Example 23. The electronic device of any of Examples 21-22, wherein the evaluation strategy includes an execution strategy and a scoring strategy, and receiving the user input includes:

[0123] receiving a first user input indicating a selection of the operation strategy from a plurality of operation strategies; and

[0124] A second user input is received, the second user input indicating selection of the scoring strategy from a plurality of scoring strategies.

[0125] Example 24. The electronic device of any one of Examples 21-23, wherein sending a request for executing the selected evaluation policy comprises:

[0126] sending a first request for executing the selected execution strategy, the execution strategy being configured to generate a model output based on the target model; and

[0127] A second request is sent for executing the selected scoring strategy, the scoring strategy being configured to generate the evaluation result based on the model output.

[0128] Example 25. The electronic device of any one of Examples 21-24, wherein obtaining the execution result of the request comprises obtaining a first execution result of the first request, and the action further comprises:

[0129] Based on the first execution result, an evaluation dataset used to evaluate the target model and a model output of the target model are updated.

[0130] Example 26. The electronic device of any one of Examples 21-25, wherein obtaining the execution result of the request comprises obtaining a second execution result of the second request, and the action further comprises:

[0131] Based on the second execution result, an evaluation dataset used to evaluate the target model and an evaluation score for the target model are updated.

[0132] Example 27. The electronic device of any one of Examples 21-26, wherein the identifier of the evaluation policy is a key value of the policy file in a data table of the database.

[0133] Example 28. The electronic device of any one of Examples 21-27, further comprising:

[0134] In response to detecting a request for executing the evaluation policy, the policy system obtains a context object for executing the evaluation policy;

[0135] The policy system executes the evaluation policy by transferring the context object to the evaluation policy; and

[0136] The policy system generates the execution result by executing the evaluation policy.

[0137] Example 29. The electronic device of any one of Examples 21-28, wherein dependencies required to execute the evaluation policy are specified in a dependency file, and publishing the evaluation policy to the policy service in the policy system comprises:

[0138] The policy file, the toolkit for the policy file, the entry function of the policy service, the configuration file of the policy service, and the dependent files are merged and published.

[0139] Example 30. The electronic device of any one of Examples 21-29, wherein the evaluation policy comprises a published functionality.

[0140] Example 31. A computer-readable storage medium having one or more computer instructions stored thereon, wherein the one or more computer instructions are executed by a processor to implement the method according to any one of Examples 1 to 10.

[0141] Example 32. A computer program product tangibly stored on a computer-readable medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of Examples 1 to 10.

[0142] Although the present disclosure has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A method for evaluating a large model, comprising: Sending a request for executing the selected evaluation strategy based on user input indicating selection of an evaluation strategy for evaluating a target model from a plurality of evaluation strategies, the evaluation strategy including an execution strategy and a plurality of scoring strategies, the plurality of evaluation strategies being published on a strategy system, the strategy system being configured to: Creating a policy file for the evaluation policy, wherein the policy file is stored in a database and is generated based on a toolkit provided by the policy system, wherein the toolkit supports one or more of data set management, environment variable calling, and metadata acquisition; Generate a dependency file by setting dependencies required for executing the evaluation strategy, wherein the dependency file is read to install dependent packages; The evaluation policy is published to the policy service in the policy system by packaging the policy file, the toolkit, the dependency file, the configuration file of the policy service, and the entry function of the policy service as a publishing file, where the policy service is a Function as a Service (FaaS). In response to receiving the request for executing the evaluation strategy, dispatching the request to the selected execution strategy by executing the entry function of the FaaS; executing the operation strategy to determine a model output of the target model; executing the plurality of scoring strategies on the model output of the target model to determine a plurality of execution results of the request; as well as The multiple execution results of the request are obtained, where the multiple execution results at least include multiple evaluation results of the target model.

2. The method according to claim 1, further comprising: The multiple execution results are displayed, each of the multiple execution results further comprising at least one of the following: model input, model output, correct answer, and execution status.

3. The method according to claim 1, further comprising: receiving a first user input indicating selection of the operation strategy from a plurality of operation strategies; as well as receiving a second user input indicating selection of the plurality of scoring strategies from a plurality of scoring strategies; The user input includes the first user input and the second user input.

4. The method of claim 3, wherein sending a request for executing the selected evaluation strategy comprises: sending a first request for executing the selected execution strategy, the execution strategy being configured to generate the model output based on the target model; as well as A second request is sent for executing the selected plurality of scoring strategies, the plurality of scoring strategies being configured to generate the plurality of evaluation results based on the model output.

5. The method according to claim 4, wherein obtaining the multiple execution results of the request comprises obtaining a first execution result of the first request, and the method further comprises: Based on the first execution result, an evaluation dataset used to evaluate the target model and a model output of the target model are updated.

6. The method according to claim 4, wherein obtaining the multiple execution results of the request comprises obtaining a second execution result of the second request, and the method further comprises: Based on the second execution result, an evaluation dataset used to evaluate the target model and an evaluation score for the target model are updated. 7 . The method according to claim 1 , wherein the identifier of the evaluation policy is a key value of the policy file in a data table of the database.

8. The method according to claim 7, further comprising: In response to detecting a request for executing the evaluation policy, the policy system obtains a context object for executing the evaluation policy; The policy system executes the evaluation policy by transmitting the context object to the evaluation policy; as well as The policy system generates the plurality of execution results by executing the evaluation policy. The method of claim 8 , wherein the evaluation strategy comprises a published functionality.

10. A device for large-scale model evaluation, comprising: A request sending module is configured to send a request for executing an evaluation strategy selected from a plurality of evaluation strategies for evaluating a target model based on user input indicating the selection of the evaluation strategy, wherein the evaluation strategy includes an operation strategy and a plurality of scoring strategies, wherein the plurality of evaluation strategies are published on a strategy system, wherein the strategy system includes: a policy creation module configured to create a policy file for the evaluation policy, wherein the policy file is stored in a database and is generated based on a toolkit provided by the policy system, wherein the toolkit supports one or more of data set management, environment variable call, and metadata acquisition; a dependency setting module configured to set dependencies required for executing the evaluation strategy, wherein the dependencies generate dependency files, and the dependency files are read to install dependency packages; and A policy publishing module is configured to publish the evaluation policy to a policy service in the policy system, wherein the policy file, the toolkit, the dependency files, the configuration file of the policy service, and the entry function of the policy service are packaged as a publishing file, and the policy service is a Function as a Service (FaaS); The policy system further performs: in response to receiving the request for executing the evaluation policy, distributing the request to the selected execution policy by executing the entry function of the FaaS; executing the operation strategy to determine a model output of the target model; executing the plurality of scoring strategies on the model output of the target model to determine a plurality of execution results of the request; and The result acquisition module is configured to acquire the multiple execution results of the request, where the multiple execution results at least include multiple evaluation results of the target model.

11. An electronic device comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, wherein when the instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 9.

12. A computer program product tangibly stored on a non-transitory computer-readable medium and comprising computer-executable instructions for performing the method according to any one of claims 1 to 9.

Images