Model evaluation method and device, electronic equipment and computer program product

By selecting and publishing strategies from multiple evaluation strategies, efficient evaluation of large models is achieved, the problem of inefficient evaluation is solved, the evaluation efficiency and accuracy are improved, and the development process is simplified.

CN119025484AActive Publication Date: 2024-11-26BEIJING VOLCANO ENGINE TECH CO LTD

Patent Information

Application Number
CN202411124729.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-26
Estimated Expiration
2044-08-15

AI Technical Summary

Technical Problem

The evaluation efficiency of large models is inefficient, requiring multiple rounds of optimization and continuous updates, resulting in low development and optimization efficiency, and low existing technology efficiency, making it difficult to meet the needs of widespread applications.

Method used

By selecting the appropriate policy from multiple evaluation strategies to evaluate the target model, using the policy system to create and publish evaluation policies, set dependencies, and obtain evaluation results, an efficient model evaluation process is achieved.

Benefits of technology

It improves the efficiency of model evaluation, shortens the development cycle, accelerates model iteration, reduces the difficulty of evaluation tasks, enables more people to participate in the evaluation process, and improves the accuracy and flexibility of evaluation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025484A_ABST
    Figure CN119025484A_ABST
Patent Text Reader

Abstract

The embodiment of the invention relates to a model evaluation method and device, electronic equipment and a computer program product. The method includes transmitting, based on a user input indicating selection of an evaluation policy for evaluating a target model from a plurality of evaluation policies, a request for executing the selected evaluation policy. In addition, the method further comprises the step of obtaining an execution result of the request, and the execution result at least comprises an evaluation result of the target model. Through the embodiment of the invention, the model evaluation can be efficiently completed, the evaluation efficiency is improved, the development period of the model is shortened, the iteration of the model is accelerated, the difficulty of developing a model evaluation task can be reduced, and more users can participate in the evaluation process of the model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more particularly to a method, device, electronic device and computer program product for model evaluation. Background Art

[0002] Through massive data training, the big model has strong generalization ability and excellent performance, and can cope with complex tasks and various application scenarios. In addition, the big model has also shown great potential and application prospects in many fields such as medicine, finance, and autonomous driving, promoting the comprehensive development and widespread application of artificial intelligence technology, and becoming an important driving force for current scientific and technological innovation.

[0003] Large models are increasingly used in various fields, and their performance and reliability directly affect the actual application effect. Model evaluation can reveal the performance differences of large models in different application scenarios and provide a basis for model optimization. Therefore, the evaluation of large models has become particularly important. Summary of the invention

[0004] Embodiments of the present disclosure provide a method, apparatus, electronic device, computer program product, and medium for model evaluation.

[0005] According to the first aspect of the present disclosure, a method for model evaluation is provided. The method includes sending a request for executing the selected evaluation strategy based on a user input indicating the selection of an evaluation strategy for evaluating a target model from a plurality of evaluation strategies, wherein the plurality of evaluation strategies are published on a policy system, and the policy system is configured to: create a policy file for the evaluation strategy, wherein the policy file is stored in a database; set the dependencies required for executing the evaluation strategy; and publish the evaluation strategy to a policy service in the policy system. In addition, the method also includes obtaining an execution result of the request, wherein the execution result at least includes an evaluation result of the target model.

[0006] According to the second aspect of the present disclosure, a device for model evaluation is provided. The device includes a request sending module, which is configured to send a request for executing the selected evaluation strategy based on a user input indicating that an evaluation strategy for evaluating a target model is selected from multiple evaluation strategies, wherein the multiple evaluation strategies are published on a policy system, and the policy system includes: a policy creation module, which is configured to create a policy file for the evaluation strategy, and the policy file is stored in a database; a dependency setting module, which is configured to set the dependencies required for executing the evaluation strategy; and a policy publishing module, which is configured to publish the evaluation strategy to a policy service in the policy system. In addition, the device also includes a result acquisition module, which is configured to obtain the execution result of the request, and the execution result at least includes the evaluation result of the target model.

[0007] According to a third aspect of the present disclosure, an electronic device is provided, comprising a processor and a memory coupled to the processor, wherein the memory has instructions stored therein, and when the instructions are executed by the processor, the electronic device executes the method according to the first aspect.

[0008] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided. The computer program product is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which, when executed, cause a computer to perform the steps of the method of the first aspect of the present disclosure.

[0009] In a fifth aspect of the present disclosure, a computer-readable storage medium is provided, wherein one or more computer instructions are stored on the computer-readable storage medium, wherein the one or more computer instructions are executed by a processor to implement the method according to the first aspect.

[0010] The purpose of this Summary is to introduce a selection of concepts in a simplified form that are further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0012] Figure 1 A schematic diagram showing an example environment in which devices and / or methods according to embodiments of the present disclosure may be implemented;

[0013] Figure 2 A flowchart of a method for model evaluation according to an embodiment of the present disclosure is shown;

[0014] Figure 3 A schematic diagram showing the architecture of a model evaluation and strategy system according to an embodiment of the present disclosure;

[0015] Figure 4A A flowchart showing a process of issuing an evaluation strategy according to an embodiment of the present disclosure;

[0016] Figure 4B A schematic diagram showing a user interface for creating a function according to an embodiment of the present disclosure is shown;

[0017] Figure 4C A schematic diagram showing an object relationship of a context object according to an embodiment of the present disclosure;

[0018] Figure 5A A flowchart of a process for performing model evaluation according to an embodiment of the present disclosure is shown;

[0019] Figure 5B A schematic diagram showing a user interface for creating an assessment task according to an embodiment of the present disclosure

[0020] Figure 5C A schematic diagram showing the execution result of a policy request according to an embodiment of the present disclosure;

[0021] Figure 6 A block diagram showing an apparatus for model evaluation according to an embodiment of the present disclosure; and

[0022] Figure 7 A block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0023] Throughout the drawings, the same or similar reference numbers denote the same or similar elements. DETAILED DESCRIPTION

[0024] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information (such as voice) involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0025] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0026] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects, unless explicitly stated. Other explicit and implicit definitions may also be included below.

[0027] As mentioned above, the development of large models is getting faster and faster, and the demand for large model evaluation is also increasing. In the related technologies of large model evaluation, users need to develop evaluation strategies for each model to be evaluated and deploy them after development is completed. This evaluation method is inefficient. In addition, large models usually require multiple rounds of optimization and continuous updates, and the process is repeated after each optimization and update is completed, which greatly affects the development and optimization efficiency of large models.

[0028] To this end, the embodiment of the present disclosure proposes a model evaluation scheme. The scheme evaluates the target model by selecting an evaluation strategy from multiple evaluation strategies, and the evaluation system can provide many evaluation strategies for selection to meet the model evaluation requirements in different situations, so that only a request to execute the evaluation strategy is sent to obtain the corresponding evaluation result, thereby completing the evaluation of the target model. Therefore, through the model evaluation scheme proposed by the embodiment of the present disclosure, it is possible to efficiently complete the model evaluation, improve the evaluation efficiency, shorten the model development cycle, speed up the iteration of the model, and also reduce the difficulty of developing the model evaluation task, so that more people can participate in the model evaluation process.

[0029] Figure 1 Schematic diagram of an example environment in which the device and / or method according to an embodiment of the present disclosure may be implemented. Figure 1 As shown, the example environment 100 may include an evaluation and policy system 110, and the evaluation and policy system 110 may include an evaluation system 120 and a policy system 130. The evaluation system 120 may receive a user input 140, and the user input 140 may indicate an evaluation policy 122 for selecting a user to evaluate a target model from a plurality of evaluation policies. For example, the evaluation policy 122 may include a target model to be evaluated, an evaluation data set, and a process for evaluating the target model, etc. The evaluation system 120 may send a policy request 124 for executing the evaluation policy 122 to the policy system 130. For example, the policy request 124 may be a hypertext transfer protocol (HTTP) request.

[0030] The policy system 130 can deploy a policy service, and the policy service can have multiple evaluation policies. In some embodiments, the policy service can be a function as a service (FaaS). When the policy system 130 receives the policy request 124, it can trigger the FaaS service to execute the corresponding evaluation policy. After executing the evaluation policy 122, the policy system 130 can transmit the execution result 126 to the evaluation system 120. The execution result 126 includes at least the evaluation result for the target model. For example, when evaluating the accuracy of the target model, the execution result 126 may include a measure of the accuracy of the target model. In some embodiments, the execution result 126 may also include the model input, model output, benchmark truth value, and execution status for each evaluation data. In some embodiments, the content of the execution result 126 can be specified in the evaluation policy 122.

[0031] It should be understood that the architecture and functions in the example environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure. Embodiments of the present disclosure may also be applied to other environments with different structures and / or functions.

[0032] The following will combine Figures 2 to 7 The process according to the embodiment of the present disclosure is described in detail. For ease of understanding, the specific data mentioned in the following description are exemplary and are not intended to limit the scope of protection of the present disclosure. It is understood that the embodiments described below may also include additional actions not shown and / or the actions shown may be omitted, and the scope of the present disclosure is not limited in this respect.

[0033] Figure 2 A flow chart of a method 200 for model evaluation according to an embodiment of the present disclosure is shown. At block 202, based on user input indicating that an evaluation strategy for evaluating a target model is selected from a plurality of evaluation strategies, a request for executing the selected evaluation strategy may be sent. Figure 1 The evaluation system 120 may receive user input 140 indicating that an evaluation strategy 122 is selected from a plurality of evaluation strategies for evaluating a target model, and then the evaluation system 120 may send a request 124 for executing the selected evaluation strategy 122. The policy system 130 may be used to create a policy file for the evaluation strategy 122 and save the policy file in a database, and may also be used to set dependencies required for executing the evaluation strategy 122, and to publish the evaluation strategy 122 to a policy service in the policy system 130.

[0034] At block 204, the execution result of the request may be obtained, the execution result at least including the evaluation result of the target model. Figure 1, the evaluation system 120 can obtain the execution result 126 of the request 124, and the execution result 126 at least includes the evaluation result of the target model.

[0035] Therefore, according to the method 200 of the embodiment of the present disclosure, it is possible to efficiently complete model evaluation, improve evaluation efficiency, shorten the model development cycle, accelerate model iteration, and also reduce the difficulty of developing model evaluation tasks, allowing more people to participate in the model evaluation process.

[0036] Figure 3 FIG. 3 is a schematic diagram of an architecture 300 of a model evaluation and strategy system according to an embodiment of the present disclosure. Figure 3 As shown, the model evaluation and strategy system 302 may include an evaluation system 304 and a strategy system 306, wherein the evaluation system 304 may manage and execute model evaluation tasks, and the strategy system 306 may manage, deploy and execute evaluation strategies. User 308 may be a user with a large model evaluation requirement, which may include an administrator and an ordinary user. Data set module 310 may receive evaluation data uploaded by user 308. For example, the evaluation data may include a topic for evaluating a large model. In addition, data set module 310 may be used to annotate evaluation data, such as information such as the correct answer, type, source, etc. for each topic in the evaluation data. In some embodiments, the original topic may be used to evaluate the large model, and the original topic may be updated during the evaluation process. In some embodiments, different evaluation tasks may use the same original topic. In some embodiments, different evaluation tasks may use different original topics.

[0037] The evaluation creation module 312 can receive user input from the user 308 to select one or more evaluation strategies from a plurality of evaluation strategies. In some embodiments, the evaluation strategy may include an operation strategy (also referred to as an API function) and a scoring strategy (also referred to as a scoring function). For example, when performing a large model evaluation, the API function may be executed first to complete each question in the evaluation data, and then the scoring function may be executed to generate an evaluation result of the large model. In some embodiments, the evaluation creation module 312 may receive the user 308's selection of the operation strategy and the selection of the scoring strategy. For example, the user may specify the API function and the scoring function. In addition, the evaluation creation module 312 may also receive additional user selections. For example, the user 308 may select one or more large models to be evaluated, and may also select one or more evaluation data sets, and so on. The evaluation creation module 312 may create an evaluation task based on the user input, and then save the evaluation task to the scheduling module 314. The evaluation strategy is divided into an operation strategy and a scoring strategy, and the operation strategy may be executed only once to obtain the output of the large model, and then different scoring strategies may be used to obtain different evaluation results, thereby avoiding repeated operation of the large model, thereby improving the evaluation efficiency.

[0038] In some embodiments, the scheduling module 314 can save one or more evaluation tasks. For example, when multiple users have created multiple evaluation tasks, the scheduling module 314 can save multiple evaluation tasks and then schedule them when conditions are met. After the scheduling module 314 is scheduled to the created evaluation task, the evaluation execution module 316 can execute the evaluation task. When the evaluation execution module 316 can execute the evaluation task, the operation strategy request module 318 can send a request (e.g., HTTP request) for executing an API function. For example, an API function (i.e., an operation strategy) can include obtaining a topic in the evaluation data, obtaining a prompt content, setting an answer to a topic, obtaining a standard answer to a topic, and obtaining other front-end parameters, etc. The operation strategy request module 318 can send a request for executing an API function to the policy service module 344 in the policy system 306 to access the selected API function.

[0039] In some embodiments, the policy service module 344 can be a FaaS service. FaaS service is a cloud computing service that allows users to write code in the form of functions and deploy it on a cloud platform. The platform is responsible for managing the execution environment, resource scheduling, and expansion of the function, and users do not need to manage the underlying infrastructure, but only need to focus on the implementation of business logic. The embodiments of the present disclosure combine large model evaluation with FaaS services, which can achieve an efficient and scalable evaluation process, reduce costs and maintenance difficulties, provide flexible and fast iteration capabilities, and ensure the isolation and security of evaluation tasks, thereby improving the overall evaluation efficiency and accuracy.

[0040] The evaluation data update module 320 can receive the result of executing the API function from the policy service module 344. For example, after the API function is executed, the execution result of the large model can be generated, and the evaluation data update module 320 can update the evaluation data set based on the execution result of the large model and store it in the database 322. For example, the execution results of different large models can be written into the evaluation data set, and the large model name is the field name of the corresponding execution result, so that the differences between the execution results of different large models can be compared. The scoring strategy request module 324 can send a request for executing a scoring function (i.e., a scoring strategy) to the policy service module 344, and the evaluation result processing module 326 can receive the result of executing the scoring function from the policy service module 344, thereby obtaining the evaluation result of the large model. The evaluation result processing module 326 can write the evaluation result of the large model into the database 322 and display the evaluation result on the user interface.

[0041] The policy management module 330 in the policy system 306 can receive a request for creating an evaluation strategy from a user 328. The user 328 can be the same user as the user 308, or can be a different user from the user 308. For example, the evaluation strategy created by a certain user can be used by the user himself or another user later (for example, authorized), and the reuse of this evaluation strategy can significantly improve the evaluation efficiency of the large model. The policy management module 330 can receive user input to execute the creation of an evaluation strategy 332, such as a python file of an API function or a Policy can be created. The policy management module 330 can execute the preservation of file content 334, such as the file content of the evaluation strategy uploaded by the user, and the toolkit 336 provided by the policy system 306 can be used in the policy file, and the toolkit 336 can support data set management, environment variable call, metadata acquisition, etc., and can improve the efficiency of the user writing function files. Then, the file of the evaluation strategy can be stored in a database 338, such as a MySQL database. For example, the evaluation policy file may be stored in a data table in the database 338 , and its identifier key (ie, ID key) in the data table may be used as an identifier for external access.

[0042] The policy management module 330 can be used to execute configuration dependencies 340. For example, when writing API functions and scoring functions, multiple external dependencies are usually required. Through the policy management module 330, the user 328 can easily add dependencies. In some embodiments, the required dependencies can be added to the dependency file, thereby achieving spatial level dependency isolation. For example, the dependency file requirements.txt can be used, and the required dependency packages (for example, specifying the name and version number of the dependency package) can be added therein, without the user having to install and configure the dependency package separately, which is convenient for users to use.

[0043] The policy management module 330 can be used to execute the release evaluation policy 342, for example, to the policy service module 344. As mentioned above, the policy service module 344 can be a FaaS service, and it can also be other types of cloud computing services or event-driven services, which is not limited by the present disclosure. In some implementations, the release file 346 may include an evaluation file 348, a toolkit 350, an entry function 352, a service configuration file 354, and a dependency file 356. For example, the evaluation file 348 can be a file of the evaluation policy obtained after executing the save file content 334, which can be an API function file or a scoring function file; the toolkit 350 can be all or part of the toolkit 336; the entry function 352 can be used to route the entry of the evaluation file 348; the configuration file 348 can be a configuration file required by the policy service module 344; the dependency file 356 can be a dependency file generated after executing the configuration dependency 340.

[0044] In addition, the policy management module 330 can publish the function function to the function plug-in market 358, and the function plug-in market 358 can save commonly used function functions and encapsulate them into function plug-ins. Function functions are function functions that may be used when writing API functions or scoring functions. For example, user 328 can encapsulate the function of comparing whether two SQL queries are consistent into a function plug-in, and the function plug-in market 358 can publish the function plug-in. Then, when writing API functions or scoring functions, you only need to call the function plug-in. This process is similar to calling a local function, which is convenient for users to write functions. For example, when the function function is published to the function plug-in market 358, the policy system 306 can provide a function plug-in list and provide a template code for calling the function plug-in. In the process of writing the evaluation strategy, by copying the template code, you can call the function plug-in just like calling a local function.

[0045] Figure 4A FIG. 4 is a flowchart showing a process 400A for publishing an evaluation strategy according to an embodiment of the present disclosure. Figure 3 The process of publishing an evaluation strategy can be described in Figure 3 The model evaluation and strategy system 302 is executed on the strategy system 306. As shown in Figure 4, at box 402, a strategy file for the evaluation strategy can be created. As mentioned above, the evaluation strategy is also called an evaluation function, which can include an operation strategy (also called an API function) and a scoring strategy (also called a scoring function). When performing a large model evaluation, the operation strategy can be executed first to complete each question in the evaluation data, and then the scoring strategy can be executed to generate the evaluation result of the large model. It should be understood that the creation of the evaluation strategy described here includes creating an operation strategy and creating a scoring strategy.

[0046] For example, after receiving user input, the policy system can create a function file in the user space where the user is located. In some embodiments, the user input includes basic information related to creating an evaluation function, such as a function name, file name, path, input parameters, and output parameters. In some embodiments, the created function file can be saved in a database (e.g., Figure 3 The database 338). For example, the created function file can be saved in a data table of a MySQL database, and the ID key serves as an identifier for external access. Figure 4B FIG. 4 is a schematic diagram of a user interface 400B for creating a function according to an embodiment of the present disclosure, as shown in FIG. Figure 4B As shown, users can specify the basic information of the created function, such as function name, file name, path, function description, input parameters, output parameters, etc.

[0047] Return to reference Figure 4AAt block 404, the file contents of the policy file may be saved. For example, the policy file may include an API function file and a scoring function file. In some embodiments, the file contents of the API function file may include an import section, a registration section, an information acquisition section, and a setting section, and the scoring function file may include a score setting section. For example, the import section may include, but is not limited to, built-in libraries, dependent files (e.g., Figure 3 The registration part may include a decorator, which may register a function as a function that needs to be published. The information acquisition part may be used to obtain relevant information when executing a function (e.g., a context object ctx), receive additional parameters defined by the function (defined in the incoming parameters), obtain a row of data in the current data set, and obtain the current evaluation task, etc.

[0048] The setting part can be used to perform one or more of the following: modify the data set, such as modifying the field name of the data set; set the answer, that is, set the variable to save the execution result of the large model (i.e., the prediction result), which can be saved in the database (e.g., Figure 3 The database 322 shown in the figure) can be saved in JSON format for subsequent parsing and processing; set logs and comments, which can be used to record information during the execution of the API function and comment on the basis of user scoring; set the layout for display; set the score of the large model, for example, you can set the score name, which can be specified synchronously in the output parameter, and you can also set this as a group name. When multiple scores are set, the same group name will be displayed together, and you can set a specific value, which is set according to the requirements of the evaluation strategy. Figure 4C A schematic diagram of an object relationship 400C of a context object according to an embodiment of the present disclosure is shown. When the above setting process is performed, one or more properties or methods in the context object ctx object may be operated.

[0049] At block 406, the dependencies required for the policy file may be configured. This process may be, for example, Figure 3 The policy management module 330 in the policy system 306 performs the configuration dependency 340 process. For example, a dependency file can be obtained in the outermost directory of the project, and all dependency packages required by the policy file can be configured in the dependency file. In this way, only the dependencies required by the policy file need to be configured in the dependency file. When the system deploys or executes the policy function, the dependency file will be automatically read and all dependency packages listed therein will be installed. Users do not need to manually install these dependency packages, which facilitates users to evaluate large models.

[0050] At block 408, the evaluation policy may be published. Figure 3 As described above, the evaluation file 348, toolkit 350, entry function 352, configuration file 354, and dependency file 356 can be packaged as a release file, and the evaluation strategy can be published to the policy service module 344 (e.g., FaaS service). For example, the policy service module can provide an interface for accessing the FaaS service (e.g., HTTP interface) and an entry point (e.g., entry function) of the FaaS service for routing and calling the evaluation function written by the user. In this way, the evaluation strategy can be published to the FaaS service and called through the service interface, so that the evaluation strategy can be run in an isolated, secure, and scalable environment.

[0051] Figure 5A FIG. 5 shows a flow chart of a process 500A for performing model evaluation according to an embodiment of the present disclosure. Figure 3 The process of performing model evaluation can be described in Figure 3 The model evaluation and strategy system 302 is executed on the evaluation system 304. At box 502, an evaluation task for model evaluation can be created. For example, user input can be received to create an evaluation task, and the user input can specify the evaluation strategy to be executed for performing model evaluation. In some embodiments, the user input can specify a task name, an evaluation data set, an operation strategy (i.e., an API function), a scoring strategy (i.e., a scoring function), etc. After completing the creation of the evaluation task, the scheduling module can make a call to execute the evaluation task. Figure 5B A schematic diagram of a user interface 500B for creating an assessment task according to an embodiment of the present disclosure is shown. Figure 5B As shown, the user specifies the basic information for creating an evaluation task, such as task name, task description, dataset, scoring function, and API function.

[0052] At block 504, a request for executing the running strategy and the scoring strategy may be sent. This process may be performed in a loop for each data in the evaluation data set. Figure 3 As described above, the operation policy execution module 318 can send a request to execute the operation policy to the policy service module 344, and the scoring policy request module 324 can send a request to execute the scoring policy to the policy service module 344. In some embodiments, an HTTP request can be sent to obtain the file path of the API function, the module where the API function is located is obtained through the file path, and the context object (ctx object) required by the API function is encapsulated, and then the ctx object is serialized to access the FaaS service by sending an HTTP request. Then, when the policy service module can distribute the request, for example, when the policy service module receives the request, the entry function (for example, Figure 3The entry function 352 described above is used to distribute the request to the specified API function. For example, when distributing a request, the policy service module can parse the object in the request to obtain the ctx object and the path of the module, deserialize the ctx object, and then obtain the method containing the decorator (for example, the registration part described in FIG. 4 ) through Python reflection by loading the module path, thereby passing the deserialized context object to the function and executing the function.

[0053] At block 506, the execution results of the running strategy and the scoring strategy can be obtained. For example, the ctx object can be deserialized, and then the data set fields, scores, comments, etc. that need to be updated can be obtained from the deserialized ctx object. Figure 3 As described above, the evaluation data update module 320 can obtain the execution result of the operation strategy, which can include the output of the large model; the evaluation result processing module 326 can obtain the execution result of the scoring strategy, which can include the score, parameters, comments, etc. of the large model. For example, Figure 5C A schematic diagram of the execution result 500C of the policy request according to an embodiment of the present disclosure is shown. The execution result 500C shows information for each evaluation data, wherein the model output is the output of the large model running each evaluation data, and the evaluation score is the score for each evaluation data. The scores of each evaluation data are summarized to generate the evaluation score of the large model. In addition, the execution result also includes the benchmark truth value and execution status for each evaluation data.

[0054] Figure 6 FIG. 6 is a block diagram of an apparatus 600 for model evaluation according to some embodiments of the present disclosure. Figure 6 As shown, the device 600 includes a request sending module 602, which is configured to send a request for executing the selected evaluation strategy based on a user input indicating that an evaluation strategy for evaluating a target model is selected from multiple evaluation strategies, wherein the multiple evaluation strategies are published on a policy system, and the policy system includes: a policy creation module, which is configured to create a policy file for the evaluation strategy, and the policy file is stored in a database; a dependency setting module, which is configured to set the dependencies required for executing the evaluation strategy; and a policy publishing module, which is configured to publish the evaluation strategy to a policy service in the policy system. In addition, the device also includes a result acquisition module 604, which is configured to obtain the execution result of the request, and the execution result at least includes the evaluation result of the target model.

[0055] Figure 7 A block diagram of an electronic device 700 is shown in accordance with certain embodiments of the present disclosure. Figure 7FIG. 1 is a block diagram of an electronic device 700 according to some embodiments of the present disclosure. The device 700 may be a device or apparatus described in an embodiment of the present disclosure. Figure 7 As shown, the device 700 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 701, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 702 or computer program instructions loaded from a storage unit 708 to a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The CPU / GPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704. Although not shown in FIG. Figure 7 As shown in FIG. 7 , device 700 may further include a coprocessor.

[0056] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0057] The various methods or processes described above may be performed by the CPU / GPU 701. For example, in some embodiments, the methods may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the CPU / GPU 701, one or more steps or actions in the methods or processes described above may be performed.

[0058] In some embodiments, the methods and processes described above may be implemented as a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0059] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0060] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0061] The computer program instructions for performing the disclosed operation may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, programming languages ​​including object-oriented programming languages, and conventional procedural programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In certain embodiments, by utilizing the state information of a computer-readable program instruction to customize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit may execute a computer-readable program instruction, thereby realizing various aspects of the present disclosure.

[0062] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0063] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0064] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the equipment, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous frames can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the functions involved. It should also be noted that each frame in the block diagram and / or flow chart, and the combination of frames in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0065] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

[0066] Some example implementations of the present disclosure are listed below.

[0067] Example 1. A method for model evaluation, comprising:

[0068] Based on a user input indicating selection of an evaluation strategy for evaluating a target model from a plurality of evaluation strategies, sending a request for executing the selected evaluation strategy, the plurality of evaluation strategies being published on a policy system, the policy system being configured to: create a policy file for the evaluation strategy, the policy file being stored in a database; set dependencies required for executing the evaluation strategy; and publish the evaluation strategy to a policy service in the policy system; and

[0069] An execution result of the request is obtained, where the execution result at least includes an evaluation result of the target model.

[0070] Example 2. The method according to Example 1 further includes:

[0071] The execution result is displayed, wherein the execution result further includes at least one of the following: model input, model output, correct answer, and execution status.

[0072] Example 3. The method of any one of Examples 1-2, wherein the evaluation strategy includes an operation strategy and a scoring strategy, and receiving the user input includes:

[0073] receiving a first user input indicating a selection of the operation strategy from a plurality of operation strategies; and

[0074] A second user input is received, the second user input indicating selection of the scoring strategy from a plurality of scoring strategies.

[0075] Example 4. The method of any one of Examples 1-3, wherein sending a request for executing the selected evaluation strategy comprises:

[0076] sending a first request for executing the selected execution strategy, the execution strategy being configured to generate a model output based on the target model; and

[0077] A second request is sent for executing the selected scoring strategy, the scoring strategy being configured to generate the evaluation result based on the model output.

[0078] Example 5. The method of any one of Examples 1-4, wherein obtaining the execution result of the request comprises obtaining a first execution result of the first request, and the method further comprises:

[0079] Based on the first execution result, an evaluation data set used to evaluate the target model and a model output of the target model are updated.

[0080] Example 6. A method according to any one of Examples 1-5, wherein obtaining the execution result of the request includes obtaining a second execution result of the second request, and the method further includes: based on the second execution result, updating an evaluation data set used to evaluate the target model and an evaluation score for the target model.

[0081] Example 7. A method according to any one of Examples 1-6, wherein the identifier of the evaluation policy is a key value of the policy file in a data table of the database.

[0082] Example 8. The method according to any one of Examples 1-7, further comprising:

[0083] In response to detecting a request for executing the evaluation policy, the policy system obtains a context object for executing the evaluation policy;

[0084] The policy system executes the evaluation policy by transmitting the context object to the evaluation policy; and

[0085] The policy system generates the execution result by executing the evaluation policy.

[0086] Example 9. The method of any one of Examples 1-8, wherein dependencies required to execute the evaluation policy are specified in a dependency file, and publishing the evaluation policy to the policy service in the policy system comprises:

[0087] The policy file, the toolkit for the policy file, the entry function of the policy file, the configuration file of the policy service, and the dependent files are merged and published.

[0088] Example 10. A method according to any of Examples 1-9, wherein the evaluation strategy includes a published functional function.

[0089] Example 11: A device for model evaluation, comprising:

[0090] a request sending module configured to send a request for executing an evaluation strategy selected from a plurality of evaluation strategies for evaluating a target model based on a user input indicating that the evaluation strategy is selected from the plurality of evaluation strategies, the plurality of evaluation strategies being published on a policy system, the policy system comprising: a policy creating module configured to create a policy file for the evaluation strategy, the policy file being stored in a database; a dependency setting module configured to set dependencies required for executing the evaluation strategy; and a policy publishing module configured to publish the evaluation strategy to a policy service in the policy system; and

[0091] The result acquisition module is configured to acquire the execution result of the request, wherein the execution result at least includes the evaluation result of the target model.

[0092] Example 12. The apparatus of Example 11, further comprising:

[0093] The result display module is configured to display the execution result, wherein the execution result also includes at least one of the following: model input, model output, correct answer, and execution status.

[0094] Example 13. The apparatus of any one of Examples 11-12, wherein the evaluation strategy comprises an operation strategy and a scoring strategy, and the input receiving module comprises:

[0095] A first input receiving module is configured to receive a first user input, wherein the first user input indicates selecting the operation strategy from a plurality of operation strategies; and

[0096] The second input receiving module is configured to receive a second user input, where the second user input indicates selecting the scoring strategy from a plurality of scoring strategies.

[0097] Example 14. The apparatus of any one of Examples 11-13, wherein the request sending module comprises:

[0098] a first request sending module configured to send a first request for executing the selected operation strategy, wherein the operation strategy is configured to generate a model output based on the target model; and

[0099] The second request sending module is configured to send a second request for executing the selected scoring strategy, wherein the scoring strategy is configured to generate the evaluation result based on the model output.

[0100] Example 15. The apparatus according to any one of Examples 11-14, wherein the result acquisition module includes a first result acquisition module configured to acquire a first execution result of the first request, and the apparatus further includes:

[0101] An evaluation data updating module is configured to update an evaluation data set used to evaluate the target model and a model output of the target model based on the first execution result.

[0102] Example 16. The apparatus according to any one of Examples 11-15, wherein the result acquisition module includes a second result acquisition module configured to acquire a second execution result of the second request, and the apparatus further includes:

[0103] The second evaluation data updating module is configured to update the evaluation data set used to evaluate the target model and the evaluation score for the target model based on the second execution result.

[0104] Example 17. An apparatus according to any one of Examples 11-16, wherein the identifier of the evaluation policy is a key value of the policy file in a data table of the database.

[0105] Example 18. The apparatus of any one of Examples 11-17, further comprising:

[0106] a context object acquisition module, configured to, in response to detecting a request for executing the evaluation policy, cause the policy system to acquire a context object for executing the evaluation policy;

[0107] an evaluation policy execution module configured to execute the evaluation policy by transmitting the context object to the evaluation policy through the policy system; and

[0108] The evaluation result generating module is configured so that the policy system generates the execution result by executing the evaluation policy.

[0109] Example 19. The apparatus of any of Examples 11-18, wherein dependencies required to execute the evaluation policy are specified in a dependency file, and publishing the evaluation policy to the policy service in the policy system comprises:

[0110] The policy file, the toolkit for the policy file, the entry function of the policy file, the configuration file of the policy service, and the dependent files are merged and published.

[0111] Example 20. An apparatus according to any of Examples 11-19, wherein the evaluation strategy includes a published functional function.

[0112] Example 21. An electronic device comprising:

[0113] Processor; and

[0114] A memory coupled to the processor, the memory having instructions stored therein, wherein when the instructions are executed by the processor, the electronic device performs actions, the actions comprising:

[0115] Based on a user input indicating a selection of an evaluation strategy for evaluating a target model from a plurality of evaluation strategies, sending a request for executing the selected evaluation strategy, the plurality of evaluation strategies being published on a policy system, the policy system being configured to:

[0116] Creating a policy file of the evaluation policy, wherein the policy file is stored in a database;

[0117] Setting up the dependencies required to execute the evaluation strategy; and

[0118] publishing the evaluation policy to a policy service in the policy system; and

[0119] An execution result of the request is obtained, where the execution result at least includes an evaluation result of the target model.

[0120] Example 22. The electronic device of Example 21, further comprising:

[0121] The execution result is displayed, wherein the execution result further includes at least one of the following: model input, model output, correct answer, and execution status.

[0122] Example 23. The electronic device of any of Examples 21-22, wherein the evaluation strategy includes an operation strategy and a scoring strategy, and receiving the user input includes:

[0123] receiving a first user input indicating a selection of the operation strategy from a plurality of operation strategies; and

[0124] A second user input is received, the second user input indicating selection of the scoring strategy from a plurality of scoring strategies.

[0125] Example 24. The electronic device of any one of Examples 21-23, wherein sending a request for executing the selected evaluation strategy comprises:

[0126] sending a first request for executing the selected execution strategy, the execution strategy being configured to generate a model output based on the target model; and

[0127] A second request is sent for executing the selected scoring strategy, the scoring strategy being configured to generate the evaluation result based on the model output.

[0128] Example 25. The electronic device of any one of Examples 21-24, wherein obtaining the execution result of the request comprises obtaining a first execution result of the first request, and the action further comprises:

[0129] Based on the first execution result, an evaluation data set used to evaluate the target model and a model output of the target model are updated.

[0130] Example 26. The electronic device of any one of Examples 21-25, wherein obtaining the execution result of the request comprises obtaining a second execution result of the second request, and the action further comprises:

[0131] Based on the second execution result, an evaluation data set used to evaluate the target model and an evaluation score for the target model are updated.

[0132] Example 27. An electronic device according to any one of Examples 21-26, wherein the identifier of the evaluation policy is a key value of the policy file in a data table of the database.

[0133] Example 28. The electronic device of any one of Examples 21-27, further comprising:

[0134] In response to detecting a request for executing the evaluation policy, the policy system obtains a context object for executing the evaluation policy;

[0135] The policy system executes the evaluation policy by transmitting the context object to the evaluation policy; and

[0136] The policy system generates the execution result by executing the evaluation policy.

[0137] Example 29. The electronic device of any one of Examples 21-28, wherein dependencies required to execute the evaluation policy are specified in a dependency file, and publishing the evaluation policy to the policy service in the policy system comprises:

[0138] The policy file, the toolkit for the policy file, the entry function of the policy file, the configuration file of the policy service, and the dependent files are merged and published.

[0139] Example 30. An electronic device according to any of Examples 21-29, wherein the evaluation strategy includes a published functionality.

[0140] Example 31. A computer-readable storage medium having one or more computer instructions stored thereon, wherein the one or more computer instructions are executed by a processor to implement the method according to any one of Examples 1 to 10.

[0141] Example 32. A computer program product tangibly stored on a computer-readable medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform a method according to any one of Examples 1 to 10.

[0142] Although the disclosure has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Instead, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for model evaluation, include: Based on a user input indicating a selection of an evaluation strategy for evaluating a target model from a plurality of evaluation strategies, sending a request for executing the selected evaluation strategy, the plurality of evaluation strategies being published on a policy system, the policy system being configured to: Creating a policy file of the evaluation policy, wherein the policy file is stored in a database; Setting up the dependencies required to execute the evaluation strategy; as well as The evaluation policy is published to a policy service in the policy system; and an execution result of the request is obtained, wherein the execution result at least includes an evaluation result of the target model.

2. The method according to claim 1, further comprising: include: The execution result is displayed, and the execution result also includes at least one of the following: model input, model output, correct answer, and execution status.

3. The method according to claim 1, wherein the evaluation strategy includes an operation strategy and a scoring strategy, and receiving the user input include: receiving a first user input indicating selection of the operation strategy from a plurality of operation strategies; as well as A second user input is received, the second user input indicating selection of the scoring strategy from a plurality of scoring strategies.

4. The method according to claim 3, wherein a request is sent for executing the selected evaluation strategy. include: sending a first request for executing the selected execution strategy, the execution strategy being configured to generate a model output based on the target model; as well as A second request is sent for executing the selected scoring strategy, the scoring strategy being configured to generate the evaluation result based on the model output.

5. The method according to claim 4, wherein obtaining the execution result of the request comprises obtaining a first execution result of the first request, and the method further include: Based on the first execution result, an evaluation data set used to evaluate the target model and a model output of the target model are updated.

6. The method according to claim 4, wherein obtaining the execution result of the request comprises obtaining a second execution result of the second request, and the method further include: Based on the second execution result, an evaluation data set used to evaluate the target model and an evaluation score for the target model are updated. 7 . The method according to claim 1 , wherein the identifier of the evaluation policy is a key value of the policy file in a data table of the database.

8. The method according to claim 7, further comprising: include: In response to detecting a request for executing the evaluation policy, the policy system obtains a context object for executing the evaluation policy; The policy system executes the evaluation policy by transmitting the context object to the evaluation policy; as well as The policy system generates the execution result by executing the evaluation policy.

9. The method of claim 8, wherein the dependencies required to execute the evaluation policy are specified in a dependency file, and the evaluation policy is published to the policy service in the policy system include: The policy file, the toolkit for the policy file, the entry function of the policy file, the configuration file of the policy service, and the dependent files are merged and published.

10. The method of claim 8, wherein the evaluation strategy comprises a published functionality.

11. A device for model evaluation, include: A request sending module is configured to send a request for executing an evaluation strategy selected from a plurality of evaluation strategies for evaluating a target model based on a user input indicating that the evaluation strategy is selected from the plurality of evaluation strategies, wherein the plurality of evaluation strategies are published on a policy system, wherein the policy system comprises: A strategy creation module, configured to create a strategy file of the evaluation strategy, wherein the strategy file is stored in a database; A dependency setting module configured to set dependencies required for executing the evaluation strategy; and a policy publishing module configured to publish the evaluation policy to a policy service in the policy system; and The result acquisition module is configured to acquire the execution result of the request, wherein the execution result at least includes the evaluation result of the target model.

12. An electronic device, include: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, wherein when the instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 10.

13. A computer program product tangibly stored on a non-transitory computer readable medium and comprising computer executable instructions for performing the method according to any one of claims 1 to 10.

Citation Information

Patent Citations

  • Method for managing storage device, electronic device and computer program product

    CN115878018A

  • Method, device and equipment for evaluating model performance, medium and program product

    CN117648140A

  • Model evaluation method, electronic equipment and storage medium

    CN117724941A

  • Dynamic evaluation and selection of file system pre-fetch policy

    US10474635B1

  • Fetching vendor specific policy events and corresponding device feature mappings from a policy server at mobile device runtime of a managed application

    US20170093877A1

Cited By

  • Personal information protection evaluation method based on FaaS and deep learning

    CN121659354A