Model evaluation methods and equipment
By implementing algorithm model evaluation on the server side and using the service agent to automatically switch the environment, the problems of evaluation complexity and low efficiency in the existing technology are solved, and efficient model evaluation management and data support are achieved.
Patent Information
- Application Number
- CN202210770160.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-30
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-06-30
AI Technical Summary
The existing algorithm model evaluation technology has a high threshold, it is difficult to achieve cross-environmental evaluation, lacks effective task management and quantitative analysis, and the mobile terminal evaluation efficiency is low.
Implement algorithm model evaluation on the server side (such as cloud server), automatically switch different environments through service agents for evaluation, and provide model evaluation services. Users can customize configuration and trigger evaluation tasks, without the need to configure the environment locally, automatically obtain analysis results and quantitative data. .
It improves model evaluation efficiency, saves labor and time costs, realizes platformization of model evaluation, accumulates evaluation task history, facilitates management and analysis, provides objective data support, has concurrency capabilities, and supports evaluation service.
Smart Images

Figure CN115145813B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of network technology, and more specifically, to a model evaluation method, a model evaluation device, a computing device, a computer-readable storage medium, and a computer program product. Background Art
[0002] Model (e.g., algorithm model) evaluation is significantly different from traditional software evaluation. The technical threshold for algorithm model evaluation is higher, and the evaluation requirements are more complex. For example, when evaluating artificial intelligence algorithm models, the reliance on large amounts of data and evaluation analysis have become challenges in algorithm model evaluation.
[0003] The demand for testing algorithm models is increasing, and model evaluation often involves many business parties. However, the current algorithm model evaluation is limited to local environment construction, followed by local testing and analysis, and its implementation is limited to scripts and manual forms. At the same time, service testing for different environments requires switching to the corresponding environment, the operation threshold requirements are relatively high, and there are problems such as decentralized task management. Summary of the invention
[0004] The present disclosure provides a model evaluation method, a model evaluation device, a computing device, a computer-readable storage medium, and a computer program product to solve the problems involved in the above-mentioned related technologies.
[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an algorithm model evaluation method, the model evaluation method comprising: creating a model evaluation task in response to receiving a model evaluation request; determining a service agent for executing the model evaluation task; and remotely calling the service agent to execute the model evaluation task through the service agent.
[0006] Optionally, determining the service agent for executing the model evaluation task includes: determining a service deployment environment corresponding to the model evaluation task, and environment configuration parameters matching the service deployment environment; and determining the service agent based on the environment configuration parameters.
[0007] Optionally, determining the service agent corresponding to the service deployment environment includes: determining the service deployment environment corresponding to the model evaluation task, and environment configuration parameters matching the service deployment environment; and determining the service agent based on the environment configuration parameters.
[0008] Optionally, in response to receiving a model evaluation request, creating a model evaluation task includes: in response to the model evaluation request, obtaining evaluation configuration parameters corresponding to the model evaluation request; outputting evaluation configuration entries based on the evaluation configuration parameters; in response to a setting operation for the evaluation configuration entry, obtaining the set evaluation configuration entry; and creating a model evaluation task based on the set evaluation configuration entry.
[0009] Optionally, the remote calling of the service agent to execute the model evaluation task through the service agent includes: constructing a remote service request according to the evaluation configuration parameters; and sending the remote service request to the service agent to request the service agent to execute the model evaluation task.
[0010] Optionally, in response to the model evaluation request, obtaining evaluation configuration parameters corresponding to the model evaluation request includes: in response to the model evaluation request, determining the model evaluation service corresponding to the model evaluation request; obtaining the evaluation configuration parameters corresponding to the model evaluation service from a preset evaluation configuration file, wherein the preset evaluation configuration file includes: preset evaluation configuration parameters respectively mapped to one or more preset model evaluation services.
[0011] Optionally, in response to a setting operation on the evaluation configuration entry, obtaining the set evaluation configuration entry includes: in response to an input operation on the evaluation configuration entry, obtaining the input evaluation configuration entry, and using the input evaluation configuration entry as the set evaluation configuration entry; and / or, in response to a selection operation on multiple configuration options under the evaluation configuration entry, determining the selected configuration option, and obtaining the set evaluation configuration entry according to the selected configuration option.
[0012] Optionally, the model evaluation method also includes: after the model evaluation task is executed, obtaining the execution result of the model evaluation task through the service agent; obtaining the model analysis type of the model evaluation task; and analyzing the execution result according to the model analysis type to generate evaluation analysis data.
[0013] Optionally, analyzing the execution result according to the model analysis type includes: rendering the execution result if the model analysis type is a renderable type; and not rendering the execution result if the model analysis type is a non-renderable type.
[0014] Optionally, after generating the evaluation and analysis data, the method further includes: quantifying the evaluation and analysis data to generate a model evaluation report.
[0015] According to a second aspect of an embodiment of the present disclosure, a model evaluation device is provided, comprising: a task creation unit, configured to create a model evaluation task in response to receiving a model evaluation request; an agent determination unit, configured to determine a service agent for executing the model evaluation task; and an agent calling unit, configured to remotely call the service agent to execute the model evaluation task through the service agent.
[0016] Optionally, the agent determination unit is further configured to: determine a service deployment environment corresponding to the model evaluation task, and environment configuration parameters matching the service deployment environment; and determine the service agent according to the environment configuration parameters.
[0017] Optionally, the task creation unit is further configured to: in response to the model evaluation request, obtain evaluation configuration parameters corresponding to the model evaluation request; output evaluation configuration entries according to the evaluation configuration parameters; in response to a setting operation for the evaluation configuration entry, obtain the set evaluation configuration entry; and create a model evaluation task according to the set evaluation configuration entry.
[0018] Optionally, the agent calling unit is further configured to: construct a remote service request according to the evaluation configuration parameters; and send the remote service request to the service agent to request the service agent to perform the model evaluation task.
[0019] Optionally, the task creation unit is further configured to: in response to the model evaluation request, determine the model evaluation service corresponding to the model evaluation request; obtain evaluation configuration parameters corresponding to the model evaluation service from a preset evaluation configuration file, wherein the preset evaluation configuration file includes: preset evaluation configuration parameters respectively mapped to one or more preset model evaluation services.
[0020] Optionally, the task creation unit is further configured to: in response to an input operation on the evaluation configuration entry, obtain the input evaluation configuration entry, and use the input evaluation configuration entry as the set evaluation configuration entry; and / or, in response to a selection operation on multiple configuration options under the evaluation configuration entry, determine the selected configuration option, and obtain the set evaluation configuration entry based on the selected configuration option.
[0021] Optionally, the model evaluation device also includes an analysis unit, which is configured to: after the model evaluation task is completed, obtain the execution result of the model evaluation task through the service agent; obtain the model analysis type of the model evaluation task; and analyze the execution result according to the model analysis type to generate evaluation analysis data.
[0022] Optionally, the analysis unit is further configured to: render the execution result if the model analysis type is a renderable type; and not render the execution result if the model analysis type is a non-renderable type.
[0023] Optionally, the model evaluation device further includes a quantification unit configured to: quantify the evaluation and analysis data to generate a model evaluation report.
[0024] According to a third aspect of an embodiment of the present disclosure, a computing device is provided, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, cause the at least one processor to execute the algorithm model evaluation method according to the present disclosure.
[0025] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is provided. When the instructions are executed by at least one processor, the at least one processor is prompted to execute the model evaluation method according to the present disclosure.
[0026] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, and instructions in the computer program product can be executed by a processor of a computer device to complete the model evaluation method according to the present disclosure.
[0027] The technical solution provided by the embodiments of the present disclosure brings at least one of the following beneficial effects. The model evaluation method, model evaluation device, computing device, computer-readable storage medium and computer program product proposed in the present disclosure can improve the model evaluation efficiency of the server, facilitate the realization of test service, improve the management of model evaluation tasks, and thus efficiently respond to frequently iterated test requirements.
[0028] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0030] Figure 1 A diagram showing an application scenario of an algorithm model evaluation method and device according to an exemplary embodiment of the present disclosure.
[0031] Figure 2 A flowchart of an algorithm model evaluation method according to an exemplary embodiment of the present disclosure is shown.
[0032] Figure 3A technical architecture diagram of an algorithm model evaluation method according to an exemplary embodiment of the present disclosure is shown.
[0033] Figure 4 Another flow chart of an algorithm model evaluation method according to an exemplary embodiment of the present disclosure is shown.
[0034] Figure 5 Another flow chart of an algorithm model evaluation method according to an exemplary embodiment of the present disclosure is shown.
[0035] Figure 6 A block diagram of an algorithm model evaluation device according to an exemplary embodiment of the present disclosure is shown.
[0036] Figure 7 A block diagram of a computing device according to an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0037] The usual algorithm model evaluation scheme is limited to local evaluation, or evaluation is performed by calling services locally. Each time a model evaluation task is executed, the corresponding environment needs to be configured locally, which has high technical threshold requirements and high configuration costs. When it comes to model evaluation in different environments, it is necessary to switch the environment of the main caller accordingly. This model evaluation method is difficult to represent the effect of the final completed model, and the environment switching operation and a large number of repeated environment configuration operations lead to increased labor costs and time-consuming and labor-intensive. Secondly, the existing evaluation method lacks standardized evaluation task management, and it is difficult to effectively utilize the various resources of the evaluation service platform. Furthermore, the existing evaluation method is not directly connected to the quantification of model evaluation, and it is difficult to efficiently provide evaluation quantitative data. In addition, the evaluation of algorithm models on mobile terminals such as mobile phones will be affected by the configuration and performance of the mobile terminal itself, resulting in poor evaluation efficiency.
[0038] Taking into account the many problems in the existing algorithm model evaluation technology, the present disclosure proposes a model evaluation method and device, which can realize algorithm model evaluation on the server side (for example, cloud server), automatically provide evaluation in different environments (for example, testing (STAGING) environment, pre-release (PRT) environment, online (PROD) environment, candidate (CANDIDATE) environment), and the evaluation process is close to the real online interface call process, saving labor costs and time costs. By realizing the platformization of model evaluation on the server side, the evaluation task history can be effectively accumulated, which is convenient for managing and tracking model evaluation tasks, making model evaluation indicators persistent, and providing basic data for subsequent model evaluation analysis. When performing model evaluation on the server side, there is no need to consider problems such as limited performance of mobile terminals. Model evaluation services can be provided to callers in the form of services, with concurrency capabilities, to realize evaluation services. Through the technical solution proposed in the present invention, users can customize the configuration of model evaluation and automatically trigger the execution of model evaluation tasks. There is no need to configure the local environment, and model evaluation services in different environments can be called at no cost. Evaluation and analysis results and quantitative results can be automatically obtained, connecting evaluation, analysis and quantification, so that model evaluation has objective data support.
[0039] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings, but the present disclosure is not limited to these embodiments.
[0040] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.
[0041] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.
[0042] Figure 1 The following is an application scenario diagram of the model evaluation method and device according to the exemplary embodiment of the present disclosure. The application scenario described in the embodiment is to more clearly illustrate the technical solution according to the embodiment of the present disclosure, and does not constitute a limitation on the technical solution provided by the embodiment of the present disclosure. A person of ordinary skill in the art can know that with the emergence of new application scenarios, the technical solution according to the embodiment of the present disclosure is also applicable to solving similar technical problems.
[0043] like Figure 1 As shown, the server 11 can communicate with the client through the network 12. The server 11 may include a cloud server. The client may include a user terminal 131 (e.g., a mobile phone), a user terminal 132 (e.g., a laptop computer), and a user terminal 133 (e.g., a tablet computer). The model evaluation method according to the embodiment of the present disclosure can be executed by the server 11. The model evaluation device according to the embodiment of the present disclosure can be applied to the server 11. The user can request the server 11 to perform model evaluation through the user terminal 131, 132 or 133.
[0044] The technical solution disclosed in the present invention can be applied to model evaluation of various algorithms (e.g., computer vision (CV) algorithms, natural language processing (NLP) algorithms, machine learning algorithms, etc.), and can be evaluated according to evaluation indicators of different algorithm models. For example, a computer vision (CV) algorithm model may include a facial key point recognition model, a portrait segmentation model, a gesture recognition model, etc.
[0045] Figure 2 A flow chart of a model evaluation method according to an exemplary embodiment of the present disclosure is shown. In an embodiment of the present disclosure, the model evaluation method can be executed by a server.
[0046] In step S201, a model evaluation request is received. For example, the server may receive a model evaluation request input by a user from a mobile terminal such as a mobile phone or a computer. In one example, when a user desires to evaluate a computer vision (CV) algorithm model, the server may receive a CV model evaluation request from the user.
[0047] In step S202, in response to receiving a model evaluation request, a model evaluation task is created. For example, in response to the model evaluation request, an evaluation configuration parameter corresponding to the model evaluation request is obtained; according to the evaluation configuration parameter, an evaluation configuration item is output; in response to a setting operation for the evaluation configuration item, the set evaluation configuration item is obtained; and according to the set evaluation configuration item, a model evaluation task is created.
[0048] According to an embodiment of the present disclosure, in response to a received model evaluation request, an evaluation configuration parameter corresponding to the model evaluation request may be obtained. For example, after receiving the model evaluation request, the server may retrieve the corresponding evaluation configuration parameter according to the model evaluation request. For example, in response to receiving a CV model evaluation request, a CV evaluation configuration parameter corresponding to the CV model evaluation request may be obtained.
[0049] In an embodiment of the present disclosure, in response to a received model evaluation request, a model evaluation service corresponding to the model evaluation request is determined; and evaluation configuration parameters corresponding to the model evaluation service are obtained from a preset evaluation configuration file. For example, the preset evaluation configuration file may include: preset evaluation configuration parameters respectively mapped to one or more preset model evaluation services.
[0050] According to an embodiment of the present disclosure, a preset evaluation configuration file is stored on the server side, and a mapping relationship between a preset model evaluation service (for example, various CV evaluation services (face key point recognition model evaluation service, portrait segmentation model evaluation service, gesture recognition model evaluation service)) and corresponding preset evaluation configuration parameters can be pre-set in the preset evaluation configuration file. According to the mapping relationship in the preset evaluation configuration file, the evaluation configuration parameters corresponding to the model evaluation service can be obtained from the preset evaluation configuration file.
[0051] The evaluation configuration parameters may include various configuration parameters related to the algorithm model evaluation. For example, the evaluation configuration parameters may include parameters related to at least one of the following items: task name, service deployment environment (for example, testing (STAGING) environment, pre-release (PRT) environment, online (PROD) environment, candidate (CANDIDATE) environment), model type (for example, GAN series algorithm type, basic vision algorithm type), model name, model test set, model testing tool, model analysis type, model quantization type.
[0052] In an embodiment of the present disclosure, in response to receiving a model evaluation request, an evaluation configuration item may be output according to the evaluation configuration parameters. For example, the server may generate and output the evaluation configuration item according to the evaluation configuration parameters for the user to customize. Therefore, the user may customize the relevant configuration of the model evaluation task, such as the service deployment environment, by setting the evaluation configuration item.
[0053] In an embodiment of the present disclosure, the evaluation configuration entry may include at least one of the following items: task name, service deployment environment, model type, model name, model test set, model test tool, model quantization type, model test type.
[0054] Then, the setting for the evaluation configuration item can be received from the user. For example, the setting for the evaluation configuration item is received by receiving input information for the evaluation configuration item. Alternatively, the setting for the evaluation configuration item is received by receiving a selection of multiple configuration options under the evaluation configuration item. In this way, the user can make customized settings for the evaluation configuration item provided by the server.
[0055] For example, the user may enter information under an evaluation configuration item that requests user input (e.g., a task name that needs to be customized by the user, an email address for the user to receive the evaluation report, etc.). The user may select a configuration option from multiple configuration options under the evaluation configuration item to customize the evaluation configuration desired by the user. In this way, the server may receive the evaluation configuration item set by the user.
[0056] In response to a setting operation for the evaluation configuration entry, the evaluation configuration entry after setting is obtained, and a model evaluation task can be created according to the evaluation configuration entry after setting. For example, in response to an input operation for the evaluation configuration entry, the input evaluation configuration entry is obtained, and the input evaluation configuration entry is used as the evaluation configuration entry after setting; and / or, in response to a selection operation for multiple configuration options under the evaluation configuration entry, the selected configuration option is determined, and the evaluation configuration entry after setting is obtained according to the selected configuration option. When the evaluation configuration entry after setting is obtained or received on the server side, the creation of the model evaluation task can be automatically triggered. In this way, the pipeline job can be called, and dynamic construction can be performed according to the obtained evaluation configuration parameters and the settings of the front end to call the service agent. For example, an RPC service call can be made through an http agent.
[0057] In step S203, a service agent for executing the model evaluation task may be determined. For example, the service agent corresponding to the created model evaluation task may be identified.
[0058] According to the embodiments of the present disclosure, a service deployment environment corresponding to a model evaluation task and environment configuration parameters matching the service deployment environment can be determined; according to the environment configuration parameters, it is determined that the service agent can be combined with Figure 3 and Figure 5 The illustrated embodiments are used to understand the operation of determining a service agent.
[0059] The service deployment environment of the model evaluation task can be identified. Model evaluation tasks all contain corresponding service deployment environments. Therefore, its service deployment environment can be identified from the model evaluation task. The service deployment environment may include a test (STAGING) environment, a pre-launch (PRT) environment, an online (PROD) environment, and a candidate (CANDIDATE) environment. Determine the service agent corresponding to the service deployment environment. For example, the corresponding service agent can be determined as a test service agent based on the test (STAGING) environment, and the corresponding service agent can be determined as a pre-launch service agent based on the pre-launch (PRT) environment.
[0060] In step S204, the service agent may be remotely called to execute the model evaluation task through the service agent. The service agent may be called through a remote procedure call (RPC). For example, an RPC service may be called through an HTTP service interface to call the service agent, thereby executing the model evaluation task in the service deployment environment. In an embodiment of the present disclosure, a service agent may be used to execute the model evaluation task. For example, a test service agent may be used to execute the corresponding model evaluation task in a test environment, and a pre-launch service agent may be used to execute the corresponding model evaluation task in a pre-launch environment.
[0061] For example, through remote procedure call (RPC), the model evaluation task is executed in the service deployment environment corresponding to the model evaluation task. In the embodiment of the present disclosure, the server can dynamically call the service in the corresponding service deployment environment through RPC to realize the execution of the model evaluation task in the corresponding service deployment environment without local call.
[0062] A service agent for executing the model evaluation task may be determined according to the service deployment environment of the model evaluation task; and the service agent may be called through a remote procedure call (RPC) to execute the model evaluation task.
[0063] In an embodiment of the present disclosure, a remote service request can be constructed according to the evaluation configuration parameters corresponding to the model evaluation request obtained; the remote service request is sent to the service agent to request the service agent to perform the model evaluation task. In this way, the service agent can be remotely called by constructing and sending the remote service request.
[0064] In addition, after the model evaluation task is completed (for example, in response to the model evaluation task being completed), the execution result of the model evaluation task can be obtained; according to the model analysis type of the model evaluation task, the execution result is analyzed to generate evaluation analysis data. Optionally, the evaluation analysis data can be quantified to generate a model evaluation report.
[0065] The model evaluation method according to an exemplary embodiment of the present disclosure will be further described below.
[0066] Figure 3 A technical architecture diagram of a model evaluation method according to an exemplary embodiment of the present disclosure is shown.
[0067] Can be implemented on the server side Figure 3 The technical architecture diagram shown in Figure 1 is as follows. Figure 3 As shown, a mapping relationship between a preset model evaluation service and a corresponding preset evaluation configuration parameter is pre-set in the "KConf service parameter dynamic configuration". The "test task configuration" module can query the evaluation configuration parameters from the "KConf service parameter dynamic configuration" module, and accordingly, the "KConf service parameter dynamic configuration" module can feedback a parameter list to the "test task configuration" module. The "test task configuration" module can determine the evaluation configuration entries (for example, entries corresponding to deploy_env, kessname, lane_name, server_params, resources, blobstore, and eval_method) according to the evaluation configuration parameters. Based on this, a model evaluation task can be created. The model evaluation task can be submitted for task creation (for example, ATTM job and the platform general node of the server-side evaluation platform). The model evaluation task can be created through the "dockerfile binding node label" module, the "kml image construction" module, etc., that is, the service creation is completed.
[0068] Then, the model evaluation task can be executed in the service deployment environment corresponding to the model evaluation task through the "container HB1:ZW:PROD" module and RPC. For example, the corresponding service proxy can be called through the "resource acquisition" module and the "request dynamic construction" module with the help of the HTTP interface (for example, calling the proxy interface (proxy api (for example, STAGING-api, PRT-api)) through HTTP POST) to match the corresponding service deployment environment (for example, STAGING environment, PRT environment under the ytech model rpc service). In addition, kessname, proto information, etc. can also be exchanged with the INFRA module. The proxy interface can send a response to the "render_type parsing_type" module. The "render_type parsing_type" module can send kessname and other information to the "kconf service-info-mapping" module and receive corresponding meta information. In addition, the data can be analyzed through the "generative keypoint classification seg parsing bboxinstance-seg" module (for example, parsing using keypoint method, rendering using keypoint method, etc.).
[0069] Optionally, the evaluation and analysis data can be quantified through the "Quantitative Calculation" module. The evaluation results can be stored through the "Task Result Structured Storage" module. The generated model evaluation report can be displayed through the "HTML Report Display". In addition, the "Manual Storage" module can be used to allow users to conveniently perform manual storage operations. In this way, the server can support multiple model evaluation functions.
[0070] As described above, the corresponding service agent can be determined according to the service deployment environment of the model evaluation task, and the model evaluation task can be executed by calling the corresponding service agent through RPC.
[0071] Figure 4 Another flow chart of a model evaluation method according to an exemplary embodiment of the present disclosure is shown.
[0072] In step S401, in response to the model evaluation task being completed, the execution result of the model evaluation task is obtained through the service agent. The obtained execution result can be used for subsequent evaluation and analysis. In step S402, the model analysis type of the model evaluation task can be obtained to determine the analysis method corresponding to the model analysis type.
[0073] In step S403, the execution results are analyzed according to the model analysis type to generate evaluation analysis data. Model evaluation tasks may all contain corresponding model analysis types. How to analyze the execution results can be determined according to the specific model analysis type. For example, if the model analysis type is a renderable type, the execution result is rendered; if the model analysis type is a non-renderable type, the execution result is not rendered. In addition, after the task is executed, the user who requested the evaluation (i.e., the task creator) can be notified in a timely manner so that the user can understand the progress of the activity.
[0074] After the evaluation and analysis data is generated, the evaluation and analysis data can be selectively quantified. In step S404, the evaluation and analysis data is quantified to generate a model evaluation report. In an embodiment of the present disclosure, the model evaluation report may include data, charts, etc. related to the model evaluation indicators. And the form of the model evaluation report may include charts, text, animations, etc. The generated model evaluation report can be stored on the server side and can be provided to the user for timely viewing by the user. In addition, the evaluation and analysis data and / or model evaluation reports of multiple algorithm models can be stored on the server side, so that different model evaluation tasks can be managed and tracked through the server side. For example, model evaluation reports of different periods can be viewed at any time. In an embodiment of the present disclosure, the execution, analysis and quantification of the model evaluation task can be connected in series through the RPC call service via the HTTP interface to achieve support for multiple functions.
[0075] The following can be combined Figure 5 To further understand the exemplary operation of the model evaluation method.
[0076] Figure 5 Another flow chart of a model evaluation method according to an exemplary embodiment of the present disclosure is shown.
[0077] exist Figure 5 In the illustrated example, CV model evaluation in a testing (STAGING) environment and a pre-release (PRT) environment is taken as an example for description, but the present disclosure is not limited thereto.
[0078] In model evaluation tasks (e.g. Figure 5 After the graph running task in the example is created, the model evaluation task claiming interface can be called back and the graph running start can be notified.
[0079] In step S501, it is determined whether the service deployment environment corresponding to the model evaluation task is a test environment or a pre-launch environment. In response to the determination that it is a test environment, according to the evaluation configuration parameters corresponding to the model evaluation request, an http post request is constructed (step S502), and the test service agent is called (step S503). In response to the determination that it is a pre-launch environment, according to the evaluation configuration parameters corresponding to the model evaluation request, an http post request is constructed (step S504), and the pre-launch service agent is called (step S505).
[0080] For the running task, it can be determined whether the non-rendering mode needs to be set. For example, in step S506, it is determined whether the corresponding field in the running task is set to the non-rendering mode. In response to determining that the corresponding field is not set to the non-rendering mode, step S507 is executed to set the corresponding field to the non-rendering mode; otherwise, step S508 is executed.
[0081] In step S508, it is determined whether the model analysis type is a model analysis type that can parse the prediction results to determine whether it is a renderable type. For example, the model analysis type may include a GAN series algorithm type and a basic visual algorithm type. The GAN series algorithm type belongs to a non-renderable type, and the basic visual algorithm type belongs to a renderable type. That is to say, for the GAN series algorithm type, no rendering is required, while for the basic visual algorithm type, the synthesis result needs to be rendered, for example, the keypoint method can be used for parsing and rendering. Therefore, in response to determining that the model analysis type is a model analysis type that can parse the prediction results, step S509 is executed to determine not to render the synthesis result. In response to determining that the model analysis type is a model analysis type that can parse the prediction results, step S510 is executed to render the synthesis result.
[0082] As described above, in response to the completion of the execution of the model evaluation task, the execution result of the model evaluation task can be obtained; according to the model analysis type of the model evaluation task, the execution result is analyzed to generate evaluation analysis data.
[0083] After the model evaluation task is executed or the evaluation analysis data is generated, the task execution completion may be notified. For example, in step S511, the running graph is notified, and the task status interface may be called back to update.
[0084] Optionally, in step S512, it may be determined whether quantization is required. Otherwise, step S512 is executed in a loop.
[0085] In response to determining that quantization is required, step S513 is executed to quantify the evaluation analysis data to generate a model evaluation report. For example, the model evaluation report may include the comparison effect before and after the algorithm model is applied, the data quantization results, the server execution log, etc.
[0086] Figure 6 A block diagram of a model evaluation device 6 according to an exemplary embodiment of the present disclosure is shown. The model evaluation device 6 may be provided in a server.
[0087] like Figure 6 As shown, the model evaluation device 6 may include a task creation unit 61, which is configured to create a model evaluation task in response to receiving a model evaluation request. The model evaluation device 6 may also include an agent determination unit 62, which is configured to determine a service agent for executing the model evaluation task. The model evaluation device 6 may also include an agent calling unit 63, which is configured to remotely call the service agent to execute the model evaluation task through the service agent.
[0088] According to an embodiment of the present disclosure, the task creation unit 61 may be further configured to: in response to the model evaluation request, obtain the evaluation configuration parameters corresponding to the model evaluation request; output the evaluation configuration items according to the evaluation configuration parameters; in response to the setting operation for the evaluation configuration items, obtain the set evaluation configuration items; create the model evaluation task according to the set evaluation configuration items. For example, the task creation unit 61 may be further configured to: in response to the model evaluation request, determine the model evaluation service corresponding to the model evaluation request; obtain the evaluation configuration parameters corresponding to the model evaluation service from the preset evaluation configuration file, wherein the preset evaluation configuration file includes: preset evaluation configuration parameters respectively mapped to one or more preset model evaluation services. The evaluation configuration parameters include parameters related to at least one of the following items: task name, service deployment environment, model type, model name, model test set, model test tool, model analysis type, model quantization type. In addition, the evaluation configuration items include at least one of the following items: task name, service deployment environment, model type, model name, model test set, model test tool, model quantization type, model test type.
[0089] According to an embodiment of the present disclosure, the task creation unit 61 may be further configured to: in response to an input operation for the evaluation configuration entry, obtain the input evaluation configuration entry, and use the input evaluation configuration entry as the set evaluation configuration entry; and / or, in response to a selection operation for multiple configuration options under the evaluation configuration entry, determine the selected configuration option, and obtain the set evaluation configuration entry based on the selected configuration option.
[0090] According to an embodiment of the present disclosure, the agent determination unit 62 may be further configured to: determine a service deployment environment corresponding to the model evaluation task, and environment configuration parameters matching the service deployment environment; and determine the service agent according to the environment configuration parameters.
[0091] According to an embodiment of the present disclosure, the agent calling unit 63 may be further configured to: construct a remote service request according to the evaluation configuration parameters; and send the remote service request to the service agent to request the service agent to perform the model evaluation task.
[0092] In addition, the model evaluation device 6 may include an analysis unit 65, which is configured to: after the model evaluation task is executed, obtain the execution result of the model evaluation task through the service agent; obtain the model analysis type of the model evaluation task; and analyze the execution result according to the model analysis type to generate evaluation analysis data. For example, the analysis unit 65 may be further configured to: if the model analysis type is a renderable type, render the execution result; if the model analysis type is a non-renderable type, do not render the execution result.
[0093] The model evaluation device 6 may include a quantification unit 66 configured to quantify the evaluation analysis data to generate a model evaluation report.
[0094] Can be combined with reference Figures 1 to 5 The model evaluation method described here understands the corresponding operations of each unit in the model evaluation device 6. For the sake of brevity, it will not be repeated here.
[0095] Figure 7 700 is a block diagram of a computing device according to an exemplary embodiment of the present disclosure. The computing device 700 may be, for example, a smart phone, a tablet computer, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer, or a desktop computer. The computing device 700 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, or other names.
[0096] Typically, the computing device 700 includes a processor 701 and a memory 702 .
[0097] The processor 701 may include one or more processing cores, such as a quad-core processor, an octa-core processor, and the like. The processor 701 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), or PLA (Programmable Logic Array). The processor 701 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 701 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 701 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.
[0098] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 702 is used to store at least one instruction, which is used to be executed by the processor 701 to implement an exemplary embodiment according to the present disclosure (for example, Figures 2 to 5 The embodiment shown in the figure) is a model evaluation method.
[0099] In some embodiments, the computing device 700 may further optionally include: a peripheral device interface 703 and at least one peripheral device. The processor 701, the memory 702 and the peripheral device interface 703 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 703 via a bus, a signal line or a circuit board. Specifically, the peripheral device may include: at least one of a radio frequency circuit 704, a touch display screen 705, a camera assembly 706, an audio circuit 707, a positioning assembly 708 and a power supply 709.
[0100] The peripheral device interface 703 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 701 and the memory 702. In some embodiments, the processor 701, the memory 702, and the peripheral device interface 703 are integrated on the same chip or circuit board; in some other embodiments, any one or more of the processor 701, the memory 702, and the peripheral device interface 703 may be implemented on a separate chip or circuit board, but the present disclosure is not limited thereto.
[0101] The radio frequency circuit 704 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 704 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 704 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 704 may also include circuits related to NFC (Near Field Communication), but the present disclosure is not limited thereto.
[0102] The display screen 705 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 705 is a touch display screen, the display screen 705 also has the ability to collect touch signals on the surface or above the surface of the display screen 705. The touch signal can be input to the processor 701 as a control signal for processing. At this time, the display screen 705 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 705 can be one, set on the front panel of the computing device 700; in other embodiments, the display screen 705 can be at least two, respectively set on different surfaces of the computing device 700 panels or folding design; in other embodiments, the display screen 705 can be a flexible display screen, set on the curved surface or folding surface of the computing device 700. Even, the display screen 705 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 705 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).
[0103] The camera assembly 706 is used to capture images or videos. Optionally, the camera assembly 706 may include a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the computing device, and the rear camera is disposed on the back of the computing device. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 706 may also include a flash. The flash may be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.
[0104] The audio circuit 707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 701 for processing, or input them into the radio frequency circuit 704 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there can be multiple microphones, which are respectively arranged at different parts of the computing device 700. The microphone can also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 701 or the radio frequency circuit 704 into sound waves. The speaker can be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 707 may also include a headphone jack.
[0105] Positioning component 708 is used to locate the current geographic location of computing device 700 to implement navigation or LBS (Location Based Service). Positioning component 708 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system, Russia's Grenas system or the European Union's Galileo system.
[0106] The power supply 709 is used to power various components in the computing device 700. The power supply 709 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 709 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.
[0107] In some embodiments, the computing device 700 further includes one or more sensors 710 , including but not limited to: an acceleration sensor 711 , a gyroscope sensor 712 , a pressure sensor 713 , a fingerprint sensor 714 , an optical sensor 715 , and a proximity sensor 716 .
[0108] The acceleration sensor 711 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the computing device 700. For example, the acceleration sensor 711 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 701 can control the touch display screen 705 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 711. The acceleration sensor 711 can also be used for collecting game or user motion data.
[0109] The gyroscope sensor 712 can detect the body direction and rotation angle of the computing device 700, and the gyroscope sensor 712 can cooperate with the acceleration sensor 711 to collect the user's 3D actions on the computing device 700. The processor 701 can implement the following functions based on the data collected by the gyroscope sensor 712: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.
[0110] The pressure sensor 713 can be set on the side frame of the computing device 700 and / or the lower layer of the touch display screen 705. When the pressure sensor 713 is set on the side frame of the computing device 700, it can detect the user's grip signal of the computing device 700, and the processor 701 performs left and right hand recognition or shortcut operations according to the grip signal collected by the pressure sensor 713. When the pressure sensor 713 is set on the lower layer of the touch display screen 705, the processor 701 controls the operability controls on the UI according to the user's pressure operation on the touch display screen 705. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.
[0111] The fingerprint sensor 714 is used to collect the user's fingerprint, and the processor 701 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 714, or the fingerprint sensor 714 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 701 authorizes the user to perform relevant sensitive operations, which include unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 714 can be set on the front, back, or side of the computing device 700. When a physical button or a manufacturer logo is set on the computing device 700, the fingerprint sensor 714 can be integrated with the physical button or the manufacturer logo.
[0112] The optical sensor 715 is used to collect the ambient light intensity. In one embodiment, the processor 701 can control the display brightness of the touch display screen 705 according to the ambient light intensity collected by the optical sensor 715. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 705 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 705 is reduced. In another embodiment, the processor 701 can also dynamically adjust the shooting parameters of the camera assembly 706 according to the ambient light intensity collected by the optical sensor 715.
[0113] The proximity sensor 716, also known as a distance sensor, is usually disposed on the front panel of the computing device 700. The proximity sensor 716 is used to collect the distance between the user and the front of the computing device 700. In one embodiment, when the proximity sensor 716 detects that the distance between the user and the front of the computing device 700 is gradually decreasing, the processor 701 controls the touch display screen 705 to switch from the screen-on state to the screen-off state; when the proximity sensor 716 detects that the distance between the user and the front of the computing device 700 is gradually increasing, the processor 701 controls the touch display screen 705 to switch from the screen-off state to the screen-on state.
[0114] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation on the computing device 700, and the computing device 700 may include more or less components than shown in the figure, or combine some components, or adopt a different arrangement of components.
[0115] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to execute the model evaluation method according to the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.
[0116] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, and instructions in the computer program product may be executed by a processor of a computer device to complete the model evaluation method according to the exemplary embodiment of the present disclosure.
[0117] According to the model evaluation method, model evaluation device, computing device, computer-readable storage medium and computer program product of the exemplary embodiments of the present disclosure, one of the following technical effects can be achieved: the model evaluation efficiency on the server side can be improved, which is conducive to the realization of test service and the management of model evaluation tasks, so as to efficiently deal with the frequent iteration of test requirements; model evaluation can be implemented on the server side (for example, a cloud server), and evaluation in different environments (for example, a test (STAGING) environment, a pre-release (PRT) environment, an online (PROD) environment, and a candidate (CANDIDATE) environment) can be automatically provided, saving labor costs and time costs; by implementing The current model evaluation platform can effectively accumulate the evaluation task history, facilitate the management and tracking of model evaluation tasks, make the model evaluation indicators persistent, and provide basic data for subsequent model evaluation analysis; when performing model evaluation on the server, there is no need to consider issues such as limited performance of mobile terminals. Model evaluation services can be provided to callers in the form of services, with concurrency capabilities, and evaluation services can be realized; users can customize the configuration of model evaluation, automatically trigger the execution of model evaluation tasks, and do not need to configure the local environment. Model evaluation services in different environments can be called at no cost, and evaluation analysis results and quantitative results can be automatically obtained, connecting evaluation, analysis and quantification, so that model evaluation has objective data support.
[0118] The exemplary embodiments of the present disclosure provide representative method strategies and / or logic, which can be implemented using one or more processing strategies (such as multi-tasking, multi-threading, etc.). Therefore, the various steps or functions shown can be performed in the order shown, performed in parallel, or omitted in some cases. Although not always explicitly shown, it will be recognized by those of ordinary skill in the art that one or more steps or functions shown can be repeatedly performed according to a specific strategy in the inventive concept of the present disclosure.
[0119] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0120] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A model evaluation method, characterized in that: The model evaluation method includes: In response to receiving the model evaluation request, creating a model evaluation task; Determining a service agent for performing the model evaluation task; Remotely calling the service agent to execute the model evaluation task through the service agent; The step of creating a model evaluation task in response to receiving a model evaluation request includes: In response to the model evaluation request, obtaining evaluation configuration parameters corresponding to the model evaluation request; Outputting evaluation configuration items according to the evaluation configuration parameters; In response to a setting operation on the evaluation configuration item, obtaining the set evaluation configuration item; Create a model evaluation task according to the evaluation configuration items after the settings; Wherein, in response to the setting operation for the evaluation configuration item, obtaining the set evaluation configuration item includes: In response to an input operation on the evaluation configuration item, acquiring the input evaluation configuration item, and using the input evaluation configuration item as the set evaluation configuration item; and / or In response to a selection operation on a plurality of configuration options under the evaluation configuration item, a selected configuration option is determined, and the set evaluation configuration item is obtained according to the selected configuration option.
2. The model evaluation method according to claim 1, characterized in that: The determining of a service agent for executing the model evaluation task includes: Determine a service deployment environment corresponding to the model evaluation task, and environment configuration parameters matching the service deployment environment; The service agent is determined according to the environment configuration parameters.
3. The model evaluation method according to claim 1, characterized in that: The remotely calling the service agent to execute the model evaluation task through the service agent includes: Constructing a remote service request according to the evaluation configuration parameters; The remote service request is sent to the service agent to request the service agent to perform the model evaluation task.
4. The model evaluation method according to claim 1, characterized in that: The step of obtaining, in response to the model evaluation request, evaluation configuration parameters corresponding to the model evaluation request includes: In response to the model evaluation request, determining a model evaluation service corresponding to the model evaluation request; Acquire evaluation configuration parameters corresponding to the model evaluation service from a preset evaluation configuration file, wherein the preset evaluation configuration file includes: preset evaluation configuration parameters respectively mapped to one or more preset model evaluation services.
5. The model evaluation method according to claim 1, characterized in that: The model evaluation method further includes: After the model evaluation task is completed, the execution result of the model evaluation task is obtained through the service agent; Obtaining the model analysis type of the model evaluation task; According to the model analysis type, the execution result is analyzed to generate evaluation analysis data.
6. The model evaluation method according to claim 5, characterized in that: Analyzing the execution result according to the model analysis type includes: If the model analysis type is a renderable type, rendering the execution result; If the model analysis type is a non-renderable type, the execution result is not rendered.
7. The model evaluation method according to claim 5, characterized in that: After generating the evaluation and analysis data, the method further includes: The evaluation and analysis data are quantified to generate a model evaluation report.
8. A model evaluation device, characterized in that: The model evaluation device comprises: A task creation unit, configured to create a model evaluation task in response to receiving a model evaluation request; an agent determining unit, configured to determine a service agent for performing the model evaluation task; An agent calling unit, configured to remotely call the service agent to execute the model evaluation task through the service agent; The task creation unit is further configured to: In response to the model evaluation request, obtaining evaluation configuration parameters corresponding to the model evaluation request; Outputting evaluation configuration items according to the evaluation configuration parameters; In response to a setting operation on the evaluation configuration item, obtaining the set evaluation configuration item; Create a model evaluation task according to the evaluation configuration items after the settings; The task creation unit is further configured to: In response to an input operation on the evaluation configuration item, acquiring the input evaluation configuration item, and using the input evaluation configuration item as the set evaluation configuration item; and / or In response to a selection operation on a plurality of configuration options under the evaluation configuration item, a selected configuration option is determined, and the set evaluation configuration item is obtained according to the selected configuration option.
9. The model evaluation device according to claim 8, characterized in that: The proxy determination unit is further configured to: Determine a service deployment environment corresponding to the model evaluation task, and environment configuration parameters matching the service deployment environment; The service agent is determined according to the environment configuration parameters.
10. The model evaluation device according to claim 8, characterized in that: The proxy calling unit is further configured as follows: Constructing a remote service request according to the evaluation configuration parameters; The remote service request is sent to the service agent to request the service agent to perform the model evaluation task.
11. The model evaluation device according to claim 8, characterized in that: The task creation unit is further configured to: In response to the model evaluation request, determining a model evaluation service corresponding to the model evaluation request; Acquire evaluation configuration parameters corresponding to the model evaluation service from a preset evaluation configuration file, wherein the preset evaluation configuration file includes: preset evaluation configuration parameters respectively mapped to one or more preset model evaluation services.
12. The model evaluation device according to claim 8, characterized in that: The model evaluation device further includes an analysis unit configured to: After the model evaluation task is completed, the execution result of the model evaluation task is obtained through the service agent; Obtaining the model analysis type of the model evaluation task; According to the model analysis type, the execution result is analyzed to generate evaluation analysis data.
13. The model evaluation device according to claim 12, characterized in that: The analysis unit is further configured to: If the model analysis type is a renderable type, rendering the execution result; If the model analysis type is a non-renderable type, the execution result is not rendered.
14. The model evaluation device according to claim 12, characterized in that: The model evaluation device also includes a quantification unit configured to quantify the evaluation analysis data to generate a model evaluation report.
15. A computing device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, When the computer executable instructions are executed by the at least one processor, they prompt the at least one processor to execute the model evaluation method as described in any one of claims 1 to 7.
16. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by at least one processor, the at least one processor is prompted to perform the model evaluation method according to any one of claims 1 to 7.
17. A computer program product, characterized in that The instructions in the computer program product can be executed by a processor of a computer device to perform the model evaluation method according to any one of claims 1 to 7.
