Method, system and device for evaluating artificial intelligence ai models

Abstract

The application discloses an evaluation method of an artificial intelligence (AI) model, and relates to the field of AI, and comprises the following steps: an evaluation data set and an AI model are acquired by a computing device, the evaluation data set comprises a plurality of evaluation data carrying labels used for representing real results corresponding to the evaluation data; the evaluation data in the evaluation data set are classified according to data features to obtain an evaluation data subset, and the values of the data features of all the evaluation data in the evaluation data subset satisfy a condition; the inference results of the AI model on the evaluation data in the evaluation data subset are determined, the inference result of each evaluation data in the evaluation data subset is compared with the label, and the accuracy of the inference of the AI model on the evaluation data subset is calculated according to the comparison result, so as to obtain the evaluation result of the AI model on the data whose value of the data feature satisfies the condition. The above method can obtain the evaluation result of the AI model on the data of a specific classification, and can better guide the optimization direction of the AI model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI), and more particularly to an evaluation method, system, and device for an AI model. Background Technology

[0002] With the continuous development of deep learning technology, AI models for various scenarios are constantly being trained, such as AI models for image classification and AI models for object recognition. However, these trained AI models may have some problems, such as low classification accuracy for all or part of the input images. Therefore, it is necessary to evaluate these trained AI models.

[0003] Current technologies cannot provide a guiding evaluation of AI models. Summary of the Invention

[0004] This application discloses an evaluation method, system, and device for AI models, which can be used to evaluate AI models more effectively.

[0005] The first aspect discloses an evaluation method for an AI model. A computing device acquires an AI model and an evaluation dataset including multiple labeled evaluation data. The evaluation data in the evaluation dataset are classified according to data features to obtain a subset of evaluation data. The subset of evaluation data is a subset of the evaluation dataset. The values ​​of the data features of all evaluation data in the subset of evaluation data satisfy a certain condition. The computing device further determines the inference result of the AI ​​model on the evaluation data in the subset of evaluation data. The inference result of each evaluation data in the subset of evaluation data is compared with the label of each evaluation data in the subset of evaluation data. The accuracy of the AI ​​model's inference on the subset of evaluation data is calculated based on the comparison result to obtain the evaluation result of the AI ​​model on the data whose data feature values ​​satisfy the condition.

[0006] The method described above yields evaluation results of the AI ​​model on data of a specific category. These evaluation results can be used to better guide further optimization of the AI ​​model. The label for each evaluation data point represents the true result corresponding to that data point.

[0007] As one possible implementation, the computing device can generate optimization suggestions for the AI ​​model. These suggestions may include training the AI ​​model with new data whose data feature values ​​satisfy the given condition. The more specific optimization suggestions for the AI ​​model based on the evaluation results obtained in this application can effectively optimize the AI ​​model, improve its reasoning ability, and avoid the problem of poor optimization results caused by technicians optimizing AI models solely based on experience.

[0008] As one possible implementation, the computing device can generate an evaluation report that includes evaluation results and / or optimization suggestions, and send the evaluation report to the user's device or system so that the user can understand the evaluation results of the AI ​​model on data of a specific category based on the evaluation report, and optimize the AI ​​model based on the evaluation report.

[0009] As one possible implementation, the computing device can acquire performance data, which can represent the performance of the hardware executing the inference process during the AI ​​model's inference of the evaluation data, and / or can represent the usage of operators included in the AI ​​model during the AI ​​model's inference of the evaluation data, so that users can understand the impact of the AI ​​model on the hardware and the usage of operators in the AI ​​model based on the performance data, and can optimize the AI ​​model accordingly based on the performance data.

[0010] As one possible implementation, performance data may include one or more of the following: CPU utilization, GPU utilization, memory usage, video memory usage, operator usage duration, and number of operators used.

[0011] As one possible implementation, the number of the aforementioned data features can be multiple, and the aforementioned conditions can include multiple sub-conditions, with a one-to-one correspondence between the multiple data features and the multiple sub-conditions. When the computing device classifies the evaluation data in the evaluation dataset according to the data features to obtain a subset of evaluation data, it can classify the evaluation data in the evaluation dataset according to the aforementioned multiple data features to obtain a subset of evaluation data. Each value of the multiple data features of all evaluation data in the subset of evaluation data satisfies the corresponding sub-condition in the aforementioned conditions. The above method classifies the evaluation dataset according to multiple data features, which can yield the evaluation results of the AI ​​model for data of a specific classification. These evaluation results can be better used to guide the further optimization direction of the AI ​​model.

[0012] As one possible implementation, the computing device can determine the inference result of the AI ​​model on the evaluation data in the evaluation dataset, and calculate the accuracy of the AI ​​model's inference on the evaluation dataset based on a comparison between the inference result and the labels of the evaluation data in the evaluation dataset, thereby obtaining the evaluation result of the AI ​​model on the global data. The above method can intuitively demonstrate the overall inference capability of the AI ​​model on global data.

[0013] As one possible implementation, the evaluation data in the evaluation dataset can be either images or audio.

[0014] The second aspect discloses an evaluation system for an AI model, the system comprising:

[0015] The input / output (I / O) module is used to acquire the AI ​​model and the evaluation dataset. The evaluation dataset includes multiple labeled evaluation data, and the label of each evaluation data is used to represent the true result corresponding to the evaluation data.

[0016] The data analysis module is used to classify the evaluation data in the evaluation dataset according to data characteristics to obtain a subset of evaluation data. The subset of evaluation data is a subset of the evaluation dataset, and the values ​​of the data characteristics of all evaluation data in the subset of evaluation data satisfy the following conditions.

[0017] The inference module is used to determine the inference result of the AI ​​model on the evaluation data in the evaluation data subset;

[0018] The data analysis module is further configured to compare the inference result of each evaluation data in the evaluation data subset with the label of each evaluation data in the evaluation data subset, and calculate the accuracy of the AI ​​model's inference of the evaluation data subset based on the comparison result, so as to obtain the evaluation result of the AI ​​model for data whose data feature values ​​meet the conditions.

[0019] As one possible implementation, the system further includes:

[0020] A diagnostic module is used to generate optimization suggestions for the AI ​​model, the optimization suggestions including: training the AI ​​model with new data whose values ​​of the data features satisfy the conditions.

[0021] As one possible implementation, the diagnostic module is also configured to generate an evaluation report, which includes the evaluation results and / or the optimization recommendations;

[0022] The I / O module is also used to send the evaluation report.

[0023] As one possible implementation, the system further includes:

[0024] A performance monitoring module is used to acquire performance data, which represents the performance of the hardware executing the inference process during the inference process of the AI ​​model on the evaluation data, or the usage of operators included in the AI ​​model during the inference process of the AI ​​model on the evaluation data.

[0025] As one possible implementation, the performance data includes one or more of the following: CPU utilization, GPU utilization, memory usage, video memory usage, operator usage duration, and number of operators used.

[0026] As one possible implementation, the inference module is further configured to determine the inference result of the AI ​​model on the evaluation data in the evaluation dataset;

[0027] The system also includes:

[0028] The model analysis module is used to calculate the accuracy of the AI ​​model's inference on the evaluation dataset based on the comparison results of the inference results of the evaluation data in the evaluation dataset and the labels of the evaluation data in the evaluation dataset, so as to obtain the evaluation results of the AI ​​model on the global data.

[0029] As one possible implementation, the number of data features is multiple, the condition includes multiple sub-conditions, and the relationship between the multiple data features and the multiple sub-conditions is one-to-one;

[0030] The data analysis module is specifically used to classify the evaluation data in the evaluation dataset according to the multiple data features to obtain a subset of evaluation data, wherein each value of the multiple data features of all evaluation data in the subset of evaluation data satisfies the corresponding sub-condition in the conditions.

[0031] As one possible implementation, the evaluation data in the evaluation dataset is either an image or audio.

[0032] The third aspect discloses a computing device including a memory and a processor, the memory for storing a set of computer instructions; the processor executes the set of computer instructions stored in the memory to cause the computing device to perform the method disclosed in the first aspect or any possible implementation thereof.

[0033] The fourth aspect discloses a computer-readable storage medium storing computer program code, which, when executed by a computing device, performs the method disclosed in the first aspect or any possible implementation thereof. The storage medium includes, but is not limited to, volatile memory, such as random access memory, and non-volatile memory, such as flash memory, hard disk drive (HDD), and solid-state drive (SSD).

[0034] The fifth aspect discloses a computer program product comprising computer program code, wherein when the computer program code is executed by a computing device, the computing device performs the methods disclosed in the first aspect or any possible implementation thereof. The computer program product may be a software installation package, and when it is necessary to use the methods disclosed in the first aspect or any possible implementation thereof, the computer program product may be downloaded and executed on a computing device.

[0035] The sixth aspect discloses an AI model evaluation method. A computing device can acquire an AI model and an evaluation dataset including multiple labeled evaluation data. The AI ​​model performs inference on the evaluation data in the evaluation dataset to obtain performance data. Based on the performance data, optimization suggestions for the AI ​​model are generated. This method provides more specific optimization suggestions for the AI ​​model based on the performance data obtained by the evaluation method of this application, avoiding the problem of poor optimization results caused by technicians optimizing the AI ​​model solely based on experience. The performance data represents the performance of the hardware executing the inference process during the AI ​​model's inference on the evaluation data, or the usage of operators included in the AI ​​model during the inference process. Optimization suggestions may include adjusting the structure of the AI ​​model, or optimizing the training of the AI ​​model's operators.

[0036] As one possible implementation, the computing device can generate an evaluation report that includes performance data and / or optimization suggestions, and send the evaluation report so that users can understand the AI ​​model's reasoning ability based on data features and optimize the AI ​​model based on the evaluation report.

[0037] As one possible implementation, the usage of operators included in the AI ​​model during the process of reasoning on the evaluation data includes: the usage duration of the AI ​​model's operators and the number of AI model's operators used.

[0038] As one possible implementation, during the process of the AI ​​model reasoning on the evaluation data, the performance of the hardware performing the reasoning process includes one or more of the following: CPU utilization, GPU utilization, memory usage, and video memory usage.

[0039] As one possible implementation, the computing device can determine the inference result of the AI ​​model on the evaluation data in the evaluation dataset, and calculate the accuracy of the AI ​​model's inference on the evaluation dataset based on a comparison between the inference result and the labels of the evaluation data in the evaluation dataset, thereby obtaining the evaluation result of the AI ​​model on the global data. The above method can intuitively demonstrate the overall inference capability of the AI ​​model on global data.

[0040] As one possible implementation, the evaluation data in the evaluation dataset can be either images or audio.

[0041] The seventh aspect discloses an evaluation system for an AI model, the system comprising:

[0042] The I / O module is used to acquire the AI ​​model and the evaluation dataset. The evaluation dataset includes multiple labeled evaluation data, and the label of each evaluation data is used to represent the actual result corresponding to the evaluation data.

[0043] The inference module is used to infer the evaluation data in the evaluation dataset using the AI ​​model;

[0044] A performance monitoring module is used to acquire performance data, which represents the performance of the hardware executing the inference process during the inference process of the AI ​​model on the evaluation data, or the usage of operators included in the AI ​​model during the inference process of the AI ​​model on the evaluation data.

[0045] The diagnostic module is used to generate optimization suggestions for the AI ​​model based on the performance data. The optimization suggestions include adjusting the structure of the AI ​​model or optimizing the training of the operators of the AI ​​model.

[0046] As one possible implementation, the diagnostic module is also configured to generate an evaluation report, which includes the performance data and / or the optimization recommendations;

[0047] The I / O module is also used to send the evaluation report.

[0048] As one possible implementation, the usage of operators included in the AI ​​model during the process of reasoning about the evaluation data includes: the usage duration of the AI ​​model's operators and the number of operators used in the AI ​​model.

[0049] As one possible implementation, during the process of the AI ​​model reasoning on the evaluation data, the performance of the hardware performing the reasoning process includes one or more of the following: CPU utilization, GPU utilization, memory usage, and video memory usage.

[0050] As one possible implementation, the inference module is also used to determine the inference result of the AI ​​model on the evaluation data in the evaluation dataset;

[0051] The system also includes:

[0052] The model analysis module is used to calculate the accuracy of the AI ​​model's inference on the evaluation dataset based on the comparison results of the inference results of the evaluation data in the evaluation dataset and the labels of the evaluation data in the evaluation dataset, so as to obtain the evaluation results of the AI ​​model on the global data.

[0053] As one possible implementation, the evaluation data in the evaluation dataset is an image or audio.

[0054] The eighth aspect discloses a computing device including a memory and a processor, the memory being used to store a set of computer instructions; the processor executing the set of computer instructions stored in the memory to cause the computing device to perform the methods disclosed in the sixth aspect or any possible implementation thereof.

[0055] A ninth aspect discloses a computer-readable storage medium storing computer program code, which, when executed by a computing device, performs the methods disclosed in the sixth aspect or any possible implementation thereof. The storage medium includes, but is not limited to, volatile memory, such as random access memory, and non-volatile memory, such as flash memory, hard disk drive (HDD), and solid-state drive (SSD).

[0056] The tenth aspect discloses a computer program product comprising computer program code, wherein when the computer program code is executed by a computing device, the computing device performs the methods disclosed in the sixth aspect or any possible implementation thereof. The computer program product may be a software installation package, and when it is necessary to use the methods disclosed in the sixth aspect or any possible implementation thereof, the computer program product may be downloaded and executed on a computing device. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of a system architecture 100 disclosed in an embodiment of this application;

[0058] Figure 2 This is a schematic diagram of another system architecture 200 disclosed in an embodiment of this application;

[0059] Figure 3 This is a schematic diagram of the deployment of an evaluation system disclosed in an embodiment of this application;

[0060] Figure 4 This is a deployment diagram of another evaluation system disclosed in an embodiment of this application;

[0061] Figure 5 This is a schematic diagram of the structure of an evaluation system disclosed in an embodiment of this application;

[0062] Figure 6 This is a flowchart illustrating an AI model evaluation method disclosed in an embodiment of this application;

[0063] Figure 7 This is a schematic diagram of a task creation interface disclosed in an embodiment of this application;

[0064] Figure 8 This is a flowchart illustrating another AI model evaluation method disclosed in an embodiment of this application;

[0065] Figure 9 This is a distribution diagram of the brightness of the annotation box for microbial detection disclosed in the embodiments of this application;

[0066] Figure 10 This is a distribution diagram showing the proportion of the area of ​​the annotation box for microbial detection disclosed in the embodiments of this application to the image.

[0067] Figure 11 This is a schematic diagram of the mAP before and after retraining of a microbial cell model disclosed in an embodiment of this application;

[0068] Figure 12 This is a curve showing the FI value versus the confidence threshold of an AI model for helmet detection disclosed in an embodiment of this application;

[0069] Figure 13 This is a PR curve of an AI model for safety helmet detection disclosed in an embodiment of this application;

[0070] Figure 14 This is a schematic diagram of the structure of another evaluation system 1500 disclosed in an embodiment of this application;

[0071] Figure 15 This is a schematic diagram of the structure of another evaluation system 1600 disclosed in the embodiments of this application;

[0072] Figure 16 This is a schematic diagram of the structure of a computing device disclosed in an embodiment of this application;

[0073] Figure 17 This is a schematic diagram of the structure of another computing device disclosed in an embodiment of this application. Detailed Implementation

[0074] This application discloses a method, system, and device for evaluating artificial intelligence (AI) models, which are used to effectively evaluate AI models. These will be described in detail below.

[0075] Currently, AI has received widespread attention from academia and industry, and it has demonstrated performance exceeding that of ordinary humans in many application areas. For example, the application of AI technology in machine vision (such as facial recognition, image classification, and object detection) has resulted in machine vision accuracy exceeding that of humans. AI technology also has good applications in natural language processing and recommendation systems.

[0076] Machine learning is a core method for achieving AI. Computers build an AI model based on existing data to address a technical problem, and then use this model to reason about unknown data to obtain a result. This method is analogous to a computer learning a certain ability (such as cognitive, discriminative, or classification abilities) like a human; therefore, it is called machine learning.

[0077] Various applications of AI, utilizing machine learning, employ various AI models (such as neural network models). An AI model is a type of mathematical algorithm model that uses machine learning concepts to solve practical problems. An AI model includes numerous parameters and computational formulas (or rules). The parameters in an AI model are numerical values ​​obtained by training the model on a dataset; for example, the parameters in an AI model are the weights of computational formulas or factors. An AI model can be divided into multiple layers or nodes. Each layer or node includes a type of computational rule and one or more parameters (used to represent a mapping, relationship, or transformation). The computational rule and one or more parameters used in each layer or node of an AI model are called an operator. An AI model can include a large number of operators. For example, in a neural network, an operator can be a single layer, such as a convolutional layer, pooling layer, or fully connected layer. Convolutional layers are used for feature extraction. Pooling layers are used for downsampling. Fully connected layers are used for feature extraction or classification. AI models include deep convolutional neural networks, residual networks (ResNet), visual geometry group (VGG) networks, Inception networks, Faster region-based convolutional neural networks (R-CNN), single-shot multibox detector (SSD) networks, and You Only Look Once (YOLO) networks, among others.

[0078] Before using an AI model to solve a specific technical problem, it needs to be trained first, then evaluated, and the evaluation results determine whether further optimization is needed. The AI ​​model can only be used if its evaluation results are good. With the continuous development of deep learning, AI platforms have gradually emerged. An AI platform is a system that provides services such as training, evaluating, and optimizing AI models to individuals or enterprises. It can receive user requirements and data through interfaces, train and optimize various AI models that meet user needs, evaluate the performance of AI models, and further optimize the models based on the evaluation results.

[0079] Currently, AI platforms train an initial AI model. After obtaining the model, the platform uses it to infer on an evaluation dataset to obtain inference results. Then, based on the inference results and the labels of the evaluation data in the dataset, the accuracy of the AI ​​model's inference is determined. Accuracy represents how close the AI ​​model's inference results to the actual results of the evaluation data in the dataset. Accuracy can be measured by many metrics, such as precision and recall. However, this method only provides researchers with the accuracy value of the AI ​​model on the entire evaluation dataset, without obtaining more specific information, such as the impact of data features on the AI ​​model's inference results. This results in relatively general evaluation results, failing to provide more information for further optimization of the AI ​​model.

[0080] Inference is the process of using an AI model to predict outcomes from evaluation data in an evaluation dataset. For example, in a face recognition task, inference might involve using an AI model to identify the names corresponding to faces in images within the evaluation dataset. Specifically, inference code can be used to call the AI ​​model to perform inference on the evaluation data in the evaluation dataset. Inference code can include calling code to invoke the AI ​​model to perform inference on the evaluation data in the evaluation dataset. Inference code can also include preprocessing code to preprocess the evaluation data in the evaluation dataset before using the calling code to invoke the AI ​​model to perform inference on the preprocessed evaluation data. Inference code can also include post-processing code for further statistical analysis and other processing of the inference results.

[0081] Data features are abstractions of the inherent characteristics or features of data, used to represent those characteristics. For example, when evaluating images, data features could include aspect ratio, color saturation, resolution, blur, brightness, and saturation. Different data points correspond to different feature values ​​under the same data feature. Based on these features, multiple data points can be classified, with each category containing data points exhibiting similar characteristics. For example, since images of different sizes have different aspect ratios, we can calculate the aspect ratio of 10 images separately to obtain a set of images with aspect ratios of [0.4, 0.3, 0.35, 0.9, 0.1, 1.2, 1.4, 0.3, 0.89, 0.7]. We can then divide these images into three categories according to their aspect ratios: one category consists of images with aspect ratios of [0-0.5], totaling 5 images; another category consists of images with aspect ratios of (0.5-1], totaling 3 images; and the third category consists of images with aspect ratios of (1-1.5], totaling 2 images.

[0082] This application discloses an AI model evaluation method, system, and device. The method can obtain the evaluation results of the AI ​​model on data of a specific category, so that the evaluation results can be used more effectively to guide the further optimization of the AI ​​model.

[0083] To better understand the AI ​​model evaluation method, system, and device disclosed in this application, the system architecture used in this application embodiment is described below. Please refer to... Figure 1 , Figure 1 This is a schematic diagram of a system architecture 100 disclosed in an embodiment of this application. For example... Figure 1 As shown, the system architecture 100 may include a training system 11, an evaluation system 12, and a terminal device 13. The training system 11 and the evaluation system 12 can provide users with training and evaluation services for AI models through the AI ​​platform.

[0084] The training system 11 is used to receive the training dataset sent by the user through the terminal device 13, train the initial AI model based on the training dataset, and send the trained AI model to the evaluation system 12.

[0085] Optionally, the training system 11 is also used to receive the task type input or selected by the user on the AI ​​platform through the terminal device 13, and determine the initial AI model according to the task type.

[0086] Optionally, the training system 11 is also used to send the received task type to the evaluation system 12.

[0087] Optionally, the training system 11 is also used to receive an initial AI model uploaded by the user through the terminal device 13.

[0088] The evaluation system 12 is used to receive the AI ​​model from the training system 11, receive the evaluation dataset uploaded by the user through the terminal device 13, use the AI ​​model to perform inference on the evaluation dataset to obtain inference results, generate an evaluation report including the evaluation results and / or optimization suggestions for the AI ​​model based on the evaluation dataset and inference results, and send the evaluation report to the terminal device 13.

[0089] Optionally, the evaluation system 12 is also used to receive task types from the training system 11.

[0090] Optionally, the evaluation system 12 is also used to receive the task type input or selected by the user on the AI ​​platform via the terminal device 13.

[0091] Terminal device 13 is used to send data and information to training system 11 and evaluation system 12 according to user operations, or to receive information sent by training system 11 or evaluation system 12.

[0092] Please see Figure 2 , Figure 2 This is a schematic diagram of another system architecture 200 disclosed in an embodiment of this application. For example... Figure 2 As shown, the system architecture 200 may include a terminal device 21 and an evaluation system 22.

[0093] Terminal device 21 is used to send the trained AI model, evaluation dataset and inference code to evaluation system 22 according to the user's operation.

[0094] The evaluation system 22 is used to receive a trained AI model, evaluation dataset and inference code from the terminal device 21, call the AI ​​model through the inference code to infer the evaluation data in the evaluation dataset to obtain the inference result, generate an evaluation report including the evaluation result and optimization suggestions for the AI ​​model based on the evaluation dataset and the inference result, and send the evaluation report to the terminal device 21.

[0095] Optionally, the evaluation system 22 is also used to receive the task type sent by the user through the terminal device 21.

[0096] It should be understood that in some embodiments, the evaluation method of the AI ​​model provided in this application is executed by an evaluation system, for example, the evaluation system may be the evaluation system 12 or the evaluation system 22 described above.

[0097] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating the deployment of an evaluation system disclosed in an embodiment of this application. For example... Figure 3 As shown, the evaluation system can be deployed in a cloud environment. A cloud environment is an entity that provides cloud services to users using basic resources under the cloud computing model. A cloud environment includes cloud data centers and cloud service platforms. Cloud data centers include a large amount of basic resources (including computing resources, storage resources, and network resources) owned by cloud service providers. The computing resources included in a cloud data center can be a large number of computing devices (e.g., servers). The evaluation system can be deployed independently on servers or virtual machines in a cloud data center, or it can be distributed across multiple servers in a cloud data center, or distributed across multiple virtual machines in a cloud data center, or distributed across servers and virtual machines in a cloud data center. For example... Figure 3As shown, the evaluation system is abstracted into an evaluation cloud service by the cloud service provider on the cloud service platform and provided to users. After users purchase this cloud service on the cloud service platform (pre-payment is possible, with settlement based on the final resource usage), the cloud environment utilizes the evaluation system deployed in the cloud data center to provide the evaluation cloud service to users. It should be understood that the functions provided by the evaluation system can also be abstracted together with the functions provided by other systems into a single cloud service. For example, the cloud service provider may abstract the AI ​​model evaluation function provided by the evaluation system, along with the initial AI model training function provided by the training system, into a single AI platform cloud service.

[0098] The evaluation system can also be deployed in edge environments, which refer to a collection of data centers or edge computing devices located close to the user. An edge environment includes one or more edge computing devices. The evaluation system can be deployed independently on edge computing devices, or distributed across multiple edge servers, multiple edge stations with computing power, or a combination of edge servers and edge stations with computing power. Furthermore, the evaluation system can be deployed in other environments, such as clusters of terminal computing devices. The evaluation system can be a software system running on computing devices such as servers. It can also be a backend system of an AI platform, serving as an AI model evaluation service provided by the evaluation system's backend.

[0099] Please see Figure 4 , Figure 4 This is a deployment diagram of another evaluation system disclosed in an embodiment of this application. Figure 4 As shown, the evaluation system provided in this application can also be deployed in a distributed manner in different environments. The evaluation system provided in this application can be logically divided into multiple parts, each with different functions. Each part of the evaluation system can be deployed in any two or three environments: terminal computing devices, edge environments, and cloud environments. Terminal computing devices include: terminal servers, smartphones, laptops, tablets, personal desktop computers, smart cameras, etc. The edge environment is an environment that includes a set of edge computing devices located close to the terminal computing devices, including: edge servers, edge stations with computing power, etc. The various parts of the evaluation system deployed in different environments or devices work together to achieve the AI ​​model evaluation function. It should be understood that this application does not restrict the specific environments in which the evaluation system is deployed; in practical applications, it can be deployed adaptively according to the computing power of the terminal computing devices, the resource availability of the edge and cloud environments, or specific application requirements.

[0100] In some embodiments, the AI ​​platform includes a training system and an evaluation system, which can be deployed in the same environment, such as a cloud environment or an edge environment. Alternatively, the training system and evaluation system can be deployed in different environments; for example, the training system can be deployed in a cloud environment, and the evaluation system in an edge environment. The training system and evaluation system can be deployed independently or in a distributed manner.

[0101] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an evaluation system 500 disclosed in an embodiment of this application. Figure 5 As shown, the evaluation system 500 may include an input / output (I / O) module 501, a dataset storage module 502, an inference module 503, a performance monitoring module 504, a model analysis module 505, a data analysis module 506, a diagnostic module 507, and a result storage module 508. The evaluation system 500 may include all or some of the modules mentioned above. The functions of each module in the evaluation system 500 will be described below.

[0102] I / O module 501 is used to receive AI models sent from the training system or terminal device, receive evaluation datasets and inference code uploaded by users through the terminal device, and send evaluation reports to the terminal device.

[0103] Optionally, the I / O module 501 is also used to receive task types uploaded by the user through the terminal device.

[0104] Dataset storage module 502 is used to store the received evaluation dataset.

[0105] The inference module 503 is used to perform inference on the evaluation dataset stored in the dataset storage module 502 or the received evaluation dataset using an AI model.

[0106] The performance monitoring module 504 is used to monitor the hardware resource usage information of the AI ​​model during inference in the inference module 503, as well as the usage time and number of operators included in the AI ​​model. The number of operators used is the number of times the operator is used during the inference process in the inference module 503. The usage time of the operator is the total time and / or average time used by each operator during the inference process in the inference module 503.

[0107] The model analysis module 505 is used to calculate the accuracy of the AI ​​model's inference results on the evaluation data in the evaluation dataset based on the inference results of the inference module 503 and the labels of the evaluation data in the evaluation dataset.

[0108] The data analysis module 506 is used to calculate the values ​​of data features of the evaluation data in the evaluation dataset under one or more data features, classify the evaluation data in the evaluation dataset according to the values ​​of the data features, obtain at least one subset of evaluation data, and calculate the accuracy of the AI ​​model's inference results for the evaluation data in each subset of evaluation data based on the inference results of the inference module 503 and the labels of the evaluation data in each subset of evaluation data.

[0109] The diagnostic module 507 is used to generate an evaluation report based on any one or more of the monitoring results from the performance monitoring module 504, the analysis results from the model analysis module 505, and the analysis results from the data analysis module 506.

[0110] The result storage module 508 is used to store the monitoring results of the performance monitoring module 504, the analysis results of the model analysis module 505, the analysis results of the data analysis module 506, and the diagnostic results of the diagnostic module 507.

[0111] Due to the functions of the above modules, the evaluation system provided in this application embodiment can provide users with services for evaluating AI models, and the evaluation system can deeply analyze the impact of different data features on AI models and other analysis results, and further provide users with AI model optimization suggestions.

[0112] based on Figure 1 or Figure 2 Please refer to the system architecture shown. Figure 6 , Figure 6 This is a flowchart illustrating an AI model evaluation method disclosed in an embodiment of this application. The AI ​​model evaluation method is applied to an evaluation system. Since the evaluation system is deployed independently or in a distributed manner on a computing device, the AI ​​model evaluation method is applied to the computing device; that is, the AI ​​model evaluation method of this application can be executed by the processor in the computing device by executing computer instructions stored in memory. Figure 6 As shown, the evaluation method for this AI model may include the following steps.

[0113] 601. Receive AI models and evaluation datasets.

[0114] An AI model is a trained model. It can be sent by the training system or uploaded by the user through a terminal device.

[0115] An evaluation dataset can include multiple evaluation data points and labels for these data points. Each evaluation data point corresponds to one or more labels, which represent the true results of the evaluation data. These evaluation data points are of the same type, such as images, videos, audio, and text. The evaluation data in the dataset may differ or be the same depending on the task type. For example, in the case of image classification or object detection, all evaluation data in the dataset are images; in the case of speech recognition, the evaluation data are audio. The labels represent the true results corresponding to the evaluation data, and their form varies depending on the task type and the evaluation data. For example, if the evaluation data is an image and the task is to identify the type of object in the image, the label of the evaluation data is the true type of the object. Another example: if the evaluation data is an image and the task is to detect objects in the image, the label can be the detection box corresponding to the object in the evaluation image. The shape of the detection box can be a rectangle, a circle, a line, or other shapes, without limitation. In essence, a label is a value with specific meaning, associated with the labeled evaluation data. This value can represent the type, location, or other aspects of the labeled evaluation data. For example, when the evaluation data is audio, the label can indicate the type of audio, such as pop music or classical music. Each evaluation data point can correspond to one label, or multiple labels.

[0116] Different AI models can be applied to different application scenarios, and the same AI model can also be applied to different application scenarios. The application scenario of an AI model may differ, and the task type of the AI ​​model may also differ. Because the task type of an AI model differs, its evaluation metrics and data features also differ. Therefore, after obtaining an AI model, we can obtain the evaluation metrics and data features corresponding to the task type of the AI ​​model, that is, obtain the evaluation metrics and data features corresponding to the task type of the AI ​​model. If the evaluation system includes multiple task types, and each task type has corresponding evaluation metrics and data features, we can obtain the evaluation metrics and data features of the AI ​​model for that task type. If the evaluation system includes only one task type, we can obtain the evaluation metrics and data features for that task type. An evaluation metric for a task type can include at least one evaluation metric, and a data feature for a task type can include at least one data feature. A data feature is an abstraction of the characteristics of the data itself. There can be one or more data features, each data feature representing one aspect of the characteristics of the evaluation data in the evaluation dataset.

[0117] In evaluation systems that include multiple task types, the task types can be pre-input or selected by the user through the I / O modules of the evaluation system. See also... Figure 7 , Figure 7 This is a schematic diagram of a task creation interface disclosed in an embodiment of this application. Figure 7 As shown, the task creation interface can include dataset, model type, model source, and inference code. Additionally, the task creation interface may include other content, which is not limited here. The box next to "Dataset" can be used by the user to upload an evaluation dataset or by the user to input the storage path of the evaluation dataset. The box next to "Model Type" can be used by the user to select the task type of the AI ​​model from the stored task types or by the user to input the task type of the AI ​​model. The box next to "Model Source" can be used by the user to upload an AI model or by the user to input the storage path of the AI ​​model. The box next to "Inference Code" can be used by the user to upload inference code or by the user to input the storage path of the inference code. It is evident that the task type of the AI ​​model is determined after task creation. The inference code is used to call the AI ​​model to perform inference on the evaluation dataset. The inference code can include calling code, which calls the AI ​​model to perform inference on the evaluation dataset. The inference code can also include preprocessing code, which preprocesses the evaluation data in the evaluation dataset, after which the calling code calls the AI ​​model to perform inference on the preprocessed evaluation dataset. The inference code can also include post-processing code, which processes the inference results to obtain the inference result.

[0118] 602. Calculate the value of the data feature for each evaluation data in the evaluation dataset.

[0119] After receiving the AI ​​model and the evaluation dataset, the value of the data feature for each evaluation data point in the evaluation dataset can be calculated. This involves calculating the value of the data feature for each evaluation data point based on the multiple evaluation data points included in the dataset and their labels. The value of a data feature is used to measure the characteristics of the data. There can be one or more data features. When there are multiple data features, the value of each of the multiple data features for each evaluation data point in the evaluation dataset can be calculated.

[0120] In the case of image classification, each evaluation data point in the evaluation dataset is an image. Data features can include common image features such as aspect ratio, the mean and standard deviation of the RGB values ​​of all images, color saturation, resolution, blur, brightness, and saturation. The aspect ratio is the ratio of the image's width to its height, and can be represented as follows:

[0121]

[0122] ImageH represents the height of the image, and ImageW represents the width of the image. The RGB mean of all images is the average of the R channel values, the average of the G channel values, and the average of the B channel values ​​in all images included in the evaluation dataset. The RGB mean T of all images is... mean It can be represented as follows:

[0123]

[0124] n represents the number of images included in the evaluation dataset. (R, G, B) i In this context, R represents the sum of the R-channel values ​​of all pixels in the i-th image included in the evaluation dataset, where (R, G, B) i In this context, G represents the sum of the G channel values ​​of all pixels in the i-th image included in the evaluation dataset, where (R, G, B) i In this context, B represents the sum of the B-channel values ​​of all pixels in the i-th image included in the evaluation dataset. The RGB mean of all images can be decomposed into the following three formulas:

[0125]

[0126]

[0127]

[0128] T mean,R T is the average value of the R channels of n images. mean,G T is the average value of the G channel of n images. mean,B R is the average value of the B channel values ​​of n images. i To evaluate the sum of the R channel values ​​of all pixels in the i-th image included in the dataset, G i To evaluate the sum of the G channel values ​​of all pixels in the i-th image included in the dataset, B i To evaluate the sum of the B-channel values ​​of all pixels in the i-th image included in the dataset. The standard deviation T of the RGB values ​​for all images. STD It can be represented as follows:

[0129]

[0130] The chromaticity of an image refers to the richness of its colors. The chromaticity (CO) of an image can be represented as follows:

[0131]

[0132] STD() calculates the standard deviation of the content within the parentheses. Image resolution is the number of pixels per inch. Image blur is the degree of blurring in the image. Image brightness is the brightness of the image; image brightness BR can be expressed as follows:

[0133]

[0134] Image saturation refers to the purity of colors in an image. Image saturation (SA) can be represented as follows:

[0135]

[0136] m is the number of pixels in an image, max(R,G,B) j Let min(R,G,B) be the maximum value among the R, G, and B channels of the j-th pixel in an image. j It is the minimum value among the R channel, G channel, and B channel values ​​of the j-th pixel in an image.

[0137] In the case of object detection, each evaluation data point in the evaluation dataset is an image. Data features can include the number of bounding boxes, the proportion of the bounding box area to the image area, the variance of the bounding box area, the distance of the bounding box from the image edge, the overlap of the bounding boxes, the aspect ratio of the image, and other bounding box-based features, as well as image resolution, image blur, image brightness, and image saturation. Bounding boxes are the labels on the training images in the training dataset. In the training images, one or more classes of objects to be identified are labeled with bounding boxes. During the training process, the AI ​​model learns the features of the objects within the bounding boxes in the training images, thereby enabling the AI ​​model to detect one or more classes of objects in the images. The proportion of the bounding box area to the image area is the percentage of the total image area. The proportion of the bounding box area to the image area can be represented as follows:

[0138]

[0139] BboxW represents the width of the label box, i.e., the width of the label box corresponding to the labels included in the evaluation data. BboxH represents the height of the label box, i.e., the height of the label box corresponding to the labels included in the evaluation data. Label box overlap is the proportion of a label box covered by other label boxes. Label box overlap (OV) can be represented as follows:

[0140]

[0141] M is the difference between the number of bounding boxes in an image and 1, C is the region of the target box within the bounding boxes in the image, area(C) is the area of ​​the target box, and G... k Let C∩G be the region of the k-th bounding box excluding the target box within the bounding boxes of this image. k The region is the overlapping area between the target bounding box and the region of the k-th bounding box, area(C∩G) k Let be the area of ​​the overlapping region between the target bounding box and the region of the k-th bounding box. The distance MA of the bounding box from the image edge can be represented as follows:

[0142]

[0143] imgx is the x-coordinate of the center point of an image, imgy is the y-coordinate of the center point of the image, x is the x-coordinate of the center point of the bounding box in the image, and y is the y-coordinate of the center point of the bounding box in the image.

[0144] In the case of text classification in natural language processing, data features can include word count, number of unique words, length, number of stop words, number of punctuation marks, number of heading words, average word length, term frequency (TF), and inverse document frequency (UDF). Word count is used to count the number of words in each line of text. Number of unique words is used to count the number of words that appear only once in each line of text. Length is used to count the storage space occupied by each line of text (including spaces, symbols, letters, etc.). Number of stop words is used to count the number of words such as "between," "but," "about," and "very." Number of punctuation marks is used to count the number of punctuation marks in each line of text. Number of uppercase words is used to count the number of uppercase words. Number of heading words is used to count the number of words whose first letter is uppercase and the rest are lowercase. Average word length is used to count the average length of each word in each line of text.

[0145] In the case of audio sound classification, data features can include short-time average zero-crossing rate, short-time energy, energy entropy, spectral centroid, spectral spread, spectral entropy, and spectral flux. The short-time average zero-crossing rate is the number of zero-crossing points of the signal within each frame, used to reflect frequency characteristics. Short-time energy is the sum of squares of the signal in each frame, used to reflect the strength of the signal energy. Energy entropy, similar to spectral entropy, describes the time-domain distribution of the signal, used to reflect continuity. The spectral centroid, also known as the first-order spectral moment, indicates that more spectral energy is concentrated in the low-frequency range; for example, speech typically has a lower spectral centroid compared to music. Spectral spread, also known as the second-order spectral central moment, describes the distribution of the signal around the spectral centroid. Spectral entropy, based on the characteristics of entropy, shows that the more uniform the distribution, the greater the entropy. Spectral entropy reflects the uniformity of each frame of signal. For example, the spectrum of a speaker is non-uniform due to the presence of formants, while the spectrum of white noise is more uniform. One application of this is voice activity detection (VAD). Spectral flux describes the changes in the spectrum between adjacent frames.

[0146] The values ​​of data features for each evaluation data point in the evaluation dataset can be calculated using methods or formulas similar to those given above.

[0147] 603. Based on the value of the data feature of each evaluation data in the evaluation dataset, divide the evaluation data in the evaluation dataset into at least one subset of evaluation data.

[0148] After calculating the value of the data feature for each evaluation data point in the evaluation dataset, the evaluation data can be divided into at least one subset based on the distribution of the data feature values ​​or according to a pre-set partitioning threshold. That is, the evaluation data in the evaluation dataset is classified according to the values ​​of the data features to obtain evaluation data subsets. There can be multiple data features for the evaluation data, and the evaluation dataset can be partitioned based on each data feature. For example, in a task type of image classification, where the data features include image brightness and image saturation, after calculating the brightness and saturation values ​​for each image in the evaluation dataset, the evaluation data can be divided into at least one subset based on the distribution of brightness values, and the evaluation data can also be divided into at least one subset based on the distribution of saturation values. When partitioning the evaluation data in the evaluation dataset according to the distribution of data feature values, it can be done by using a threshold, by using a percentage, or by other methods; no limitation is placed here.

[0149] For example, let's illustrate this by dividing the data into percentages. Data features include image brightness, and the evaluation dataset consists of 100 images. These 100 images can first be sorted in descending or ascending order of brightness values. Then, the sorted 100 images can be divided into four evaluation data subsets based on percentages. Each of these four subsets can contain 25 images. The percentage division can be either equal or uneven.

[0150] For example, let's take threshold-based segmentation as an illustration. Data features include image brightness, and the evaluation dataset consists of 100 images. These 100 images can first be sorted in descending or ascending order of brightness value. Then, images with brightness values ​​greater than or equal to a first threshold can be grouped into the first evaluation data subset; images with brightness values ​​less than the first threshold but greater than or equal to a second threshold can be grouped into the second evaluation data subset; images with brightness values ​​less than the second threshold but greater than or equal to a third threshold can be grouped into the third evaluation data subset; and images with brightness values ​​less than the third threshold can be grouped into the fourth evaluation data subset. The first, second, and third thresholds decrease sequentially. The number of images included in the first, second, third, and fourth data subsets can be the same or different.

[0151] The values ​​of the data features of all evaluation data in each subset of the evaluation data obtained after partitioning satisfy the same set of conditions. The conditions may be: the values ​​of the data features of all evaluation data in the subset are within a specific numerical range (e.g., the brightness values ​​of all evaluation data images are within the range of 0-20%), or the values ​​of the data features of all evaluation data in the subset conform to a specific characteristic (e.g., the aspect ratio of all evaluation data images is an even number).

[0152] In another embodiment, the evaluation dataset can be divided according to multiple data features to obtain at least one evaluation data subset. The values ​​of multiple data features in the evaluation data subset satisfy multiple sub-conditions within the same set of conditions; that is, the value of each data feature in the evaluation data subset satisfies one sub-condition corresponding to that data feature. For example, if the evaluation data is an image, its data features include two aspects: image brightness and image aspect ratio. Images in the evaluation dataset whose brightness is within a first threshold range and whose aspect ratio is within a second threshold range can be divided into an evaluation data subset. That is, the values ​​of the two data features corresponding to all evaluation data in this subset each satisfy one corresponding sub-condition. The evaluation data subset is a subset of the evaluation dataset; that is, the evaluation data included in the evaluation data subset is a portion of the evaluation data included in the evaluation dataset.

[0153] 604. Use an AI model to infer the evaluation data in at least one subset of the evaluation data to obtain the inference result.

[0154] After obtaining the AI ​​model and evaluation dataset, or after dividing the evaluation data in the evaluation dataset into at least one subset according to the distribution of data feature values ​​under data features for each evaluation data point, the AI ​​model can be used to infer the evaluation data of each subset to obtain inference results. Evaluation data from each subset can be input into the AI ​​model for inference. Inference code can be used to call the AI ​​model to infer the evaluation data in the subset. The inference code may include calling code to invoke the AI ​​model to infer the evaluation data in the subset. Before using the AI ​​model to infer the evaluation data in the subset, to ensure consistency in certain aspects of the evaluation data (e.g., to ensure consistent image sizes when the evaluation data is images), the evaluation data in the subset can be preprocessed. The inference code may also include preprocessing code for preprocessing the evaluation data in the subset. After using the AI ​​model to infer the evaluation data in the subset, the inference results may need to be processed. Optionally, the inference code may also include post-processing code for post-processing the inference results. The pre-processing code, calling code, and post-processing code are executed sequentially. Figure 1 In the corresponding system architecture, the inference code is developed based on the AI ​​model. Figure 2 In the corresponding system architecture, the inference code is provided by the client.

[0155] It is worth noting that in some other embodiments, when performing the method of evaluating the AI ​​model, the order of steps 603 and 604 above may not be followed. Instead, the AI ​​model may be used to infer all the evaluation data in the evaluation dataset to obtain the inference results of all the evaluation data in the evaluation dataset. Then, the evaluation dataset may be divided into at least one subset of evaluation data according to the distribution of the data feature values ​​of each evaluation data in the evaluation dataset under the data features, and the inference results corresponding to the evaluation data in each subset of evaluation data may be obtained.

[0156] 605. Compare the inference result of each evaluation data with the label of each evaluation data, calculate the accuracy of the AI ​​model's inference for each subset of evaluation data based on the comparison result, and obtain the evaluation result.

[0157] After using an AI model to infer the evaluation data in at least one subset of evaluation data, the inference result for each evaluation data point can be compared with its label. If the inference result and label match, the AI ​​model's inference is considered accurate, and the comparison is correct. If they do not match, the inference is considered inaccurate, and the comparison is incorrect. Based on the comparison results, the accuracy of the AI ​​model's inference for each subset of evaluation data can be calculated, yielding the evaluation result. When calculating the accuracy of the AI ​​model's inference for each subset of evaluation data, the evaluation metric value for the inference result of the AI ​​model for each subset of evaluation data within the at least one evaluation data set can be calculated based on the comparison results, under the given evaluation metric. Accuracy can be measured using one or more evaluation metrics for the AI ​​model.

[0158] When the task type is image classification, evaluation metrics can include confusion matrix, accuracy, precision, recall, receiver operating characteristic (ROC) curve, F1 score, etc. When image classification is binary, the categories can include positive and negative classes. Samples can be classified as true positive (TP), true negative (TN), false positive (FP), and false negative (FN) based on their true and predicted categories. TP represents the number of true positive samples predicted as positive by the AI ​​model, i.e., the number of samples whose first label is positive and whose inference result is positive. TN represents the number of true negative samples predicted as negative by the AI ​​model, i.e., the number of samples whose first label is negative and whose inference result is negative. FP represents the number of true samples predicted as positive by the AI ​​model but actually negative; that is, the number of samples whose first label is negative and whose inference result is positive. FN represents the number of true samples predicted as negative by the AI ​​model but actually positive; that is, the number of samples whose first label is positive and whose inference result is negative. The confusion matrix includes TP, TN, FP, and FN, as shown in Table 1.

[0159]

[0160] Table 1 Confusion Matrix

[0161] Accuracy is the proportion of correctly predicted samples out of the total number of samples. In the case of binary image classification, accuracy AC can be expressed as follows:

[0162]

[0163] Precision is the proportion of correctly predicted positive samples out of all samples predicted positive. In the case of binary image classification, precision (PR) can be expressed as follows:

[0164]

[0165] Recall is the proportion of correctly predicted positive samples out of all positive samples. In the case of binary image classification, recall (RE) can be expressed as follows:

[0166]

[0167] The F1 value is the ratio of the arithmetic mean to the geometric mean. The F1 value can be expressed as follows:

[0168]

[0169] The ROC curve is a curve with the true positive ratio (TPR) on the vertical axis and the false positive ratio (FPR) on the horizontal axis. TPR is the proportion of true positive samples predicted as positive out of all true positive samples. FPR is the proportion of true negative samples predicted as positive out of all true negative samples. In the case of binary image classification, FPR and TPR can be represented as follows:

[0170]

[0171]

[0172] When the task type is object detection, evaluation metrics can include mean average precision (mAP) and the precision-recall (PR) curve. The PR curve is a curve with recall on the x-axis and precision on the y-axis. mAP is the mean average precision (AP), and AP is the area enclosed by the PR curve. mAP and AP can be represented as follows:

[0173]

[0174]

[0175] Q is the number of labels, AP(q) is the average precision of the q-th label, N is the number of predicted bounding boxes, and RE idx For the predicted idx-th bounding box, RE idx-1 For the predicted recall rate of the (idx-1)th bounding box, PR idx Let be the accuracy of the predicted idx-th bounding box.

[0176] For text classification in natural language processing, evaluation metrics can include accuracy, precision, recall, and F1 score. For sound classification in audio processing, evaluation metrics can include accuracy, precision, recall, and F1 score.

[0177] The evaluation index value under the evaluation metric can be calculated according to the above formula, or it can be calculated according to other methods, which are not limited here. The evaluation result can include the evaluation index value of the AI ​​model's inference result on the evaluation data in the evaluation data subset corresponding to each data feature under the evaluation metric. For an evaluation metric and a data feature, multiple data feature values ​​under this data feature can correspond to one evaluation index value under this evaluation metric. The evaluation result can also include the phenomena obtained from the evaluation index value of the AI ​​model's inference result on the evaluation data in the evaluation data subset corresponding to each data feature under the evaluation metric, such as the fact that image brightness has a significant impact on accuracy. For example, if the task type is face detection, the data features include the proportion of the area of ​​the bounding box to the image, and the evaluation metric includes recall, the evaluation results can be shown in Table 2:

[0178]

[0179] Table 2 Evaluation Results

[0180] Optionally, after performing steps 601-605 above, the method may further include: generating optimization suggestions for the AI ​​model based on the evaluation results. These suggestions may be based on the current evaluation results of the AI ​​model on each subset of evaluation data, recommending the addition of new data that meets the same set of conditions as the evaluation data in one or more subsets of evaluation data to continue training the AI ​​model. Typically, the accuracy of the current AI model's inference on this subset of evaluation data does not yet meet the model's requirements, or the accuracy of the current AI model's inference on this subset of evaluation data is lower than that of other subsets of evaluation data. For example, for the evaluation results in Table 2, the optimization suggestion could be to train the AI ​​model with new data where the area of ​​the bounding box occupies 0%-20% of the image. It should be understood that the new data obtained based on the optimization suggestions for continued training can be newly collected data, or data whose data feature values ​​in the original training data have been adjusted.

[0181] Optionally, the sensitivity of data features to evaluation metrics can be determined based on the evaluation results. Specifically, regression analysis can be performed on the values ​​of data features and the evaluation metric values ​​under the evaluation metrics, based on the inference results of the AI ​​model for each subset of evaluation data corresponding to each data feature. This yields the sensitivity of data features to the evaluation metrics. In other words, the values ​​of data features can be used as input, and the evaluation metric values ​​under the evaluation metrics, based on the inference results of the AI ​​model for each subset of evaluation data corresponding to each data feature, can be used as output for regression analysis to obtain the sensitivity of data features to the evaluation metrics. For example, using linear regression f(z...)... t ) = W T z t The value of a set of data features is z t A vector, such as one containing four dimensions of image brightness, sharpness, resolution, and saturation, uses the evaluation index value of the inference result of the evaluation data subset corresponding to the data features under the evaluation index as f(z). t The fitted W vector is the influence weight of each data feature on each evaluation index, i.e., the sensitivity.

[0182] After calculating the sensitivity of each data feature to each evaluation metric, optimization suggestions for the AI ​​model can be generated based on this sensitivity. A data feature with a sensitivity greater than a certain value can be considered to have a significant impact on the evaluation metric, and corresponding optimization suggestions can be generated for this phenomenon. For example, if image brightness significantly affects accuracy, it can be suggested to continue training the AI ​​model with images whose brightness values ​​fall within one or more ranges. Since the current AI model's inference accuracy for these images still has room for improvement, training the current AI model with new data based on this optimization suggestion will likely improve the AI ​​model's inference ability.

[0183] Optionally, the above method may further include: generating an evaluation report and sending the evaluation report. The evaluation report may include at least one of evaluation results and optimization suggestions. After obtaining the evaluation results by calculating the accuracy of the AI ​​model's inference for each subset of evaluation data based on the comparison results, and / or after generating optimization suggestions for the AI ​​model based on the evaluation results, an evaluation report including the evaluation results and / or optimization suggestions can be generated.

[0184] Optionally, the above method may further include: calculating the accuracy of the AI ​​model's overall inference on the evaluation dataset. Specifically, the inference results of the AI ​​model on the evaluation data in the evaluation dataset can be determined first. Then, the inference results of each evaluation data are compared with the labels of each evaluation data. Finally, the accuracy of the AI ​​model's inference on the evaluation dataset is calculated based on the comparison results, thus obtaining the AI ​​model's evaluation result on the global data. The difference here is that it is not necessary to divide the evaluation dataset into multiple subsets of evaluation data. Instead, the evaluation dataset is calculated as a whole. Since all evaluation data in the evaluation dataset are unselected data, evaluating the AI ​​model's inference ability on the global data—that is, the AI ​​model's inference ability on any type of data that can be used as input to the AI ​​model—can be assessed by evaluating the AI ​​model's overall inference ability on the evaluation dataset. In this application, the global data refers to data obtained without classification based on any data feature, and it can represent any type of data that can be used as input to the AI ​​model.

[0185] Optionally, the aforementioned evaluation report may also include the accuracy of the AI ​​model's inference on the evaluation dataset.

[0186] Optionally, the above method may further include: obtaining performance parameters. During the inference process using the AI ​​model on the evaluation data in the evaluation dataset, hardware resource usage information and the usage duration and number of operators included in the AI ​​model can be monitored to obtain performance parameters. During the inference process using the AI ​​model on the evaluation dataset, hardware resource usage information and the usage duration and number of operators included in the AI ​​model can be monitored. Hardware resources may include a central processing unit (CPU), a graphics processing unit (GPU), physical memory, GPU video memory, etc. A performance monitoring process can be used to monitor the inference process. Specifically, GPU performance monitoring tools, such as the NVIDIA System Management Interface (SMI), can be called to collect GPU utilization and video memory usage. CPU performance monitoring tools, such as topvmstatiostat, can be called to collect CPU utilization and video memory usage. Operator performance monitoring tools, such as profilers, can be called to collect the usage duration and number of operators included in the AI ​​model.

[0187] Optionally, the aforementioned optimization suggestions may also include optimization suggestions generated based on performance parameters. After obtaining the performance parameters, optimization suggestions for the AI ​​model can be generated based on the performance parameters. Performance optimization suggestions for the AI ​​model can be generated based on hardware resource usage information, the usage duration of operators included in the AI ​​model, the number of operators used, and a performance tuning knowledge base. The performance tuning knowledge base may include phenomena corresponding to hardware resource usage information, phenomena corresponding to operator usage, and performance optimization methods corresponding to the phenomena corresponding to hardware resource usage information and operator usage. For example, if the phenomenon corresponding to hardware resource usage information is high memory consumption, the performance optimization suggestion could be to adjust the precision of the AI ​​model parameters to 8-bit quantization, or to enable operator fusion. As another example, if the phenomenon corresponding to hardware resource usage information is high memory consumption, the performance optimization method corresponding to this phenomenon could be to adjust the precision of the AI ​​model parameters to half-precision or int8 quantization.

[0188] Optionally, the above steps can be performed multiple times, i.e., multiple evaluations. The execution steps are the same each time, the difference being that the evaluation dataset used each time is slightly different. For example, the first evaluation dataset used might be the dataset uploaded by the user or sent by the terminal device. Subsequent evaluation datasets might be datasets with adjusted data features from the received dataset, but the adjusted datasets will not affect the visual effect. Adjustments could include adding noise, changing the brightness value of some data in the evaluation dataset, or adjusting other data features; there are no limitations on this. Afterwards, these multiple evaluation reports and optimization suggestions can be combined to obtain more accurate suggestions and reports, thereby improving the robustness of the evaluation. For example, if the second evaluation dataset has added noise compared to the received dataset, the accuracy and precision of the second evaluation report will be lower than the first evaluation report, indicating that noise has a significant impact on the AI ​​model. Therefore, noise interference should be avoided as much as possible.

[0189] Optionally, in this embodiment, the evaluation of the AI ​​model can also utilize engine-related tools, such as the profiler tool provided by TensorFlow or MXNet, to analyze the AI ​​model's structure, the operators it includes, the time complexity of the operators, and the space complexity of the operators. The AI ​​model's structure may include residual structures, multi-level feature extraction, etc. The optimization suggestions may also include suggestions for modifying the AI ​​model's structure based on the above analysis. For example, if the analysis reveals that the AI ​​model does not include a batch normalization (BN) layer, it may introduce the risk of overfitting; therefore, a suggestion to add a BN layer can be generated. As another example, if the AI ​​model's structure includes multi-level features for feature extraction and classification, and the bounding boxes to be identified include multiple scales, it may be impossible to identify bounding boxes at all scales, only some scales. The time and space complexity of the operators can be linear or exponential. If the operator's space complexity is exponential, it indicates that the AI ​​model's structure is relatively complex, and a pruning suggestion can be generated, i.e., adjusting the AI ​​model's structure.

[0190] The aforementioned suggestions and reports can be provided to users via a GUI, a JavaScript object notation (JSON) document, or sent to the user's terminal device.

[0191] Please see Figure 8 , Figure 8This is a flowchart illustrating another AI model evaluation method disclosed in an embodiment of this application. The AI ​​model evaluation method is applied to an evaluation system. Figure 8 As shown, the evaluation method for this AI model may include the following steps.

[0192] 801. Obtain AI models and evaluation datasets.

[0193] For a detailed description of step 801, please refer to step 601.

[0194] 802. Use AI models to infer the evaluation data in the evaluation dataset.

[0195] For a detailed description of step 802, please refer to step 604. The difference between step 802 and step 604 is that step 802 involves reasoning about the evaluation data in the evaluation dataset without needing to partition the evaluation dataset, while step 604 involves reasoning about the evaluation data in the evaluation dataset divided into at least one subset of evaluation data, which requires first partitioning the evaluation data in the evaluation dataset into at least one subset of evaluation data.

[0196] 803. Obtain performance data.

[0197] During the inference process of using an AI model to infer the evaluation data in the evaluation dataset, the performance of the hardware during the inference process can be monitored, i.e., the hardware resource usage information, as well as the usage time and number of operators included in the AI ​​model, to obtain performance parameters. In other words, performance data is used to represent the performance of the hardware executing the inference process during the AI ​​model's inference of the evaluation data, or the usage of the operators included in the AI ​​model during the inference process. The operator usage information refers to the usage time of each operator in the AI ​​model during the inference process or the number of times each operator is used in the AI ​​model. A detailed description of step 803 can be found in the relevant description above.

[0198] 804. Generate optimization suggestions for the AI ​​model based on performance data.

[0199] After obtaining the performance data, optimization suggestions for the AI ​​model can be generated based on the performance data. Optimization suggestions may include adjusting the structure of the AI ​​model, or optimizing the training of the AI ​​model's operators. A detailed description of step 804 can be found in the relevant description above.

[0200] Optionally, the above method may further include: generating an evaluation report and sending the evaluation report. After generating optimization suggestions for the AI ​​model based on the performance data, an evaluation report can be generated and sent, either to a terminal device or to the user's email address. The evaluation report may include at least one of the performance data and optimization suggestions.

[0201] Optionally, the above method may further include: calculating the accuracy of the AI ​​model's inference on the evaluation dataset. Specifically, the inference result of the AI ​​model on the evaluation data in the evaluation dataset can be determined first, then the inference result of each evaluation data is compared with the label of each evaluation data, and finally the accuracy of the AI ​​model's inference on the evaluation dataset is calculated based on the comparison result. A detailed description can be found in the relevant description above.

[0202] The following example demonstrates the steps performed by an AI model on an evaluation dataset where the evaluation data consists of microbial images and the task is object detection. After inference, the AI ​​model infers the detected epithelial cells, subspores, cocci, leukocytes, spores, fungi, and clue cells. When data features include image brightness and evaluation metrics include F1 scores, the evaluation report can include the FI values ​​of the AI ​​model for four subsets of evaluation data divided according to brightness value distribution, as shown in Table 3.

[0203]

[0204] Table 3 shows the F1 values ​​of the four evaluation data subsets divided according to the brightness value distribution. In step 603, the microbial images can be arranged in descending or ascending order of brightness value. Then, the first 25% (0-25%) of the evaluation data is determined as the first evaluation data subset, the next 25% (25%-50%) as the second evaluation data subset, the next 25% (50%-75%) as the third evaluation data subset, and the last 25% (75%-100%) as the fourth evaluation data subset. Then, in step 605, the F1 values ​​for epithelial cells, subspores, cocci, leukocytes, spores, fungi, and clue cells in the first to fourth evaluation data subsets are calculated respectively. Furthermore, in step 605, after calculating the F1 values ​​of epithelial cells, subspores, cocci, leukocytes, spores, fungi, and clue cells in the first and fourth evaluation data subsets, the mAP of the F1 values ​​of epithelial cells, subspores, cocci, leukocytes, spores, fungi, and clue cells in the first and fourth data subsets can also be calculated, as well as the standard deviation (STD) of the F1 values ​​of epithelial cells, subspores, cocci, leukocytes, spores, fungi, and clue cells in all evaluation data, i.e., sensitivity. Table 3 shows that image brightness has a significant impact on epithelial cells and clue cells. Accordingly, suggestions can be given for training the AI ​​model with images having brightness values ​​between 25% and 50% and images with brightness values ​​between 50% and 75%. When data features include the size of the bounding boxes and evaluation metrics include F1 values, the evaluation results in the evaluation report can include the FI values ​​of the AI ​​model for the evaluation data of the four evaluation data subsets divided according to the bounding box size distribution, as shown in Table 4.

[0205]

[0206] Table 4 shows the F1 scores of four subsets of live human assessment data divided according to the distribution of bounding box size. The process for Table 4 is similar to that in Table 3 and will not be detailed here. Table 4 concludes that bounding box size has a significant impact on epithelial cells and clue cells. Accordingly, images with bounding box sizes increased by 0-25%, 25%-50%, and 50%-75% can be used to train the AI ​​model. Please refer to [link to relevant documentation]. Figure 9 , Figure 9 This is a distribution diagram of the brightness of the annotation boxes for microbial detection disclosed in the embodiments of this application. For example... Figure 9 As shown, the brightness of the area within the labeled box is mostly concentrated between 50 and 170. Please refer to [link / reference]. Figure 10 , Figure 10 This is a distribution diagram showing the proportion of the area of ​​the labeled boxes for microbial detection disclosed in the embodiments of this application to the image. For example... Figure 10 As shown, the area of ​​the labeled boxes mostly accounts for a proportion of the image between 0 and 0.05. The evaluation report may also include performance data; the hardware resource usage information in the acquired performance data is shown in Table 5.

[0207] Hardware resource usage information peak mean GPU utilization 65％ 30％ CPU utilization 60％ 40％ Physical memory 390M 270M GPU memory 1570M 1240M

[0208] Table 5 Hardware Resource Usage Information

[0209] Table 5 shows that memory consumption is relatively high. Therefore, it is recommended to adjust the parameter precision in the AI ​​model to half-precision or int8 quantization. The usage of operators in the obtained performance data is shown in Table 6.

[0210] Operator Total usage time Average usage time Quantity used Detection box generation (contrib_Proposal) 1329.748ms 120.886ms 11 Convolution and activation 1221.938ms 9.257ms 132 Convolution, activation, and pooling. 1162.373ms 23.722ms 49 Fully connected, activated 260.557ms 13.028ms 20 Normalization (softmax) 138.426ms 12.584ms 11 Dimensionality reduction (flatten) 130.858ms 13.086ms 10 Reshape 32.838ms 2.985ms 11

[0211] Table 6. Operator Usage

[0212] Table 6 shows that the detection box generation operator is time-consuming. Therefore, suggestions for optimizing the detection box generation operator are provided. After one evaluation, the AI ​​model for the microbial cells can be retrained based on the suggestions given above. Please refer to [link to relevant documentation]. Figure 11 , Figure 11 This is a schematic diagram of the mAP (mAP) before and after retraining of an AI model corresponding to a microbial cell, as disclosed in an embodiment of this application. Figure 11 As shown, the mAP before retraining was 0.4421. After randomly scaling the image, the mAP after retraining was 0.4482, and after adjusting the image brightness, the mAP after retraining was 0.45. It is evident that the AI ​​model retrained according to the recommendations is superior to the model before retraining.

[0213] The following example demonstrates the execution of the above steps using a trained AI model whose evaluation data consists of images of people and whose task is object detection. After inferring from the evaluation data in the dataset, the AI ​​model outputs five categories: no helmet, wearing a white helmet, wearing a yellow helmet, wearing a red helmet, and wearing a blue helmet. Please refer to [link / reference]. Figure 12 , Figure 12 This is a curve showing the FI value versus confidence threshold of an AI model for helmet detection disclosed in an embodiment of this application. The F1 value is calculated based on the comparison results to determine the accuracy of the AI ​​model's inference for each subset of evaluation data. Figure 12 As shown, the F1 score first increases and then decreases as the confidence threshold increases. The F1 score is highest when the confidence threshold is 0.37; therefore, the confidence threshold can be set to 0.37. Please refer to [link to relevant documentation]. Figure 13 , Figure 13This application discloses a PR curve for an AI model used for helmet detection. The PR curve is calculated by determining the evaluation result through a step of calculating the accuracy of the AI ​​model's inference for each subset of evaluation data based on comparison results. Figure 13 As shown, the PR curves for the five types of detection results are different. When data features include ambiguity and evaluation metrics include recall, the evaluation report can include the recall values ​​of the AI ​​model for the four evaluation data subsets divided according to ambiguity distribution, as shown in Table 7:

[0214]

[0215] Table 7 shows the recall values ​​of the evaluation data for the four evaluation data subsets divided according to the blurriness distribution. Table 7 indicates that image blurriness has a significant impact on the absence of a safety helmet. Accordingly, suggestions can be given for training the AI ​​model on images with blurriness between 50% and 85% and images with blurriness between 85% and 100%. When data features include the number of bounding boxes and the evaluation metric includes recall, the evaluation report can include the recall values ​​of the AI ​​model on the four evaluation data subsets divided according to the number of bounding boxes, as shown in Table 8.

[0216]

[0217] Table 8 shows the recall values ​​of the evaluation data for the four evaluation data subsets divided according to the distribution of the number of bounding boxes. Table 8 shows that the blur of the image has a significant impact on the images of those without helmets, those with yellow helmets, and those with white helmets. Accordingly, we can give a suggestion to increase the number of bounding boxes in images between 85% and 100% for training the AI ​​model.

[0218] Please see Figure 14 , Figure 14 This is a schematic diagram of another evaluation system 1400 disclosed in an embodiment of this application. Figure 14 As shown, the evaluation system 1400 may include an I / O module 1401, a data analysis module 1402, and an inference module 1403.

[0219] Optionally, the assessment system 1400 may also include a diagnostic module 1404.

[0220] Optionally, the evaluation system 1400 may also include a performance monitoring module 1405.

[0221] Optionally, the evaluation system 1400 may also include a model analysis module 1406.

[0222] Detailed descriptions of the I / O module 1401, data analysis module 1402, inference module 1403, performance monitoring module 1405, and model analysis module 1406 in the evaluation system 1400 can be found in [reference needed]. Figure 6 Corresponding method implementation examples.

[0223] Please see Figure 15 , Figure 15 This is a schematic diagram of the structure of another evaluation system 1500 disclosed in the embodiments of this application. For example... Figure 15 As shown, the evaluation system 1500 may include an I / O module 1501, an inference module 1502, a performance monitoring module 1503, and a diagnostic module 1504.

[0224] Optionally, the evaluation system 1500 may also include a model analysis module 1505.

[0225] Detailed descriptions of the I / O module 1501, inference module 1502, performance monitoring module 1503, diagnostic module 1504, and model analysis module 1505 in the evaluation system 1500 can be found in [reference needed]. Figure 8 Corresponding method implementation examples.

[0226] Please see Figure 16 , Figure 16 This is a schematic diagram of the structure of a computing device disclosed in an embodiment of this application. Figure 16 As shown, the computing device 1600 includes a memory 1601, a processor 1602, a communication interface 1603, and a bus 1604. The memory 1601, processor 1602, and communication interface 1603 are interconnected via the bus 1604.

[0227] The memory 1601 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1601 can store programs. When the program stored in the memory 1601 is executed by the processor 1602, the processor 1602 and the communication interface 1603 are used to execute the aforementioned program. Figure 6 or Figure 8 This provides a method for users to evaluate AI models. The memory 1601 can also store the evaluation dataset.

[0228] The processor 1602 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits.

[0229] The communication interface 1603 uses transceiver modules, such as, but not limited to, transceivers, to enable communication between the computing device 1600 and other devices or communication networks. For example, an evaluation dataset can be obtained through the communication interface 1603.

[0230] Bus 1604 may include a pathway for transmitting information between various components of computing device 1600 (e.g., memory 1601, processor 1602, communication interface 1603).

[0231] Since the modules in the evaluation system 500, evaluation system 1400, and evaluation system 1500 provided in this application can be distributed and deployed on multiple computers in the same or different environments, please refer to... Figure 17 , Figure 17 This is a schematic diagram of the structure of another computing device disclosed in an embodiment of this application. Figure 17 The computing device shown includes multiple computers, each comprising a memory, a processor, a communication interface, and a bus. The memory, processor, and communication interface are interconnected via the bus.

[0232] The memory can be ROM, static storage, dynamic storage, or RAM. Memory can store programs; when executed by a processor, the processor and communication interface perform the portion of the evaluation system that evaluates the AI ​​model for the user. Memory can also store evaluation datasets; for example, a portion of the memory's storage resources can be divided into a dataset storage module to store the evaluation datasets required by the evaluation system, and a portion can be divided into a results storage module to store evaluation reports.

[0233] The processor can be a general-purpose CPU, microprocessor, ASIC, GPU, or one or more integrated circuits.

[0234] Communication interfaces use transceiver modules, such as, but not limited to, transceivers, to enable communication between a computer and other devices or communication networks. For example, an evaluation dataset can be obtained through a communication interface.

[0235] A bus can include a pathway for transmitting information between various components of a computer (e.g., memory, processor, communication interface).

[0236] Each of the aforementioned computers establishes a communication path through a communication network. Each computer runs any one or more modules of evaluation system 500, evaluation system 1400, and evaluation system 1500. Any computer can be a computer in a cloud data center (e.g., a server), a computer in an edge data center, or a terminal computing device.

[0237] The descriptions of the processes corresponding to the above-mentioned figures each have their own emphasis. For parts of a process that are not described in detail, please refer to the relevant descriptions of other processes.

[0238] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product provided for evaluation includes one or more computer instructions for evaluation, which, when loaded and executed on a computer, produce, in whole or in part, the results according to the embodiments of the present invention. Figure 6 or Figure 8 The process or function described.

[0239] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium stores computer program instructions that provide evaluation. The computer-readable storage medium may be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., SSD).

Claims

1. A method for evaluating an artificial intelligence (AI) model, characterized in that, include: The computing device acquires the AI ​​model and the evaluation dataset, and acquires one or more evaluation metrics corresponding to the AI ​​model. The evaluation dataset includes multiple labeled evaluation data, and the label of each evaluation data is used to represent the true result corresponding to the evaluation data. The type of the evaluation data is image, video, audio or text. The computing device calculates the data features of each evaluation data in the evaluation dataset, classifies the evaluation data in the evaluation dataset according to the data features, and obtains a subset of evaluation data. The subset of evaluation data is a subset of the evaluation dataset, and the values ​​of the data features of all evaluation data in the subset of evaluation data satisfy the following conditions. The computing device determines the inference result of the AI ​​model on the evaluation data in the evaluation data subset, compares the inference result of each evaluation data in the evaluation data subset with the label of each evaluation data in the evaluation data subset, and calculates the accuracy of the AI ​​model's inference on the evaluation data subset based on the comparison result and the one or more evaluation indicators, so as to obtain the evaluation result of the AI ​​model on the data whose data feature values ​​meet the conditions, wherein the accuracy includes the values ​​corresponding to one or more evaluation indicators; The computing device generates optimization suggestions for the AI ​​model based on the evaluation results. The optimization suggestions include training the AI ​​model with new data whose values ​​satisfy the conditions.

2. The method of claim 1, further comprising: The computing device acquires performance data, which represents the performance of the hardware executing the inference process during the AI ​​model's inference of the evaluation data, and / or the usage of operators included in the AI ​​model during the AI ​​model's inference of the evaluation data.

3. The method as described in any one of claims 1 or 2, characterized in that, The method further includes: The computing device determines the inference result of the AI ​​model on the evaluation data in the evaluation dataset; The computing device calculates the accuracy of the AI ​​model's inference on the evaluation dataset based on the comparison results of the inference results of the evaluation data in the evaluation dataset and the labels of the evaluation data in the evaluation dataset, so as to obtain the evaluation results of the AI ​​model on the global data.

4. The method according to any one of claims 1-2, characterized in that, The number of data features is multiple, and the condition includes multiple sub-conditions. The relationship between the multiple data features and the multiple sub-conditions is one-to-one. The computing device classifies the evaluation data in the evaluation dataset according to data characteristics to obtain a subset of evaluation data, including: The computing device classifies the evaluation data in the evaluation dataset according to the plurality of data features to obtain a subset of evaluation data, wherein each value of the plurality of data features of all evaluation data in the subset of evaluation data satisfies the corresponding sub-condition in the conditions.

5. An evaluation method for an artificial intelligence (AI) model, characterized in that, include: The computing device acquires the AI ​​model and the evaluation dataset, which includes multiple labeled evaluation data. The label of each evaluation data is used to represent the actual result corresponding to the evaluation data. The evaluation data can be in the form of images, videos, audio, or text. The computing device uses the AI ​​model to infer the evaluation data in the evaluation dataset; The computing device acquires performance data, which represents the performance of the hardware executing the inference process during the inference process of the AI ​​model on the evaluation data, and / or the usage of operators included in the AI ​​model during the inference process of the AI ​​model on the evaluation data. The usage of operators includes: the usage time of the operators in the AI ​​model and the number of operators used in the AI ​​model. The computing device generates optimization suggestions for the AI ​​model based on the performance data. The optimization suggestions include: adjusting the structure of the AI ​​model, and / or optimizing the training of the operators of the AI ​​model.

6. An evaluation system for an artificial intelligence (AI) model, characterized in that, The system includes: The input / output (I / O) module is used to acquire the AI ​​model and the evaluation dataset, and to acquire one or more evaluation metrics corresponding to the AI ​​model. The evaluation dataset includes multiple labeled evaluation data, and the label of each evaluation data is used to represent the actual result corresponding to the evaluation data. The type of the evaluation data is image, video, audio, or text. The data analysis module is used to calculate the data features of each evaluation data in the evaluation dataset, classify the evaluation data in the evaluation dataset according to the data features, and obtain an evaluation data subset. The evaluation data subset is a subset of the evaluation dataset, and the values ​​of the data features of all evaluation data in the evaluation data subset satisfy the following conditions. The inference module is used to determine the inference result of the AI ​​model on the evaluation data in the evaluation data subset; The data analysis module is further configured to compare the inference result of each evaluation data in the evaluation data subset with the label of each evaluation data in the evaluation data subset, and calculate the accuracy of the AI ​​model's inference of the evaluation data subset based on the comparison result and the one or more evaluation indicators, so as to obtain the evaluation result of the AI ​​model for data whose data feature values ​​meet the conditions, wherein the accuracy includes the values ​​corresponding to one or more evaluation indicators; A diagnostic module is used to generate optimization suggestions for the AI ​​model based on the evaluation results. The optimization suggestions include training the AI ​​model with new data whose values ​​satisfy the conditions.

7. The system as described in claim 6, characterized in that, The system also includes: A performance monitoring module is used to acquire performance data, which represents the performance of the hardware executing the inference process during the inference process of the AI ​​model on the evaluation data, and / or the usage of operators included in the AI ​​model during the inference process of the AI ​​model on the evaluation data.

8. The system as described in claim 6 or 7, characterized in that, The inference module is also used to determine the inference result of the AI ​​model on the evaluation data in the evaluation dataset; The system also includes: The model analysis module is used to calculate the accuracy of the AI ​​model's inference on the evaluation dataset based on the comparison results of the inference results of the evaluation data in the evaluation dataset and the labels of the evaluation data in the evaluation dataset, so as to obtain the evaluation results of the AI ​​model on the global data.

9. The system as described in claim 6 or 7, characterized in that, The number of data features is multiple, and the condition includes multiple sub-conditions. The relationship between the multiple data features and the multiple sub-conditions is one-to-one. The data analysis module is specifically used to calculate the data features of each evaluation data in the evaluation dataset, classify the evaluation data in the evaluation dataset according to the multiple data features, and obtain an evaluation data subset, wherein each value of the multiple data features of all evaluation data in the evaluation data subset satisfies the corresponding sub-condition in the conditions.

10. An evaluation system for an artificial intelligence (AI) model, characterized in that, The system includes: The input / output (I / O) module is used to acquire the AI ​​model and the evaluation dataset. The evaluation dataset includes multiple labeled evaluation data, where the label of each evaluation data represents the actual result corresponding to the evaluation data. The evaluation data can be in the form of images, videos, audio, or text. The inference module is used to infer the evaluation data in the evaluation dataset using the AI ​​model; A performance monitoring module is used to acquire performance data, which represents the performance of the hardware executing the inference process during the inference process of the AI ​​model on the evaluation data, or the usage of operators included in the AI ​​model during the inference process of the AI ​​model on the evaluation data. The usage of operators includes the usage time of the AI ​​model's operators and the number of operators used in the AI ​​model. The diagnostic module is used to generate optimization suggestions for the AI ​​model based on the performance data. The optimization suggestions include: adjusting the structure of the AI ​​model, and / or optimizing the training of the operators of the AI ​​model.

11. A computing device, characterized in that, The computing device includes a memory and a processor, the memory being used to store a set of computer instructions; The processor executes a set of computer instructions stored in the memory to perform the method of any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program code, which, when executed by a computing device, performs the method described in any one of claims 1 to 5.

Images