Methods, apparatus, electronic devices and storage media for evaluating the performance of terminal equipment

Abstract

This invention proposes a method, apparatus, electronic device, and storage medium for evaluating the performance of a terminal device. The method includes: receiving a user's instruction to initiate an AI performance evaluation process; controlling and executing each individual AI performance test, and determining the score for each individual AI performance test; determining a comprehensive AI performance score for the terminal device based on the scores of all individual AI performance tests, and thus evaluating the performance of the terminal device. This method evaluates the AI ​​performance of the terminal device through multiple individual AI performance tests, enabling the testing of the device's actual performance when handling different AI applications, and providing a more comprehensive assessment of the terminal device's AI performance.

Description

Technical Field

[0001] This invention relates to the field of computer application technology, and in particular to a method, apparatus, electronic device, and storage medium for evaluating the performance of terminal devices. Background Technology

[0002] Existing AI performance testing methods for terminal devices typically only test the average latency and accuracy of AI applications such as image classification and object recognition, and evaluate the device's AI performance based on the test results. However, existing methods can only test the performance of terminal devices running a small subset of AI applications. AI technology has many other application areas; therefore, current terminal device AI performance testing methods cannot measure the comprehensive performance of terminal devices when running multiple other AI applications. Furthermore, the AI ​​model loading performance of terminal devices directly affects the user experience when using AI applications, but current AI performance testing methods cannot measure the model loading performance level of terminal devices. Additionally, when scoring and evaluating the AI ​​performance of terminal devices, current AI performance testing methods mainly use average processing latency and accuracy, ignoring the computational complexity of the AI ​​neural network model itself and the number of parameters in the AI ​​model. This results in users not being able to accurately understand the AI ​​performance of the terminal device. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention proposes a method, apparatus, electronic device, and storage medium for evaluating the performance of terminal devices. By conducting various individual AI performance tests, the AI ​​performance of the terminal device can be evaluated, enabling the testing of the device's actual performance when handling different AI applications and providing a more comprehensive assessment of the terminal device's AI performance.

[0004] To achieve the above objectives, the present invention provides a method for evaluating the performance of a terminal device, comprising:

[0005] Receive user instructions to initiate the AI ​​performance evaluation process;

[0006] Control and execute each AI performance test individually, and determine the score for each of the AI ​​performance tests individually;

[0007] Based on the scores of all the individual AI performance tests, a comprehensive AI performance score for the terminal device is determined, and the performance of the terminal device is evaluated.

[0008] Optionally, the execution process for each of the aforementioned individual AI performance tests includes:

[0009] Load the test dataset for the AI ​​performance single test;

[0010] The AI ​​neural network model corresponding to the AI ​​performance single test is loaded from the storage system of the terminal device into memory, the model initialization process is completed, and the model loading time is recorded.

[0011] The loaded model is used to process multiple test data in the test dataset to obtain the test output results information for each test data and the average processing latency data of the terminal device.

[0012] The test output results of each test data are compared with the true value of the test data, and the accuracy test result of the terminal device is calculated according to the accuracy calculation formula corresponding to the single AI performance test.

[0013] Based on the AI ​​model loading time, the average processing latency data of the terminal device, and the accuracy test results of the terminal device, the score obtained by the terminal device in the AI ​​performance single test is calculated.

[0014] Optionally, the score calculation formula for the AI ​​performance single test is as follows:

[0015] S i =P i +T i *w i

[0016] Among them, S i P represents the performance score of the terminal device in the i-th AI performance test. i T represents the data processing performance score of the terminal device in the i-th AI performance test. i w is the model loading performance score of the terminal device in the i-th AI performance test. i The score ratio weight of the i-th AI performance single test;

[0017] The formula for calculating the data processing performance score is as follows:

[0018]

[0019] Among them, P i MAC is the data processing performance score of the terminal device in the i-th AI performance test. i L represents the number of multiplication and addition operations included in the AI ​​neural network model in the i-th AI performance single test; i It is the average latency data of the terminal device in processing each unit of data in the i-th AI performance test;

[0020] The formula for calculating the model loading performance score is:

[0021]

[0022] Among them, T i M represents the model loading performance score of the terminal device in the i-th AI performance test. i s represents the number of model parameters of the AI ​​neural network model in the i-th AI performance single test, that is, the amount of parameter data for the model weights in the AI ​​neural network model; i The time taken by the terminal device to complete the AI ​​neural network model loading and model initialization process in the i-th AI performance single test.

[0023] Optionally, the AI ​​performance tests may include any one of the following: face recognition test, voice keyword recognition test, image classification test, object recognition test, super-resolution test, human pose recognition test, and semantic segmentation test.

[0024] Optionally, if the AI ​​performance test includes a human pose recognition test, the execution process for the human pose recognition AI performance test includes:

[0025] The AI ​​performance test for human posture recognition is run, and key points of the human body are identified as posture recognition feature points. The accuracy of the key points is determined by the similarity of the key points, and the AI ​​human posture recognition capability of the terminal device is evaluated.

[0026] Optionally, the formula for calculating the keypoint similarity OKS is:

[0027]

[0028] Where p represents a person in the true values ​​of the test data, p i This represents the i-th key point of the p-th person;

[0029] Let represent the Euclidean distance between the i-th keypoint of the p-th person and the i-th keypoint in the output result, and its calculation formula is:

[0030]

[0031] Among them, (x' i ,y' i () represents the detection result coordinates of the i-th key point in the test output information. Let be the coordinates of the i-th keypoint of the p-th person in the true value;

[0032] v pi Indicate the visibility of the i-th keypoint of the p-th person;

[0033] S p The scale factor for the p-th person is calculated using the following formula: w and h are the width and height of the detection box for the p-th person, respectively;

[0034] σ i The normalization factor for the keypoint with id i is determined by the standard deviation between manually labeled and true keypoints in all test datasets.

[0035] δ(*) means that if condition * is true, then δ(*) = 1, otherwise δ(*) = 0. It is used to determine whether a key point is a point that has been marked in the truth value.

[0036] Optionally, if the AI ​​performance test includes a speech keyword recognition test, the execution process for the speech keyword recognition AI performance test includes:

[0037] Run the AI ​​performance test for voice keyword recognition, and evaluate the performance level of the terminal device in recognizing voice keywords by the voice recognition accuracy, including:

[0038] The formula for calculating the speech recognition accuracy is as follows:

[0039]

[0040] Among them, Acuraccy _0 Let n0 be the speech recognition accuracy, n0 be the number of incorrectly recognized samples, and t0 be the total number of test data.

[0041] Using the speech recognition accuracy as a reference, the data processing performance of the speech keyword recognition AI performance single test is scored, and the score of the terminal device in the speech keyword recognition AI performance single test is calculated by combining the model loading performance score.

[0042] Optionally, if the AI ​​performance test includes a face recognition test, the face recognition AI performance test execution process includes:

[0043] The face recognition AI performance single test is run to evaluate the performance of the terminal device in processing AI face recognition tasks based on the face recognition accuracy, including:

[0044] The formula for calculating the accuracy of facial recognition is as follows:

[0045]

[0046] Among them, Acuraccy _1Let n be the face recognition accuracy, n1 be the number of incorrectly identified samples, and t1 be the total number of test data.

[0047] Using the face recognition accuracy as a reference, the data processing performance of the face recognition AI performance test is scored, and the score of the terminal device in the face recognition AI performance test is calculated by combining the model loading performance score.

[0048] Optionally, if the AI ​​performance test includes a semantic segmentation test, the semantic segmentation AI performance test execution process includes:

[0049] Run the single-item test of the semantic segmentation AI, and evaluate the performance of the terminal device in processing AI semantic segmentation tasks by using the average intersection-union ratio (AUC) metric.

[0050] The average intersection-union ratio is used as a reference to score the data processing performance of the semantic segmentation AI performance test, and the score of the terminal device in the semantic segmentation AI performance test is calculated by combining the model loading performance score.

[0051] Optionally, evaluating the performance of the terminal device in processing AI semantic segmentation tasks using the average intersection-union ratio (AUC) includes:

[0052] Load the test dataset for the semantic segmentation AI performance single test, construct an output matrix of the categories of the true values ​​and the categories of the predicted values ​​in the test dataset, and generate a confusion matrix for the semantic segmentation AI performance single test by comparing the value of each pixel in the output matrix with the value of each pixel in the true value matrix.

[0053] After obtaining the confusion matrix, the Intersection over Union (IoU) is calculated. The formula for calculating IoU is:

[0054]

[0055] Among them, IoU i V represents the IoU value for the i-th class. i Let R be the value in the i-th row and i-th column of the confusion matrix. i C is the sum of all elements in the i-th row of the confusion matrix. i This is the sum of all elements in the i-th column of the confusion matrix;

[0056] The average IoU value of the test datasets for all categories used in the semantic segmentation AI performance single test is the MIoU (mean intersection-union ratio).

[0057] Optionally, if the AI ​​performance test includes an image classification test, the image classification AI performance test execution process includes:

[0058] The image classification AI performance test is run to evaluate the performance of the terminal device in processing AI image classification tasks based on the image classification accuracy, including:

[0059] Load the test dataset, i.e., the test image set, for the single performance test of the image classification AI;

[0060] After processing a test image, multiple predicted categories and their corresponding probabilities are output. As needed, the first few categories in the output probability values ​​for each test image are compared with the ground truth value for that image. If any category matches the ground truth value, the image is considered correctly classified. After processing all test data, the number of correctly identified images is counted. The formula for calculating the image classification accuracy is:

[0061]

[0062] Where p is the number of correctly identified images, and n is the total number of test images;

[0063] Using the image classification accuracy as a reference, the data processing performance of the image classification AI performance test is scored, and the score of the terminal device in the image classification AI performance test is calculated by combining the model loading performance score.

[0064] Optionally, if the AI ​​performance test includes a super-resolution test, the execution process for the super-resolution AI performance test includes:

[0065] Run the single test of super-resolution AI performance and evaluate the AI ​​super-resolution computing performance of the terminal device through the peak signal-to-noise ratio index;

[0066] Using the peak signal-to-noise ratio as a reference, the data processing performance of the super-resolution AI performance test is scored, and the score of the terminal device in the super-resolution AI performance test is calculated by combining the model loading performance score.

[0067] Optionally, load the test dataset, i.e., the test image set, for the single test of super-resolution AI performance;

[0068] Before testing, the original test data was converted to low-resolution data using a downsampling method. During testing, the low-resolution data was used. After testing, the high-resolution image output by the model was compared with the original image to obtain the peak signal-to-noise ratio (PSNR). The formula for calculating the PSNR is as follows:

[0069]

[0070] Among them, MAXI This represents the maximum value of a single pixel in the image.

[0071] MSE, or Mean Squared Error, is used to reflect the degree of difference between the estimator and the estimated quantity. The formula for calculating MSE is:

[0072]

[0073] Where I and K are the pixel values ​​in the i-th row and j-th column of the original test image and the output image, respectively, and m and n are the horizontal and vertical resolutions.

[0074] Optionally, if the AI ​​performance test includes an object recognition test, the execution process for the object recognition AI performance test includes:

[0075] Run the single test of the AI ​​performance of object recognition, and evaluate the AI ​​object recognition computing performance of the terminal device by the mAP average accuracy.

[0076] Using the mAP average accuracy as a reference, the data processing performance of the object recognition AI performance single test is scored, and the score of the terminal device in the object recognition AI performance single test is calculated by combining the model loading performance score.

[0077] Optionally, determining the overall AI performance score of the terminal device based on the scores of all the individual AI performance tests includes:

[0078] The overall AI performance score of the terminal device is calculated by taking the weighted average of the scores of each individual AI performance test.

[0079] In another aspect, the present invention provides a performance evaluation apparatus for a terminal device, employing the aforementioned performance evaluation method for a terminal device, comprising:

[0080] The receiving module is used to receive user commands to initiate the AI ​​performance evaluation process;

[0081] The control module is used to control and execute each AI performance test and determine the score for each AI performance test.

[0082] The evaluation module is used to determine the overall AI performance score of the terminal device based on the scores of all the individual AI performance tests, and to evaluate the performance of the terminal device.

[0083] In another aspect, the present invention provides a storage medium for storing a computer program for executing the performance evaluation method of the terminal device described above.

[0084] In another aspect, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned performance evaluation method for the terminal device.

[0085] As can be seen from the above solutions, the advantages of the present invention are:

[0086] The performance evaluation method for terminal devices provided by this invention uses multiple individual AI performance tests to evaluate the AI ​​performance of the terminal device. By determining the score of each of the aforementioned individual AI performance tests, and based on the scores of all the individual AI performance tests, a comprehensive AI performance score is determined for the terminal device, thereby evaluating the performance of the terminal device. This method can test the actual performance of the device when handling different AI applications, providing a more comprehensive evaluation of the terminal device's AI performance, and thus better helping users choose a terminal device with suitable AI performance according to their own needs. Attached Figure Description

[0087] Figure 1 This is a schematic diagram of the system architecture for terminal performance evaluation methods;

[0088] Figure 2 This is a flowchart of a single AI performance test in an embodiment of this application;

[0089] Figure 3 A framework diagram of a performance evaluation device for terminal equipment;

[0090] Figure 4 This is a block diagram of a portion of the structure of the terminal device provided in the embodiments of this application;

[0091] in:

[0092] 100 - Terminal equipment;

[0093] 400 - Performance Evaluation Device;

[0094] 410 - Receiver module;

[0095] 420 - Control Module;

[0096] 430 - Loading modules;

[0097] 440 - Calculation Module;

[0098] 450 - Statistics Module;

[0099] 460-Evaluation Module;

[0100] 500 - Terminal Equipment;

[0101] 501-IO component;

[0102] 502-Processor;

[0103] 503 - Controller;

[0104] 504-AI computing unit;

[0105] 505 - Storage System;

[0106] 506 - Memory;

[0107] 507 - Memory;

[0108] 508 - Monitor;

[0109] 509-bus;

[0110] 510 - Other external devices. Detailed Implementation

[0111] To make the above features and effects of the present invention clearer and easier to understand, specific embodiments are described below, and detailed descriptions are provided in conjunction with the accompanying drawings.

[0112] As mentioned earlier, existing AI performance testing methods for terminal devices typically only test the performance level of devices when processing some AI applications such as image classification and object recognition. The test indicators are usually average latency data and accuracy data, and the AI ​​performance score is used as the test conclusion to describe the AI ​​performance level of the terminal device.

[0113] However, existing technologies typically evaluate the performance of terminal devices when running a few AI applications such as image classification and object recognition, without using other mainstream AI applications for AI performance testing. In addition, the AI ​​model loading performance of terminal devices directly affects the user experience when using AI applications, but current AI performance testing methods for terminal devices cannot measure the performance of terminal devices in terms of model loading performance. Furthermore, existing AI performance scoring methods mainly use average processing latency and accuracy to score, ignoring the computational complexity of AI neural network models. This results in users not being able to accurately understand the AI ​​performance of terminal devices.

[0114] To address the aforementioned technical problems, this invention provides a performance evaluation method for terminal devices. This method includes seven different AI performance tests: face recognition, speech keyword recognition, image classification, object recognition, super-resolution, human pose recognition, and semantic segmentation. Each test utilizes an AI neural network model with varying computational complexity. Furthermore, this invention provides corresponding AI performance evaluation metrics for the terminal devices, assessing AI model loading performance and AI data processing performance. Finally, this invention provides a comprehensive performance score calculation method for the terminal devices. This method combines the number of AI model parameters and the computational complexity of the AI ​​model to calculate the scores for each AI performance test, resulting in a more balanced weighting of each test score in the overall AI performance score. This scoring method can more accurately describe the performance of the terminal device in each test.

[0115] Next, we will introduce the terminal performance evaluation methods. (See [link to relevant documentation]). Figure 1 , Figure 1 This is a schematic diagram of the system architecture for the terminal device performance evaluation method provided in this application embodiment. The system architecture includes a terminal device 100, on which an evaluation program can be installed and run, allowing users to assess the performance of AI applications running on the terminal device 100. Users send a start test command to the terminal device 100 by opening the evaluation program on the terminal device 100, and the terminal device 100 will initiate the AI ​​performance evaluation process upon receiving the user's evaluation command.

[0116] During this AI performance evaluation process, the terminal device 100 will sequentially execute the seven AI performance tests included in this AI performance evaluation method. During the execution of each AI performance test, the test results generated during the evaluation process will be displayed, and the performance evaluation data for each AI performance test will be statistically analyzed. Then, the AI ​​performance score for each AI performance test will be calculated. After completing all seven AI performance tests, the terminal device 100 will calculate the comprehensive AI performance score of the terminal device 100 based on the AI ​​performance scores of all AI performance tests, and display the performance evaluation result to the user. Based on the performance evaluation result, the user can understand the performance level of the terminal device when running the seven AI performance tests, thereby helping the user to select a terminal device according to their own needs.

[0117] A method for evaluating the performance of a terminal device, wherein the terminal device 100 performs the following process, including:

[0118] S1. Receive user instructions to initiate the AI ​​performance evaluation process;

[0119] S2. Control and execute each AI performance test individually, and determine the score for each of the AI ​​performance tests individually;

[0120] S3. Based on the scores of all the individual AI performance tests, determine the comprehensive AI performance score of the terminal device and evaluate the performance of the terminal device.

[0121] In a practical implementation, the scores of each AI performance test item can be weighted and averaged to calculate the overall AI performance score of the terminal device. The formula for calculating the overall AI performance score is as follows:

[0122]

[0123] In the formula, n is the number of individual AI performance tests, and S i It is the performance score of the i-th AI performance test.

[0124] The terminal device performance evaluation method provided in this embodiment uses multiple AI performance tests to evaluate the AI ​​performance of the terminal device. By determining the score of each AI performance test, and based on the scores of all AI performance tests, a comprehensive AI performance score is determined for the terminal device, thus evaluating the performance of the terminal device. This method can test the actual performance of the device when handling different AI applications, providing a more comprehensive evaluation of the terminal device's AI performance, thereby better helping users choose a terminal device with suitable AI performance according to their needs.

[0125] The AI ​​performance single-item test process in this application embodiment will be described in detail below with reference to the accompanying drawings. See also: Figure 2 , Figure 2 This is a flowchart of an AI performance single-item test in an embodiment of this application. The execution process of each AI performance single-item test includes:

[0126] S201, Load the test dataset for the AI ​​performance single test;

[0127] The test dataset contains both test data and ground truth values. Different AI performance tests use different test datasets. Test data can be images or audio, while ground truth values ​​are the actual information contained in each test data point, such as the category and location of objects in images, or the content of audio data. Before each AI performance test begins, the corresponding test dataset needs to be loaded into the terminal device's memory. At the start of the test, each unit of test data is input into the AI ​​neural network model for processing. After each AI performance test, the ground truth values ​​are compared with the AI ​​model's output data, and the accuracy rate of the device processing the AI ​​model is calculated according to the accuracy calculation formula for each AI performance test.

[0128] S202, load the AI ​​neural network model corresponding to the AI ​​performance single test from the storage system of the terminal device into memory, complete the model initialization process, and count the model loading time.

[0129] The time required to load AI neural network models varies significantly across different devices, primarily due to differences in hardware configuration and software implementation methods. AI neural network model loading time refers to the time required for a terminal device to load the network structure data, weight data, configuration data, and other relevant data of the AI ​​neural network model into memory and complete the initialization process. Longer loading times result in a poorer user experience. Existing technologies typically only assess a device's AI data processing performance and AI application accuracy, without considering the loading performance of the AI ​​neural network model. Therefore, this embodiment uses AI neural network model loading performance as one of the evaluation indicators when assessing the AI ​​performance of terminal devices, thereby providing a more accurate evaluation of the device's AI performance.

[0130] AI neural network models consist of neural network structure information and neural network weight data. Each AI neural network model is composed of multiple layers of different neural network operators (neural network layers). Neural network operators can be understood as specific algorithmic functions required to perform AI computations, such as convolution operators, fully connected operators, pooling operators, etc. Neural network structure information is used to configure which neural network operators a model includes, the number of operators, their order, and their configuration. Neural network weight information consists of the parameter data corresponding to each operator in each layer of the AI ​​neural network model; the parameter data in the weights is determined in advance through the neural network training process. By using neural network structure information and neural network weight data, AI neural network computations can be implemented on terminal devices. Since the number of neural network operators, computational load, and number of parameters vary among different AI neural network models, the amount of neural network weight data also varies. Before testing, since the AI ​​model is stored in the memory of the terminal device, the AI ​​model data needs to be loaded into memory during AI performance testing to shorten the loading time of the model data. At the same time, the terminal device also needs to complete the AI ​​model initialization work. Model initialization is to configure and warm up the computing and storage resources of the terminal device according to the network structure of the AI ​​model in order to prepare for the subsequent S203 steps.

[0131] It should be noted that in the specific implementation, different models and types of terminal devices may have different memory access performances and may use different software methods or interfaces during the S202 process. This results in different time requirements for loading the same AI neural network model on different terminal devices. The longer the loading time of the AI ​​model, the longer the user has to wait. Therefore, the loading time of the terminal device directly affects the user's experience of using AI applications on that terminal device. In order to reflect the performance difference of loading AI neural network models between different terminal devices in the AI ​​performance score, this embodiment will count the delay of the terminal device in completing the S202 process and use it as the evaluation criterion. At the same time, it will combine the number of parameters of the AI ​​neural network model to calculate the performance score of the terminal device loading the AI ​​neural network model.

[0132] S203, using the loaded AI neural network model to process multiple test data in the test dataset, and obtaining the test output result information of each test data and the average processing latency data of the terminal device;

[0133] In the specific implementation, after preparing the test dataset and AI neural network model, the AI ​​performance single test can be started. By inputting the test data into the AI ​​neural network algorithm and combining it with the corresponding AI neural network weight data for calculation, this process requires calling the processor of the terminal device to complete the calculation. In this embodiment, the timing will start when processing the first test data and end when outputting the output result of the last test data, thereby obtaining the total time for the hardware to process all test data. The average processing delay of the terminal device in this AI performance single test is calculated and recorded according to the number of test data. The formula for calculating the average processing delay is shown in formula (2):

[0134]

[0135] In formula (2), L avg This is the average processing latency, where n is the total number of test data sets, and l is the average processing latency. i It is the delay result of the terminal device processing the i-th test data.

[0136] During the testing process, the output data obtained by the terminal device after processing each test data is recorded, that is, the prediction result obtained by the AI ​​neural network model after calculating each unit of data. The prediction result data obtained by different AI performance single tests are different. The prediction result data can be the category, position, etc. of objects in the image, or information contained in the voice data. This application will calculate the accuracy value of the terminal device in the AI ​​performance single test based on all the prediction results output by each AI performance single test.

[0137] S204, compare the test output result information of each test data with the true value of the test data, and calculate the accuracy test result of the terminal device according to the accuracy calculation formula corresponding to the AI ​​performance single test.

[0138] In practice, since the test content and application areas of different AI performance tests are not the same, the formulas for calculating the accuracy of different AI performance tests are different.

[0139] S205, based on the AI ​​model loading time, the average processing latency data of the terminal device, and the accuracy test results of the terminal device, calculate the score of the AI ​​performance single test obtained by the terminal device in the AI ​​performance single test.

[0140] In the specific implementation, the score calculation formula for the AI ​​performance single test is shown in formula (3).

[0141] S i =P i +T i *wi (3)

[0142] In formula (3), S i P represents the performance score of the terminal device in the i-th AI performance test. i T represents the data processing performance score of the terminal device in the i-th AI performance test. i w represents the model loading performance score of the terminal device in the i-th AI performance test. i The weighted percentage of the score for the i-th AI performance test.

[0143] The formula for calculating the data processing performance score is shown in formula (4).

[0144]

[0145] In formula (4), P i MAC is the data processing performance score of the terminal device in the i-th AI performance test. i Let L represent the number of Multiply-Accumulate (MAC) operations in the AI ​​neural network model within the i-th AI performance test. Currently, the computational complexity of mainstream AI neural network models typically depends on the number of MAC operations; the more MAC operations, the more complex the AI ​​neural network model, and the longer it usually takes to complete. i It is the latency data of the terminal device in processing each unit of data on average in the i-th AI performance test. Formula (4) can reflect the time required for the terminal device to process each unit of MAC operation on average. The stronger the computing power of the terminal device, the less time it takes, and the higher the data processing performance score.

[0146] The formula for calculating the model loading performance score is shown in formula (5).

[0147]

[0148] In formula (5), T i M represents the model loading performance score of the terminal device in the i-th AI performance test. i s represents the number of model parameters of the AI ​​neural network model in the i-th AI performance test, specifically the amount of parameter data for the model weights in the AI ​​neural network model, measured in MB. i This is the time taken by the terminal device to complete the loading and initialization of the AI ​​neural network model in the i-th AI performance test. Formula (4) reflects the average time taken by the terminal device to load each MB of parameter data. The stronger the performance of the terminal device in loading the AI ​​model, the shorter the time taken, and the higher the calculated score.

[0149] The AI ​​performance individual test score reflects the comprehensive performance of a terminal device when actually completing AI tasks, allowing users to intuitively understand the AI ​​performance level of different terminal devices. Since the data processing performance score and the model loading performance score use different data units, and these two scores reflect the performance level of the terminal device in two different dimensions, the total AI performance score cannot be obtained by simply adding the two scores. Instead, a weighted average can be applied to one performance score to merge the two scores into a single dimension, which can then be used to evaluate the overall AI performance of the terminal device. The score weight is mainly used to determine the proportion of these two scores in the total score. The score weight comprehensively considers the numerical scale of the two performance scores of mainstream terminal devices, as well as the impact of the AI ​​model loading process and AI computation process on user experience. This application, after extensive research, actual testing, and statistical calculations, has finally determined the score weight corresponding to each AI performance individual test. Table 1 shows the score weight corresponding to each individual test in this application.

[0150] Table 1 shows the score percentage weight for each AI performance test item in this application.

[0151] Serial Number AI performance single test Score proportion weight 1 Face recognition test 13.8 2 Voice keyword recognition test 0.46 3 Image classification test 0.56 4 Object recognition test 0.90 5 Super-resolution test 25.86 6 Human pose recognition test 1.38 7 Semantic segmentation test 3.57

[0152] The seven AI performance tests used in this evaluation method will be introduced below.

[0153] ①When the AI ​​performance test includes an object recognition test, the execution process of the object recognition AI performance test includes:

[0154] The object recognition AI performance test is run, and the AI ​​object recognition computing performance of the terminal device is evaluated using the mAP (mean average accuracy). Using the mAP as a reference, the data processing performance of the object recognition AI performance test is scored, and the terminal device's score in the object recognition AI performance test is calculated by combining the model loading performance score. Specifically:

[0155] A single-item test of AI object recognition performance was conducted. The terminal device displayed the output results for each unit of data to the user and recorded the test output results. After the test, the performance score of the terminal device in this test was calculated. Object recognition technology can help terminal devices find specific objects (such as people, animals, and plants) in images, and can also provide reference data for the camera to automatically adjust shooting parameters, thereby enabling the terminal device to obtain a better shooting experience. The purpose of this test is to evaluate the AI ​​object recognition performance of the terminal device. The AI ​​neural network model used in this test is MobileNet SSD. This model can recognize 90 categories of objects in the COCO dataset, find all target objects in the image, determine their category and location, and output recognition accuracy information. This test uses mAP (Mean Average Precision) to evaluate the accuracy level of the terminal device when processing this model, and uses this data as a reference to score the data processing performance of the terminal device when performing AI object recognition tasks. The score of the terminal device in this single-item AI performance test is calculated by combining the model loading performance score.

[0156] Next, we will introduce the specific calculation method of mAP accuracy. First, let's introduce the metrics involved in mAP calculation: IoU (Intersection-over-Union), which is a standard for measuring the accuracy of detecting corresponding objects in a specific dataset. IoU represents the overlap rate between the candidate bounding boxes output by the AI ​​model and the ground truth bounding boxes, that is, the ratio of their intersection to their union. The higher the correlation, the higher the value. Ideally, the IoU value is 1 when there is complete overlap; True positives (TP): The number of instances correctly classified as positive, that is, the number of instances that are actually positive and were classified as positive by the classifier (number of samples); False positives (FP): The number of instances incorrectly classified as positive, that is, the number of instances that are actually negative but were classified as positive by the classifier; False Negatives (FN): The number of instances incorrectly classified as negative, i.e., instances that are actually positive but were classified as negative by the classifier. After processing an image, the object detection algorithm outputs the confidence score for all supporting sample categories. The confidence score describes the probability that the sample is a positive sample. For example, there is a 99% probability that sample A is a positive example and a 1% probability that sample B is a positive example. By choosing an appropriate threshold, such as 50%, the detection output is divided. Instances with a probability greater than 50% are considered positive examples, and those less than 50% are considered negative examples. This yields a set of positive samples below the 50% threshold. Then, based on this set of positive samples, an IoU threshold of 0.5 is set. Instances with an IoU greater than this threshold are considered TP, and others are considered FP. Finally, the number of true positive samples in the test samples is subtracted from the number of TP to obtain FN.

[0157] Precision, calculated as the number of actual positive samples in the predicted sample divided by the total number of positive samples, is expressed as Precision = TP / (TP + FP). Recall, calculated as the number of actual positive samples in the predicted sample divided by the total number of predicted samples, is expressed as Recall = TP / (TP + FN). Generally, a higher recall rate corresponds to a lower precision. AP, or Average Precision, is the average AP value calculated over multiple validation sets, serving as a metric for measuring detection accuracy in this single test. The PR curve is a two-dimensional curve with Precision and Recall as the vertical and horizontal axes. It is plotted by selecting different confidence thresholds to represent the precision and recall rates. Since higher precision corresponds to lower recall, when recall reaches 1, it corresponds to the lowest probability score of the positive samples. The minimum precision value is calculated by dividing the number of positive samples by the total number of samples greater than or equal to this threshold. The area enclosed by the PR curve is the AP value. Generally speaking, the better the classifier, the higher the AP value. In object detection, a PR curve can be drawn for each class based on Precision and Recall. AP is the area under the curve, and mAP is the average AP of all classes.

[0158] ②In cases where the AI ​​performance test includes a human pose recognition test, the execution process of the human pose recognition AI performance test includes:

[0159] The AI ​​performance test for human posture recognition is run, using key points of the human body as posture recognition feature points. The accuracy of the key points is determined by their similarity, thus evaluating the AI ​​human posture recognition capability of the terminal device. Specifically:

[0160] A single-item test of AI performance in human pose recognition is conducted. The terminal device displays the output results for each unit of data to the user and records the test output results. After the test, the performance score of the terminal device in this test is calculated. Human pose recognition technology finds the posture of a person in an image or video by identifying the position of key human body parts. In VR (Virtual Reality) and AR (Augmented Reality) applications, it can help terminal devices quickly identify human-related information. This single-item test mainly evaluates the AI ​​human pose recognition capability of the terminal device. The test uses 50 test data from the COCO Keypoints dataset. The PoseNet model used can recognize 17 key human body parts, including the nose, eyes, ears, shoulders, elbows, and legs. After calculation, the corresponding human body part number, position, and accuracy are output. This test primarily uses the Object Keypoint Similarity (OKS) metric to calculate the Mean Average Precision (mAP), and then uses this mAP value to evaluate the accuracy performance of the terminal device when processing the model. This mAP value is used as a reference to score the data processing performance of the terminal device in performing AI human pose recognition, and is combined with the model loading performance score to calculate the terminal device's score in this AI performance test. The calculation method for Object Keypoint Similarity (OKS) will be introduced next.

[0161] Keypoint similarity (OKS) is used to describe the accuracy of keypoints obtained in the output results in human pose recognition tasks. Its calculation formula is shown in formula (6).

[0162]

[0163] Where p represents a person in the truth value, p i Let i represent the i-th key point of the p-th person.

[0164] Let represent the Euclidean distance between the i-th keypoint of the p-th person in the true value and the i-th keypoint in the output result.

[0165] The formula for calculating Euclidean distance is shown in formula (7).

[0166]

[0167] In formula (7) (x' i ,y' i () represents the detection coordinates of the i-th keypoint in the output. Let be the coordinates of the i-th key point of the p-th person in the true value.

[0168] v pi This represents the visibility of the i-th keypoint of the p-th person. A visibility of 1 indicates that the keypoint is unobstructed and has been labeled.

[0169] S p Let represent the scale factor for the p-th person, which is the square root of the area of ​​the detection box for the p-th person, and its calculation formula is: w and h are the width and height of the detection box for the p-th person, respectively.

[0170] σ i σ represents the normalization factor for the keypoint with id i. This factor is calculated by comparing the standard deviation between the manually labeled ground truth keypoints and the true values ​​across all sample sets. The larger the σ, the more difficult it is to label this type of keypoint.

[0171] δ(*) means that if condition * is true, then δ(*) = 1, otherwise δ(*) = 0. It is used to determine whether a key point is a point that has been marked in the truth value.

[0172] OKS is used to determine the similarity of key points of a identified person. If there are M targets in the test image (the target is a person), and the model outputs test results for N targets, then the similarity between the key points of each of the M targets and the N results output by the model will be calculated. Finally, an M-row N-column similarity matrix will be obtained. The position (i, j) in the matrix represents the OKS similarity between the i-th person in the ground truth and the j-th person predicted by the algorithm. The maximum value in each row of the matrix is ​​used as the OKS similarity value corresponding to the i-th person.

[0173] Based on the OKS similarity matrix above, we already know the OKS scores of all targets (targets appearing in the ground truth) in a certain image. The test set contains many images, and each image contains many targets. At this point, AP (Average Precision) is used to measure the accuracy of the AI ​​model across all test images. Calculating AP requires a given threshold t. If the current OKS is greater than t, it means the skeleton of the person has been successfully detected correctly. If it is less than t, it indicates a detection failure, false positive, or missed detection. Therefore, for all OKS values, we count the number of values ​​greater than t and calculate the ratio of these values ​​to all OKS values. For example, if there are 100 OKS values, and 30 of them are greater than the threshold t, then the AP value is 30 / 100 = 0.3. mAP (mean Average Precision) is the average AP value. Specifically, it is calculated by giving different thresholds t, calculating the AP for each threshold, and then finding the average of all AP values.

[0174] ③ In the case where the AI ​​performance test includes a speech keyword recognition test, the execution process of the speech keyword recognition AI performance test includes:

[0175] The speech keyword recognition AI performance test is run. The speech recognition accuracy rate is used to evaluate the performance level of the terminal device in recognizing speech keywords. The speech recognition accuracy rate is used as a reference to score the data processing performance of the speech keyword recognition AI performance test. The score of the terminal device in the speech keyword recognition AI performance test is calculated by combining the model loading performance score. Specifically:

[0176] A single-item test of AI performance for speech keyword recognition was conducted. The terminal device displayed the output results of each unit of data to the user and recorded the test output results. After the test, the performance score of the terminal device in this test was calculated. Currently, many terminal devices have intelligent voice assistants. However, the intelligent voice assistants on the terminal devices usually require the user to say the corresponding keywords before they are activated. The recognition function of such speech keywords usually requires the help of AI technology. This test is mainly used to evaluate the performance level of the terminal device in recognizing speech keywords. The AI ​​model used in this test can recognize the basic speech of 10 different words, such as "yes", "no", "up", "down", "left", "right", etc. The input test data is an audio segment with a duration of 1 second. The AI ​​model can analyze the audio information and output the words in the audio. After the test is completed, the program will calculate the speech recognition accuracy of this test. The formula for calculating the speech recognition accuracy is shown in formula (8).

[0177]

[0178] In formula (8), Acuraccy represents the speech recognition accuracy, n represents the number of incorrectly recognized samples, and t represents the total number of test data. This method uses this accuracy data as a reference to score the data processing performance of the terminal device when performing AI speech recognition tasks, and combines it with the model loading performance score to calculate the terminal device's score in this AI performance test.

[0179] ④ When the AI ​​performance test includes a face recognition test, the execution process of the face recognition AI performance test includes:

[0180] The facial recognition AI performance test was run to evaluate the performance of the terminal device in processing AI facial recognition tasks based on the facial recognition accuracy.

[0181] Using the facial recognition accuracy rate as a reference, a score is assigned to the data processing performance of the facial recognition AI performance test. This score is then combined with the model loading performance score to calculate the terminal device's overall score in the facial recognition AI performance test. Specifically:

[0182] A single performance test of AI face recognition was run. The terminal device displayed the output results of each unit of data to the user and recorded the test output results. After the test, the performance score of the terminal device in this test was calculated. AI face recognition is a biometric application that extracts and recognizes facial feature information through AI algorithms. This technology has become a method for unlocking a large number of terminal devices. This test can evaluate the performance of the terminal device in processing AI face recognition tasks. The test is based on the ssd_mobilenet model, which can recognize and classify the faces of 62 celebrities in the LFW dataset. After processing, the AI ​​model will output the face location, celebrity name, and recognition accuracy information. After the test, the program will calculate the face recognition accuracy of this test. The formula for calculating the face recognition accuracy is shown in formula (9).

[0183]

[0184] In formula (9), Acuraccy is the face recognition accuracy, n is the number of incorrectly identified samples, and t is the total number of test data. This method uses this accuracy data as a reference to score the data processing performance of the terminal device when performing AI face recognition tasks, and combines it with the model loading performance score to calculate the terminal device's score in this AI performance test.

[0185] ⑤ In the case where the AI ​​performance test includes a semantic segmentation test, the execution process of the semantic segmentation AI performance test includes:

[0186] Run the single-item test of the semantic segmentation AI, and evaluate the performance of the terminal device in processing AI semantic segmentation tasks by using the average intersection-union ratio (AUC) metric.

[0187] The average intersection-union ratio (OCR) is used as a reference to score the data processing performance of the semantic segmentation AI performance test, and the score of the terminal device in the semantic segmentation AI performance test is calculated by combining the model loading performance score. Specifically:

[0188] A semantic segmentation AI performance test is run. The terminal device displays the output results for each unit of data to the user and records the test output results. After the test, the performance score of the terminal device in this test is calculated. Speech segmentation technology can map each pixel in an image to a category, such as "person," "object," or "background." In VR / AR applications, it can help the terminal device identify foreground or background regions in an image, so as to segment and occlude parts of the image. This test is based on the DeepLab model. DeepLab can identify 21 categories of information, including "background," in the VOC dataset. After completing the calculation, the model outputs the category information of each pixel in the original image. After the test is completed, the program uses MIoU (Mean Intersection-over-Union) to evaluate the accuracy performance of the terminal device when processing the model. This data is used as a reference to score the data processing performance of the terminal device when performing AI semantic segmentation tasks. The score of the terminal device in this AI performance test is calculated by combining the model loading performance score. The specific calculation method of MIoU will be introduced next.

[0189] The semantic segmentation algorithm used in this test has an input resolution of 257x257, capable of detecting 21 different categories. The output data is a 257x257 matrix, with each matrix cell representing the category to which the pixel belongs. MIoU is a metric used to evaluate the accuracy of semantic segmentation algorithms. MIoU calculation requires a confusion matrix. After processing an image, this method creates a 21x21 matrix. The vertical axis represents the category contained in the ground truth matrix, and the horizontal axis represents the predicted category. By comparing the values ​​of each pixel in the output matrix with the values ​​of each pixel in the ground truth matrix, the confusion matrix for this AI performance test is calculated. For example, if a pixel corresponds to category 4 in the ground truth matrix, but the output pixel is classified as category 3, then the value in the 4th row and 3rd column of the confusion matrix is ​​incremented by 1. Therefore, the values ​​in the diagonal elements of the confusion matrix represent the number of correctly predicted categories. After obtaining the confusion matrix, first calculate the IoU (Intersection-over-Union), which is the crossover-union ratio. The formula for calculating IoU is shown in formula (10).

[0190]

[0191] In formula (10), IoU i V represents the IoU value for the i-th class. i Let R be the value in the i-th row and i-th column of the confusion matrix. i C is the sum of all elements in the i-th row of the confusion matrix. iThe MIoU is the sum of all elements in the i-th column of the confusion matrix. In this test method, there are 21 classes of data, and the average of the IoU values ​​of these 21 classes is the MIoU. This method will use this MIoU accuracy data as a reference to score the data processing performance of terminal devices when performing AI semantic segmentation tasks.

[0192] ⑥ When the AI ​​performance test includes an image classification test, the execution process of the image classification AI performance test includes:

[0193] Run the image classification AI performance test to evaluate the performance of the terminal device in processing AI image classification tasks based on the image classification accuracy.

[0194] Using the image classification accuracy as a reference, a score is assigned to the data processing performance of the image classification AI performance test, and this score is combined with the model loading performance score to calculate the terminal device's score in the image classification AI performance test. Specifically:

[0195] This test performs a single-item performance test on AI image classification. The terminal device displays the output results for each unit of data to the user and records the test results. After the test, the terminal device's performance score is calculated. Image classification technology helps terminal devices identify people, objects, or locations in images, allowing for targeted image optimization. This test primarily evaluates the performance of the terminal device in performing AI image classification tasks. The MobileNetV2 model used in this test can classify 1000 classes of images from the ImageNet dataset. This test uses 50 images from ImageNet as test data. After performing calculations on the test data, the model outputs classification information for the image content. This test uses TOP5 accuracy to evaluate the accuracy level of the terminal device when processing the model, and uses this data as a reference to score its data processing performance in performing AI image classification tasks. Next, we will introduce the calculation method of TOP5 accuracy. After processing an image, the image classification algorithm will output multiple predicted categories and their corresponding probabilities. When calculating TOP5 accuracy, the top five categories in the output probability values ​​of each test image are compared with the true value of the image. If one of the categories matches the true value, the image is considered to be correctly classified. After processing all test data, the number of correctly identified images is counted. The formula for calculating the TOP5 accuracy of image classification is shown in formula (11).

[0196]

[0197] In formula (11), p is the number of correctly identified images, and n is the total number of test images. This method uses this accuracy data as a reference to score the data processing performance of the terminal device when performing the A image classification task, and combines the model loading performance score to calculate the terminal device's score in this AI performance test.

[0198] ⑦ When the AI ​​performance test includes a super-resolution test, the execution process of the super-resolution AI performance test includes:

[0199] Run the single test of super-resolution AI performance and evaluate the AI ​​super-resolution computing performance of the terminal device through the peak signal-to-noise ratio index;

[0200] Using the peak signal-to-noise ratio (PSNR) as a reference, the data processing performance of the super-resolution AI performance test is scored, and the score of the terminal device in the super-resolution AI performance test is calculated by combining the model loading performance score. Specifically:

[0201] A single-item test of super-resolution AI performance was conducted. The terminal device displayed the output results for each unit of data to the user and recorded the test output results. After the test, the performance score of the terminal device in this test was calculated. Super-resolution technology enables terminal devices to reconstruct corresponding high-resolution images from low-resolution images, effectively enhancing the image quality of photos. This test mainly evaluates the AI ​​super-resolution computing performance of the terminal device. The AI ​​neural network model used in the test is ESRGAN, which generates details in images and can generate super-resolution images with 4 times higher pixels based on low-resolution images. The 20 test data in this test came from the DIV2K dataset. Before the test, the original data was converted into low-resolution data using a downsampling method. During the test, the low-resolution data was used for testing. After the test, the high-resolution image output by the model was compared with the original image, and the PSNR (Peak Signal to Noise Ratio) was calculated. This data was used as a reference to score the data processing performance of the terminal device when performing AI super-resolution tasks. The score of the terminal device in this single-item AI performance test was calculated by combining the model loading performance score.

[0202] Formula (12) is the formula for calculating PSNR.

[0203]

[0204] MAX in formula (12) I is the maximum value of a single pixel in the image. MSE is the mean squared error, which reflects the degree of difference between the estimator and the estimated quantity. Formula (13) is the formula for calculating MSE.

[0205]

[0206] In formula (13), I and K are the pixel values ​​of the i-th row and j-th column of the original image and the output image, respectively, and m and n are the horizontal and vertical resolutions.

[0207] After calculating the PSNR of the RGB pixels, the values ​​are summed and averaged. If the average is greater than 40, the accuracy is 1. If it is less than 40, the accuracy is PSNR / 40.

[0208] In summary, the performance evaluation method for terminal devices provided in this invention includes seven different AI performance tests: face recognition, speech keyword recognition, image classification, object recognition, super-resolution, human pose recognition, and semantic segmentation. Each test utilizes AI neural network models with varying computational complexity and different test datasets. Finally, this performance evaluation method comprehensively assesses the AI ​​performance of the terminal device by combining the results of these seven AI performance tests. This method can test the actual performance of the terminal device when handling these different AI applications, providing a more comprehensive evaluation of the terminal device's AI performance and thus better helping users choose a terminal device with suitable AI performance based on their needs.

[0209] Meanwhile, this invention provides an AI performance evaluation index for terminal devices. This index evaluates both AI model loading performance and AI data processing performance. AI model loading performance comprehensively assesses the performance of the terminal device when loading an AI model, while AI data processing performance evaluates the performance of the terminal device when using an AI neural network model to process test data. The score for each test item is calculated by combining the AI ​​model loading performance score and the AI ​​data processing performance score. This performance evaluation index can evaluate the AI ​​performance of a terminal device from both AI model loading performance and AI data processing performance perspectives. The evaluation content covers the entire process of AI application operation, providing a more comprehensive reflection of the terminal device's performance level throughout the entire AI application process.

[0210] Meanwhile, the overall performance score of this invention is the average of the scores of seven individual AI performance tests. Each individual AI performance test score mainly consists of an AI model loading performance score and an AI data processing performance score. The AI ​​model loading performance score is mainly calculated using AI model loading latency data combined with the number of AI model parameters. The AI ​​data processing performance score is mainly calculated using AI model computation latency data, AI model accuracy data, combined with the number of MAC operations of the AI ​​model. When calculating the individual AI performance test scores, a score ratio weight is designed for each individual test to describe the degree of impact of AI model loading performance and AI data processing performance on the overall AI performance of the terminal device. This performance evaluation scoring calculation method can describe the AI ​​performance of the terminal device from two aspects: AI model loading performance and AI data processing performance. This method combines the number of AI model parameters and AI computational complexity to calculate the individual AI performance test scores, making the proportion of each individual test score in the overall AI performance score more balanced. This scoring calculation method can more accurately describe the performance of the terminal device in each individual test.

[0211] Compared to existing technologies, the evaluation method of this invention tests more comprehensively and extensively, employing multiple AI performance tests to evaluate the AI ​​performance of terminal devices. This allows for testing the actual performance of the device when handling different AI applications, providing a more complete assessment of the terminal device's AI performance and helping users better select the appropriate AI-performance terminal device based on their needs. The performance evaluation indicators proposed in this invention evaluate the AI ​​performance of terminal devices from two aspects: AI model loading performance and AI data processing performance. The evaluation content covers the entire process of AI application operation, providing a more comprehensive reflection of the device's performance level throughout the entire AI application process. The performance evaluation score calculation method proposed in this invention describes the AI ​​performance of terminal devices from both AI model loading performance and AI data processing performance perspectives. This method combines the number of AI model parameters and the complexity of AI computation to calculate the score for each AI performance test, resulting in a more balanced weighting of each test score in the overall AI performance score. This scoring calculation method provides a more accurate description of the terminal device's performance in each individual test.

[0212] Based on the above method embodiments, this invention provides a performance evaluation device 400 for terminal devices, such as... Figure 3 As shown, the device may include:

[0213] The receiving module 410 is used to receive user instructions to start the AI ​​performance evaluation process;

[0214] The control module 420 is used to control and execute each AI performance test and determine the score for each AI performance test. Specifically, for different AI performance tests, it controls the loading of the AI ​​neural network model, controls the calculation of the AI ​​neural network model, and calculates the results of each test. After completing all AI performance tests, it summarizes the test output results and scores of all AI performance tests.

[0215] Loading module 430 is used to prepare the test data required for each AI performance test, including loading the test dataset and AI neural network model required for each test, and completing the initialization process of the AI ​​neural network model.

[0216] The calculation module 440 is used to complete the AI ​​calculation process for each AI performance test, including inputting each unit of test data into the AI ​​neural network model, completing the calculation process of the corresponding AI algorithm for different AI test contents, and outputting the calculation result for each unit of data.

[0217] The statistics module 450 is used to collect test data for each AI performance test, including the time required to load the AI ​​neural network model, the time required to process all test data, the AI ​​calculation result for each unit of data, and to calculate the AI ​​performance test score based on the above statistics.

[0218] The evaluation module 460 is used to determine the overall AI performance score by averaging the scores of each individual AI performance test, and to display all test results to the user, including the test output results of all individual AI performance tests, the scores of individual AI performance tests, and the overall AI performance score.

[0219] This application also provides a computer-readable non-volatile storage medium for storing program code and data, which are used to implement the AI ​​performance evaluation methods described in the foregoing embodiments.

[0220] To implement the embodiments of the present invention, this application also provides a computer program product or computer program. The computer program product or computer program may contain one or more computer instructions, which are stored in a readable non-volatile storage medium. A computer reads the computer instructions from the non-volatile storage medium and processes these computer instructions through a processor to implement the AI ​​performance evaluation method described in the embodiments of this application. The code of the computer program may be written in one or more computer languages. These computer languages ​​may be object-oriented programming languages, such as Java and C++, or procedural programming languages, such as C.

[0221] This invention also provides a terminal device for implementing the above-described AI performance evaluation method. Figure 4 The diagram shown is a block diagram of a portion of the structure of the terminal device provided in the embodiments of this application.

[0222] refer to Figure 4 The terminal device 500 includes the following components: an I / O component 501, a processor 502, a storage system 505, a display 508, and a bus 509. The I / O component 501 is used to connect and communicate with external devices. The storage system 505 is used to store relevant data of the computer program of this application example and the execution of each module. The bus 509 is used to connect all components and realize communication between components. The display 508 is used to display the test results to the user. The processor 502 is used to control and process the computer program described in this application example. When the processor executes the computer program described in this patent, the AI ​​performance evaluation method of this application is realized.

[0223] Storage system 505 is used to store data related to the computer program of this application and various modules. It consists of memory 506 and main memory 507. Memory 506 can store and read / write data, application programs, and other related files required for running the computer program indefinitely. It may include non-volatile memory, such as SSD hard disk, HDD hard disk, flash memory, etc., and may also include optical storage devices and disk storage devices. Main memory 507 is short-term memory. Compared with memory 506, it has a faster read / write speed and is used to quickly obtain the data, program, and related files required by the program during the execution of the computer program. When the terminal device is turned off or the power is cut off, the data in memory 507 will not be retained. Main memory 507 may include volatile memory or dynamic random access memory.

[0224] The IO component 501 is used to enable the terminal device 500 to connect and communicate with other external devices 510. The external device 510 can be a keyboard, mouse, pointing device, etc., or a device that can communicate with the terminal device 500.

[0225] Display 508 is used to display the intermediate results generated during the AI ​​performance evaluation program test to the user. After the AI ​​performance test is completed, it is used to display all the test results to the user, including the test output results of all AI performance individual tests, the scores of AI performance individual tests, and the overall AI performance score.

[0226] Bus 509 is used to connect all components in the terminal device and realize the communication function between components. Bus 509 can be one or more structures, such as memory bus, I / O bus, processor bus.

[0227] The processor 502 is used to process the computer program in this application example, and includes a controller 503 and an AI computing unit 504. The controller 503 is mainly used to control all components of the terminal device, and may include controlling the storage system 505 to load the computer program and related data, controlling the I / O component 501 to communicate with other external devices, controlling the AI ​​computing unit 504 to complete the calculation of the AI ​​algorithm, and controlling the display 508 to display the output results of the test program. The AI ​​computing unit 504 is used to complete and accelerate the calculation process of the AI ​​algorithm in the computer program. It can be a CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array), NPU (Neural Network Processing Unit), MLU (Machine Learning Processing Unit), ASIC (Application Specific Integrated Circuit), DSP (Digital Signal Processor), or other programmable computing devices. It can also be a combination of one or more of the above devices. The processor 502 executes all individual AI performance tests, data processing, and result display in the computer program running in the storage system 505, thereby realizing the AI ​​performance evaluation method described in this application.

[0228] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for evaluating the performance of a terminal device, characterized in that, include: Receive user instructions to initiate the AI ​​performance evaluation process; Control and execute each AI performance individual test, and determine the score for each of the AI ​​performance individual tests, including: Load the test dataset for the AI ​​performance single test; The AI ​​neural network model corresponding to the single AI performance test is loaded from the storage system of the terminal device into memory to complete the initialization process of the AI ​​neural network model, and the model loading time is recorded. The loaded AI neural network model is used to process multiple test data in the test dataset to obtain the test output result information for each test data and the average processing latency data of the terminal device. The test output results of each test data are compared with the true value of the test data, and the accuracy test result of the terminal device is calculated according to the accuracy calculation formula corresponding to the single AI performance test. Using the accuracy test results of the terminal device as a reference, and combining the average processing latency data of the terminal device, the data processing performance score of the AI ​​performance single test is calculated. Calculate the model loading performance score based on the model loading time; By combining the data processing performance score and the model loading performance score, the score of the AI ​​performance single test obtained by the terminal device in the AI ​​performance single test is calculated; The formula for calculating the score of the AI ​​performance test is as follows: S i =P i +T i *w i Among them, S i P represents the performance score of the terminal device in the i-th AI performance test. i T represents the data processing performance score of the terminal device in the i-th AI performance test. i w is the model loading performance score of the terminal device in the i-th AI performance test. i The score ratio weight of the i-th AI performance single test; The formula for calculating the data processing performance score is as follows: Among them, P i MAC is the data processing performance score of the terminal device in the i-th AI performance test. i L represents the number of multiplication and addition operations included in the AI ​​neural network model in the i-th AI performance single test; i It is the average processing latency data of the terminal device in the i-th AI performance single test, i.e., the average processing latency data; The formula for calculating the model loading performance score is as follows: Among them, T i M represents the model loading performance score of the terminal device in the i-th AI performance test. i s represents the number of model parameters of the AI ​​neural network model in the i-th AI performance single test, that is, the amount of parameter data for the model weights in the AI ​​neural network model; i The time taken by the terminal device to complete the AI ​​neural network model loading and model initialization process in the i-th AI performance single test is the model loading time. Based on the scores of all the individual AI performance tests, a comprehensive AI performance score for the terminal device is determined, and the performance of the terminal device is evaluated.

2. The method according to claim 1, characterized in that, The AI ​​performance tests include any one of the following: face recognition test, voice keyword recognition test, image classification test, object recognition test, super-resolution test, human pose recognition test, and semantic segmentation test.

3. The method according to claim 2, characterized in that, When the AI ​​performance test includes a human pose recognition test, the execution process of the human pose recognition AI performance test includes: The AI ​​performance test for human posture recognition is run, and key points of the human body are identified as posture recognition feature points. The accuracy of the key points is determined by the similarity of the key points, and the AI ​​human posture recognition capability of the terminal device is evaluated.

4. The method according to claim 3, characterized in that, The formula for calculating the keypoint similarity (OKS) is as follows: Where p represents a person in the true values ​​of the test data, p i This represents the i-th key point of the p-th person; Let represent the Euclidean distance between the i-th keypoint of the p-th person and the i-th keypoint in the output result, and its calculation formula is: Among them, (x' i ,y' i () represents the detection result coordinates of the i-th key point in the test output information. Let be the coordinates of the i-th keypoint of the p-th person in the true value; v pi Indicate the visibility of the i-th keypoint of the p-th person; S p The scale factor for the p-th person is calculated using the following formula: w and h are the width and height of the detection box for the p-th person, respectively; σ i The normalization factor for the keypoint with id i is determined by the standard deviation between manually labeled and true keypoints in all test datasets. δ(*) means that if condition * is true, then δ(*) = 1, otherwise δ(*) = 0. It is used to determine whether a key point is a point that has been marked in the truth value.

5. The method according to claim 2, characterized in that, When the AI ​​performance test includes a speech keyword recognition test, the execution process of the speech keyword recognition AI performance test includes: Run the AI ​​performance test for voice keyword recognition, and evaluate the performance level of the terminal device in recognizing voice keywords by the voice recognition accuracy, including: The formula for calculating the speech recognition accuracy is as follows: Where Acuraccy_0 is the speech recognition accuracy, n0 is the number of samples that were misrecognized, and t0 is the total number of test data. Using the speech recognition accuracy as a reference, the data processing performance of the speech keyword recognition AI performance single test is scored, and the score of the terminal device in the speech keyword recognition AI performance single test is calculated by combining the model loading performance score.

6. The method according to claim 2, characterized in that, When the AI ​​performance test includes a face recognition test, the execution process for the face recognition AI performance test includes: The face recognition AI performance single test is run to evaluate the performance of the terminal device in processing AI face recognition tasks based on the face recognition accuracy, including: The formula for calculating the accuracy of facial recognition is as follows: Where Acuraccy_1 is the face recognition accuracy, n1 is the number of incorrectly identified samples, and t1 is the total number of test data. Using the face recognition accuracy as a reference, the data processing performance of the face recognition AI performance test is scored, and the score of the terminal device in the face recognition AI performance test is calculated by combining the model loading performance score.

7. The method according to claim 2, characterized in that, In the case where the AI ​​performance test includes a semantic segmentation test, the execution process of the semantic segmentation AI performance test includes: Run the single-item test of the semantic segmentation AI, and evaluate the performance of the terminal device in processing AI semantic segmentation tasks by using the average intersection-union ratio (AUC) metric. The average intersection-union ratio is used as a reference to score the data processing performance of the semantic segmentation AI performance test, and the score of the terminal device in the semantic segmentation AI performance test is calculated by combining the model loading performance score.

8. The method according to claim 7, characterized in that, The evaluation of the performance of the terminal device in processing AI semantic segmentation tasks using the average intersection-union ratio (AUC) includes: Load the test dataset for the semantic segmentation AI performance single test, construct an output matrix of the categories of the true values ​​and the categories of the predicted values ​​in the test dataset, and generate a confusion matrix for the semantic segmentation AI performance single test by comparing the value of each pixel in the output matrix with the value of each pixel in the true value matrix. After obtaining the confusion matrix, the Intersection over Union (IoU) is calculated. The formula for calculating IoU is: Among them, IoU i V represents the IoU value for the i-th class. i Let R be the value in the i-th row and i-th column of the confusion matrix. i C is the sum of all elements in the i-th row of the confusion matrix. i This is the sum of all elements in the i-th column of the confusion matrix; The average IoU value of the test datasets for all categories used in the semantic segmentation AI performance single test is the MIoU (mean intersection-union ratio).

9. The method according to claim 2, characterized in that, In the case where the AI ​​performance test includes an image classification test, the execution process for the image classification AI performance test includes: The image classification AI performance test is run to evaluate the performance of the terminal device in processing AI image classification tasks based on the image classification accuracy, including: Load the test dataset, i.e., the test image set, for the single performance test of the image classification AI; After processing a test image, multiple predicted categories and their corresponding probabilities are output. As needed, the first few categories in the output probability values ​​for each test image are compared with the ground truth value for that image. If any category matches the ground truth value, the image is considered correctly classified. After processing all test data, the number of correctly identified images is counted. The formula for calculating the image classification accuracy is: Where p is the number of correctly identified images, and n is the total number of test images; Using the image classification accuracy as a reference, the data processing performance of the image classification AI performance test is scored, and the score of the terminal device in the image classification AI performance test is calculated by combining the model loading performance score.

10. The method according to claim 2, characterized in that, When the AI ​​performance test includes a super-resolution test, the execution process of the super-resolution AI performance test includes: Run the single test of super-resolution AI performance and evaluate the AI ​​super-resolution computing performance of the terminal device through the peak signal-to-noise ratio index; Using the peak signal-to-noise ratio as a reference, the data processing performance of the super-resolution AI performance test is scored, and the score of the terminal device in the super-resolution AI performance test is calculated by combining the model loading performance score.

11. The method according to claim 10, characterized in that, Load the test dataset, i.e., the test image set, for the single test of super-resolution AI performance; Before testing, the original test data was converted to low-resolution data using a downsampling method. During testing, the low-resolution data was used. After testing, the high-resolution image output by the model was compared with the original image to obtain the peak signal-to-noise ratio (PSNR). The formula for calculating the PSNR is as follows: Among them, MAX I This represents the maximum value of a single pixel in the image. MSE, or Mean Squared Error, is used to reflect the degree of difference between the estimator and the estimated quantity. The formula for calculating MSE is: Where I and K are the pixel values ​​in the i-th row and j-th column of the original test image and the output image, respectively, and m and n are the horizontal and vertical resolutions.

12. The method according to claim 2, characterized in that, When the AI ​​performance test includes an object recognition test, the execution process for the object recognition AI performance test includes: Run the single test of the AI ​​performance of object recognition, and evaluate the AI ​​object recognition computing performance of the terminal device by the mAP average accuracy. Using the mAP average accuracy as a reference, the data processing performance of the object recognition AI performance single test is scored, and the score of the terminal device in the object recognition AI performance single test is calculated by combining the model loading performance score.

13. The method according to any one of claims 1-12, characterized in that, The determination of the overall AI performance score of the terminal device based on the scores of all the individual AI performance tests includes: The overall AI performance score of the terminal device is calculated by taking the weighted average of the scores of each individual AI performance test.

14. A performance evaluation device for a terminal device, characterized in that, The performance evaluation method for the terminal device according to any one of claims 1-13 includes: The receiving module is used to receive user commands to initiate the AI ​​performance evaluation process; The control module, used to control and execute each AI performance test individually, and to determine the score for each of the AI ​​performance tests individually, includes: Load the test dataset for the AI ​​performance single test; The AI ​​neural network model corresponding to the single AI performance test is loaded from the storage system of the terminal device into memory to complete the initialization process of the AI ​​neural network model, and the model loading time is recorded. The loaded AI neural network model is used to process multiple test data in the test dataset to obtain the test output result information for each test data and the average processing latency data of the terminal device. The test output results of each test data are compared with the true value of the test data, and the accuracy test result of the terminal device is calculated according to the accuracy calculation formula corresponding to the single AI performance test. Using the accuracy test results of the terminal device as a reference, and combining the average processing latency data of the terminal device, the data processing performance score of the AI ​​performance single test is calculated. Calculate the model loading performance score based on the model loading time; By combining the data processing performance score and the model loading performance score, the score of the AI ​​performance single test obtained by the terminal device in the AI ​​performance single test is calculated; The formula for calculating the score of the AI ​​performance test is as follows: S i =P i +T i *w i Among them, S i P represents the performance score of the terminal device in the i-th AI performance test. i T represents the data processing performance score of the terminal device in the i-th AI performance test. i w is the model loading performance score of the terminal device in the i-th AI performance test. i The score ratio weight of the i-th AI performance single test; The formula for calculating the data processing performance score is as follows: Among them, P i MAC is the data processing performance score of the terminal device in the i-th AI performance test. i L represents the number of multiplication and addition operations included in the AI ​​neural network model in the i-th AI performance single test; i It is the average processing latency data of the terminal device in the i-th AI performance single test, i.e., the average processing latency data; The formula for calculating the model loading performance score is as follows: Among them, T i M represents the model loading performance score of the terminal device in the i-th AI performance test. i s represents the number of model parameters of the AI ​​neural network model in the i-th AI performance single test, that is, the amount of parameter data for the model weights in the AI ​​neural network model; i The time taken by the terminal device to complete the AI ​​neural network model loading and model initialization process in the i-th AI performance single test is the model loading time. The evaluation module is used to determine the overall AI performance score of the terminal device based on the scores of all the individual AI performance tests, and to evaluate the performance of the terminal device.

15. A storage medium, characterized in that, A computer program for storing a method for performing a performance evaluation of a terminal device according to any one of claims 1-13.

16. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the performance evaluation method for the terminal device according to any one of claims 1-13.

Images