Operator performance evaluation method, device, electronic device and storage medium

Through automated deployment and unified data format operator performance evaluation methods, the problems of high manual interference and testing costs in AI operator performance evaluation are solved, the evaluation accuracy and efficiency are improved, and it is suitable for a variety of environments and hardware conditions.

CN114564374BActive Publication Date: 2024-11-22BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210157139.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-21
Publication Date
2024-11-22
Estimated Expiration
2042-02-21

AI Technical Summary

Technical Problem

The prior art has problems such as high manual interference, high testing cost and low accuracy in the performance evaluation of AI operators. Especially when deploying and testing under different environments and hardware conditions, manual operation and script development are required, resulting in inefficiency.

Method used

Provide an operator performance evaluation method. By determining the AI ​​operator to be evaluated, automatically deploying and testing, using a unified service deployment strategy and data format, using electronic devices for data acquisition and analysis, generating performance evaluation data, and reducing manual interference , improve the accuracy of evaluation.

Benefits of technology

It realizes automated deployment and testing of AI operator performance evaluation, reduces the evaluation cost and personnel interference factors of testers, improves the accuracy and efficiency of performance evaluation, and is suitable for deployment and testing under different environments and hardware conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114564374B_ABST
    Figure CN114564374B_ABST
Patent Text Reader

Abstract

The present disclosure provides an operator performance evaluation method, device, electronic device and storage medium, which relate to the field of computer technology, and specifically to the field of artificial intelligence technology such as AI development and cloud platform. The specific implementation scheme is: determine the artificial intelligence AI operator to be evaluated; deploy the service of the operator to obtain the operator service; obtain the operator data set; test the operator according to the data set and the operator service to generate the performance evaluation data of the operator. In this way, the evaluation cost of the tester can be reduced, while reducing the interference factor of the personnel and improving the accuracy of the performance evaluation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to the field of artificial intelligence technology such as AI development and cloud platforms, and in particular to an operator performance evaluation method, device, electronic device, and storage medium. Background Art

[0002] Currently, many applications and some public cloud services need to be packaged and developed based on AI (Artificial Intelligence) technology. Therefore, the server needs to deploy an AI operator service cluster for upper-layer business encapsulation to provide external services. In this way, it is necessary to evaluate the performance of each AI operator and generate performance data before cluster capacity planning can be carried out conveniently.

[0003] Traditional AI algorithm performance testing mainly relies on testers to operate stress testing tools to issue stress, and continuously increase the request concurrency to test the extreme pressure of AI operator services. Finally, relevant resource data, success rate data, etc. are counted to form the final performance data. Summary of the invention

[0004] The present disclosure provides an operator performance evaluation method, device, electronic device and storage medium.

[0005] According to one aspect of the present disclosure, there is provided an operator performance evaluation method, comprising:

[0006] Determine the operator to be evaluated;

[0007] Performing service deployment on the operator to obtain the operator service;

[0008] Obtaining a data set of the operator;

[0009] The operator is tested according to the data set and the operator service to generate performance evaluation data of the operator.

[0010] According to another aspect of the present disclosure, there is provided an operator performance evaluation device, comprising:

[0011] A determination module, used to determine the artificial intelligence AI operator to be evaluated;

[0012] A service deployment module, used to perform service deployment on the operator to obtain the operator service;

[0013] An acquisition module, used to acquire a data set of the operator;

[0014] A testing module is used to test the operator according to the data set and the operator service to generate performance evaluation data of the operator.

[0015] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the operator performance evaluation method described in the above-mentioned embodiment.

[0019] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the operator performance evaluation method described in the above-mentioned embodiment.

[0020] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the steps of the operator performance evaluation method described in the above-mentioned embodiment are implemented.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 A flowchart of an operator performance evaluation method provided in an embodiment of the present disclosure;

[0024] Figure 2 A flowchart of another operator performance evaluation method provided by an embodiment of the present disclosure;

[0025] Figure 3 A flowchart of another operator performance evaluation method provided by an embodiment of the present disclosure;

[0026] Figure 4 A flowchart of another operator performance evaluation method provided by an embodiment of the present disclosure;

[0027] Figure 5 A flowchart of another operator performance evaluation method provided by an embodiment of the present disclosure;

[0028] Figure 6 A flowchart of another operator performance evaluation method provided by an embodiment of the present disclosure;

[0029] Figure 7 A flowchart of another operator performance evaluation method provided by an embodiment of the present disclosure;

[0030] Figure 8 A schematic diagram of the time consumption of operator processing provided in an embodiment of the present disclosure;

[0031] Fig. 9 A flowchart of another operator performance evaluation method provided by an embodiment of the present disclosure;

[0032] Fig.10 A schematic diagram of an operator request success rate provided in an embodiment of the present disclosure;

[0033] Fig.11 A flowchart of another operator performance evaluation method provided by an embodiment of the present disclosure;

[0034] Fig.12 A block diagram of an operator performance evaluation device provided by an embodiment of the present disclosure; and

[0035] Fig.13 The present invention is a block diagram of an electronic device for the operator performance evaluation method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0037] The following describes the operator performance evaluation method, device, electronic device, and storage medium of the embodiments of the present disclosure with reference to the accompanying drawings.

[0038] Artificial intelligence is a discipline that studies the use of computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level technical fields and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies include computer vision technology, speech recognition technology, natural language processing technology, as well as deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0039] Cloud computing platform, also known as cloud platform, refers to services based on hardware resources and software resources, providing computing, network and storage capabilities. Cloud computing platforms can be divided into three categories: storage-based cloud platforms that focus on data storage, computing-based cloud platforms that focus on data processing, and comprehensive cloud computing platforms that take into account both computing and data storage processing.

[0040] The operator performance evaluation method provided in the embodiments of the present disclosure may be executed by an electronic device, which may be a PC (Personal Computer), a tablet computer or a PDA, a mobile phone or a server, etc., without any limitation here.

[0041] In the embodiment of the present disclosure, a processing component, a storage component and a driving component may be provided in the electronic device. Optionally, the driving component and the processing component may be integrated, the storage component may store an operating system, an application program or other program modules, and the processing component implements the operator performance evaluation method provided in the embodiment of the present disclosure by executing the application program stored in the storage component.

[0042] Figure 1 A flowchart of an operator performance evaluation method provided in an embodiment of the present disclosure.

[0043] The operator performance evaluation method of the embodiment of the present disclosure can also be executed by the operator performance evaluation device provided by the embodiment of the present disclosure. The device can be configured in an electronic device to determine the artificial intelligence AI operator to be evaluated, and deploy the service of the operator to obtain the operator service, and then obtain the data set of the operator, and test the operator according to the data set and the operator service to generate performance evaluation data of the operator, which can reduce the evaluation cost of the tester, while reducing the interference factor of the personnel and improving the accuracy of the performance evaluation.

[0044] As a possible situation, the operator performance evaluation method of the embodiment of the present disclosure may also be executed on the server side. The server may be a cloud server, and the operator performance evaluation method may be executed on the cloud.

[0045] As another possible scenario, the operator performance evaluation method of the embodiment of the present disclosure may be executed by an operator performance evaluation platform, and the operator performance evaluation platform may be deployed in an electronic device (eg, a server), and the operator performance evaluation platform may be run based on the electronic device.

[0046] like Figure 1 As shown, the operator performance evaluation method may include:

[0047] Step 101, determine the artificial intelligence AI operator to be evaluated.

[0048] It should be noted that the operator described in this embodiment (ie, the artificial intelligence AI operator) may be a service-oriented AI operator.

[0049] In the embodiment of the present disclosure, relevant R&D personnel can send (submit) the service-oriented operator to an electronic device. After receiving and confirming the operator, the electronic device can store the operator in its own storage space for subsequent calling (use).

[0050] Step 102: Deploy the service for the operator to obtain the operator service.

[0051] In the embodiments of the present disclosure, the operator may be automatically deployed for service according to a preset service deployment strategy to obtain the operator service, wherein the preset service deployment strategy may be calibrated according to actual conditions.

[0052] Specifically, after receiving the above-mentioned operator to be evaluated, the electronic device may perform service deployment on the operator according to a preset service deployment strategy to obtain the operator service.

[0053] Step 103: Obtain the operator data set.

[0054] It should be noted that the data set described in this embodiment may be composed of videos, pictures, texts, etc. collected by relevant testers.

[0055] In the disclosed embodiment, the relevant tester may send (submit) the above data set to an electronic device. After receiving the data set, the electronic device may store the data set in its own storage space for subsequent calling (use).

[0056] Specifically, after obtaining the operator service, the electronic device can obtain (call out) the data set of the above operator from its own storage space.

[0057] As a possible scenario, after obtaining the operator service, the electronic device can directly receive the data set of the above operator submitted by the relevant tester.

[0058] Step 104: Test the operator according to the data set and the operator service to generate performance evaluation data of the operator.

[0059] In the disclosed embodiment, after obtaining a data set, the electronic device first encapsulates the data set according to a preset strategy to generate a request instruction set for an operator service, wherein the request instruction set may include multiple request instructions. The electronic device may then test the operator according to the request instruction set and the operator service to generate performance evaluation data for the operator.

[0060] It should be noted that the test described in this embodiment may be a stress test.

[0061] Furthermore, after generating the performance evaluation data of the above-mentioned operator, the electronic device may provide the performance evaluation data of the operator to relevant personnel, for example, by sending it via text message, displaying it on a display screen of the electronic device, sending it via email, etc.

[0062] It should be noted that the performance evaluation data of the operator (AI operator) described in this embodiment is the basis for subsequent large-scale deployment of PaaS (Platform as a Service). For GPU (Graphics Processing Unit) operators, dynamic scheduling will be performed based on performance evaluation data, including decisions such as how much resources to allocate and whether to enable mixed distribution, so as to make full use of the underlying computing resources. It has been used in real online systems, and the scale of deployed GPU cards can be 3000+ cards. Through this stored performance evaluation data information, detailed resource allocation and sufficient mixed distribution are performed on the underlying operators to achieve multiple instances of one card.

[0063] In the disclosed embodiment, the artificial intelligence AI operator to be evaluated is first determined, and the service deployment of the operator is performed to obtain the operator service, and then the operator data set is obtained, and the operator is tested according to the data set and the operator service to generate the performance evaluation data of the operator. In this way, the evaluation cost of the tester can be reduced, while the human interference factor is reduced, and the accuracy of the performance evaluation is improved.

[0064] In related technologies, the obstacles to the automated deployment of operators (e.g., AI operators) mainly include the following:

[0065] 1. Deployment form: Currently, there are physical machines, virtual machines, containers, etc., and the deployment method of each form is different;

[0066] 2. Basic environment: The service needs to rely on libraries to run, such as glibc (C runtime library), operating system kernel version, etc. The Python (computer programming language) service also needs to rely on the Python runtime, dependency packages, etc.;

[0067] 3. GPU hardware dependency: Most AI operator services require GPU and other hardware devices for acceleration, so they are also subject to restrictions such as GPU graphics card driver version, CUDA (Compute Unified Device Architecture) version, CuDNN (GPU acceleration library for deep neural networks) version, etc. At the same time, different models also have restrictions on graphics card models. For example, if a certain AI operator is only compatible with NVIDIA P4 and T4, it will cause startup errors on a machine with an A10 card;

[0068] 4. Service startup method: Different AI operator services have different startup methods. For example, some can be started with just a few commands, while others require some environment variable configuration and service configuration modification before they can be started.

[0069] In order to solve the above problems, in the embodiments of the present disclosure, a unified standard method can be adopted to make each environment meet the standard and finally complete the automation of deployment. The main standards involved are shown in Table 1 below:

[0070]

[0071]

[0072] Table 1

[0073] After meeting the above standards, relevant R&D personnel only need to specify the GPU model when sending (submitting) the service-oriented operator to the electronic device to complete the automated deployment without human intervention, which can effectively reduce human interference factors and increase the deployment speed.

[0074] In order to clearly explain the above embodiment, in one embodiment of the present disclosure, as Figure 2 As shown, the service deployment of the operator to obtain the operator service may include:

[0075] Step 201, obtaining a specified model of a graphics processor GPU of an operator.

[0076] In the disclosed embodiment, relevant R&D personnel may send (submit) the GPU specified model of the operator to the electronic device while sending (submitting) the serviced operator to the electronic device.

[0077] Step 202, determine the deployment strategy according to the GPU specified model.

[0078] It should be noted that the deployment strategy described in this embodiment may be generated (designed) in advance and pre-stored in the storage space of the electronic device to facilitate retrieval and application.

[0079] Step 203: Deploy the operator according to the deployment strategy to obtain the operator service. The deployment strategy can be calibrated according to the actual situation.

[0080] Specifically, when the relevant R&D personnel send (submit) the service-oriented operator to the electronic device, they can also send (submit) the GPU specified model of the operator to the electronic device. After receiving the operator and its corresponding GPU specified model, the electronic device can first determine the deployment strategy from its own storage space according to the GPU specified model, and automatically deploy the service to the operator according to the deployment strategy to obtain the operator service. In this way, the automatic deployment of the operator service can be completed without human intervention, which can effectively reduce human interference factors and improve the deployment speed.

[0081] In related technologies, operators are tested manually by testers. The main reasons are as follows:

[0082] 1. The AI ​​operator service interface formats are different: For the input format, testers need to generate a stress test data set that is adapted to the service for each interface before they can perform stress testing operations; for the output format, testers need to parse different formats to determine whether the request is successful, and thus determine whether it is necessary to continue the next round of stress testing.

[0083] 2. Judgment of the end of stress testing: After the stress testing program is running, it is necessary to count some technical indicators, such as changes in success rate, latency, etc., to determine whether the stress testing needs to be stopped. Currently, this process requires testers to develop statistical scripts, perform manual statistics, and make manual judgments.

[0084] In view of the above two problems, relevant solutions are designed in the embodiments of the present disclosure to optimize:

[0085] 1. Unify the input standards of AI operator services (for example, the standards of request instructions in the following request instruction set), and uniformly adopt the http (Hyper Text Transfer Protocol) format. The specific input fields can be shown in Table 2 below:

[0086]

[0087] Table 2

[0088] 2. Unify the output standards of AI operator services (for example, the standards of the following test data, resource data, and response data), and use the json format. The specific output fields can be shown in Table 3 below:

[0089]

[0090] Table 33. Error codes may also be unified, as shown in Table 4 below:

[0091]

[0092]

[0093] Table 4

[0094] In order to clearly explain the above embodiment, in one embodiment of the present disclosure, as Figure 3 As shown, the operator is tested based on the data set and operator service to generate performance evaluation data for the operator, which may include:

[0095] Step 301: Test the operator according to the data set and the operator service to obtain test data.

[0096] In the disclosed embodiment, after obtaining a data set, the electronic device first encapsulates the data set according to a preset strategy to generate a request instruction set for an operator service, wherein the request instruction set may include multiple request instructions. The electronic device may then test the operator according to the request instruction set and the operator service to generate test data for the operator.

[0097] Step 302: Analyze the test data to obtain performance evaluation data of the operator.

[0098] In the embodiment of the present disclosure, the test data may be analyzed according to a preset analysis algorithm to generate performance evaluation data of the operator, wherein the preset analysis algorithm may be calibrated according to actual conditions, and no limitation is made here.

[0099] Specifically, after acquiring the above test data, the electronic device may directly analyze the test data according to a preset analysis algorithm to generate performance evaluation data of the operator.

[0100] In order to clearly explain the above embodiment, in one embodiment of the present disclosure, as Figure 4 As shown, the operator is tested according to the data set and operator service to obtain test data, which may include:

[0101] Step 401: encapsulate the data set to generate a request instruction set for the operator service.

[0102] In the embodiment of the present disclosure, the data set may be packaged according to a preset packaging algorithm to generate a request instruction set for the operator service. The preset packaging algorithm may be calibrated according to actual conditions.

[0103] Specifically, after obtaining the data set, the electronic device can encapsulate each data in the data set into a request instruction set of the above-mentioned operator service according to a preset encapsulation algorithm, and each request instruction in the request instruction set is in the request format required by the above-mentioned operator (ie, operator service).

[0104] It should be noted that the fields in Table 2 described in the above embodiment may be fields included in the request instruction. Referring to Table 2, the data field in the request instruction may be the actual input data of the operator (i.e., instruction parameters), which can call different operator services.

[0105] Step 402: Select at least one request instruction from the request instruction set as a target request instruction according to the performance evaluation strategy. There may be multiple target request instructions, and the performance evaluation strategy may be calibrated according to actual conditions.

[0106] Step 403: Call the operator service according to the target request instruction to generate test data for the operator.

[0107] It should be noted that the fields in Table 2 described in the above embodiment may be fields included in the test data. As shown in Table 3, the result field in the test data may be encapsulated into a unified return, and the error code may be standardized and returned in a unified standard format.

[0108] Specifically, after obtaining the request instruction set of the above operator service, the electronic device can select at least one request instruction from the request instruction set as the target request instruction according to the performance evaluation strategy, and call the operator service according to the input data (i.e., instruction parameters) corresponding to the data field in the target request instruction to generate the test data of the operator. The test data may include the return data (i.e., the data in Table 3 above) during the call execution of the operator service, as well as the execution time (i.e., processing time) and execution success rate of this operator service.

[0109] In another embodiment of the present disclosure, Figure 5 As shown, analyzing the test data to obtain the performance evaluation data of the operator may include:

[0110] Step 501, determine whether the test data meets the fuse condition. The fuse condition can be calibrated according to the actual situation.

[0111] Step 502: If the test data meets the fuse condition, the resource data and response data generated by the operator service during the calling process are obtained.

[0112] It should be noted that the resource data described in this embodiment may include CPU (central processing unit) standard cores, memory usage, GPU memory usage, GPU utilization, etc. when the operator service is running (i.e., when the operator service is called for execution). The response data described in this embodiment may include the average time consumption, percentile time consumption (50, 60, 70, 80, 90, 95, 99, 999 percentiles), success rate, etc. during the test.

[0113] Step 503: Analyze the resource data and the response data to generate performance evaluation data of the operator.

[0114] In the embodiments of the present disclosure, resource data and response data may be analyzed according to analysis algorithms such as a stress testing analysis algorithm and a performance evaluation algorithm to generate performance evaluation data of the operator.

[0115] Specifically, after acquiring the above resource data and response data, the electronic device may directly analyze the resource data and response data according to a stress testing analysis algorithm or a performance evaluation algorithm to generate performance evaluation data of the operator.

[0116] As a possible scenario, the resource data and response data may also be analyzed according to the analysis model to generate performance evaluation data of the operator. It should be noted that the analysis model described in this embodiment may be trained in advance and pre-stored in the storage space of the electronic device for easy retrieval and application.

[0117] The training and generation of the analysis model can be performed by a related training server, which can be a cloud server or a computer host. A communication connection is established between the training server and an electronic device that can execute the operator performance evaluation method provided by the embodiment of the present disclosure, and the communication connection can be at least one of a wireless network connection and a wired network connection. The training server can send the trained analysis model to the electronic device so that the electronic device can call it when needed, thereby greatly reducing the computing pressure of the electronic device.

[0118] Specifically, after acquiring the above-mentioned resource data and response data, the electronic device can call out (acquire) the analysis model from its own storage space, and input the resource data and response data into the analysis model, thereby analyzing the resource data and response data through the analysis model to obtain the performance evaluation data of the above-mentioned operator output by the analysis model.

[0119] As another possible scenario, the electronic device may also use an analysis tool (eg, a plug-in) to analyze the resource data and response data to generate performance evaluation data of the operator.

[0120] In related technologies, the reasons why operator testing cannot be automated include the following:

[0121] 1. For resource data, it is necessary to manually count the resource usage in each interval, so as to calculate the average resource usage in this phase after a round of stress testing is completed;

[0122] 2. For response data, it is necessary to manually record the time taken for each request, whether it is successful, and other information. After a round of stress testing is completed, the average time taken, percentile time taken, and success rate of the phase can be counted;

[0123] 3. The format of the generated performance evaluation data is not uniform and the storage methods are diverse, which is not conducive to viewing historical data.

[0124] In order to solve the above problems, the operator performance evaluation method provided in the embodiment of the present disclosure can be solved by the following steps (methods):

[0125] 1. A collection Agent (intelligent agent) can be introduced as an executor (module) for obtaining resource data and response data generated by the operator service during the call process. It mainly has the following two functions: ①, deployed in the operator (for example, AI operator) service container, it can automatically collect resource indicators (that is, resource data) every preset time (for example, 5s) for upload; ②, deployed in the gateway container, the gateway writes the request time consumption, error and other information into the log file in the specified format. The Agent (intelligent agent) collects the logs generated by the gateway in real time for upload.

[0126] 2. For resource data, after receiving each reported resource amount, store it first, and then calculate the average resource usage of the time period after a round of stress testing is completed;

[0127] 3. For statistics of response data, Agent can parse out key log fields according to the specified log format, including time consumption, error code, etc. Then, after a round of stress testing, the number of log entries, time consumption, error code, etc. are aggregated and counted, and the call volume, average time consumption and percentile value, and success rate data are output;

[0128] 4. After a round of stress testing is completed, the statistical resource data and response data are stored;

[0129] 5. After all stress testing rounds are completed, the performance evaluation data of this stress testing can be output.

[0130] Specifically, after obtaining the above test data, the electronic device can parse the test data and determine whether the test data meets the fuse condition according to the parsing result. If so, the resource data and response data generated by the operator service during the invocation process can be obtained through the above steps (methods), that is, the performance evaluation data of the operator can be obtained. In this way, the operator's automated testing can be realized, thereby greatly shortening the operator's testing time.

[0131] Further, in one embodiment of the present disclosure, Figure 6 As shown in the figure, the operator is tested according to the data set and operator service to generate performance evaluation data of the operator, which may also include:

[0132] Step 601: If the test data does not satisfy the fuse condition, at least one request instruction is reselected from the request instruction set as a target request instruction according to the performance evaluation strategy.

[0133] It should be noted that the performance evaluation strategy described in this embodiment may include a stress test strategy, that is, selecting a request instruction as a target request instruction from a request instruction set according to different concurrency increments.

[0134] For example, if one request instruction was selected from the request instruction set in the last test, two request instructions may be selected from the request instruction set again this time; if two request instructions were selected from the request instruction set in the last test, three request instructions may be selected from the request instruction set again this time.

[0135] Step 602, calling the operator service according to the re-selected target request instruction to regenerate the test data of the operator until the re-generated test data meets the fuse condition.

[0136] Specifically, after determining that the above test data does not meet the fuse condition, the electronic device can reselect at least one request instruction from the request instruction set as the target request instruction according to the performance evaluation strategy, and call the operator service according to the reselected target request instruction to regenerate the operator's test data until the regenerated test data meets the fuse condition. In this way, the operator's automated stress test can be implemented, thereby greatly shortening the operator's test time.

[0137] In order to clearly explain the above embodiment, in one embodiment of the present disclosure, as Figure 7 As shown, judging whether the test data meets the fuse condition may include:

[0138] Step 701: parse the test data to obtain the current processing time of the operator service.

[0139] In the embodiment of the present disclosure, the electronic device may parse the test data according to a preset parsing algorithm to obtain the current processing time of the operator service. The preset parsing algorithm may be calibrated according to actual conditions.

[0140] As a possible scenario, the electronic device may parse the test data according to the parsing model to obtain the current processing time of the operator service. It should be noted that the parsing model described in this embodiment may be trained in advance and pre-stored in the storage space of the electronic device for easy retrieval and application.

[0141] As another possible situation, the electronic device may also use a parsing tool (eg, a plug-in) to parse the test data to obtain the current processing time of the operator service.

[0142] Step 702: Obtain the previous historical processing time of the operator service, and calculate the difference between the current processing time and the historical processing time.

[0143] Step 703: If the difference is greater than the time threshold, it is determined that the test data meets the fuse condition. The time threshold can be calibrated according to actual conditions.

[0144] Specifically, after obtaining the current processing consumption of the operator service, the electronic device can obtain the previous historical processing consumption of the operator service from its own storage space, and calculate the absolute value of the difference between the current processing consumption and the historical processing consumption, and determine whether the absolute value is greater than the time threshold. If so, it can be said that the operator service request delay data has changed significantly, and it can be considered that the operator limit has been reached and the stress test (i.e., stress testing) needs to be stopped. At this time, it can be determined that the test data meets the fuse condition.

[0145] See also Figure 8 When the concurrent requests of the operator service are 6 (that is, the operator service is called by 6 request instructions at the same time), the processing time (operator time) suddenly increases significantly, far exceeding the previous average time. Therefore, it can be considered that there are a lot of queues in the operator service at this time, which leads to the increase in time. Therefore, concurrency 5 is the limit concurrency, and the stress test needs to be stopped.

[0146] This can improve the accuracy of operator performance evaluation. At the same time, the operator performance evaluation data has historical data for each iteration (stress test), which is convenient for later users to backtrack and view.

[0147] In addition, in one embodiment of the present disclosure, Fig. 9 As shown, judging whether the test data meets the fuse condition may include:

[0148] Step 901: parse the test data to obtain the current request success rate of the operator service.

[0149] In an embodiment of the present disclosure, the electronic device may parse the test data according to the above-mentioned parsing algorithm to obtain the current request success rate of the operator service, or simultaneously obtain the current processing time and the current request success rate of the operator service.

[0150] As a possible scenario, the electronic device may parse the test data according to the above-mentioned parsing model to obtain the current request success rate of the operator service, or simultaneously obtain the current processing time and the current request success rate of the operator service.

[0151] As another possible scenario, the electronic device may also use the above-mentioned parsing tool (eg, plug-in) to parse the test data to obtain the current request success rate of the operator service, or simultaneously obtain the current processing time and current request success rate of the operator service.

[0152] Step 902: Obtain the previous historical request success rate of the operator service, and calculate the difference between the current request success rate and the historical request success rate.

[0153] Step 903: If the difference is greater than the success rate threshold, it is determined that the test data meets the fuse condition.

[0154] Specifically, after obtaining the current request success rate of the operator service, the electronic device can obtain the previous historical request success rate of the operator service from its own storage space, and calculate the absolute value of the difference between the current request success rate and the historical request success rate, and determine whether the absolute value is greater than the success rate threshold. If so, it can be said that the operator service request success rate data has changed significantly, and it can be considered that the operator limit has been reached and the stress test needs to be stopped. At this time, it can be determined that the test data meets the fuse condition.

[0155] See also Fig.10 When the concurrent requests of the operator service are 6 (that is, the operator service is called by 6 request instructions at the same time), the request success rate suddenly starts to drop, and the change is very large. Therefore, it can be considered that at this time, a large number of queued requests of the operator service timed out, which eventually led to request failure. Therefore, concurrency 5 is the limit concurrency, and the stress test needs to be stopped.

[0156] This can improve the accuracy of operator performance evaluation. At the same time, the operator performance evaluation data has historical data for each iteration (stress test), which is convenient for later users to backtrack and view.

[0157] Further, in one embodiment of the present disclosure, Fig.11 As shown, the resource data and response data are analyzed to generate performance evaluation data of the operator, which may include:

[0158] Step 1101, splitting the resource data and the response data respectively to obtain target resource data, target response data, reference resource data and reference response data.

[0159] In the disclosed embodiment, the electronic device may split the resource data and the response data according to a preset splitting algorithm to obtain target resource data and target response data, as well as reference resource data and reference response data. The preset splitting algorithm may be calibrated according to actual conditions.

[0160] As a possible scenario, the electronic device may split the resource data and the response data according to the split model to obtain target resource data and target response data, as well as reference resource data and reference response data. It should be noted that the split model described in this embodiment may be trained in advance and pre-stored in the storage space of the electronic device for easy retrieval and application.

[0161] As another possible scenario, the electronic device may also use a splitting tool (eg, a plug-in) to split the resource data and the response data, respectively, to obtain target resource data and target response data, as well as reference resource data and reference response data.

[0162] It should be noted that the target resource data and target response data described in the above embodiments may be the resource data and response data obtained in the penultimate round during the entire test (stress test) of the operator (i.e., the resource data and response data generated (obtained) under extreme concurrency).

[0163] Step 1102: Perform performance evaluation on the operator according to the target resource data, the target response data, the reference resource data and the reference response data to generate performance evaluation data.

[0164] Specifically, after obtaining the target resource data and target response data, as well as the reference resource data and reference response data, the electronic device can perform a performance evaluation on the operator according to the preset performance evaluation algorithm, target resource data, target response data, reference resource data and reference response data to generate performance evaluation data. Among them, the preset performance evaluation algorithm can be calibrated according to the actual situation. In this way, the performance evaluation data of the operator can be obtained, which can lay the foundation for the subsequent large-scale deployment of PaaS, and at the same time is conducive to making full use of the underlying computing resources.

[0165] Fig.12 A block diagram of an operator performance evaluation device provided in an embodiment of the present disclosure.

[0166] The operator performance evaluation device of the embodiment of the present disclosure can be configured in an electronic device to determine the artificial intelligence AI operator to be evaluated, and deploy the service of the operator to obtain the operator service, and then obtain the data set of the operator, and test the operator according to the data set and the operator service to generate performance evaluation data of the operator, which can reduce the evaluation cost of the tester, while reducing the personnel interference factor and improving the accuracy of the performance evaluation.

[0167] like Fig.12 As shown, the operator performance evaluation device 1200 may include: a determination module 1210 , a service deployment module 1220 , an acquisition module 1230 and a testing module 1240 .

[0168] Among them, the determination module 1210 is used to determine the artificial intelligence AI operator to be evaluated.

[0169] It should be noted that the operator described in this embodiment (ie, the artificial intelligence AI operator) may be a service-oriented AI operator.

[0170] In the embodiment of the present disclosure, relevant R&D personnel may send (submit) the serviced operator to the determination module 1210. After receiving and determining the operator, the determination module 1210 may store the operator in the storage space of the electronic device for subsequent calling (use).

[0171] The service deployment module 1220 is used to perform service deployment on operators to obtain operator services.

[0172] In the embodiment of the present disclosure, the service deployment module 1220 can automatically deploy services to operators according to a preset service deployment strategy to obtain operator services, wherein the preset service deployment strategy can be calibrated according to actual conditions.

[0173] Specifically, after the determination module 1210 determines the above-mentioned operator to be evaluated, the service deployment module 1220 may perform service deployment on the operator according to a preset service deployment strategy to obtain the operator service.

[0174] The acquisition module 1230 is used to acquire the data set of the operator.

[0175] It should be noted that the data set described in this embodiment may be composed of videos, pictures, texts, etc. collected by relevant testers.

[0176] In the disclosed embodiment, the relevant tester may send (submit) the above data set to the acquisition module 1230. After receiving the data set, the acquisition module 1230 may store the data set in the storage space of the electronic device for subsequent calling (use).

[0177] Specifically, after the service deployment module 1220 obtains the operator service, the acquisition module 1230 may acquire (call out) the data set of the above operator from the storage space of the electronic device.

[0178] As a possible scenario, after the service deployment module 1220 obtains the operator service, the acquisition module 1230 may directly receive the data set of the above operator submitted by the relevant tester.

[0179] The testing module 1240 is used to test the operator according to the data set and the operator service to generate performance evaluation data of the operator.

[0180] In the embodiment of the present disclosure, after the acquisition module 1230 obtains the data set, the test module 1240 may first encapsulate the data set according to a preset strategy to generate a request instruction set for an operator service, wherein the request instruction set may include multiple request instructions. Then, the test module 1240 may test the operator according to the request instruction set and the operator service to generate performance evaluation data for the operator.

[0181] It should be noted that the test described in this embodiment may be a stress test.

[0182] Furthermore, after generating the performance evaluation data of the above-mentioned operator, the testing module 1240 may provide the performance evaluation data of the operator to relevant personnel, for example, by sending via text message, displaying via a display screen of an electronic device, sending via email, etc.

[0183] It should be noted that the performance evaluation data of the operator (AI operator) described in this embodiment is the basis for subsequent large-scale deployment of PaaS. For GPU operators, dynamic scheduling will be performed based on performance evaluation data, including decisions such as how much resources to allocate and whether to enable mixed distribution, so as to make full use of the underlying computing resources. It has been used in real online systems, and the scale of deployed GPU cards can be 3000+ cards. Through this stored performance evaluation data information, detailed resource allocation and sufficient mixed distribution are performed on the underlying operators to achieve multiple instances of one card.

[0184] In the disclosed embodiment, the AI ​​operator to be evaluated is first determined by the determination module, and the service deployment module is used to deploy the operator to obtain the operator service, and the operator data set is obtained by the acquisition module, and then the operator is tested according to the data set and the operator service by the test module to generate the performance evaluation data of the operator. In this way, the evaluation cost of the tester can be reduced, the interference factor of the personnel can be reduced, and the accuracy of the performance evaluation can be improved.

[0185] In one embodiment of the present disclosure, Fig.12 As shown, the testing module 1240 may include: a testing unit 1241 and an analyzing unit 1242 .

[0186] The testing unit 1241 is used to test the operator according to the data set and the operator service to obtain test data.

[0187] The analysis unit 1242 is used to analyze the test data to obtain performance evaluation data of the operator.

[0188] In one embodiment of the present disclosure, the service deployment module 1220 is specifically used to: obtain a graphics processor GPU specified model of the operator; determine a deployment strategy according to the GPU specified model; and perform service deployment on the operator according to the deployment strategy to obtain the operator service.

[0189] In one embodiment of the present disclosure, the test unit 1241 is specifically used to: encapsulate the data set to generate a request instruction set for the operator service; select at least one request instruction from the request instruction set as a target request instruction according to a performance evaluation strategy; and call the operator service according to the target request instruction to generate test data for the operator.

[0190] In one embodiment of the present disclosure, the analysis unit 1242 is specifically used to: determine whether the test data meets the fuse condition; if the test data meets the fuse condition, obtain the resource data and response data generated by the operator service during the calling process; analyze the resource data and the response data to generate performance evaluation data of the operator.

[0191] In one embodiment of the present disclosure, the analysis unit 1242 is specifically used to: split the resource data and the response data respectively to obtain target resource data, target response data, reference resource data and reference response data; perform performance evaluation on the operator according to the target resource data, target response data, reference resource data and reference response data to generate performance evaluation data.

[0192] In one embodiment of the present disclosure, the test module 1240 is also used to: if the test data does not meet the fuse condition, re-select at least one request instruction from the request instruction set as the target request instruction according to the performance evaluation strategy; call the operator service according to the re-selected target request instruction to regenerate the test data of the operator until the regenerated test data meets the fuse condition.

[0193] In one embodiment of the present disclosure, the analysis unit 1242 is also used to: parse the test data to obtain the current processing time of the operator service; obtain the previous historical processing time of the operator service, and calculate the difference between the current processing time and the historical processing time; if the difference is greater than the time threshold, it is determined that the test data meets the fuse condition.

[0194] In one embodiment of the present disclosure, the analysis unit 1242 is also used to: parse the test data to obtain the current request success rate of the operator service; obtain the previous historical request success rate of the operator service, and calculate the difference between the current request success rate and the historical request success rate; if the difference is greater than the success rate threshold, it is determined that the test data meets the fuse condition.

[0195] It should be noted that the above explanation of the operator performance evaluation method embodiment is also applicable to the operator performance evaluation device of this embodiment, and will not be repeated here.

[0196] The operator performance evaluation device of the embodiment of the present disclosure first determines the artificial intelligence AI operator to be evaluated through the determination module, and deploys the service of the operator through the service deployment module to obtain the operator service, and obtains the operator data set through the acquisition module, and then tests the operator according to the data set and the operator service through the test module to generate the performance evaluation data of the operator. In this way, the evaluation cost of the tester can be reduced, the interference factor of the personnel can be reduced, and the accuracy of the performance evaluation can be improved.

[0197] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0198] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0199] Fig.13 A schematic block diagram of an example electronic device 1300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0200] like Fig.13 As shown, the device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1302 or a computer program loaded from a storage unit 1308 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the device 1300 can also be stored. The computing unit 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0201] A number of components in the device 1300 are connected to the I / O interface 1305, including: an input unit 1306, such as a keyboard, a mouse, etc.; an output unit 1307, such as various types of displays, speakers, etc.; a storage unit 1308, such as a disk, an optical disk, etc.; and a communication unit 1309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1309 allows the device 1300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0202] The computing unit 1301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1301 performs the various methods and processes described above, such as the operator performance evaluation method. For example, in some embodiments, the operator performance evaluation method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1308. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1300 via the ROM 1302 and / or the communication unit 1309. When the computer program is loaded into the RAM 1303 and executed by the computing unit 1301, one or more steps of the operator performance evaluation method described above may be performed. Alternatively, in other embodiments, the computing unit 1301 may be configured to execute the operator performance evaluation method in any other appropriate manner (eg, by means of firmware).

[0203] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0204] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0205] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0206] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0207] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0208] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0209] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0210] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An operator performance evaluation method, comprising: Determine the artificial intelligence (AI) operator to be evaluated; Performing service deployment on the operator to obtain the operator service; Obtaining a data set of the operator; Testing the operator according to the data set and the operator service to generate performance evaluation data of the operator; The performing service deployment on the operator to obtain the operator service includes: Get the GPU model of the operator. Determine a deployment strategy according to the GPU specified model, wherein the deployment strategy is determined according to the unified basic environment dependency, container deployment, unified CPU and GPU environment, and unified use of encapsulated scripts to start the deployment strategy; Performing service deployment on the operator according to the deployment strategy to obtain the operator service; The basic environment dependencies include unified deployment of machine kernels, unified provision of basic environment dependency libraries, unified provision of CUDA and CDNN version dependencies, and restrictions on GPU hardware models.

2. The method according to claim 1, wherein: The testing the operator according to the data set and the operator service to generate performance evaluation data of the operator includes: Testing the operator according to the data set and the operator service to obtain test data; The test data is analyzed to obtain performance evaluation data of the operator.

3. The method according to claim 2, wherein: The testing the operator according to the data set and the operator service to obtain test data includes: Encapsulating the data set to generate a request instruction set for the operator service; Selecting at least one request instruction from the request instruction set as a target request instruction according to a performance evaluation strategy; The operator service is called according to the target request instruction to generate test data for the operator.

4. The method according to claim 3, wherein: The analyzing the test data to obtain the performance evaluation data of the operator includes: Determine whether the test data meets the fuse condition; If the test data meets the fuse condition, the resource data and response data generated by the operator service during the invocation process are obtained; The resource data and the response data are analyzed to generate performance evaluation data of the operator.

5. The method according to claim 4, wherein: The analyzing the resource data and the response data to generate performance evaluation data of the operator includes: The resource data and the response data are respectively split to obtain target resource data, target response data, reference resource data and reference response data; The performance of the operator is evaluated according to the target resource data, the target response data, the reference resource data and the reference response data to generate the performance evaluation data.

6. The method according to claim 4, wherein: The testing the operator according to the data set and the operator service to generate performance evaluation data of the operator further includes: If the test data does not satisfy the fuse condition, reselecting at least one request instruction from the request instruction set as a target request instruction according to the performance evaluation strategy; The operator service is called according to the re-selected target request instruction to regenerate the test data of the operator until the re-generated test data meets the fuse condition.

7. The method according to claim 4, wherein: The determining whether the test data satisfies the fuse condition includes: Parsing the test data to obtain the current processing time of the operator service; Obtain the previous historical processing time of the operator service, and calculate the difference between the current processing time and the historical processing time; If the difference is greater than the time threshold, it is determined that the test data meets the fuse condition.

8. The method according to claim 4, wherein: The determining whether the test data satisfies the fuse condition includes: Parsing the test data to obtain the current request success rate of the operator service; Obtain the previous historical request success rate of the operator service, and calculate the difference between the current request success rate and the historical request success rate; If the difference is greater than the success rate threshold, it is determined that the test data meets the fuse condition.

9. An operator performance evaluation device, comprising: A determination module, used to determine the artificial intelligence AI operator to be evaluated; A service deployment module, used to perform service deployment on the operator to obtain the operator service; An acquisition module, used to acquire a data set of the operator; A testing module, used to test the operator according to the data set and the operator service to generate performance evaluation data of the operator; Wherein, the service deployment module is specifically used to: Get the GPU model of the operator. Determine a deployment strategy according to the GPU specified model, wherein the deployment strategy is determined according to the unified basic environment dependency, container deployment, unified CPU and GPU environment, and unified use of encapsulated scripts to start the deployment strategy; Performing service deployment on the operator according to the deployment strategy to obtain the operator service; The basic environment dependencies include unified deployment of machine kernels, unified provision of basic environment dependency libraries, unified provision of CUDA and CDNN version dependencies, and restrictions on GPU hardware models.

10. The device according to claim 9, wherein: The test module comprises: A testing unit, used to test the operator according to the data set and the operator service to obtain test data; The analysis unit is used to analyze the test data to obtain performance evaluation data of the operator.

11. The device according to claim 10, wherein: The testing unit is specifically used for: Encapsulating the data set to generate a request instruction set for the operator service; Selecting at least one request instruction from the request instruction set as a target request instruction according to a performance evaluation strategy; The operator service is called according to the target request instruction to generate test data for the operator.

12. The device according to claim 11, wherein: The analysis unit is specifically used for: Determine whether the test data meets the fuse condition; If the test data meets the fuse condition, the resource data and response data generated by the operator service during the invocation process are obtained; The resource data and the response data are analyzed to generate performance evaluation data of the operator.

13. The device according to claim 12, wherein: The analysis unit is specifically used for: The resource data and the response data are respectively split to obtain target resource data, target response data, reference resource data and reference response data; The performance of the operator is evaluated according to the target resource data, the target response data, the reference resource data and the reference response data to generate the performance evaluation data.

14. The device according to claim 12, wherein: The test module is also used for: If the test data does not satisfy the fuse condition, reselecting at least one request instruction from the request instruction set as a target request instruction according to the performance evaluation strategy; The operator service is called according to the re-selected target request instruction to regenerate the test data of the operator until the re-generated test data meets the fuse condition.

15. The device according to claim 12, wherein: The analysis unit is further used for: Parsing the test data to obtain the current processing time of the operator service; Obtain the previous historical processing time of the operator service, and calculate the difference between the current processing time and the historical processing time; If the difference is greater than the time threshold, it is determined that the test data meets the fuse condition.

16. The device according to claim 12, wherein: The analysis unit is further used for: Parsing the test data to obtain the current request success rate of the operator service; Obtain the previous historical request success rate of the operator service, and calculate the difference between the current request success rate and the historical request success rate; If the difference is greater than the success rate threshold, it is determined that the test data meets the fuse condition.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the operator performance evaluation method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the operator performance evaluation method according to any one of claims 1 to 8.

19. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the operator performance evaluation method according to any one of claims 1 to 8.