Reasoning device testing method, device, equipment, storage medium and program product

Abstract

The present application discloses an inference device testing method, device, equipment, storage medium and program product, wherein the method includes: obtaining the remaining resource information of the test cluster, and obtaining the test parameters and test priority of each of at least two test tasks; the remaining resource information is at least used to characterize the working status of the inference device in the test cluster; based on the remaining resource information and the test parameters and test priority of each of the test tasks, an inference device is allocated to each of the test tasks; through the allocated inference device, the test task is executed based on the test parameters of the test task, and the test result corresponding to the inference device is obtained.

Description

Technical Field

[0001] The present application relates to the technical field of inference device testing, and in particular to an inference device testing method, device, equipment, storage medium and computer program product. Background Art

[0002] As graphics processing units (GPUs) demand for parallel computing capabilities, inference device tests are increasingly adding parallel tests to evaluate the performance of graphics cards when processing multiple parallel tasks.

[0003] When the related technology performs parallel testing on graphics cards through model reasoning, different models may require different video memory and reasoning time. Using the same test parameters may result in insufficient video memory for some models or too long model reasoning time, resulting in the inability to complete the continuous integration (CI) task. How to reasonably allocate video memory and time when performing parallel testing on graphics cards has become an urgent problem to be solved. Summary of the invention

[0004] In view of this, the embodiments of the present application at least provide a method, device, equipment, storage medium and program product for testing an inference device.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] On the one hand, an embodiment of the present application provides an inference device testing method, which is applied to a test cluster including at least one inference device, and the method includes: obtaining the remaining resource information of the test cluster, and obtaining the test parameters and test priority of each test task in at least two test tasks; the remaining resource information is at least used to characterize the working status of the inference device in the test cluster; based on the remaining resource information and the test parameters and test priority of each test task, an inference device is assigned to each test task; through the assigned inference device, the test task is executed based on the test parameters of the test task, and the test result corresponding to the inference device is obtained.

[0007] On the other hand, an embodiment of the present application provides an inference device testing device, including: an acquisition module, used to obtain the remaining resource information of the test cluster, and obtain the test parameters and test priority of each test task in at least two test tasks; the remaining resource information is at least used to characterize the working status of the inference device in the test cluster; an allocation module, used to allocate an inference device to each test task based on the remaining resource information and the test parameters and test priority of each test task; a testing module, used to execute the test task based on the test parameters of the test task through the allocated inference device, and obtain the test result corresponding to the inference device.

[0008] On the other hand, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0009] On the other hand, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements part or all of the steps in the above method when executed by a processor.

[0010] On the other hand, an embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements some or all of the steps in the above method.

[0011] In the embodiment of the present application, by obtaining the remaining resource information of the test cluster, the working status of each inference device can be monitored in real time, the accuracy of resource allocation can be ensured, and the situation of idle or overloaded resources can be effectively avoided; at the same time, by obtaining the test parameters and priority of each test task, the test task requirements can be deeply understood, which provides an important basis for resource allocation, and then the most suitable inference device can be allocated to each test task, which not only ensures the priority execution of high-priority tasks, but also achieves the optimal utilization of resources. Finally, by performing tasks according to the test parameters through the assigned inference device, the accuracy and reliability of the test results can be ensured, and solid data support is provided for the performance evaluation of the inference device. Overall, through this series of refined operations, the present application not only significantly improves the work efficiency of the test cluster, but also ensures the accuracy and reliability of the test results, laying a solid foundation for the optimization and upgrading of the inference device.

[0012] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0014] Figure 1 A schematic diagram of the implementation process of a method for testing an inference device provided in an embodiment of the present application Figure 1 ;

[0015] Figure 2 A schematic diagram of the implementation process of a method for testing an inference device provided in an embodiment of the present application Figure 2 ;

[0016] Figure 3 A schematic diagram of the implementation process of a method for testing an inference device provided in an embodiment of the present application Figure 3 ;

[0017] Figure 4 A schematic diagram of the implementation process of a method for testing an inference device provided in an embodiment of the present application Figure 4 ;

[0018] Figure 5 A schematic diagram of configuration parameter records in a configuration file provided in an embodiment of the present application;

[0019] Figure 6 A sixth schematic diagram of a test unit creation process provided in an embodiment of the present application;

[0020] Figure 7 A schematic diagram of a timing table provided in an embodiment of the present application;

[0021] Figure 8 A schematic diagram of the structure of an inference device testing device provided in an embodiment of the present application;

[0022] Fig. 9 A hardware entity diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0024] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific order for the objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing this application and are not intended to limit this application.

[0026] In the current era of AI big models, the development of artificial intelligence is advancing at an astonishing speed. Various large-scale pre-trained models such as GPT-4 and BERT have made significant progress in natural language processing, computer vision and other fields. These models usually require a lot of computing resources and data for training and reasoning, which has promoted the demand for high-performance computing devices, especially graphics processing units (GPUs). With the rapid development of GPU technology in recent years, corporate R&D often has low efficiency, insufficient testing, and waste of resources when using domestic graphics cards for performance parallel testing and stability parallel testing of daily model reasoning. This application aims to solve this problem.

[0027] In the case of limited resources such as laboratory test cluster machines (PC|Server) and graphics cards, and the graphics card (chip) companies need to run many tasks for daily CI testing, including graphics card bottom-level drivers, middle-level operator libraries, middle-level math acceleration libraries, upper-level neural network frameworks, upper-level neural network model training, upper-level neural network model reasoning, etc. Each CI task consumes machine resources and graphics card resources.

[0028] The following are several reasons why performance and stability parallel testing of model inference on graphics cards is inefficient, testing is insufficient, and resources are wasted:

[0029] (1) Different deep learning AI frameworks and models require different amounts of video memory for reasoning. Using the same test parameters will result in insufficient video memory for some models, making it impossible to complete daily continuous integration (CI) tasks.

[0030] (2) Different deep learning AI frameworks and models require different amounts of time for reasoning. Using the same test parameters will cause some models to take too long to complete daily CI tasks.

[0031] (3) Different graphics cards have different memory sizes and performance. Therefore, it is impossible to use the same set of parameters to perform stability tests on different graphics cards in parallel.

[0032] (4) When some other CIs are running, only part of the graphics card's video memory is used, so there is a situation where the video memory is not fully utilized.

[0033] (5) As the graphics card driver and other software versions are updated, the performance of the model will change, and the model test parameters also need to change dynamically.

[0034] Based on this, how to efficiently and reasonably utilize resources to perform performance testing and stability testing of neural network models is the problem that this application solves.

[0035] The embodiment of the present application provides a method for testing an inference device, which is applied to a test cluster including at least one inference device. In some embodiments, the test cluster may include at least one node, and the node may include one or more of the inference devices. The method for testing an inference device may be executed by a control node (which may be any node) in the test cluster, and the control node may be a server, a laptop, a tablet computer, a desktop computer, a smart TV, a set-top box, a mobile device (such as a mobile phone, a portable video player, a personal digital assistant, a dedicated messaging device, a portable gaming device), or other device with data processing capabilities.

[0036] In some embodiments, the above-mentioned reasoning device is a computing device for performing reasoning tasks, such as a graphics processing unit (GPU) or other dedicated accelerator card, which can efficiently process the reasoning calculation of the deep learning model. In the actual implementation scenario, the reasoning device includes but is not limited to a high-performance central processing unit (CPU), a graphics processor, a field programmable gate array (FPGA), and an application-specific integrated circuit (ASIC).

[0037] Figure 1 A schematic diagram of the implementation process of a method for testing an inference device provided in an embodiment of the present application Figure 1 ,like Figure 1 As shown, the method includes the following steps S101 to S103:

[0038] Step S101, obtaining the remaining resource information of the test cluster, and obtaining the test parameters and test priority of each of at least two test tasks; the remaining resource information is at least used to characterize the working status of the reasoning device in the test cluster.

[0039] The remaining resource information of the test cluster is at least used to characterize the working state of the inference device in the test cluster, and the working state can characterize whether the inference device is idle, and can also characterize the remaining information of the video memory of the inference device. In some embodiments, the above-mentioned remaining resource information can include but is not limited to the GPU idle rate, the number of idle GPUs, the remaining amount of video memory in the GPU, and the GPU video memory idle rate.

[0040] Among them, the test task is an AI model (or algorithm task) that needs to be inferred by an inference device, and each test task contains specific test parameters and test priorities. In some embodiments, the test parameters are used to characterize the test task's demand for computing resources, and may include but are not limited to the resource requirements of the test task, the number of cycles of the test task, and the number of parallel processes running the test task. Among them, the resource requirements of the test task are the amount of hardware resources for running the test task (which can be the number of inference devices or the amount of video memory resources of the inference device); the number of cycles is the number of times a test task needs to be tested in a loop; the number of parallel processes is the number of parallel processes used in the process of running the test task in parallel.

[0041] In some embodiments, a corresponding test priority needs to be set for each test task in advance, and the test priority is an indicator to measure the importance and urgency of the test task. Exemplarily, it can be divided into high priority, medium priority and low priority, and the priority determines the order of the test tasks in resource allocation.

[0042] Step S102: allocating an inference device to each of the test tasks based on the remaining resource information and the test parameters and test priority of each of the test tasks.

[0043] Among them, the test priority can be used to determine the order of test task allocation; the test parameters are used to determine the resource requirements in the test task allocation process, such as the number of idle inference devices required by the test task, or the remaining video memory resources required by the test task.

[0044] In some embodiments, the test task can be allocated based on the idle state of the inference device. The above allocation process may include: real-time or periodic monitoring of the idle state of each inference device, that is, whether it is executing a task or in an idle waiting state; sorting according to the test priority of the test task, and the test task with a high priority has priority to obtain the inference device resources; obtaining the test task with the highest test priority, if there is an idle inference device, and the idle inference device can meet the resource requirements of the test task (here, the specific video memory requirements are not considered temporarily, only whether the inference device is idle, that is, how many idle inference devices are required for the test task), then the test task is allocated to the idle inference device. In addition, if there is no idle inference device, or the idle inference device cannot meet the resource requirements of the test task, it enters the waiting state until the resource requirements of the test task are met. During the execution process, the idle state of the inference device is continuously monitored. If a certain inference device completes the current task and releases resources, the unexecuted test tasks are checked and reallocated according to the test priority.

[0045] Exemplarily, assume that there are three reasoning devices (A, B, C) and five test tasks (T1, T2, T3, T4, T5), and the test parameters of each test task indicate the need for an idle reasoning device, where T1 is the highest priority and T5 is the lowest priority. The allocation process can be: real-time monitoring of the idle status of the three reasoning devices A, B, and C (initially, the three reasoning devices A, B, and C are all busy); when the system detects that the A device is idle, it checks the task queue and finds that the test task T1 is the highest priority and the A device can meet its resource requirements (i.e., an idle reasoning device), so T1 is allocated to the A device. Subsequently, the B device also becomes idle, but at this time T2, T3, T4, and T5 are all waiting for allocation. Since T2 has the highest priority and the B device can meet its needs, T2 is allocated to the B device. Assume that the C device is still busy at this time, and T3, T4, and T5 continue to wait. When device C completes the task and releases resources, the system checks the task queue again and finds that T3 has the highest priority and device C is idle, so it assigns T3 to device C, and so on.

[0046] As another example, assume that there are three reasoning devices (A, B, C) and five test tasks (T1, T2, T3, T4, T5). T1 and T3 require three idle reasoning devices to run simultaneously; T2, T4, and T5 only require one idle reasoning device to run. Initially, the three reasoning devices A, B, and C are all idle. The system checks the task queue and finds that T1 has the highest priority and requires three idle reasoning devices. Since there are only three reasoning devices and they are all idle, T1 is assigned to the three devices A, B, and C. After T1 starts executing, the three devices A, B, and C become busy. Assume that after T1 is executed, the three devices A, B, and C release resources and become idle. The system checks the task queue again and finds that T2 has the highest priority and requires an idle reasoning device, so T2 is assigned to the A device (or B device, or C device). After T2 starts executing, the A device becomes busy, and the B and C devices are idle. At this time, the system checks the remaining tasks and finds that T3 is the highest priority waiting task, but it requires three idle reasoning devices, which cannot be satisfied at present. Therefore, it enters the waiting state until T2 is executed and the three devices A, B, and C become idle. T3 is assigned to the three devices A, B, and C, and so on. It can be seen that in the embodiment of the present application, during the execution of T2, although there are two idle reasoning devices, T4 or T5 is not executed first in consideration of the test priority.

[0047] In other embodiments, tasks may also be allocated based on the remaining video memory resources of the inference device, and the above allocation process may include: real-time or periodic monitoring of the remaining video memory resources of each inference device, that is, the amount of video memory currently available for each inference device. Sorting according to the test priority of the test tasks, high-priority test tasks are given priority to obtain inference device resources; obtaining the test task with the highest test priority, and if the remaining video memory resources meet the resource requirements of the test task, the test task is allocated to the corresponding inference device. If an inference device that meets the video memory requirements cannot be found, a waiting state is entered until the resource requirements of the test task are met.

[0048] Among them, a test task can be run in different reasoning devices, or can be run only in the same reasoning device.

[0049] Exemplarily, assume that the inference device includes: A (remaining video memory 10GiB), B (remaining video memory 8GiB), C (remaining video memory 6GiB), and there are 5 test tasks: T1 (requires video memory 6GiB), T2 (requires video memory 4GiB), T3 (requires video memory 9GiB), T4 (requires video memory 3GiB), T5 (requires video memory 5GiB); among them, T1>T2>T3>T4>T5 (test priority T1 is the highest and T5 is the lowest). The allocation process includes: the system obtains the highest priority test task T1, which requires video memory 6GiB. After checking, it is found that the remaining video memory of inference devices A and B can meet the needs of T1. Therefore, T1 is allocated to one of the inference devices that meets the requirements (for example, A), and the remaining video memory of A is updated to 4GiB. Then, continue to check the remaining tasks T2, T3, T4, and T5, and obtain the second highest priority task T2, which requires video memory 4GiB. At this time, inference devices B (8 GiB remaining) and C (6 GiB) can both meet the requirements of T2. Therefore, T2 is assigned to one of the inference devices (for example, B), and the remaining video memory of B is updated to 4 GiB. At this time, inference devices A (4 GiB remaining), B (4 GiB remaining), and C (6 GiB) all have tasks to execute. The system checks the remaining task T3 (9 GiB of video memory is required). In the scenario where the test tasks can only be run on the same inference device, there is currently no inference device with remaining video memory that can meet the requirements of T3. Therefore, T3 enters a waiting state, waiting for an inference device to release resources. Afterwards, assuming that inference device A first completes the execution of T1 and releases resources, its remaining video memory is restored to the initial value of 10GiB, so T3 is allocated to inference device A, and A's remaining video memory is updated to 1GiB, and so on; in the scenario where the test task can run on different inference devices, task T3 can be allocated to 4GiB of inference device A, 4GiB of inference device B, and 1GiB in inference device C. At this time, inference devices A and B have no remaining video memory, and there are 5GiB remaining in inference device C, and the allocation process of subsequent tasks continues.

[0050] Step S103: Execute the test task based on the test parameters of the test task through the assigned inference device to obtain a test result corresponding to the inference device.

[0051] In some embodiments, after the allocation of the inference device is completed, the test environment can be initialized on the allocated inference device according to the requirements of the test parameters, including loading the model, configuring the input data, setting the number of loops and the number of parallel processes, etc.

[0052] Taking the test parameters including the number of loops and the number of parallel processes as an example, the execution process of a test task may include: setting the number of loops M and the number of parallel processes N, and creating N parallel processes on the inference device assigned to this test task according to the number of parallel processes N; assigning the test task to each parallel process, and each process will independently perform M loop inferences on the test data set; wherein, during the process of each parallel process executing the test task, the process will read the test data, perform model inference, and record the running results, repeating the above steps M times until all loops are completed; after all parallel processes are completed, the M×N running results generated are collected; combined with the M×N running results, statistical analysis is performed (such as calculating accuracy, recall rate, etc.) to obtain the test results corresponding to the final inference device.

[0053] In the embodiment of the present application, by obtaining the remaining resource information of the test cluster, the working status of each inference device can be monitored in real time, the accuracy of resource allocation can be ensured, and the situation of idle or overloaded resources can be effectively avoided; at the same time, by obtaining the test parameters and priority of each test task, the test task requirements can be deeply understood, which provides an important basis for resource allocation, and then the most suitable inference device can be allocated to each test task, which not only ensures the priority execution of high-priority tasks, but also achieves the optimal utilization of resources. Finally, by performing tasks according to the test parameters through the assigned inference device, the accuracy and reliability of the test results can be ensured, and solid data support is provided for the performance evaluation of the inference device. Overall, through this series of refined operations, the present application not only significantly improves the work efficiency of the test cluster, but also ensures the accuracy and reliability of the test results, laying a solid foundation for the optimization and upgrading of the inference device.

[0054] Figure 2 This is a schematic diagram of the implementation process of a reasoning device testing method provided in an embodiment of the present application. Figure 2 , the method can be executed by a processor of a computer device. Figure 1 , the remaining resource information includes first resource information, and the first resource information is used to indicate the number of inference devices in the test cluster that are in an idle state; the test parameter includes a first parameter, and the first parameter is used to characterize the number of inference devices required to execute the test task; Figure 1 Step S102 in can be updated to step S201, combining Figure 2 The steps shown are explained.

[0055] Step S201: When the first resource information is greater than or equal to the total amount of inference devices required, a corresponding number of inference devices are allocated to each of the test tasks based on the first parameters of each of the test tasks; the total amount of inference devices required is the sum of the first parameters of each of the test tasks.

[0056] Among them, the first resource information is the number of inference devices that are currently idle in the test cluster, which is used to evaluate the remaining computing capacity of the test cluster so as to allocate resources for the test task; the above-mentioned test parameters include a first parameter, which is used to characterize the number of inference devices required to execute the test task.

[0057] In the above embodiment, when the remaining resources (i.e., the number of idle inference devices) of the test cluster are sufficient to meet the needs of all tasks to be tested, step S201 expects to ensure that each test task can obtain sufficient computing resources by reasonably allocating resources, so as to efficiently complete the inference task. The implementation scheme is to allocate a corresponding number of inference devices to each test task based on the first parameter (the number of required inference devices) of each test task.

[0058] In some embodiments, it is necessary to obtain the first parameters of all test tasks (the number of required inference devices), and then add these first parameters to obtain the total number of inference devices required when all test tasks are executed in parallel. Afterwards, compare the first resource information (the number of idle inference devices) with the total number of inference devices required. If the first resource information is greater than or equal to the total number of inference devices required, it means that the cluster has sufficient resources to meet the needs of parallel execution of all test tasks; on the premise of meeting the requirements, allocate a corresponding number of inference devices to each test task according to its first parameter. The allocation process can be based on certain strategies, such as giving priority to tasks with high priority, or allocating tasks according to their urgency.

[0059] For example, suppose there is a test cluster with 10 idle inference devices (GPUs), and there are 3 tasks to be tested, of which Task 1 requires 3 GPUs, Task 2 requires 4 GPUs, and Task 3 requires 2 GPUs. According to the resource allocation strategy (i.e., step S201), the system will first calculate the total demand for GPUs for all tasks, and the total demand is 9 GPUs. Subsequently, the system will check whether the number of currently idle GPUs meets this demand, and find that 10 idle GPUs are sufficient to meet the demand for 9 GPUs, so it is judged that the resources are sufficient. Finally, the system will allocate 3 GPUs to Task 1, 4 GPUs to Task 2, and 2 GPUs to Task 3 according to the specific needs of each task, thereby ensuring that each task can obtain the required computing resources to efficiently complete the inference task.

[0060] In the above embodiment, by reasonably allocating resources, it is ensured that each test task can obtain sufficient computing resources, thereby improving the resource utilization of the cluster; at the same time, since each task has obtained sufficient resources, multiple test tasks can be executed in parallel, thereby shortening the overall test time and improving test efficiency.

[0061] In some embodiments, the allocating an inference device to each of the test tasks based on the remaining resource information and the test priority of each of the test tasks further includes step S202.

[0062] Step S202: When the first resource information is less than the total amount of the inference device requirement and greater than the preset number of devices, a corresponding number of inference devices are allocated to each of the test tasks in order of test priority from high to low based on the first real-time resource information and the first parameter of each of the test tasks.

[0063] The first real-time resource information is real-time first resource information in response to allocating reasoning devices for the current test priority. The number of preset devices can be set to 1 or to the minimum first parameter.

[0064] In an embodiment of the present application, the remaining resource information includes first resource information, and the first resource information is used to indicate the number of idle reasoning devices in the test cluster; furthermore, the first real-time resource information is actually the number of idle reasoning devices (first resource information) obtained in real time in the process of trying to allocate reasoning devices for a certain test task, so as to reflect the dynamic changes in the resource allocation process. Step S202 is expected to solve the problem of how to reasonably allocate reasoning devices to each test task according to the test priority and real-time resource status of the test task when resources are limited (i.e., the remaining resources are not enough to meet the needs of all test tasks, but exceed the preset minimum allocation unit). By giving priority to high-priority tasks and updating resource information in real time during the allocation process, it is ensured that resources are efficiently and fairly allocated.

[0065] In some embodiments, in the process of implementing resource allocation, the system will first compare the current remaining resources with the total amount of inference device requirements of all test tasks to determine whether the resources are in a state of "insufficient but exceeding the preset minimum allocation unit". Subsequently, the system will sort the test tasks from high to low according to their test priority. For each sorted test task, the system will check the number of inference devices required based on the real-time remaining resource information and allocate the corresponding resources to it. After each allocation, the system will immediately update the remaining resource information to ensure that the resource status is accurate when subsequent tasks are allocated.

[0066] In some embodiments, when the first real-time resource information is less than the first parameter of the current test priority, a first waiting state is entered and the first real-time resource information is monitored; when the first real-time resource information meets the first parameter of the current test priority, a corresponding number of inference devices are allocated to the test task of the current test priority.

[0067] In an embodiment of the present application, in the process of allocating a corresponding number of reasoning devices to each of the test tasks based on the first real-time resource information and the first parameter of each of the test tasks in turn, there may be a situation where the first real-time resource information is less than the first parameter of the current test priority. In the current case, it is necessary to enter a first waiting state, in which the system will not immediately allocate resources to the test task, but will wait for the resources to become available. It is understandable that the current test priority refers to the test priority of the test task currently being considered for allocation (the test task with no allocated resources and the highest test priority) in the process of allocating resources.

[0068] In the above embodiment, when the system detects that the idle resources are insufficient to meet the needs of the current test task, it does not immediately reject the task or allocate insufficient resources, but enters a waiting state and continuously monitors the changes in resources. Once the resources become sufficient, the system will immediately allocate the required number of inference devices to the task. In this way, in the case of limited resources, it can be achieved that high-priority test tasks can obtain the required inference device resources first.

[0069] Exemplarily, when the test cluster has 10 idle GPUs and the preset minimum allocation unit is 2 GPUs, it faces three test tasks (Task A requires 5 GPUs, high priority; Task B requires 6 GPUs, medium priority; Task C requires 3 GPUs, low priority). The above resource allocation process may include: the system checks the current first real-time resource information, that is, 10 idle GPUs, and compares this number with the first parameter of each test task (that is, the number of required GPUs). For Task A, since it requires 5 GPUs and has the highest priority, the system will try to allocate it. Since 10 GPUs are greater than the 5 GPUs required by Task A, the system will meet the needs of Task A and allocate 5 GPUs to it, leaving 5 GPUs. Next, the system will consider Task B. Since Task B requires 6 GPUs, and the current number of remaining GPUs is 5, it is not enough to meet the needs of Task B. Therefore, the system will enter the first waiting state and continuously monitor the changes in the first real-time resource information, that is, waiting for other tasks to release GPU resources or new GPU resources to be added to the cluster. During the waiting process, if the system detects that there are enough GPU resources released so that the number of idle GPUs reaches or exceeds 6, the system will immediately allocate the required 6 GPUs to Task B. Finally, the system will consider Task C. Since Task C has the lowest priority and requires 3 GPUs, and the system cannot allocate enough resources to Task C when the requirements of Task B have not been met. Therefore, Task C will also be placed in the waiting queue, waiting for resources to become available.

[0070] In the above embodiment, it can be achieved that high-priority test tasks can obtain resources first, thereby improving the efficiency of resource utilization and task execution. Through the management of the waiting state, the situation of immediately rejecting tasks when resources are insufficient is avoided, and dynamic monitoring and allocation of resources are achieved. The task allocation strategy can be adjusted in time according to resource changes, thereby improving the response speed and stability of the system.

[0071] In some embodiments, the remaining resource information also includes second resource information, and the second resource information is used to indicate the remaining reasoning resources of each of the reasoning devices; the test parameters include a second parameter, and the second parameter is used to characterize the amount of reasoning resources required to execute the test task; and the allocation of reasoning devices to each of the test tasks based on the remaining resource information and the test priority of each of the test tasks also includes step S203.

[0072] Step S203: When the first resource information is less than the preset number of devices and the time to enter the first waiting state exceeds the first time threshold, the corresponding amount of inference resources is allocated to each test task to be assigned in order of test priority from high to low, based on the second real-time resource information and the second parameter of each test task to be assigned.

[0073] The second real-time resource information is real-time second resource information in response to allocating the amount of inference resources for the current test priority.

[0074] In the embodiment of the present application, the above-mentioned remaining resource information is used to describe the resource status of each inference device (such as GPU) in the system that is currently not in use. Among them, the second resource information specifically refers to the remaining video memory resources of these inference devices, that is, the video memory space in the GPU that has not been occupied by other test tasks. The second resource information reflects the available status of the video memory of the inference device before processing a new task. The test parameters are a set that describes the resource requirements of the test task, and the second parameter specifically represents the amount of video memory resources required to execute the test task.

[0075] In step S202, resource allocation is based on the number of idle inference devices (first real-time resource information) and the number of inference devices required for the test task (first parameter). However, in the case of tight resources, this allocation method may result in uneven distribution of video memory resources or the test task cannot be allocated in time. Therefore, when the first real-time resource information (i.e., the number of idle inference devices) is less than the preset number of devices and the system has entered the first waiting state, in order to improve the flexibility and efficiency of allocation, the system will change the allocation condition from the number of idle inference devices to the idle video memory resources (i.e., the second real-time resource information and the second parameter).

[0076] In some embodiments, when the system detects that the first real-time resource information is less than the preset number of devices, resulting in the inability to meet the task requirements and entering the first waiting state, it will check whether the video memory requirements (second parameters) of the current highest priority task match the real-time available video memory resources (second real-time resource information) according to the test priority order of the test task. If they match, the corresponding video memory resource amount is immediately allocated to the task; if they do not match, the system enters the second waiting state and continuously monitors the changes in the second real-time resource information until the video memory resources meet the video memory requirements of the current highest priority task.

[0077] Exemplarily, assume that a test cluster contains 4 GPUs, each with 8GIB of video memory. At this point, there are two test tasks to be assigned: Task A requires at least 2 GPUs (actually requires 8GIB of video memory) and has a high priority; Task B requires 1 GPU (4GIB of video memory) and has a low priority. First, the number of idle GPUs is checked as the first resource information, and it is found that there are no idle GPUs (that is, the first resource information is 0, which is less than the preset number of devices of at least 2 GPUs required for Task A). Therefore, the system enters the first waiting state and continuously monitors the remaining video memory of each GPU as the second real-time resource information (assuming that GPU1 to GPU4 all have 3GIB of video memory remaining). During the waiting period, if the system detects that the waiting time exceeds the preset first time threshold (for example, 10 minutes) and no idle GPUs are released, the system will allocate resources from high to low according to the test priority of the test task. That is, resources are allocated to task A first. Based on the second parameter required by task A (the amount of inference resources required to execute the test task, i.e., 8GIB of video memory), 8GIB of video memory can be allocated to task A in the second real-time resource information (GPU1 to GPU4 all have 3GIB of video memory remaining, for a total of 12GIB of video memory remaining). For example, 3GIB of video memory in GPU1 and GPU2 can be allocated to task A, and 2GIB of video memory in GPU3 can be allocated to task A. In this way, 8GIB has been allocated to task A, meeting the running requirements of task A.

[0078] In some embodiments, when the second real-time resource information is less than the second parameter of the current test priority, a second waiting state is entered and the second real-time resource information is monitored; when the second real-time resource information meets the second parameter of the current test priority, a corresponding amount of inference resources is allocated to the test task of the current test priority.

[0079] Continuing with the above example, in the first waiting state, assuming that GPU1 to GPU4 all have 1GIB of video memory remaining, it can be seen that the second real-time resource information is that the current test cluster has a total of 4GIB of video memory remaining. In the process of allocating resources for Task A, since the 4GIB of video memory remaining is less than the 8GIB of video memory required by Task A, it is necessary to enter the second waiting state.

[0080] In some embodiments, after entering the second waiting state, continue to monitor the second real-time resource information; when the second real-time resource information meets the second parameter of the current test priority, allocate the corresponding reasoning resource amount to the test task of the current test priority; when the second real-time resource information is less than the second parameter of the current test priority and the time to enter the second waiting state exceeds the first time threshold, stop the allocation process.

[0081] In the embodiment of the present application, when video memory resources are tight, the present application can make dynamic adjustments based on real-time video memory resources and task priorities to ensure that high-priority tasks get video memory resources first, while reducing idleness and waste of video memory resources. In addition, by introducing the second waiting state, the system can more accurately control the video memory resource allocation process, further improving video memory utilization and the success rate of task execution.

[0082] In some embodiments, the test parameters include a third parameter, and the third parameter is used to determine the number of parallel processes of the test task; the third parameter is related to the allocation process of the reasoning device.

[0083] Among them, the third parameter is used to determine the number of parallel processes of the test task. The number of parallel processes refers to the number of test task instances or computing tasks running simultaneously. It can be understood that different numbers of parallel processes can correspond to different execution speeds and resource consumption of test tasks, and thus can adapt to different test requirements and environmental conditions.

[0084] The allocation process of the inference device refers to the process of allocating the inference device to the test task in the above steps S201 to S203. The third parameter (number of parallel processes) is closely related to the allocation process of the inference device. In an embodiment of the present application, when the remaining resource information of the allocation process indicates that there are more remaining resources in the test cluster, the third parameter can be set to a larger value, so that the test task will be executed with a higher degree of parallelism; on the contrary, when the remaining resource information of the allocation process indicates that there are fewer remaining resources in the test cluster, the third parameter can be set to a smaller value, thereby reducing the degree of parallelism of the test task.

[0085] In some embodiments, when the inference devices of the test task are allocated based on the number of inference devices required for the test task, the third parameter of the test task is the first parallel number; when the inference devices of the test task are allocated based on the amount of inference resources required for the test task, the third parameter of the test task is the second parallel number; the first parallel number is greater than the second parallel number.

[0086] Among them, when the inference device of the test task is allocated based on the number of inference devices required for the test task, it means that there are completely idle inference devices in the current test cluster, which can further indicate that there are more remaining inference resources in the current test cluster and can support a higher degree of parallelism. Therefore, the third parameter of the test task can be set to a larger first parallel number; in this way, the computing power of the inference device can be maximized and the execution efficiency of the test task can be improved.

[0087] Among them, the allocation based on the number of inference devices is different. When the inference device of the test task is allocated based on the amount of inference resources required by the test task, it means that the inference device resources in the current test cluster are relatively tight. In this case, due to resource limitations, it may not be possible to support a high degree of parallelism like the first strategy. In order to avoid excessive consumption and waste of resources, the system will set the third parameter (number of parallel processes) of the test task to a smaller second parallel number. In this way, while achieving the parallel execution of test tasks, it is possible to reasonably control resource consumption and optimize resource utilization.

[0088] Since the allocation strategy based on the number of inference devices usually occurs when there are more remaining resources, and the allocation strategy based on the amount of inference resources is more conservative to adapt to resource constraints, the first parallel number is greater than the second parallel number.

[0089] In an embodiment of the present application, the number of parallel processes is dynamically adjusted according to the requirements of the test task and the allocation method of the inference device, thereby improving the execution efficiency and resource utilization of the test task. When allocating based on the number of inference devices, by increasing the number of parallel processes, the computing power of multiple inference devices is fully utilized; when allocating based on the amount of inference resources, by reducing the number of parallel processes, over-allocation and waste of resources are avoided. This dynamic adjustment mechanism enables the system to respond to the needs of different test tasks more flexibly, improving the overall test efficiency and resource utilization.

[0090] The first parallel number is the maximum number of the test tasks that can be run in parallel in the target number of inference devices; the second parallel number is determined based on the first parallel number and business conditions.

[0091] In some embodiments, the first parallel number refers to the maximum number of processes that can run in parallel in the test task when a set number of inference devices (such as graphics cards) are used, and these processes can normally complete the inference task without causing resources (such as video memory) to be exhausted or performance to be significantly reduced. Exemplarily, taking the yolov3 model in the PyTorch framework as an example, the test can be performed by starting multiple processes and gradually increasing the number of processes. During the test, the operation of the yolov3 model under each number of processes is recorded. When the number of processes increases to a certain value, if continued increase causes the video memory to be full or the model to run unstably, then the maximum number of processes that can run normally is used as the maximum number of parallel processes of the yolov3 model on the inference device, that is, the first parallel number.

[0092] In some embodiments, the second parallel number is a smaller number of parallel processes set based on the first parallel number and business conditions. The above business conditions may include, but are not limited to: the urgency of the test task, resource budget, execution time requirements, etc. Exemplarily, when the maximum number of parallel processes (first parallel number) is not less than 4, half of the maximum number of parallel processes can be set as the minimum number of parallel processes (second parallel number) to maintain a certain degree of parallelism when resources are sufficient. However, when the maximum number of parallel processes is less than 4, in order to avoid performance degradation due to too low a degree of parallelism, the minimum number of parallel processes can be set to 2. Of course, these settings can be adjusted according to actual business needs and resource conditions.

[0093] Figure 3 This is a schematic diagram of the implementation process of a reasoning device testing method provided in an embodiment of the present application. Figure 3 , the method can be executed by a processor of a computer device. Figure 1 , Figure 1 Step S103 in can be updated to step S301 to step S302, combining Figure 3 The steps shown are explained.

[0094] Step S301: Create a test unit corresponding to the test task based on the inference device assigned to the test task.

[0095] In the embodiment of the present application, one test priority corresponds to one test task. Accordingly, when a test unit is created for a test task, one test priority corresponds to one test unit.

[0096] The above-mentioned test unit refers to an independent test instance set for a test task (test priority), that is, different test units are independent of each other.

[0097] In some embodiments, the test unit created for the test task has the inference resources allocated for the test task. In the allocation process based on the inference device, the inference resources owned by the test unit created for the test task are at least one idle inference device allocated for the test task; in the allocation process based on the amount of inference resources, the inference resources owned by the test unit created for the test task are the remaining inference resources (video memory) allocated for the test task. It can be understood that the allocated remaining inference resources can be located in the same inference device or in multiple inference devices.

[0098] Step S302: The test unit executes the test tasks in parallel based on the third parameter to obtain the test result.

[0099] In some embodiments, after creating the test unit, the software objects to be tested can be installed for each test unit, such as graphics card driver, runtime environment, operator library dnn, machine learning library (such as PyTorch, TensorFlow plug-in, etc.). At the same time, configure test data (such as neural network model weight file, test set) and test code, and select or integrate test display tools (such as pytest, allure, influxdb and grafana, etc.) to automatically generate test reports and visually monitor test data.

[0100] In an embodiment of the present application, the third parameter is used to determine the number of parallel processes of the test task. During the execution process, a corresponding number of parallel processes can be configured according to the number of parallel processes of the test task, and the test task can be executed in parallel based on the configured parallel processes to obtain the test results. During the parallel test execution process, the test results of each process in the test unit can be collected, and then the test results for the reasoning device under the test task can be determined. Combined with the test results of each test task, the final test result for the reasoning device can be obtained.

[0101] In some embodiments, after executing a test task through a process, the test result obtained can be used to characterize whether the reasoning device meets the preset requirements; wherein, the test result can be determined based on the running result obtained after the process runs the test task, or can be determined based on the running time of the process running the test task, or can be determined based on the resource usage when the process runs the test task. After obtaining the test result corresponding to each process, the test result for the reasoning device under the test task can be obtained based on the relevant statistical algorithm. For example, when the proportion of the test results of the process that meets the preset requirements exceeds the preset proportion, the test result for the reasoning device under the test task is determined to meet the preset requirements; then, based on the relevant statistical algorithm, combined with the test results of each test task, the final test result for the reasoning device is obtained.

[0102] In some embodiments, the test task is used to execute the reasoning task of at least one model to be inferred, and the third parameter includes the number of parallel operations corresponding to each model to be inferred; the above-mentioned test result can be obtained by executing the test task in parallel based on the third parameter by the test unit through steps S3021 and S3022.

[0103] Step S3021: In the process of executing the reasoning tasks of each model to be inferred, the test unit executes the reasoning process of the model to be inferred in parallel based on the parallel quantity corresponding to the model to be inferred, so as to obtain the reasoning result of the model to be inferred.

[0104] In some embodiments, the model to be inferred is a machine learning or deep learning model that requires reasoning calculations; the parallel number refers to the number of reasoning processes or threads that can run simultaneously when executing the reasoning task; the test task is used to execute at least one reasoning task of the model to be inferred in sequence; that is, after the test unit completes the reasoning task of one model to be inferred, it will start the reasoning task of the next model to be inferred.

[0105] In some embodiments, the third parameter includes the number of parallel operations corresponding to each of the models to be inferred, and the number of parallel operations corresponding to different models to be inferred may be the same or different.

[0106] For example, assume that the test task includes Model 1 and Model 2, where Model 1 requires 3 parallel processes to perform the reasoning task, and Model 2 requires 4 parallel processes. When the test unit performs the reasoning task of Model 1, it will first create 3 parallel processes, and simultaneously perform the reasoning calculation of Model 1 in these 3 processes. When the reasoning task of Model 1 is completed, the test unit will release the resources of these 3 processes, and create 4 new parallel processes according to the needs of Model 2 to perform the reasoning task of Model 2.

[0107] Step S3022: Generate the test result based on the inference result of each model to be inferred.

[0108] Among them, the above-mentioned reasoning results are the output data or results generated by the model to be inferred after performing the reasoning task.

[0109] In some embodiments, after executing the reasoning task of each model to be inferred, the test unit will collect the reasoning results of the model. It can be understood that the number of collected reasoning results is the same as the parallel number of the model to be inferred. For example, assume that the test task includes model 1 and model 2, where model 1 requires 3 parallel processes to perform the reasoning task, and model 2 requires 4 parallel processes. After completing the test task, 3 reasoning results corresponding to model 1 and 4 reasoning results corresponding to model 2 can be obtained.

[0110] In some embodiments, the inference results obtained by each model to be inferred can be compared with the preset standard results to determine whether each inference result meets the requirements. After the statistics are completed, it can be obtained whether the inference device meets the requirements when executing the inference process of each model to be inferred. It can also be obtained whether the inference device meets the requirements when executing each test task. Finally, the test result of whether the inference device meets the preset requirements can be obtained.

[0111] In some embodiments, generating the test result based on the inference result of each model to be inferred includes: generating the test result based on the inference result of each model to be inferred and the standard result of each model to be inferred.

[0112] Among them, the standard result is the result obtained by reasoning the model to be inferred in an inference device that meets the requirements; and the test result is used to determine whether the inference device meets the requirements.

[0113] In the embodiment of the present application, the standard result is the result obtained after executing the reasoning process for a certain model to be inferred in an inference device that meets the requirements. This result is used as a benchmark to evaluate the performance or accuracy of other inference devices when performing the same reasoning task under the same conditions. It is understandable that the requirements here can be standards set based on actual business needs, which is not limited by this application.

[0114] It is understandable that the above standard results can be obtained by executing the reasoning process of the model to be inferred in the same operating environment as the reasoning device to be tested (ensuring that all environmental parameters such as other hardware configurations, software versions, etc., except for the reasoning device itself, remain consistent), and the obtained reasoning result is used as the standard result of the model to be inferred. Subsequently, the same reasoning process is executed on the reasoning device to be tested, and the obtained reasoning result is compared with the standard result generated previously. A comparison result is generated based on the comparison result, and the comparison result is used to evaluate whether the reasoning device meets the preset performance or accuracy requirements during the reasoning process of the model to be inferred.

[0115] Exemplarily, assuming that there is an image recognition model as the model to be inferred, in order to generate the standard result of the model, the model can be run on an inference server known to meet the requirements, and its inference result can be recorded, wherein the inference result can be the output data of the model to be inferred for a specific sample, or it can be the accuracy rate for multiple samples (for example, 95%). Subsequently, the same image recognition model is run on the inference device to be tested, and the output data for a specific sample and / or the recognition accuracy rate obtained are compared with the standard result (the inference result on the inference server known to meet the requirements). If the recognition accuracy rate on the device to be tested is close to 95%, it is considered to meet the requirements, or if the similarity between the output data for a specific sample on the device to be tested and the output data for a specific sample on the inference server meets the preset threshold, it is considered to meet the requirements.

[0116] In the embodiment of the present application, it is possible to objectively and accurately evaluate whether the inference device meets the preset requirements. By comparing the difference between the actual inference results and the standard results, the performance or accuracy level of the inference device can be quantitatively reflected, thereby providing strong data support for the subsequent performance optimization of the inference device.

[0117] In some embodiments, the test parameters include a fourth parameter, and the fourth parameter is used to determine the number of loop executions of the test task.

[0118] In some embodiments, the number of loop executions is positively correlated with the test priority of the test task; in other embodiments, the number of loop executions is negatively correlated with the time consumption of the test task; in yet other embodiments, the number of loop executions is positively correlated with the test priority of the test task and negatively correlated with the time consumption of the test task.

[0119] Taking the example that the number of loop executions is positively correlated with the test priority of the test task and negatively correlated with the time consumption of the test task, before executing the test task, the number of loop executions can be calculated based on the test priority and time consumption of the test task.

[0120] For example, suppose there are three test tasks, task A has a high priority and a short time consumption, task B has a medium priority and a medium time consumption, and task C has a low priority and a long time consumption. According to the setting of the fourth parameter, task A may obtain the highest number of loop executions, such as 10 times; task B is second, such as 5 times; and task C is the least, such as 2 times. In this way, under limited test resources, task A with high priority and short time consumption can get more test opportunities, thereby improving the overall test efficiency.

[0121] In some embodiments, the test task may include at least one model to be inferred. Accordingly, the fourth parameter may actually include the number of loop inferences of each model to be inferred.

[0122] In an embodiment of the present application, by setting the fourth parameter, the number of cyclic executions of the test task is combined with its priority and time consumption, so as to implement a strategy in which the higher the priority and the shorter the time consumption, the more times the task is executed, thereby improving the test efficiency.

[0123] Figure 4 This is a schematic diagram of the implementation process of a reasoning device testing method provided in an embodiment of the present application. Figure 4 , the method can be executed by a processor of a computer device. Figure 1 The method further comprises step S401, combining Figure 4 The steps shown are explained.

[0124] Step S401: Update the test parameters of the test task based on the execution data of the test task.

[0125] In the embodiment of the present application, the previously set test parameters can be updated based on the actual execution of the test task. The execution data of the above test task is the data obtained after the test cluster executes the test task.

[0126] In some embodiments, the above-mentioned test parameters may include at least one of the following: a third parameter and a fourth parameter; wherein the third parameter is used to determine the number of parallel processes of the test task, and the fourth parameter is used to determine the number of loop executions of the test task. By dynamically adjusting the test parameters (the third parameter and the fourth parameter), the execution mode and scale of the test task can be more flexibly controlled to adapt to different test requirements and scenarios. The adjustment of the third parameter can optimize the parallelism of the test task and improve the test efficiency; the adjustment of the fourth parameter can simulate the system performance under long-term operation and evaluate the stability and durability of the system. Through the synergistic effect of these two parameters, it is possible to achieve refined control of the test task execution process and improve the accuracy and effectiveness of the test.

[0127] In some embodiments, during the execution of the test task, the system collects and analyzes the execution data of the test task in real time, including key indicators such as response time, error rate, and resource utilization. Based on these data, the third parameter and the fourth parameter can be automatically adjusted. For example, if it is found that the system performs poorly under high concurrency, the third parameter value can be appropriately reduced, that is, the number of parallel processes can be reduced to reduce the burden on the system; if it is desired to evaluate the stability of the system under long-term operation, the fourth parameter value can be increased, that is, the number of loop executions of the test task can be increased.

[0128] In the embodiment of the present application, by dynamically adjusting the test parameters (third parameter and fourth parameter) of the test task, it is possible to achieve refined control over the test task execution process and improve the accuracy and effectiveness of the test. At the same time, it is also possible to balance the relationship between test efficiency and resource consumption based on historical execution data to avoid wasting test resources.

[0129] In some embodiments, the execution data includes a plurality of execution sample data; and the test results can be obtained by executing the test tasks in parallel by the test unit based on the third parameter through steps S4011 and S4012.

[0130] Step S4011: determine upper confidence limit data of multiple execution sample data of the test task within a preset confidence interval.

[0131] In some embodiments, the above-mentioned execution sample data may include at least one of the following: time consumption data and resource usage data. The upper confidence limit data is the upper limit value of multiple execution sample data within a preset confidence interval at a given confidence level. In statistics, a confidence interval is an interval estimate of a population parameter, which shows the extent to which the true value of this parameter has a certain probability of falling around the measurement result. The higher the confidence level, the wider the interval, and vice versa, the narrower it is, wherein the upper confidence limit and the lower confidence limit represent the upper endpoint and the lower endpoint of the confidence interval, respectively.

[0132] In some embodiments, the upper confidence limit is related to an average value, a fluctuation, and an execution timestamp of a plurality of execution sample data.

[0133] In some embodiments, determining the upper confidence limit data of multiple execution sample data of the test task within a preset confidence interval includes: determining the average execution data of the multiple execution sample data based on the execution timestamp of each execution sample data; the influence weight of the execution sample data on the average execution data is negatively correlated with the time difference between the execution timestamp and the current time; and determining the upper confidence limit data using the average execution data, the confidence level, the fluctuation data of the multiple execution sample data, and the number of samples of the multiple execution sample data.

[0134] In some embodiments, firstly, based on the execution timestamp of each execution sample data, the average execution data of multiple execution sample data is calculated, and the influence weight of each execution sample data on the average execution data is assigned, and this weight is negatively correlated with the time difference between the execution timestamp of the execution sample data and the current time, that is, the more recent the data is, the greater the influence on the average execution data. Then, using the calculated average execution data, the preset confidence level, the fluctuation data of the execution sample data and the number of samples, the upper confidence limit data is calculated by statistical methods.

[0135] In some embodiments, the above average execution data can be calculated by a weighted average method, which includes: setting the execution timestamp to have a smaller time difference from the current time, and the execution sample data has a larger weight. For example, if there are n days of data, the weight of the execution sample data of the most recent day is n, followed by n-1, and so on, until the weight of the execution sample data of the furthest day is 1. The calculation formula of the weighted average method is shown in formula (1).

[0136]

[0137] in, is the average execution data, w i is the weight of the i-th execution sample data, x i is the value of the i-th execution sample data, and n is the number of execution sample data.

[0138] In some other embodiments, the above average execution data can also be calculated by exponential smoothing method, which includes: setting the average execution data to be most affected by the execution sample data of the previous day, and then decreasing in sequence. Setting the smoothing coefficient α, the weight of the most recent day is α, the weight of the previous day is α·(1-α), and so on. The closer α is to 1, the higher the weight of recent data. The calculation formula of the exponential smoothing method is shown in formula (2).

[0139] S t =α·y t +(1-α)·S t-1 Formula (2);

[0140] Among them, S t is the smoothed value calculated at the tth iteration, α is the smoothing coefficient, y t is the sample data of the execution calculated for the tth iteration, S t-1is the smoothing value calculated at the t-1th iteration. In actual calculation, the first execution sample data among the multiple execution sample data is used as the initial smoothing value. Starting from the second execution sample data among the multiple execution sample data, the following iterative calculation is performed: the current execution sample data is multiplied by the smoothing coefficient α, and then the previous smoothing value is multiplied by (1-α), and the calculated new smoothing value is used as the smoothing value calculated for this iteration; the iterative process of the next execution sample data is continued until the calculation of multiple execution sample data is completed, and the final smoothing value is used as the average execution data.

[0141] In some embodiments, the fluctuation data of the multiple execution sample data may be a standard deviation; the above-mentioned determination of the upper confidence limit data using the average execution data, the confidence level, the fluctuation data of the multiple execution sample data and the number of samples of the multiple execution sample data may be implemented by formula (3).

[0142]

[0143] Among them, conf is the upper confidence limit data, is the average execution data, Z α / 2 is the quantile corresponding to the critical value under the standard normal distribution (for example, for a 95% confidence level, Z α / 2 ≈1.96), σ is the fluctuation data (standard deviation) of the multiple execution sample data, and n is the sample size of the data on the execution sample.

[0144] Step S4012: Update the test parameters of the test task using the test priority and upper confidence limit data of the test task.

[0145] In the embodiment of the present application, the test parameters of the test task can be dynamically adjusted based on the statistical characteristics of the execution sample data, thereby improving the test efficiency while ensuring the test accuracy. The present application can flexibly adjust the test strategy according to different test requirements and data characteristics, effectively deal with the uncertainty in the test process, and improve the overall efficiency of the test work.

[0146] In some embodiments, the execution sample data includes resource usage data; the test parameters include a third parameter, and the third parameter is used to determine the number of parallel processes of the test task; the method of updating the test parameters of the test task using the test priority and upper confidence limit data of the test task includes: determining the updated third parameter based on the inference resource amount of the inference device and the upper confidence limit of the resource usage data.

[0147] The inference resource amount of the inference device is the video memory capacity of the inference device. The updated third parameter can be set as the quotient of the inference resource amount of the inference device and the upper confidence limit of the resource usage data.

[0148] Exemplarily, the updated third parameter can be obtained by formula (4).

[0149]

[0150] Where D is the updated third parameter, G_MAX is the inference resource amount of the inference device, g conf is the upper confidence limit for the resource usage data.

[0151] In some embodiments, the third parameter includes a first parallel number and a second parallel number; the determining of the updated third parameter based on the inference resource amount of the inference device and the upper confidence limit of the resource usage data includes: determining the first parallel data based on the inference resource amount of the inference device and the upper confidence limit of the resource usage data; determining the second parallel number based on the quotient of the first parallel number and a preset adjustment parameter and the minimum parallel number.

[0152] The first parallel number can be determined by formula (5), and the second parallel number can be determined by formula (6).

[0153]

[0154]

[0155] Wherein, Parallel_max_dst is the new first parallel data, G_MAX is the inference resource amount of the inference device, and g conf is the upper confidence limit of the resource usage data; Parallel_min_dst is the new second parallel number, k is the minimum parallel number, and p is the preset adjustment parameter.

[0156] In some embodiments, the updated third parameter is actually the new number of parallel operations for each model to be inferred in the test task. Accordingly, the parameters used in calculating the number of parallel operations are also the parameters of the model to be inferred, such as resource usage data, the upper confidence limit of resource usage data, the minimum number of parallel operations, and p is a preset adjustment parameter.

[0157] It should be noted that, in the process of calculating the updated third parameter, it is not necessary to refer to the test priority of the test task.

[0158] In the embodiment of the present application, the maximum inference resource (G_MAx) of the inference device can be calculated according to the upper confidence limit (g conf) to allocate reasonably; and, since the upper confidence limit of resource usage data (g conf ) is dynamically changed based on actual execution data. Therefore, the present application can dynamically adjust the third parameter (the first parallel number and the second parallel number) according to the current resource demand situation, thereby maintaining the flexibility and responsiveness of the system. At the same time, since the minimum parallel number (k) is set, even in the most extreme resource allocation scenario, there will be a minimum parallel number (k) guarantee, thereby maintaining the parallel execution process of the test cluster for the test task.

[0159] In some embodiments, the execution sample data includes time-consuming data; the test parameters include a fourth parameter, and the fourth parameter is used to determine the number of cyclic executions of the test task; the method of updating the test parameters of the test task using the test priority and upper confidence limit data of the test task includes: determining the updated fourth parameter using the maximum test duration, the test priority of the test task and the upper confidence limit of the time-consuming data.

[0160] Among them, the time-consuming data is the execution time of the test task, which can be in seconds. In the case where the test task includes at least one model to be inferred, the time-consuming data actually includes the execution time of each model to be inferred. In some embodiments, the above-mentioned maximum test duration is the maximum duration for which model reasoning can be tested in parallel for stability, in seconds. Exemplarily, the maximum test duration can be the product of the maximum daily running time and the resource utilization rate, wherein the maximum daily running time can be set to 86400 seconds (i.e. 1 day), and the resource utilization rate can be set to 0.5, so that the maximum test duration obtained is 43200 seconds (i.e. half a day).

[0161] In some embodiments, a distribution weight can be set in advance for each test priority, and the higher the test priority, the greater the distribution weight. Exemplarily, the distribution weight of high priority can be set to 1.5, the distribution weight of medium priority can be set to 1, and the distribution weight of low priority can be set to 0.5.

[0162] In some embodiments, the above-mentioned determination of the updated fourth parameter by utilizing the maximum test duration, the test priority of the test task and the upper confidence limit of the time-consuming data can be implemented by formula (7).

[0163]

[0164] Among them, Loop_dst is the updated fourth parameter, T is the maximum test duration in seconds, weight is the allocation weight set according to the test priority, t conf is the upper confidence limit for time usage.

[0165] In the embodiment of the present application, the updated fourth parameter is determined by using the maximum test duration, the test priority of the test task and the upper confidence limit of the time-consuming data, so that the test duration can be reasonably allocated according to the test tasks of different priorities, ensuring that high-priority tasks get more testing opportunities. For example, high-priority tasks will obtain more loop execution times due to their larger allocation weights, thereby improving their test adequacy; while low-priority tasks will be reduced accordingly, but still within an acceptable range. In addition, by setting the maximum test duration to the maximum duration of the model reasoning parallel stability test, such as half a day (43200 seconds), the test time can be effectively limited to avoid wasting test resources. At the same time, according to the priority of the test task and the upper confidence limit of the time-consuming data, the number of loop executions is dynamically adjusted to achieve the optimal configuration of test resources. The technical effect of the overall solution is to improve the test efficiency while ensuring the test quality.

[0166] In some embodiments, the test task is used to perform an inference task of at least one model to be inferred, and the time consumption data includes the model consumption of the model to be inferred; the updated fourth parameter is determined by using the maximum test duration of the test task, the test priority of the test task and the upper confidence limit of the time consumption data, including: determining the test duration of the test task by using the weight corresponding to the maximum test duration and the test priority of the test task; determining the average test duration based on the test duration and the number of models to be inferred; determining the number of cyclic inferences of the inference model by using the average test duration and the upper confidence limit of the model consumption of the model to be inferred.

[0167] Exemplarily, the number of loop reasoning of the reasoning model can be obtained by formula (8).

[0168]

[0169] Among them, T·weight is the test time of the test task, T·weight / model_count is the average test time, and Loop_dst is the number of loop inferences of the inference model.

[0170] In the embodiment of the present application, the average test time is calculated based on the test time and the number of models to be inferred, so that each model can obtain a relatively fair test opportunity, avoiding the problem of uneven resource allocation. Furthermore, the average test time and the upper confidence limit of the model time of the model to be inferred are used to determine the number of loop inferences of the inference model, which not only takes into account the average test requirements of the model, but also takes into account the uncertainty of the model time, thereby ensuring the stability and accuracy of the test process.

[0171] The following describes the application of the inference device testing method provided in an embodiment of the present application in a graphics card testing scenario.

[0172] When related technologies perform parallel testing on graphics cards through model reasoning, since different models may require different video memory, reasoning time, etc., using the same test parameters may result in insufficient video memory for some models or too long model reasoning time, resulting in the inability to complete the continuous integration task.

[0173] In some embodiments, the yolov3 model, yolov5s model, resnet50 model, vgg16 model, bilstm model and crnn model in the PyTorch framework and the yolov3 model, yolov5s model, resnet50 model, vgg16 model, bilstm model and crnn model in the TensorFlow framework can be used to test the graphics card. The following takes the example of using the yolov3 model, yolov5s model, resnet50 model, vgg16 model, bilstm model and crnn model in the PyTorch framework and the TensorFlow framework to test the graphics card sku1.

[0174] In some embodiments, before performing a graphics card test according to each model, the configuration parameters of each model may be determined first. The configuration parameters of each model may be recorded in a configuration file in a text file format. The configuration parameters may include the deep learning framework name (Framework), model name (Model), maximum number of parallel processes (Parallel_max), minimum number of parallel processes (Parallel_min), number of loops (Loop), test priority (Priority) and timeout (Timeout) corresponding to each model.

[0175] In some embodiments, the test priority of each model can be set according to business requirements. The test priority includes high priority, medium priority and low priority.

[0176] In some embodiments, the maximum number of parallel processes and the minimum number of parallel processes corresponding to each model can be determined based on the maximum number of parallel processes that each model can carry when performing graphics card testing and the actual requirements when performing graphics card testing.

[0177] Taking the yolov3 model in the PyTorch framework as an example, in some embodiments, multiple processes can be started, and the graphics card test can be performed through the yolov3 model in the PyTorch framework, and the number of processes can be increased successively until the video memory in each graphics card is full. In the process of increasing the number of processes, the corresponding maximum number of parallel processes on which the yolov3 model can run normally is used as the maximum number of parallel processes of the yolov3 model.

[0178] In some embodiments, when the maximum number of parallel processes is not less than 4, half of the maximum number of parallel processes may be set as the minimum number of parallel processes, and when the maximum number of parallel processes is less than 4, the minimum number of parallel processes may be set to 2. In some embodiments, the minimum number of parallel processes may also be set according to actual requirements when performing graphics card testing.

[0179] In some embodiments, after determining the maximum number of parallel processes and the minimum number of parallel processes corresponding to each model, the maximum number of parallel processes and the minimum number of parallel processes of each model can be recorded in a configuration file, and the configuration file can be named model_test_cfg_GPU_sku1.csv.

[0180] In some embodiments, the total duration of the graphics card test does not exceed 12 hours, and the single time consumption of each model for a graphics card test can be determined respectively, and the number of cycles for each model to perform the graphics card test can be determined in combination with the test priority of each model. It can be understood that when a graphics card test of a certain model takes a long time and the test priority of the model is low, the number of cycles corresponding to the model is small; when a graphics card test of a certain model takes a short time and the test priority of the model is high, the number of cycles corresponding to the model is large. After determining the number of cycles corresponding to each model, the number of cycles corresponding to each model can be recorded in a configuration file.

[0181] In some embodiments, the timeout time (in seconds) reserved for each model when performing the graphics card test can be determined based on the total time required for each model to perform the graphics card test for the number of cycles corresponding to the number of cycles. The timeout time of a model is the total time of the model multiplied by a preset redundancy parameter (such as 1.2), resulting in a timeout time greater than the total time during the test. After determining the timeout time corresponding to each model, the timeout time corresponding to each model can be recorded in a configuration file.

[0182] Take the video memory size of graphics card sku1 as 16GiB as an example. In some embodiments, the configuration parameter record in the configuration file is as follows Figure 5 As shown, Figure 5The first column records the framework name (i.e. the name of the deep learning framework), the second column records the model name, the third column records the minimum parallel process (i.e. the minimum number of parallel processes), the fourth column records the maximum parallel process (i.e. the maximum number of parallel processes), the fifth column records the number of loops, the sixth column records the priority (i.e. the test priority), and the seventh column records the timeout. Taking the second row as an example, the deep learning framework name recorded is PyTorch, the model name is yolov3, the minimum number of parallel processes is 8, the maximum number of parallel processes is 16, the number of loops is 20, the test priority is low priority, and the timeout is 6000 (in seconds); taking the fourth row as an example, the deep learning framework name recorded is TensorFlow, the model name is yolov5s, the minimum number of parallel processes is 8, the maximum number of parallel processes is 16, the number of loops is 25, the test priority is high priority, and the timeout is 5000 (in seconds).

[0183] In the embodiment of the present application, after determining the configuration parameters of each model, a test unit can be created through the test pod creation module. Figure 6 , which shows a flow chart for creating a test unit.

[0184] Step S601, query the resource information of all nodes in the kubernetes cluster, and record the number of idle graphics cards, the remaining video memory of each graphics card, the graphics card model, CPU and memory of each node.

[0185] Step S602: Match a graphics card node with sufficient video memory and computing power according to the test configuration requirements.

[0186] In some embodiments, the node resource information in the kubernetes cluster can be queried and recorded. Then, according to the test configuration requirements, a graphics card node with sufficient video memory and computing power is matched. The node resource information includes the number of idle graphics cards, remaining video memory, graphics card model, CPU and memory of each node.

[0187] In some embodiments, the graphics card test may be performed first according to the maximum number of parallel processes. The following will take the case where the graphics card test is performed according to the maximum number of parallel processes, where the high priority group requires a maximum of x graphics cards, the medium priority group requires a maximum of y graphics cards, and the low priority group requires a maximum of z graphics cards as an example. For ease of understanding, the following will be described separately in multiple scenarios.

[0188] Scenario 1: When the number of idle graphics cards is greater than or equal to the sum of x, y, and z, graphics card tests of the high priority group, medium priority group, and low priority group can be performed simultaneously.

[0189] Step S603: Perform resource configuration on the YAML files of the pods of the three test priority groups, and allocate x, y, and z graphics cards, as well as CPU, memory and other resources respectively.

[0190] Step S604: Create Pod_High_max, Pod_Medium_max, and Pod_Low_max for each test priority group at the same time.

[0191] Exemplarily, resources can be configured for the high priority group, the medium priority group, and the low priority group, and x, y, and z graphics cards are allocated to the high priority group, the medium priority group, and the low priority group, respectively, and resources such as CPU and memory are allocated. Test unit Pods can also be created for the high priority group, the medium priority group, and the low priority group, respectively. The Pod corresponding to the high priority group can be named Pod_High_max, the Pod corresponding to the medium priority group can be named Pod_Medium_max, and the Pod corresponding to the low priority group can be named Pod_Low_max. In some embodiments, the naming and properties of the Pod can be defined by a YAML file.

[0192] Scenario 2: The number of idle graphics cards is greater than 1 but less than the sum of x, y, and z. At least one priority group can be tested.

[0193] Step S605: Perform resource configuration for the Pod_High_max.yaml of the pod in the High test priority group, and allocate x graphics cards, as well as CPU, memory and other resources.

[0194] Step S606: Create a Pod for the High test priority group: Pod_High_max.

[0195] For example, it can be determined whether the number of idle graphics cards is not less than x. If the number of idle graphics cards is not less than x, resources can be configured for the high priority group, and x graphics cards, CPU, memory and other resources can be allocated to the high priority group. A test unit Pod can also be created for the high priority group, and the Pod corresponding to the high priority group can be named Pod_High_max.

[0196] Step S607: Perform resource configuration for the Pod_Medium_max.yaml of the pod of the Medium test priority group to allocate y graphics cards, as well as CPU, memory and other resources.

[0197] Step S608: Create a Pod for the Medium test priority group: Pod_Medium_max.

[0198] Next, you can determine whether the number of remaining idle graphics cards is not less than y. If the number of idle graphics cards is not less than y, you can configure resources for the medium priority group, allocate y graphics cards to the medium priority group, and allocate resources such as CPU and memory. You can also create a test unit Pod for the medium priority group, and name the Pod corresponding to the medium priority group Pod_Medium_max.

[0199] Step S609: Perform resource configuration for the Pod_Low_max.yaml of the pod in the Low test priority group to allocate z graphics cards, as well as CPU, memory and other resources.

[0200] Step S610: Create a Pod for the Low test priority group: Pod_Low_max.

[0201] Finally, it can be determined whether the number of remaining idle graphics cards is not less than z. If the number of idle graphics cards is not less than z, resources can be configured for the low-priority group, z graphics cards can be allocated to the low-priority group, and resources such as CPU and memory can be allocated. A test unit Pod can also be created for the low-priority group, and the Pod corresponding to the low-priority group can be named Pod_Low_max.

[0202] In other embodiments, Pod_High_max, Pod_Medium_max, and Pod_Low_max may be created in sequence according to the number of idle graphics cards. When the number of idle graphics cards is not less than x, Pod_High_max is created; when the number of idle graphics cards is less than x, it is determined whether the number of idle graphics cards is not less than y. Secondly, when the number of idle graphics cards is not less than y, Pod_Medium_max is created; when the number of idle graphics cards is less than y, it is determined whether the number of idle graphics cards is not less than z. Thirdly, when the number of idle graphics cards is not less than z, Pod_Low_max is created. When the number of idle graphics cards is less than z, queue up for test resources. When the waiting time reaches the preset timeout time, the graphics card test may be performed according to the minimum number of parallel processes.

[0203] Scenario 3: The number of idle graphics cards is less than 1, resources are relatively scarce, and only the remaining video memory is available. The test task needs to be tested according to the minimum resource requirements. Generally speaking, scenario 3 is executed after scenario 2 is not satisfied and the queue waiting for test resources times out. The following example will be used to illustrate the case where the graphics card test is performed according to the minimum number of parallel processes, and the high priority group requires a maximum of i (GiB) video memory, the medium priority group requires a maximum of j (GiB) video memory, and the low priority group requires a maximum of k (GiB) video memory.

[0204] Step S611: Perform resource configuration for the Pod_High_min.yaml of the pod in the High test priority group, and allocate i (GiB) video memory, as well as CPU, memory and other resources.

[0205] Step S612: Create a Pod for the High test priority group: Pod_High_min.

[0206] In some embodiments, it may be prioritized to determine whether the video memory of the remaining graphics cards is not less than i (GiB). In the case where the video memory of the remaining graphics cards is not less than i (GiB), resources may be configured for the high priority group, i (GiB) video memory may be allocated to the high priority group, and resources such as CPU and memory may be allocated. A test unit Pod may also be created for the high priority group, and the Pod corresponding to the high priority group may be named Pod_High_min. In some embodiments, the naming and properties of the Pod may be defined through a YAML file.

[0207] Step S613: Perform resource configuration for the Pod_Medium_min.yaml of the pod of the Medium test priority group, and allocate j (GiB) of video memory, as well as CPU, memory and other resources.

[0208] Step S614: Create a Pod for the Medium test priority group: Pod_Medium_min.

[0209] Next, you can determine whether the video memory of the remaining graphics cards is not less than j (GiB). If the video memory of the remaining graphics cards is not less than j (GiB), you can configure resources for the medium priority group, allocate j (GiB) video memory to the medium priority group, and allocate resources such as CPU and memory. You can also create a test unit Pod for the medium priority group, and name the Pod corresponding to the medium priority group Pod_Medium_min.

[0210] Step S615: Perform resource configuration for the Pod_Low_min.yaml of the pod in the Low test priority group, and allocate k (GiB) of video memory, CPU, memory and other resources.

[0211] Step S616, create a Pod for the Low test priority group: Pod_Low_min. Finally, it can be determined whether the video memory of the remaining graphics cards is not less than k (GiB). If the video memory of the remaining graphics cards is not less than k (GiB), resources can be configured for the low priority group, k (GiB) video memory can be allocated to the low priority group, and resources such as CPU and memory can be allocated. A test unit Pod can also be created for the low priority group, and the Pod corresponding to the low priority group can be named Pod_Low_min.

[0212] In other embodiments, Pod_High_min, Pod_Medium_min, and Pod_Low_min may be created in sequence according to the video memory of the remaining graphics card. When the video memory of the remaining graphics card is not less than i (GiB), Pod_High_min is created; when the video memory of the remaining graphics card is less than i (GiB), it is determined whether the video memory of the remaining graphics card is not less than j (GiB). Secondly, when the video memory of the remaining graphics card is not less than j (GiB), Pod_Medium_min is created; when the video memory of the remaining graphics card is less than j (GiB), it is determined whether the video memory of the remaining graphics card is not less than k (GiB). Thirdly, when the video memory of the remaining graphics card is not less than k (GiB), Pod_Low_min is created. When the video memory of the remaining graphics card is less than k (GiB), the test unit Pod is queued for test resources. When the waiting time reaches the preset timeout time, the test unit Pod is not created.

[0213] In the embodiment of the present application, after the test unit is created, the test can be started through the test running module.

[0214] In some embodiments, the test environment can be deployed first. For example, the software objects to be tested can be installed for the test unit Pod. The software objects include graphics card drivers, runtimes, operator libraries dnn, machine learning libraries PyTorch, TensorFlow plug-ins, etc. Test data, test codes, and test display tools can also be configured for the test unit Pod. Among them, the test data can be a neural network model weight file, a test set, and the like. The test display tool can be integrated with grafana through pytest, allure, influxdb, and the test display tool can realize the automatic generation of test reports and the visual monitoring of test data. The test display tool can also be selected or integrated according to actual needs, and this application does not specifically limit the test display tool.

[0215] Next, you can execute the graphics card test of each test unit Pod in turn. You can select the corresponding number of parallel processes according to the Pod configuration. For example, Pod_High_max corresponds to the maximum number of parallel processes, and Pod_High_min corresponds to the minimum number of parallel processes. In the process of executing the graphics card test of each test unit Pod in turn, each model needs to loop and execute the test a number of times corresponding to the number of loops.

[0216] After the test is finished, the inference accuracy data obtained from each graphics card test can be compared with the accuracy data of Golden, and the comparison results can be output. The average inference performance data and other test result data of each graphics card test can be recorded in a time series table. The time series table can be named historical_metrics_sku1.csv.

[0217] like Figure 7 As shown, Figure 7 A schematic diagram of a timing table provided for an embodiment of the present application. Among them, the first column records the framework name (i.e., the name of the deep learning framework), the second column records the model name, the third column records the date (i.e., the date of the graphics card test), the fourth column records the average time consumed (i.e., the average time consumed by the model to perform the graphics card test multiple times), the fifth column records the average video memory consumption (i.e., the average video memory consumption of the model to perform the graphics card test multiple times, in GiB), the sixth column records whether the test passes, the seventh column records the error message when the test fails, the eighth column records the Node id where the test Pod is located, and the ninth column records the GPU id applied for by the test Pod.

[0218] After the test is completed, the test results can be displayed through the above test display tool.

[0219] In an embodiment of the present application, after the test is completed, the inference parameters of all models can be dynamically updated and optimized through the model inference parameter update module.

[0220] In some embodiments, the model inference parameter updating module is used to update the maximum number of parallel processes Parallel_max, the minimum number of parallel processes Parallel_min, and the number of loops Loop.

[0221] In the process of determining the sample mean of computing time usage and video memory usage over a period of time, the sample mean of time usage and video memory usage over a period of time can be calculated by quantifying historical data, for example, a weighted average or exponential smoothing value can be used. Compared with the solution of using only the average value over a period of time, this application uses algorithms such as weighted average and exponential smoothing, so that the data of the last two days has a greater impact on the adjustment, while the early data has a smaller impact, and the parameters are updated more real-time and accurately.

[0222] For the weighted average method, the weight of the more recent data can be set to be greater. For example, if there are n days of data, the weight of the most recent data is n, the weight of the next is n-1, and so on until the weight of the most distant day's data is 1. The calculation formula of the weighted average method is shown in formula (9).

[0223]

[0224] in, is the weighted average, w i is the weight of the ith data point, x i is the value of the ith data point, and n is the number of data points.

[0225] For the exponential smoothing method, the current value can be set to be most affected by the data of the previous day, and then decrease in order. Set the smoothing coefficient α, the weight of the most recent day is α, the weight of the previous day is α·(1-α), and so on. The closer α is to 1, the higher the weight of recent data. The calculation formula of the exponential smoothing method is shown in formula (10).

[0226] S t =α·y t +(1-α)·S t-1 Formula (10);

[0227] Among them, S t is the smoothing value of the tth period, α is the smoothing coefficient, y t is the actual value of period t, S t-1 is the smoothed value of the t-1 period.

[0228] In the process of determining the upper confidence limits of the computing time usage and the video memory usage, respectively, this can be achieved through formula (11).

[0229]

[0230] conf is the upper confidence limit, is the sample mean of time or video memory usage, which can be calculated by the algorithm in historical data quantification. α / 2 is the quantile corresponding to the critical value under the standard normal distribution (for example, for a 95% confidence level, Z α / 2 ≈1.96), σ is the sample standard deviation of time usage and memory usage, and n is the sample size.

[0231] In the process of dynamically calculating new configuration parameters, the model test priority needs to be considered to more reasonably allocate resources for high-priority tasks. For example, the weight of high priority can be set to 1.5, the weight of medium priority can be set to 1, and the weight of low priority can be set to 0.5.

[0232] In combination with the above embodiments, the sample mean (including the sample mean of time memory usage and the sample mean of video memory usage) obtained by quantifying historical data and the above upper confidence limit calculation method can obtain the upper confidence limit of time memory usage and the upper confidence limit of video memory usage.

[0233] The new maximum number of parallel processes can be obtained through formula (12).

[0234]

[0235] Among them, Parallel_max_dst is the new maximum number of parallel processes, G_MAX is the GPU memory capacity, g confis the upper confidence limit of video memory usage.

[0236] The new minimum number of parallel processes can be obtained through formula (13).

[0237]

[0238] Among them, Parallel_max_dst is the new maximum number of parallel processes, and Parallel_min_dst is the new minimum number of parallel processes.

[0239] The new number of cycles can be obtained by formula (14).

[0240]

[0241] Among them, Loop_dst is the new number of loops, T_MAX is the maximum daily running time in seconds, UTILIZATION_RATE is the resource utilization rate. For this method, the total duration of the model inference parallel stability test is set to no more than 12 hours (which can be adjusted according to actual conditions), weight is the weight obtained according to the task priority, model_count is the total number of models, t conf is the upper confidence limit for time usage.

[0242] This application can be scientific, comprehensive, automated, intuitive, efficient, and reasonably use resources to perform performance and stability tests on neural network models, and then test the general computing capabilities of graphics cards. Evaluate whether the various indicators of the graphics card can meet business requirements, and optimize the graphics card hardware and software drivers based on the test results. The advantages of each module provided by this application will be explained below.

[0243] Model reasoning parameter configuration module: Taking into account that different deep learning AI frameworks and different models require different video memory for reasoning, using the same test parameters will cause some models to have insufficient video memory and be unable to complete daily CI tasks. The model reasoning parameter configuration module in this application can configure the test concurrency number differently according to the video memory consumption of different models; Taking into account that different deep learning AI frameworks and different models require different time for reasoning, using the same test parameters will cause some models to take too long and be unable to complete daily CI tasks. The model reasoning parameter configuration module in this application can configure the test cycle number differently according to the running time of different models; Taking into account that different graphics cards have different corresponding video memory sizes and different performance, it is impossible to apply the same set of parameters to different graphics cards for parallel stability testing. The model reasoning parameter configuration module in this application can generate independent model reasoning parameter configurations based on different models of graphics cards. In addition, the configuration file (model_test_cfg_GPU_skux.csv) in this application includes the parameters of all models with different requirements, reasonably allocates test resources for different models, and is also the basis for subsequent test resource scheduling. It can be understood that when implementing this application, not all models are put together, and then a fixed large number of graphics cards are applied for when applying for resources, but the models are grouped according to their priority and the number of graphics cards applied for is dynamically adjusted.

[0244] Test pod creation module: When some other CIs are running, only part of the video memory of the requested graphics card is used, so there is a situation where the video memory is not fully utilized. The test pod creation module in this application can adaptively adjust the test pod creation process and adaptively adjust resource requirements when resources are sufficient or scarce; traditional k8s scheduling is based on the total amount of resources, while the test pod creation module in this application is based on the priority of the model, giving priority to models with high priority, so that high-priority tasks can be completed first; traditional k8s scheduling will keep waiting if no resources are available, but here you can apply for the minimum resource after the maximum resource application times out, and there is a redundancy guarantee.

[0245] Model reasoning parameter update module: With the update of graphics card driver and other software versions, the performance of the model will change, and the model test parameters also need to change dynamically. The model reasoning parameter update module in this application can dynamically update and optimize the reasoning parameters of all models.

[0246] Therefore, it can be seen that the present application can be scientific, comprehensive, automated, intuitive, efficient, and reasonably use resources to perform performance testing and stability testing of neural network models, evaluate whether various indicators of the graphics card can meet business requirements, and optimize the graphics card hardware and software drivers based on the test results; the present application is based on general scenarios of general computing of graphics cards: "reasoning of neural network models", "parallel stability", and the testing methods are scientific and effective; the present application can efficiently and reasonably use the test cluster resources to avoid waste of resources; the present application can automatically update the test parameters to avoid the need for manual maintenance of the test parameters as the software versions such as the graphics card driver are updated; the present application can continuously optimize and improve the graphics card hardware and software drivers based on the test results.

[0247] Based on the foregoing embodiments, an embodiment of the present application provides an inference device testing device, which includes the units included and the modules included in the units, which can be implemented by a processor in a computer device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0248] Figure 8 A schematic diagram of the structure of an inference device test device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the inference device test device 800 includes: an acquisition module 810, an allocation module 820, and a test module 830, wherein:

[0249] The acquisition module 810 is used to acquire the remaining resource information of the test cluster and acquire the test parameters and test priority of each of the at least two test tasks; the remaining resource information is at least used to characterize the working state of the reasoning device in the test cluster;

[0250] An allocation module 820, configured to allocate an inference device to each of the test tasks based on the remaining resource information and the test parameters and test priority of each of the test tasks;

[0251] The testing module 830 is used to execute the test task based on the test parameters of the test task through the assigned reasoning device to obtain the test result corresponding to the reasoning device.

[0252] In some embodiments, the remaining resource information includes first resource information, and the first resource information is used to indicate the number of inference devices in the test cluster that are in an idle state; the test parameters include a first parameter, and the first parameter is used to characterize the number of inference devices required to execute the test task; the allocation module 820 is also used to: when the first resource information is greater than or equal to the total amount of inference device requirements, and based on the first parameter of each test task, allocate a corresponding number of inference devices to each test task; the total amount of inference device requirements is the sum of the first parameters of each test task.

[0253] In some embodiments, the allocation module 820 is also used for: when the first resource information is less than the total amount of the inference device demand and greater than the preset number of devices, in accordance with the order of test priority from high to low, based on the first real-time resource information and the first parameter of each test task, allocating a corresponding number of inference devices to each test task; the first real-time resource information is the real-time first resource information in response to the allocation of inference devices for the current test priority; wherein, when the first real-time resource information is less than the first parameter of the current test priority, entering a first waiting state and monitoring the first real-time resource information; when the first real-time resource information meets the first parameter of the current test priority, allocating a corresponding number of inference devices to the test task of the current test priority.

[0254] In some embodiments, the remaining resource information also includes second resource information, and the second resource information is used to indicate the remaining reasoning resources of each of the reasoning devices; the test parameters include a second parameter, and the second parameter is used to characterize the amount of reasoning resources required to execute the test task; the allocation module 820 is also used to: when the first resource information is less than the preset number of devices and the time to enter the first waiting state exceeds the first time threshold, in order of test priority from high to low, allocate corresponding reasoning resources to each test task to be allocated based on the second real-time resource information and the second parameter of each test task to be allocated; the second real-time resource information is real-time second resource information in response to allocating reasoning resources to the current test priority; wherein, when the second real-time resource information is less than the second parameter of the current test priority, enter the second waiting state and monitor the second real-time resource information; when the second real-time resource information meets the second parameter of the current test priority, allocate corresponding reasoning resources to the test task of the current test priority.

[0255] In some embodiments, the test parameters include a third parameter, and the third parameter is used to determine the number of parallel processes of the test task; the third parameter is related to the allocation process of the reasoning device.

[0256] In some embodiments, when the inference devices of the test task are allocated based on the number of inference devices required for the test task, the third parameter of the test task is the first parallel number; when the inference devices of the test task are allocated based on the amount of inference resources required for the test task, the third parameter of the test task is the second parallel number; the first parallel number is greater than the second parallel number.

[0257] In some embodiments, the first parallel number is the maximum number of the test tasks that can be run in parallel in the target number of inference devices; the second parallel number is determined based on the first parallel number and business conditions.

[0258] In some embodiments, the test module 830 is further used to: create a test unit corresponding to the test task based on the inference device assigned to the test task; and execute the test task in parallel based on the third parameter by the test unit to obtain the test result.

[0259] In some embodiments, the test task is used to execute the reasoning task of at least one model to be inferred, and the third parameter includes the parallel number corresponding to each model to be inferred; the test module 830 is also used to: in the process of executing the reasoning task of each model to be inferred, execute the reasoning process of the model to be inferred in parallel based on the parallel number corresponding to the model to be inferred by the test unit to obtain the reasoning result of the model to be inferred; generate the test result based on the reasoning result of each model to be inferred.

[0260] In some embodiments, the test module 830 is also used to: generate the test result based on the reasoning result of each model to be inferred and the standard result of each model to be inferred; wherein the standard result is the result obtained by reasoning the model to be inferred in an inference device that meets the requirements; and the test result is used to determine whether the inference device meets the requirements.

[0261] In some embodiments, the test parameters include a fourth parameter, and the fourth parameter is used to determine the number of loop executions of the test task; the number of loop executions is positively correlated with the test priority of the test task, and negatively correlated with the time consumption of the test task.

[0262] In some embodiments, the apparatus further comprises an updating module, wherein the updating module is configured to update a test parameter of the test task based on execution data of the test task.

[0263] In some embodiments, the execution data includes multiple execution sample data; the update module is also used to: determine the upper confidence limit data of the multiple execution sample data of the test task in a preset confidence interval; and update the test parameters of the test task using the test priority and upper confidence limit data of the test task.

[0264] In some embodiments, the update module is also used to: determine the average execution data of the multiple execution sample data based on the execution timestamp of each execution sample data; the influence weight of the execution sample data on the average execution data is negatively correlated with the time difference between the execution timestamp and the current time; and determine the upper confidence limit data using the average execution data, confidence level, fluctuation data of the multiple execution sample data and the number of samples of the multiple execution sample data.

[0265] In some embodiments, the execution sample data includes resource usage data; the test parameters include a third parameter, and the third parameter is used to determine the number of parallel processes of the test task; the update module is also used to determine the updated third parameter based on the inference resource amount of the inference device and the upper confidence limit of the resource usage data.

[0266] In some embodiments, the third parameter includes a first parallel number and a second parallel number; the update module is also used to: determine the first parallel data based on the inference resource amount of the inference device and the upper confidence limit of the resource usage data; determine the second parallel number based on the quotient of the first parallel number and a preset adjustment parameter, and the minimum parallel number.

[0267] In some embodiments, the execution sample data includes time-consuming data; the test parameters include a fourth parameter, and the fourth parameter is used to determine the number of loop executions of the test task; the update module is also used to determine the updated fourth parameter using the maximum test duration, the test priority of the test task and the upper confidence limit of the time-consuming data.

[0268] In some embodiments, the test task is used to execute the reasoning task of at least one model to be inferred, and the time consumption data includes the model consumption of the model to be inferred; the update module is also used to: determine the test time of the test task using the maximum test time and the weight corresponding to the test priority of the test task; determine the average test time based on the test time and the number of models to be inferred; determine the number of cyclic reasoning of the reasoning model using the average test time and the upper confidence limit of the model consumption of the model to be inferred.

[0269] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiment of the present application can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0270] It should be noted that in the embodiment of the present application, if the above-mentioned reasoning device test method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly reflected in the form of a software product that contributes to the relevant technology. The software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.

[0271] An embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0272] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0273] An embodiment of the present application provides a computer program, including a computer-readable code. When the computer-readable code is run in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0274] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.

[0275] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.

[0276] Fig. 9 A hardware entity diagram of a computer device provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the hardware entity of the computer device 900 includes: a processor 901 and a memory 902, wherein the memory 902 stores a computer program that can be run on the processor 901, and the processor 901 implements the steps in the method of any of the above embodiments when executing the program.

[0277] The memory 902 stores computer programs that can be run on the processor. The memory 902 is configured to store instructions and applications executable by the processor 901. It can also cache data to be processed or processed by the processor 901 and various modules in the computer device 900 (for example, image data, audio data, voice communication data, and video communication data). This can be achieved through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0278] When the processor 901 executes the program, the steps of any of the above-mentioned inference device testing methods are implemented. The processor 901 generally controls the overall operation of the computer device 900.

[0279] An embodiment of the present application provides a computer storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps of the inference device testing method of any of the above embodiments.

[0280] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0281] The processor may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that the electronic device that implements the functions of the processor may also be other, and the embodiments of the present application are not specifically limited.

[0282] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM) and the like; it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0283] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.

[0284] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0285] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0286] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0287] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units. A person of ordinary skill in the art can understand that all or part of the steps of the above method embodiments can be completed by hardware related to program instructions, and the above program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above method embodiments are executed; and the above storage medium includes: mobile storage devices, read-only memory (ROM), disks or optical disks, etc. Various media that can store program codes.

[0288] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0289] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A method for testing an inference device, characterized in that: Applied to a test cluster including at least one inference device, the method comprises: Obtaining the remaining resource information of the test cluster, and obtaining the test parameters and test priority of each of the at least two test tasks; the remaining resource information is at least used to characterize the working state of the reasoning device in the test cluster; Allocating an inference device to each of the test tasks based on the remaining resource information and the test parameters and test priority of each of the test tasks; The test task is executed based on the test parameters of the test task through the assigned inference device to obtain the test result corresponding to the inference device.

2. The method according to claim 1, characterized in that The remaining resource information includes first resource information, and the first resource information is used to indicate the number of inference devices in the test cluster that are in an idle state; the test parameter includes a first parameter, and the first parameter is used to characterize the number of inference devices required to execute the test task; The allocating an inference device to each of the test tasks based on the remaining resource information and the test parameters and test priority of each of the test tasks includes: When the first resource information is greater than or equal to the total amount of inference device requirements, a corresponding number of inference devices are allocated to each test task based on the first parameter of each test task; the total amount of inference device requirements is the sum of the first parameters of each test task.

3. The method according to claim 2, characterized in that The allocating an inference device to each of the test tasks based on the remaining resource information and the test priority of each of the test tasks also includes: In the case where the first resource information is less than the total amount of the inference device demand and greater than the preset number of devices, a corresponding number of inference devices are allocated to each of the test tasks in descending order of test priority based on the first real-time resource information and the first parameter of each of the test tasks; the first real-time resource information is the real-time first resource information in response to the allocation of the inference device for the current test priority; Among them, when the first real-time resource information is less than the first parameter of the current test priority, the first waiting state is entered and the first real-time resource information is monitored; when the first real-time resource information meets the first parameter of the current test priority, a corresponding number of inference devices are allocated to the test task of the current test priority.

4. The method according to claim 3, characterized in that The remaining resource information also includes second resource information, and the second resource information is used to indicate the remaining reasoning resources of each of the reasoning devices; the test parameter includes a second parameter, and the second parameter is used to characterize the amount of reasoning resources required to execute the test task; The allocating an inference device to each of the test tasks based on the remaining resource information and the test priority of each of the test tasks also includes: In the case where the first resource information is less than the preset number of devices and the time to enter the first waiting state exceeds the first time threshold, a corresponding amount of inference resources is allocated to each test task to be allocated in order of test priority from high to low, based on the second real-time resource information and the second parameter of each test task to be allocated; the second real-time resource information is real-time second resource information in response to allocating the amount of inference resources for the current test priority; Among them, when the second real-time resource information is less than the second parameter of the current test priority, enter the second waiting state and monitor the second real-time resource information; when the second real-time resource information meets the second parameter of the current test priority, allocate the corresponding reasoning resource amount to the test task of the current test priority.

5. The method according to any one of claims 1 to 4, characterized in that: The test parameters include a third parameter, which is used to determine the number of parallel processes of the test task; the third parameter is related to the allocation process of the reasoning device.

6. The method according to claim 5, characterized in that When the inference devices of the test task are allocated based on the number of inference devices required for the test task, the third parameter of the test task is the first parallel number; when the inference devices of the test task are allocated based on the amount of inference resources required for the test task, the third parameter of the test task is the second parallel number; the first parallel number is greater than the second parallel number.

7. The method according to claim 6, characterized in that The first parallel number is the maximum number of the test tasks that can be run in parallel in the target number of inference devices; the second parallel number is determined based on the first parallel number and business conditions.

8. The method according to claim 5, characterized in that The assigned inference device executes the test task based on the test parameters of the test task to obtain a test result corresponding to the inference device, including: Creating a test unit corresponding to the test task based on the inference device assigned to the test task; The test unit executes the test tasks in parallel based on the third parameter to obtain the test result.

9. The method according to claim 5, characterized in that The test task is used to execute the reasoning task of at least one model to be inferred, and the third parameter includes the parallel quantity corresponding to each model to be inferred; The performing the test tasks in parallel based on the third parameter by the test unit to obtain the test result includes: In the process of executing the reasoning task of each model to be inferred, the test unit executes the reasoning process of the model to be inferred in parallel based on the parallel quantity corresponding to the model to be inferred, so as to obtain the reasoning result of the model to be inferred; The test result is generated based on the inference result of each of the models to be inferred.

10. The method according to claim 9, characterized in that The generating the test result based on the inference result of each of the models to be inferred includes: The test result is generated based on the inference result of each model to be inferred and the standard result of each model to be inferred; wherein the standard result is the result obtained by inferring the model to be inferred in an inference device that meets the requirements; and the test result is used to determine whether the inference device meets the requirements.

11. The method according to any one of claims 1 to 4, characterized in that: The test parameters include a fourth parameter, which is used to determine the number of loop executions of the test task; the number of loop executions is positively correlated with the test priority of the test task, and negatively correlated with the time consumption of the test task.

12. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Based on the execution data of the test task, a test parameter of the test task is updated.

13. The method according to claim 12, characterized in that The execution data includes a plurality of execution sample data; and updating the test parameters of the test task based on the execution data of the test task includes: Determine upper confidence limit data of a plurality of execution sample data of the test task within a preset confidence interval; The test parameters of the test task are updated using the test priority and upper confidence limit data of the test task.

14. The method according to claim 13, characterized in that The step of determining upper confidence limit data of a plurality of execution sample data of the test task within a preset confidence interval includes: Based on the execution timestamp of each execution sample data, average execution data of the plurality of execution sample data are determined; the influence weight of the execution sample data on the average execution data is negatively correlated with the time difference between the execution timestamp and the current time; The upper confidence limit data is determined using the average execution data, the confidence level, the fluctuation data of the plurality of execution sample data, and the sample size of the plurality of execution sample data.

15. The method according to claim 13, characterized in that The execution sample data includes resource usage data; the test parameter includes a third parameter, and the third parameter is used to determine the number of parallel processes of the test task; the test parameter of the test task is updated by using the test priority and upper confidence limit data of the test task, including: The updated third parameter is determined based on the inference resource amount of the inference device and the upper confidence limit of the resource usage data.

16. The method according to claim 15, characterized in that The third parameter includes a first parallel quantity and a second parallel quantity; the determining the updated third parameter based on the inference resource amount of the inference device and the upper confidence limit of the resource usage data includes: determining the first parallel data based on the inference resource amount of the inference device and the upper confidence limit of the resource usage data; The second parallel quantity is determined based on a quotient of the first parallel quantity and a preset adjustment parameter and a minimum parallel quantity.

17. The method according to claim 13, characterized in that The execution sample data includes time-consuming data; the test parameters include a fourth parameter, and the fourth parameter is used to determine the number of loop executions of the test task; the test parameters of the test task are updated using the test priority and upper confidence limit data of the test task, including: The updated fourth parameter is determined using the maximum test duration, the test priority of the test task and the upper confidence limit of the time-consuming data.

18. The method according to claim 17, characterized in that The test task is used to perform at least one reasoning task of a model to be inferred, and the time consumption data includes the model time consumption of the model to be inferred; the use of the maximum test duration of the test task, the test priority of the test task and the upper confidence limit of the time consumption data to determine the updated fourth parameter includes: Determine the test duration of the test task by using the maximum test duration and the weight corresponding to the test priority of the test task; Determine an average test duration based on the test duration and the number of models to be inferred; The average test time and the upper confidence limit of the model time of the model to be inferred are used to determine the number of cyclic inferences of the inference model.

19. A test device for an inference device, characterized in that: include: An acquisition module, used to acquire remaining resource information of the test cluster, and acquire test parameters and test priority of each of at least two test tasks; The remaining resource information is at least used to characterize the working status of the reasoning device in the test cluster; An allocation module, configured to allocate an inference device to each of the test tasks based on the remaining resource information and the test parameters and test priority of each of the test tasks; The test module is used to execute the test task based on the test parameters of the test task through the assigned reasoning device to obtain the test result corresponding to the reasoning device.

20. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the inference device testing method according to any one of claims 1 to 18 is implemented.

21. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the inference device testing method according to any one of claims 1 to 17 is implemented.

22. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the inference device testing method according to any one of claims 1 to 17 is implemented.

Images