Automatic test system and method, related equipment and program product

Abstract

The embodiment of the invention provides an automatic test system and method, related equipment and a program product, and the automatic test system comprises a test scheduling module; the test scheduling module is used for determining a to-be-executed current test case in the to-be-tested queue; for the current test case, when it is judged that the current remaining resources in the current test environment cannot support execution of the current test case, marking the current test case, and updating the next test case as the current test case; when it is judged that all the test cases are marked, the test cases are sorted based on the resource requirements of the test cases, and when the current remaining resources are increased, the current test case to be executed is determined; the test case in the to-be-tested queue is the test case of the currently tested heterogeneous acceleration card matched with the current test environment. According to the automatic test system provided by the embodiment of the invention, the resource utilization rate of the heterogeneous accelerator card for executing the test case can be improved.

Description

Technical Field

[0001] This invention relates to the field of chip testing, specifically to an automated testing system, method, related equipment, and program products. Background Technology

[0002] With the rapid development of artificial intelligence, high-performance computing (HPC), and large-scale deep learning models, heterogeneous accelerated computing architecture has become a core solution for improving computing power density. Heterogeneous accelerated computing cards are the core hardware components of heterogeneous accelerated computing architectures, and due to their parallel computing capabilities, they are widely used in data centers, scientific simulations, and AI (Artificial Intelligence) training and inference scenarios. However, the iterative acceleration of heterogeneous accelerated cards themselves, the fragmentation of driver versions for heterogeneous accelerated cards, and the frequent updates of software tools related to testing heterogeneous accelerated cards have led to multiple systemic challenges in testing heterogeneous accelerated cards.

[0003] In related technologies, low resource utilization is a problem when testing heterogeneous accelerator cards. Therefore, there is an urgent need for a testing system that can improve the resource utilization of heterogeneous accelerator cards executing test cases. Summary of the Invention

[0004] This invention provides an automated testing system, method, related equipment, and program products to improve the resource utilization of heterogeneous accelerator cards executing test cases.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions.

[0006] In a first aspect, embodiments of the present invention provide an automatic testing system applied to heterogeneous accelerator cards, comprising: a test scheduling module; The test scheduling module is used to determine the current test cases to be executed in the test queue; for the current test case, when it is determined that the current remaining resources in the current test environment of the automated test system cannot support the execution of the current test case, the current test case is marked, and the next test case to be executed in the test queue is updated to the current test case; and, when it is determined that all test cases in the test queue are marked, the test cases are sorted according to the resource requirements of each test case, and when the current remaining resources increase, the current test case to be executed is determined from the sorted test cases; the test cases in the test queue are: test cases of heterogeneous accelerator cards currently being tested that are adapted to the current test environment.

[0007] In a second aspect, embodiments of the present invention provide an automatic testing method, applied to the automatic testing system as described in the first aspect, comprising: Identify the current test cases to be executed in the test queue; For the current test case, when it is determined that the remaining resources in the current test environment of the automated test system cannot support the execution of the current test case, the current test case is marked, and the next test case to be executed in the test queue is updated to the current test case; Furthermore, when it is determined that all test cases in the test queue are marked, the test cases are sorted according to their resource requirements. When the remaining resources increase, the current test case to be executed is determined from the sorted test cases. The test cases in the test queue are test cases of the heterogeneous accelerator cards currently being tested that are adapted to the current test environment.

[0008] Thirdly, embodiments of the present invention provide a computer program product, including a computer program that, when executed, implements the automatic testing method as described in the second aspect.

[0009] Fourthly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a program, and the processor calls the program stored in the memory to execute the automatic testing method as described in the second aspect.

[0010] Fifthly, embodiments of the present invention provide a storage medium storing a program that, when executed, implements the automatic testing method as described in the second aspect.

[0011] This invention provides an automated testing system applied to heterogeneous accelerator cards, comprising: a test scheduling module; the test scheduling module is used to determine the current test cases to be executed in the test queue; for the current test case, when it is determined that the current remaining resources in the current test environment of the automated testing system cannot support the execution of the current test case, the current test case is marked, and the next test case to be executed in the test queue is updated to the current test case; and, when it is determined that all test cases in the test queue are marked, the test cases are sorted according to the resource requirements of each test case, and when the current remaining resources increase, the current test case to be executed is determined from the sorted test cases; the test cases in the test queue are: test cases of the heterogeneous accelerator cards currently being tested that are adapted to the current test environment.

[0012] As can be seen, in the automatic testing system provided by this embodiment of the invention, after the test scheduling module determines the test queue, the test queue typically consists of multiple test cases to be executed. The current test case is the test case to be executed, which is the determined target object to be executed. For the current test case, it is determined whether the remaining resources of the current test environment can support the execution of the current test case. Since the test queue contains multiple test cases, the testing process for the test queue is continuous, and the resources in the current test environment are in a dynamic state. Therefore, for the current test case, it is possible to determine whether the remaining resources of the current test environment can support the execution of the current test case. The system first checks the remaining resources in the current test environment to determine if they meet the resource requirements of the current test case. If not, the current test case is marked, and the next test case to be executed in the test queue is updated to the current test case, and the process continues. If it is determined that all test cases in the test queue are marked, meaning the current remaining resources cannot meet the resource requirements of any test case in the queue, then the test cases are sorted based on their resource requirements. After updating the current remaining resources, the current test case to be executed is determined from the sorted test queue. Therefore, the execution of test cases can be dynamically adjusted based on the real-time status of the remaining resources, thereby avoiding test interruptions due to insufficient resources and reducing the probability of test interruptions. It is evident that the automatic testing system provided in this embodiment of the invention can improve the resource utilization of the test environment executing test cases, and also improve the resource utilization of heterogeneous accelerator cards in the test environment. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0014] Figure 1 This is a flowchart illustrating a heterogeneous accelerator card testing method in related technologies; Figure 2 This is a schematic diagram of the structure of an automatic testing system provided in an embodiment of the present invention; Figure 3 A functional diagram of the test case management module, performance information acquisition module, and fault diagnosis and recovery module provided in an embodiment of the present invention; Figure 4 A flowchart illustrating the automatic testing method provided in an embodiment of the present invention; Figure 5 This is another flowchart illustrating the automatic testing method provided in an embodiment of the present invention.

[0015] Figure label: 1. Automated Testing System; 11. Test Environment Setup Module; 12. Test Scheduling Module; 121. Test Case Management Module; 122. Performance Information Acquisition Module; 123. Fault Diagnosis and Recovery Module; 13. Test Result Summary Module. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0017] With the rapid development of artificial intelligence, high-performance computing (HPC), and large-scale deep learning models, heterogeneous accelerated computing architectures have become a core solution for improving computing power density. Heterogeneous accelerator cards, represented by GPUs (Graphics Processing Units) and DCUs (Deep Computing Units), are widely used in data centers, scientific simulations, and AI training and inference scenarios due to their parallel computing capabilities.

[0018] Heterogeneous accelerator cards refer to systems (hardware devices or hardware / software combinations that carry accelerated computing tasks) that simultaneously contain multiple types of heterogeneous accelerator cards, or heterogeneous accelerator cards of the same type but with different parameters. These heterogeneous accelerator cards have differences in computing power, memory bandwidth, and resource allocation logic, and their respective resource usage will change with the task.

[0019] However, most automated testing systems in related technologies adopt a fixed load mode, lacking a dynamic feedback mechanism of "sensing-adjustment": they cannot sense changes in resource usage in real time, nor can they dynamically adjust the testing strategy based on perceived changes in resource usage. This mismatch between static testing and dynamic hardware leads to inaccurate test results (either failing to measure extreme performance or obtaining distorted data), and unstable testing processes (easily causing hardware overload failures of heterogeneous accelerator cards), ultimately failing to effectively verify the true performance and compatibility of heterogeneous accelerator cards.

[0020] Fixed load mode refers to the system having a fixed amount of test tasks and a fixed execution rhythm in advance. For example, regardless of whether the heterogeneous accelerator card is currently idle or busy, the test is executed according to the fixed rules of "running 100 inference tasks per second" and "loading 10GB of test data each time", without adjusting the amount of tasks, execution speed or task allocation method throughout the process.

[0021] Resource usage changes refer to the real-time hardware resource consumption status of the automated testing system during test case execution. This mainly includes three core indicators: 1. Computing resources: such as GPU / DCU core utilization (%), for example, 80% of the cores of a heterogeneous accelerator card are currently running other tasks, leaving only 20% idle; 2. Memory resources: such as the used / remaining capacity of the accelerator card's video memory (VRAM, Video Random Access Memory), for example, 12GB of video memory is used for a DCU, leaving only 4GB idle; 3. Scheduling resources: such as the length of the test queue, for example, currently 5 tasks are queued for execution, and new tasks need to be queued.

[0022] In one of the technical solutions related to the technology, please refer to Figure 1 , Figure 1 This is a flowchart illustrating a heterogeneous accelerator card testing method in related technologies. The testing method for the heterogeneous accelerator card includes the following main steps: Step S11, Hardware Identification: Detect the product model of the heterogeneous accelerator card under test; Step S12, Driver Matching: Retrieve the corresponding driver from the preset driver library according to the product model; Step S13, Environment Deployment: Automatically install drivers and configure to obtain the basic test environment; Step S14: Test execution: Run the preset test case set and output the results.

[0023] Figure 1 The core limitations of the heterogeneous accelerator card testing method mentioned include at least the lack of real-time resource monitoring and dynamic scheduling capabilities for the heterogeneous accelerator card testing system.

[0024] To address the above problems, embodiments of the present invention provide an automated testing system, please refer to... Figure 2 , Figure 2 This is a schematic diagram of an automatic testing system provided in an embodiment of the present invention. The automatic testing system is applied to heterogeneous accelerator cards.

[0025] like Figure 2 As shown, the automatic testing system 1 includes: a test scheduling module 12; The test scheduling module 12 is used to determine the current test cases to be executed in the test queue; for the current test cases, when it is determined that the current remaining resources in the current test environment of the automatic test system cannot support the execution of the current test cases, the current test cases are marked, and the next test case to be executed in the test queue is updated to the current test cases; and, when it is determined that all test cases in the test queue are marked, the test cases are sorted according to the resource requirements of each test case, and when the current remaining resources increase, the current test cases to be executed are determined from the sorted test cases; the test cases in the test queue are: test cases of heterogeneous accelerator cards currently being tested that are adapted to the current test environment.

[0026] When it is determined that the remaining resources do not support the execution of the current test case, i.e., the resources requested by the current test case are not met, the current test case is marked. This is to prevent the marked test case from being traversed and queried again when the remaining resources are not updated. This can save unnecessary repeated traversal and querying, thereby saving testing time and improving testing efficiency.

[0027] Meanwhile, when it is determined that the remaining resources do not support the execution of the current test case, the next test case in the test queue is used as the current test case, and the current test case is judged and executed, thereby reducing the probability of test interruption.

[0028] It's easy to understand that if it's determined that the remaining resources support the execution of the current test case, then the current test case can be executed directly. Simultaneously, the next test case in the test queue will be used as the current test case to continue resource assessment.

[0029] It's important to note that "current remaining resources" refers to the real-time resources in the test environment at the time of executing each test case in the test queue; in other words, "current remaining resources" is a variable. For example, when test case 'a' is determined to be the current test case in the test queue, the current remaining resources are s1. If it's determined that the current remaining resources s1 can support the execution of test case 'a', then the current remaining resources a1 provide the necessary resources for test case 'a', for example, a1. Then, if the next test case 'b' becomes the current test case, and the current remaining resources do not increase, the current remaining resources for test case 'b' will be s1 - a1.

[0030] It is evident that, without an increase in remaining resources, the resources are decreasing and will not meet the execution requirements of the marked test cases. Therefore, in this embodiment of the invention, the marked test cases are processed: based on the resource requirements of each test case, the test cases are sorted, and then, when the remaining resources increase, the sorted test cases are re-evaluated. This avoids repeated traversal of already evaluated test cases and allows for timely re-execution of marked test cases.

[0031] As can be seen, in the automatic testing system 1 provided in this embodiment of the invention, after the test scheduling module 12 determines the test queue, the test queue usually consists of multiple test cases to be executed. The current test case is the test case to be executed, which is the determined target object to be executed. For the current test case, it is determined whether the current remaining resources of the current test environment can support the execution of the current test case. Since the test queue contains multiple test cases, the testing process for the test queue is continuous, and the resources in the current test environment are in a dynamic state. Therefore, for the current test case, it is possible to base the current test on the available resources. The system determines whether the remaining resources in the current test environment can meet the resource requirements of the current test case. If not, the current test case is marked, and the next test case to be executed in the test queue is updated to the current test case, and the determination continues. If it is determined that all test cases in the test queue are marked, meaning the current remaining resources cannot meet the resource requirements of any test case in the test queue, then the test cases are sorted based on resource requirements. After updating the current remaining resources, the current test case to be executed is determined from the sorted test queue. Therefore, the execution of test cases can be dynamically adjusted according to the real-time status of the remaining resources, thereby avoiding test interruptions due to insufficient resources and reducing the probability of test interruptions. It is evident that the automatic testing system provided in this embodiment can improve the resource utilization of the test environment executing test cases, and also improve the resource utilization of heterogeneous accelerator cards in the test environment.

[0032] Please continue to refer to this. Figure 2 The test scheduling module 12 includes: a test case management module 121, a performance information acquisition module 122, and a fault diagnosis and recovery module 123.

[0033] Please refer to Figure 3 , Figure 3 This is a functional diagram of the test case management module, performance information collection module, and fault diagnosis and recovery module provided in the embodiments of the present invention.

[0034] according to Figure 3 As can be seen, the test case management module 121 provided in this embodiment of the invention includes the following functions: tag management function and resource usage management function.

[0035] In related technologies, testing solutions for heterogeneous accelerator cards typically bind test cases to specific versions of GPU drivers or products, lacking systematic support for compatibility across multiple versions and scenarios.

[0036] The functions of the test case management module 121 include: Function 1: Tag Management. This function refers to the ability to classify and manage compatibility across multiple versions and scenarios by tagging test cases. For example, for a heterogeneous accelerator card with versions V1.0 and V2.0, there are 10 test cases. Three test cases are only applicable to the V1.0 version of the heterogeneous accelerator card, five test cases are applicable to both versions V1.0 and V2.0, and two test cases are only applicable to the V2.0 version. The three test cases applicable to the V1.0 version are tagged as "V1.0". The five test cases applicable to both versions V1.0 and V2.0 are tagged as "V1.0 and V2.0". The two test cases applicable to the V2.0 version are tagged as "V2.0". This achieves multi-version compatibility classification and management.

[0037] The above-described method of managing test cases can reduce the memory space occupied by test cases, improve the reusability of test cases, and provide systematic support for multi-version and multi-scenario compatibility of test cases for heterogeneous accelerator cards.

[0038] In one embodiment, the test case management module 121 is used to obtain all test cases supported by the automatic testing system; establish a mapping table according to the support relationship between each test case and the heterogeneous accelerator card and the test environment; filter test cases from the mapping table based on the heterogeneous accelerator card currently being tested and the current test environment; and form a queue to be tested based on the filtered test cases.

[0039] The process of establishing a mapping table is the process of tagging the test cases. It is understood that the establishment of the mapping table provided in this embodiment of the invention is one of the implementation methods for tagging the test cases, and is not intended to limit the implementation of tagging the test cases in this embodiment of the invention.

[0040] Test cases depend on the test environment, which in turn depends on heterogeneous accelerator cards. Furthermore, test case requirements define the compatibility standards between the test environment and the accelerator cards. Therefore, this embodiment of the invention establishes a mapping table based on the support relationships between the test cases, heterogeneous accelerator cards, and the test environment (e.g., the driver for the heterogeneous accelerator card in the test environment, or the Device Development Kit: DTK). Then, based on the heterogeneous accelerator card currently being tested and the current test environment, a set of test cases for the current test is selected from the mapping table. In other words, the selected test cases are applicable to the heterogeneous accelerator card being tested and can run in the current test environment. This reduces the problem of test case execution failures due to poor compatibility between test cases and the test environment, allowing the focus to be on testing the functionality or performance of the corresponding supported or tested heterogeneous accelerator cards, thus improving testing efficiency.

[0041] Function 2: Resource Usage Management Function: This function is used to request corresponding resources for different types of test cases.

[0042] For example, the aforementioned tag management function can also be based on the testing purpose of the test cases. For instance, if there are 10 test cases, 7 of which are for functional testing and the other 3 are for performance testing, then the 7 test cases with the testing purpose of functional testing are tagged as functional testing, and the 3 test cases with the testing purpose of performance testing are tagged as performance testing.

[0043] When test cases are for functional testing of heterogeneous accelerator cards (such as verifying whether "the heterogeneous accelerator card can correctly process data operations" or "whether calling the interface of the heterogeneous accelerator card returns the correct result"), it is not necessary for each test case to occupy a separate heterogeneous accelerator card. Instead, multiple different functional test cases (such as "accelerated operation of test data A", "accelerated operation of test data B", "error handling of test abnormal data") can take turns or use the same heterogeneous accelerator card included in the automated testing system in parallel.

[0044] When the test case is a test case for performance testing of heterogeneous accelerator cards (such as testing "the computing speed of heterogeneous accelerator cards under full load", "the maximum throughput of heterogeneous accelerator cards in system calls", "the response latency of heterogeneous accelerator cards under high concurrency"), a single test case can completely monopolize all available hardware resources in the automated testing system.

[0045] To enable the setup of the current test environment, in one embodiment, the automatic test system 1 further includes a test environment setup module 11; the test environment setup module 11 is used to set up a basic test environment for the test environment, so as to provide it to the test case management module 121 to form the current test environment; The test environment setup module 11 is used to build the basic test environment for the test environment, including: Obtain the compatibility verification basis of the local testing environment of the automatic testing system 1; Obtain the different models of heterogeneous accelerator cards included in the automatic testing system 1; For each model of heterogeneous accelerator card, based on the compatibility verification criteria, it is checked whether the local test environment of the automatic test system 1 supports the testing of the heterogeneous accelerator card; if so, the local test environment is used as the basic test environment; if not, the corresponding matching test environment setup information is obtained according to the model of the heterogeneous accelerator card, and the basic test environment is built based on the test environment setup information and the local test environment.

[0046] The test environment setup module 11 can first detect the local test environment of the automatic test system, such as the operating system version running in the local test environment, and collect underlying related information such as kernel version, glibc version, and CPU information, thereby forming a complete compatibility verification basis for the local test environment, and detecting whether the local test environment supports the testing of heterogeneous accelerator cards, ensuring that the testing of heterogeneous accelerator cards can be carried out smoothly.

[0047] The test environment setup module 11 of the automatic testing system 1 provided in this embodiment of the invention can directly use the local test environment as the basic test environment when it is determined that the local test environment supports testing heterogeneous accelerator cards. This allows the test environment to be provided to the test case management module 121 to form the current test environment for testing the heterogeneous accelerator cards. When the local test environment cannot support testing heterogeneous accelerator cards, test environment setup information is obtained based on the model of the heterogeneous accelerator card, and the test environment setup information is installed into the local test environment to obtain the basic test environment. This maximizes the utilization of the local test environment of the automatic testing system 1 and expands the compatibility of the basic test environment by testing different models of heterogeneous accelerator cards. This allows the basic test environment to support an increasing number of heterogeneous accelerator card models. Furthermore, when the same heterogeneous accelerator card needs to be tested later, there is no need to repeatedly set up the basic test environment, thus improving testing efficiency.

[0048] In one embodiment, the test environment setup information may include, for example, the driver for the heterogeneous accelerator card, the tool suite, and the dependency software packages for the test environment corresponding to the heterogeneous accelerator card.

[0049] It is easy to understand that during the testing phase of heterogeneous accelerator cards, there is a clear difference between the testing environment on which the heterogeneous accelerator card itself depends and the testing environment on which the test cases used to test the heterogeneous accelerator card depend. The core reason is that the "dependency target" and "functional positioning" of the testing environment on which the heterogeneous accelerator card itself depends and the testing environment on which the test cases depend are completely different: the former is to enable the heterogeneous accelerator card to "run", while the latter is to enable the test cases to "verify". In other words, the former is the "basic prerequisite" for testing, while the latter is the "verification tool / scenario carrier" for testing.

[0050] The test environment upon which the heterogeneous accelerator card depends can be understood as the operating support environment for the heterogeneous accelerator card. This is the minimum test environment that allows the host to recognize the heterogeneous accelerator card, load its drivers correctly, and provide basic computing capabilities. Its core objective is to ensure the heterogeneous accelerator card itself is in a working state; it is the underlying foundation for all subsequent tests, which is the current test environment described above.

[0051] The test environment on which test cases depend can be understood as the tool / scenario environment for verifying the functionality / performance of heterogeneous accelerator cards. This means the environment that allows test cases (such as functional test cases, performance test cases, and stability test cases) to execute normally, output valid results, and accurately locate problems. Its core objective is to enable test cases to verify heterogeneous accelerator cards; it is the upper-level tool carrier for testing.

[0052] Therefore, after the current test environment is successfully set up, please continue to refer to... Figure 3 The test case management module also includes the following functions: Function 3: Test Case Dependency Management: Used to configure the environment dependencies of test cases.

[0053] The core objective of the test case dependency management function is to clarify which environment dependencies a test case needs to be configured to run in the current test environment, and to transform the requirements into test environment dependencies configured for the test case (the instruction manual for subsequent execution), without involving the actual test environment setup operation.

[0054] Function 4: Containerization Management Function: This function mainly enables automatic image pulling, test container creation, and environment dependency installation based on the test environment dependencies configured for the test cases.

[0055] Essentially, it involves executing specific actions related to the test environment dependencies configured for the test cases. For example, the specific steps might be: Pull Image: Download the corresponding system image from the image repository according to the image specified in the test environment dependencies configured for the test cases (e.g., ubuntu:20.04). This is the foundation for building the test environment and directly determines the operating system version.

[0056] Create a test container: Based on the pulled base image, create an independent container instance according to the resource limits and accelerator card association rules in the test environment dependencies configured for the test cases. This step provides an isolated environment for installing test environment dependencies to avoid conflicts with other tasks.

[0057] Install test environment dependencies: Inside the created container, install the corresponding software according to the software dependency list included in the test environment dependencies configured for the test cases using commands. This step directly satisfies the core dependencies of the test cases, enabling the test cases to run.

[0058] In summary, the aforementioned setup of the current test environment is to create a test environment that can run on heterogeneous accelerator cards, while functions three and four of the test case management module are to create a test environment that can run on test cases.

[0059] In one embodiment, the aforementioned test environment setup information may further include, for example, the operating system version. This test environment setup information is used during the setup of the current test environment; that is, it is essentially a list of requirements for the test environment that the heterogeneous accelerator card can run on. As discussed above, the test environment dependencies configured for test cases also involve operating system requirements.

[0060] In actual testing of heterogeneous accelerator cards, a potential scenario is that the configuration (e.g., operating system version) required by the heterogeneous accelerator card in the test environment may conflict with the configuration (e.g., operating system version) in the test case's test environment. In this case, it may be necessary to identify which test cases conflict with the configuration required by the heterogeneous accelerator card and adjust them to ensure the required configuration does not conflict with the heterogeneous accelerator card. Alternatively, the test environment may need to be rebuilt to meet both the test cases' and the heterogeneous accelerator card's configuration requirements. However, either approach increases the likelihood of errors during testing and reduces testing efficiency.

[0061] Therefore, the automatic testing system 1 provided in this embodiment of the invention can filter test cases based on the current testing environment before configuring test environment dependencies for test cases. The filtered test cases can be adapted to the current testing environment, which enables the automatic testing system 1 provided in this embodiment of the invention to reduce the possibility of errors during the testing process while improving testing efficiency.

[0062] In one implementation, after forming a queue to be tested based on the selected test cases, the test case management module 121 is further configured to initialize a resource pool based on the heterogeneous accelerator card currently being tested and the allocatable resources in the basic test environment, forming a current test environment for the heterogeneous accelerator card currently being tested, and the resources in the initialized resource pool are used as resources for the test cases in the queue to be tested.

[0063] The system initializes the allocatable resources in the basic test environment, completes the initialization of the resource pool, and binds the resource pool to the heterogeneous accelerator card currently being tested. This ultimately forms a dedicated test environment for the heterogeneous accelerator card, i.e., the current test environment. This facilitates precise matching between the current test environment and the heterogeneous accelerator card, reducing problems that may arise during resource allocation for test cases of the heterogeneous accelerator card due to incompatibility. It allows the system to focus on testing the functionality or performance of the heterogeneous accelerator card, thereby improving the testing efficiency of the heterogeneous accelerator card.

[0064] During testing, the causes of test environment anomalies are extremely complex, essentially stemming from a breach in the stability / consistency of the test environment. This could be due to deviations in any aspect, such as the test environment's own configuration, external dependencies, or test operations. These factors can ultimately prevent the test from proceeding normally or result in distorted information. However, related technologies primarily rely on analyzing logs after the heterogeneous accelerator card tests are completed, making dynamic anomaly detection and root cause localization during the testing process difficult. Therefore, this invention proposes a functional module for real-time anomaly detection.

[0065] Please continue to refer to this. Figure 2 The test scheduling module 12 provided in this embodiment of the invention includes: a performance information acquisition module 122 and a fault diagnosis and recovery module 123; During the execution of the current test case by the test scheduling module 12, the performance information acquisition module 122 is used to collect the performance status information of the current test environment and send the performance status information of the current test environment to the fault diagnosis and recovery module 123. The fault diagnosis and recovery module 123 is used to determine whether the current test environment is abnormal based on the performance status information, and to handle the abnormality when the current test environment is abnormal.

[0066] The test scheduling module 12 provided in this embodiment of the invention can monitor the performance status information of the current test environment in real time during the test process, and handle the abnormality in real time when the current test environment is abnormal. It can realize dynamic abnormality detection and root cause localization during the test process, and improve test efficiency.

[0067] Please continue to refer to this. Figure 3 The performance information acquisition module 122 provided in this embodiment of the invention mainly functions as real-time monitoring of device status information, real-time monitoring of operating system status information, real-time monitoring of heterogeneous accelerator card status information, and real-time monitoring of other performance-related information.

[0068] Please continue to refer to this. Figure 3 Corresponding to the aforementioned performance information acquisition module 122, the fault diagnosis and recovery module 123 provided in this embodiment of the invention mainly functions as real-time diagnosis and repair of abnormal equipment status, real-time diagnosis and repair of abnormal operating system status, real-time diagnosis and repair of abnormal heterogeneous accelerator card status, and real-time diagnosis and repair of other performance-related abnormal statuses.

[0069] For example, the performance status information of the current test environment may include: device performance status information, operating system performance status information, and heterogeneous accelerator card performance status information; The anomalies include: driver issues and / or cross-chip interconnection issues; The solution to this driver problem is to reload the driver. The method for handling the cross-chip connection problem is to rebuild the chip connection.

[0070] Furthermore, considering both testing efficiency and anomaly handling, the processing method provided in this embodiment of the invention is not for each heterogeneous accelerator card, but for a specific type of anomaly. Therefore, after anomaly handling, changes in the heterogeneous accelerator card or testing environment may prevent the anomaly from being handled, or in other words, the state of the testing environment after the anomaly has been handled may be uncertain.

[0071] Therefore, after handling the anomaly, the fault diagnosis and recovery module 123 is further configured to detect the post-anomaly status information of the current test environment, and based on the post-anomaly status information, determine whether the current test environment after handling the anomaly supports continuing subsequent testing. If not, the test is terminated and a warning signal is output.

[0072] The above steps ensure that after the aforementioned performance information acquisition module 122 and the fault diagnosis and recovery module 123, the current test environment can continue to support the execution of subsequent test cases, thereby improving test efficiency while ensuring the stability of the current test environment.

[0073] Please continue to refer to this. Figure 1 In one embodiment, the automatic testing system 1 provided by the present invention further includes: a test result summary module 13; The test result aggregation module 13 is used to collect the test results obtained by executing the current test case in the current test environment. The performance information acquisition module 122 is also used to send the performance status information to the test result summary module 13; The fault diagnosis and recovery module 123 is also used to send the status information after the anomaly handling to the test result summary module 13; The test result summary module 13 is also used to summarize and output the performance status information and the status information after exception handling along with the test results.

[0074] The test result summary module 13 summarizes the test results, performance status information, and post-anomaly handling status information, providing a data foundation for subsequent adjustments to the test environment, so as to better support the testing of the heterogeneous accelerator card.

[0075] After the resource pool of the test environment is initialized, it will continuously allocate resources to the test cases. Therefore, after the test results are obtained, the test scheduling module 12 is also used to: release the resources used by the test cases that have obtained the test results to the resource pool and update the current remaining resources.

[0076] Updating the current remaining resources can trigger continued judgment and execution of each test sequence after sorting. For example, when releasing the resources used by the test case that has obtained the test result (test completed) to the resource pool, the current remaining resources are increased to trigger continued detection and execution of each test sequence after sorting.

[0077] In other implementations, increasing the current remaining resources can also be achieved, for example, by adding a new heterogeneous accelerator card of the same model to the current test environment.

[0078] After a test case has finished executing, quickly releasing resources for subsequent test cases can reduce resource waste.

[0079] Based on the same inventive concept, this invention also provides an automatic testing method, the flowchart of which is shown below. Figure 4 As shown, Figure 4 This is a flowchart illustrating an automatic testing method provided in an embodiment of the present invention. The automatic testing method is applied to the automatic testing system described in any of the foregoing embodiments.

[0080] like Figure 4 As shown, the method includes the following steps: Step S41: Determine the current test cases to be executed in the test queue.

[0081] Step S42: For the current test case, when it is determined that the remaining resources in the current test environment of the automatic test system cannot support the execution of the current test case, mark the current test case and update the next test case to be executed in the test queue to the current test case.

[0082] Step S43: When it is determined that all test cases in the test queue are marked, sort the test cases according to their resource requirements. When the current remaining resources increase, determine the current test case to be executed from the sorted test cases.

[0083] The test cases in the queue to be tested are: test cases of the heterogeneous accelerator cards currently being tested that are adapted to the current test environment.

[0084] As can be seen, in the automatic testing method provided by the embodiments of the present invention, after determining the test queue, the test queue is usually composed of multiple test cases to be executed. The current test case is the test case to be executed, which is the determined target object to be executed. For the current test case, it is determined whether the current remaining resources of the current test environment can support the execution of the current test case. Since the test queue contains multiple test cases, the testing process for the test queue is continuous, and the resources in the current test environment are in a dynamic state. Therefore, for the current test case, it can be determined whether the current remaining resources of the current test environment can meet the resource requirements of the current test case based on the current remaining resources. If they do, the current test case is executed. If they do not, the current test case is marked. When it is determined that all test cases in the test queue are marked, that is, the current remaining resources cannot meet the resource requirements of any test case in the test queue, the test cases are sorted based on the resource requirements. After the current remaining resources are updated, the current test case to be executed is determined in the sorted test queue. Therefore, the execution of test cases can be dynamically adjusted based on the real-time remaining resource situation, thereby avoiding test interruptions due to insufficient resources to execute the current test cases. It is evident that the automated testing method provided by this embodiment of the invention can improve the resource utilization of the test environment executing test cases, and simultaneously improve the resource utilization of heterogeneous accelerator cards within the test environment.

[0085] Please refer to Figure 5 , Figure 5 This is another flowchart illustrating the automatic testing method provided in an embodiment of the present invention.

[0086] like Figure 5 As shown, the method includes the following steps: Step S51: Determine the current test cases to be executed in the test queue.

[0087] Step S52: For the current test case, determine whether the remaining resources in the current test environment of the automatic test system can support the execution of the current test case. If yes, proceed to step S53; otherwise, proceed to step S54.

[0088] Step S53: Execute the current test case to obtain the test result, and update the next test case to be executed in the test queue to the current test case.

[0089] After step S53 is completed, step S52 is executed to continue judging the current test case until all test cases in the test queue are executed.

[0090] Step S54: Mark the current test case and update the next test case to be executed in the test queue to the current test case.

[0091] After step S54 is executed, step S52 is executed to continue judging the current test case. When it is determined that the remaining resources do not support the execution of the current test case, it is marked again, and step S55 is executed.

[0092] Step S55: When it is determined that all test cases in the test queue have been marked, sort the test cases according to their resource requirements. When the remaining resources increase, determine the current test case to be executed from the sorted test cases.

[0093] Step S55 determines the marking status of test cases during the execution of step S54, so that when the remaining test cases increase, the current test cases that cannot be executed can be re-determined, and the process returns to step S52 for execution.

[0094] In one embodiment, during the execution of the current test case, the automated testing method further includes: Collect performance status information of the current test environment; based on the performance status information, determine whether the current test environment has experienced any abnormalities; If so, then handle the exception.

[0095] In one embodiment, after the step of handling the exception, the method further includes: detecting the post-exception status information of the current test environment; based on the post-exception status information, determining whether the current test environment after handling the exception supports continuing subsequent testing; if not, ending the test and outputting a warning signal.

[0096] Based on the same inventive concept, embodiments of the present invention also provide a computer program product, including a computer program, which, when executed, implements the automatic testing method as described in any of the foregoing embodiments.

[0097] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory stores a program, and the processor calls the program stored in the memory to execute the automatic testing method as described in any of the foregoing embodiments.

[0098] Based on the same inventive concept, embodiments of the present invention also provide a storage medium storing a program that, when executed, implements the automatic testing method as described in any of the foregoing embodiments.

[0099] The foregoing describes multiple embodiments of the present invention. The optional methods described in each embodiment can be combined and cross-referenced without conflict, thereby extending to a variety of possible embodiments. These can all be considered as embodiments disclosed or made public by the present invention.

[0100] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. An automatic testing system, characterized in that, Applied to heterogeneous accelerator cards, including: a test scheduling module; The test scheduling module is used to determine the current test cases to be executed in the test queue; for the current test case, when it is determined that the current remaining resources in the current test environment of the automated test system cannot support the execution of the current test case, the current test case is marked, and the next test case to be executed in the test queue is updated to the current test case; and, when it is determined that all test cases in the test queue are marked, the test cases are sorted according to the resource requirements of each test case, and when the current remaining resources increase, the current test case to be executed is determined from the sorted test cases; the test cases in the test queue are: test cases of heterogeneous accelerator cards currently being tested that are adapted to the current test environment.

2. The automatic testing system as described in claim 1, characterized in that, The test scheduling module includes: a test case management module; The test case management module is used to obtain all test cases supported by the automatic testing system; establish a mapping table according to the support relationship between each test case and the heterogeneous accelerator card and the test environment; filter test cases from the mapping table based on the heterogeneous accelerator card currently being tested and the current test environment; and form a queue to be tested based on the filtered test cases.

3. The automatic testing system as described in claim 2, characterized in that, Also includes: Test environment setup module; The test environment setup module is used to build the basic test environment for the test environment, so as to provide it to the test case management module to form the current test environment; The test environment setup module is used to build the basic test environment, including: Obtain the compatibility verification basis of the local testing environment of the automated testing system; Obtain the different models of heterogeneous accelerator cards included in the automated testing system; For each model of heterogeneous accelerator card, based on the compatibility verification criteria, it is checked whether the local test environment of the automatic test system supports the testing of the heterogeneous accelerator card; if so, the local test environment is used as the basic test environment; if not, the corresponding matching test environment setup information is obtained according to the model of the heterogeneous accelerator card, and the basic test environment is built based on the test environment setup information and the local test environment.

4. The automatic testing system as described in claim 3, characterized in that, The test environment setup information includes: the driver for the heterogeneous accelerator card, the tool suite, and the dependent software packages for the test environment corresponding to the heterogeneous accelerator card.

5. The automatic testing system as described in claim 3, characterized in that, After forming a queue to be tested based on the selected test cases, the test case management module is further used to initialize a resource pool based on the heterogeneous accelerator card currently being tested and the allocable resources in the basic test environment, forming a current test environment for the heterogeneous accelerator card currently being tested, and the resources in the initialized resource pool are used as resources for the test cases in the queue to be tested.

6. The automatic testing system as described in claim 1, characterized in that, The test scheduling module includes: a performance information acquisition module and a fault diagnosis and recovery module; During the execution of the current test case by the test scheduling module, the performance information acquisition module is used to collect the performance status information of the current test environment and send the performance status information of the current test environment to the fault diagnosis and recovery module. The fault diagnosis and recovery module is used to determine whether the current test environment is abnormal based on the performance status information, and to handle the abnormality when the current test environment is abnormal.

7. The automatic testing system as described in claim 6, characterized in that, After handling the anomaly, the fault diagnosis and recovery module is further configured to detect the post-anomaly status information of the current test environment, and based on the post-anomaly status information, determine whether the current test environment supports continuing subsequent tests. If not, the test is terminated and a warning signal is output.

8. The automatic testing system as described in claim 7, characterized in that, The performance status information of the current test environment includes: device performance status information, operating system performance status information, and heterogeneous accelerator card performance status information; the anomalies include: driver problems and / or cross-chip interconnection problems; The method for handling the driver problem is: reloading the driver; The method for handling the cross-chip connection problem is to rebuild the chip connection.

9. The automatic testing system as described in claim 6, characterized in that, Also includes: Test results summary module; The test result aggregation module is used to collect the test results obtained by executing the current test case in the current test environment. The performance information acquisition module is also used to send the performance status information to the test result aggregation module; The fault diagnosis and recovery module is also used to send the status information after the anomaly handling to the test result summary module; The test result summary module is also used to summarize and output the performance status information and the post-anomaly handling status information along with the test results.

10. The automatic testing system as described in claim 9, characterized in that, After obtaining the test results, the test scheduling module is also used to: release the resources used by the test cases that obtained the test results to the resource pool, and increase the current remaining resources.

11. An automatic testing method, applied to the automatic testing system as described in any one of claims 1-10, characterized in that, include: Identify the current test cases to be executed in the test queue; For the current test case, when it is determined that the remaining resources in the current test environment of the automated test system cannot support the execution of the current test case, the current test case is marked, and the next test case to be executed in the test queue is updated to the current test case; Furthermore, when it is determined that all test cases in the test queue are marked, the test cases are sorted according to their resource requirements. When the remaining resources increase, the current test case to be executed is determined from the sorted test cases. The test cases in the test queue are test cases of the heterogeneous accelerator cards currently being tested that are adapted to the current test environment.

12. The automatic testing method as described in claim 11, characterized in that, During the execution of the current test case, the automated testing method further includes: Collect performance status information of the current test environment; based on the performance status information, determine whether the current test environment has experienced any abnormalities; If so, then handle the exception.

13. The automatic testing method as described in claim 12, characterized in that, After the step of handling the exception, the method further includes: detecting the post-exception status information of the current test environment; based on the post-exception status information, determining whether the current test environment after handling the exception supports continuing subsequent testing; if not, ending the test and outputting a warning signal.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed, it implements the automatic testing method as described in any one of claims 11-13.

15. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a program, and the processor calling the program stored in the memory to execute the automatic testing method as described in any one of claims 11-13.

16. A storage medium, characterized in that, The storage medium stores a program that, when executed, implements the automatic testing method as described in any one of claims 11-13.

Images