System and method for stress testing artificial intelligence systems on heterogeneous architectures

Abstract

The application provides a method and system for stress testing of an artificial intelligence system with a heterogeneous architecture. The method for stress testing of an artificial intelligence system with a heterogeneous architecture includes: performing statistics on historical test data of previous stress tests in a historical database to determine the correlation between test environments, test case types, fault modes, and test durations; generating a personalized test strategy for each of a plurality of test environments under test of the artificial intelligence system with a heterogeneous architecture under test based on the correlation between the test environments, the test case types, the fault modes, and the test durations; and performing stress testing on the plurality of test environments under test in parallel based on the personalized test strategy of each of the test environments under test.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence operation and maintenance technology, specifically to a system and method for stress testing of artificial intelligence systems with heterogeneous architectures. Background Technology

[0002] With the explosive growth in demand for artificial intelligence (AI) computing power, AI systems (AI data centers) commonly adopt heterogeneous architectures. Heterogeneous architectures may involve the mixed deployment of servers from different vendors (such as Dell, H3C, Ningchang, and Sugon), running different operating systems (such as Ubuntu, Debian, Red Hat, and CentOS), and / or various AI accelerator cards (such as Moffett S40, S30, and S4). Under heterogeneous architectures, ensuring system stability through stress testing presents significant challenges. Summary of the Invention

[0003] In one aspect, embodiments of this application provide a method for stress testing a heterogeneous artificial intelligence system, comprising: statistically analyzing historical test data from previous stress tests in a historical database to determine the correlation between test environments, test case types, failure modes, and test durations; generating personalized test strategies for each of multiple test environments of the heterogeneous artificial intelligence system under test based on the correlation between the test environments, test case types, failure modes, and test durations; and performing stress tests on the multiple test environments in parallel based on the generated personalized test strategies for each test environment.

[0004] In another aspect, embodiments of this application provide a system for stress testing a heterogeneous artificial intelligence system, comprising: a historical database analysis module configured to statistically analyze historical test data from previous stress tests in a historical database to determine the correlation between test environments, test case types, failure modes, and test durations; a strategy generation module configured to generate personalized test strategies for each of multiple test environments of the heterogeneous artificial intelligence system under test based on the correlation between the test environments, test case types, failure modes, and test durations; and a test execution module configured to perform stress tests on the multiple test environments in parallel based on the generated personalized test strategies for each test environment.

[0005] In another aspect, embodiments of this application provide a non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for stress testing an artificial intelligence system with a heterogeneous architecture, according to embodiments of this application.

[0006] In another aspect, embodiments of this application provide a computer program product storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for stress testing an artificial intelligence system with a heterogeneous architecture, according to embodiments of this application.

[0007] The system and method for stress testing a heterogeneous artificial intelligence system according to embodiments of this application can determine personalized testing strategies for each test environment under test and perform stress tests on multiple test environments in parallel by analyzing historical test data. This significantly reduces testing time while ensuring testing reliability, and greatly improves the efficiency and intelligence level of the system testing. Attached Figure Description

[0008] When read in conjunction with the accompanying drawings, various aspects of this disclosure are best understood through the following detailed description. It should be understood that the drawings are for illustrative and descriptive purposes only and not for limiting purposes. In the drawings:

[0009] Figure 1 This is a schematic diagram of the stress test operation architecture according to an embodiment of this application.

[0010] Figure 2 This is a flowchart of a method for stress testing a heterogeneous artificial intelligence system according to an embodiment of this application.

[0011] Figure 3 This is a flowchart illustrating the generation of a personalized testing strategy according to an embodiment of this application.

[0012] Figure 4 This is a flowchart illustrating a dynamic test plan for determining the first test in each test environment according to an embodiment of this application.

[0013] Figure 5 This is a flowchart illustrating the determination of the optimal test duration interval for each test environment in historical test data, according to an embodiment of this application.

[0014] Figure 6 This is a flowchart illustrating the process of determining the optimal execution time for each test environment under test from the optimal test duration range, according to an embodiment of this application.

[0015] Figure 7 This is a flowchart illustrating the determination of the applicable variant type for performing a second test in each test environment according to embodiments of this application.

[0016] Figure 8 This is a flowchart for determining the preferred variant type of the second test in each test environment in historical test data according to an embodiment of this application.

[0017] Figure 9This is a flowchart illustrating the determination of the applicable variant type of the second test in each test environment under test, based on preferred variant types according to embodiments of this application.

[0018] Figure 10 This is a flowchart of a method for determining the priority scores of each of the multiple variant types of the second test according to an embodiment of this application.

[0019] Figure 11 This is a block diagram of a system for stress testing a heterogeneous artificial intelligence system according to an embodiment of this application.

[0020] Figure 12 This is a schematic diagram of a computing device that enables stress testing of a heterogeneous artificial intelligence system according to embodiments of this application. Detailed Implementation

[0021] The following disclosure provides numerous different embodiments or examples for implementing various features of the provided subject matter. Specific examples of components and arrangements are described below to simplify this disclosure. Of course, these are merely examples and not limiting.

[0022] In the heterogeneous architecture of AI systems, stress testing is no longer about verifying the performance of a single device, but rather involves collaborative verification between different devices. Currently, most stress testing solutions in the industry have evolved from testing methods for homogeneous hardware architectures, typically employing full-load stress testing, also known as full-chain / full-stack stress testing. This method applies saturated loads to all components / processes of the system, from front-end to back-end, and from hardware to software, to verify extreme stability and bottlenecks. For example, in full-load stress testing, each server needs to undergo a complete load test and a full set of memory tests mechanically.

[0023] Through long-term practice and analysis, the inventors of this application have discovered the following prominent pain points and defects in the full-scale stress testing method: 1) Extremely long testing time and low efficiency: In heterogeneous hardware architectures, each iteration of full-scale stress testing takes up to 50 hours, which seriously slows down the pace of hardware deployment, operation and maintenance verification, and fault reproduction, becoming a bottleneck in the R&D and delivery process; 2) Huge resource consumption and high cost: Long-term full-load stress testing continuously occupies valuable AI computing power servers, making them unusable for model training or inference tasks, resulting in huge opportunity costs; 3) Poor test targeting and insufficient intelligence: Traditional "flood irrigation" testing fails to utilize historical test data, ignores the unique fault modes of different hardware and software combinations, under-tests high-fault areas, and over-tests stable areas, which is a "blind" testing strategy; 4) Static and rigid strategy, unable to evolve: Test cases and durations are pre-defined statically and cannot be dynamically adjusted and optimized based on the effectiveness of historical test results, making it impossible to achieve self-iteration and improvement of the testing strategy.

[0024] In view of the problems existing in the existing stress testing schemes, this application proposes a system and method for stress testing artificial intelligence systems, which can significantly shorten the testing time, reduce resource consumption, improve the testing focus, and have self-learning capabilities to meet the quality assurance requirements under the heterogeneous hardware architecture of artificial intelligence.

[0025] Figure 1 This is a schematic diagram of the stress test execution architecture according to an embodiment of this application. Figure 1 As shown, the stress test execution architecture 100 according to an embodiment of this application includes a test management platform 101 and an AI system (also called an AI data center) 102 that communicates with the test management platform 101. The communication between the test management platform 101 and the AI ​​system 102 can be implemented using direct communication methods based on network protocols (e.g., through application programming interfaces (APIs) or container orchestration engines), cloud-native and service mesh-based communication methods (e.g., through the control plane of a service network or a cloud-native API gateway), communication methods based on automated operation and maintenance tools (e.g., through CI / CD pipelines), and dedicated protocol and hardware-level communication methods (e.g., through the Intelligent Platform Management Interface (IPMI)).

[0026] AI system 102 includes a heterogeneous test environment cluster 1021, which includes test environment 1, test environment 2, test environment 3, and test environment 4. It should be understood that although the figure shows the heterogeneous test environment cluster 1021 comprising four test environments, the embodiments of this application are not limited to this, and may include more or fewer test environments depending on requirements. Test environment 1, test environment 2, test environment 3, and test environment 4 may each include hardware-software combinations with different configurations, such as consisting of different types of servers, operating systems, and accelerator cards. For example, test environment 1 includes a Dell server, Ubuntu system, and S40 accelerator card for large-scale training scenarios; test environment 2 includes an H3C server, Red Hat system, and S30 accelerator card for high-concurrency inference scenarios; test environment 3 includes a Ningchang server, CentOS system, and S4 accelerator card for edge inference scenarios; and test environment 4 includes a Sugon server, Debian system, and S40 accelerator card for mixed load scenarios. It should be understood that these are merely examples, and the embodiments of this application are not limited thereto; other test environments with different configurations may be used. It should also be understood that the AI ​​system 102 may include any other components well-known in the art (e.g., supporting storage, network infrastructure, etc.), the description of which has been omitted to avoid unnecessarily obscuring this application.

[0027] As the "command and dispatch center" for stress testing, the Test Management Platform 101 is the core of decision-making and control for the entire stress testing system. The Test Management Platform 101 is responsible not only for the full lifecycle management of test cases, including case creation, version iteration, scenario-based classification, and reuse, but also, based on the heterogeneous architecture characteristics of the AI ​​System 102, intelligently allocates differentiated testing strategies for different hardware-software combinations (such as multi-vendor servers, multiple operating systems, and multiple accelerator cards) in the heterogeneous test environment cluster 1021. For example, it configures long-term high-load stability testing strategies for training test environments and high-concurrency peak impact strategies for inference test environments.

[0028] In addition, the test management platform 101 is also responsible for the automated orchestration and dynamic scheduling of test tasks. It can automatically adjust the number of stress nodes according to the cluster resource load to avoid waste of test resources. During test execution, the test management platform 101 will collect full-link monitoring data from the AI ​​system 102 in real time, including hardware indicators, system status, business performance, etc., and complete bottleneck location, root cause tracing and capacity assessment through the built-in intelligent analysis engine. Finally, it generates a visualized test report to provide data support for the optimization and capacity planning of heterogeneous architecture.

[0029] AI System 102, serving as the "test environment carrier" for stress testing, is an isolated test environment that fully replicates production-grade configurations. It fully supports all hardware and software resources of the heterogeneous test environment cluster 1021, simulating load characteristics under real business scenarios. The core function of AI System 102 is to verify the compatibility, stability, and fault tolerance of heterogeneous test environments under extreme scenarios such as high pressure and failures in a controlled, isolated environment. It supports both benchmark performance testing of a single test environment and end-to-end stress testing of multiple test environments, ensuring the consistency and reference value of test results with the production environment.

[0030] The following will combine Figures 2-5 A method and system for stress testing a heterogeneous artificial intelligence system according to embodiments of this application are described in more detail.

[0031] Figure 2 This is a flowchart illustrating a method for stress testing a heterogeneous artificial intelligence system according to an embodiment of this application. Figure 2 As shown, the method 200 for stress testing a heterogeneous artificial intelligence system according to an embodiment of this application includes steps S201-S203.

[0032] In step S201, the historical test data of each stress test in the historical database are statistically analyzed to determine the correlation between the test environment, test case type, failure mode and test duration.

[0033] In step S202, based on the correlation between test environment, test case type, failure mode and test duration, a personalized test strategy is generated for each of the multiple test environments of the heterogeneous AI system under test.

[0034] In step S203, stress tests are performed in parallel on the multiple test environments based on the personalized test strategies generated for each test environment.

[0035] In some implementations, historical test data may include: test environment data, test process data, and test result data. Test environment data includes: server brand and model, operating system type and version, accelerator card model and driver version; test process data includes: test case type, test duration, and test parameters; and test result data includes: whether a fault occurred, the type of fault, and the time when the fault occurred.

[0036] In some implementations, test case types can include multiple variants of a first test for testing environment stability and a second test for testing environment defects. The fault detection in the first test has a clear time correlation. The fault detection in the second test targets hardware defects in the test environment. In some implementations, the first test can be a stress test, and the second test can be a memtest test. Stress tests are used to test the stability of the environment, such as testing the interoperability of servers, operating systems, and accelerator cards; their fault triggering has a clear time correlation. Memtest tests detect physical defects in the environment's hardware itself (such as address line failures, data bit flips, and parity errors), and belong to hardware functional testing.

[0037] In some implementations, such as Figure 3 As shown, generating a personalized testing strategy may include: step S301, determining the optimal execution time for the first test in each of the multiple test environments based on the correlation between the test environment, test case type, failure mode, and test duration; and step S302, determining the applicable variant type for the second test in each of the multiple test environments based on the correlation between the test environment, test case type, failure mode, and test duration through weighted random screening.

[0038] In some implementations, such as Figure 4 As shown, step S301 of determining the dynamic test plan for the first test in each of the multiple test environments under test may include: step S3011, determining the optimal test duration range for the first test in each test environment from historical test data based on the correlation between test environment, test case type, failure mode, and test duration; step S3012, determining the optimal execution duration for the first test in each of the multiple test environments under test based on the optimal test duration range for the first test in each test environment from historical test data; and step S3013, generating a dynamic test plan for each of the multiple test environments under test based on the optimal execution duration for the first test in each of the multiple test environments under test, with a period not exceeding a predetermined total duration and the optimal execution duration as the period.

[0039] In some implementations, such as Figure 5As shown, step S3011, which determines the optimal test duration interval for the first test under each test environment in the historical test data, may include: step S3011-1, for each test environment in the historical test data, determining the total number of failures occurring in all first tests corresponding to that test environment; step S3011-2, assigning each failure occurring in each first test corresponding to that test environment to a corresponding duration interval within at least one preset duration interval based on the time point of the failure occurrence; step S3011-3, counting the number of failures within each duration interval in at least one duration interval, and based on each The failure rate of each time interval is determined by the number of failures within the time interval and the total number of failures; step S3011-4, based on the failure rate of each time interval in at least one time interval, the cumulative failure rate of that time interval is determined, the cumulative failure rate is the sum of the failure rate of that time interval and the failure rates of all time intervals before that time interval; and step S3011-5, the cumulative failure rate of each time interval is compared with a preset cumulative failure rate threshold, and the time intervals in which the cumulative failure rate reaches or exceeds the cumulative failure rate threshold are determined as the optimal test time intervals for the first test under the test environment.

[0040] In some implementations, such as Figure 6 As shown, step S3012, which determines the optimal execution time for the first test in each of the multiple test environments, may include: step S3012-1, matching each test environment with each test environment in the historical database; and step S3012-2, taking the longest time interval of the optimal test duration for the first test in the test environment matched with the test environment in the historical database as the optimal execution time for the first test in the test environment.

[0041] As an example, for a certain test environment, there are 10 sets of stress test data in the historical test data, with a total of 26 failures. Four preset duration intervals are set: 0-2 hours, 2-4 hours, 4-6 hours, and 6-24 hours. The number of failures in each duration interval and its percentage relative to the total number of failures are calculated. For example, the failure percentage in the 0-2 hour interval is 15.4%, in the 2-4 hour interval it is 26.9%, in the 4-6 hour interval it is 34.6%, and in the 6-24 hour interval it is 11.5% (in this example, the cumulative failure percentage reaches 88.4%, and the remaining 11.6% is the "implicit stable sample" corresponding to the failure-free test, which does not affect the duration optimization conclusion). In this example, nearly 77% of the failures are concentrated in the 0-6 hour interval, with the 4-6 hour interval having the highest failure percentage (34.6%), while the 6-24 hour interval has only 11.5% failure percentage. Based on this, if the cumulative failure rate is set to ≥75% as the preset threshold, the optimal test duration range can be determined to be 4-6 hours.

[0042] The optimal test duration range is determined to be 4-6 hours, meaning that stability failures exceeding a preset proportion (e.g., 75%) are concentrated within the first 6 hours of the test. After 6 hours, the rate of increase in failure detection significantly decreases. By finding a test environment matching the environment under test in historical test data, the longest duration of this optimal test duration range (e.g., 6 hours) can be set as the optimal execution duration for that test environment. Assuming a predetermined total duration of, for example, 24 hours, a stress test can be performed on the test environment under test every 6 hours within 24 hours, thus providing a dynamic test plan. According to the test embodiment of this application, by exploring the distribution pattern of historical test durations and eliminating invalid time consumption, test efficiency can be improved compared to the traditional fixed test duration method.

[0043] In some implementations, such as Figure 7 As shown, step S302, which determines the applicable variant type for the second test in each of the multiple test environments under test through weighted random screening, further includes: step S3021, determining the fault detection rate, detection time, and core fault detection items for each variant type of the second test in each test environment based on the correlation between the test environment, test case type, fault mode, and test duration; step S3022, determining the preferred variant type of the second test in each test environment based on the fault detection rate, detection time, and core fault detection items for each variant type of the second test in each test environment; and step S3023, determining the applicable variant type for the second test in each of the multiple test environments under test based on the preferred variant type of the second test in each test environment.

[0044] In some implementations, such as Figure 8 As shown, step S3022, which determines the preferred variant type of the second test under each test environment based on the fault detection rate, detection time, and core fault detection items of each variant type of the second test in each test environment, includes: step S3022-1, determining the overlap rate between the core fault detection items of the multiple variant types of the second test based on the core fault detection items of each variant type of the second test, and determining the adaptability of the multiple variant types of the second test to the test environment; step S3022-2, selecting a suitable variant type from the variant types with an overlap rate higher than a preset overlap rate threshold based on the fault detection rate, detection time, or adaptability of each variant type. The test involves several steps: S3022-3, determining the priority scores of each variant type in the second test based on their respective fault detection rates, detection times, preset fault detection rate weights, and preset detection time weights; S3022-4, determining the probability interval corresponding to the fault detection probability of each candidate variant type based on their respective priority scores; and S3022-5, generating at least one random number and selecting a variant type from the candidate variant types based on the probability interval corresponding to the generated random number, as the preferred variant type for the second test under this test environment.

[0045] In some implementations, such as Figure 9 As shown, step S3023, which determines the applicable variant type for performing a second test in each of the multiple test environments from the preferred variant type, includes: step S3023-1, matching each test environment in the multiple test environments with each test environment in the historical database; and step S3023-2, taking the preferred variant type of the second test in the test environment in the historical database that matches the test environment as the applicable variant type for performing a second test in the test environment.

[0046] In some implementations, such as Figure 10As shown, step S3022-3, which determines the priority score of each variant type of the second test based on their respective fault detection rate, detection time, preset fault detection rate weight, and preset detection time weight, includes: step S3022-31, for each variant type in the second test, multiplying the fault detection rate of that variant type by the preset fault detection rate weight as the fault detection rate score of that variant type; step S3022-32, determining the difference between 1 and the ratio of the detection time of that variant type to the longest detection time among the multiple variant types; step S3033-33, multiplying the difference by the preset detection time weight as the detection time score of that variant type; and step S3022-34, using the sum of the fault detection rate score and the detection time score of that variant type as the priority score of that variant type.

[0047] As an example, for a certain test environment (e.g., a hardware cluster tagged as: DDR5 multi-channel ECC memory cluster), the historical test data includes five variant types of the second test, such as MemTest86, MemTest Pro, memtester, Sugon customized version of MemTest, and MemTest64. The historical data for these five variant types are shown in Table 1 below. The preset fault detection rate weight is 60%, and the preset detection time weight is 40%.

[0048] Table 1

[0049]

[0050] From Table 1 above, for a specific variant type, the fault detection rate is the percentage of the number of tests that detected faults to the total number of tests.

[0051] From Table 1 above, the priority of MemTest86 can be determined as: (0.85×0.6) + (1-40 / 240)×0.4 = 0.51 + 0.33 = 0.843. Similarly, the priority score of MemTest Pro is 0.652; the priority score of memtester is 0.77; the priority score of the Dawning customized version of MemTest is 0.845; and the priority score of MemTest64 is 0.54.

[0052] Table 1 above also shows that the overlap rate of core fault detection items between MemTest86 and MemTestPro is 3 / 5 = 60% (3 overlapping detection items: address lines, data bit flipping, parity check). Since MemTestPro takes 4.5 times longer than MemTest86 but only improves the detection rate by 7%, it is excluded. The overlap rate of core fault detection items between MemTest86 and MemTest64 is 2 / 3 ≈ 67% (2 overlapping detection items: address lines, data bit flipping). Since MemTest64 takes 6 times longer than MemTest86 but only improves the detection rate by 5%, it is excluded. The overlap rate of core fault detection items between the Dawning customized version of MemTest and MemTest86 is 2 / 3 ≈ 67% (2 overlapping detection items: address lines, parity check), but the Dawning customized version adds "DDR5 multi-channel compatibility detection" (adapting to DDR5 memory in this environment), so the Dawning customized version is retained. The Dawning-customized versions of MemTest, MemTest86, and memtester were selected as candidate variant types.

[0053] The total weight score of the candidate variant types, Dawning Customized MemTest, MemTest86, and memtester, is 2.458. The fault detection probability of Dawning Customized MemTest is approximately 34.4% (0.845 / 2.458); the fault detection probability of MemTest86 is approximately 34.3%; and the fault detection probability of memtester is approximately 31.3%. Two selections were performed using a random number generator (0-1 interval): the first random number was 0.28, falling within the probability interval [0, 0.344), thus selecting Dawning Customized MemTest; the second random number was 0.55, falling within the probability interval [0.344, 0.687), thus selecting MemTest86. Therefore, the preferred variant types for the second test in this testing environment are Dawning Customized MemTest and MemTest86.

[0054] Based on the preferred variant type, the applicable variant types for the second test under the test environment can be the Dawning customized version of MemTest and MemTest86, thereby covering: address line failure, data bit flip, parity error, and DDR5 multi-channel compatibility error.

[0055] In some implementations, if it is also necessary to detect cache consistency errors for the test environment under test, since the above two variant types do not fully cover the differences, memtester is added based on priority score, but memtester does not cover cache consistency errors; further, MemTestPro (which covers cache consistency errors) is added based on priority score, which is the second test option for the test environment under test: Sugon customized version MemTest, MemTest86, and MemTestPro (which only executes the "cache consistency detection" sub-item, greatly reducing the time consumption).

[0056] It should be understood that the above is merely illustrative, and the embodiments of this application are not limited to the specific values ​​and selections mentioned above. Generally, 2-3 preferred memtest variants can be selected for each test environment. Through the above method, core fault detection items can be covered for each test environment under test, avoiding vulnerabilities caused by fixed test patterns. Simultaneously, test cases are significantly simplified, leading to a substantial reduction in the testing time for each round, thereby improving testing efficiency. Furthermore, by concurrent execution for multiple test environments under test, a leap in efficiency can be achieved.

[0057] In some implementations, a reference is returned. Figure 2 As shown, the method 200 for stress testing a heterogeneous AI system according to an embodiment of this application may further include: step S204, collecting test data from stress testing multiple test environments under test, and updating a historical database based on the collected data. The collected test data includes, but is not limited to, test pass / fail, generated logs, performance data, etc. Through feedback updates, a closed-loop learning system is formed. The continuously enriched historical test data makes the analysis steps more accurate, thereby driving the continuous optimization of the generated test strategy and forming a virtuous cycle of self-improvement.

[0058] The method for stress testing a heterogeneous AI system according to embodiments of this application analyzes historical test data to determine personalized testing strategies for each test environment and executes stress tests on multiple test environments in parallel. This significantly reduces testing time while ensuring testing reliability, greatly improving the efficiency of system testing. Furthermore, by reducing invalid or redundant test loads, the computing resources of the AI ​​system can be released back to production or R&D tasks, improving resource utilization efficiency and thus reducing operating costs.

[0059] Figure 11 This is a block diagram of a system for stress testing a heterogeneous artificial intelligence system according to an embodiment of this application. Figure 11As shown, the system 1100 for stress testing a heterogeneous AI system according to an embodiment of this application includes: a historical database analysis module 1101, configured to statistically analyze historical test data from previous stress tests in a historical database to determine the correlation between test environments, test case types, failure modes, and test durations; a strategy generation module 1102, configured to generate personalized test strategies for each of the multiple test environments of the heterogeneous AI system under test based on the correlation between test environments, test case types, failure modes, and test durations; and a test execution module 1103, configured to perform stress tests on multiple test environments in parallel based on the generated personalized test strategies for each test environment.

[0060] In some implementations, system 1100 may also include: a feedback update module 1104, configured to collect test data from stress tests on multiple test environments under test, and to update a historical database based on the collected data.

[0061] It should be understood that Figure 11 Each module can be referred to in the preceding method embodiments, and will not be repeated here. The system for stress testing heterogeneous AI systems according to embodiments of this application, by analyzing historical test data, can determine personalized testing strategies for each test environment and execute stress tests on multiple test environments in parallel. This significantly reduces testing time while ensuring testing reliability, greatly improving system testing efficiency. Furthermore, by reducing invalid or redundant test loads, the computing resources of the AI ​​system can be released back to production or R&D tasks, improving resource utilization efficiency and thus reducing operating costs.

[0062] Figure 12 This is a schematic diagram of a computing device capable of performing stress tests on a heterogeneous artificial intelligence system according to embodiments of this application. For example... Figure 12 As shown, computing device 1200 may include bus 1202 or other communication mechanism for transmitting information, and one or more processors 1204 coupled to bus 1202 for processing information. The one or more processors 1204 may include, for example, one or more general-purpose microprocessors.

[0063] like Figure 12As shown, in some implementations, computing device 1200 may further include main memory 1206 coupled to bus 1202. Main memory 1206 is used to store information (e.g., a historical database) and instructions executed by one or more processors 1204, such as random access memory (RAM), cache, and / or other dynamic storage devices. Main memory 1206 may also be used to store temporary variables or other intermediate information during the execution of instructions executed by one or more processors 1204. When these instructions are stored in storage media accessible to one or more processors 1204, they can cause computing device 1200 to become a dedicated machine customized to perform the operations specified in the instructions. Storage device 1208 may include non-volatile and / or volatile storage media. Non-volatile storage media may include, for example, optical disks or magnetic disks. Volatile storage media may include dynamic memory. Common forms of storage media may include, for example, floppy disks, hard disks, solid-state drives, magnetic tapes, or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, DRAM, PROM, EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge, or their networking versions.

[0064] like Figure 12 As shown, in some embodiments, computing device 1200 may further include one or more communication interfaces or network interfaces 1210 coupled to bus 1202. Network interface 1210 may provide bidirectional data communication coupling to one or more network links connected to one or more networks. As another example, network interface 1210 may be a local area network (LAN) card to provide data communication connectivity to a LAN-compatible (or WAN component communicating with a WAN) network. Wireless links may also be implemented.

[0065] The execution of certain operations can be distributed across processors rather than residing within a single machine, but rather deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0066] Each of the processes, methods, and algorithms described in the preceding sections may be embodied in code modules executed by one or more computer systems or computer processors including computer hardware, and may be fully or partially automated by these code modules. The processes and algorithms may be implemented, partially or fully, in dedicated circuit systems.

[0067] When the functions disclosed herein are implemented as software functional units and sold or used as stand-alone products, they may be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, may be embodied in the form of a software product. The software product may be stored in a storage medium and includes several instructions that cause a computing device (which may be a personal computer, server, network device, etc.) to perform all or some steps of the methods of the embodiments of this application. The storage medium may include a flash drive, portable hard disk drive, ROM, RAM, magnetic disk, optical disk, other media operable to store program code, or any combination thereof.

[0068] According to embodiments of this application, a system is provided that includes a processor and a non-transitory computer-readable storage medium storing instructions, which are executable by the processor to cause the system to perform operations corresponding to steps in any method of the embodiments disclosed above. According to embodiments of this application, a non-transitory computer-readable storage medium or computer program product storing instructions, which are executable by one or more processors to cause the one or more processors to perform operations corresponding to steps in any method of the embodiments disclosed above.

[0069] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Additionally, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and their associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in an order other than that specifically disclosed, or multiple blocks or states may be combined into a single block or state. Example blocks or states may be executed sequentially, in parallel, or in some other manner. Certain blocks or states may be added to or removed from the disclosed example embodiments. The exemplary systems and components described herein may be configured differently than described. For example, certain components may be added to, removed from, or rearranged compared to the disclosed example embodiments.

[0070] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. The algorithm may be included in program code or instructions stored in memory (e.g., the aforementioned non-transitory computer-readable storage medium). This algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not explicitly turn the computer into an execution function but may learn from training data to produce a predictive model of the execution function.

[0071] The various operations of the exemplary methods described herein can be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, these processors can constitute an engine of processor implementation that operates to perform one or more of the operations or functions described herein.

[0072] Similarly, the methods described herein may be implemented at least in part by a processor, wherein one or more specific processors are instances of hardware. For example, at least some operations of the methods may be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors may also be operable to support the execution of relevant operations in a “cloud computing” environment or as the execution of relevant operations in a “Software as a Service” (SaaS) context. For example, at least some operations may be performed by a group of computers (as an example of a machine containing processors), wherein these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0073] In this specification, although individual operations of one or more methods are illustrated and described as separate operations, one or more of these individual operations may be performed simultaneously, and not necessarily in the order illustrated. Structures and functions presented as separate components in the example configurations may be implemented as composite structures or components. Similarly, structures and functions presented as single components may be implemented as single components. These and other variations, modifications, additions, and improvements fall within the scope of this document. Therefore, this specification and its drawings should be considered illustrative rather than restrictive.

[0074] As used herein, “or” is inclusive rather than exclusive unless explicitly indicated by the context. Therefore, in this document, “A, B, or C” means “A, B, A and B, A and C, B and C, or A, B, and C” unless explicitly indicated by the context. Furthermore, “and” is combined and separate unless explicitly indicated by the context. Therefore, in this document, “A and B” means “A and B, combined or separate” unless explicitly indicated by the context.

[0075] The terms “comprising” or “including” are used to indicate the presence of a subsequently claimed feature, but do not exclude the presence of other features. Unless otherwise specifically stated or otherwise understood in the context in which they are used, conditional language such as “may,” “can,” “may,” and “can” is generally intended to convey that certain embodiments include certain features, components, and / or steps that are not included in other embodiments. Therefore, this conditional language is generally not intended to imply that one or more embodiments necessarily require a certain feature, component, and / or step in any way, or that one or more embodiments must include such features, components, and / or steps.

[0076] Although the general outline of the subject matter has been described with reference to specific exemplary embodiments, various modifications and changes may be made to these embodiments without departing from the broad scope of embodiments of this disclosure. Where more than one embodiment is disclosed, these embodiments of the subject matter may be referred to individually or collectively herein as the term "invention," this is for convenience only and is not intended to automatically limit the scope of this application to any single disclosure or concept.

[0077] The embodiments illustrated herein are described in detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. Therefore, the term "implementation" is not intended to be limiting, and the scope of the various embodiments is defined only by the appended claims and their equivalents.

Claims

1. A method for stress testing a heterogeneous artificial intelligence system, characterized in that... include: Statistical analysis is performed on historical test data from each stress test in the historical database to determine the correlation between test environment, test case type, failure mode, and test duration. Based on the correlation between the test environment, test case type, failure mode and test duration, a personalized test strategy is generated for each of the multiple test environments of the heterogeneous AI system under test. as well as Based on the personalized testing strategy generated for each test environment, stress tests are performed in parallel on the multiple test environments. The test case types include multiple variant types of a first test for testing environmental stability and a second test for testing environmental defects. The generation of the personalized testing strategy includes: Based on the correlation between the test environment, test case type, failure mode, and test duration, a dynamic test plan is determined for performing the first test in each of the multiple test environments; and Based on the correlation between the test environment, test case type, failure mode, and test duration, a weighted random selection is used to determine the applicable variant type for the second test in each of the multiple test environments. The applicable variant types for performing the second test in each of the plurality of test environments are determined through the weighted random screening, including: Based on the correlation between the test environment, test case type, fault mode and test duration, the fault detection rate, detection time and core fault detection items of each variant type of the second test in each test environment in the historical test data are determined. Based on the fault detection rate, detection time, and core fault detection items of multiple variant types of the second test in each test environment, the preferred variant type of the second test in each test environment is determined; and Based on the preferred variant type of the second test in each test environment, determine the applicable variant type for performing the second test in each of the plurality of test environments, and The preferred variant type of the second test in each test environment is determined based on the fault detection rate, detection time, and core fault detection items of each variant type of the second test in each test environment, including: Based on the core fault detection items of each of the multiple variant types of the second test, determine the overlap rate between the core fault detection items of the multiple variant types of the second test, and determine the adaptability of the multiple variant types of the second test to the test environment. From the variant types whose overlap rate is higher than a preset overlap rate threshold, a suitable variant type is selected as a candidate variant type based on the fault detection rate, detection time, or adaptability of each variant type. Based on the fault detection rate, detection time, preset fault detection rate weight, and preset detection time weight of each of the multiple variant types of the second test, the priority score of each of the multiple variant types of the second test is determined. Based on the priority scores of each candidate variant type, the probability interval corresponding to the fault detection probability of each candidate variant type is determined; and At least one random number is generated, and based on the probability interval corresponding to the generated random number, a candidate variant type is selected from the candidate variant types as the preferred variant type for the second test in this test environment.

2. The method according to claim 1, characterized in that, The historical test data includes: test environment data, test process data, and test result data, among which, The test environment data includes: server brand and model, operating system type and version, accelerator card model and driver version; The test process data includes: test case type, test duration, test parameters; and The test result data includes: whether a fault occurred, the type of fault, and the time when the fault occurred.

3. The method according to claim 1, characterized in that, The first test is the stress test, and the second test is the memtest test.

4. The method according to claim 1, characterized in that, Determining a dynamic test plan for the first test in each of the plurality of test environments includes: Based on the correlation between the test environment, test case type, failure mode and test duration, the optimal test duration range for performing the first test in each test environment in the historical test data is determined. Based on the optimal test duration range for the first test in each test environment from the historical test data, the optimal execution duration for the first test in each of the multiple test environments is determined; and Based on the optimal execution time of the first test in each of the plurality of test environments, a dynamic test plan is generated for each of the plurality of test environments, with a period not exceeding a predetermined total time and the optimal execution time as the period.

5. The method according to claim 4, characterized in that, Determining the optimal test duration range for the first test under each test environment in the historical test data includes: For each test environment in the historical test data, determine the total number of failures that occurred in all the first tests corresponding to that test environment; Each fault occurring in each of the first tests corresponding to the test environment is assigned to a corresponding time interval in at least one preset time interval based on the time point at which the fault occurred. The number of failures in each time interval within the at least one time interval is counted, and the failure percentage of each time interval is determined based on the number of failures in each time interval and the total number of failures; Based on the failure rate of each time interval within the at least one time interval, a cumulative failure rate for that time interval is determined, wherein the cumulative failure rate is the sum of the failure rate of that time interval and the failure rates of all time intervals preceding that time interval; and The cumulative failure rate of each time interval is compared with a preset cumulative failure rate threshold, and the time interval in which the cumulative failure rate reaches or exceeds the cumulative failure rate threshold is determined as the optimal test time interval for the first test under the test environment.

6. The method according to claim 4, characterized in that, Determining the optimal execution time for the first test in each of the plurality of test environments includes: For each of the multiple test environments, match the test environment with each test environment in the historical database; and The longest time interval of the optimal test duration range for the first test in the historical database that matches the test environment to be tested is taken as the optimal execution time for the first test in the test environment to be tested.

7. The method according to claim 1, characterized in that, Based on the preferred variant type of the second test in each of the plurality of test environments, the applicable variant types for performing the second test in each test environment include: For each of the multiple test environments: The test environment to be tested is matched with each test environment in the historical database; and The preferred variant type of the second test in the test environment that matches the test environment under the historical database is used as the applicable variant type for performing the second test in the test environment under the test environment.

8. The method according to claim 1, characterized in that, Based on the fault detection rate, detection time, and preset fault detection rate weight and preset detection time weight for each of the multiple variant types of the second test, the priority scores for each of the multiple variant types of the second test are determined as follows: For each of the multiple variant types in the second test: The product of the fault detection rate of this variant type and the preset fault detection rate weight is used as the fault detection rate score of this variant type; Determine the difference between the detection time of 1 and the ratio of the detection time of this variant type to the longest detection time among the plurality of variant types; The product of the difference and the preset detection time weight is used as the detection time score for this variant type; and The sum of the fault detection rate score and the detection time score for that variant type is used as the priority score for that variant type.

9. The method according to claim 1, further comprising: Collect test data from stress testing the multiple test environments under test, and update the historical database based on the collected data.

10. A system for stress testing of heterogeneous artificial intelligence systems, characterized in that... include: The historical database analysis module is configured to perform statistical analysis on historical test data from each stress test in the historical database to determine the relationship between test environment, test case type, failure mode and test duration. The strategy generation module is configured to generate personalized test strategies for each of the multiple test environments of the heterogeneous AI system under test based on the correlation between the test environment, test case type, fault mode and test duration. as well as The test execution module is configured to perform stress tests on the multiple test environments in parallel based on a personalized test strategy generated for each test environment. The test case types include multiple variant types of a first test for testing environmental stability and a second test for testing environmental defects. The strategy generation module is further configured as follows: Based on the correlation between the test environment, test case type, failure mode, and test duration, a dynamic test plan is determined for performing the first test in each of the multiple test environments; and Based on the correlation between the test environment, test case type, failure mode, and test duration, a weighted random selection is used to determine the applicable variant type for the second test in each of the multiple test environments. The strategy generation module is further configured as follows: Based on the correlation between the test environment, test case type, fault mode and test duration, the fault detection rate, detection time and core fault detection items of each variant type of the second test in each test environment in the historical test data are determined. Based on the fault detection rate, detection time, and core fault detection items of multiple variant types of the second test in each test environment, the preferred variant type of the second test in each test environment is determined; and Based on the preferred variant type of the second test in each test environment, determine the applicable variant type for performing the second test in each of the plurality of test environments, and The strategy generation module is further configured as follows: Based on the core fault detection items of each of the multiple variant types of the second test, determine the overlap rate between the core fault detection items of the multiple variant types of the second test, and determine the adaptability of the multiple variant types of the second test to the test environment. From the variant types whose overlap rate is higher than a preset overlap rate threshold, a suitable variant type is selected as a candidate variant type based on the fault detection rate, detection time, or adaptability of each variant type. Based on the fault detection rate, detection time, preset fault detection rate weight, and preset detection time weight of each of the multiple variant types of the second test, the priority score of each of the multiple variant types of the second test is determined. Based on the priority scores of each candidate variant type, the probability interval corresponding to the fault detection probability of each candidate variant type is determined; and At least one random number is generated, and based on the probability interval corresponding to the generated random number, a candidate variant type is selected from the candidate variant types as the preferred variant type for the second test in this test environment.

11. The system according to claim 10, characterized in that... Also includes: The feedback update module is configured to collect test data from stress testing the multiple test environments under test, and to update the historical database based on the collected data.

12. A non-transitory computer-readable medium, characterized in that... The system stores instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1-9.

13. A computer program product, characterized in that... Includes instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1-9.

Images