Graphic processor multidimensional test method of artificial intelligence server

Abstract

The invention discloses a multi-dimensional test method for a graphics processor of an artificial intelligence server, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining an interconnection topological structure and a multi-dimensional test demand of a graphics processor cluster, and then preliminarily generating a test case based on a language model according to the interconnection topological structure and the multi-dimensional test demand, and optimizing the fault input mode of the initial test case through reinforcement learning to obtain a final test case, and testing. According to the method, the log is learned through a large model, the distribution of the load of the graphics processor in time and space is fully learned, the high-pertinence test case is automatically generated in combination with an interconnection topological structure, the fault is automatically injected in combination with the reinforcement learning technology, manual participation of test case writing is reduced, and the test efficiency is improved. The problem that in the prior art, the test case writing efficiency is low, and the rapid iteration requirement of the artificial intelligence server cannot be met is solved, the generation efficiency of the test case is effectively improved, and the iteration speed of the artificial intelligence server is increased.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence server technology, and in particular to a multi-dimensional testing method for the graphics processor of an artificial intelligence server and an artificial intelligence server. Background Technology

[0002] Currently, the testing of graphics processors (GPUs) for AI servers employs testing schemes based on traditional graphics rendering workload expansion, along with some preliminary attempts for multi-GPU systems. For example, improved graphics rendering testing tools are used to simulate AI workloads by adding parallel computing tasks. However, these methods are severely inadequate in simulating core AI computational modes such as tensor parallelism and mixed-precision computing. In multi-GPU collaborative testing, some research attempts to test multi-GPU systems using simple task allocation algorithms, but these cannot effectively address the complex communication bottlenecks and load balancing issues between GPUs during distributed training. Most importantly, all of the above testing methods require manual writing of test cases. For instance, testing a 64-card cluster requires hundreds of hours of manual writing, and accuracy is difficult to guarantee, making it unsuitable for the rapid iteration pace of AI servers. Summary of the Invention

[0003] This application provides a multi-dimensional testing method for the graphics processor of an artificial intelligence server and an artificial intelligence server, so as to at least solve the problem of low efficiency in test case writing in related technologies, which cannot meet the rapid iteration needs of artificial intelligence servers.

[0004] This application provides a multi-dimensional testing method for the graphics processing unit (GPU) of an artificial intelligence server, comprising: acquiring the interconnection topology and testing requirements of a GPU cluster, wherein the testing requirements include multiple sub-requirements for testing dimensions; generating test case groups corresponding to different sub-requirements based on the interconnection topology and the testing requirements using a pre-trained language model, thereby obtaining multiple first test case groups, wherein the language model is trained using historical task logs of the artificial intelligence server; determining the fault injection nodes and fault injection types of all test cases in each of the first test case groups using a reinforcement learning algorithm, and correcting the corresponding test cases based on the fault injection nodes and fault injection types, thereby obtaining multiple second test case groups; sending the test cases in each of the second test case groups to the corresponding GPUs in the GPU cluster and running them, monitoring the running status of each GPU, and obtaining test results.

[0005] This application also provides an artificial intelligence server, including: a memory for storing a computer program; and a processor for implementing the steps of the graphics processor multidimensional testing method of any of the above-mentioned artificial intelligence servers when executing the computer program.

[0006] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the graphics processor multidimensional testing method of any of the above-described artificial intelligence servers.

[0007] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the graphics processor multidimensional testing method for any of the above-described artificial intelligence servers.

[0008] This application, through the combination of deep learning and reinforcement learning, automatically generates highly targeted test cases covering preset multi-dimensional scenarios. Specifically, it uses a large deep learning model to deeply understand and learn the historical work logs of the AI ​​server, extracting the load change trends of the graphics processor under different scenarios. Based on this, it automatically generates test cases for testing needs. Furthermore, it uses reinforcement learning technology to inject faults at the optimal time and space to simulate various unexpected situations in actual operation scenarios, achieving full automation of graphics processor testing. Therefore, it can solve the technical problem of low test case writing efficiency in related technologies, which cannot meet the rapid iteration needs of AI servers, and achieve the technical effect of improving the efficiency of test case generation and thus improving the iteration speed of AI servers. Attached Figure Description

[0009] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A flowchart illustrating a multi-dimensional testing method for a graphics processor of an artificial intelligence server provided in an embodiment of this application;

[0011] Figure 2 A schematic diagram of a four-dimensional intelligent testing architecture provided in an embodiment of this application;

[0012] Figure 3 A flowchart illustrating an intelligent test case generation and fault injection method provided in this application embodiment;

[0013] Figure 4 A schematic diagram of a multi-graphics processor collaborative test scheduling system provided in this application embodiment;

[0014] Figure 5 A schematic diagram of a real-time monitoring and intelligent analysis platform provided in an embodiment of this application;

[0015] Figure 6 This is a schematic diagram of a hardware acceleration system provided in an embodiment of this application. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0017] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0018] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] The embodiments of this application provide a multi-dimensional testing method for the graphics processor of an artificial intelligence server. The method is described in detail below, in conjunction with the execution flow of the multi-dimensional testing method for the graphics processor of an artificial intelligence server (the technical terms involved must be explained).

[0020] In one embodiment of this application, such as Figure 1 As shown, the multi-dimensional testing method for the graphics processor of the aforementioned artificial intelligence server includes:

[0021] Step S101: Obtain the interconnection topology and test requirements of the graphics processor cluster, wherein the test requirements include test sub-requirements of multiple test dimensions;

[0022] Understandably, the above steps are based on network theory and computer system architecture principles, aiming to understand the connection methods and communication paths within a graphics processing unit (GPU) cluster, extract the transmission efficiency and communication performance of data streams through the interconnection topology, so as to ensure the collaborative performance of multiple GPUs and thus improve the accuracy of testing. At the same time, test requirements are the basis for determining the generation of subsequent test cases.

[0023] In practice, using self-developed or existing tools, the interconnection topology of the graphics processor cluster is automatically identified, including but not limited to three-dimensional hybrid topology, ring topology and tree topology. Then, based on the working characteristics of the artificial intelligence server, multiple test dimensions are determined, and sub-requirements under each dimension are determined, such as functional testing, performance testing, security testing and reliability testing.

[0024] The test architecture of this application is as follows: Figure 2 As shown, this application sets up performance testing, functional testing, security testing, and reliability testing dimensions.

[0025] Among them, the functional testing dimension deeply integrates the artificial intelligence model operator library, comprehensively covering a variety of core artificial intelligence computing units, i.e. operators, such as convolution, pooling, and attention mechanisms. By calling various operators in the operator library, test instructions are sent to the graphics processor, and the accuracy of its calculation results is strictly verified to ensure that the graphics processor can accurately and stably execute basic calculations in various AI workloads.

[0026] The performance dimension test dynamically creates dynamic load models to highly simulate the complex mixed load of computation, memory, and communication during the training process of various large models. By flexibly adjusting the tensor shape (covering 1×1×1024 - 8192×8192×64) and computational accuracy (FP32 / FP16 / BF16), the test accurately evaluates the computing performance, data processing speed, and communication efficiency of the graphics processor under different load conditions.

[0027] In terms of reliability testing, stress testing methods such as temperature cycling and voltage fluctuation are used to simulate the operating state of the graphics processor under extreme working environments. Environmental test chambers can be used to precisely control the working environment, and professional power supplies can be used for voltage control. By running under different extreme conditions, errors or crashes are detected to verify the stability of the graphics processor.

[0028] In terms of security testing, a high-precision hardware Trojan detection module is embedded at the graphics processor hardware level. Side-channel analysis and logic locking technologies are used to conduct a comprehensive and in-depth investigation of security risks in the graphics processor hardware.

[0029] By accurately identifying the topology and clarifying the testing requirements, the above operations provide a precise direction and basis for generating subsequent test cases, ensuring the relevance and effectiveness of the testing.

[0030] Step S102: Using a pre-trained language model, test case groups corresponding to different test sub-requirements are generated based on the interconnection topology and test requirements, resulting in multiple first test case groups. The language model is trained using the historical task logs of the artificial intelligence server.

[0031] Understandably, pre-training is a strategy for training deep learning models. Its core lies in using large-scale datasets to initially train the model, enabling it to learn general feature representations. This process is similar to the basic learning stage humans undergo before learning new knowledge, accumulating experience through extensive reading and observation.

[0032] In this application, the language model is used to generate test cases that meet specific sub-requirements based on the input interconnection topology and test requirements, using deep learning and natural language processing techniques. In optional embodiments, the above-mentioned large model can adopt the Transformer model.

[0033] In practice, a language model trained on historical task logs of an AI server is used. The interconnection topology and test requirements are taken as input. Through the model’s reasoning and generation mechanism, the logs are subjected to deep learning and mining. Based on the distribution characteristics of the workload in time and space, a series of targeted test cases are constructed, resulting in the first test case group mentioned above.

[0034] The above operations generate test cases that closely match real-world application scenarios and cover various dimensions, thereby improving the comprehensiveness and relevance of testing.

[0035] Step S103: Use reinforcement learning algorithm to determine the fault injection node and fault injection type of all test cases in each first test case group, and correct the corresponding test cases according to the fault injection node and fault injection type to obtain multiple second test case groups.

[0036] Understandably, reinforcement learning is a goal-oriented machine learning algorithm that learns the best action strategy through multiple interactions with the environment. In the above steps, reinforcement learning algorithms are used to determine the most appropriate time and place for fault injection, as well as the specific fault type, in order to improve the fault tolerance and stability testing capabilities of graphics processing units.

[0037] In practice, reinforcement learning algorithms are used to optimize the fault injection strategy for each test case group. Specifically, based on the graphics processor's running status, historical fault data, and current test requirements, it is dynamically determined which nodes to inject what type of fault, and then the test cases are modified to add fault injection instructions until the injection strategy corresponding to each test case is determined, thus obtaining the aforementioned second test case group.

[0038] The above embodiments make the optimized test cases closer to the actual possible failure scenarios, so as to achieve more accurate reliability testing.

[0039] In one embodiment, the test case generation process is as follows: Figure 3As shown, a large number of real AI task logs are first collected and used as training data to input a large model. Through deep learning and mining of the logs, the large model fully learns the distribution of AI workloads in time and space. Then, the large model generates a high-precision test case set based on the graphics processor architecture and testing requirements. Subsequently, reinforcement learning algorithms are used to intelligently optimize the actual and types of fault injection, which can accurately cover common or potential fault modes such as memory errors and computing unit failures, so as to simulate fault conditions in the real operating environment, obtain fault input strategies, and perform fault injection according to the fault injection strategies.

[0040] Step S104: Send the test cases in each second test case group to the corresponding graphics processor in the graphics processor cluster and run them, monitor the running status of each graphics processor, and obtain the test results.

[0041] Understandably, based on distributed computing and real-time monitoring technologies, the test cases in the second test case group are distributed to the graphics processors in the graphics processor cluster for execution, and the running status of the graphics processors is monitored in real time to capture abnormal behavior and performance bottlenecks in a timely manner.

[0042] In practice, a distributed test scheduling system is used to evenly distribute test cases in the second test case group to the graphics processors in the cluster, ensuring that each graphics processor undertakes the corresponding test tasks. At the same time, the system starts the monitoring module to continuously monitor indicators such as core frequency, memory bandwidth, temperature, voltage, and faults at a high sampling frequency.

[0043] Through the above operations, real-time monitoring ensures the immediate capture and analysis of faults, improving the speed and accuracy of fault location.

[0044] Through the above embodiments, by combining deep learning and reinforcement learning, highly targeted test cases are automatically generated, covering preset multi-dimensional scenarios. Specifically, by deeply understanding and learning the historical work logs of the AI ​​server through a large deep learning model, the load change trend of the graphics processor under different scenarios is extracted. Based on this, test cases for testing needs are automatically generated. Furthermore, by using reinforcement learning technology, faults are injected at the optimal time and space to simulate various emergencies in actual operation scenarios, thus achieving full automation of graphics processor testing. Therefore, it can solve the technical problem of low test case writing efficiency in related technologies, which cannot meet the rapid iteration needs of AI servers, and achieve the technical effect of improving the efficiency of test case generation and thus improving the iteration speed of AI servers.

[0045] To train the aforementioned large model, in one optional implementation, before generating test case sets corresponding to different test sub-requirements based on the architecture data and test requirements using a pre-trained language model, the method further includes:

[0046] Step S201: Extract the interconnect topology of the graphics processor cluster and the architecture data of each graphics processor from the historical task logs.

[0047] Understandably, the above steps are based on data mining and computer network theory, aiming to extract useful information from the historical operation logs of artificial intelligence servers, especially the interconnection topology of the graphics processor cluster and the hardware architecture details of individual graphics processors, to assist in the generation of test cases.

[0048] In practice, log parsing tools are used to read and parse historical task logs to extract the interconnection information of the graphics processing unit (GPU) cluster, including the topology formed by the GPUs through various connection methods. Secondly, the hardware specifications of individual GPUs are analyzed, including GPU model, number of cores, memory size, and memory type. This provides the necessary input parameters for subsequent model training, ensuring that the model can generate test cases based on real hardware configuration information.

[0049] Through the above operations, the accurately extracted hardware architecture data and interconnection topology ensure the relevance of the generated test cases.

[0050] Step S202: Extract the computational task parameters, running time, tensor shape, computational accuracy, working environment, and load distribution sequence of each graphics processor when running different computational tasks from the historical task log according to each test sub-requirement;

[0051] Understandably, based on data mining and feature engineering, specific operating parameters of the graphics processor when performing various computing tasks are extracted from historical task logs, providing rich input features for model training.

[0052] In practice, data mining algorithms are used to filter historical task log fragments related to each test sub-requirement. From these fragments, data such as the graphics processing unit's runtime computational task parameters (e.g., model type, batch size), runtime, tensor shape, and computational precision, as well as runtime environmental parameters (e.g., temperature, voltage), are extracted. This data will serve as feature inputs for training a language model to learn the graphics processing unit's behavior patterns under different conditions.

[0053] Step S203: Using the interconnection topology, architecture data, and corresponding computational task parameters, runtime, tensor shape, computational accuracy, and working environment as input parameters, and the load distribution sequence as label data, train the language model.

[0054] It is understandable that the text understanding and generation capabilities obtained by the pre-trained language model through unsupervised learning can be used to understand and generate test cases that conform to specific hardware configurations and workloads through retraining.

[0055] All the extracted data are used as input parameters, with the load distribution sequence (the load variation of the graphics processing unit when performing different tasks) serving as the training objective of the model. Deep learning training methods such as backpropagation are used to optimize the model parameters, enabling the model to generate test cases that meet different test sub-requirements based on the input architecture data and topology, i.e., the first test case group.

[0056] Through the above operations and training, the language model can generate highly targeted test cases that take into account the actual working environment and hardware capabilities of the graphics processor, thereby improving the simulation of the test scenario and the reliability of the test results.

[0057] Through the above embodiments, not only can the actual hardware configuration and operating status of the graphics processing unit (GPU) cluster be accurately extracted, but a language model capable of intelligently generating test cases can also be trained based on this information. This big data and deep learning-based testing method ensures high quality and relevance in the subsequent test case generation during the preparation phase. It covers multi-dimensional testing requirements including functionality, performance, reliability, and security, and can generate personalized test cases based on the actual operating environment and hardware characteristics of the GPU, greatly improving the comprehensiveness and efficiency of testing. Furthermore, this method can be continuously optimized as new historical task logs accumulate, constantly improving the quality of test cases and the generalization ability of the model, providing long-term development space and technical support for GPU testing of AI servers.

[0058] In an optional implementation, to generate a test case set, step S102 above includes:

[0059] Step S1021: Input the test sub-requirements and interconnection topology into the language model to obtain multiple predicted load distributions, which correspond one-to-one with the graphics processor.

[0060] Understandably, language models, such as those based on the Transformer architecture, possess powerful text understanding and generation capabilities. By inputting specific structured data, such as test requirements and the interconnect topology of a graphics processing unit (GPU) cluster, the model can predict the load distribution of the GPU during different tests based on its internal statistical learning mechanisms. These predictions, based on historical data patterns and the model's generalization ability, can guide the generation of test cases.

[0061] In practice, the GPU interconnect topology and test requirements (including sub-requirements such as functionality, performance, reliability, and security) extracted from historical task logs are used as input. Through the model's inference mechanism, the expected load distribution of each GPU is predicted when different test sub-requirements are executed. By analyzing a large amount of historical task data, the model learns the operating characteristics of GPUs under different conditions, thereby generating a refined predicted load distribution.

[0062] Through the above operations, the predictive capabilities of the language model were used to simulate the load distribution of the graphics processing unit when performing specific tests in advance, providing a reference for the generation and optimization of subsequent test cases.

[0063] Step S1022: Generate corresponding target test instructions based on each predicted load distribution to obtain a corresponding test instruction group, wherein the target test instruction is an input instruction that makes the load of the graphics processor consistent with the predicted load distribution.

[0064] Understandably, based on the predicted load distribution, target test instructions are generated through algorithmic design. These instructions are designed to simulate the predicted load conditions, ensuring that the graphics processing unit (GPU) can achieve the expected load level when executing tests. This process involves understanding and controlling the performance of the GPU cluster.

[0065] In practice, based on the predicted load distribution obtained from the above steps, specific test instructions are designed. These instructions include parameters such as data shape, data type, and task configuration to ensure that the load on the graphics processor when executing the instructions is consistent with the predicted distribution. The generation of test instructions needs to consider the graphics processor's computing power, video memory capacity, and interconnect bandwidth to generate instructions that both meet testing requirements and are practically feasible.

[0066] Through the above operations, the generated target test instructions can ensure that the load distribution of the graphics processor during test execution is very close to the prediction of the language model, which improves the controllability of the test and the accuracy of the prediction. At the same time, it can also cover the performance range of the graphics processor cluster and ensure the comprehensiveness of the test.

[0067] Step S1023: Combine the test sub-requirements and test instruction groups to obtain the first test case group.

[0068] Understandably, test case design is a crucial step in software testing. It combines test requirements and test instructions to guide test execution in a concrete way.

[0069] In practice, test sub-requirements are combined with the generated target test instruction sets to form specific test case sets. Each test case set contains a complete set of test instructions designed to comprehensively evaluate a specific test sub-requirement. The generation of test cases needs to fully consider the overall performance and architectural characteristics of the graphics processing unit cluster to ensure comprehensive test coverage and the rationality of the test instructions.

[0070] Through the above embodiments, a series of highly targeted test cases were generated by utilizing a pre-trained language model combined with the interconnection topology of the graphics processing unit (GPU) cluster and specific testing requirements. This method not only improves the accuracy and efficiency of testing but also effectively covers the comprehensive testing needs of GPUs in terms of functionality, performance, reliability, and security. Compared to traditional testing methods, this approach significantly reduces the time and effort required for manually designing test cases. Simultaneously, through model prediction and instruction optimization, it ensures the quality of test cases and the reliability of experimental results, providing an innovative solution for GPU testing of AI servers and contributing to accelerating the technological development and iteration of GPUs and AI servers.

[0071] To determine the fault injection strategy, in one optional implementation, step S103 above includes:

[0072] Step S1031: Construct a simulation environment based on the test cases. The simulation environment is used to simulate the operating environment of the graphics processor.

[0073] Understandably, a simulation environment is a virtual system created using computer simulation technology, capable of replicating the working environment and behavioral patterns of a real graphics processing unit (GPU) cluster. By constructing a simulation environment, extensive fault injection testing can be performed without interfering with the operation of the actual system.

[0074] In practical implementation, a self-developed or open-source graphics processor (GPU) simulation framework is used to establish a detailed GPU operation simulation model based on the interconnect topology and operating environment characteristics of the GPU cluster. The model includes key parameters such as GPU operating status, data transfer rate, and memory access mode, as well as communication protocols and load distribution strategies between GPUs. By adjusting the simulation parameters, the operating status of the GPU under different load and environmental conditions can be simulated.

[0075] Through the above operations, the construction of the simulation environment allows for free fault injection testing in a controlled virtual environment, avoiding the risk of damaging the actual system operation due to testing, while also enabling a comprehensive and in-depth exploration of the behavior of the graphics processor under various conditions.

[0076] Step S1032: Construct a state space based on the interconnect topology and the working environment of the graphics processor cluster. The state space is used to describe the variable parameters of the graphics processor cluster and the working environment.

[0077] Step S1033: Construct an action space based on a preset fault mode. The action space includes the fault input type under each node of the state space.

[0078] Understandably, reinforcement learning algorithms maximize the reward function by taking actions in the state space. The state space describes all possible states the system can be in, while the action space defines all possible actions the system can take.

[0079] In practical implementation, the state space includes the interconnection topology of the graphics processing unit (GPU) cluster, the core frequency, memory usage, temperature, voltage, and other parameters of each GPU, as well as the external operating environment (such as temperature fluctuations and voltage fluctuations). The action space defines the specific types and injection nodes for fault injection, such as injecting memory errors, computing unit failures, network packet loss, and other fault modes, as well as on which GPU or communication link the fault is injected.

[0080] Step S1034: Construct a reward function. The value of the reward function is directly proportional to the complexity of the fault mode monitored during the simulation, the hardware requirements of the graphics processor, and the response and recovery speed of the graphics processor cluster. The value of the function is inversely proportional to the frequency of occurrence of the fault mode in actual operation.

[0081] Understandably, the reward function is a core component in reinforcement learning, determining the direction and goal of the algorithm's learning. A reasonable reward function can guide the algorithm to explore and learn more effectively.

[0082] In its implementation, the reward function design considers several factors: the complexity of the failure mode (the more complex the failure mode, the higher the reward), the hardware requirements of the graphics processing unit (the greater the impact of the failure, the higher the reward), the response and recovery speed of the graphics processing unit cluster (failure modes with fast response and strong recovery capabilities receive higher rewards), and the frequency of the failure mode in actual operation (the lower the frequency, the higher the reward). This reward function design incentivizes the algorithm to prioritize learning failure modes that are unlikely to occur in actual operation but have a significant impact.

[0083] Through the above operations, the design of the reward function encourages the algorithm to prioritize learning and exploring fault modes that have a significant impact on the system but are less common, thereby improving the comprehensiveness and accuracy of fault detection.

[0084] Step S1035: Run the reinforcement learning algorithm in the simulation environment according to the state space, action space and reward function until the function value is greater than the first threshold, and extract the fault input node and fault injection type from the current fault injection strategy.

[0085] Understandably, reinforcement learning, through iterative learning, gradually converges to the optimal policy, that is, the action taken in a given state can maximize long-term reward.

[0086] In practical implementation, within a simulation environment, reinforcement learning algorithms learn, through continuous trial and error (exploration) and policy optimization (expitation), how to inject which type of fault in a given state to maximize the reward function. The learning process continues until the reward function value exceeds a pre-set first threshold, at which point the algorithm can be considered to have found a set of efficient fault injection strategies.

[0087] Through the above embodiments, the optimal node and type for fault injection in a graphics processing unit (GPU) cluster can be intelligently determined. This method not only improves the comprehensiveness and accuracy of fault detection but also reduces the impact of testing on the actual system, ensuring efficient and safe testing. Compared to traditional manual fault injection methods, this method significantly reduces manpower and time costs. Furthermore, by combining simulation environments and reinforcement learning algorithms, it achieves autonomous optimization of fault injection strategies, opening up new pathways for GPU testing technology.

[0088] To balance the load on each graphics processor, in one optional implementation, step S104 includes:

[0089] Step S10401: Traverse each second test case group. When there is a communication dependency between each graphics processor during the test case execution, use the topology discovery algorithm to identify the interconnection topology between graphics processors with communication dependencies in the graphics processor cluster.

[0090] Understandably, in network and computer system architectures, topology discovery algorithms are used to identify connection patterns and attributes between nodes. In graphics processing unit (GPU) clusters, such algorithms can help identify communication paths and data dependencies between GPUs, thereby understanding the communication structure within the cluster.

[0091] In practice, each test case in the second test case group contains specific functional, performance, reliability, or security test instructions. When the test cases are executed, communication dependencies may exist within the graphics processing unit (GPU) cluster; that is, some GPUs need to receive data or signals from other GPUs to complete their test tasks. Self-developed or open-source topology discovery algorithms, such as those based on network scanning, handshake protocols, or ping commands, are used to identify the interconnection topology between GPUs in the GPU cluster that have communication dependencies.

[0092] Step S10402: Generate the corresponding resource distribution map based on the interconnection topology;

[0093] As is understandable, a resource distribution map is a graphical representation of the distribution and communication paths of resources (such as graphics processors, memory, and bandwidth) within a graphics processing unit (GPU) cluster, based on its topology. It helps in understanding and optimizing resource allocation and usage.

[0094] In practical implementation, a resource distribution map is generated using the aforementioned interconnection topology. Each node in the map represents a graphics processor (GPU), and edges represent communication paths between GPUs. The weights of the edges reflect communication bandwidth or latency information. The resource distribution map visually displays the distribution of resources and communication dependencies within the GPU cluster, providing data support for subsequent test case allocation.

[0095] Step S10403: Using a reinforcement learning algorithm, with the goal of minimizing communication latency, a target allocation strategy is generated based on the resource distribution map and communication dependencies. The target allocation strategy includes a representation of the correspondence between test cases and graphics processors.

[0096] As is understandable, reinforcement learning is a machine learning method that enables agents to learn optimal behavioral policies through interaction with the environment in a dynamic context. Here, the goal is to find an allocation strategy that minimizes communication latency during test case execution by the graphics processing unit (GPU) cluster, thereby improving testing efficiency.

[0097] In practical implementation, the resource distribution map and the communication dependencies of test cases are used as input to construct the state space of the reinforcement learning algorithm. The action space is defined as the allocation relationship between test cases and graphics processors, i.e., determining which graphics processor executes which test case. The reward function is designed to be inversely proportional to the communication latency; that is, the lower the latency, the higher the reward. The algorithm iterates multiple times, continuously trying different allocation strategies and optimizing the strategy based on the feedback from the reward function until it finds the optimal strategy that minimizes the communication latency.

[0098] Through the above operations, the optimal allocation strategy learned by the reinforcement learning algorithm can significantly reduce the communication latency of the graphics processing unit (GPU) cluster when executing test cases, improving testing efficiency, resource utilization balance, and accuracy. This solves the problem of wasted computing power in existing multi-GPU collaborative testing technologies, which struggle to handle complex communication and load balancing between GPUs.

[0099] Step S10404: Send each test case to the corresponding graphics processor for execution according to the target allocation strategy.

[0100] Understandably, based on the target allocation strategy learned through reinforcement learning, test cases are assigned to specific graphics processors for execution, and the running status of the graphics processor cluster is monitored to evaluate test results and failure conditions.

[0101] In practice, based on the optimal allocation strategy described above, each test case in the second test case group is sent to the corresponding graphics processor for execution. Simultaneously, a real-time monitoring system is activated to monitor key indicators such as the graphics processor's core frequency, memory bandwidth, temperature, and voltage at a high sampling rate, as well as the cluster's communication status, ensuring that any abnormal behavior can be detected promptly.

[0102] Through the above embodiments, reinforcement learning algorithms and topology discovery techniques were employed to achieve intelligent test case allocation and execution optimization for graphics processing unit (GPU) clusters. This method not only reduces communication latency during test case execution and improves testing efficiency, but also enables timely fault detection and location through real-time monitoring and intelligent analysis, enhancing testing accuracy and fault response capabilities.

[0103] To balance the load on each graphics processor, in an optional implementation, step S104 further includes:

[0104] Step S10405: Traverse each second test case group and determine the test sub-requirements corresponding to the second test case group if there is no communication dependency between the graphics processors during the test case execution.

[0105] Understandably, test case design typically revolves around specific testing dimensions, including functional testing, performance testing, reliability testing, and security testing. Test cases for different testing dimensions may have varying requirements for the operating environment and resources of the graphics processing unit (GPU) cluster.

[0106] Through the above embodiments, the test cases in each second test case group are traversed, their test objectives are analyzed, and the test sub-requirements corresponding to the test cases are determined, such as whether it is necessary to test specific operators, whether it is necessary to evaluate the performance of the graphics processor under specific loads, whether it is necessary to verify the stability of the graphics processor under extreme conditions, or whether it is necessary to test the security of the graphics processor.

[0107] Step S10406: When the test dimension of the test sub-requirement is security test, determine that test cases of different test dimensions are not allowed to run in parallel.

[0108] Understandably, during test execution, test cases for different test dimensions may require different running modes. Security tests typically need to be run separately to ensure that they do not pose a security risk to other test cases or the system; while test cases for other test dimensions (such as functional, performance, and reliability tests) can be run in parallel to improve testing efficiency.

[0109] In practice, if the testing dimension of a sub-requirement is security testing, the execution mode is determined to be non-parallel execution. This means that all test cases involving security testing must be executed individually, with only one test case running at a time.

[0110] Step S10407: If the test dimension of the test sub-requirement is not security test, determine the running mode so that test cases of different test dimensions can run in parallel;

[0111] In practice, if the testing dimension of a sub-requirement is not security testing, the running mode is set to allow parallel execution. This means that test cases for dimensions other than security testing can be executed simultaneously on the graphics processing unit (GPU) cluster, thereby making full use of GPU resources and shortening the testing cycle.

[0112] By performing the above operations, differentiating between different testing dimensions and determining the corresponding operating modes, the execution of test cases can be effectively managed, ensuring the independence of security testing while improving the efficiency of functional, performance, and reliability testing.

[0113] Step S10408: Send each test case to the corresponding graphics processor for execution according to the test dimension.

[0114] Understandably, by rationally allocating test cases to different graphics processors in the graphics processor cluster based on the test dimensions and operating modes of the test cases, optimal utilization of test resources is ensured.

[0115] In practical implementation, the graphics processing unit (GPU) cluster's scheduling system, based on the aforementioned operating mode—that is, security test cases run independently while test cases for other test dimensions run in parallel—sends test cases to the corresponding GPUs for execution. The scheduling system needs to consider the GPU's current load, availability, and the resource requirements of the test cases to achieve efficient resource allocation and scheduling.

[0116] The above embodiments not only effectively determine the specific testing requirements of each test case, but also enable the formulation of reasonable operating modes based on different testing dimensions, ensuring both testing security and efficiency. The independent operation of security testing ensures that no new security risks are introduced during the testing process, while the parallel operation of test cases in other testing dimensions fully utilizes graphics processor resources, significantly shortening the testing cycle and improving overall testing efficiency. This also demonstrates the advantages of intelligent test case scheduling, which, while ensuring test quality, can address the complexity and diversity of graphics processor cluster testing through dynamic resource allocation and operating mode adjustments, providing an efficient, secure, and intelligent solution for graphics processor testing of artificial intelligence servers.

[0117] In one embodiment, a multi-graphics processor collaborative scheduling system, such as Figure 4 As shown, the topology-aware scheduling module is used to identify the interconnect topology of the graphics processor, run intelligent algorithms based on the characteristics of the topology to dynamically allocate test tasks, thereby optimizing the data transmission path, reducing communication latency, improving the execution efficiency of test tasks, and ensuring the efficiency and smoothness of the testing process.

[0118] The load balancing algorithm module, based on deep reinforcement learning technology, achieves cross-GPU load balancing by monitoring the load of each GPU and adjusting task allocation according to load changes.

[0119] The parallel testing strategy module supports the parallel execution of functional testing, performance testing, and reliability testing. By planning and scheduling test tasks, it shortens the testing cycle and improves testing efficiency.

[0120] To obtain the above test results, in an optional implementation, step S104 further includes:

[0121] Step S10409: Receive monitoring data from various sensors deployed on each graphics processor via a high-speed peripheral interface sampler. The monitoring data includes at least the core frequency, memory bandwidth, and operating temperature of the graphics processor.

[0122] Understandably, modern graphics processing units (GPUs) incorporate various sensors to monitor key metrics such as core frequency, memory bandwidth, and operating temperature. High-speed peripheral interface samplers (such as sampling modules connected via the PCIe bus) can acquire data at extremely high frequencies (e.g., 1MHz) to ensure detailed real-time information about the GPU's operation.

[0123] In practice, sensors are deployed on each graphics processor (GPU) in the GPU cluster to monitor its operational status. These sensors transmit real-time data to a central monitoring system via high-speed PCIe samplers, including but not limited to metrics such as core frequency, memory bandwidth, and operating temperature. The sampling frequency is chosen to be sufficiently high to capture transient changes during GPU operation, ensuring the timeliness and completeness of the data.

[0124] Step S10410: Wavelet transform is used to process the monitoring data to extract abnormal signals from the monitoring data and obtain the first abnormal signal;

[0125] Understandably, wavelet transform is a signal processing technique particularly well-suited for analyzing instantaneous changes in non-stationary signals. It decomposes a signal in the time domain into a series of wavelet coefficients, thereby identifying outliers in the signal.

[0126] In practice, the graphics processor monitoring data collected in the above steps is input into a wavelet transform algorithm to decompose the data layer by layer and extract detailed information at each time scale. By comparing the wavelet coefficients with a normal baseline, abnormal signals exceeding a threshold are identified. These abnormal signals refer to those that suddenly change or deviate from the normal range, such as sudden drops or increases in core frequency, instantaneous fluctuations in memory bandwidth, and abnormal increases in operating temperature.

[0127] Through the above operations, wavelet transform technology can effectively identify transient anomalies in monitoring data, improve the accuracy and timeliness of anomaly signal detection, and play an important role in rapid response and prevention of graphics processor failures.

[0128] Step S10411: Input the monitoring data into the Long Short-Term Memory network for processing in order to predict the changing trend of the monitoring data;

[0129] Understandably, Long Short-Term Memory (LSTM) networks are a variant of Recurrent Neural Networks (RNNs) that are particularly well-suited for processing and predicting time-series data. They can remember long-term dependencies, making them highly effective for predicting trend changes in monitored data.

[0130] In practice, the monitoring data from the above steps is input into a pre-trained LSTM network to predict trends in the graphics processor's operating status. By learning from historical data, the network can predict the trends of various graphics processor indicators over a future period, such as whether the core frequency will continue to rise, whether the memory bandwidth will tend to saturate, and whether the operating temperature will rise under specific loads.

[0131] Step S10412: If the difference between the trend and the monitoring data is greater than the second threshold, the monitoring data is determined as the second abnormal signal.

[0132] Understandably, by comparing monitoring data with predicted trends, abnormal signals that do not conform to expected behavior can be identified, which often indicate impending failures or performance problems.

[0133] In practice, the monitoring data change trend predicted by the LSTM network is compared with the actual collected monitoring data. If the difference between the actual data change and the prediction result exceeds a preset second threshold, the monitoring data is marked as a second abnormal signal. This threshold can be adjusted according to the type of graphics processor and testing requirements to ensure accurate detection of anomalies.

[0134] By comparing the predicted and actual monitoring data through the above steps, situations that do not conform to expected behavior can be identified in a timely manner, which is crucial for fault early warning and performance optimization.

[0135] Step S10413: Generate test results based on the first abnormal signal and the second abnormal signal.

[0136] Understandably, by combining the first anomalous signal identified by wavelet transform with the second anomalous signal generated by comparing LSTM prediction with monitoring data, the operating status and performance of the graphics processor can be comprehensively evaluated.

[0137] In practice, the first and second abnormal signals identified will be summarized and combined with the execution status of test cases and other monitoring data to generate a detailed test report. The report should include changes in various indicators of the graphics processor during the test, descriptions of abnormal signals, possible cause analysis, and suggested debugging and optimization steps.

[0138] The above embodiments demonstrate a graphics processing unit (GPU) operation status monitoring system that integrates real-time high-speed data acquisition, wavelet transform anomaly signal identification, LSTM trend prediction, and comprehensive anomaly signal analysis. This system can not only monitor various key indicators of the GPU in real time, but also accurately identify anomaly signals, predict data trends, evaluate system performance, and provide early warnings of faults.

[0139] In one embodiment, the intelligent monitoring and analysis platform, such as Figure 5 As shown, a high-performance sampler performs real-time, high-speed sampling at a high frequency to collect data such as the graphics processor core frequency, memory bandwidth, and temperature sensor readings. Then, wavelet transform and LSTM neural network technologies are integrated. Wavelet transform efficiently processes high-frequency transient signals, quickly capturing instantaneous anomalies during graphics processor operation. The LSTM neural network learns the normal operating mode of the graphics processor to perform in-depth analysis and prediction of the collected data, achieving highly accurate anomaly localization. Finally, the monitoring results are displayed in real-time through a 3D visualization interface.

[0140] To obtain the aforementioned testing requirements, in one optional implementation, step S101 includes:

[0141] Step S1011: Integrate the operators called by different artificial intelligence models to obtain a model operator library, analyze the functional characteristics and input / output standards of each operator in the model operator library, and generate the first test sub-requirement based on the functional characteristics and input / output standards.

[0142] Understandably, the operation of artificial intelligence models relies on a series of basic mathematical operations or "operators," such as convolution, matrix multiplication, and activation functions. Understanding the characteristics of the operators used in the model (such as functionality, input / output formats, etc.) is crucial for ensuring the correctness and performance optimization of the graphics processing unit cluster.

[0143] In practice, operators used by various AI models are integrated to build a comprehensive model operator library, which includes all operators involved in common CNN (Convolutional Neural Network), RNN, Transformer models, etc. Then, the functional characteristics and input / output standards of each operator in the library are analyzed, and based on this information, the first test sub-requirements are generated, that is, to test the functionality and accuracy of each operator.

[0144] Step S1012: Create a dynamic load model to simulate the operation of different artificial intelligence models, determine the monitoring parameters of the dynamic load model, and generate a second test sub-requirement based on the dynamic load model and the monitoring parameters.

[0145] Understandably, dynamic load models can test the performance and resource management capabilities of graphics processing unit clusters by simulating the computing power, memory, and communication loads of artificial intelligence models during training or inference.

[0146] In practical implementation, a dynamic load model is constructed to simulate the operation of complex models, including changes in tensor shape, data type, and computational intensity. Simultaneously, monitoring parameters for model operation are determined, such as GPU core frequency, memory bandwidth, and PCIe communication rate, thereby generating a second test sub-requirement. The goal is to evaluate the performance of the GPU cluster under different load conditions.

[0147] Step S1013: Use stress testing methods to simulate the extreme operating environment of the graphics processor, and generate a third test sub-requirement based on the extreme operating environment;

[0148] Understandably, stress testing is a testing method that assesses the stability and durability of a system by applying extreme conditions. In graphics processor testing, extreme operating environments can be simulated by changing environmental parameters (such as temperature and voltage).

[0149] In practical implementation, stress testing methods are employed, such as simulating cyclic temperature changes from -20°C to 85°C using an environmental test chamber, or allowing voltage fluctuations within ±15% using an adjustable power supply, thereby simulating the extreme operating environment of the graphics processor. Based on these environmental conditions, a third test sub-requirement is generated—testing the stable operation and fault recovery capabilities of the graphics processor under extreme conditions.

[0150] Step S1014: Generate attack scenarios during the operation of different artificial intelligence models, and generate the fourth test sub-requirement based on the attack scenarios;

[0151] Understandably, security testing focuses on a system's ability to protect itself against malicious attacks or security threats. For artificial intelligence systems, this includes aspects such as data encryption, model protection, and hardware security.

[0152] In practical implementation, based on the hardware characteristics of the graphics processing unit (GPU), a fourth test sub-requirement was designed and generated—simulating different types of malicious attack scenarios, such as data tampering, side-channel attacks, and DoS attacks. These tests were used to evaluate the GPU cluster's response capabilities and recovery mechanisms when subjected to attacks.

[0153] Step S1015: Integrate the first test sub-requirement, the second test sub-requirement, the third test sub-requirement, and the fourth test sub-requirement into a test requirement.

[0154] In practice, organize and integrate the first to fourth test sub-requirements to ensure that no critical test points are omitted. The test requirements document should detail the objectives, test methods, expected results, and required test environment and tools for each test sub-requirement.

[0155] Through the above embodiments, a multi-dimensional graphics processor cluster testing strategy was constructed, aiming to comprehensively evaluate the performance of graphics processors in terms of functionality, performance, reliability, and security. This method not only covers the basic computing power of graphics processors but also evaluates their responsiveness under complex loads, extreme environments, and malicious attacks, providing a comprehensive perspective for the optimization and maintenance of graphics processor clusters.

[0156] To balance the load across the graphics processors, in one optional implementation, after sending the test cases from each second test case group to the corresponding graphics processor in the graphics processor cluster and running them, the method further includes:

[0157] Step S301: Monitor the current load status of each graphics processor and adjust the test cases running on each graphics processor according to the current load status until the utilization difference between each graphics processor is less than the third threshold.

[0158] Understandably, the load status of a graphics processing unit (GPU) directly impacts its performance and resource utilization efficiency. Monitoring the GPU load status allows for real-time understanding of GPU usage, which is crucial for balancing resource allocation across GPUs in the cluster. Dynamically adjusting test case execution based on the real-time GPU load status ensures balanced GPU utilization within the cluster, preventing some GPUs from being overloaded while others remain idle.

[0159] In practice, a load monitoring module is deployed within the graphics processing unit (GPU) cluster to monitor key performance indicators (KPIs) of each GPU in real time, such as GPU utilization, memory usage, and computation task queue length. This data helps us understand the GPU's operational status, including idle time, light load operation, and full load operation. After monitoring the GPU load status in real time, the execution plan of test cases is dynamically adjusted based on the actual utilization of each GPU using a load balancing algorithm. This includes test case scheduling, priority adjustment, and task migration to minimize the utilization difference between GPUs and keep it below a predetermined third threshold (e.g., 5%).

[0160] Through the above embodiments, dynamically adjusting the execution of test cases results in a more even distribution of resources within the graphics processing unit (GPU) cluster. This avoids the problem of some GPUs being overloaded while others remain idle, improving overall computational efficiency and task processing speed. Balanced load distribution mitigates the risk of GPU overload and reduces performance fluctuations and failure probabilities caused by localized overheating or resource contention. Automated load monitoring and test case adjustment reduce manual intervention, improving the automation level and continuity of testing. Balanced load conditions help improve the accuracy of test results, avoiding test biases caused by resource imbalances and ensuring that tests accurately reflect the performance and stability of the GPUs. By improving resource utilization efficiency and reducing test interruptions, the overall testing cycle is shortened, accelerating the development and maintenance process of AI servers and meeting the needs of rapid iteration.

[0161] In one embodiment, the hardware acceleration testing module of this application is as follows: Figure 6As shown, a hardware acceleration card based on a high-performance field-programmable gate array (FPGA) simulates the massive data input scenario of an artificial intelligence server during actual operation, thereby accelerating the generation of test instances in this application. The real-time data verification unit, equipped with a hardware-accelerated hash verification module, rapidly calculates the hash value of the graphics processor output using a FPGA-accelerated hash algorithm when monitoring test results, and compares it with the expected result. The fault injection control unit utilizes the programmable characteristics of the FPGA, employing nanosecond-level precision fault input control to accurately simulate various hardware faults of the graphics processor.

[0162] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0163] Embodiments of this application also provide a multi-dimensional testing device for the graphics processor of an artificial intelligence server, comprising:

[0164] The first acquisition unit acquires the interconnection topology and testing requirements of the graphics processor cluster, wherein the testing requirements include sub-requirements for multiple testing dimensions.

[0165] The first generation unit is used to generate test case groups corresponding to different test sub-requirements based on the interconnection topology and test requirements using a pre-trained language model, resulting in multiple first test case groups. The language model is trained using the historical task logs of the artificial intelligence server.

[0166] The first processing unit is used to use a reinforcement learning algorithm to determine the fault injection nodes and fault injection types of all test cases in each first test case group, and to correct the corresponding test cases according to the fault injection nodes and fault injection types to obtain multiple second test case groups.

[0167] The second processing unit is used to send the test cases in each second test case group to the corresponding graphics processor in the graphics processor cluster and run them, monitor the running status of each graphics processor, and obtain the test results.

[0168] Through the above embodiments, by combining deep learning and reinforcement learning, highly targeted test cases are automatically generated, covering preset multi-dimensional scenarios. Specifically, by deeply understanding and learning the historical work logs of the AI ​​server through a large deep learning model, the load change trend of the graphics processor under different scenarios is extracted. Based on this, test cases for testing needs are automatically generated. Furthermore, by using reinforcement learning technology, faults are injected at the optimal time and space to simulate various emergencies in actual operation scenarios, thus achieving full automation of graphics processor testing. Therefore, it can solve the technical problem of low test case writing efficiency in related technologies, which cannot meet the rapid iteration needs of AI servers, and achieve the technical effect of improving the efficiency of test case generation and thus improving the iteration speed of AI servers.

[0169] To train the aforementioned large model, in one optional implementation, the apparatus further includes:

[0170] The second acquisition unit is used to extract the interconnection topology of the graphics processor cluster and the architecture data of each graphics processor from the historical task log before generating test case groups corresponding to different test sub-requirements based on architecture data and test requirements using a pre-trained language model.

[0171] The third acquisition unit is used to extract the computational task parameters, running time, tensor shape, computational accuracy, working environment and load distribution sequence of each graphics processor when running different computational tasks from the historical task log according to the requirements of each test sub-test.

[0172] The training unit is used to train a language model with the interconnect topology, architecture data, and corresponding computational task parameters, runtime, tensor shape, computational accuracy, and working environment as input parameters, and the load distribution sequence as label data.

[0173] In an optional implementation, to generate a test case set, the first generation unit includes:

[0174] The input module is used to input the test sub-requirements and interconnection topology into the language model to obtain multiple predicted load distributions, which correspond one-to-one with the graphics processor.

[0175] The generation module is used to generate corresponding target test instructions based on each predicted load distribution, and obtain the corresponding test instruction group. The target test instruction is the input instruction that makes the load of the graphics processor consistent with the predicted load distribution.

[0176] The processing module is used to combine test sub-requirements and test instruction groups to obtain the first test case group.

[0177] To determine the fault injection strategy, in one optional implementation, the first processing unit includes:

[0178] The first building module is used to build a simulation environment based on test cases. The simulation environment is used to simulate the running environment of the graphics processor.

[0179] The second building module is used to construct a state space based on the interconnect topology and the working environment of the graphics processor cluster. The state space is used to describe the variable parameters of the graphics processor cluster and the working environment.

[0180] The third construction module is used to construct the action space according to the preset fault mode. The action space includes the fault input type under each node of the state space.

[0181] The fourth module is used to construct the reward function. The value of the reward function is directly proportional to the complexity of the fault mode monitored during the simulation, the hardware requirements of the graphics processor, and the response and recovery speed of the graphics processor cluster. The value of the function is inversely proportional to the frequency of occurrence of the fault mode in actual operation.

[0182] The first acquisition module is used to run a reinforcement learning algorithm in the simulation environment based on the state space, action space and reward function until the function value is greater than the first threshold, and to extract the fault input node and fault injection type from the current fault injection strategy.

[0183] To balance the load on each graphics processor, in one optional implementation, the second processing unit includes:

[0184] The second acquisition module is used to traverse each second test case group and, when there are communication dependencies between the graphics processors during the test case execution, to identify the interconnection topology between graphics processors with communication dependencies in the graphics processor cluster through a topology discovery algorithm.

[0185] The first processing module is used to generate a corresponding resource distribution map based on the interconnection topology;

[0186] The first computing module is used to generate a target allocation strategy based on the resource distribution map and communication dependencies, with the goal of minimizing communication latency, using a reinforcement learning algorithm. The target allocation strategy includes a representation of the correspondence between test cases and graphics processors.

[0187] The first sending module is used to send each test case to the corresponding graphics processor for execution according to the target allocation strategy.

[0188] To balance the load on each graphics processor, in one optional implementation, the second processing unit further includes:

[0189] The third acquisition module is used to traverse each second test case group and determine the test sub-requirements corresponding to the second test case group when there is no communication dependency between the graphics processors during the test case execution.

[0190] The first determining module is used to determine, when the testing dimension of the test sub-requirement is security testing, that test cases of different testing dimensions are not allowed to run in parallel.

[0191] The second determination module is used to determine whether test cases of different test dimensions can run in parallel when the test dimension of the test sub-requirement is not security test.

[0192] The second sending module is used to send each test case to the corresponding graphics processor for execution according to the test dimension.

[0193] To obtain the above test results, in one optional implementation, the second processing unit further includes:

[0194] The fourth acquisition module is used to receive monitoring data from various sensors deployed on each graphics processor through a high-speed peripheral interface sampler. The monitoring data includes at least the core frequency, memory bandwidth and operating temperature of the graphics processor.

[0195] The second processing module is used to process the monitoring data using wavelet transform to extract abnormal signals from the monitoring data and obtain the first abnormal signal.

[0196] The second calculation module is used to input the monitoring data into the long short-term memory network for processing in order to predict the changing trend of the monitoring data;

[0197] The third determination module is used to determine the monitoring data as a second abnormal signal when the difference between the changing trend and the monitoring data is greater than the second threshold.

[0198] The third processing module is used to generate test results based on the first and second abnormal signals.

[0199] To obtain the aforementioned testing requirements, in one optional implementation, the first obtaining unit includes:

[0200] The fifth acquisition module is used to integrate the operators called by different artificial intelligence models to obtain a model operator library, analyze the functional characteristics and input / output standards of each operator in the model operator library, and generate the first test sub-requirements based on the functional characteristics and input / output standards.

[0201] The sixth acquisition module is used to create a dynamic load model for simulating the operation of different artificial intelligence models, determine the monitoring parameters of the dynamic load model, and generate the second test sub-requirement based on the dynamic load model and the monitoring parameters.

[0202] The seventh acquisition module is used to simulate the extreme operating environment of the graphics processor using stress testing methods, and generate the third test sub-requirement based on the extreme operating environment;

[0203] The eighth acquisition module is used to generate attack scenarios during the operation of different artificial intelligence models, and generate the fourth test sub-requirement based on the attack scenarios;

[0204] The fourth processing module is used to integrate the first test sub-requirement, the second test sub-requirement, the third test sub-requirement, and the fourth test sub-requirement into a test requirement.

[0205] To balance the load on each graphics processor, in one optional implementation, the above-mentioned apparatus further includes:

[0206] The third processing unit is used to monitor the current load status of each graphics processor after sending the test cases in each second test case group to the corresponding graphics processor in the graphics processor cluster and running them, and to adjust the test cases running on each graphics processor according to the current load status until the utilization difference between each graphics processor is less than the third threshold.

[0207] For a description of the features in the embodiment corresponding to the multi-dimensional testing device for the graphics processor of the artificial intelligence server, please refer to the relevant description of the embodiment corresponding to the multi-dimensional testing method for the graphics processor of the artificial intelligence server, which will not be repeated here.

[0208] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above embodiments of the graphics processor multidimensional testing method for an artificial intelligence server.

[0209] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the multidimensional testing method for the graphics processor of an artificial intelligence server when it is run.

[0210] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0211] The embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the multidimensional testing method for a graphics processor of an artificial intelligence server.

[0212] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above embodiments of the multidimensional testing method for a graphics processor of an artificial intelligence server.

[0213] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0214] The above provides a detailed description of a multi-dimensional testing method for the graphics processor of an artificial intelligence server provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A multi-dimensional testing method for the graphics processing unit (GPU) of an artificial intelligence server, characterized in that, include: Obtain the interconnect topology and testing requirements of the graphics processing unit cluster, including sub-requirements for multiple testing dimensions; A pre-trained language model is used to generate test case groups corresponding to different test sub-requirements based on the interconnection topology and the test requirements, resulting in multiple first test case groups. The language model is trained using historical task logs of an artificial intelligence server. A reinforcement learning algorithm is used to determine the fault injection nodes and fault injection types of all test cases in each first test case group, and the corresponding test cases are corrected according to the fault injection nodes and fault injection types to obtain multiple second test case groups. The test cases in each of the second test case groups are sent to the corresponding graphics processors in the graphics processor cluster and run. The running status of each graphics processor is monitored to obtain the test results.

2. The multidimensional testing method for graphics processors according to claim 1, characterized in that, Before generating test case sets corresponding to different test sub-requirements based on the interconnection topology and the test requirements using a pre-trained language model, the method further includes: Extract the interconnect topology of the graphics processor cluster and the architecture data of each graphics processor from the historical task logs; According to the test sub-requirements, extract the computational task parameters, running time, tensor shape, computational accuracy, working environment, and load distribution sequence of each graphics processor running different computational tasks from the historical task log; use the interconnect topology, the architecture data, and the corresponding computational task parameters, running time, tensor shape, computational accuracy, and working environment as input parameters, and the load distribution sequence as label data to train the language model.

3. The multi-dimensional testing method for graphics processors according to claim 1, characterized in that, Using a pre-trained language model, test case sets corresponding to different test sub-requirements are generated based on the interconnection topology and the test requirements, resulting in multiple first test case sets, including: The test sub-requirements and the interconnection topology are input into the language model to obtain multiple predicted load distributions, each of which corresponds to a graphics processor. Based on each of the predicted load distributions, corresponding target test instructions are generated to obtain corresponding test instruction groups, wherein the target test instruction is an input instruction that makes the load of the graphics processor consistent with the predicted load distribution; The test sub-requirements and the test instruction group are combined to obtain the first test case group.

4. The multidimensional testing method for graphics processors according to claim 1, characterized in that, Reinforcement learning algorithms are used to determine the fault injection nodes and fault injection types for each test case, including: A simulation environment is constructed based on the test cases, and the simulation environment is used to simulate the operating environment of the graphics processor; A state space is constructed based on the interconnection topology and the operating environment of the graphics processor cluster. The state space is used to describe the variable parameters of the graphics processor cluster and the operating environment. An action space is constructed based on a preset fault mode, and the action space includes the fault input type under each node of the state space; A reward function is constructed, the value of which is directly proportional to the complexity of the fault mode monitored during the simulation, the hardware requirements of the graphics processor, and the response and recovery speed of the graphics processor cluster, and the value of which is inversely proportional to the frequency of occurrence of the fault mode in actual operation. In the simulation environment, a reinforcement learning algorithm is run based on the state space, the action space, and the reward function until the function value is greater than a first threshold. Then, the fault input node and the fault injection type are extracted from the current fault injection strategy.

5. The multi-dimensional testing method for graphics processors according to claim 1, characterized in that, Sending the test cases from each of the second test case groups to the corresponding graphics processors in the graphics processor cluster and running them includes: Traverse each of the second test case groups, and when there is a communication dependency between the graphics processors in the test case cluster during runtime, use a topology discovery algorithm to identify the interconnection topology between the graphics processors in the cluster that have the communication dependency. Generate a corresponding resource distribution map based on the interconnection topology; A reinforcement learning algorithm is used to minimize communication latency as the optimization objective. Based on the resource distribution map and the communication dependencies, a target allocation strategy is generated. The target allocation strategy includes a representation of the correspondence between test cases and the graphics processor. Each test case is sent to the corresponding graphics processor for execution according to the target allocation strategy.

6. The multidimensional testing method for graphics processors according to claim 1, characterized in that, Sending the test cases from each of the second test case groups to the corresponding graphics processors in the graphics processor cluster and running them further includes: Traverse each of the second test case groups, and determine the test sub-requirements corresponding to the second test case group if there is no communication dependency between the graphics processors during the test case execution. When the test dimension of the test sub-requirement is security testing, test cases with different test dimensions are not allowed to run in parallel. If the test dimension of the test sub-requirement is not a security test, the operation mode is determined to allow test cases of different test dimensions to run in parallel. Each test case is sent to the corresponding graphics processor for execution according to the test dimensions.

7. The multidimensional testing method for graphics processors according to any one of claims 1 to 6, characterized in that, Monitor the operating status of each graphics processor to obtain test results, including: The high-speed peripheral interface sampler receives monitoring data from various sensors deployed on each of the graphics processors, and the monitoring data includes at least the core frequency, memory bandwidth and operating temperature of the graphics processor. Wavelet transform is used to process the monitoring data to extract abnormal signals from the monitoring data, thereby obtaining a first abnormal signal; The monitoring data is input into a long short-term memory network for processing in order to predict the changing trend of the monitoring data; If the difference between the changing trend and the monitoring data is greater than a second threshold, the monitoring data is identified as a second abnormal signal. The test results are generated based on the first abnormal signal and the second abnormal signal.

8. The multidimensional testing method for graphics processors according to any one of claims 1 to 6, characterized in that, The testing requirements for obtaining the graphics processing unit cluster include: The operators called by different artificial intelligence models are integrated to obtain a model operator library. The functional characteristics and input / output standards of each operator in the model operator library are analyzed, and the first test sub-requirement is generated based on the functional characteristics and input / output standards. Create a dynamic load model to simulate the operation of different artificial intelligence models, determine the monitoring parameters of the dynamic load model, and generate a second test sub-requirement based on the dynamic load model and the monitoring parameters; The extreme operating environment of the graphics processor is simulated using a stress testing method, and a third test sub-requirement is generated based on the extreme operating environment. Generate attack scenarios during the operation of different artificial intelligence models, and generate a fourth test sub-requirement based on the attack scenarios; The first test sub-requirement, the second test sub-requirement, the third test sub-requirement, and the fourth test sub-requirement are integrated into the test requirement.

9. The multidimensional testing method for graphics processors according to any one of claims 1 to 6, characterized in that, After sending the test cases from each of the second test case groups to the corresponding graphics processors in the graphics processor cluster and running them, the method further includes: Monitor the current load status of each graphics processor and adjust the test cases running on each graphics processor according to the current load status until the utilization difference between the graphics processors is less than a third threshold.

10. An artificial intelligence server, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the graphics processor multidimensional testing method as described in any one of claims 1 to 9 when executing the computer program.

Images