Artificial intelligence cluster fault processing method and device, equipment and medium

Abstract

The invention discloses an artificial intelligence cluster fault processing method and device, equipment and a medium, and the method comprises the steps: generating an environment snapshot of a current operation environment of an artificial intelligence cluster under the condition that the artificial intelligence cluster breaks down; in a resource pool corresponding to the artificial intelligence cluster, determining a target hardware resource and a target software environment matched with the environment snapshot; according to the target hardware resource and the target software environment, fault reproduction of the artificial intelligence cluster is carried out, an operable reproduction scheme is provided for artificial intelligence cluster faults which are difficult to reproduce, a relatively stable debugging basis is provided for developers, a complete process from fault information collection to field reproduction is formed, and the development efficiency is improved. The method is helpful for improving the success rate of fault reproduction, and plays a positive role in shortening the average fault repair time and improving the operation and maintenance efficiency of the artificial intelligence cluster.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method, apparatus, device, and medium for handling faults in an artificial intelligence cluster. Background Technology

[0002] Against the backdrop of rapid development in artificial intelligence (AI) technology, AI clusters, as the core infrastructure for large-scale model training and inference tasks, are crucial for business continuity due to their stable operation. However, AI clusters typically consist of a large number of heterogeneous hardware resources, and their operating environment involves complex software stacks, dependency libraries, and configuration parameters. In actual operation, they are highly susceptible to failures due to various factors such as hardware malfunctions, software conflicts, resource contention, or configuration anomalies. When a failure occurs, the complexity and transient nature of the cluster's operating environment at the time of the failure often make it impossible to accurately reproduce the failure scenario locally or in a test environment during subsequent troubleshooting. This results in lengthy fault diagnosis, severely impacting the operational efficiency of AI clusters and the speed of business recovery. Summary of the Invention

[0003] In view of the above problems, embodiments of the present invention are proposed to provide an artificial intelligence cluster fault handling method, apparatus, device and medium that overcomes the problem that the fault scenarios cannot be accurately reproduced in local or test environments, resulting in long fault diagnosis time and seriously affecting the operation and maintenance efficiency and business recovery speed of artificial intelligence clusters, or at least partially solves the above problems.

[0004] In a first aspect, embodiments of the present invention provide a method for handling faults in an artificial intelligence cluster, the method comprising: In the event of a failure in the AI ​​cluster, an environment snapshot of the current operating environment of the AI ​​cluster is generated; In the resource pool corresponding to the artificial intelligence cluster, target hardware resources and target software environment that match the environment snapshot are determined; Based on the target hardware resources and the target software environment, reproduce the fault of the artificial intelligence cluster.

[0005] Optionally, the environment snapshot includes hardware configuration snapshot information and software configuration snapshot information. The step of determining the target hardware resources and target software environment matching the environment snapshot in the resource pool corresponding to the artificial intelligence cluster includes: In the resource pool corresponding to the artificial intelligence cluster, target hardware resources that match the hardware configuration snapshot information are determined, and target software environments that match the software configuration snapshot information are determined.

[0006] Optionally, the software configuration snapshot information includes code information and / or software stack information; the hardware configuration snapshot information includes hardware context information and / or configuration parameter information of the artificial intelligence cluster.

[0007] Optionally, the target hardware resource includes a first target hardware resource, and determining the target hardware resource matching the hardware configuration snapshot information in the resource pool corresponding to the artificial intelligence cluster includes: Based on the hardware context information, determine the model of at least one processing unit, the number of processing units, and the network topology of the processing units: In the resource pool corresponding to the artificial intelligence cluster, a first target hardware resource is determined according to a preset resource matching rule, which matches the model of the at least one processing unit, the number of the processing units, and the network topology of the processing units; the preset resource matching rule is used to guide how to perform resource matching.

[0008] Optionally, the target hardware resource includes a second target hardware resource, and the method further includes: If, in the resource pool corresponding to the artificial intelligence cluster, a first target hardware resource matching the model, quantity, and network topology of the at least one processing unit fails to be determined according to a preset resource matching rule, at least one candidate processing unit model, quantity, and network topology are determined based on the code information. In the resource pool corresponding to the artificial intelligence cluster, a second target hardware resource is determined according to a preset resource matching rule, which matches the model of the at least one candidate processing unit, the number of the candidate processing units, and the network topology relationship of the candidate processing units.

[0009] Optionally, determining the model of at least one candidate processing unit, the number of candidate processing units, and the network topology of the candidate processing units based on the code information includes: Obtain historical error information and the model, quantity, and network topology of the historical processing units corresponding to the historical error information; In the historical error messages, a target historical error message that matches the error message is identified; The model of the historical processing unit corresponding to the target historical error information, the number of the historical processing units, and the network topology relationship of the historical processing units are used as at least one candidate processing unit model, candidate processing unit number, and candidate processing unit network topology relationship.

[0010] Optionally, determining the target software environment that matches the software configuration snapshot information includes: Based on the software stack information, determine the target driver version; Based on the code information, determine the target code; The target software environment is determined based on the target driver version and the target code.

[0011] Optionally, the method further includes: after the fault of the artificial intelligence cluster is reproduced, releasing the target hardware resources if the release conditions are met; the release conditions include the idle time of the target hardware resources reaching a preset threshold or receiving a manual release instruction.

[0012] Optionally, the method further includes: After successfully reproducing the fault of the artificial intelligence cluster based on the target hardware resources and the target software environment, the fault of the artificial intelligence cluster is recorded as a historical fault. The repaired software environment for the historical fault is used as the new target software environment, and the fault repair verification of the historical fault is performed based on the new target software environment and the target hardware resources.

[0013] Secondly, embodiments of the present invention provide an artificial intelligence cluster fault handling device, the device comprising: The environment snapshot generation module is used to generate an environment snapshot of the current operating environment of the artificial intelligence cluster in the event of a failure in the artificial intelligence cluster. The resource and environment determination module is used to determine the target hardware resources and target software environment that match the environment snapshot in the resource pool corresponding to the artificial intelligence cluster. The fault reproduction module is used to reproduce the faults of the artificial intelligence cluster based on the target hardware resources and the target software environment.

[0014] Optionally, the environment snapshot includes hardware configuration snapshot information and software configuration snapshot information, and the resource and environment determination module includes: The hardware resource and software environment determination submodule is used to determine, within the resource pool corresponding to the artificial intelligence cluster, the target hardware resources that match the hardware configuration snapshot information, and the target software environment that matches the software configuration snapshot information.

[0015] Optionally, the target hardware resource includes a first target hardware resource, and the hardware resource and software environment determination submodule includes: The processing unit information acquisition unit is used to determine, based on the hardware context information, the model of at least one processing unit, the number of processing units, and the network topology of the processing units. The first resource determination unit is used to determine, in the resource pool corresponding to the artificial intelligence cluster, a first target hardware resource that matches the model of the at least one processing unit, the number of the processing units, and the network topology of the processing units according to a preset resource matching rule; the preset resource matching rule is used to guide how to perform resource matching.

[0016] Optionally, the target hardware resource includes a second target hardware resource, and the device further includes: The candidate resource determination submodule is used to determine the model, quantity, and network topology of at least one candidate processing unit in the resource pool corresponding to the artificial intelligence cluster, according to the code information, in the event that the first target hardware resource matching the model, quantity, and network topology of the at least one processing unit fails to be determined according to the preset resource matching rules. The second resource determination submodule is used to determine, in the resource pool corresponding to the artificial intelligence cluster, a second target hardware resource that matches the model of the at least one candidate processing unit, the number of the candidate processing units, and the network topology relationship of the candidate processing units, according to a preset resource matching rule.

[0017] Optionally, the candidate resource determination submodule includes: The historical data determination unit is used to obtain historical error information and the model, quantity, and network topology relationship of the historical processing unit corresponding to the historical error information. The target historical data determination unit is used to determine the target historical error information that matches the error information in the historical error information; The candidate resource matching unit is used to take the model of the historical processing unit corresponding to the target historical error information, the number of the historical processing units, and the network topology relationship of the historical processing units as at least one candidate processing unit model, candidate processing unit number, and candidate processing unit network topology relationship.

[0018] Optionally, the fault reproduction module includes: The driver version determination submodule is used to determine the target driver version based on the software stack information. The target code determination submodule is used to determine the target code based on the code information. The target software environment determination submodule is used to determine the target software environment based on the target driver version and the target code.

[0019] Optionally, the device further includes: The resource release submodule is used to release the target hardware resources after the fault of the artificial intelligence cluster is reproduced, provided that the release conditions are met; the release conditions include the target hardware resources being idle for a preset threshold or receiving a manual release command.

[0020] Optionally, the device further includes: The historical fault recording submodule is used to record the fault of the artificial intelligence cluster as a historical fault after the fault of the artificial intelligence cluster is successfully reproduced based on the target hardware resources and the target software environment. The repair verification submodule is used to take the repaired software environment for the historical fault as the new target software environment, and perform fault repair verification on the historical fault based on the new target software environment and the target hardware resources.

[0021] Thirdly, embodiments of the present invention provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0022] Fourthly, embodiments of the present invention provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0023] In this embodiment of the invention, by generating an environment snapshot of the current operating environment of the AI ​​cluster in the event of a failure, the complex and instantaneous state of the cluster's operating environment at the time of the failure can be captured completely and accurately. Within the resource pool corresponding to the AI ​​cluster, target hardware resources and target software environments matching the environment snapshot are determined, ensuring the consistency between the resource environment used for fault reproduction and the environment at the time of the failure. Finally, by reproducing the AI ​​cluster's failure based on the target hardware resources and target software environment, accurate reproduction of the fault scenario is achieved in a local or test environment. This effectively solves the technical problem of long fault diagnosis time, impacting the operational efficiency and business recovery speed of the AI ​​cluster due to the inability to accurately reproduce the fault scenario. It provides developers with a relatively stable debugging foundation, forming a complete process from fault information collection to on-site reproduction, which helps improve the success rate of fault reproduction and has a positive effect on shortening the average fault repair time and improving the operational efficiency of the AI ​​cluster. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the steps of an embodiment of an artificial intelligence cluster fault handling method according to the present invention; Figure 2This is a flowchart illustrating the steps of another embodiment of the artificial intelligence cluster fault handling method of the present invention; Figure 3 This is a schematic diagram of a scenario of an embodiment of an artificial intelligence cluster fault handling method of the present invention; Figure 4 This is a structural block diagram of an embodiment of an artificial intelligence cluster fault handling device according to the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0027] The following description, in conjunction with the accompanying drawings, details the artificial intelligence cluster fault handling method, apparatus, equipment, and medium provided by the embodiments of the present invention through specific examples and application scenarios.

[0028] Reference Figure 1 The diagram illustrates a flowchart of an embodiment of an artificial intelligence cluster fault handling method according to the present invention, which may specifically include the following steps: Step 101: In the event of a failure in the AI ​​cluster, generate an environment snapshot of the current operating environment of the AI ​​cluster.

[0029] When an AI cluster experiences a failure, an environment snapshot of the cluster's current operating environment is generated. This snapshot is a comprehensive dataset designed to fully record the system state at the time of the failure, providing a complete data foundation for subsequent failure analysis. Its specific content typically includes, but is not limited to, software environment-related information, hardware resource-related information, and AI cluster configuration-related information. Software environment-related information refers to the complete software stack and its configuration status that supports various tasks within the AI ​​cluster (such as model training and inference services). This covers precise version information from the underlying operating system, drivers, runtime libraries, and programming language interpreters / compilers to the upper-level AI frameworks and their dependent libraries, as well as the configuration parameters of these software programs. Examples include the operating system kernel version, GPU (Graphics Processing Unit) driver version, underlying library version, interpreter version, and the precise version numbers of machine learning frameworks and their dependent libraries. This information can be obtained by calling system commands, reading environment variables, or parsing the metadata of container images. This ensures the consistency of software dependencies in the reproduction environment.

[0030] Hardware resource-related information refers to the real-time operating status and topology of relevant computing, storage, and network hardware within the cluster when a failure occurs. Specifically, it may include the model and unique identifier of the server and GPU, the real-time utilization and temperature of the CPU (Central Processing Unit) and GPU, details of video memory usage, memory usage and quantity, and the connection status and communication matrix of the high-speed network. This information can be obtained by calling hardware management interfaces, tools provided by the manufacturer, system performance monitoring tools, and high-speed network management software. This information is used to locate potential hardware defects or resource conflicts and to ensure the consistency of hardware dependencies in the reproduction.

[0031] The configuration information for the AI ​​cluster refers to the complete set of parameters required to start and run AI tasks and manage the cluster itself. This includes task-related hyperparameters (such as learning rate and batch size), distributed training startup commands and node configurations, environment variables, container orchestration strategies, and cluster management software configurations. This information can be obtained by recording startup commands, parsing configuration files, or querying the cluster management platform database. This information is crucial for understanding the task execution context and analyzing configuration errors. Step 101 can be triggered by various mechanisms, such as system monitoring alarms, user manual commands, or application-layer anomalies. This method can be applied to scenarios such as intelligent operation and maintenance and fault diagnosis, disaster recovery and system reconstruction, compliance auditing and tracing, and R&D testing and continuous integration using AI clusters.

[0032] Step 102: In the resource pool corresponding to the artificial intelligence cluster, determine the target hardware resources and target software environment that match the environment snapshot.

[0033] Based on the environment snapshot generated in step 101, a target environment that is highly consistent with or equivalent to the one at the time of the fault is prepared from the set of schedulable resources of the artificial intelligence cluster for subsequent fault reproduction. This includes two key dimensions: the determination of the target hardware resources and the determination of the target software environment.

[0034] Determining the target hardware resources and target software environment is a comprehensive decision-making and resource scheduling process. The system first parses the hardware resource information and software environment information from the environment snapshot, extracting key features. Then, based on these features, it filters, matches, and evaluates resources within the resource pool to identify the candidate resource combination that best reproduces the original fault environment. Finally, it selects and allocates (or prepares) a set of idle or rapidly deployable target hardware resources and target software environment specifically for this fault reproduction task. The resource pool broadly refers to the set of all allocable computing resources uniformly managed and scheduled by the cluster management system, including but not limited to physical servers, GPU cards, virtualization instances, container resources, network and storage resources, etc. It may contain dedicated test environment resources or temporarily isolated production environment backup resources.

[0035] In some cases, if no resources can be found in the resource pool that are completely identical to the original environment in every detail, the system can prioritize matching based on a preliminary analysis of the fault symptoms and snapshot information (e.g., inferring that the fault is more likely caused by a specific software configuration or specific hardware characteristics). For example, for suspected software-related faults, absolute consistency of the software environment can be prioritized, while hardware resources can meet downgrade compatibility or minimum computing power requirements; for suspected hardware-related faults, critical hardware components can be prioritized. If the root cause of the fault is complex and involves software and hardware interaction, a high degree of matching between the two should still be ensured as much as possible.

[0036] For example, the determination of target hardware resources can primarily rely on the hardware resource information contained in the environment snapshot, as mentioned above. The system will use preset matching and filtering rules to match and filter hardware resource information within the resource pool, identifying physical or virtual resources that are identical or highly similar to the faulty environment in key hardware components. This specifically includes, but is not limited to: identical CPU and GPU models and quantities, consistent GPU computing power specifications, identical memory (including CPU memory and GPU video memory) capacity and configuration, similar storage types and performance characteristics, and consistent network topology and bandwidth capabilities (such as identical PCIe (Peripheral Component Interconnect Express) topology, network interface card models, and high-speed network interconnection methods). If the snapshot records specific hardware serial numbers or firmware version information, matching will also be given priority when conditions permit, to ensure consistency in hardware-level behavior.

[0037] In another example, the determination of the target software environment can primarily rely on the software environment-related information that can be included in the environment snapshot, as mentioned above. The system will use preset matching and filtering rules to match and filter the software environment-related information in the resource pool, identifying an isolated environment capable of accurately reproducing the faulty software configuration. This may require the target software environment to have the exact same operating system version and its patches, GPU driver version, CUDA toolkit version, various underlying system library versions, programming language interpreter / compiler version, and the precise version of the artificial intelligence framework and all its dependent libraries installed as recorded in the snapshot. The specific content can be flexibly adjusted according to business needs and the type of fault.

[0038] Step 103: Based on the target hardware resources and the target software environment, reproduce the fault of the artificial intelligence cluster.

[0039] By comprehensively utilizing the target hardware resources and target software environment determined in the preceding steps, the fault reproduction of the artificial intelligence cluster is performed. This step is the final verification stage, aiming to automatically reconstruct the fault scene and attempt to trigger the same phenomena in order to observe phenomena that are the same as or similar to the original fault, thereby providing direct observation objects and data for subsequent fault localization and root cause analysis.

[0040] Specifically, the fault reproduction process is a systematic operational procedure; for example, this procedure typically includes the following key steps: The first step is environment preparation and configuration. Based on the target hardware resources and target software environment determined in step 102, the system will automatically or under the guidance of operations personnel allocate and isolate the target hardware resources in the resource pool, and then deploy and configure the target software environment on them. This includes ensuring that the physical connections and network configurations of the target hardware resources are correct, and that the operating system, drivers, artificial intelligence frameworks, and dependent libraries in the target software environment are precisely installed and set according to the versions and configuration parameters recorded in the environment snapshot. This process may involve technologies such as containerized deployment, virtual machine cloning, or automated script deployment on physical machines.

[0041] Secondly, there is the reproduction of data and code. If, after the target environment is ready, it is necessary to deploy the data being processed at the time of the failure (or a representative sample thereof), as well as the AI ​​model code, training scripts, or inference service code (corresponding to a specific version, such as the version identified by the code repository commit hash value) to the target software environment, then during the process of obtaining the environment snapshot, attention should be paid to recording the above content, thereby ensuring that the data and code versions used are completely consistent with those at the time of the failure.

[0042] Next comes task execution and state reproduction. Based on the information about the AI ​​cluster configuration recorded in the environment snapshot (such as training hyperparameters, distributed training start commands and parameters, environment variables, task start commands, etc.), the same AI task (such as model training task, inference service request, etc.) is started and executed in the target environment in the same or as similarly as possible to the way it occurred at the time of the original failure. This may include reproducing the same task start process, parameter settings, input data sequence, and possible concurrent user request patterns.

[0043] Finally, fault observation and recording are performed. During task execution, the system closely monitors the status of target hardware resources, such as CPU / GPU utilization, memory usage, network traffic, target software environment runtime logs, and the task's output information. If, during the reproduction process, conditions consistent with or highly similar to the original fault symptoms, such as program crashes, service unresponsiveness, abnormal results, specific error codes, or sudden performance drops, the fault is considered successfully reproduced. At this point, detailed monitoring data, log information, and error stack traces from the reproduction process are recorded as a basis for further in-depth analysis.

[0044] In other examples, if the expected fault is not observed on the first attempt, the reproduction strategy may need to be adjusted based on the initial observations. For example, some environmental variables may need to be matched more precisely, the order or size of data input may need to be adjusted, or the target environment configuration may need to be checked for any unnoticed differences. Multiple attempts may be made until the environmental factors are successfully reproduced or eliminated.

[0045] By generating an environment snapshot of the current operating environment of the AI ​​cluster in the event of a failure, the complex and instantaneous state of the cluster's operating environment at the time of the failure can be captured completely and accurately. Within the resource pool corresponding to the AI ​​cluster, target hardware resources and target software environments matching the environment snapshot are identified, ensuring the consistency of the resource environment used to reproduce the failure with that at the time of the failure. Finally, by reproducing the AI ​​cluster's failure based on the target hardware resources and target software environment, accurate reproduction of the failure scenario is achieved in a local or test environment. This effectively solves the technical problem of long fault diagnosis time, impacting the operational efficiency and business recovery speed of AI clusters due to the inability to accurately reproduce failure scenarios. It provides developers with a relatively stable debugging foundation, forming a complete process from fault information collection to on-site reproduction, which helps improve the success rate of fault reproduction and has a positive effect on shortening the average fault repair time and improving the operational efficiency of AI clusters.

[0046] Reference Figure 2 The flowchart illustrates another embodiment of the artificial intelligence cluster fault handling method of the present invention, which may specifically include the following steps and contents: Step 201: In the event of a failure in the AI ​​cluster, generate an environment snapshot of the current operating environment of the AI ​​cluster; for an explanation of step 201, please refer to the explanation of step 101, which will not be repeated here.

[0047] Step 202: In the resource pool corresponding to the artificial intelligence cluster, determine the target hardware resources and target software environment that match the environment snapshot; for an explanation of step 202, please refer to the explanation of step 102, which will not be repeated here.

[0048] In one embodiment, the environment snapshot includes hardware configuration snapshot information and software configuration snapshot information, and step 202 may include the following sub-steps: Sub-step S11: In the resource pool corresponding to the artificial intelligence cluster, determine the target hardware resources that match the hardware configuration snapshot information, and determine the target software environment that matches the software configuration snapshot information.

[0049] Based on the hardware configuration snapshot information and software configuration snapshot information extracted from the environment snapshot, a bidirectional matching process is performed in the resource pool to determine the target hardware resources and target software environment for fault reproduction. Specifically, for determining the target hardware resources, the system will filter physical or virtual resources with the same or highly similar key hardware components from the resource pool based on the hardware configuration snapshot information, such as the same model and number of CPUs and GPUs, consistent memory (including video memory) configuration, and similar storage and network characteristics. For determining the target software environment, the system will filter or construct an isolated environment from the resource pool based on the software configuration snapshot information, installing an operating system, drivers, artificial intelligence frameworks, and their dependent libraries that are completely identical to the snapshot records. This process involves scanning resources within the resource pool, applying matching rules, and evaluating and selecting the best candidate resources to ensure that the determined target hardware resources and target software environment can together form a basis sufficient to reproduce the original fault.

[0050] In one embodiment, the software configuration snapshot information includes code information and / or software stack information; the hardware configuration snapshot information includes hardware context information and / or configuration parameter information of the artificial intelligence cluster.

[0051] The software stack information includes the container image identifiers and version numbers of the accompanying drivers that the current operating environment of the AI ​​cluster depends on. This information is typically obtained by querying the container runtime interface or parsing the image metadata file. Its core purpose is to ensure that the fault reproduction environment has completely consistent software binary dependencies and underlying driver support with the original environment. Hardware context information specifically includes the specific model of the processing unit, the number deployed, and its network topology connections within the cluster. This information is collected by calling hardware management interfaces and cluster management tools. Its core purpose is to provide a highly consistent physical hardware foundation in terms of computing power, interconnect bandwidth, and communication paths for fault reproduction. Code information specifically includes the commit hash value that uniquely identifies the code version, the actual code content loaded at the time of the fault, and detailed error messages generated during runtime. This information is obtained by interacting with the version control system and reading runtime memory or log files. Its core purpose is to ensure the logical consistency of the executed code and to provide a benchmark for comparing fault phenomena. Configuration data is specified as the running parameters specified when starting the AI ​​task, the data storage paths for the model and dataset, and the environment configuration parameters of the operating system or container. This information is obtained by parsing the task startup script, configuration files, or cluster database. Its core purpose is to completely reproduce the initial state and external conditions of the task's operation.

[0052] When an AI cluster experiences a failure, an environment snapshot of the cluster's current operating environment is generated. This snapshot is a comprehensive dataset designed to fully record the system state at the time of the failure, providing a complete data foundation for subsequent failure analysis. Its content includes four main categories of information. Software stack information refers to the complete software environment supporting the operation of AI tasks, along with its versions and configuration status. This includes, for example, the operating system kernel version, driver version, underlying library version, interpreter version, and the precise version numbers of machine learning frameworks and their dependent libraries. This information can be obtained by calling system commands, reading environment variables, or parsing the metadata of container images. This information ensures the consistency of software dependencies in the reproduction environment.

[0053] Hardware context information refers to the real-time operating status and topology of relevant computing, storage, and network hardware within the cluster when a failure occurs. Specifically, it may include the model and unique identifier of the server and GPU, the real-time utilization and temperature of the CPU and GPU, details of video memory usage, memory usage and quantity, and the connection status and communication matrix of the high-speed network. It can be obtained by calling hardware management interfaces, tools provided by the manufacturer, system performance monitoring tools, and high-speed network management software. This information is used to locate potential hardware defects or resource conflicts and to ensure the consistency of hardware dependencies in the reproduction.

[0054] Code information refers to key identifiers of the AI ​​model code that caused the failure and its execution status, including but not limited to the commit hash of the code repository, the storage path and checksum of the training script, the version of the model structure definition file, and, where possible, the function call stack information at the time of the exception. This information can be obtained through integration with a version control system, embedding log points in the code, or automatic capture by the framework's built-in or custom code modules when exceptions occur. This information ensures that the code used for reproduction is completely consistent with the code at the time of the failure. Configuration parameter information for the AI ​​cluster refers to the complete set of parameters used to start and configure this AI task, such as training hyperparameters, the startup command and parameters for distributed training, environment variables, and container orchestration file content. This can be achieved by recording the task startup command, parsing configuration files, or retrieving information from the cluster management platform's database.

[0055] In one embodiment, the target hardware resource includes a first target hardware resource, and sub-step S11 may include the following sub-steps: Step S111: Based on the hardware context information, determine the model of at least one processing unit, the number of processing units, and the network topology of the processing units. Key feature parameters for hardware resource matching are extracted from the hardware context information. These parameters form the basis for subsequent resource matching. Specifically, processing units mainly refer to the hardware components in the AI ​​cluster that undertake core computing tasks, such as GPUs and / or CPUs. Model refers to the specific model identifier of these processing units, such as a specific series of GPU or CPU models; quantity refers to the number of processing units of that model deployed; network topology refers to the connection methods, communication link types (such as PCIe, InfiniBand), bandwidth configuration, and data flow between these processing units and with other network devices. By parsing the hardware context information, the system can clearly identify the specific configuration and interconnection status of critical processing resources in the cluster when a fault occurs.

[0056] Step S112: In the resource pool corresponding to the artificial intelligence cluster, a first target hardware resource is determined according to a preset resource matching rule to match the model of the at least one processing unit, the number of processing units, and the network topology of the processing units; the preset resource matching rule is used to guide how to perform resource matching.

[0057] Based on the three key dimensions determined in step S111—processing unit model, quantity, and network topology—the first target hardware resources that meet the requirements are selected from the resource pool. The preset resource matching rules are the core judgment logic for this selection process, which include matching priority and similarity requirements for the above three dimensions.

[0058] For example, the preset resource matching rules can stipulate the following: First, perform precise or highly compatible matching of processing unit models to ensure that the processing unit models of candidate resources are the same as or functionally equivalent to those in the fault environment; second, perform consistency verification of the number of processing units, requiring that the number of the same model of processing units provided by the candidate resources be completely consistent with that of the fault environment to ensure matching of computing scale; finally, perform a similarity comparison of network topology relationships to ensure that the interconnection method, communication bandwidth, and latency characteristics between processing units in the candidate resources are consistent with or highly similar to the network topology relationships recorded in the fault environment. By applying the above preset resource matching rules, the system will screen and determine the hardware resources in the resource pool that are closest to the fault environment in terms of core computing power, processing scale, and communication characteristics, as the first target hardware resources. This provides a basic guarantee for accurately reproducing faults that may be related to processing units and their interconnections under similar hardware conditions. This matching process can be implemented by calling the resource query interface and topology discovery interface of the cluster management system.

[0059] In one embodiment, the target hardware resource includes a second target hardware resource, and sub-step S11 may include the following: If, in the resource pool corresponding to the artificial intelligence cluster, a first target hardware resource matching the model, quantity, and network topology of the at least one processing unit fails to be determined according to a preset resource matching rule, then, based on the code information, at least one candidate processing unit model, quantity, and network topology are determined; and in the resource pool corresponding to the artificial intelligence cluster, a second target hardware resource matching the model, quantity, and network topology of the at least one candidate processing unit is determined according to the preset resource matching rule.

[0060] This embodiment aims to provide a degradation matching strategy. When a completely identical ideal resource, i.e., the first target hardware resource, cannot be found, the system can intelligently find a functionally similar alternative environment based on the specific characteristics of the fault. Specifically, the implementation can include two stages: Candidate resource configuration generation: If the first target hardware resource fails based on hardware context information, the system then analyzes the code information. By parsing relevant content contained in the code information, such as error code information, exception type information, and stack trace information, the system infers the hardware characteristics that the fault may depend on, and generates a set of candidate processing unit models, quantities, and candidate network topologies. For example, if the error message points to a driver vulnerability of a specific GPU model, then the candidate model is locked to that model; if the error is related to multi-card communication, then the candidate topology must maintain a certain interconnect bandwidth. Second target hardware resource matching: Subsequently, the generated candidate computing unit models, quantities, and network topologies are used as new inputs, and the system is again filtered in the resource pool corresponding to the artificial intelligence cluster according to preset resource matching rules to finally determine the second target hardware resource.

[0061] In one embodiment, the step of 'determining the model of at least one candidate processing unit, the number of candidate processing units, and the network topology relationship of the candidate processing units based on the code information' may further include the following: Obtain historical error information and the model, quantity, and network topology of the historical processing units corresponding to each historical error information; determine the error information based on the code information; identify the target historical error information that matches the historical error information from the historical error information; and use the model, quantity, and network topology of the historical processing units corresponding to the target historical error information as at least one candidate processing unit model, candidate processing unit quantity, and candidate processing unit network topology.

[0062] The process involves acquiring aggregated historical error messages and information about the corresponding historical processing units. The current error message is then matched against records in the historical database. By executing an error message matching algorithm (e.g., similarity calculation based on error codes, exception types, or stack trace features), historical error messages highly similar to the current error are selected. The model, quantity, and network topology of these historical processing units are then directly used as candidate configurations for the current fault reproduction task. Essentially, this approach leverages validated historical solutions to guide the construction of a new environment, aiming to find a functionally replaceable hardware environment known to trigger similar fault phenomena, thus providing a foundation for subsequent reproduction and debugging.

[0063] In another example, besides determining the model, quantity, and candidate network topology of candidate processing units based on the code information, a pre-defined expert domain knowledge dictionary can be consulted. This dictionary defines known hardware compatibility and substitution relationships (e.g., GPU model A can be replaced by model B when running a specific algorithm); or official replacement guidelines and compatibility lists provided by processing unit manufacturers can be accessed; or a joint decision can be made by combining the above multiple information sources. By introducing an intelligent degradation mechanism based on error analysis, it is ensured that the entire fault reproduction process can continue rather than be interrupted even when ideal reproduction resources are unavailable. This provides developers with more diagnostic opportunities and increases the possibility of locating the root cause of complex faults.

[0064] In one embodiment, sub-step S11 may include the following: Based on the software stack information, determine the target driver version; based on the code information, determine the target code; based on the target driver version and the target code, determine the target software environment.

[0065] Drivers, especially GPU drivers, are crucial bridges connecting the operating system, upper-layer software (such as AI frameworks), and hardware acceleration units (such as GPUs). Their version compatibility and features are essential for the stable operation of AI tasks. Therefore, the system will parse the version identifiers of drivers recorded in the software stack information, such as the version number of the GPU driver, to determine the target driver version. This target driver version should be completely consistent with the driver version used by the AI ​​cluster at the time of the failure to ensure consistency of behavior at the software-hardware interaction level. Target code is the core component of the AI ​​model that caused the failure or was executing at the time of the failure. The system will retrieve and determine the corresponding code entity from the version control system or a designated storage location based on identifiers recorded in the code information, such as the commit hash of the code repository, a specific version or checksum of the training script or model structure definition file, and identify it as the target code. This target code should be completely consistent with the code actually running at the time of the failure, including its logic, structure, and dependencies, to ensure consistency of execution logic. The target software environment contains more than just the target driver version and target code; it is a collection of all the software components and their configurations required for the target code to run correctly with the support of the target driver version. This typically includes operating system versions compatible with the target driver version, AI framework versions compatible with both the target code and the target driver version, and the various underlying libraries and programming language interpreter / compiler versions that the framework depends on. The system will filter existing isolated environments from the resource pool that meet all these conditions, based on the compatibility requirements of the target driver version and whether the target code's dependency declarations on the runtime environment can be included in the code information. Alternatively, it will automatically deploy a new environment containing all the aforementioned components with precisely matching versions, identifying this as the target software environment. Ensuring the integrity and version consistency of the target software environment is fundamental to accurately reproducing software-related faults in subsequent steps.

[0066] Step 203: Based on the target hardware resources and the target software environment, reproduce the fault of the artificial intelligence cluster. For an explanation of step 203, please refer to the explanation of step 103, which will not be repeated here.

[0067] Step 204: After the fault of the artificial intelligence cluster is reproduced, the target hardware resource is released if the release conditions are met; the release conditions include the target hardware resource being idle for a preset threshold or receiving a manual release command.

[0068] After the fault in the AI ​​cluster is reproduced, the system will continuously monitor the usage status of the target hardware resources. When the idle time of the target hardware resources reaches a preset threshold, or a manual release command is received through the management interface, the system will automatically trigger the resource release procedure. This release procedure calls the cluster manager's resource management application interface to remove the isolation status of the target hardware resource, re-mark it as available, and return it to the resource pool. The core purpose of this management process is to achieve automated lifecycle management of computing resources, avoid resource silos and waste caused by forgetting to release resources, and thus significantly improve the utilization efficiency and economic benefits of the entire AI cluster resources.

[0069] In one embodiment, the method may further include the following: After successfully reproducing the fault of the artificial intelligence cluster based on the target hardware resources and the target software environment, the fault of the artificial intelligence cluster is recorded as a historical fault; the software environment after repair for the historical fault is obtained; the software environment after repair for the historical fault is used as the new target software environment, and the historical fault is repaired and verified based on the new target software environment and the target hardware resources.

[0070] After successfully reproducing the fault, the remediation plan is further verified to ensure its effectiveness. This step is typically performed after the fault has been successfully reproduced, the root cause has been located, and a corresponding patch, update, or configuration adjustment has been developed.

[0071] For example, this step may include the following key elements: First, historical faults are recorded. After successfully reproducing a fault based on the target hardware resources and target software environment, the system formally records the relevant information of the fault, such as the description of the fault phenomenon, the time of occurrence, the original environment snapshot identifier, the reproduction steps, and the preliminary analysis conclusions, as a historical fault record. This is helpful for fault tracking, management, statistical analysis, and the construction of a knowledge base.

[0072] Secondly, obtain the repaired software environment. For recorded historical faults, developers or system administrators will propose and implement corresponding remedial measures. These measures typically involve software-level changes, such as updating specific software library versions, fixing defects in application code, adjusting configuration parameters, and upgrading or downgrading drivers. The new set of software configurations resulting from these changes is called the repaired software environment. Obtaining this repaired software environment can be achieved in various ways, such as pulling the repaired code branch from a version control system, obtaining the updated software installation package or image, or reading files containing the repair configuration.

[0073] Finally, the repaired software environment for the historical fault is used as the new target software environment for fault repair verification. While keeping the target hardware resources unchanged, the original target software environment is replaced with the acquired repaired software environment. Subsequently, in this newly configured environment (i.e., the combination of target hardware resources and the repaired software environment), the same tasks and operation steps as in the historical fault reproduction process are executed again. The system will monitor the task execution process to observe whether the previously recorded historical fault phenomena reappear. If, after multiple executions of the same operations, the historical fault phenomena no longer reappear and the system runs stably, the repaired software environment can be determined to be effective, and the historical fault has been resolved; otherwise, it indicates that the repair has not met expectations, and the repair plan needs to be re-examined. Through this step, the actual effect of the fault repair measures can be effectively verified, ensuring that the repair plan can truly solve the problem, thereby improving the reliability and stability of the AI ​​cluster. This process is of great significance for fault verification in the continuous integration / continuous deployment (CI / CD) process and for ensuring the stable operation of the production environment.

[0074] In one embodiment, this method can be applied to fault reproduction and diagnosis in large-scale distributed training scenarios. The following detailed explanation uses a specific scenario as an example: There is an AI training cluster containing multiple computing nodes (e.g., 16 computing nodes), which is running a precision training task for a large language model. This task employs a distributed training architecture and uses a specific communication library for inter-node data communication. During training, a communication timeout error causes the training task to fail.

[0075] Following the failure, the system automatically triggers a snapshot generation process to capture critical environmental information at the time of the failure. This snapshot contains the following core information: software stack information, specifically recording the container image identifier running the training task, such as the container image's SHA256 hash value; and the driver version running on the underlying host, such as v3.5.0; hardware context information, specifically recording the hardware configuration used by the training task, such as the GPU accelerator card model, the number of compute nodes participating in training, the number of GPU accelerator cards configured on each node, and the network topology used by the cluster; code information, specifically recording the source code information of the training task, such as the version control repository address, the specific commit version identifier of the code used at the time of the failure, and the script path that started the training task; and configuration parameter information of the AI ​​cluster, specifically recording the dataset path that the training task depends on and the configuration file path used during training. This snapshot will serve as a baseline reference for subsequent failure reproduction.

[0076] Users can initiate the entire reproduction process by clicking 'One-Click Reproduction' (or other corresponding UI buttons, 'One-Click Reproduction' is used as an example here) on the corresponding fault event record page in the issue tracking system. After the 'One-Click Reproduction' command is issued, the system first parses the hardware context information recorded in the aforementioned snapshot. Based on the parsing results, the target hardware resource specifications required to reproduce the fault are determined: for example, 16 compute nodes are required, each equipped with a specified model of GPU accelerator card, and configured with a specific type of network interconnection. Subsequently, 16 idle server nodes that meet the above specifications are selected from the resource pool corresponding to the artificial intelligence cluster, and these nodes are reserved and identified as the target hardware resources for this fault reproduction.

[0077] After the target hardware resources are successfully reserved, an automated deployment process is executed on these nodes. First, based on the container image identifier recorded in the snapshot, the specified version of the container image is pulled and loaded on each target node. Next, according to the network topology recorded in the snapshot, the network between the target nodes is configured to ensure that the network environment is consistent with that at the time of the failure. Then, based on the version control repository address and specific commit version identifier recorded in the code information, the corresponding version of the training code is pulled from the code repository to the target nodes. Finally, the system automatically locates and executes the training script recorded in the snapshot, thereby initiating the reproduction process of the training task. The entire deployment and execution process requires no manual intervention, achieving a high degree of automation.

[0078] After the reproduction task is initiated, the system continuously monitors its running status. If the training task encounters the same error as the original fault again during execution, the system determines that the fault reproduction is successful. Subsequently, the system automatically associates and updates the reproduction results (including whether the fault was successfully reproduced) and the log information generated during the reproduction process to the corresponding issue tracking system event for developers to view. In addition, to facilitate in-depth debugging by developers, they can log in to the target hardware resource environment of the reproduced fault through the Secure Shell protocol or other similar protocols to directly analyze the fault scene and locate the problem. Through the above process, faults caused by complex environment configurations or software dependencies in large-scale distributed training scenarios can be reproduced efficiently and accurately, providing strong support for quickly locating and resolving problems.

[0079] Reference Figure 3 The diagram illustrates a scenario of an embodiment of the artificial intelligence cluster fault handling method of the present invention, the details of which are as follows: First, when a failure occurs in AI cluster 100, the central controller 2001 of AI cluster server 200 is automatically notified. Then, the central controller 2001 sends an environment snapshot generation command to the event management system 2002. Upon receiving the command, the event management system 2002 automatically captures the current hardware resources (i.e., the hardware resources used) and current software environment of AI cluster 100 at the time of the failure. After capture, the event management system 2002 records this environment snapshot and stores it along with the failure log. Subsequently, when a user or other system sends a failure reproduction command to the central controller 2001, the central controller 2001 retrieves and parses the corresponding environment snapshot from the event management system 2002 based on the failure ID in the reproduction command. Based on the parsing results, it matches the resources required for failure reproduction from the total hardware resource pool and software resource pool of AI cluster server 200, and constructs a temporary reproduction environment based on these matched resources. Then, the failure of AI cluster 100 is fully reproduced in the reproduction environment.

[0080] By generating an environment snapshot of the current operating environment of the AI ​​cluster in the event of a failure, the complex and instantaneous state of the cluster's operating environment at the time of the failure can be captured completely and accurately. Within the resource pool corresponding to the AI ​​cluster, target hardware resources and target software environments matching the environment snapshot are identified, ensuring the consistency of the resource environment used to reproduce the failure with that at the time of the failure. Finally, by reproducing the AI ​​cluster's failure based on the target hardware resources and target software environment, accurate reproduction of the failure scenario is achieved in a local or test environment. This effectively solves the technical problem of long fault diagnosis time, impacting the operational efficiency and business recovery speed of AI clusters due to the inability to accurately reproduce failure scenarios. It provides developers with a relatively stable debugging foundation, forming a complete process from fault information collection to on-site reproduction, which helps improve the success rate of fault reproduction and has a positive effect on shortening the average fault repair time and improving the operational efficiency of AI clusters.

[0081] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0082] Reference Figure 4 The diagram shows a structural block diagram of an embodiment of an artificial intelligence cluster fault handling device according to the present invention, the details of which are as follows: The environment snapshot generation module 301 is used to generate an environment snapshot of the current operating environment of the artificial intelligence cluster in the event of a failure in the artificial intelligence cluster. The resource and environment determination module 302 is used to determine the target hardware resources and target software environment that match the environment snapshot in the resource pool corresponding to the artificial intelligence cluster. The fault reproduction module 303 is used to reproduce the faults of the artificial intelligence cluster based on the target hardware resources and the target software environment.

[0083] Optionally, the environment snapshot includes hardware configuration snapshot information and software configuration snapshot information, and the resource and environment determination module includes: The hardware resource and software environment determination submodule is used to determine, within the resource pool corresponding to the artificial intelligence cluster, the target hardware resources that match the hardware configuration snapshot information, and the target software environment that matches the software configuration snapshot information.

[0084] Optionally, the target hardware resource includes a first target hardware resource, and the hardware resource and software environment determination submodule includes: The processing unit information acquisition unit is used to determine, based on the hardware context information, the model of at least one processing unit, the number of processing units, and the network topology of the processing units. The first resource determination unit is used to determine, in the resource pool corresponding to the artificial intelligence cluster, a first target hardware resource that matches the model of the at least one processing unit, the number of the processing units, and the network topology of the processing units according to a preset resource matching rule; the preset resource matching rule is used to guide how to perform resource matching.

[0085] Optionally, the target hardware resource includes a second target hardware resource, and the device further includes: The candidate resource determination submodule is used to determine the model, quantity, and network topology of at least one candidate processing unit in the resource pool corresponding to the artificial intelligence cluster, according to the code information, in the event that the first target hardware resource matching the model, quantity, and network topology of the at least one processing unit fails to be determined according to the preset resource matching rules. The second resource determination submodule is used to determine, in the resource pool corresponding to the artificial intelligence cluster, a second target hardware resource that matches the model of the at least one candidate processing unit, the number of the candidate processing units, and the network topology relationship of the candidate processing units, according to a preset resource matching rule.

[0086] Optionally, the candidate resource determination submodule includes: The historical data determination unit is used to obtain historical error information and the model, quantity, and network topology relationship of the historical processing unit corresponding to the historical error information. The target historical data determination unit is used to determine the target historical error information that matches the error information in the historical error information; The candidate resource matching unit is used to take the model of the historical processing unit corresponding to the target historical error information, the number of the historical processing units, and the network topology relationship of the historical processing units as at least one candidate processing unit model, candidate processing unit number, and candidate processing unit network topology relationship.

[0087] Optionally, the fault reproduction module includes: The driver version determination submodule is used to determine the target driver version based on the software stack information. The target code determination submodule is used to determine the target code based on the code information. The target software environment determination submodule is used to determine the target software environment based on the target driver version and the target code.

[0088] Optionally, the device further includes: The resource release submodule is used to release the target hardware resources after the fault of the artificial intelligence cluster is reproduced, provided that the release conditions are met; the release conditions include the target hardware resources being idle for a preset threshold or receiving a manual release command.

[0089] Optionally, the device further includes: The historical fault recording submodule is used to record the fault of the artificial intelligence cluster as a historical fault after the fault of the artificial intelligence cluster is successfully reproduced based on the target hardware resources and the target software environment. The repair verification submodule is used to take the repaired software environment for the historical fault as the new target software environment, and perform fault repair verification on the historical fault based on the new target software environment and the target hardware resources.

[0090] The AI ​​cluster fault handling device in this embodiment of the invention can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This embodiment of the invention does not impose specific limitations.

[0091] The artificial intelligence cluster fault handling device in this embodiment of the invention can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this embodiment of the invention does not impose specific limitations.

[0092] The artificial intelligence cluster fault handling device provided in this embodiment of the invention can achieve Figures 1 to 3 To avoid repetition, the various processes in the method embodiments will not be described again here.

[0093] By generating an environment snapshot of the current operating environment of the AI ​​cluster in the event of a failure, the complex and instantaneous state of the cluster's operating environment at the time of the failure can be captured completely and accurately. Within the resource pool corresponding to the AI ​​cluster, target hardware resources and target software environments matching the environment snapshot are identified, ensuring the consistency of the resource environment used to reproduce the failure with that at the time of the failure. Finally, by reproducing the AI ​​cluster's failure based on the target hardware resources and target software environment, accurate reproduction of the failure scenario is achieved in a local or test environment. This effectively solves the technical problem of long fault diagnosis time, impacting the operational efficiency and business recovery speed of AI clusters due to the inability to accurately reproduce failure scenarios. It provides developers with a relatively stable debugging foundation, forming a complete process from fault information collection to on-site reproduction, which helps improve the success rate of fault reproduction and has a positive effect on shortening the average fault repair time and improving the operational efficiency of AI clusters.

[0094] This invention also provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the various processes of the above-described embodiment of an artificial intelligence cluster fault handling method and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0095] This invention also provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the various processes of the above-described embodiment of an artificial intelligence cluster fault handling method and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0096] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0097] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0098] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0099] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A method for handling faults in an artificial intelligence cluster, characterized in that, The method includes: In the event of a failure in the AI ​​cluster, an environment snapshot of the current operating environment of the AI ​​cluster is generated; In the resource pool corresponding to the artificial intelligence cluster, target hardware resources and target software environment that match the environment snapshot are determined; Based on the target hardware resources and the target software environment, reproduce the fault of the artificial intelligence cluster.

2. The artificial intelligence cluster fault handling method according to claim 1, characterized in that, The environment snapshot includes hardware configuration snapshot information and software configuration snapshot information. The step of determining the target hardware resources and target software environment matching the environment snapshot within the resource pool corresponding to the artificial intelligence cluster includes: In the resource pool corresponding to the artificial intelligence cluster, target hardware resources that match the hardware configuration snapshot information are determined, and target software environments that match the software configuration snapshot information are determined.

3. The artificial intelligence cluster fault handling method according to claim 2, characterized in that, The software configuration snapshot information includes code information and / or software stack information; the hardware configuration snapshot information includes hardware context information and / or configuration parameter information of the artificial intelligence cluster.

4. The artificial intelligence cluster fault handling method according to claim 3, characterized in that, The target hardware resources include first target hardware resources. The step of determining the target hardware resources matching the hardware configuration snapshot information in the resource pool corresponding to the artificial intelligence cluster includes: Based on the hardware context information, determine the model of at least one processing unit, the number of processing units, and the network topology of the processing units: In the resource pool corresponding to the artificial intelligence cluster, a first target hardware resource is determined according to a preset resource matching rule, which matches the model of the at least one processing unit, the number of the processing units, and the network topology of the processing units; the preset resource matching rule is used to guide how to perform resource matching.

5. The artificial intelligence cluster fault handling method according to claim 4, characterized in that, The target hardware resources include second target hardware resources. The step of determining the target hardware resources that match the hardware configuration snapshot information within the resource pool corresponding to the artificial intelligence cluster includes: If, in the resource pool corresponding to the artificial intelligence cluster, a first target hardware resource matching the model, quantity, and network topology of the at least one processing unit fails to be determined according to a preset resource matching rule, at least one candidate processing unit model, quantity, and network topology are determined based on the code information. In the resource pool corresponding to the artificial intelligence cluster, a second target hardware resource is determined according to a preset resource matching rule, which matches the model of the at least one candidate processing unit, the number of the candidate processing units, and the network topology relationship of the candidate processing units.

6. The artificial intelligence cluster fault handling method according to claim 5, characterized in that, The step of determining the model, quantity, and network topology of at least one candidate processing unit based on the code information includes: Obtain historical error information and the model, quantity, and network topology of the historical processing units corresponding to the historical error information; Based on the code information, determine the error message; In the historical error messages, a target historical error message that matches the error message is identified; The model of the historical processing unit corresponding to the target historical error information, the number of the historical processing units, and the network topology relationship of the historical processing units are used as at least one candidate processing unit model, candidate processing unit number, and candidate processing unit network topology relationship.

7. The artificial intelligence cluster fault handling method according to claim 3, characterized in that, The determination of the target software environment that matches the software configuration snapshot information includes: Based on the software stack information, determine the target driver version; Based on the code information, determine the target code; The target software environment is determined based on the target driver version and the target code.

8. The artificial intelligence cluster fault handling method according to claim 1, characterized in that, The method further includes: after the fault of the artificial intelligence cluster is reproduced, releasing the target hardware resources if the release conditions are met; the release conditions include the idle time of the target hardware resources reaching a preset threshold or receiving a manual release command.

9. The artificial intelligence cluster fault handling method according to claim 2, characterized in that, The method further includes: After successfully reproducing the fault of the artificial intelligence cluster based on the target hardware resources and the target software environment, the fault of the artificial intelligence cluster is recorded as a historical fault. The repaired software environment for the historical fault is used as the new target software environment, and the fault repair verification of the historical fault is performed based on the new target software environment and the target hardware resources.

10. An artificial intelligence cluster fault handling device, characterized in that, The device includes: The environment snapshot generation module is used to generate an environment snapshot of the current operating environment of the artificial intelligence cluster in the event of a failure in the artificial intelligence cluster. The resource and environment determination module is used to determine the target hardware resources and target software environment that match the environment snapshot in the resource pool corresponding to the artificial intelligence cluster. The fault reproduction module is used to reproduce the faults of the artificial intelligence cluster based on the target hardware resources and the target software environment.

11. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the artificial intelligence cluster fault handling method as described in claims 1-9.

12. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions, which, when executed by a processor, implement the steps of the artificial intelligence cluster fault handling method as described in claims 1-9.

Images