A reliability simulation analysis method and medium for LLM training clusters

Abstract

This invention discloses a reliability simulation analysis method and medium for LLM training clusters. The method involves acquiring and processing the LLM training cluster to be simulated using a multi-dimensional system abstract modeling approach to obtain a standardized simulation model. The method then uses the basic component FIT calculation method to calculate the joint reliability index of the standardized model. At least one type of fault injection event is acquired in real time and injected into the standardized simulation model for discrete simulation, monitoring the reliability index of the current cluster operating parameters and the current fault propagation path. Finally, data statistical analysis is used to obtain and feed back a quantitative report on the reliability simulation analysis of the LLM training cluster. This method solves the problems of high testing costs and difficulties caused by relying on actual clusters for physical testing, improves the accuracy and flexibility of cluster reliability analysis, and reduces testing costs.

Description

Technical Field

[0001] This invention relates to the field of simulation analysis technology, and in particular to a reliability simulation analysis method and medium for LLM training clusters. Background Technology

[0002] In the field of LLM (Large Language Model) training clusters and reliable evaluation and simulation technology, the rapid development of deep learning technology has driven the evolution of neural network models towards ultra-large scale and high complexity, upgrading from conventional deep learning models to large-scale models with hundreds of billions or trillions of parameters. This places stringent demands on the computing power, storage capacity, network transmission capacity, and stability of the clusters. As the scale of LLM training clusters continues to expand, the probability of various anomalies and failures is showing a significant upward trend, and the speed of failure propagation is fast and the scope of impact is wide. Specifically, at the hardware level, problems such as computing node downtime, chip aging, and network link interruption are prone to occur; at the software level, issues such as task scheduling failures, algorithm adaptation anomalies, and system vulnerability triggering frequently occur; at the system level, pain points such as unbalanced load, unreasonable topology design, and untimely fault handling are faced. These problems overlap, seriously reducing the utilization rate and computing performance of intelligent computing clusters, becoming a key bottleneck restricting the efficient development and stable operation of large-scale artificial intelligence models.

[0003] In the process of developing this invention, the inventors discovered the following shortcomings in the existing technology: Traditional methods for assessing the reliability of intelligent computing clusters mainly rely on physical testing of actual clusters, which has many limitations. These limitations include: high testing costs, requiring significant investment in hardware resources; long testing cycles, making it difficult to adapt to the rapid iteration needs of clusters; difficulty in reproducing fault scenarios, making it hard to cover various extreme and abnormal situations; and a lack of quantitative support in the assessment results, failing to provide precise guidance for cluster hardware and software design. Summary of the Invention

[0004] This invention provides a reliability simulation analysis method and medium for LLM training clusters, which improves the accuracy and flexibility of cluster reliability analysis and reduces testing costs.

[0005] According to one aspect of the present invention, a reliability simulation analysis method for LLM training clusters is provided, comprising: The current standardized simulation model is obtained by acquiring and processing the LLM training cluster to be simulated and analyzed using a preset multi-dimensional system abstract modeling method. By using the pre-defined FIT calculation method for basic components, the joint reliability index of the current standardized model is obtained by calculating each basic component in the current standardized simulation model. At least one type of fault injection event is acquired in real time and injected into the current standardized simulation model for discrete simulation, thereby monitoring and obtaining the reliability index of the current cluster operating parameters and the current fault propagation path. Using a pre-defined data statistical analysis method, the joint reliability index of the current standardized model, the reliability index of the current cluster operating parameters, and the current fault propagation path are processed respectively to obtain and feed back a quantitative report of reliability simulation analysis corresponding to the LLM training cluster to be simulated and analyzed.

[0006] According to another aspect of the present invention, a reliability simulation analysis apparatus for LLM training clusters is provided, comprising: The current standardized simulation model determination module is used to acquire and process the LLM training cluster to be simulated and analyzed, and obtain the current standardized simulation model through a preset multi-dimensional system abstract modeling method. The current standardized model joint reliability index determination module is used to calculate the joint reliability index of the current standardized model by using the preset basic component FIT calculation method. The current cluster operating parameter reliability index and current fault propagation path determination module is used to acquire at least one type of fault injection event in real time, inject it into the current standardized simulation model for discrete simulation, and monitor and obtain the current cluster operating parameter reliability index and current fault propagation path. The reliability simulation analysis quantification report determination and feedback module is used to process the joint reliability index of the current standardized model, the reliability index of the current cluster operating parameters, and the current fault propagation path through a preset data statistical analysis method, and obtain and feed back the reliability simulation analysis quantification report corresponding to the LLM training cluster to be simulated and analyzed.

[0007] According to another aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the reliability simulation analysis method for LLM training clusters as described in any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the reliability simulation analysis method for LLM training clusters as described in any embodiment of the present invention.

[0009] The technical solution of this invention obtains and processes the current LLM training cluster to be simulated and analyzed using a preset multi-dimensional system abstract modeling method to obtain a current standardized simulation model; calculates the joint reliability index of the current standardized model by using a preset basic component FIT calculation method; acquires at least one type of fault injection event in real time and injects it into the current standardized simulation model for discrete simulation, monitoring and obtaining the reliability index of the current cluster operating parameters and the current fault propagation path; processes the joint reliability index of the current standardized model, the reliability index of the current cluster operating parameters, and the current fault propagation path using a preset data statistical analysis method to obtain and feed back a quantitative reliability simulation analysis report corresponding to the current LLM training cluster to be simulated and analyzed; solves the problem of high testing cost and difficulty caused by relying on actual clusters for physical testing, improves the accuracy and flexibility of cluster reliability analysis, and reduces testing cost and difficulty.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a reliability simulation analysis method for LLM training clusters provided according to Embodiment 1 of the present invention; Figure 2 This is a detailed flowchart of a reliability simulation analysis method for LLM training clusters provided in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the structure of a reliability simulation analysis device for LLM training clusters provided in Embodiment 3 of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device provided according to Embodiment 4 of the present invention. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0014] It should be noted that the terms "target," "current," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0015] It is worth noting that the information collected in the technical solution of this application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse; if the user chooses to refuse, the process will proceed to the expert decision-making process.

[0016] Example 1 Figure 1 The flowchart of a reliability simulation analysis method for LLM training clusters is provided in Embodiment 1 of the present invention. This embodiment is applicable to scenarios of reliability simulation analysis of LLM training clusters. The method can be executed by a reliability simulation analysis device for LLM training clusters, which can be implemented in hardware and / or software.

[0017] Correspondingly, such as Figure 1 As shown, the method includes: S110. Obtain and process the current LLM training cluster to be simulated and analyzed using a preset multi-dimensional system abstract modeling method to obtain the current standardized simulation model.

[0018] Among them, large-scale LLM training clusters, as the core infrastructure supporting the training and inference of such large models, have become increasingly complex in their system composition. They integrate massive heterogeneous computing nodes, high-speed storage devices, complex network topologies, and multi-level software stacks, involving multiple interactive logics at the three core levels of hardware, software, and system.

[0019] One approach to multi-dimensional system abstract modeling is to analyze the LLM training cluster and model it from the perspectives of hardware architecture, software system, and system operation. Current standardized simulation models are obtained by modeling and simulating the LLM training cluster.

[0020] Optionally, the step of acquiring and processing the LLM training cluster to be simulated and analyzed using a preset multi-dimensional system abstract modeling method to obtain a current standardized simulation model includes: performing multi-dimensional analysis on the acquired LLM training cluster to be simulated and analyzed to obtain hardware architecture description parameters, software system description parameters, and system operation characteristic description parameters; and performing simulation processing on the hardware architecture description parameters, software system description parameters, and system operation characteristic description parameters using the multi-dimensional system abstract modeling method to obtain a current standardized simulation model; wherein, the current standardized simulation model includes at least one of the following: a device performance threshold unit, a load dynamic change model unit, and a fault anomaly mode library unit.

[0021] In this embodiment, by performing multi-dimensional analysis on the LLM training cluster to be simulated and analyzed, corresponding hardware architecture description parameters, software system description parameters, and system operation characteristic description parameters can be obtained. Specifically, the hardware architecture description parameters can include description parameters of the hardware architecture such as computing nodes, storage devices, and network topology. The software system description parameters can include description parameters of the software system such as scheduling algorithms, task models, and system stack levels. The system operation characteristic description parameters can be parameters describing the characteristics of the LLM training cluster during runtime.

[0022] Furthermore, a multi-dimensional system abstraction modeling method can be used to extract key parameters from each dimension of hardware architecture description parameters, software system description parameters, and system operation characteristic description parameters, and then simulate and construct the current standardized simulation model. Specifically, the current standardized simulation model can include core units such as device performance thresholds, load dynamic change models, and fault anomaly mode libraries, ensuring that the simulation model highly matches the actual cluster operating state.

[0023] The advantage of this setup is that by using a multi-dimensional system abstract modeling method to model the LLM training cluster parameters obtained from the analysis, it is possible to accurately model the LLM training cluster that needs performance analysis. It is also possible to better determine and analyze the reliability of the actual LLM training cluster through the established standardized simulation model, thereby reducing testing costs and difficulties and allowing for better early determination of the reliability of the actual LLM training cluster.

[0024] S120. Using the preset basic component FIT calculation method, calculate each basic component in the current standardized simulation model to obtain the joint reliability index of the current standardized model.

[0025] FIT (Failures In Time) can refer to a quantified failure rate (i.e., the number of failures that occur within a certain time period) for an indivisible basic hardware component in a cluster (e.g., a single GPU chip, a network card, or a power module). Specifically, FIT is a standardized unit for measuring the reliability of electronic components, defined as the failure rate per unit of time (FIT). The number of failures that occur within equipment hours (approximately one billion hours).

[0026] The FIT calculation method for basic components can be a method for calculating the FIT values ​​corresponding to different basic components. Basic components can be the smallest hardware unit corresponding to the intelligent computing cluster, serving as the basic unit for reliability calculation. The current standardized model joint reliability index can be an indicator describing the current mean time between failures (MTBF), current availability, and current performance degradation rate of the cluster.

[0027] Optionally, the step of calculating the joint reliability index of the current standardized model by using a preset basic component FIT calculation method for each smallest hardware unit component in the current standardized simulation model includes: defining and setting the basic components of the cluster in the current standardized simulation model; wherein the basic components are the smallest hardware unit components; performing multi-level modeling processing on each basic component by using a multi-level abstract modeling method, and combining the basic component FIT calculation method to obtain the joint reliability index of the current standardized model.

[0028] In this embodiment, it is first necessary to determine the basic components of one or more clusters corresponding to the current standardized simulation model. Then, the basic component-level FIT modeling, node and supernode-level modeling and cluster-level modeling in the multi-level abstract modeling method are combined to perform modeling, and the corresponding joint reliability index of the current standardized model is calculated.

[0029] Specifically, the multi-level abstract modeling method includes basic component-level FIT. Modeling includes node and supernode level modeling and cluster level modeling. The method employs a multi-level abstract modeling approach to perform multi-level modeling processing on each basic component, and combines this with a basic component FIT calculation method to obtain the joint reliability index of the current standardized model. This includes: performing basic component-level FIT modeling on each basic component using the multi-level abstract modeling approach to determine the basic component FIT value and the number of basic components deployed at each level; using node and supernode level modeling, based on the basic component FIT value and the number of basic components deployed at each level, aggregating and calculating the node-level FIT values ​​of various basic components using the basic component FIT calculation method; summing the node-level FIT values ​​of all nodes corresponding to any supernode using supernode level modeling, and summing the obtained current interconnect component FIT values ​​corresponding to the supernode again to obtain the supernode-level FIT value; summing the sum of all supernode-level FIT values, as well as the northbound network FIT value, storage subsystem FIT value, and memory pool FIT value using cluster level modeling to obtain the current cluster-level total FIT value; and using the formula... The current mean time between failures (MTBF) is calculated; where, The current cluster-level total FIT value; obtain the current cluster average repair time using the formula. The current availability is calculated using the following methods: MTTR (Mean Time To Repair) is the average time to repair the cluster; the current failure impact parameters are obtained and combined with the current mean time between failures (MTTR) to calculate the current performance degradation rate; based on the current MTTR, current availability, and current performance degradation rate, the current standardized model joint reliability index is determined.

[0030] In this embodiment, a multi-level abstract modeling method is used to perform basic component-level FIT modeling on each basic component to determine the basic components. The value and the number of basic components deployed at each tier. For example, the basic component FIT value can be defined as... The number of basic deployment components at each level can be set to .

[0031] Furthermore, through node and supernode level modeling, using various , The parameters are calculated by aggregating and calculating the node-level FIT values ​​of various basic components using the basic component FIT calculation method. Specifically... .in, Let FIT be the value of the i-th basic component; Let i represent the number of deployments of the i-th type of basic component at each level; i represents the i-th type of basic component. This represents the node-level FIT value corresponding to each node.

[0032] Correspondingly, by using supernode-level modeling, the node-level FIT values ​​of all nodes corresponding to any supernode can be summed, and then the FIT values ​​of the current interconnect components corresponding to the supernode can be summed again to obtain the supernode-level FIT value. The current interconnect component FIT value can include the FIT values ​​of additional interconnect components such as cable trays and DPU boards included with the supernode. The corresponding supernode-level FIT value can then be calculated.

[0033] A cable tray is a physical structure used to carry and organize high-speed data cables and power lines within the cabinets or supernodes of a computing cluster. Its fit-in-the-tray (FIT) effect may not originate from its own circuitry (which is typically a passive component), but rather from potential poor contact, signal attenuation, or physical damage at the cable and connector fixing points, bends, or interfaces within the tray.

[0034] The DPU (Data Processing Unit) board is a dedicated processor that offloads infrastructure tasks such as networking, storage, and security. It can be a PCB board that integrates one or more DPU chips and their peripheral circuitry (memory, power supply, or PCIe interfaces, etc.). In a supernode, it is responsible for data transfer between nodes and network protocol processing. Its FIT (Fitness Injection) contribution includes all potentially faulty components on the board, such as the DPU chip, memory, power management module, PCIe interface, and solder joints.

[0035] Furthermore, through cluster-level modeling, for example, taking an 8192-card cluster, the sum of all supernode-level FIT values, as well as the northbound network FIT value, storage subsystem FIT value, and memory pool FIT value, can be summed to obtain the current cluster-level total FIT value, which can be set as follows: Accordingly, it can be based on Through formula Calculate the current Mean Time Between Failures (MTBF); obtain the current Mean Time To Repair (MTTR) of the cluster using the formula. The system calculates current availability, obtains current failure impact parameters, and combines them with the current mean time between failures (MTBF) to calculate the current performance degradation rate. In other words, the current standardized model joint reliability index can be determined based on the calculated current MTBF, current availability, and current performance degradation rate.

[0036] Additionally, basic components with a FIT value of less than 1% can be ignored in the modeling process to improve simulation efficiency.

[0037] The advantage of this setup is that it allows for theoretical modeling based on an abstract model, and the quantification of reliability indicators for each basic component of the cluster through mathematical algorithms. This includes standardized model joint reliability indicators such as mean time between failures (MTBF), availability, and performance degradation rate. This enables accurate calculation of standardized model joint reliability indicators, thereby improving the accuracy of cluster reliability analysis.

[0038] S130. Real-time acquisition of at least one type of fault injection event, injection into the current standardized simulation model for discrete simulation, and monitoring to obtain the reliability index of the current cluster operating parameters and the current fault propagation path.

[0039] In this embodiment, while using mathematical algorithms to quantify the reliability indicators of each basic component of the cluster through theoretical modeling, a discrete event simulation mode is also adopted to simulate the actual operation process of the cluster, supporting the automatic injection of various faults according to preset rules, and real-time monitoring of the reliability indicators of cluster operation parameters and fault propagation paths.

[0040] Specifically, the process involves: acquiring at least one type of fault injection event in real time, injecting it into the current standardized simulation model for discrete simulation, and monitoring the reliability indicators of the current cluster operating parameters and the current fault propagation path. This includes: acquiring at least one type of fault injection event in real time; sequentially acquiring one type of target fault injection event and injecting it into the current standardized simulation model for discrete simulation, while monitoring the state to obtain the reliability indicators of the current cluster operating parameters and the current fault propagation path, so as to realize the reproduction of LLM training cluster fault scenarios based on the injected fault injection events; determining whether all fault injection events have been traversed, and if so, outputting the reliability indicators of the current cluster operating parameters and the current fault propagation path corresponding to each type of fault injection event; otherwise, returning to the operation of sequentially acquiring one type of target fault injection event.

[0041] In this embodiment, different types of fault injection events can be acquired and automatically injected into the current standardized simulation model for discrete simulation according to settings. Status monitoring is then performed to obtain the reliability indicators of the current cluster operating parameters and the current fault propagation path. This enables the reproduction of LLM training cluster fault scenarios based on injected fault events, thereby obtaining the reliability indicators of the current cluster operating parameters and the current fault propagation path corresponding to different types of fault events. Alternatively, fault injection events can also be performed manually.

[0042] The advantages of this setup are: it integrates a flexible fault injection mechanism, which can easily simulate and reproduce various fault scenarios, while outputting quantitative analysis reports to provide accurate quantitative support for the design and operation and maintenance optimization of intelligent computing cluster software and hardware, thereby improving the efficiency and pertinence of cluster reliability optimization.

[0043] S140. Using a preset data statistical analysis method, process the current standardized model joint reliability index, the current cluster operating parameter reliability index, and the current fault propagation path respectively, and obtain and feed back a reliability simulation analysis quantitative report corresponding to the LLM training cluster to be simulated and analyzed.

[0044] Among them, the data statistical analysis method can be a weighted average fusion analysis of the reliability indicators of each basic component calculated by mathematical algorithms using theoretical modeling, as well as the reliability indicators of cluster operation parameters and fault propagation paths determined by discrete event simulation mode.

[0045] Optionally, the step of processing the current standardized model joint reliability index, the current cluster operating parameter reliability index, and the current fault propagation path using a preset data statistical analysis method to obtain and feed back a reliability simulation analysis quantification report corresponding to the LLM training cluster to be simulated and analyzed, includes: calculating a target weighted average joint reliability index by performing a weighted average calculation on the current standardized model joint reliability index and the current cluster operating parameter reliability index using the weighted average calculation sub-method in the data statistical analysis method; processing the target weighted average joint reliability index and the current fault propagation path to obtain a reliability simulation analysis quantification report corresponding to the LLM training cluster to be simulated and analyzed; and providing real-time visual feedback of the reliability simulation analysis quantification report to the user.

[0046] In this embodiment, according to user needs, different weights can be set for the results of theoretical modeling and discrete event simulation using a weighted average calculation sub-method, thereby determining the corresponding target weighted average joint reliability index.

[0047] Specifically, the discrete event simulation process records the entire process of fault occurrence, propagation, and handling, enabling rapid reproduction of complex fault scenarios and accurate verification of cluster fault perception, detection, and recovery capabilities. It also analyzes the impact of different fault modes on cluster utilization and computing performance. Furthermore, by combining the target weighted average joint reliability index with the current fault propagation path, a multi-dimensional quantitative report of reliability simulation analysis can be generated and fed back.

[0048] The advantages of this setup are: reliability assessment can be completed without relying on actual large-scale LLM training clusters; the dual-mode assessment architecture combining theoretical modeling and discrete event simulation allows for rapid acquisition of quantitative reliability indicators through theoretical calculations, and also enables the simulation to reproduce the entire failure process, achieving a comprehensive and efficient assessment of cluster reliability, overcoming the limitations of traditional single assessment methods. This significantly reduces testing costs, shortens the assessment cycle, and flexibly covers various extreme failure scenarios, addressing the limitations of traditional physical testing.

[0049] The technical solution of this invention obtains and processes the current LLM training cluster to be simulated and analyzed using a preset multi-dimensional system abstract modeling method to obtain a current standardized simulation model; calculates the joint reliability index of the current standardized model by using a preset basic component FIT calculation method; acquires at least one type of fault injection event in real time and injects it into the current standardized simulation model for discrete simulation, monitoring the reliability index of the current cluster operating parameters and the current fault propagation path; processes the joint reliability index of the current standardized model, the reliability index of the current cluster operating parameters, and the current fault propagation path using a preset data statistical analysis method to obtain and feed back a reliability simulation analysis quantitative report corresponding to the current LLM training cluster to be simulated and analyzed; reliability assessment can be completed without relying on actual large-scale LLM training clusters, significantly reducing testing costs and shortening the assessment cycle, while flexibly covering various extreme fault scenarios, solving the limitations of traditional physical testing, improving the accuracy and flexibility of cluster reliability analysis, and reducing testing costs and difficulties.

[0050] Example 2 Figure 2 This invention provides a detailed flowchart of a reliability simulation analysis method for LLM training clusters, as described in Embodiment 2. This embodiment optimizes the above embodiment. After processing the current standardized model joint reliability index, the current cluster operating parameter reliability index, and the current fault propagation path using a preset data statistical analysis method to obtain and feed back a quantitative reliability simulation analysis report corresponding to the LLM training cluster to be simulated, the method further includes confirming the completion of the reliability simulation analysis operation.

[0051] S210. Obtain and process the current LLM training cluster to be simulated and analyzed using a preset multi-dimensional system abstract modeling method to obtain the current standardized simulation model.

[0052] S220. Using the preset basic component FIT calculation method, calculate each basic component in the current standardized simulation model to obtain the joint reliability index of the current standardized model.

[0053] S230. Real-time acquisition of at least one type of fault injection event, injection into the current standardized simulation model for discrete simulation, and monitoring to obtain the reliability index of the current cluster operating parameters and the current fault propagation path.

[0054] S240. Using a preset data statistical analysis method, process the current standardized model joint reliability index, the current cluster operating parameter reliability index, and the current fault propagation path respectively, and obtain and feed back a reliability simulation analysis quantitative report corresponding to the LLM training cluster to be simulated and analyzed.

[0055] S250. Receive the optimization and adjustment results corresponding to the reliability simulation analysis quantification report in real time; wherein, the optimization and adjustment results include at least one LLM training cluster optimization and adjustment strategy.

[0056] S260. Adjust the LLM training cluster according to the optimization and adjustment strategy of each LLM training cluster, and return to execute the operation of calculating each smallest hardware unit component in the current standardized simulation model through the preset basic component FIT calculation method to obtain the joint reliability index of the current standardized model, until the reliability simulation analysis quantification report meets the user's requirements, then confirm that the reliability simulation analysis operation of the current LLM training cluster to be simulated and analyzed is completed.

[0057] In this embodiment, the parameters in the LLM training cluster can be adjusted according to one or more LLM training cluster optimization adjustment strategies in the optimization adjustment results. After the adjustment is completed, it is also necessary to return to the operation of calculating each smallest hardware unit component in the current standardized simulation model through the preset basic component FIT calculation method to obtain the joint reliability index of the current standardized model, and then generate the optimization adjustment results corresponding to the new reliability simulation analysis quantitative report.

[0058] Once the reliability simulation analysis quantification report meets the user's requirements, the reliability simulation analysis operation of the current LLM training cluster to be simulated is confirmed to be complete; otherwise, it is necessary to continue to adjust the parameters in the LLM training cluster according to the optimization and adjustment strategy of each LLM training cluster.

[0059] The technical solution of this invention involves acquiring and processing the LLM training cluster to be simulated using a preset multi-dimensional system abstract modeling method to obtain a current standardized simulation model; calculating the joint reliability index of the current standardized model using a preset basic component FIT calculation method; acquiring at least one type of fault injection event in real time and injecting it into the current standardized simulation model for discrete simulation, monitoring the reliability index of the current cluster operating parameters and the current fault propagation path; and using a preset data statistical analysis method to analyze the joint reliability index of the current standardized model and the reliability index of the current cluster operating parameters. The current fault propagation path is processed to obtain and feed back a reliability simulation analysis quantification report corresponding to the LLM training cluster to be simulated and analyzed. The optimization and adjustment results corresponding to the reliability simulation analysis quantification report are received in real time. The LLM training cluster is adjusted according to the optimization and adjustment strategies for each LLM training cluster, and then the operation of calculating the joint reliability index of the current standardized model for each smallest hardware unit component using the preset basic component FIT calculation method is executed until the reliability simulation analysis quantification report meets the user's requirements. At this point, the reliability simulation analysis operation for the current LLM training cluster to be simulated and analyzed is confirmed to be complete. The optimization and adjustment results corresponding to the reliability simulation analysis quantification report can be used to adjust the LLM training cluster, thereby generating new reliability simulation analysis results. This can better improve the accuracy and flexibility of cluster reliability analysis and is applicable to various extreme fault scenarios, highlighting the advantages of simulation models in saving testing costs and reducing testing difficulty.

[0060] Example 3 Figure 3 This is a schematic diagram of a reliability simulation analysis device for LLM training clusters provided in Embodiment 2 of the present invention. The reliability simulation analysis device for LLM training clusters provided in this embodiment can be implemented by software and / or hardware, and can be configured in a terminal device or server to implement a reliability simulation analysis method for LLM training clusters according to an embodiment of the present invention. Figure 3 As shown, the device includes: a current standardized simulation model determination module 310, a current standardized model joint reliability index determination module 320, a current cluster operation parameter reliability index and current fault propagation path determination module 330, and a reliability simulation analysis quantitative report determination and feedback module 340.

[0061] Among them, the current standardized simulation model determination module 310 is used to obtain and process the current standardized simulation model by a preset multi-dimensional system abstract modeling method based on the LLM training cluster to be simulated and analyzed. The current standardized model joint reliability index determination module 320 is used to calculate the joint reliability index of the current standardized model by using the preset basic component FIT calculation method. The current cluster operating parameter reliability index and current fault propagation path determination module 330 is used to acquire at least one type of fault injection event in real time, inject it into the current standardized simulation model for discrete simulation, and monitor and obtain the current cluster operating parameter reliability index and current fault propagation path. The reliability simulation analysis quantification report determination and feedback module 340 is used to process the current standardized model joint reliability index, the current cluster operating parameter reliability index and the current fault propagation path through a preset data statistical analysis method, and obtain and feed back the reliability simulation analysis quantification report corresponding to the LLM training cluster to be simulated and analyzed.

[0062] The technical solution of this invention obtains and processes the current LLM training cluster to be simulated and analyzed using a preset multi-dimensional system abstract modeling method to obtain a current standardized simulation model; calculates the joint reliability index of the current standardized model by using a preset basic component FIT calculation method; acquires at least one type of fault injection event in real time and injects it into the current standardized simulation model for discrete simulation, monitoring and obtaining the reliability index of the current cluster operating parameters and the current fault propagation path; processes the joint reliability index of the current standardized model, the reliability index of the current cluster operating parameters, and the current fault propagation path using a preset data statistical analysis method to obtain and feed back a quantitative reliability simulation analysis report corresponding to the current LLM training cluster to be simulated and analyzed; solves the problem of high testing cost and difficulty caused by relying on actual clusters for physical testing, improves the accuracy and flexibility of cluster reliability analysis, and reduces testing cost and difficulty.

[0063] Based on the above embodiments, a reliability simulation analysis operation confirmation module is also included. This module is specifically used to: after processing the current standardized model joint reliability index, the current cluster operating parameter reliability index, and the current fault propagation path using a preset data statistical analysis method to obtain and feed back a reliability simulation analysis quantification report corresponding to the LLM training cluster to be simulated and analyzed, receive in real time the optimization adjustment results corresponding to the reliability simulation analysis quantification report; wherein the optimization adjustment results include at least one LLM training cluster optimization adjustment strategy; adjust the LLM training cluster according to each LLM training cluster optimization adjustment strategy, and return to execute the operation of calculating each smallest hardware unit component in the current standardized simulation model using a preset basic component FIT calculation method to obtain the current standardized model joint reliability index, until the reliability simulation analysis quantification report meets the user's requirements, then confirm the completion of the reliability simulation analysis operation for the current LLM training cluster to be simulated and analyzed.

[0064] Based on the above embodiments, the current standardized simulation model determination module 310 can be specifically used to: perform multi-dimensional analysis on the obtained LLM training cluster to be simulated and analyzed to obtain hardware architecture description parameters, software system description parameters, and system operation characteristic description parameters; and perform simulation processing on the hardware architecture description parameters, software system description parameters, and system operation characteristic description parameters through the multi-dimensional system abstract modeling method to obtain the current standardized simulation model; wherein, the current standardized simulation model includes at least one of the following: a device performance threshold unit, a load dynamic change model unit, and a fault anomaly mode library unit.

[0065] Based on the above embodiments, the current standardized model joint reliability index determination module 320 can be specifically used to: define and set the basic components of the cluster in the current standardized simulation model; wherein, the basic components are the smallest hardware unit components; through a multi-level abstract modeling method, perform multi-level modeling processing on each basic component, and combine the basic component FIT calculation method to obtain the current standardized model joint reliability index.

[0066] Based on the above embodiments, the multi-level abstract modeling method includes basic component-level FIT modeling, node and supernode-level modeling, and cluster-level modeling.

[0067] Based on the above embodiments, the current standardized model joint reliability index determination module 320 can also be specifically used for: performing basic component-level FIT modeling on each basic component using a multi-level abstract modeling method to determine the basic component FIT value and the number of basic components deployed at each level; aggregating and calculating the node-level FIT values ​​of various basic components based on the basic component FIT values ​​and the number of basic components deployed at each level using a basic component FIT calculation method, and summing the node-level FIT values ​​of all nodes corresponding to any supernode using supernode-level modeling, and summing the obtained current interconnect component FIT values ​​corresponding to the supernode again to obtain the supernode-level FIT value; summing the sum of all supernode-level FIT values, the northbound network FIT value, the storage subsystem FIT value, and the memory pool FIT value using cluster-level modeling to obtain the current cluster-level total FIT value; and using the formula... The current mean time between failures (MTBF) is calculated; where, The current cluster-level total FIT value; obtain the current cluster average repair time using the formula. The current availability is calculated using the following methods: MTTR (Mean Time To Repair) is the average time to repair the cluster; the current failure impact parameters are obtained and combined with the current mean time between failures (MTTR) to calculate the current performance degradation rate; based on the current MTTR, current availability, and current performance degradation rate, the current standardized model joint reliability index is determined.

[0068] Based on the above embodiments, the current cluster operating parameter reliability index and current fault propagation path determination module 330 can be specifically used for: acquiring at least one type of fault injection event in real time; sequentially acquiring a type of target fault injection event and injecting it into the current standardized simulation model for discrete simulation and state monitoring to obtain the current cluster operating parameter reliability index and current fault propagation path, so as to realize the reproduction of LLM training cluster fault scenarios based on injected fault injection events; determining whether all fault injection events have been traversed, if so, outputting the current cluster operating parameter reliability index and current fault propagation path corresponding to each type of fault injection event; if not, returning to execute the operation of sequentially acquiring a type of target fault injection event.

[0069] Based on the above embodiments, the reliability simulation analysis quantification report determination and feedback module 340 can be specifically used to: calculate the weighted average of the current standardized model joint reliability index and the current cluster operating parameter reliability index using the weighted average calculation sub-method in the data statistical analysis method, to obtain a target weighted average joint reliability index; process the target weighted average joint reliability index and the current fault propagation path to obtain a reliability simulation analysis quantification report corresponding to the LLM training cluster to be simulated and analyzed; and provide real-time visual feedback of the reliability simulation analysis quantification report to the user.

[0070] The reliability simulation analysis device for LLM training clusters provided in this embodiment of the invention can execute the reliability simulation analysis method for LLM training clusters provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0071] Example 4 Figure 4 A schematic diagram of an electronic device 10, which can be used to implement Embodiment 4 of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0072] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0073] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0074] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as reliability simulation analysis methods for LLM training clusters.

[0075] In some embodiments, the reliability simulation analysis method for LLM training clusters can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the reliability simulation analysis method for LLM training clusters described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the reliability simulation analysis method for LLM training clusters by any other suitable means (e.g., by means of firmware).

[0076] The method includes: acquiring and processing the current LLM training cluster to be simulated and analyzed using a preset multi-dimensional system abstract modeling method to obtain a current standardized simulation model; calculating the joint reliability index of the current standardized model by using a preset basic component FIT calculation method; acquiring at least one type of fault injection event in real time and injecting it into the current standardized simulation model for discrete simulation, monitoring and obtaining the reliability index of the current cluster operating parameters and the current fault propagation path; and processing the joint reliability index of the current standardized model, the reliability index of the current cluster operating parameters, and the current fault propagation path using a preset data statistical analysis method to obtain and feed back a reliability simulation analysis quantitative report corresponding to the current LLM training cluster to be simulated and analyzed.

[0077] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0078] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0079] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0080] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0081] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0082] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0083] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0084] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

[0085] Example 5 Embodiment 5 of the present invention also provides a computer-readable storage medium, wherein the computer-readable instructions, when executed by a computer processor, are used to execute a reliability simulation analysis method for an LLM training cluster. The method includes: acquiring and processing the current LLM training cluster to be simulated using a preset multi-dimensional system abstract modeling method to obtain a current standardized simulation model; calculating the joint reliability index of the current standardized model using a preset basic component FIT calculation method; acquiring at least one type of fault injection event in real time, injecting it into the current standardized simulation model for discrete simulation, and monitoring the reliability index of the current cluster operating parameters and the current fault propagation path; and processing the joint reliability index of the current standardized model, the reliability index of the current cluster operating parameters, and the current fault propagation path using a preset data statistical analysis method to obtain and feed back a quantitative reliability simulation analysis report corresponding to the current LLM training cluster to be simulated.

[0086] Of course, the computer-executable instructions provided in the embodiments of the present invention, which include a computer-readable storage medium, are not limited to the method operations described above, but can also perform related operations in the reliability simulation analysis method for LLM training clusters provided in any embodiment of the present invention.

[0087] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0088] It is worth noting that in the above embodiments of the reliability simulation analysis method for LLM training clusters, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0089] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A reliability simulation analysis method for LLM training clusters, characterized in that, include: The current standardized simulation model is obtained by acquiring and processing the LLM training cluster to be simulated and analyzed using a preset multi-dimensional system abstract modeling method. By using the pre-defined FIT calculation method for basic components, the joint reliability index of the current standardized model is obtained by calculating each basic component in the current standardized simulation model. At least one type of fault injection event is acquired in real time and injected into the current standardized simulation model for discrete simulation, thereby monitoring and obtaining the reliability index of the current cluster operating parameters and the current fault propagation path. Using a pre-defined data statistical analysis method, the joint reliability index of the current standardized model, the reliability index of the current cluster operating parameters, and the current fault propagation path are processed respectively to obtain and feed back a quantitative report of reliability simulation analysis corresponding to the LLM training cluster to be simulated and analyzed.

2. The method of claim 1, wherein, After processing the current standardized model joint reliability index, the current cluster operating parameter reliability index, and the current fault propagation path using a preset data statistical analysis method to obtain and feed back a reliability simulation analysis quantification report corresponding to the LLM training cluster to be simulated and analyzed, the method further includes: The optimization and adjustment results corresponding to the reliability simulation analysis quantification report are received in real time; wherein, the optimization and adjustment results include at least one LLM training cluster optimization and adjustment strategy; The LLM training cluster is adjusted according to the optimization and adjustment strategy of each LLM training cluster, and then the operation of calculating each smallest hardware unit component in the current standardized simulation model by means of the preset basic component FIT calculation method is performed to obtain the joint reliability index of the current standardized model is carried out until the reliability simulation analysis quantification report meets the user's requirements. Then the reliability simulation analysis operation of the current LLM training cluster to be simulated is confirmed to be completed.

3. The method of claim 2, wherein, The process of acquiring and processing the current LLM training cluster to be simulated and analyzed using a preset multi-dimensional system abstract modeling method to obtain the current standardized simulation model includes: The obtained LLM training cluster to be simulated and analyzed is analyzed in multiple dimensions to obtain hardware architecture description parameters, software system description parameters, and system operation characteristic description parameters. Using the multi-dimensional system abstract modeling method, the hardware architecture description parameters, software system description parameters, and system operation characteristic description parameters are simulated to obtain the current standardized simulation model. The current standardized simulation model includes at least one of the following: a device performance threshold unit, a load dynamic change model unit, and a fault anomaly mode library unit.

4. The method of claim 3, wherein, The method of calculating the FIT (Fixed Integrity Index) of basic components in the current standardized simulation model is used to calculate the joint reliability index of the current standardized model, including: In the current standardized simulation model, the basic components of the cluster are defined and configured; wherein, the basic components are the smallest hardware unit components; By using a multi-level abstract modeling method, each basic component is modeled in a multi-level manner, and combined with the basic component FIT calculation method, the joint reliability index of the current standardized model is obtained.

5. The method of claim 4, wherein, The multi-level abstract modeling method includes basic component-level FIT modeling, node and supernode-level modeling, and cluster-level modeling. The method employs a multi-level abstract modeling approach to perform multi-level modeling processing on each basic component, and combines this with the basic component FIT calculation method to obtain the joint reliability index of the current standardized model, including: Using a multi-level abstract modeling method, basic component-level FIT modeling is performed on each basic component to determine the basic component FIT value and the number of basic components deployed at each level. By modeling at the node and supernode levels, based on the FIT value of the basic components and the number of basic components deployed at each level, the node-level FIT value of various basic components is aggregated and calculated using the basic component FIT calculation method. Then, by modeling at the supernode level, the node-level FIT values ​​of all nodes corresponding to any supernode are summed, and the FIT values ​​of the current interconnected components corresponding to the supernode are summed again to obtain the supernode-level FIT value. By summing the FIT values ​​of all supernodes, as well as the FIT values ​​of the northbound network, storage subsystem, and memory pool, the total FIT value of the current cluster is obtained through cluster-level modeling. According to the current cluster-level total FIT value, a current mean time between failures MTBF is calculated by the formula wherein is the current cluster-level total FIT value; To obtain the current average repair time of the cluster, use the formula... This is used to calculate the current availability; where MTTR is the average time to repair the cluster. Obtain the current fault impact parameters and calculate the current performance degradation rate by combining them with the current mean time between failures (MTBF). The current standardized model joint reliability index is determined based on the current mean time between failures, current availability, and current performance degradation rate.

6. The method of claim 5, wherein, At least one type of fault injection event is acquired in real time and injected into the current standardized simulation model for discrete simulation. The reliability indicators of the current cluster operating parameters and the current fault propagation path are monitored and obtained, including: Real-time acquisition of at least one type of fault injection event; A type of target fault injection event is sequentially acquired and injected into the current standardized simulation model for discrete simulation and state monitoring to obtain the reliability index of the current cluster operation parameters and the current fault propagation path, so as to realize the reproduction of LLM training cluster fault scenarios based on injected fault injection events. Determine whether all fault injection events have been traversed. If so, output the current cluster operating parameter reliability index and the current fault propagation path corresponding to each type of fault injection event. If not, return to the previous operation and execute the operation of sequentially obtaining one type of target fault injection event.

7. The method of claim 6, wherein, The process involves using a pre-defined data statistical analysis method to process the joint reliability index of the current standardized model, the reliability index of the current cluster operating parameters, and the current fault propagation path, respectively, to obtain and feed back a quantitative reliability simulation analysis report corresponding to the LLM training cluster to be simulated and analyzed, including: The weighted average calculation sub-method in the data statistical analysis method is used to calculate the weighted average of the current standardized model joint reliability index and the current cluster operation parameter reliability index to obtain the target weighted average joint reliability index. The target weighted average joint reliability index and the current fault propagation path are processed to obtain a quantitative report on reliability simulation analysis corresponding to the LLM training cluster to be simulated and analyzed. The reliability simulation analysis and quantitative report will be provided to users in real time with visual feedback.

8. A reliability simulation analysis device for an LLM training cluster, characterized by, include: The current standardized simulation model determination module is used to acquire and process the LLM training cluster to be simulated and analyzed, and obtain the current standardized simulation model through a preset multi-dimensional system abstract modeling method. The current standardized model joint reliability index determination module is used to calculate the joint reliability index of the current standardized model by using the preset basic component FIT calculation method. The current cluster operating parameter reliability index and current fault propagation path determination module is used to acquire at least one type of fault injection event in real time, inject it into the current standardized simulation model for discrete simulation, and monitor and obtain the current cluster operating parameter reliability index and current fault propagation path. The reliability simulation analysis quantification report determination and feedback module is used to process the joint reliability index of the current standardized model, the reliability index of the current cluster operating parameters, and the current fault propagation path through a preset data statistical analysis method, and obtain and feed back the reliability simulation analysis quantification report corresponding to the LLM training cluster to be simulated and analyzed.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a reliability simulation analysis method for LLM training clusters as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute a reliability simulation analysis method for an LLM training cluster as described in any one of claims 1-7.

Images