Anti-open-circuit optimization method and device for AI computing architecture and medium

Abstract

The invention discloses an anti-open circuit optimization method and device for an AI computing architecture and a medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: monitoring operation state parameters of each link in a current communication path in real time, and dynamically updating the operation state parameters to a congestion lookup table; based on the congestion lookup table, calculating a comprehensive abnormal index and a stability index of the current communication path to predict a link disconnection probability of the current communication path in a future time period; when the disconnection probability exceeds a preset threshold value, screening a minimum risk path in the network topology through a Hamiltonian path analysis method and replacing the minimum risk path; according to the task type of the AI calculation task, calculating a dynamic waterline threshold value of a plurality of links corresponding to the minimum risk path, and dynamically adjusting a flow load waterline threshold value of the plurality of links based on the dynamic waterline threshold value; and when a certain link is congested or disconnected, calculating an optimal flow recovery scheme by using a convex optimization algorithm, and executing recovery.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, and medium for optimizing circuit breaker prevention for AI computing architecture. Background Art

[0002] In artificial intelligence (AI) computing architectures, the stability and throughput of network communications have a crucial impact on overall computing efficiency. AI training and inference tasks typically require multiple computing nodes (such as GPUs, TPUs, or AI accelerator chips) to work together, and data transmission between these computing units relies on high-speed interconnection networks such as InfiniBand, NVLink, or Ethernet. However, with the increasing complexity of AI tasks and the expansion of computing scale, network congestion and link disconnection problems have become increasingly prominent, seriously affecting the throughput and stability of the system. Existing network routing protocols (such as OSPF, BGP, and ECMP) are mainly optimized for traditional data centers and wide area networks. Their disconnection detection and recovery strategies usually rely on passive heartbeat detection or traffic statistics, which exhibit the following limitations in AI computing architectures:

[0003] (1) High detection delay: Relying on periodic detection or passive feedback, it cannot quickly respond to sudden link failures.

[0004] (2) Poor adaptability: Unable to dynamically adjust according to the traffic pattern of AI computing tasks.

[0005] (3) Inaccurate traffic scheduling: Existing load balancing solutions are usually based on static strategies and are difficult to adapt to the dynamic traffic changes of AI tasks.

[0006] Especially in large-scale AI computing architectures (such as NVIDIA DGX server clusters, cloud-based AI training environments, and supercomputing centers), network disconnections can lead to GPU computing synchronization failures (such as obstruction of AllReduce and AllToAll operations), RDMA transmission interruptions, and task delays, seriously affecting the performance of AI computing tasks. Summary of the Invention

[0007] The embodiments of the present application provide a method, device, and medium for optimizing circuit disconnection prevention for an AI computing architecture, to solve the following technical problem: how to efficiently detect circuit disconnections and optimize traffic flow to improve the stability and throughput of the AI ​​computing architecture.

[0008] In the first aspect, an embodiment of the present application provides an anti-circuit-break optimization method for an AI computing architecture, the method comprising: when an AI computing task is executed, real-time monitoring of the operating status parameters of each link in the current communication path, and dynamically updating the operating status parameters to a congestion query table; wherein the operating status parameters include at least traffic, packet loss rate, average delay, and number of congestion markings; based on historical data and real-time data in the congestion query table, calculating the comprehensive anomaly index and stability index of the current communication path, and based on the comprehensive anomaly index and stability index, predicting the link disconnection probability of the current communication path in the future period; when the disconnection probability exceeds a preset threshold, the Hamiltonian path analysis method is used to screen and replace the minimum risk path in the network topology; according to the task type of the AI ​​computing task, the dynamic waterline thresholds of several links corresponding to the minimum risk path are calculated, and the traffic load waterline thresholds of several links are dynamically adjusted based on the dynamic waterline thresholds; when a link is congested or disconnected, the convex optimization algorithm is used to calculate the optimal traffic recovery plan, and recovery is performed.

[0009] In one embodiment of the present application, the calculation of the comprehensive abnormality index is expressed by the following formula:

[0010] Path id,src,dst ={(i1,j1),(i2,j2),…,(i N ,j N )}

[0011]

[0012] Among them, Path id,src,dst A is the actual transmission path of the current communication path id from the source node src to the destination node dst, and each pair of adjacent (i, j) represents the adjacent links in the actual transmission path; id,src,dst (t) is the comprehensive abnormality index of the current communication path id when communicating from the source node src to the destination node dst at time t; w1, w2, w3 are weight coefficients, and w1+w2+w3=1; is the packet loss rate at time t when the current communication path id is communicating from the source node src to the destination node dst; is the preset packet loss rate threshold; is the average round-trip delay from the source node src to the destination node dst at time t for the current communication path id; α is a tolerance adjustment parameter used to relax or tighten the penalty for increased RTT; is the historical average round-trip delay; The number of congestion marks at time t when the current communication path id is communicating from the source node src to the destination node dst; The preset congestion marking threshold.

[0013] In one embodiment of the present application, the calculation of the stability index is represented by the following formula:

[0014]

[0015] Among them, LSI id,src,dst (t) is the stability index of the current communication path id at time t when communicating from the source node src to the destination node dst, which measures the degree of link abnormality. The larger the value, the more serious the abnormality. i,j (t) is the stability index of link (i, j) at time t; A i,j (t) is the comprehensive anomaly index of link (i, j) at time t; λ is the sensitivity adjustment parameter (λ>0): increasing the λ value increases the sensitivity to anomalies, while decreasing the λ value decreases the sensitivity to anomalies; the LSI value ranges from [0, 1], with values ​​closer to 1 indicating a more stable link, and vice versa.

[0016] In one embodiment of the present application, based on the comprehensive anomaly index and stability index, the link disconnection probability of the current communication path in the future time period is predicted, specifically including: reverse weighting the real-time stability index of each link in the current communication path to determine the instability weight of each link; based on the instability weight of each link, calculating the future anomaly index of the current communication path in the future time period; based on the future anomaly index, using a preset Sigmoid function, determining the link disconnection probability.

[0017] In one embodiment of the present application, the minimum risk path is screened in the network topology by using the Hamiltonian path analysis method, specifically including: constructing a network topology subgraph that excludes the main path edge set; using the Hamiltonian path algorithm to traverse all paths and calculate the cumulative disconnection risk value of each path; wherein the cumulative disconnection risk value is the sum of the predicted disconnection probabilities of each link in the path; and selecting the path with the smallest cumulative risk value as the minimum risk path.

[0018] In one embodiment of the present application, the dynamic waterline thresholds of several links corresponding to the minimum risk path are calculated according to the task type of the AI ​​computing task, specifically including: determining whether the communication mode of the AI ​​computing task is AllReduce or AllToAll; if it is an AllReduce task, determining the task type weight coefficient and the flow change adjustment factor value of the AllReduce task type based on the flow change rate and the short-term fluctuation standard deviation; if it is an AllToAll task, determining the task type weight coefficient and the flow change adjustment factor value of the AllToAll task type based on the flow change rate; if it is neither an AllReduce task nor an AllToAll task, querying the preset dynamic waterline threshold calculation mapping table to determine the corresponding task type weight coefficient and the flow change adjustment factor value; based on the task type weight coefficient and the flow change adjustment factor value, calculating the dynamic waterline threshold through the preset dynamic waterline model.

[0019] In one embodiment of the present application, the dynamic waterline model is represented by the following formula:

[0020]

[0021] in, is the dynamic waterline threshold, is the static reference waterline constant, γ task (t) is the task type coefficient at time t, δ load (t) is the flow change adjustment factor;

[0022] Based on the traffic change rate and short-term fluctuation standard deviation, the traffic change adjustment factor value of the AllReduce task type is determined, which is expressed by the following formula:

[0023]

[0024] in, is the traffic change adjustment factor value of the link (i, j) of the AllReduce task type at time t; is the real-time traffic of link (i, j) at time t; is the instantaneous traffic change rate of link (i, j) at time t; σ T(t) is the standard deviation of the flow rate in the observation window at time t; α∈[0.3,0.6], β∈[0.1,0.3] are the sensitivity coefficients;

[0025] The traffic change adjustment factor value of the AllToAll task type is determined based on the traffic change rate, which is expressed by the following formula:

[0026]

[0027] in, is the traffic change adjustment factor value of the link (i, j) of the AllToAll task type at time t; μ∈[0.2,0.4] is the sensitivity coefficient.

[0028] In one embodiment of the present application, a convex optimization algorithm is used to calculate the optimal flow restoration solution, which is expressed by the following formula:

[0029]

[0030] Among them, x p Set of backup paths The load distribution of each path reconfigured in the above example is that each path p corresponds to a unit switching cost c. p .

[0031] In a second aspect, an embodiment of the present application also provides an anti-circuit-breaking optimization device for an AI computing architecture, the device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute an anti-circuit-breaking optimization method for an AI computing architecture such as any one of the above items.

[0032] On the third aspect, an embodiment of the present application also provides a non-volatile computer storage medium for anti-circuit short circuit optimization for AI computing architecture, which stores computer-executable instructions. When the computer-executable instructions are executed, an anti-circuit short circuit optimization method for AI computing architecture such as any of the above items is implemented.

[0033] The embodiments of the present application provide a method, device, and medium for optimizing circuit breaker prevention for AI computing architecture, which have the following beneficial effects:

[0034] 1. Accurate and rapid disconnection detection: Based on congestion lookup tables and time series analysis, it improves the real-time performance of link fault detection and reduces interruptions to AI training and inference tasks caused by network anomalies.

[0035] 2. Optimal traffic allocation: Hamiltonian path analysis is used to calculate the optimal traffic transfer path, and convex optimization methods are used to implement intelligent traffic scheduling, thereby improving GPU interconnection throughput and reducing packet loss.

[0036] 3. Dynamic adaptation to network load: A dynamic waterline adjustment mechanism is used to optimize waterline parameters based on AI task traffic patterns (such as AllReduce and AllToAll) to avoid secondary congestion caused by traffic bursts.

[0037] 4. Improve GPU computing resource utilization: Combine stage-aware scheduling and priority management strategies to optimize bandwidth allocation between AI training tasks and improve the balanced utilization of computing resources.

[0038] 5. Reduce AI computing latency and QoS degradation risks: Predict traffic bottlenecks in advance and perform path switching with minimal impact to ensure stable operation of AI cloud computing tasks.

[0039] 6. Improve RDMA transmission efficiency: Optimize RDMA data flow in AI computing architecture, reduce communication congestion, and improve AI training throughput and inference response speed.

[0040] 7. Overall, the solution of the present invention effectively improves the network stability of the AI ​​computing architecture, reduces the impact of network anomalies on AI training and reasoning tasks, and ensures efficient computing performance and low-latency data transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0042] Figure 1 A flow chart of an anti-circuit-break optimization method for an AI computing architecture provided in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of the internal structure of an anti-circuit-break optimization device for an AI computing architecture provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0045] Currently, most data centers and AI computing clusters use ECMP (Equal-Cost Multi-Path) and centralized traffic scheduling for traffic management. In the ECMP solution, network devices use a hashing algorithm to evenly distribute traffic to multiple equal-cost paths to improve throughput. However, this method cannot dynamically perceive the link status. Once a link fails or becomes congested, the traffic scheduling policy cannot be adjusted in time, which may cause computing tasks to be blocked or large-scale timeouts. In addition, centralized traffic scheduling solutions usually rely on the controller to periodically collect network status and perform traffic scheduling based on historical data, which results in a lag in detection and adjustment.

[0046] Some high-performance computing (HPC) environments use InfiniBand with Adaptive Routing technology, which dynamically adjusts data flow based on link load. However, this approach primarily relies on local load information for decision-making and lacks global awareness, making it difficult to achieve optimal traffic distribution in complex AI computing architectures.

[0047] In addition, some AI data centers and supercomputing environments have introduced RDMA over Converged Ethernet (RoCE) technology, which uses zero-copy transmission to improve data throughput. However, RoCE relies on the stability of data center network traffic and is susceptible to congestion control mechanisms when encountering link disconnection or traffic bursts, resulting in reduced training throughput.

[0048] Although the above technical solutions have optimized the network performance in the AI ​​computing architecture to a certain extent, the following shortcomings still exist:

[0049] (1) High latency in link failure detection: ECMP and adaptive routing solutions rely on passive detection and cannot quickly respond to sudden link failures, which may cause AI training tasks to be interrupted.

[0050] (2) Lack of AI task perception: Existing network scheduling methods are only based on network topology or traffic balancing, and fail to optimize data flow for AI computing tasks (such as AllReduce and AllToAll).

[0051] (3) Load adjustment lag: Centralized traffic scheduling is usually based on historical data and is difficult to adapt to sudden traffic changes in GPU computing architecture, which may lead to unbalanced computing resources.

[0052] (4) Optimization methods are not sophisticated enough: Technologies such as RoCE rely on TCP congestion control and are susceptible to burst traffic, resulting in a decrease in computing throughput. They fail to combine mathematical optimization methods to improve traffic distribution strategies.

[0053] In view of the above problems, the embodiments of the present application provide an anti-circuit-break optimization method, device and medium for AI computing architecture to solve the following technical problems: how to efficiently detect circuit breaks and optimize traffic to improve the stability and throughput of the AI ​​computing architecture.

[0054] The technical solutions proposed in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0055] Figure 1 This is a flow chart of an anti-circuit-breaking optimization method for AI computing architecture provided in an embodiment of the present application. Figure 1 As shown, the embodiment of the present application provides an anti-circuit-breaking optimization method for AI computing architecture, which specifically includes the following steps:

[0056] Step 101: When the AI ​​computing task is executed, the operating status parameters of each link in the current communication path are monitored in real time, and the operating status parameters are dynamically updated to the congestion query table.

[0057] In this embodiment, the operating status parameters at least include traffic flow, packet loss rate, average delay, and congestion marking times.

[0058] Step 102: Based on the historical data and real-time data in the congestion query table, calculate the comprehensive abnormality index and stability index of the current communication path, and based on the comprehensive abnormality index and stability index, predict the link disconnection probability of the current communication path in the future period.

[0059] First, it should be noted that the parameters in the embodiments of the present application are defined as shown in Tables 1 and 2 below. Table 1 defines the variables for the end-to-end complete communication path, and Table 2 defines the variables for each link (i, j) in the communication path:

[0060]

[0061]

[0062] Table 1

[0063]

[0064] Table 2

[0065] In one embodiment of the present application, the calculation of the comprehensive abnormality index is expressed by the following formula:

[0066] Path id,src,dst ={(i1,j1),(i2,j2),…,(i N ,j N )}

[0067]

[0068] Among them, Path id,src,dst A is the actual transmission path of the current communication path id from the source node src to the destination node dst, and each pair of adjacent (i, j) represents the adjacent links in the actual transmission path; id,src,dst (t) is the comprehensive abnormality index of the current communication path id when communicating from the source node src to the destination node dst at time t; w1, w2, w3 are weight coefficients, and w1+w2+w3=1; is the packet loss rate at time t when the current communication path id is communicating from the source node src to the destination node dst; is the preset packet loss rate threshold; is the average round-trip delay from the source node src to the destination node dst at time t for the current communication path id; α is a tolerance adjustment parameter used to relax or tighten the penalty for increased RTT; is the historical average round-trip delay; The number of congestion marks when the current communication path id is communicating from the source node src to the destination node dst at time t; The preset congestion marking threshold.

[0069] In this embodiment, if the calculated A id,src,dst (t)≥1, which means that the path is determined to have entered a potential abnormal state.

[0070] In one embodiment of the present application, the calculation of the stability index is represented by the following formula:

[0071]

[0072] Among them, LSI id,src,dst (t) is the stability index of the current communication path id when communicating from the source node src to the destination node dst at time t, which measures the degree of link abnormality. The larger the value, the more serious the abnormality. i,j (t) is the stability index of link (i, j) at time t; A i,j (t) is the comprehensive anomaly index of link (i, j) at time t; λ is the sensitivity adjustment parameter (λ>0): increasing the λ value increases the sensitivity to anomalies, while decreasing the λ value decreases the sensitivity to anomalies; the LSI value ranges from [0, 1], with values ​​closer to 1 indicating a more stable link, and vice versa.

[0073] In one embodiment of the present application, based on the comprehensive anomaly index and stability index, the link disconnection probability of the current communication path in the future time period is predicted, specifically including: reverse weighting the real-time stability index of each link in the current communication path to determine the instability weight of each link; based on the instability weight of each link, calculating the future anomaly index of the current communication path in the future time period; based on the future anomaly index, using a preset Sigmoid function, determining the link disconnection probability.

[0074] In this embodiment, the actual transmission path corresponding to the end-to-end link (id, src, dst) is:

[0075] Path id,src,dst ={(i1,j1),(i2,j2),…,(i N ,j N )}

[0076] Each pair (i, j) represents an adjacent link between switches. The status of these neighbor links affects the stability of the overall path and the probability of anomalies. The status of neighbor links is used to estimate the future anomaly index:

[0077]

[0078] Weight w i,j (t) Using the reverse weighting method of the link stability index (LSI), unstable neighbors have greater influence:

[0079]

[0080] This definition ensures that, assuming the total weight is 1, links with lower LSI (more unstable) have a greater impact on anomaly estimation.

[0081] Use the Sigmoid function to convert the anomaly index into a probability value in the [0,1] interval:

[0082]

[0083] Among them, A id,src,dst (t+h) is the future anomaly index; τ is the threshold for anomaly judgment, which can usually be set to 1; β is the sensitivity coefficient. The larger the value, the more sensitive it is to abnormal changes.

[0084] Step 103: When the disconnection probability exceeds a preset threshold, the minimum risk path is screened in the network topology by using the Hamiltonian path analysis method and replaced.

[0085] In one embodiment of the present application, the minimum risk path is screened in the network topology by using the Hamiltonian path analysis method, specifically including: constructing a network topology subgraph that excludes the main path edge set; using the Hamiltonian path algorithm to traverse all paths and calculate the cumulative disconnection risk value of each path; wherein the cumulative disconnection risk value is the sum of the predicted disconnection probabilities of each link in the path; and selecting the path with the smallest cumulative risk value as the minimum risk path.

[0086] In this embodiment, in the current communication path The probability of anomaly at future time t+h Exceeding the preset threshold P th The present invention triggers the preselection of alternative routes. The alternative route search is based on the Hamiltonian path principle, excluding the current communication path set. Figure G after ′ =(V,E\E primary ), find the set of Hamiltonian paths from the source node src to the destination node dst The best path that minimizes the following equation is

[0087]

[0088] The total path risk Defined as:

[0089]

[0090] Step 104: Calculate the dynamic watermark thresholds of several links corresponding to the minimum risk path according to the task type of the AI ​​computing task, and dynamically adjust the traffic load watermark thresholds of the several links based on the dynamic watermark thresholds.

[0091] In one embodiment of the present application, the dynamic waterline thresholds of several links corresponding to the minimum risk path are calculated according to the task type of the AI ​​computing task, specifically including: determining whether the communication mode of the AI ​​computing task is AllReduce or AllToAll; if it is an AllReduce task, determining the task type weight coefficient and the flow change adjustment factor value of the AllReduce task type based on the flow change rate and the short-term fluctuation standard deviation; if it is an AllToAll task, determining the task type weight coefficient and the flow change adjustment factor value of the AllToAll task type based on the flow change rate; if it is neither an AllReduce task nor an AllToAll task, querying the preset dynamic waterline threshold calculation mapping table to determine the corresponding task type coefficient and the flow change adjustment factor value; based on the task type coefficient and the flow change adjustment factor value, calculating the dynamic waterline threshold through the preset dynamic waterline model.

[0092] In one embodiment of the present application, the dynamic waterline model is represented by the following formula:

[0093]

[0094] in, is the dynamic waterline threshold, is the static reference waterline constant, γ task (t) is the task type coefficient at time t, δ load (t) is the flow change adjustment factor;

[0095] Based on the traffic change rate and short-term fluctuation standard deviation, the traffic change adjustment factor value of the AllReduce task type is determined, which is expressed by the following formula:

[0096]

[0097] in, is the traffic change adjustment factor value of the link (i, j) of the AllReduce task type at time t; is the real-time traffic of link (i, j) at time t; is the instantaneous traffic change rate of link (i, j) at time t; σ T(t) is the standard deviation of the flow rate in the observation window at time t; α∈[0.3,0.6], β∈[0.1,0.3] are the sensitivity coefficients;

[0098] The traffic change adjustment factor value of the AllToAll task type is determined based on the traffic change rate, which is expressed by the following formula:

[0099]

[0100] in, is the traffic change adjustment factor value of the link (i, j) of the AllToAll task type at time t; μ∈[0.2,0.4] is the sensitivity coefficient.

[0101] In this embodiment, in, is the task type weight coefficient of the link (i, j) of the AllReduce task type at time t, is the task type weight coefficient of the link (i, j) of the AllToAll task type at time t.

[0102] This dynamic watermark adjustment strategy adaptively limits link capacity based on task types and real-time traffic behavior. When the system performs traffic transfer on a backup route, it pre-calibrates the watermark on the alternate path to avoid congestion on the backup link due to miscalculation, achieving more stable and intelligent traffic transfer control.

[0103] Step 105: When a link is congested or disconnected, the convex optimization algorithm is used to calculate the optimal traffic recovery solution and perform the recovery.

[0104] In this embodiment, a convex optimization algorithm is used to calculate the optimal traffic recovery plan to ensure that when a link anomaly is detected in the network, the packet loss rate is too high, or a potential interruption risk is predicted, the present invention initiates an optimal traffic recovery strategy based on a convex optimization model to achieve routing reconfiguration with minimal disturbance. Traditional static rerouting methods can easily cause uneven traffic redistribution, leading to secondary congestion or inter-GPU communication interruptions. However, this invention uses mathematical optimization methods to solve for the transfer configuration within the global minimum impact range, ensuring that the recovery operation minimizes interference with AI training or inference tasks.

[0105] Specifically, let the total load of the original traffic source-destination pair (src, dst) be When the primary path fails, the system will start from the backup path. Reconfigure the load distribution amount x on each path p Each path p corresponds to a unit switching cost c p, then the recovery optimization problem is modeled as follows:

[0106]

[0107] Constraints include:

[0108] 1. Flow conservation:

[0109] 2. Link capacity and waterline limitations:

[0110] 3. Non-negative conditions:

[0111] in, is the dynamic watermark threshold of link (i, j) at that moment, which may change due to the previous watermark adjustment mechanism.

[0112] Furthermore, the present invention integrates the “stage-aware scheduling” mechanism to incorporate the latency sensitivity of GPU tasks into the recovery cost adjustment: for a path with a load-bearing task type of task(p), its latency sensitivity weight is π task(p) ∈[0,1], the adjusted cost is:

[0113]

[0114] Tasks with high sensitivity (such as AllReduce) will have lower transfer costs due to their larger weights, and will be redistributed first in the optimization solution. The final optimization solution is It can be directly applied to switching devices and SDN controllers to dynamically adjust traffic transfer configurations and achieve flexible and task-aware traffic recovery, ensuring uninterrupted operation of AI systems and a more balanced overall network load.

[0115] The congestion query table and traffic optimization scheme of the present invention not only optimizes the network disconnection detection and traffic scheduling in the AI ​​computing architecture, but also broadens its application scope and development potential, especially its integrated application on the SONiC switch product (open source network operating system). First, this scheme can collect link status and queue congestion in real time through the telemetry mechanism of the SONiC switch, and perform intelligent traffic adjustment in combination with the congestion query table to optimize the communication efficiency between GPU computing nodes. Secondly, the present invention can be combined with the traffic scheduling strategy (QoS Management) of SONiC, and through the dynamic waterline adjustment mechanism, the traffic threshold can be adaptively adjusted to avoid secondary congestion in the GPU interconnection network and improve the stability of RDMA transmission. In addition, this scheme can utilize the ECMP load balancing function of SONiC, combined with Hamiltonian path analysis to calculate the optimal traffic path, and dynamically select the optimal link during the data transmission process of the AI ​​computing task to ensure high throughput and low packet loss. Furthermore, the present invention can perform policy deployment through the control plane of SONiC to optimize the traffic scheduling of GPU computing tasks, so that AI training tasks can still maintain efficient execution when the network topology changes. In general, the present invention, through deep integration with the SONiC switch, enables the network layer of the AI ​​computing architecture to have intelligent traffic scheduling capabilities, improve throughput, reduce computing task interruptions, and provide a more stable network environment for AI training and reasoning.

[0116] The above is an embodiment of the method proposed in this application. Based on the same inventive concept, this application embodiment also provides an anti-circuit-breaking optimization device for AI computing architecture, whose structure is as follows: Figure 2 shown.

[0117] Figure 2 This is a schematic diagram of the internal structure of an anti-circuit-breaking optimization device for an AI computing architecture provided in an embodiment of the present application. Figure 2 As shown, the equipment includes:

[0118] at least one processor 201;

[0119] and, a memory 202 communicatively coupled to the at least one processor;

[0120] The memory 202 stores instructions that can be executed by at least one processor, and the instructions are executed by the at least one processor 201 to enable the at least one processor 201 to:

[0121] When executing AI computing tasks, the operating status parameters of each link in the current communication path are monitored in real time, and the operating status parameters are dynamically updated to the congestion query table. The operating status parameters include at least traffic flow, packet loss rate, average delay, and number of congestion markings.

[0122] Based on the historical data and real-time data in the congestion query table, the comprehensive anomaly index and stability index of the current communication path are calculated. Based on the comprehensive anomaly index and stability index, the link disconnection probability of the current communication path in the future period is predicted;

[0123] When the disconnection probability exceeds the preset threshold, the Hamiltonian path analysis method is used to screen the minimum risk path in the network topology and replace it;

[0124] Based on the task type of the AI ​​computing task, the dynamic watermark thresholds of several links corresponding to the minimum risk path are calculated, and the traffic load watermark thresholds of several links are dynamically adjusted based on the dynamic watermark thresholds.

[0125] When a link is congested or disconnected, the convex optimization algorithm is used to calculate the optimal traffic recovery plan and perform the recovery.

[0126] Some embodiments of the present application provide corresponding Figure 1 A non-volatile computer storage medium optimized for circuit breaking prevention of AI computing architecture stores computer executable instructions, wherein the computer executable instructions are configured as follows:

[0127] When executing AI computing tasks, the operating status parameters of each link in the current communication path are monitored in real time, and the operating status parameters are dynamically updated to the congestion query table. The operating status parameters include at least traffic flow, packet loss rate, average delay, and number of congestion markings.

[0128] Based on the historical data and real-time data in the congestion query table, the comprehensive anomaly index and stability index of the current communication path are calculated. Based on the comprehensive anomaly index and stability index, the link disconnection probability of the current communication path in the future period is predicted;

[0129] When the disconnection probability exceeds the preset threshold, the Hamiltonian path analysis method is used to screen the minimum risk path in the network topology and replace it;

[0130] Based on the task type of the AI ​​computing task, the dynamic watermark thresholds of several links corresponding to the minimum risk path are calculated, and the traffic load watermark thresholds of several links are dynamically adjusted based on the dynamic watermark thresholds.

[0131] When a link is congested or disconnected, the convex optimization algorithm is used to calculate the optimal traffic recovery plan and perform the recovery.

[0132] The various embodiments in this application are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences from the other embodiments. In particular, the IoT device and media embodiments are generally similar to the method embodiments, so their description is relatively simple. For relevant portions, refer to the description of the method embodiments.

[0133] The system and medium provided in the embodiments of the present application correspond one-to-one to the method. Therefore, the system and medium also have similar beneficial technical effects to their corresponding methods. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the system and medium will not be repeated here.

[0134] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0135] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0136] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0138] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0139] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0140] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0141] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0142] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A method for optimizing the circuit breaker for AI computing architecture, characterized in that: The method comprises: When executing AI computing tasks, the operating status parameters of each link in the current communication path are monitored in real time, and the operating status parameters are dynamically updated to the congestion query table; wherein the operating status parameters include at least traffic flow, packet loss rate, average delay, and number of congestion markings; Calculating a comprehensive anomaly index and a stability index of the current communication path based on historical data and real-time data in the congestion query table, and predicting a link disconnection probability of the current communication path in a future period based on the comprehensive anomaly index and the stability index; When the disconnection probability exceeds a preset threshold, the minimum risk path is screened in the network topology by Hamiltonian path analysis and replaced; Calculate dynamic watermark thresholds of several links corresponding to the minimum risk path according to the task type of the AI ​​computing task, and dynamically adjust the traffic load watermark thresholds of the several links based on the dynamic watermark thresholds; When a link is congested or disconnected, the convex optimization algorithm is used to calculate the optimal traffic recovery plan and perform the recovery.

2. The anti-circuit-breaking optimization method for AI computing architecture according to claim 1, characterized in that: The calculation of the comprehensive abnormality index is expressed by the following formula: Path id,src,dst ={(i1,j1),(i2,j2),…,(i N ,j N )} Among them, Path id,src,dst A is the actual transmission path of the current communication path id from the source node src to the destination node dst, and each pair of adjacent (i, j) represents the adjacent links in the actual transmission path; id,src,dst (t) is the comprehensive abnormality index of the current communication path id when communicating from the source node src to the destination node dst at time t; w1, w2, w3 are weight coefficients, and w1+w2+w3=1; is the packet loss rate at time t when the current communication path id is communicating from the source node src to the destination node dst; is the preset packet loss rate threshold; is the average round-trip delay from the source node src to the destination node dst at time t for the current communication path id; α is a tolerance adjustment parameter used to relax or tighten the penalty for increased RTT; is the historical average round-trip delay; The number of congestion marks when the current communication path id is communicating from the source node src to the destination node dst at time t; The preset congestion marking threshold.

3. The anti-circuit-breaking optimization method for AI computing architecture according to claim 2, characterized in that: The calculation of the stability index is expressed by the following formula: Among them, LSI id,src,dst (t) is the stability index of the current communication path id at time t when communicating from the source node src to the destination node dst, which measures the degree of link abnormality. The larger the value, the more serious the abnormality. i,j (t) is the stability index of link (i, j) at time t; A i,j (t) is the comprehensive anomaly index of link (i, j) at time t; λ is the sensitivity adjustment parameter (λ>0): increasing the λ value increases the sensitivity to anomalies, while decreasing the λ value decreases the sensitivity to anomalies; the LSI value ranges from [0, 1], with values ​​closer to 1 indicating a more stable link, and vice versa.

4. The anti-circuit-breaking optimization method for AI computing architecture according to claim 3, characterized in that: Predicting the link disconnection probability of the current communication path in a future time period based on the comprehensive anomaly index and the stability index, specifically including: Inversely weighting the real-time stability index of each link in the current communication path to determine an instability weight of each link; Calculating a future anomaly index of the current communication path in a future period based on the instability weight of each link; Based on the future anomaly index, the link disconnection probability is determined using a preset Sigmoid function.

5. The anti-circuit-breaking optimization method for AI computing architecture according to claim 1, characterized in that: The Hamiltonian path analysis method is used to screen the minimum risk path in the network topology, including: Construct a network topology subgraph excluding the main path edge set; Using the Hamiltonian path algorithm to traverse all paths, the cumulative disconnection risk value of each path is calculated; wherein the cumulative disconnection risk value is the sum of the predicted disconnection probabilities of each link in the path; The path with the smallest cumulative risk value is selected as the minimum risk path.

6. The anti-circuit-breaking optimization method for AI computing architecture according to claim 1, characterized in that: Calculating the dynamic waterline thresholds of several links corresponding to the minimum risk path according to the task type of the AI ​​computing task, specifically including: Determine whether the communication mode of the AI ​​computing task is AllReduce or AllToAll; If it is an AllReduce task, determine the task type weight coefficient and traffic change adjustment factor value of the AllReduce task type based on the traffic change rate and short-term fluctuation standard deviation; If it is an AllToAll task, determine the task type weight coefficient and traffic change adjustment factor value of the AllToAll task type based on the traffic change rate; If it is neither an AllReduce task nor an AllToAll task, query the preset dynamic watermark threshold calculation mapping table to determine the corresponding task type weight coefficient and traffic change adjustment factor value; Based on the task type weight coefficient and the flow change adjustment factor value, the dynamic waterline threshold is calculated through a preset dynamic waterline model.

7. The anti-circuit-breaking optimization method for AI computing architecture according to claim 6, characterized in that: The dynamic waterline model is expressed by the following formula: in, is the dynamic waterline threshold, is the static reference waterline constant, γ task (t) is the task type coefficient at time t, δ load (t) is the flow change adjustment factor; Based on the traffic change rate and short-term fluctuation standard deviation, the traffic change adjustment factor value of the AllReduce task type is determined, which is expressed by the following formula: in, is the traffic change adjustment factor value of the link (i, j) of the AllReduce task type at time t; is the real-time traffic of link (i, j) at time t; is the instantaneous traffic change rate of link (i, j) at time t; σ T(t) is the standard deviation of the flow rate in the observation window at time t; α∈[0.3,0.6], β∈[0.1,0.3] are the sensitivity coefficients; The traffic change adjustment factor value of the AllToAll task type is determined based on the traffic change rate, which is expressed by the following formula: in, is the traffic change adjustment factor value of the link (i, j) of the AllToAll task type at time t; μ∈[0.2,0.4] is the sensitivity coefficient.

8. The anti-circuit-breaking optimization method for AI computing architecture according to claim 1, characterized in that: The convex optimization algorithm is used to calculate the optimal flow restoration solution, which is expressed by the following formula: Among them, x p Set of backup paths The load distribution of each path reconfigured in the above example is that each path p corresponds to a unit switching cost c. p .

9. A circuit breaker optimization device for AI computing architecture, characterized in that: The device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; In which, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute an anti-circuit breaking optimization method for an AI computing architecture as described in any one of claims 1-8.

10. A non-volatile computer storage medium optimized for circuit breaker prevention in an AI computing architecture, storing computer-executable instructions, characterized in that: When the computer-executable instructions are executed, an anti-circuit-breaking optimization method for an AI computing architecture as described in any one of claims 1 to 8 is implemented.

Images