Intelligent positioning and self-recovery system for fault of supercomputing center computing power cluster

Abstract

The application discloses a kind of intelligent positioning and self-healing system of fault of intelligent calculation center computing power cluster, including computing node and edge programmable switch, intelligent network card acquires computing node hardware state derivative and quantization, encapsulation into service message execution in-band piggyback transmission, edge programmable switch extracts state feature in data plane and constructs feature matrix, and based on local network node data, hardware comparison is carried out to diagnose fault node;After diagnosis, intercept fault flow, and issue customized congestion notification message to source computing node, which carries hot standby node network parameters and remote memory access key value, after receiving the message by source intelligent network card, the fault node parameters in context are covered using direct memory access mechanism on the original queue, and the packet sequence number state machine pointer is kept continuous.This application realizes data plane fault diagnosis and fast hot switching of underlying communication endpoint, reduces the perception degree of upper application to node fault, and improves the continuity and reliability of computing power cluster data interaction.

Description

Technical Field

[0001] This invention relates to the field of intelligent computing center network communication technology, specifically to an intelligent fault location and self-healing system for intelligent computing center computing power clusters. Background Technology

[0002] With the development of artificial intelligence training and high-performance computing, the scale of intelligent computing center computing clusters continues to expand. In order to meet the communication bandwidth requirements of distributed computing, intelligent computing center clusters generally adopt remote direct memory access protocol based on converged Ethernet to build the underlying network. Computing nodes perform high-frequency data interaction through network cards and switches. The stable operation of the underlying hardware of the nodes and the communication links is the foundation for ensuring the overall computing efficiency of the cluster.

[0003] In the existing intelligent computing center operation and maintenance architecture, network status monitoring and fault isolation usually rely on an independent out-of-band management network and a centralized control platform. The hardware operating parameters of the computing nodes are periodically collected by the edge management software and uploaded to the centralized controller through an independent management link. The centralized controller summarizes the global status data and performs software calculation and analysis. When it is determined that a node has failed or the network is congested, the centralized controller then sends traffic isolation commands and routing switching configurations to downstream switches and computing nodes through the management network to restore the underlying data network communication.

[0004] Existing technologies rely on independent out-of-band management networks and centralized control platforms to handle faults, which suffers from delays in acquiring and processing status data. Reporting status data through independent out-of-band management networks introduces physical communication delays, and centralized controllers face software inference bottlenecks when dealing with concurrently uploaded status data from a large number of nodes. This results in a time lag between the occurrence of hardware anomalies at underlying nodes and the control platform completing analysis and issuing instructions. This out-of-band network-based monitoring mechanism cannot achieve real-time synchronization of underlying hardware status. In high-speed data interaction environments, these acquisition and processing delays can cause data backlogs to faulty nodes, leading to congestion in local network links. Consequently, the upper-layer distributed computing framework may be interrupted due to communication timeouts, failing to guarantee the continuity of data interaction tasks within the computing power cluster. Summary of the Invention

[0005] To address the problems mentioned in the background, this invention provides a fault intelligent location and self-healing system for a computing power cluster in an intelligent computing center. The system is deployed in the intelligent computing center cluster and includes multiple computing nodes and an edge programmable switch. The multiple computing nodes are connected to the edge programmable switch via optical communication links.

[0006] The computing node is used to provide data processing computing power. The computing node is equipped with a central processing unit, a graphics processing unit and a smart network card. The smart network card is used to process network data packets entering and leaving the computing node. The smart network card is equipped with a hardware micro-state acquisition module, an in-band feature encoding module, a context hot overwrite module and an on-chip static random access memory.

[0007] The edge programmable switch is used to provide line-speed forwarding of data packets and perform logical operations on the network data plane. The edge programmable switch is equipped with a state feature matrix maintenance module, a distributed consensus positioning module, a flow control isolation module, and a customized signaling generation module.

[0008] The hardware micro-state acquisition module is used to bypass and monitor the internal bus transaction latency and egress queue depth of the computing node, and calculate the rate of change of the internal bus transaction latency and egress queue depth based on a preset time sliding window to generate hardware state derivatives.

[0009] The in-band feature encoding module is used to quantize the hardware state derivative into state feature values ​​and encapsulate the state feature values ​​into the extended header of the service data packets flowing out of the smart network card.

[0010] The state feature matrix maintenance module is used to parse the extended header of the service data packet, extract the state feature values, write the state feature values ​​into the on-chip static random access memory inside the edge programmable switch, and update the state feature matrix corresponding to multiple data streams.

[0011] The distributed consensus positioning module is used to read the state feature values ​​of each node recorded in the state feature matrix and perform logical comparison operations. When the operation results of multiple data streams meet the preset trigger threshold, the target computing node that generates abnormal state feature values ​​is determined to be a faulty node.

[0012] The flow control isolation module is used to generate forwarding action rules on the data plane of the edge programmable switch to intercept subsequent service data packets whose destination address is a faulty node, extract the packet sequence number of the intercepted service data packet, construct a reverse acknowledgment message based on the packet sequence number and send it to the source computing node that sent the intercepted service data packet, so as to release the local sending buffer space.

[0013] The customized signaling generation module is used to query the pre-configured cluster hot standby routing table inside the edge programmable switch to obtain the corresponding hot standby node network parameters, construct a customized congestion notification message and write the hot standby node network parameters into it, and send it to the source computing node that has a communication connection with the faulty node.

[0014] The context hot overwrite module is used to receive the customized congestion notification message, read the queue pair context stored inside the smart network card through the direct memory access mechanism, overwrite the original fault node network parameters in the queue pair context with the hot standby node network parameters, and keep the packet sequence number state machine pointer in the queue pair context unchanged.

[0015] Preferably, the hardware micro-state acquisition module is integrated into the hardware circuit of the smart network card, and includes a delay sampling unit, a queue listening unit, a sliding window register group, and a differential operation unit.

[0016] The delay sampling unit is used to connect to the internal PCIe bus interface of the computing node, extract the sending timestamp and receiving timestamp of the bus transaction, and calculate the difference between the sending timestamp and receiving timestamp to obtain the current bus transaction delay.

[0017] The queue monitoring unit is used to connect to the data link layer outgoing channel of the smart network card and read the read / write pointer difference of the outgoing queue to obtain the current outgoing queue depth.

[0018] The sliding window register group is used to receive the system clock pulse signal, latch the current bus transaction delay and the current exit queue depth of the current sampling period, and retain the historical bus transaction delay and historical exit queue depth of the previous sampling period.

[0019] The differential operation unit is used to perform discrete differential operations using a hardware subtractor to obtain the exit queue depth difference and the bus transaction latency difference. An internal hardware multiplier multiplies the exit queue depth difference and the bus transaction latency difference by preset weight normalization coefficients to obtain an accumulated sum. A shift register is then used to shift the accumulated sum to the right. The logical operations on the bits output the derivative of the hardware state, and the calculation formula is as follows:

[0020]

[0021] in, Indicates the first Hardware state derivative for each sampling period; Indicates the first The difference in exit queue depth over each sampling period; Indicates the first The bus transaction latency difference over each sampling period; and These are the preset weight normalization coefficients; This represents the number of clock cycles within the sliding time window. It is a positive integer.

[0022] Preferably, the in-band feature coding module includes a nonlinear quantization unit, a message header parsing unit, and a feature field rewriting unit.

[0023] The nonlinear quantization unit is used to receive the hardware state derivative, perform data interval matching on the hardware state derivative using the internal hardware comparator array, and map the multi-byte high-width hardware state derivative into low-bit-width feature values ​​based on the built-in segmented quantization function.

[0024] The message header parsing unit is used to read the service data message entering the data transmission pipeline, parse the network layer and transport layer headers to identify the protocol type flag, and locate the basic transport header inside the service data message when it conforms to the Remote Direct Memory Access Protocol standard based on Converged Ethernet.

[0025] The feature field rewriting unit is used to directly overwrite the feature value into the reserved bits of the basic transmission header through hardware bitmasking operations, and recalculate the cyclic redundancy check code of the service data packet at the data link layer and replace the original check bits.

[0026] Preferably, the state feature matrix maintenance module includes a message parsing and feature extraction unit, a flow table matching and hash mapping unit, an on-chip matrix storage unit, and a time-sliding update unit.

[0027] The message parsing and feature extraction unit is used to parse the basic transmission header of the service data message, extract the status feature values ​​carried in the reserved bits, and extract the source Internet Protocol address and destination Internet Protocol address of the service data message.

[0028] The flow table matching and hash mapping unit is used to perform hash operations on the source Internet Protocol address and the destination Internet Protocol address to generate a flow identifier, and use the flow identifier as the index address of the data plane memory;

[0029] The on-chip matrix storage unit is used to address the rows of the hardware register array according to the stream identifier, write the state feature values ​​into the column where the current time slice is located, and construct the state feature matrix;

[0030] The time-sliding update unit is used to trigger a block shift operation based on the feature data bit width of the hardware register array according to the timestamp pulse, clear historical state feature values ​​that exceed the preset time window, and latch the latest state feature value at the first position of the register.

[0031] Preferably, the distributed consensus positioning module includes a multi-stream feature reading unit, a confidence calculation unit, and a threshold diagnosis unit.

[0032] The multi-stream feature reading unit is used to read the state feature matrix through hardware addressing instructions, extract the state feature values ​​of multiple service data streams whose destination addresses point to the same destination computing node, and extract the state feature values ​​of other adjacent computing nodes other than the destination computing node.

[0033] The confidence calculation unit is used to perform cumulative operations on the state feature values ​​of multiple service data streams using a hardware adder tree to obtain the sum of congestion features, and to perform cumulative operations on the state feature values ​​of other adjacent computing nodes using a hardware adder tree to obtain the sum of background features. It then reads a preset inverse multiplication table, multiplies the sum of background features by the inverse parameter of the total number of adjacent computing nodes to obtain the average background feature value, and uses a hardware subtractor to subtract the average background feature value from the sum of congestion features to output an anomaly consensus score. The formula for the calculation is as follows:

[0034]

[0035] in, Indicates the destination compute node Abnormal consensus score; This indicates that the destination address points to the destination compute node. The total number of active business data streams; Indicates the first The status characteristic values ​​carried by each active business data stream; This indicates that the edge programmable switch is directly connected to the destination computing node, excluding the destination computing node. The total number of adjacent computing nodes other than; Indicates the first The state characteristic values ​​carried by each adjacent computing node; Represents the total number of adjacent computing nodes. The preset reciprocal parameter;

[0036] The threshold confirmation unit uses a hardware comparator to compare the anomaly consensus score with a trigger threshold; when the score exceeds the trigger threshold, it confirms the target computing node. It is the epicenter node of the fault in the physical network.

[0037] Preferably, the flow control isolation module includes an action rule generation unit and a traffic interception buffer unit.

[0038] The action rule generation unit is used to receive the confirmed signal to obtain the network protocol address of the faulty node, and write the interception rule into the matching action flow table maintained by the underlying three-state content addressing memory. The matching field of the interception rule is the network protocol address of the faulty node, and the execution action field is redirection to the isolation queue.

[0039] The traffic interception buffer unit is used to extract the destination network protocol address of subsequent service data packets entering the data forwarding pipeline. When it matches the network protocol address of the faulty node, it triggers the interception rule and transfers the intercepted service data packets to the on-chip isolated buffer queue divided by static random access memory inside the edge programmable switch.

[0040] Preferably, the flow control isolation module further includes a message parameter extraction unit and a message forgery confirmation unit;

[0041] The message parameter extraction unit is used to read the service data messages intercepted in the on-chip isolation buffer queue, parse the transport layer header, and extract the packet sequence number, source Internet Protocol address, destination Internet Protocol address, source queue pair number, and destination queue pair number.

[0042] The acknowledgment message forgery unit is used to invoke the underlying message generation pipeline, swap the source and destination Internet Protocol addresses (IPAs) and write them into the IP header of the newly generated message, swap the source and destination queue pair numbers and write them into the base transport header of the newly generated message, set the opcode field of the base transport header of the newly generated message to the acknowledgment opcode, and synchronously write the extracted packet sequence number into the packet sequence number field of the base transport header of the newly generated message to generate a reverse acknowledgment message.

[0043] Preferably, the customized signaling generation module includes a parameter query unit, a signaling encapsulation unit, and a signaling distribution unit.

[0044] The parameter query unit is used to receive the original network protocol address of the faulty node, and to perform hardware retrieval operation in the static random access memory with the cluster hot standby mapping table pre-programmed, using the original network protocol address to obtain the corresponding hot standby node target medium access control address, target Internet Protocol address, target queue pair number and target remote memory access key value.

[0045] The signaling encapsulation unit is used to generate the header basic structure of the standard congestion notification message. After the acknowledgment extended transmission header, a predefined format vendor extended header is appended. The target medium access control address, target Internet Protocol address, target queue pair number, and target remote memory access key value of the hot standby node are written into the vendor extended header in bit order. Specific reserved flag bits are modified to generate a customized congestion notification message.

[0046] The signaling distribution unit is used to extract the source network protocol addresses of all source computing nodes whose destination addresses point to the faulty node from the active connection tracking table, and use the hardware message replication engine to replicate multiple independent unicast signaling messages and send them to the corresponding source computing nodes respectively.

[0047] Preferably, the context hot overwrite module includes a signaling parsing unit, a memory location unit, and a status overwrite unit.

[0048] The signaling parsing unit is used to parse the basic transmission header of the received customized congestion notification message to identify specific reserved identifier bits, and extract the target medium access control address, target Internet Protocol address, target queue pair number and target remote memory access key value of the hot standby node carried in the vendor extended header.

[0049] The memory positioning unit is used to extract the original destination Internet Protocol address and the original queue pair number to generate a joint index, and use the joint index to generate a direct memory access instruction to address the queue pair context mapping table in the static random access memory of the smart network card, and locate the original queue pair context storage block corresponding to the current communication connection.

[0050] The state overwrite unit is used to physically overwrite the original fault node network parameters in the original queue pair context storage block in the on-chip static random access memory using the extracted parameters, and locks the packet sequence number state machine pointer and the expected receive sequence number storage bit in the original queue pair context storage block during the overwrite process.

[0051] Preferably, after the state overwrite unit completes the network parameter overwrite, the data transmission engine of the smart network card is used to read the updated queue pair context vector to encapsulate the subsequent service data packets, and write the target medium access control address, target Internet Protocol address, target queue pair number and target remote memory access key value of the hot standby node into the protocol layer header corresponding to the subsequent service data packets, so that the subsequent service data packets are addressed and transmitted to the hot standby node along the physical network link.

[0052] The present invention, by adopting the above technical solution, can bring the following beneficial effects:

[0053] 1. This invention generates hardware state derivatives by bypassing the monitoring of the bus transaction latency and egress queue depth of the computing node inside the smart network interface card (NIC). These derivatives are then quantized into feature values ​​using an in-band feature encoding module and encapsulated into the extended header of the service data packet for piggyback transmission. This achieves synchronization of the underlying hardware operating status without occupying independent network bandwidth resources, eliminating the acquisition delay problem caused by the reporting of status data from traditional out-of-band management networks. This provides high-timeliness feature data support for subsequent network data plane fault location.

[0054] 2. This invention extracts feature values ​​of multiple service flows from the data plane pipeline of an edge programmable switch and stores them in an on-chip static random access memory to construct a feature matrix. At the same time, it utilizes a distributed consensus positioning module to perform hardware cross-comparison calculations and trigger threshold determination based on data from adjacent nodes within the local network. This achieves a closed loop of fault node diagnosis logic at the network's underlying hardware plane, avoiding the communication overhead caused by uploading monitoring information to a centralized controller for software inference, and shortening the response time for network layer diagnosis and isolation.

[0055] 3. This invention sends customized congestion notification messages carrying hot standby node network parameters and remote memory access keys to the source computing node. The source smart network card uses direct memory access mechanism to overwrite the faulty node parameters in the original queue context in physical storage, while maintaining the continuity of packet sequence number state machine pointers. This achieves fast switching of communication endpoints at the underlying hardware level, significantly reducing the perception of local hardware failures by upper-layer applications and shortening fault recovery time. This improves the continuity and reliability of data interaction tasks in computing clusters under node failure scenarios. Attached Figure Description

[0056] Figure 1 This is a system architecture diagram of the present invention;

[0057] Figure 2 This is a flowchart illustrating the overall closed-loop workflow of the fault self-healing mechanism of this invention.

[0058] Figure 3 This is a logical structure diagram of the hardware micro-state acquisition module of the present invention;

[0059] Figure 4 This is a diagram illustrating the state feature quantization and in-band piggyback logic processing of the present invention.

[0060] Figure 5 This is a logic diagram for constructing and maintaining the state feature matrix of this invention;

[0061] Figure 6 This is the consensus positioning operation logic diagram of the present invention;

[0062] Figure 7 This is a logic diagram for intercepting and isolating data plane fault traffic in this invention;

[0063] Figure 8 This invention provides a reverse verification of a forged logic diagram;

[0064] Figure 9 This is a logic diagram for the customized notification signaling construction and distribution of the present invention;

[0065] Figure 10 This is a logic diagram of hot overwrite of network interface card context and state machine preservation in this invention. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] Example 1

[0068] Please refer to the appendix. Figure 1 The present invention provides a fault intelligent location and self-healing system for a computing power cluster of an intelligent computing center, wherein the system is deployed in the intelligent computing center cluster 10;

[0069] The intelligent computing center cluster 10 includes multiple computing nodes 100 and edge programmable switches 200. The multiple computing nodes 100 are connected to the edge programmable switches 200 through optical communication links. The multiple computing nodes 100 and the edge programmable switches 200 form a non-blocking network topology at the physical layer, which meets the communication bandwidth requirements of large-scale distributed computing within the intelligent computing center cluster 10.

[0070] The computing node 100 is used to provide data processing computing power. The computing node 100 is equipped with a central processing unit 101, a graphics processing unit 102, and a smart network card 110. The central processing unit 101 and the graphics processing unit 102 interact with the smart network card 110 through the high-speed PCIe bus inside the computing node 100. The smart network card 110 directly reads the hardware communication status of the computing node 100 externally, avoiding occupying the computing power resources of the central processing unit 101. The smart network card 110 is used to process the network data packets entering and leaving the computing node 100. The smart network card 110 is equipped with a hardware micro-state acquisition module 111, an in-band feature encoding module 112, and a context hot overwrite module 113 to realize the seamless acquisition of the hardware status on the end side and the low-level overwrite of the communication state machine.

[0071] The edge programmable switch 200 provides line-rate forwarding of data packets and performs logical operations on the network data plane. Internally, the edge programmable switch 200 is equipped with on-chip static random access memory as the underlying storage medium, supporting nanosecond-level data feature query and matrix writing operations. The edge programmable switch 200 internally includes a state feature matrix maintenance module 201, a distributed consensus positioning module 202, a flow control isolation module 203, and a customized signaling generation module 204. These modules are fully deployed in the data plane pipeline of the edge programmable switch 200, avoiding time delays caused by control plane interactions and ensuring microsecond-level response for fault isolation.

[0072] Example 2

[0073] Please refer to the appendix. Figure 2 Based on Embodiment 1 and the aforementioned system, this invention provides a fault location and self-healing method based on in-band state derivatives and RDMA hardware context hot-swapping, comprising the following operational steps:

[0074] In step S100, the hardware micro-state acquisition module 111 inside the smart network card 110 bypasses and monitors the internal bus transaction delay and egress queue depth of the computing node 100. The hardware micro-state acquisition module 111 calculates the rate of change of the internal bus transaction delay and egress queue depth based on a preset time sliding window, and generates the hardware state derivative. The formula for calculating the hardware state derivative is as follows:

[0075]

[0076] in, This represents the hardware state derivative at the current time t; Indicates the time interval of the time sliding window; Indicates in The change in the egress queue depth of smart network interface card 110 over a period of time; This represents the change in bus transaction latency within node 100 over a time interval Δt. and These are the preset weight normalization coefficients;

[0077] In the hardware implementation, the above continuous-time formula is discretized with the system clock period as the smallest granularity; assuming the sliding time window contains... For each clock cycle, the equivalent hardware discrete calculation formula is:

[0078] In step S200, the in-band feature encoding module 112 quantizes the hardware state derivative into state feature values, and encapsulates the state feature values ​​into the extended header of the service data packet flowing out of the smart network card 110. The operation of encapsulating the state feature values ​​into the service data packet by the in-band feature encoding module 112 realizes the in-band compressed transmission of hardware state feature data, avoiding additional network management communication overhead.

[0079] In step S300, the service data packet with status feature values ​​is transmitted to the edge programmable switch 200. The status feature matrix maintenance module 201 parses the extended header of the service data packet, extracts the status feature values, and writes the status feature values ​​into the on-chip static random access memory of the edge programmable switch 200 to update the status feature matrix corresponding to multiple data streams.

[0080] In step S400, the distributed consensus positioning module 202 reads the state feature values ​​of each node recorded in the state feature matrix and performs logical comparison operations. When the operation results of multiple data streams meet the preset trigger threshold, the distributed consensus positioning module 202 determines that the target computing node that generates the abnormal state feature value is the fault node. The distributed consensus positioning module 202 performs logical comparison operations inside the edge programmable switch 200 and deploys the fault determination logic on the network data plane, effectively shortening the positioning time of the fault node.

[0081] Step S500: After the fault node is determined, the flow control isolation module 203 generates forwarding action rules on the data plane of the edge programmable switch 200. The flow control isolation module 203 intercepts subsequent service data packets whose destination address is the fault node according to the forwarding action rules.

[0082] In step S600, the flow control isolation module 203 extracts the packet sequence number of the intercepted service data packet, constructs a reverse confirmation message based on the packet sequence number, and sends the reverse confirmation message to the source computing node 100 that sent the intercepted service data packet. The source computing node 100 receives the reverse confirmation message and releases its local sending buffer space. The operation of the source computing node 100 to release its local sending buffer space eliminates the backlog of the sending queue of the source computing node 100, and suppresses the generation and reverse propagation of priority-based flow control (PFC) pause frames from the source.

[0083] In step S700, the customized signaling generation module 204 queries the pre-configured cluster hot standby routing table inside the edge programmable switch 200 to obtain the corresponding hot standby node network parameters. The customized signaling generation module 204 constructs a customized congestion notification message and writes the hot standby node network parameters into the customized congestion notification message. The customized signaling generation module 204 sends the customized congestion notification message to the source computing node 100 that has a communication connection with the faulty node.

[0084] In step S800, the context hot overwrite module 113 receives the customized congestion notification message and reads the queue pair context stored internally by the smart network card 110 through the direct memory access mechanism. The context hot overwrite module 113 uses the hot standby node network parameters to overwrite the original fault node network parameters in the queue pair context, while keeping the packet sequence number state machine pointer in the queue pair context unchanged. The source computing node 100 uses the updated queue pair context to send subsequent service data packets. The subsequent service data packets are addressed and transmitted to the hot standby node based on the updated hot standby node network parameters. The operation of the context hot overwrite module 113 overwriting the original fault node network parameters and keeping the packet sequence number state machine pointer unchanged completes the hot replacement of the communication state machine at the underlying hardware layer, maintaining the continuity of the upper layer service connection.

[0085] Example 3

[0086] Please refer to the appendix. Figure 3 Based on Embodiment 1 and Embodiment 2, step S100 further explains the process of collecting and calculating hardware state derivatives.

[0087] The hardware micro-state acquisition module 111 is integrated into the hardware circuit of the smart network card 110. It includes a delay sampling unit 1111, a queue listening unit 1112, a sliding window register group 1113, and a differential operation unit 1114.

[0088] The delay sampling unit 1111 collects the sending timestamp of the DMA bus transaction initiated by the smart network card 110 and the corresponding completion message receiving timestamp through the physical link pin between the smart network card 110 and the internal PCIe bus interface of the computing node 100, and calculates the difference between the two to obtain the current bus transaction delay; or, the delay sampling unit 1111 obtains the round-trip delay statistics of the current internal PCIe link of the computing node 100 by reading the PCIe performance monitoring unit register exposed in the central processing unit 101 or the root complex, and uses it as the current bus transaction delay; the above hardware sampling method avoids the processing delay caused by the central processing unit 101 participating in software status polling, and transmits the current bus transaction delay to the sliding window register group 1113;

[0089] The queue listening unit 1112 is connected to the data link layer outgoing channel of the smart network card 110. It reads the difference between the read and write pointers of the outgoing queue to obtain the current outgoing queue depth and transmits the current outgoing queue depth synchronously to the sliding window register group 1113. The mechanism of the queue listening unit 1112 directly reading the register pointer realizes zero-overhead real-time tracking of network transmission congestion status.

[0090] The sliding window register group 1113 receives the system clock pulse signal, latches the current bus transaction delay and the current exit queue depth of the current sampling period according to the system clock pulse signal, and retains the historical bus transaction delay and historical exit queue depth of the previous sampling period.

[0091] The differential operation unit 1114 is connected to the sliding window register group 1113. It uses a hardware subtractor to perform discrete differential operations, obtaining the exit queue depth difference and the bus transaction delay difference, thereby converting the simulated physical change trend into discrete digital characteristic parameters. The differential operation unit 1114 multiplies the exit queue depth difference and the bus transaction delay difference by preset weight normalization coefficients using an internal hardware multiplier to obtain the sum. To replace the large-area floating-point divider circuit with hardware logic gates, the clock cycle number of the sliding time window is set to... The difference operation unit 1114 uses a shift register to perform a right shift on the accumulated sum. The bitwise logical operations directly output the hardware state derivative. The hardware discrete calculation formula executed by the differential operation unit 1114 is as follows:

[0092]

[0093] in, This represents the hardware state derivative for the k-th sampling period; Indicates the first The difference in exit queue depth over each sampling period; Indicates the first The bus transaction latency difference over each sampling period; and These are the preset weight normalization coefficients; This represents the number of clock cycles within the sliding time window. It is a positive integer;

[0094] The differential operation unit 1114 uses hardware gate circuits to perform subtraction, multiplication and shift operations. It outputs hardware state derivatives based entirely on the logic gates at the bottom layer of the hardware physical layer, completely eliminating the nanosecond-level instruction cycle loss caused by relying on software threads to perform division operations.

[0095] Furthermore, after generating the hardware state derivative in step S100, the in-band feature encoding module 112 performs state feature quantization and in-band piggybacking of the message in step S200. Please refer to the appendix. Figure 4 The in-band feature coding module 112 is integrated into the smart network card 110, and its internal components include a non-linear quantization unit 1121, a packet header parsing unit 1122, and a feature field rewriting unit 1123.

[0096] The nonlinear quantization unit 1121 receives the hardware state derivative output by the differential operation unit 1114, and performs data interval matching on the hardware state derivative using its internal hardware comparator array. The nonlinear quantization unit 1121 has a built-in piecewise quantization function, which maps the multi-byte high-bit-width hardware state derivative to low-bit-width feature values. The mapping formula of the piecewise quantization function is as follows:

[0097]

[0098] in, Indicates the first The characteristic values ​​output in each sampling period; Indicates the first The hardware state derivative of the input for each sampling period; , as well as The preset step threshold is used in the internal hardware comparator array of the nonlinear quantization unit 1121.

[0099] The nonlinear quantization unit 1121 performs the above mapping operation, compressing the continuously fluctuating hardware state derivative into a discrete feature value that occupies only 2 bits, which significantly reduces the effective payload generated by the state data attached to the service data packet and releases the available bandwidth for service data transmission.

[0100] The header parsing unit 1122 is connected to the data transmission pipeline of the smart network card 110. It reads the service data packets entering the data transmission pipeline, parses their network layer and transport layer headers to identify the protocol type flags. When the service data packet conforms to the Remote Direct Memory Access Protocol standard (RoCEv2) based on converged Ethernet, the header parsing unit 1122 calculates the offset and locates the base transport header inside the service data packet.

[0101] The feature field rewriting unit 1123 is connected to the message header parsing unit 1122, obtains the feature value output by the nonlinear quantization unit 1121, and directly overwrites the feature value into the reserved bits of the basic transmission header through hardware bitmasking operation. After the overwriting is completed, the feature field rewriting unit 1123 recalculates the cyclic redundancy check code of the service data packet at the data link layer and replaces the original check bits. Then, the modified service data packet is injected into the physical layer transmission link.

[0102] The feature field rewriting unit 1123 writes the feature value into the reserved bits of the basic transmission header, so that the underlying hardware operating status is transmitted concurrently with the normalized intelligent computing node communication data stream. The in-band piggyback transmission mechanism of the feature value realizes the physical path unification of node status information and business data stream, and eliminates the millisecond-level communication delay caused by the independent management network collecting status data.

[0103] Example 4

[0104] See attached document Figure 5 Based on the second embodiment, after the state feature values ​​encapsulated in step S200 arrive at the edge programmable switch 200 with the service message, the state feature matrix maintenance module 201 performs the matrix construction and maintenance operation in step S300.

[0105] The state feature matrix maintenance module 201 is integrated into the edge programmable switch 200. Internally, it includes a packet parsing and feature extraction unit 2011, a flow table matching and hash mapping unit 2012, an on-chip matrix storage unit 2013, and a time-sliding update unit 2014. The specific working steps of these units during system operation are as follows:

[0106] In step S301, the message parsing and feature extraction unit 2011 is connected to the data receiving port of the edge programmable switch 200, reads the service data packets entering the edge programmable switch 200, parses its basic transmission header to extract the status feature values ​​carried in the reserved bits, and extracts the source Internet Protocol address and destination Internet Protocol address of the service data packet. The operation of the message parsing and feature extraction unit 2011 to extract status feature values ​​and address information realizes the lossless offloading of in-band status data in the network plane.

[0107] In step S302, the flow table matching and hash mapping unit 2012 receives the source Internet Protocol address and destination Internet Protocol address extracted by the message parsing and feature extraction unit 2011, performs a hash operation on the two to generate a flow identifier, and uses the flow identifier as the index address of the data plane memory. The flow table matching and hash mapping unit 2012 performs a hash operation to compress the high-width address information into a fixed-length index, thereby improving the retrieval and matching speed of the underlying hardware memory.

[0108] In step S303, the on-chip matrix storage unit 2013 is deployed in the on-chip static random access memory of the edge programmable switch 200. Internally, it is divided into multiple hardware register arrays. These hardware register arrays are physically divided into a two-dimensional structure. The rows of the two-dimensional structure correspond to stream identifiers generated by hash operations, and the columns correspond to discrete time slices of a sliding time window. The on-chip matrix storage unit 2013 receives the stream identifiers and state feature values, addresses the corresponding row of the hardware register array based on the stream identifier, and writes the state feature values ​​into the column of the corresponding hardware register array where the current time slice is located. This constructs a state feature matrix corresponding to multiple service data streams in the underlying physical space. The storage unit 2013 performs write operations based on the on-chip static random access memory, eliminating communication latency caused by off-chip memory calls; the number of rows of the hardware register array is pre-configured according to the number of computing nodes directly connected to the edge programmable switch 200 and the maximum number of concurrent service data streams, and the number of columns is pre-configured according to the number of discrete time slices contained in the sliding time window; the flow table matching and hash mapping unit 2012 uses a combined hash function based on the source Internet Protocol address, destination Internet Protocol address and transport layer port number in the hash hash operation, and configures a collision list pointer for each row of the hardware register array in the on-chip matrix storage unit 2013 to handle hash collisions where different data streams are mapped to the same flow identifier;

[0109] In step S304, the time sliding update unit 2014 is connected to the on-chip matrix storage unit 2013. It has a hardware clock generator inside that periodically generates timestamp pulses. The time sliding update unit 2014 triggers a block shift operation based on the feature data bit width of the hardware register array in the on-chip matrix storage unit 2013 according to the timestamp pulses. The block shift operation clears the historical state feature values ​​that exceed the preset time window and latches the latest state feature value at the first position of the register. The block shift update operation of the time sliding update unit 2014 maintains the temporal locality of the state feature matrix and ensures that the subsequent consensus positioning logic has accurate time data support.

[0110] Furthermore, after the state feature matrix is ​​established, the distributed consensus location module 202 performs fault node consensus location and diagnosis in S400. Please refer to the appendix. Figure 6 The distributed consensus positioning module 202 is integrated into the edge programmable switch 200. It includes a multi-stream feature reading unit 2021, a confidence calculation unit 2022, and a threshold diagnosis unit 2023. The specific working steps of each unit in the distributed consensus positioning module 202 during system operation are as follows:

[0111] In step S401, the multi-stream feature reading unit 2021 is connected to the on-chip matrix storage unit 2013. It reads the state feature matrix in the on-chip matrix storage unit 2013 through hardware addressing instructions, extracts the state feature values ​​of multiple service data streams whose destination addresses point to the same destination computing node, and simultaneously extracts the state feature values ​​of other adjacent computing nodes other than the destination computing node. The multi-stream feature reading unit 2021 performs parallel reading operations to obtain the state data of multiple communication links in the local network topology, providing a data foundation for data plane cross-comparison.

[0112] In step S402, the confidence calculation unit 2022 receives the state feature values ​​extracted by the multi-stream feature reading unit 2021, and uses a hardware adder tree to perform an accumulation operation on the state feature values ​​of multiple service data streams to obtain the sum of congestion features. Simultaneously, it uses the hardware adder tree to perform an accumulation operation on the state feature values ​​of other adjacent computing nodes to obtain the sum of background features. Since the hardware pipeline does not include a divider, the confidence calculation unit 2022 reads the on-chip pre-set inverse multiplication table, multiplies the sum of background features by the inverse parameter of the total number of corresponding adjacent computing nodes to obtain the average background feature value, and then uses a hardware subtractor to subtract the average background feature value from the sum of congestion features, outputting the anomaly consensus score. The formula for the logical operation performed by the confidence calculation unit 2022 is as follows:

[0113]

[0114] in, This represents the abnormal consensus score of the target computation node n; This represents the total number of active service data streams whose destination address points to the destination compute node n; Indicates the first The status characteristic values ​​carried by each active business data stream; This represents the total number of adjacent computing nodes directly connected to the edge programmable switch 200, excluding the destination computing node n. Indicates the first The state characteristic values ​​carried by each adjacent computing node; This represents the pre-defined reciprocal parameter of the total number of adjacent computing nodes M;

[0115] The confidence calculation unit 2022 eliminates background noise interference caused by minor fluctuations in the local network through hardware accumulation and difference calculation operations, thereby improving the accuracy of abnormal state feature localization.

[0116] It should be noted that the aforementioned status characteristic values ​​are carried by the source computing node when sending service data packets, reflecting the rate of change of the source network card egress queue depth and internal bus transaction latency. When the destination computing node n experiences hardware failure or severe congestion, its data receiving and processing capabilities decrease, causing packets destined for the destination computing node n to accumulate in the source network card egress queue, resulting in an abnormal increase in the source hardware status derivative. Therefore, by statistically analyzing the source status characteristic values ​​carried by multiple service data streams with destination addresses pointing to the same destination computing node n, and comparing them with the background characteristic values ​​pointing to other normal adjacent nodes, it is possible to indirectly infer the anomaly of the destination computing node n on the edge programmable switch data plane, without requiring the destination computing node n to directly report its own status.

[0117] In step S403, the threshold confirmation unit 2023 receives the abnormal consensus score output by the confidence calculation unit 2022. Its internal hardware registers are pre-programmed with trigger thresholds. The threshold confirmation unit 2023 uses a hardware comparator to compare the abnormal consensus score with the trigger threshold. When the abnormal consensus score is greater than the trigger threshold, the target computing node is confirmed. The threshold diagnosis unit 2023 is the epicenter node of the fault in the physical network and outputs a diagnosis signal to the flow control isolation module 203. The comparison and judgment operation is completed entirely within the hardware pipeline of the edge programmable switch 200, avoiding the time delay caused by uploading to the centralized controller for software inference and shortening the response time of the underlying network fault.

[0118] Once a faulty node is identified in step S400, the flow control isolation module 203 immediately performs faulty traffic interception and isolation in step S500 on the data plane. Please refer to the appendix. Figure 7The flow control isolation module 203 is integrated into the edge programmable switch 200. It includes an action rule generation unit 2031 and a traffic interception buffer unit 2032. The specific working steps of each unit of the flow control isolation module 203 during system operation are as follows:

[0119] In step S501, the action rule generation unit 2031 connects to the distributed consensus positioning module 202, receives the diagnosis signal output by the distributed consensus positioning module 202, parses the diagnosis signal to obtain the network protocol address of the faulty node, and then accesses the tri-state content addressing memory at the bottom layer of the edge programmable switch 200. It writes an interception rule into the matching action flow table maintained by the tri-state content addressing memory. The matching field of the interception rule is the network protocol address of the faulty node, and the execution action field of the interception rule is redirection to the isolation queue. The operation of the action rule generation unit 2031 in writing the interception rule physically blocks the data path to the faulty node at the bottom layer data plane, preventing the faulty node from continuously receiving data and causing the end-side congestion to worsen.

[0120] In step S502, the traffic interception buffer unit 2032 is deployed in the data forwarding pipeline of the edge programmable switch 200, parses the subsequent service data packets entering the data forwarding pipeline, extracts the destination network protocol address of the subsequent service data packets, and compares the destination network protocol address in the matching action flow table in the tri-state content addressing memory.

[0121] In step S503, when the destination network protocol address is consistent with the network protocol address of the faulty node, the traffic interception buffer unit 2032 triggers the execution action field corresponding to the interception rule to intercept subsequent service data packets and transfers the intercepted service data packets to the on-chip isolation buffer queue divided by static random access memory inside the edge programmable switch 200. The operation of the traffic interception buffer unit 2032 to transfer service data packets cuts off the data flow to the faulty node, while leaving the original packet data in the on-chip isolation buffer queue, providing basic data support for subsequent modules to read packet header parameters.

[0122] Furthermore, while traffic interception is completed in step S500, the reverse confirmation forgery logic inside the flow control isolation module 203 executes step S600 to construct a reverse confirmation message to release the source buffer. Please refer to the appendix. Figure 8 The flow control isolation module 203 also includes a message parameter extraction unit 2033 and a message forgery confirmation unit 2034. The specific working steps of each unit inside the flow control isolation module 203 during system operation are as follows:

[0123] In step S601, the message parameter extraction unit 2033 connects to the on-chip isolation buffer queue inside the edge programmable switch 200, reads the service data packets intercepted in the on-chip isolation buffer queue, parses the transport layer header of the service data packets, and extracts the packet sequence number, source Internet Protocol address, destination Internet Protocol address, source queue pair number, and destination queue pair number of the service data packets. The message parameter extraction unit 2033 extracts the packet sequence number, address parameters, and queue number parameters, and obtains the core hardware state machine variables required by the source computing node 100 to maintain the remote direct memory access connection.

[0124] Step S602: The message forgery unit 2034 is connected to the message parameter extraction unit 2033. The message generation pipeline at the bottom layer of the edge programmable switch 200 is invoked to perform source and destination field swapping on the extracted source Internet Protocol address and destination Internet Protocol address, and write the swapped Internet Protocol address into the Internet Protocol header of the newly generated message; the source queue pair number and destination queue pair number are swapped on the extracted source queue pair number, and write the swapped queue pair number into the basic transmission header of the newly generated message; the opcode field of the basic transmission header of the newly generated message is set to the acknowledgment opcode; the extracted packet sequence number is synchronously written into the packet sequence number field of the basic transmission header of the newly generated message. The message forgery unit 2034 generates a reverse acknowledgment message based on the source and destination field swapping, opcode modification, and synchronous packet sequence number writing. This operation directly simulates the protocol stack response at the bottom layer of the destination network card at the intermediate node of the network.

[0125] Step S603: The message forgery unit 2034 performs addressing according to the destination Internet Protocol address after source-destination swap, and sends the reverse confirmation message to the source computing node 100 that sent the original service data message through the physical downlink port of the edge programmable switch 200.

[0126] In step S604, the smart network interface card 110 of the source computing node 100 receives the reverse confirmation message, extracts the packet sequence number from the reverse confirmation message, and compares the packet sequence number with the expected sequence number in the local hardware queue state machine of the smart network interface card 110. When the packet sequence number matches the expected sequence number, the smart network interface card 110 determines that the original service data packet has been transmitted, executes the dequeue instruction, and releases the corresponding local transmission buffer space. The operation of the smart network interface card 110 to release the local transmission buffer space clears the data congestion backlog at the data link layer of the source computing node 100, and blocks the generation triggering condition of priority-based flow control (PFC) pause frame from a physical mechanism, thus avoiding the global computing power shutdown of the intelligent computing center cluster 10 caused by local network failure.

[0127] It should be noted that the business data packets that are intercepted and temporarily stored in the on-chip isolated cache queue can be analyzed offline by the operation and maintenance personnel through the out-of-band management channel after the faulty node recovers. Alternatively, in scenarios with application-layer fault tolerance mechanisms (such as the checkpoint and retransmission mechanism of the distributed training framework), the upper-layer application can restore the data in that interval through data retransmission or state rollback. The hot standby node and the faulty node are pre-configured with a shared remote memory access address space, and this address space is kept synchronized with the data through the cluster storage consistency protocol. This ensures that after the source end switches to the hot standby node, the hot standby node has the same memory context as before the switchover of the faulty node, and can continue to receive subsequent business data packets.

[0128] Please refer to the appendix. Figure 9 After the faulty node is isolated and the source buffer is released, the custom signaling generation module 204 executes step S700, distributing link switching instructions to the affected source. The custom signaling generation module 204 is integrated into the edge programmable switch 200 and includes a parameter query unit 2041, a signaling encapsulation unit 2042, and a signaling distribution unit 2043. The specific working steps of each unit within the custom signaling generation module 204 during system operation are as follows:

[0129] In step S701, the parameter query unit 2041 connects to the distributed consensus positioning module 202, receives the original network protocol address of the faulty node, and accesses the static random access memory configured inside the edge programmable switch 200. The static random access memory is pre-programmed with a cluster hot standby mapping table. The parameter query unit 2041 uses the original network protocol address to perform hardware retrieval operations in the cluster hot standby mapping table to obtain the target medium access control address, target Internet Protocol address, target queue pair number, and target remote memory access key value of the corresponding hot standby node. The parameter query unit 2041 directly retrieves the network parameters of the hot standby node in the underlying static random access memory, avoiding interaction with the central controller to request routing data, and providing address data support with zero waiting delay for subsequent network path hot switching.

[0130] In step S702, the signaling encapsulation unit 2042 connects to the parameter query unit 2041, calls the message generation engine of the edge programmable switch 200, generates the basic structure of the header of the standard congestion notification message, appends a predefined format vendor extension header after the acknowledgment extension transmission header of the standard congestion notification message, and writes the original queue pair number of the faulty node, the target medium access control address of the hot standby node, the target Internet Protocol address, the target queue pair number, and the target remote memory access key value into the vendor extension header in bit order. At the same time, the original network protocol address of the faulty node is written into the original destination address field of the vendor extension header. The signaling encapsulation unit 2042 modifies the specific reserved flag bits in the basic transmission header of the standard congestion notification message to generate a customized congestion notification message containing underlying state machine overwrite semantics. The operation of the signaling encapsulation unit 2042 in appending the vendor extension header and modifying the reserved flag bits achieves protocol-compatible transmission of the underlying communication state machine switching parameters while maintaining the legality of the underlying remote direct memory access protocol header parsing.

[0131] In step S703, the signaling distribution unit 2043 connects to the signaling encapsulation unit 2042, reads the active connection tracking table maintained in the tri-state content addressing memory inside the edge programmable switch 200, extracts the source network protocol addresses of all source computing nodes 100 whose destination addresses point to the faulty node from the active connection tracking table, and uses the hardware message replication engine to copy the customized congestion notification message into multiple independent unicast signaling messages according to the number of extracted source network protocol addresses.

[0132] In step S704, the signaling distribution unit 2043 overwrites the source network protocol address into the destination address field of multiple independent unicast signaling messages, and sends the multiple independent unicast signaling messages to the corresponding source computing nodes 100 through the physical downlink port of the edge programmable switch 200. The message copying and multi-path transmission operation of the signaling distribution unit 2043 ensures that all source computing nodes 100 affected by network congestion of the faulty node can synchronously receive the underlying link switching instruction.

[0133] Finally, refer to the appendix Figure 10 After the source computing node receives the customized congestion notification message sent in step S700, the context hot overwrite module 113 executes step S800 to complete the hardware layer hot switching of the communication endpoint. The context hot overwrite module 113 is integrated into the smart network card 110 of the source computing node 100, and its internal components include a signaling parsing unit 1131, a memory positioning unit 1132, and a state overwrite unit 1133. The specific working steps of each unit in the context hot overwrite module 113 during system operation are as follows:

[0134] In step S801, the signaling parsing unit 1131 connects to the data receiving link of the smart network card 110, receives the customized congestion notification message sent by the edge programmable switch 200, parses the basic transmission header of the customized congestion notification message to identify specific reserved identifier bits, and when the specific reserved identifier bits meet the preset reading conditions, the signaling parsing unit 1131 extracts the hot standby node target media access control address, target Internet Protocol address, target queue pair number and target remote memory access key value carried in the vendor extension header of the customized congestion notification message;

[0135] In step S802, the memory location unit 1132 is connected to the signaling parsing unit 1131. It extracts the original destination Internet Protocol address of the faulty node from the original destination address field of the manufacturer's extended header, generates a joint index using the original destination Internet Protocol address and the original queue pair number, and locates the original queue pair context storage block corresponding to the current communication connection by addressing the queue pair context mapping table in the on-chip static random access memory of the smart network card 110 through direct memory access instructions. The memory location unit 1132 executes the operation of direct memory access instructions to locate the original queue pair context storage block, eliminating the instruction scheduling overhead of the central processing unit 101 participating in memory addressing, and realizing high-speed direct access to the underlying hardware space.

[0136] In step S803, the state overwrite unit 1133 is connected to the memory location unit 1132 and receives the target medium access control address, target Internet Protocol address, target queue pair number, and target remote memory access key value of the hot standby node extracted by the signaling parsing unit 1131. Using these parameters, it physically overwrites the original faulty node network parameters in the original queue pair context storage block in the on-chip static random access memory. During the network parameter overwriting process, the state overwrite unit 1133 locks the packet sequence number state machine pointer and the expected receive sequence number storage bit in the original queue pair context storage block. The hardware mapping logic formula for the network parameter update executed by the state overwrite unit 1133 is as follows:

[0137]

[0138] in, This represents the updated queue pair context vector; Indicates the target media access control address of the hot standby node; Indicates the target Internet Protocol address of the hot standby node; Indicates the target queue pair number of the hot standby node; This represents the remote memory access key value of the hot standby node target; Indicates the currently sent packet sequence number that is locked; Indicates the currently expected packet sequence number to be locked;

[0139] In step S804, after the state overwrite unit 1133 completes the network parameter overwrite, the data transmission engine of the smart network card 110 reads the updated queue pair context vector and encapsulates the subsequent service data packets. It writes the target medium access control address, target Internet Protocol address, target queue pair number, and target remote memory access key value of the hot standby node into the protocol layer header corresponding to the subsequent service data packets. The subsequent service data packets are addressed and transmitted to the hot standby node along the physical network link. The state overwrite unit 1133 overwrites the network parameters and locks the packet sequence number state machine pointer. The operation completes the physical switching of the communication endpoint at the hardware layer of the smart network card 110, reduces the probability of communication timeout interruption of the artificial intelligence training framework running on the upper layer of the source computing node 100, and improves the continuity and fault tolerance of data interaction between the source computing node 100 and the destination computing node.

Claims

1. A fault intelligent location and self-healing system for a computing power cluster in an intelligent computing center, deployed in an intelligent computing center cluster (10), characterized in that, include: Multiple computing nodes (100) and an edge programmable switch (200), wherein the multiple computing nodes (100) are connected to the edge programmable switch (200) via optical communication links; The computing node (100) is used to provide data processing computing power. The computing node (100) is equipped with a central processing unit (101), a graphics processing unit (102), and a smart network card (110). The smart network card (110) is used to process network data packets entering and leaving the computing node (100). The smart network card (110) is equipped with a hardware micro-state acquisition module (111), an in-band feature encoding module (112), a context hot overwrite module (113), and an on-chip static random access memory. The edge programmable switch (200) is used to provide line-speed forwarding of data packets and perform logical operations on the network data plane. The edge programmable switch (200) is equipped with a state feature matrix maintenance module (201), a distributed consensus positioning module (202), a flow control isolation module (203), and a customized signaling generation module (204). The hardware micro-state acquisition module (111) is used to bypass and monitor the internal bus transaction delay and egress queue depth of the computing node (100), and calculate the rate of change of the internal bus transaction delay and egress queue depth based on a preset time sliding window to generate hardware state derivatives. The in-band feature encoding module (112) is used to quantize the hardware state derivative into state feature values ​​and encapsulate the state feature values ​​into the extended header of the service data packet flowing out of the smart network card (110); The state feature matrix maintenance module (201) is used to parse the extended header of the service data packet, extract the state feature values, write the state feature values ​​into the on-chip static random access memory inside the edge programmable switch (200), and update the state feature matrix corresponding to multiple data streams. The distributed consensus positioning module (202) is used to read the state feature values ​​of each node recorded in the state feature matrix and perform logical comparison operations. When the operation results of multiple data streams meet the preset trigger threshold, the target computing node that generates abnormal state feature values ​​is determined to be a faulty node. The flow control isolation module (203) is used to generate forwarding action rules on the data plane of the edge programmable switch (200) to intercept subsequent service data packets whose destination address is the faulty node, extract the packet sequence number of the intercepted service data packet, construct a reverse confirmation message based on the packet sequence number and send it to the source computing node (100) that sent the intercepted service data packet, so as to release the local sending buffer space. The customized signaling generation module (204) is used to query the pre-configured cluster hot standby routing table inside the edge programmable switch (200) to obtain the corresponding hot standby node network parameters, construct a customized congestion notification message and write the hot standby node network parameters into it, and send it to the source computing node (100) that has a communication connection with the faulty node. The context hot overwrite module (113) is used to receive the customized congestion notification message, read the queue pair context stored inside the smart network card (110) through the direct memory access mechanism, overwrite the original fault node network parameters in the queue pair context with the hot standby node network parameters, and keep the packet sequence number state machine pointer in the queue pair context unchanged.

2. The intelligent fault location and self-healing system for the intelligent computing center computing power cluster according to claim 1, characterized in that, The hardware micro-state acquisition module (111) is integrated into the hardware circuit of the smart network card (110), and includes a delay sampling unit (1111), a queue listening unit (1112), a sliding window register group (1113), and a differential operation unit (1114). The delay sampling unit (1111) is used to connect to the internal PCIe bus interface of the computing node (100), extract the sending timestamp and receiving timestamp of the bus transaction, and calculate the difference between the sending timestamp and receiving timestamp to obtain the current bus transaction delay. The queue monitoring unit (1112) is used to connect to the data link layer outgoing channel of the smart network card (110) and read the read-write pointer difference of the outgoing queue to obtain the current outgoing queue depth; The sliding window register group (1113) is used to receive the system clock pulse signal, latch the current bus transaction delay and the current exit queue depth of the current sampling period, and retain the historical bus transaction delay and historical exit queue depth of the previous sampling period. The differential operation unit (1114) is used to perform discrete differential operations using a hardware subtractor to obtain the exit queue depth difference and the bus transaction delay difference. An internal hardware multiplier multiplies the exit queue depth difference and the bus transaction delay difference by preset weight normalization coefficients to obtain an accumulated sum. A shift register is then used to shift the accumulated sum to the right. The logical operations on the bits output the derivative of the hardware state, and the calculation formula is as follows: in, Indicates the first Hardware state derivative for each sampling period; Indicates the first The difference in exit queue depth over each sampling period; Indicates the first The bus transaction latency difference over each sampling period; and These are the preset weight normalization coefficients; This represents the number of clock cycles within the sliding time window. It is a positive integer.

3. The intelligent fault location and self-healing system for the intelligent computing center computing power cluster according to claim 1, characterized in that, The in-band feature coding module (112) includes a nonlinear quantization unit (1121), a message header parsing unit (1122), and a feature field rewriting unit (1123); The nonlinear quantization unit (1121) is used to receive the hardware state derivative, perform data interval matching on the hardware state derivative using the internal hardware comparator array, and map the multi-byte high-width hardware state derivative to a low-bit-width feature value based on the built-in segmented quantization function. The message header parsing unit (1122) is used to read the service data message entering the data transmission pipeline, parse the network layer and transport layer headers to identify the protocol type flag, and locate the basic transport header inside the service data message when it conforms to the Remote Direct Memory Access Protocol standard based on converged Ethernet. The feature field rewriting unit (1123) is used to directly overwrite the feature value into the reserved bits of the basic transmission header through hardware bitmasking operation, and recalculate the cyclic redundancy check code of the service data packet at the data link layer and replace the original check bits.

4. The intelligent fault location and self-healing system for the intelligent computing center computing power cluster according to claim 1, characterized in that, The state feature matrix maintenance module (201) includes a message parsing and feature extraction unit (2011), a flow table matching and hash mapping unit (2012), an on-chip matrix storage unit (2013), and a time-sliding update unit (2014). The message parsing and feature extraction unit (2011) is used to parse the basic transmission header of the service data message, extract the status feature values ​​carried in the reserved bits, and extract the source Internet Protocol address and destination Internet Protocol address of the service data message. The flow table matching and hash mapping unit (2012) is used to perform a hash operation on the source Internet Protocol address and the destination Internet Protocol address to generate a flow identifier, and use the flow identifier as the index address of the data plane memory; The on-chip matrix storage unit (2013) is used to address the rows of the hardware register array according to the stream identifier, write the state feature values ​​into the column where the current time slice is located, and construct a state feature matrix; The time-sliding update unit (2014) is used to trigger a block shift operation based on the feature data bit width of the hardware register array according to the timestamp pulse, clear historical state feature values ​​that exceed the preset time window, and latch the latest state feature value at the first position of the register.

5. The intelligent fault location and self-healing system for the intelligent computing center computing power cluster according to claim 1, characterized in that, The distributed consensus positioning module (202) internally includes a multi-stream feature reading unit (2021), a confidence calculation unit (2022), and a threshold diagnosis unit (2023); The multi-stream feature reading unit (2021) is used to read the state feature matrix through hardware addressing instructions, extract the state feature values ​​of multiple service data streams whose destination addresses point to the same destination computing node, and extract the state feature values ​​of other adjacent computing nodes other than the destination computing node. The confidence calculation unit (2022) is used to perform cumulative operations on the state feature values ​​of multiple service data streams using a hardware adder tree to obtain the sum of congestion features, and to perform cumulative operations on the state feature values ​​of other adjacent computing nodes using a hardware adder tree to obtain the sum of background features. It then reads a preset inverse multiplication table, multiplies the sum of background features by the inverse parameter of the total number of adjacent computing nodes to obtain the average background feature value, and uses a hardware subtractor to subtract the average background feature value from the sum of congestion features to output an anomaly consensus score. The formula for the calculation is as follows: in, Indicates the destination compute node Abnormal consensus score; This indicates that the destination address points to the destination compute node. The total number of active business data streams; Indicates the first The status characteristic values ​​carried by each active business data stream; This indicates that the edge programmable switch (200) directly connects to the destination computing node, excluding the destination computing node. The total number of adjacent computing nodes other than; Indicates the first The state characteristic values ​​carried by each adjacent computing node; Represents the total number of adjacent computing nodes. The preset reciprocal parameter; The threshold confirmation unit (2023) is used to compare the anomaly consensus score with the trigger threshold using a hardware comparator. When the score is greater than the trigger threshold, the target computing node is confirmed. It is the epicenter node of the fault in the physical network.

6. The intelligent fault location and self-healing system for the intelligent computing center computing power cluster according to claim 1, characterized in that, The flow control isolation module (203) includes an action rule generation unit (2031) and a traffic interception buffer unit (2032); The action rule generation unit (2031) is used to receive the confirmed signal to obtain the network protocol address of the faulty node, and write the interception rule into the matching action flow table maintained by the underlying three-state content addressing memory. The matching field of the interception rule is the network protocol address of the faulty node, and the execution action field is redirection to the isolation queue. The traffic interception buffer unit (2032) is used to extract the destination network protocol address of subsequent service data packets entering the data forwarding pipeline. When it matches the network protocol address of the faulty node, it triggers the interception rule and transfers the intercepted service data packets to the on-chip isolated buffer queue divided by static random access memory inside the edge programmable switch (200).

7. The intelligent fault location and self-healing system for the intelligent computing center computing power cluster according to claim 6, characterized in that, The flow control isolation module (203) also includes a message parameter extraction unit (2033) and a message forgery confirmation unit (2034); The message parameter extraction unit (2033) is used to read the service data message intercepted in the on-chip isolation buffer queue, parse the transport layer header, and extract the packet sequence number, source Internet Protocol address, destination Internet Protocol address, source queue pair number and destination queue pair number. The acknowledgment message forgery unit (2034) is used to call the underlying message generation pipeline, perform source and destination field swapping on the extracted source Internet Protocol address and destination Internet Protocol address and write them into the Internet Protocol header of the newly generated message, perform source queue pair number swapping on the extracted source queue pair number and destination queue pair number and write them into the basic transport header of the newly generated message, set the opcode field of the basic transport header of the newly generated message to the acknowledgment response opcode, and synchronously write the extracted packet sequence number into the packet sequence number field of the basic transport header of the newly generated message to generate a reverse acknowledgment message.

8. The intelligent fault location and self-healing system for the intelligent computing center computing power cluster according to claim 1, characterized in that, The customized signaling generation module (204) includes a parameter query unit (2041), a signaling encapsulation unit (2042), and a signaling distribution unit (2043). The parameter query unit (2041) is used to receive the original network protocol address of the faulty node, and to perform hardware retrieval operation in the static random access memory with the cluster hot standby mapping table pre-programmed, using the original network protocol address to obtain the corresponding hot standby node target medium access control address, target Internet Protocol address, target queue pair number and target remote memory access key value. The signaling encapsulation unit (2042) is used to generate the header basic structure of the standard congestion notification message. After the response extended transmission header, a predefined format vendor extended header is appended. The target medium access control address, target Internet Protocol address, target queue pair number and target remote memory access key value of the hot standby node are written into the vendor extended header in bit order. Specific reserved flag bits are modified to generate a customized congestion notification message. The signaling distribution unit (2043) is used to extract the source network protocol addresses of all source computing nodes (100) whose destination addresses point to the faulty node from the active connection tracking table, and use the hardware message replication engine to replicate multiple independent unicast signaling messages and send them to the corresponding source computing nodes (100).

9. The intelligent fault location and self-healing system for the intelligent computing center computing power cluster according to claim 1, characterized in that, The context hot overwrite module (113) includes a signaling parsing unit (1131), a memory location unit (1132), and a status overwrite unit (1133); The signaling parsing unit (1131) is used to parse the basic transmission header of the received customized congestion notification message to identify specific reserved identifier bits, and extract the target medium access control address, target Internet Protocol address, target queue pair number and target remote memory access key value of the hot standby node carried in the vendor extended header. The memory positioning unit (1132) is used to extract the original destination Internet Protocol address and the original queue pair number to generate a joint index, and use the joint index to generate a direct memory access instruction to address the queue pair context mapping table in the on-chip static random access memory of the smart network card (110) and locate the original queue pair context storage block corresponding to the current communication connection. The state overwrite unit (1133) is used to physically overwrite the original fault node network parameters in the original queue pair context storage block in the on-chip static random access memory using the extracted parameters, and lock the packet sequence number state machine pointer and the expected received sequence number storage bit in the original queue pair context storage block during the overwriting process.

10. The intelligent fault location and self-healing system for the intelligent computing center computing power cluster according to claim 9, characterized in that, After the state overwrite unit (1133) completes the network parameter overwrite, the data transmission engine of the smart network card (110) is used to read the updated queue pair context vector to encapsulate the subsequent service data packets, and write the target medium access control address, target Internet Protocol address, target queue pair number and target remote memory access key value of the hot standby node into the protocol layer header corresponding to the subsequent service data packets, so that the subsequent service data packets are addressed and transmitted to the hot standby node along the physical network link.

Images