A method for head-of-line blocking of an interconnect switch chip with ai compute nodes

Abstract

The application discloses a method for head-of-line blocking of an interconnection exchange chip of an AI computing node, and belongs to the technical field of AI computing node interconnection and chips. According to the method, OQs corresponding to each export port in the Crossbar architecture of the interconnection exchange chip are grouped according to the export port, whether there is to-be-scheduled data in each OQ group and whether the OQ group fails to be scheduled due to no credit are counted every beat, if a certain OQ group continuously satisfies the above two conditions for N beats, it is determined that the GPU connected with the corresponding export port is faulty, then the Drop mode register of the export port is enabled, so that the export port enters the Drop mode to discard the buffered message and return the credit, the OQ and the Input FIFO successively obtain the credit to schedule the message, and finally the head-of-line blocking is removed. The method can accurately identify the GPU fault and avoid misjudgment, quickly remove the blocking and does not affect normal business, does not need to change the hardware architecture, does not increase the cost and design complexity, is suitable for the core business scene of the AI computing node, and guarantees the business continuity.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of AI computing node interconnection and chip technology, in particular to a method for head blocking of an interconnection exchange chip of an AI computing node. BACKGROUND

[0002] The AI computing node is the core hardware architecture in the current AI training and inference field, and usually needs to realize full interconnection of hundreds of GPUs through an interconnection exchange chip to meet the communication requirements of high bandwidth and low latency. The mainstream interconnection exchange chip currently generally adopts a Crossbar switching architecture, and is matched with an end-to-end credit backpressure protocol (such as a UALink type protocol) to realize lossless transmission. A message needs to be buffered through an Input FIFO (input buffer queue) first, and then is dispatched to the OQ (output buffer queue) of the corresponding export through routing lookup, and is completed after the GPU returns the credit; the core service flow is concentrated in “ALL-TO-ONE” (multiple GPUs collect data to a single GPU) and “ONE-TO-ALL” (a single GPU distributes data to multiple GPUs), and the continuity of data interaction between all GPUs needs to be guaranteed.

[0003] The existing interconnection exchange chip has obvious technical shortcomings when dealing with GPU related exceptions: first, the OQ is not grouped and managed according to the export port, and it is difficult to accurately associate all message buffers and scheduling states corresponding to a specific export port, and it is difficult to locate the fault source; second, there is a lack of continuous statistical judgment mechanism for “existence of data to be scheduled” and “no credit scheduling failure” of the OQ group, which cannot effectively distinguish between GPU faults and temporary credit delays, and is prone to fault misjudgment or omission; third, there is no directional processing scheme for the fault export, when a certain GPU fault causes the corresponding OQ to be unable to schedule, the Input FIFO will cause the subsequent normal target message to be blocked by the head fault target message due to the first-in-first-out characteristic, and eventually cause the communication interruption of the whole network GPU, and it is impossible to guarantee the normal business flow during the unblocking process, which seriously restricts the business stability of the AI computing node. SUMMARY

[0004] The purpose of the present application is to overcome one or more shortcomings of the prior art, and to provide a method for head blocking of an interconnection exchange chip of an AI computing node.

[0005] The purpose of the present application is achieved by the following technical solutions:

[0006] A method for head blocking of an interconnection exchange chip of an AI computing node is provided, comprising the following steps:

[0007] Step 1: Group the OQs (Output Queues) corresponding to each egress port in the Crossbar architecture by egress port, obtaining an OQ group corresponding to each egress port, each OQ group containing all OQs of input ports pointing to the egress port;

[0008] Step 2: Perform statistical monitoring on each OQ group per beat, respectively recording whether there is data to be scheduled in the OQ group and whether scheduling fails due to lack of credit;

[0009] Step 3: Determine whether the fault condition is met: if the OQ group corresponding to a certain egress port continuously meets the condition of having data to be scheduled and scheduling failure due to lack of credit for N beats, determine that the GPU connected to the egress port is faulty, where N is the fault triggering threshold;

[0010] Step 4: For the egress port determined to be faulty, set the Drop mode register of the egress port in the Crossbar to the enabled state;

[0011] Step 5: After the egress port reads that the Drop mode register is in the enabled state, it enters Drop mode, discards the packets in its own cache, and returns credit to the corresponding OQ group;

[0012] Step 6: After the OQ group obtains credit, it schedules the packets in the cache to the egress port (the packets are discarded), and returns credit to the Input FIFO of the interconnection switch chip;

[0013] Step 7: After the Input FIFO obtains credit, it schedules the head packet to the corresponding OQ group, resolving the problem of subsequent normal packets being unable to be scheduled due to head packet blocking, and achieving head blocking resolution.

[0014] Crossbar: Crossbar switch architecture is a typical N x N matrix structure, each input port is connected to all output ports through a cross node;

[0015] Output Queue: Buffer queue corresponding to the destination port of the packet in the Crossbar structure;

[0016] OQ: Abbreviation of Output Queue;

[0017] FIFO: First-In First-Out, a data processing principle, meaning that the first data to enter is the first to be processed or output;

[0018] Input FIFO: Queue for buffering when a packet enters, the input port uses a First-In First-Out (FIFO) queue to manage the data packets to be forwarded;

[0019] Head-of-Line Blocking, HOL Blocking: queue head blocking.

[0020] Further, the time per beat in step 2 is defined as: 1 beat corresponds to one clock cycle of the interconnection switch chip, and the clock cycle is consistent with the interconnection clock cycle of the GPU in the AI computing node.

[0021] Further, the setting step of the failure trigger threshold N in step 3 includes: setting N to 3 beats by default; if the credit in the AI computing node adopts the mode of returning delay of other messages, setting N to 5-8 beats. The threshold value can be modified according to the application environment and business characteristics.

[0022] Further, the specific steps of the statistical monitoring in step 2 include:

[0023] Step 21: Determine whether the OQ group has data to be scheduled: if at least one OQ in the OQ group stores a message, it is determined that the OQ group has data to be scheduled;

[0024] Step 22: Determine whether the OQ group fails to schedule due to lack of credit: if all OQs in the OQ group cannot schedule messages due to not receiving the credit returned by the GPU, it is determined that the OQ group fails to schedule due to lack of credit.

[0025] Further, the setting step of the Drop mode register in step 4 includes: the Drop mode register is located in the output port control unit of the Crossbar, which is a 1-bit register; after the failure judgment module of the interconnection switch chip determines that the GPU corresponding to the export port fails, the register is set to the enabled state; the message processing module of the export port reads the state of the register every beat to determine whether to enter the Drop mode.

[0026] Further, the specific steps of the returned credit in step 5 include: the number of credits returned by the export port is consistent with the number of discarded messages; the number of credits returned by the OQ group to the previous Input FIFO is consistent with the number of messages scheduled to the export port by the OQ group.

[0027] Further, it further includes step 8: after the export port enters the Drop mode, the operation of discarding messages and returning credits is continuously performed until there is no message to be discarded in the cache of the export port, ensuring that all messages in the cache of the previous stage pointing to the GPU failure are processed.

[0028] Further, the step 9 is further included: when the faulty GPU completes self-fault recovery, the GPU connects the port state of the interconnection switch chip to down; after the interconnection switch chip detects that the port state is down, the Drop mode register of the corresponding export port is set to the disabled state, and the export port exits the Drop mode; then the GPU and the port of the interconnection switch chip start link-up negotiation, and after the negotiation is successful, normal communication is restored.

[0029] Further, the steps 1 to 7 adapt the service flow scene of the AI computing node, and the service flow scene includes but is not limited to an ALL-TO-ONE scene in which multiple GPUs collect data to a single GPU and a ONE-TO-ALL scene in which a single GPU distributes data to multiple GPUs.

[0030] Further, in the process of executing the steps 1 to 7, the Crossbar architecture of the interconnection switch chip is not changed, and no cache component of the Input FIFO or the OQ is added, and only the head-of-line blocking is removed through statistics, judgment and control at a logic level.

[0031] The beneficial effects of the application are as follows:

[0032] (1) The GPU fault is accurately identified through the OQ grouping statistics and fault threshold judgment technology according to the export port, the misjudgment caused by temporary no credit is avoided, and accurate basis is provided for the head-of-line blocking removal;

[0033] (2) The head-of-line blocking is quickly removed through the message discarding and credit return synchronization technology in the Drop mode, and the normal service flow forwarding is not affected, and the AI computing node service continuity is guaranteed;

[0034] (3) Through the statistics, judgment and control at the logic level, the hardware architecture of the switch chip is not changed, and the hardware cost and design complexity are not increased, and the scene demand of the AI computing node is adapted. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 It is a whole technical scheme flow chart of the head-of-line blocking of the AI supercomputing node interconnection switch chip.

[0036] Figure 2 It is an OQ grouping and statistics monitoring flow chart according to the export port.

[0037] Figure 3 It is an OQ fault judgment and fault trigger threshold setting flow chart.

[0038] Figure 4 It is an OQ trigger fault recovery and head-of-line blocking removal flow chart.

[0039] Figure 5 To exit the fault recovery and GPU-port communication reconstruction flowchart. DETAILED DESCRIPTION

[0040] The technical solutions of the present application will be described below in connection with embodiments, obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0041] Embodiment 1:

[0042] A method for head-of-line blocking of an interconnection switch chip of an AI computing node is provided, comprising the following steps:

[0043] Step 1: Group the OQ (Output Queue) corresponding to each exit port in the Crossbar architecture of the interconnection switch chip according to the exit port, to obtain an OQ group corresponding to the exit port one by one, each OQ group containing all the OQs of the input port pointing to the exit port;

[0044] Step 2: Perform statistical monitoring on each OQ group per beat, and record whether there is data to be scheduled and whether scheduling fails due to lack of credit in the OQ group, respectively;

[0045] Step 3: Determine whether the fault condition is met: if the OQ group corresponding to a certain exit port continuously meets the conditions of "there is data to be scheduled" and "scheduling fails due to lack of credit" for N beats, it is determined that the GPU connected to the exit port is faulty, where N is the fault triggering threshold;

[0046] Step 4: For the exit port determined to be faulty, set the Drop mode register of the exit port in the Crossbar to the enabled state;

[0047] Step 5: After the exit port reads that the Drop mode register is in the enabled state, it enters the Drop mode, discards the packets in its own cache, and returns credit to the corresponding OQ group;

[0048] Step 6: After the OQ group obtains the credit, it schedules the packets in the cache to the exit port (the packets are discarded), and returns credit to the Input FIFO of the interconnection switch chip;

[0049] Step 7: After the Input FIFO obtains the credit, it schedules the head packet to the corresponding OQ group, and resolves the problem that subsequent normal packets cannot be scheduled due to head packet blocking, thereby realizing the resolution of head-of-line blocking.

[0050] In step 2, the time of "each beat" is defined as: 1 beat corresponds to one clock cycle of the interconnection switch chip, and the clock cycle is consistent with the interconnection clock cycle of the GPU in the AI computing node.

[0051] In step 3, the setting step of the fault trigger threshold N includes: setting N to 3 beats by default; if the credit in the AI computing node adopts the "ride with other message delay return" mode, setting N to 5-8 beats. The threshold value can be modified according to the application environment and business characteristics.

[0052] In step 2, the specific steps of "statistical monitoring" include:

[0053] Step 21: Determine whether the OQ group has data to be scheduled: if at least one OQ in the OQ group stores a message, it is determined that the OQ group has data to be scheduled.

[0054] Step 22: Determine whether the OQ group fails to schedule due to lack of credit: if all OQs in the OQ group cannot schedule messages due to not receiving the credit returned by the GPU, it is determined that the OQ group fails to schedule due to lack of credit.

[0055] In step 4, the setting step of the "Drop mode register" includes: the Drop mode register is located in the output port control unit of the Crossbar, which is a 1-bit register; after the interconnection switch chip determines that the GPU corresponding to the export port is faulty, the fault judgment module sets the register to the enabled state; the message processing module of the export port reads the state of the register every beat to determine whether to enter the Drop mode.

[0056] In step 5, the specific steps of "returning credit" include: the number of credits returned by the export port is consistent with the number of discarded messages; the number of credits returned by the OQ group to the previous Input FIFO is consistent with the number of messages scheduled by the OQ group to the export port.

[0057] In addition, step 8 includes: after the export port enters the Drop mode, the operation of "discarding messages-returning credit" is continuously performed until there is no message to be discarded in the cache of the export port, ensuring that all previous cache messages pointing to the faulty GPU are processed.

[0058] In addition, step 9 includes: when the faulty GPU completes its own fault recovery, the state of the port connected to the interconnection switch chip of the GPU is switched to down; after the interconnection switch chip detects that the port state is down, the Drop mode register of the corresponding export port is set to the disabled state, and the export port exits the Drop mode; then the port of the GPU and the interconnection switch chip start link-up negotiation, and after the negotiation is successful, the normal communication is restored.

[0059] Wherein, steps 1 to 7 adapt the business flow scenario of the AI computing node, including but not limited to "ALL-TO-ONE scenario of multiple GPUs collecting data to a single GPU" and "ONE-TO-ALL scenario of a single GPU distributing data to multiple GPUs".

[0060] Wherein, during the execution of steps 1 to 7, the Crossbar architecture of the interconnection switch chip is not changed, and no cache component of Input FIFO or OQ is added, and only through the logic level statistics, judgment and control, the head blocking is removed.

[0061] Embodiment 2: Head blocking removal based on OQ grouping statistics under AI training ALL-TO-ONE business flow:

[0062] This embodiment is applied to a supercomputing node in the AI training scenario, which realizes the full interconnection of 144 GPUs through the interconnection switch chip of the Crossbar architecture, and the core business flow is "ALL-TO-ONE" - all GPUs collect intermediate data to a single target GPU set for training, and an end-to-end instant credit backpressure protocol is adopted. In the initial state, all GPUs and the ports of the switch chip are in the link-up state, and there is no cache backlog in the Input FIFO (input cache queue) and the OQ (Output Queue, output cache queue), and the message forwarding process is stable: after the message is cached by the Input FIFO, the target exit port is determined through routing lookup, and then it is scheduled to the OQ of the corresponding exit; after the GPU returns the credit, the OQ sends the message to the GPU, and returns the credit to the Input FIFO, allowing the subsequent message to continue to be scheduled.

[0063] Implementation process:

[0064] OQ grouping according to exit port:

[0065] All OQs in the Crossbar architecture of the interconnection switch chip are processed by one-to-one grouping according to "exit port": each exit port corresponds to an independent OQ group, and the OQ group contains all OQs of the input port pointing to the exit port. Taking exit port 8 as an example, the OQ group corresponding to the exit port 8 covers all OQs of input ports 0~port143 whose target is port 8; after grouping, each OQ group is configured with an independent statistical module to ensure that the subsequent state statistics result is strictly bound to the exit port, avoiding misjudgment caused by cross-port interference.

[0066] 2OQ group per-beat statistics monitoring:

[0067] 1 "beat" is defined as one clock cycle of the interconnection exchange chip, and the clock cycle is consistent with the interconnection clock cycle of the GPU, ensuring that the statistical frequency is synchronized with the credit interaction cycle of the GPU, avoiding state misreading caused by clock deviation. The statistical module performs two core judgments on each OQ group every beat, as shown in Figure 2 Figure 2 The structure and flowchart of the OQ grouping statistical monitoring are shown in the figure, which clearly shows the grouping logic and statistical judgment steps:

[0068] Data existence judgment: If at least one OQ in an OQ group stores a message (regardless of the size of the cache), it is determined that the OQ group "has data to be scheduled". For example, in the OQ group of the export port 8, only the input port 10 points to the OQ of port 8, which stores 3 messages, and the rest of the input ports point to the OQ of port 8 without messages, and the statistical module still marks the OQ group "has data to be scheduled";

[0069] Scheduling failure judgment: If all OQs in an OQ group cannot schedule messages because they have not received the credit returned by the GPU (none of the OQs can trigger message sending), it is determined that the OQ group "fails to schedule due to lack of credit". For example, in the OQ group of the export port 8, all OQs pointed to by the input ports 0~143 of port 8 have not received credit, and the statistical module marks the OQ group "scheduling failure" and accumulates the number of consecutive failures.

[0070] After each beat, the statistical module generates a corresponding record of "beat number-data existence state-scheduling failure state-consecutive failure number", which provides real-time data support for subsequent fault judgment.

[0071] OQ group fault judgment:

[0072] This embodiment adopts "immediate credit return" (without delay), and the fault trigger threshold is set to 3 beats by default - when the GPU is working normally, it will return credit every beat, and the probability of failure without credit for 3 consecutive beats is > 99%, which can effectively exclude temporary no credit caused by accidental factors. When the GPU connected to the export port 8 stops returning credit due to hardware failure, the OQ group of the export port 8 starts to meet "existence of data to be scheduled" and "failure to schedule due to lack of credit" from the first beat, and the number of consecutive failures is accumulated from 1; At the third beat, the number of consecutive failures reaches the threshold, and the fault judgment module formally judges that the GPU connected to the export port 8 is faulty, triggering the subsequent blocking release process.

[0073] Drop mode register enable:

[0074] ​The fault judgment module sends a command to the output port control unit of Crossbar to set the Drop mode register corresponding to the export port 8 to an enabled state. The register is a 1-bit configuration unit, only allowing the fault judgment module to write the state (to ensure configuration safety), and the message processing module of the export port 8 reads the state of the register every beat (to ensure real-time synchronization of the state), avoiding the misapplication of fault processing instructions to other normal export ports.

[0075] The export port enters the Drop mode and returns credits:

[0076] After the message processing module of the export port 8 reads that the Drop mode register is enabled, it immediately enters the Drop mode: discarding the messages in its own cache that are targeted at the port 8 in the order of “first-in, first-out”, and returning one credit to the OQ group of the export port 8 for each discarded message. The number of returned credits is strictly consistent with the number of discarded messages, ensuring credit balance and avoiding redundancy or loss. For example, after discarding 12 messages, the export port 8 returns 12 credits to the OQ group, and the available credit value of the OQ group increases from 0 to 12, providing the condition for scheduling cached messages.

[0077] OQ group scheduling and previous-stage credit feedback:

[0078] After the OQ group of the export port 8 obtains 12 credits, it immediately schedules the messages in the cache to the export port 8 (the messages are discarded by the Drop mode), and returns credits to the Input FIFO of the corresponding input port – the number of returned credits is consistent with the number of messages scheduled by the OQ group. For example, the input port 10 points to the OQ of the port 8, which schedules 3 messages, and returns 3 credits to the Input FIFO of the port 10; the input port 20 points to the OQ of the port 8, which schedules 2 messages, and returns 2 credits to the Input FIFO of the port 20, forming a credit closed loop between the OQ group and the Input FIFO.

[0079] Input FIFO removes head-of-line blocking:

[0080] After the Input FIFO of each input port obtains credits, it immediately schedules the head message: taking the Input FIFO of the input port 10 as an example, the head message is “a message targeted at the port 8”, which is scheduled to the OQ of the export port 8 (discarded) after obtaining 3 credits. Subsequent “messages targeted at normal export ports such as the port 5 and the port 12” can enter the routing lookup process and be scheduled to the OQ of the corresponding normal export port, and then be sent to the target GPU through credit interaction. All Input FIFOs of the input ports perform the same logic, and the head-of-line blocking state is completely removed within 2-3 beats.

[0081] As Figure 1 shown, Figure 1 is the overall flowchart of the scheme, covering the whole process of the embodiment from OQ statistical monitoring, fault judgment, Drop mode triggering to head blocking removal, each step forms a closed loop in Figure 1 , directly reflecting the core idea of "blocking removal based on logical statistics and control", and the Crossbar architecture is not changed throughout the process, and no Input FIFO or OQ cache components are added.

[0082] The embodiment realizes the following effects through OQ packet statistics and logical control: 1. Strong hardware compatibility: the Crossbar architecture of the interconnection switch chip is not changed throughout the process, and no Input FIFO or OQ cache components are added, avoiding the increase of hardware cost and the increase of design complexity, and adapting to large-scale switch chips such as 144 ports; 2. High blocking removal efficiency: from fault judgment to whole network blocking removal, only 5-8 beats are consumed, which is much lower than the processing delay of existing virtual output queue (VOQ) technology, meeting the core demand of "low latency" for AI training; 3. Small business impact range: only the packets of the target fault GPU are discarded, and the "ALL-TO-ONE" data set business of the normal GPU is uninterrupted, and the business continuity reaches 100%; 4. Low misjudgment rate: based on the judgment condition of "OQ group full amount without credit", combined with the 3-beat default threshold, the misjudgment rate is reduced to 0, avoiding false discarding caused by accidental no credit.

[0083] Embodiment 3: Head blocking removal and recovery in the credit delay scenario of AI inference ONE-TO-ALL business flow:

[0084] The embodiment is applied to a supercomputing node in the AI inference scenario, which uses a 144-port Crossbar interconnection switch chip, and the core business flow is "ONE-TO-ALL" - a single source GPU distributes inference model parameters to all other GPUs. Unlike embodiment 1, in this scenario, the credit uses the "ride with other packet delay return" method: the credit is not encapsulated as a packet alone, but is attached to the idle field of the data packet and returned with the data, which prolongs the credit return period to about 4 beats, and needs to be adjusted by the threshold to adapt to this scenario to avoid misjudgment. In the initial state, the source GPU (corresponding to input port 15) sends inference parameters to the exit ports 0~port143 (except port 15), the OQ group credit interaction of each normal exit is stable, and there is only a short no credit state due to credit delay.

[0085] Implementation process:

[0086] OQ group and basic statistics:

[0087] Following the OQ grouping logic of Example 1: all OQs corresponding to all egress ports are grouped by egress port, and each OQ group contains all OQs from input ports pointing to that egress port. In this example, packets from the source GPU are only routed to the OQ groups of each egress port after being buffered by the Input FIFO of input port 15. Therefore, only the OQs from input port 15 pointing to each egress port contain packets, while the OQs of other input ports are not buffered. The statistics module focuses on monitoring the OQ status associated with input port 15 to reduce invalid statistics overhead.

[0088] OQ Group Per-Shot Statistics and Credit Delay Adaptation:

[0089] The statistics module performs status statistics on the OQ group associated with input port15 every cycle:

[0090] First step: The OQ group of the exit port25 (the exit corresponding to the target GPU) (only the OQ storage of port25 pointing to port15 contains 4 packets) "has data to be scheduled", "scheduling failed" because no credit was received (credit was not returned with the data packet), the number of consecutive failures = 1;

[0091] Second beat: The state remains unchanged, and the number of consecutive failures = 2;

[0092] Third beat: The state remains unchanged, and the number of consecutive failures is 3;

[0093] Fourth beat: The state remains unchanged, and the number of consecutive failures = 4;

[0094] During this stage, the statistics module continuously records the status and does not rush to determine the fault—because the credit return cycle is 4 beats, the "no credit" in the first 4 beats is a normal delay, and the threshold adjustment is needed to eliminate false judgments.

[0095] Fault determination of OQ group based on threshold adjustment:

[0096] In response to the business characteristics of "credit-based delayed return," the fault trigger threshold has been adjusted from the default 3 steps to 5 steps. For example... Figure 3 As shown, Figure 3 The OQ fault judgment flowchart clearly shows the logic of "threshold setting - state accumulation - fault triggering": The threshold setting module adjusts the threshold to 5 beats based on the credit delay characteristics, and the state accumulation module continuously counts the number of consecutive failures; at the 5th beat, the OQ group of the exit port25 still meets the conditions of "there is data to be scheduled" and "scheduling failed due to lack of credit", and the number of consecutive failures = 5, reaching the threshold. The fault judgment module officially determines that the GPU connected to the exit port25 is faulty (at this time, the OQ groups of other normal exit ports have received credits, and the number of consecutive failures is reset to 0, further verifying that port25 is the only faulty exit).

[0097] Drop mode register configuration:

[0098] The fault judgment module sends an instruction to the output port control unit of the Crossbar to set the Drop mode register corresponding to the egress port 25 to an enabled state. As shown in Figure 4 , Figure 4 is an OQ-triggered fault recovery mechanism flowchart. The Drop mode register is a 1-bit binary register, and its physical location is in the output port control unit of the Crossbar. Only the fault judgment module of the interconnection switch chip has the write permission (to ensure configuration safety), and the message processing module of the egress port 25 has the permission to read every beat (to ensure real-time synchronization of the state). After the message processing module of the egress port 25 reads the "enabled state" at the 6th beat, it immediately triggers the Drop mode.

[0099] Continuous Drop mode and cache clearing:

[0100] After the egress port 25 enters the Drop mode, it continuously discards the messages in its own cache that are targeted for the port 25 in the order of "first-in, first-out". It returns one credit to the OQ group for every message discarded, until the cache is completely emptied (a total of 18 messages are discarded, and 18 credits are returned). At the same time, after the OQ group (the OQ of the input port 15 pointing to the port 25) obtains 18 credits, it continuously schedules the messages in the cache to the egress port 25 (all of which are discarded), and returns 18 credits to the Input FIFO of the input port 15, ensuring that the fault target messages in the previous stage cache are completely processed, avoiding residual messages causing secondary congestion.

[0101] Fault exit and communication reconstruction:

[0102] When the faulty GPU completes its software reset recovery, the state of its connected switch chip port switches from "up" to "down". As shown in Figure 5 , Figure 5 is an exit fault recovery normal flowchart. After the port state detection module of the switch chip real-time monitors that the state of the port 25 is "down", it sends an instruction to the Drop mode register to set it to a disabled state, and the egress port 25 exits the Drop mode. Then, the port of the GPU and the port 25 of the switch chip start link-up negotiation, including consistency verification of parameters such as protocol version, rate, and credit interaction mechanism. After successful negotiation, the port state returns to "up", and the credit interaction and message forwarding process return to normal.

[0103] Service recovery and stability verification:

[0104] After the communication is restored, the source GPU re-sends the inference parameters to port 25, and the packet is normally sent to the GPU after being cached by the Input FIFO, route lookup and OQ scheduling; the statistical module continuously monitors the state of each OQ group, and the number of consecutive failures of all OQ groups remains 0, verifying that the network is stable and the "ONE-TO-ALL" data distribution service is fully restored.

[0105] The present embodiment is aimed at the AI inference scene of "credit hitch delay return", and through threshold adjustment and complete recovery process, the following effects are achieved: 1. Strong scene adaptability: the threshold adjustment function makes the method adapt to the credit delay scene, and the misjudgment rate is reduced to 0, solving the problem that the prior art cannot distinguish between "temporary no credit" and "fault no credit"; 2. High processing efficiency: from the occurrence of the fault to the recovery of the service, only 12 taps (5 taps for judgment + 5 taps for discarding + 2 taps for negotiation) are consumed, and according to the clock period of the switching chip, the total time consumption is <10ns, meeting the demand of AI inference for "low latency"; 3. Strong hardware compatibility: the Crossbar architecture is not changed throughout the process, no new cache components are added, and the hardware cost is not increased, and the method is suitable for large switching chips such as 144 ports; 4. High service continuity: only the packets to the faulty GPU are discarded, and the inference parameter distribution of the source GPU to the other 143 normal GPUs is uninterrupted, the service continuity is 99.3%, and the real-time performance of the AI inference task is guaranteed.

[0106] The above only describes the preferred embodiments of the present application, and it should be understood that the present application is not limited to the forms disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concepts described herein by the above-mentioned teaching or related technical or knowledge. The modifications and changes made by those skilled in the art without departing from the spirit and scope of the present application shall be within the scope of protection of the appended claims of the present application.

Claims

1. A method for head-of-line blocking of interconnect switching chips in AI computing nodes, characterized in that, Includes the following steps: Step 1: Group the OQ corresponding to each output port in the Crossbar architecture of the interconnect switching chip according to the output port to obtain OQ groups that correspond one-to-one with the output port. Each OQ group contains all OQs that input ports point to that output port. Step 2: Perform statistical monitoring on each OQ group every time, and record whether there is any data to be scheduled in the OQ group and whether the scheduling fails due to lack of credit. Step 3: Determine if the fault condition is met: If the OQ group corresponding to a certain exit port meets the condition of having data to be scheduled and the scheduling fails due to lack of credit for N consecutive times, then the GPU connected to that exit port is determined to be faulty, where N is the fault trigger threshold. Step 4: For the exit port that is determined to be faulty, set the Drop mode register of the exit port to the enabled state in Crossbar; Step 5: After the exit port reads that the Drop mode register is enabled, it enters Drop mode, discards the packets in its own buffer, and returns credits to the corresponding OQ group. Step 6: After obtaining credits, the OQ group schedules the packets in the buffer to the egress port and simultaneously returns credits to the Input FIFO of the interconnect switching chip; Step 7: After the Input FIFO obtains the credit, it schedules the message at the head of the queue to the corresponding OQ group, thus resolving the problem of subsequent normal messages being unable to be scheduled due to the blocking of the message at the head of the queue, and realizing the unblocking of the message at the head of the queue.

2. The method according to claim 1, characterized in that, In step 2, the time for each beat is defined as follows: one beat corresponds to one clock cycle of the interconnect switching chip, and this clock cycle is consistent with the interconnect clock cycle of the GPU in the AI ​​computing node.

3. The method according to claim 1, characterized in that, The steps for setting the fault trigger threshold N in step 3 include: N is set to 3 beats by default; if the credit in the AI ​​computing node uses the method of piggybacking on other messages to return with a delay, then N is set to 5-8 beats.

4. The method according to claim 1, characterized in that, The specific steps of statistical monitoring in step 2 include: Step 21: Determine if there is any data to be scheduled in the OQ group: If at least one OQ in the OQ group stores a message, then it is determined that there is data to be scheduled in the OQ group. Step 22: Determine if the scheduling of the OQ group fails due to lack of credit: If all OQs in the OQ group are unable to schedule messages because they have not received credit from the GPU, then the scheduling of the OQ group is determined to have failed due to lack of credit.

5. The method according to claim 1, characterized in that, The steps for setting the Drop mode register in step 4 include: the Drop mode register is located in the output port control unit of the Crossbar and is a 1-bit register; after the fault judgment module of the interconnect switching chip determines that the GPU corresponding to the output port is faulty, the register is set to the enabled state; the message processing module of the output port reads the state of the register every time to determine whether to enter Drop mode.

6. The method according to claim 1, characterized in that, The specific steps for returning credits in step 5 include: the number of credits returned by the exit port is consistent with the number of dropped messages; the number of credits returned by the OQ group to the upstream Input FIFO is consistent with the number of messages scheduled to the exit port by the OQ group.

7. The method according to claim 1, characterized in that, It also includes step 8: After the exit port enters Drop mode, it continues to perform the message drop and credit return operation until there are no messages to be dropped in the cache of the exit port, ensuring that all front-end cache messages pointing to GPU failure are processed.

8. The method according to claim 1, characterized in that, The process also includes step 9: After the faulty GPU completes its own fault recovery, the port status of the GPU connected to the interconnect switch chip is switched to down; after the interconnect switch chip detects that the port status is down, it sets the Drop mode register of the corresponding egress port to the disabled state, and the egress port exits Drop mode; then the GPU and the port of the interconnect switch chip start link-up negotiation, and normal communication is restored after the negotiation is successful.

9. The method according to claim 1, characterized in that, Steps 1 to 7 are adapted to the business flow scenarios of AI computing nodes. These business flow scenarios include, but are not limited to, the ALL-TO-ONE scenario of multiple GPUs distributing data to a single GPU set and the ONE-TO-ALL scenario of a single GPU distributing data to multiple GPUs.

10. The method according to claim 1, characterized in that, During the execution of steps 1 to 7, the Crossbar architecture of the interconnect switching chip is not modified, and no new Input FIFO or OQ cache components are added. The head-of-line blocking is resolved only through logic-level statistics, judgment and control.

Images