Model training method, supernode system, electronic device, medium, and program product

Abstract

The application provides a model training method, a supernode system, an electronic device, a medium and a program product. The model training method comprises: performing reverse propagation calculation by a first supernode to obtain a local model gradient corresponding to a tensor held by the first supernode according to a received error and the tensor held by the first supernode; sending the local model gradient to a connected line card by a plurality of first supernodes; slicing the received local model gradient into a plurality of cells by the line card, and spraying the plurality of cells to each net board; sending the received plurality of cells to a line card connected to a second supernode by the net board; recombining the received cells into a local model gradient by the line card connected to the second supernode, and sending the recombined local model gradient to each second supernode; and updating a locally stored model parameter according to the received local model gradient and the locally stored local model gradient by the second supernode. The application can improve the reliability of model training of the supernode system.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model training method, a supernode system, electronic equipment, media, and program products. Background Technology

[0002] With the rise of Artificial Intelligence (AI) and High Performance Computing (HPC), data centers are increasingly demanding larger-scale computing clusters. In large-scale computing clusters, frequent and high-throughput data transfers are required between Graphics Processing Units (GPUs).

[0003] In related technologies, multiple GPUs are aggregated into a supernode system, and the supernodes are connected point-to-point with multiple other supernodes in the same supernode system through multiple independent physical links. The network architecture of this supernode system is complex. When model training is performed in this supernode system, the training interruption time is too long when a supernode's running state is abnormal, resulting in insufficient reliability. Summary of the Invention

[0004] This application provides a model training method, a supernode system, an electronic device, a medium, and a program product, which simplifies the connection structure between supernodes, shortens the time when supernodes are in abnormal operating states, and thus improves the reliability of the supernode system.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a model training method applied to a supernode system. The supernode system includes: multiple supernodes, multiple line cards, and multiple network boards. The line cards and the network boards constitute a fully interconnected structure. Each of the multiple supernodes is connected to at least one of the line cards.

[0007] The method includes:

[0008] The first supernode performs backpropagation calculation based on the received error and the tensor it holds to obtain the local model gradient corresponding to the tensor held by the first supernode. The tensor is obtained by splitting the parameters of the model to be trained, and the first supernode is any one of the plurality of supernodes.

[0009] The local model gradient is sent to the connected line card through multiple first supernodes;

[0010] The line card slices the received local model gradient into multiple information cells, and sprays the multiple information cells onto each of the meshes;

[0011] The network board sends the received multiple cells to the line card connected to the second supernode;

[0012] The line card connected to the second supernode reassembles the received information cells into the local model gradient, and sends the reassembled local model gradient to each of the second supernodes, wherein the second supernode is a supernode other than the first supernode among the plurality of supernodes;

[0013] The second supernode updates the locally stored model parameters based on the received local model gradient and the locally stored local model gradient.

[0014] This application provides a supernode system for model training, including:

[0015] The first rack includes multiple supernodes.

[0016] The second rack includes multiple line cards.

[0017] The third rack includes multiple mesh panels.

[0018] The supernode includes:

[0019] The first supernode is used to perform backpropagation calculation based on the received error and the tensor it holds to obtain the local model gradient corresponding to the tensor held by the first supernode, wherein the tensor is obtained by splitting the parameters of the model to be trained, the first supernode is any one of the plurality of supernodes, and to send the local model gradient to the connected line card.

[0020] The second supernode is used to update the locally stored model parameters based on the received local model gradient and the locally stored local model gradient.

[0021] The line card includes:

[0022] The first line card is used to slice the received local model gradient into multiple information cells and spray the multiple information cells onto each of the mesh plates;

[0023] The second line card is used to reassemble the received information cells into the local model gradient, and send the reassembled local model gradient to each of the second super nodes;

[0024] The network board is used to send the received plurality of cells to the second line card connected to the second supernode.

[0025] This application provides an electronic device for use as a control node in a supernode system, the electronic device comprising:

[0026] Memory is used to store executable instructions or computer programs.

[0027] The processor, when executing computer-executable instructions or computer programs stored in the memory, controls the supernodes, line cards, and network boards in the supernode system to implement the above-mentioned model training method.

[0028] This application provides a model training device, stored in the memory of an electronic device, comprising:

[0029] The calculation module is used to calculate the local model gradient corresponding to the tensor held by the first supernode by backpropagation based on the received error and the tensor held by the first supernode. The tensor is obtained by splitting the parameters of the model to be trained, and the first supernode is any one of the plurality of supernodes.

[0030] The first sending module is used to send the local model gradient to the connected line card through multiple first super nodes;

[0031] The spraying module is used to slice the received local model gradient into multiple information cells through the line card, and spray the multiple information cells onto each of the screen plates;

[0032] The second transmitting module is used to transmit the received plurality of cells to the line card connected to the second supernode via the network board;

[0033] The third sending module is used to reassemble the received information cells into the local model gradient through the line card connected by the second supernode, and send the reassembled local model gradient to each of the second supernodes, wherein the second supernode is a supernode other than the first supernode among the plurality of supernodes;

[0034] The update module is used to update the locally stored model parameters through the second supernode based on the received local model gradient and the locally stored local model gradient.

[0035] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for controlling the supernodes, line cards, and network boards in a supernode system when executed by a processor, thereby implementing the aforementioned model training method.

[0036] This application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, they control the supernodes, line cards, and network boards in the supernode system to implement the above-mentioned model training method.

[0037] The embodiments of this application have the following beneficial effects:

[0038] By connecting multiple line cards and network boards in the supernode system into a fully interconnected structure, complete equivalence of multiple line cards is achieved. That is, a supernode can be connected to any line card in a single-track manner and can communicate with other supernodes connected to the line card. This eliminates the need to establish physical links between supernodes in a multi-track connection mode, simplifies the network structure of the supernode system, and improves the system reliability. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the supernode system architecture provided in the embodiments of this application;

[0040] Figure 2 This is a schematic diagram of the structure of a supernode provided in an embodiment of this application;

[0041] Figure 3 This is a schematic diagram of the connection structure of the graphics processor chip inside the supernode provided in the embodiments of this application;

[0042] Figure 4 This is a structural block diagram of the electronic device provided in the embodiments of this application;

[0043] Figure 5 This is a first flowchart illustrating the model training method according to an embodiment of this application;

[0044] Figure 6 This is a schematic diagram of the second process of the model training method according to an embodiment of this application;

[0045] Figure 7 This is a schematic diagram of the third process of the model training method according to an embodiment of this application;

[0046] Figure 8 This is a schematic diagram of the fourth process of the model training method according to an embodiment of this application;

[0047] Figure 9 This is a schematic diagram of the fifth process of the model training method according to an embodiment of this application;

[0048] Figure 10 This is a schematic diagram of the internal structure of a supernode in the relevant technology shown in the embodiments of this application;

[0049] Figure 11 This is a schematic diagram of a multi-level leaf-ridge topology architecture of the related technology shown in the embodiments of this application;

[0050] Figure 12 This is a schematic diagram illustrating the structure of adding redundant supernodes to a multi-level leaf-ridge topology, as shown in the embodiments of this application.

[0051] Figure 13 This is a schematic diagram of the structure of a supernode system in an application scenario provided by an embodiment of this application;

[0052] Figure 14 This is a schematic diagram comparing the internal architecture of the supernode system provided in this application embodiment with that of a traditional chassis switch;

[0053] Figure 15 This is a schematic diagram illustrating the principle of a supernode system deployed in a standard chassis in an application scenario provided by an embodiment of this application.

[0054] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0057] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0058] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0059] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0060] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0061] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0062] 1) Supernode: A supernode is a high-density computing server unit or logical collection that integrates a far greater number of high-performance GPUs (e.g., 32, 64, or more) than a traditional standard server (e.g., 8 GPUs). Supernodes enable memory sharing and real-time communication between computing units via an internal, extremely high-speed dedicated bus (non-Ethernet protocol), forming a vast unified computing domain. Externally, it appears as a single, large network access endpoint, aiming to confine frequent computing communication within the node and reduce dependence on external networks.

[0063] 2) Supernode Cluster: A large-scale computing system consisting of multiple supernodes interconnected through a high-performance supernode network. In this cluster, hundreds or thousands of supernodes work collaboratively to run AI models with extremely large parameter counts (such as the MoE model). The cluster achieves parallel communication between nodes through a network architecture, aiming to break through the physical limits of single-point computing power and provide a global computing environment with extremely low latency and ultra-high throughput. It serves as the infrastructure for training large models.

[0064] 3) A packet is a logical data unit used for data exchange and transmission in a computer network, typically following a standardized network protocol format (such as an Ethernet frame). In this embodiment, a packet specifically refers to the raw data stream generated by a computing node (such as a server or GPU) and sent to the network access side. It has a variable length and contains header information for routing (such as source IP and destination IP) and a payload carrying service data. Packets are the primary object processed by the access side (service side) of the supernode system.

[0065] 4) Graphics Processing Unit (GPU) refers to a high-performance electronic circuit or chip specifically designed to perform parallel computing tasks, such as the core computing accelerator used for training and inference of artificial intelligence models. GPUs can efficiently handle matrix operations and floating-point operations and are the source of massive amounts of communication data (such as gradient synchronization and parameter updates). Multiple GPUs are typically combined into a supernode via a high-speed interconnect bus to collaboratively complete large-scale distributed computing tasks. GPUs can also be packaged as accelerator cards.

[0066] 5) Bandwidth refers to the maximum amount of data that a communication link or network interface can transmit per unit of time, usually measured in bits per second (bps). In this embodiment, bandwidth refers not only to the rated transmission rate of the physical interface (e.g., 400Gbps), but also specifically to the data throughput capacity between various components within the supernode system (e.g., between line cards and network boards). High bandwidth design is a key performance indicator to ensure that large-scale clusters do not experience congestion and packet loss during fully interconnected communication.

[0067] 6) A cell is a fixed-length data slice transmitted within a supernode system (device). A cell is a special data format distinct from a long "message". When the original message enters the supernode system, it is segmented and encapsulated into several fixed-length cells, with a specific sequence number and an internal addressing header (identifier of the target line card) appended.

[0068] 7) Line Card (NCP): A line card is a service access component in a supernode system (device), also known as a service access point (SAP). On one hand, it provides a standard Ethernet interface to connect to external computing resources and handles incoming and outgoing packets. On the other hand, it performs packet-to-cell conversion, slicing received packets into cells and sending them to the core switching components via internal interconnect interfaces. The line card acts as a bridge between the service plane and the switching plane, undertaking crucial functions such as traffic access, packet slicing and reassembly.

[0069] 8) Fabric Card: This refers to the core switching component in a supernode system (device), also known as an NCF. The fabric card does not directly connect to external servers or GPUs; instead, it connects to multiple line cards via a high-bandwidth internal backplane or fiber optic cable. The main function of the fabric card is to receive cells from the line cards and perform high-speed, stateless forwarding and switching based on the cell headers, delivering the cells to the destination line card. Fabric cards constitute the switching matrix of the entire supernode system, determining the system's total switching capacity and redundancy.

[0070] 9) Rack: A standardized physical framework for housing and organizing multiple server nodes or supernodes and other related electronic equipment. Its functions include providing structural support, centralized power supply, heat dissipation management, and cabling space; it is the basic physical unit for server deployment within a data center.

[0071] 10) High-speed serial bus: A communication channel for point-to-point, high-bandwidth data transmission between devices or chips. For example, a high-speed serial bus can be a bus that conforms to the fifth generation peripheral component rapid interconnect (PCIe Gen5) standard, and the channel width can be configured to 8 channels (X8).

[0072] 11) Expansion (EXP) card: A hardware board installed on a server node or supernode to enable physical connection between the supernode and other external server nodes.

[0073] 12) Full Mesh: Also known as a full mesh topology or fully connected structure, this is a network topology where every node in the network has a direct point-to-point physical connection to all other nodes. For example, a supernode's GPU has a direct point-to-point physical connection to all other GPUs, or a line card has a direct point-to-point physical connection to all network boards. A significant characteristic of this structure is that communication between any two nodes requires only one hop (Single Hop), thus providing extremely low communication latency and extremely high path redundancy.

[0074] 13) A checkpoint is a snapshot or archive of the model's state that is periodically saved during the model training process.

[0075] In related technologies, during MoE training, model parameters are distributed and stored across multiple experts on different supernodes. Data streams (tokens) need to be dynamically routed globally to the corresponding experts for computation based on a gating mechanism. This results in large-scale data distribution and aggregation (Dispatch & Combine) across nodes during each layer's forward and backward propagation. Because this process has an extremely high communication-to-computation ratio and is highly sensitive to synchronization latency, insufficient network interconnection performance can lead to prolonged data transfer across nodes, causing computing units to idle for extended periods and severely reducing training efficiency. Therefore, a high-bandwidth, low-latency, physically fully connected network is necessary to support the non-blocking, high-speed exchange of massive amounts of data between any supernodes to match the throughput requirements of sparse model training. High-bandwidth, low-latency fully connected relationships need to be established between supernodes. Ideally, to support an all-to-all aggregated communication mode, any two supernodes in the cluster should be physically directly connected to ensure data transmission without intermediate hops and bandwidth convergence. However, as the cluster size increases (e.g., reaching thousands of nodes), the number of cables and ports required to achieve a physically mesh-like full interconnection grows exponentially, making it virtually infeasible in engineering. To address this, a multi-level leaf-spine topology is proposed. This architecture deploys multiple high-density core switches and uniformly connects the uplinks of each supernode to the left and right core switches, constructing a non-blocking switching network. This switching network effectively simulates the fully connected structure between supernodes, allowing for model training within this simulated fully connected structure.

[0076] However, in order to maximize network throughput and control construction costs, the downlink ports of the core switch in the leaf-spine topology architecture are usually strictly planned and fully occupied by worker nodes (supernodes), preventing redundant nodes from connecting to the supernode system. When a worker node (supernode) in the supernode system fails, the system cannot automatically switch over. Maintenance personnel must manually disconnect the connection cable of the failed node and reconnect the physical resources to a new redundant supernode, or be forced to change the original network topology. Because model training has the characteristic of data parallelism, that is, all GPUs must perform gradient synchronization at the end of each iteration step, if one supernode is abnormal, the entire training job will be interrupted. The leaf-spine topology architecture relies on cold backup through physical changes, which increases the mean time to repair (MTTR) and reduces the mean time between failures (MTBF) when large-scale intelligent computing clusters (supernode systems) are training models, thus reducing the stability of the supernode system.

[0077] This application provides a model training method, a supernode system, an electronic device, a medium, and a program product, which simplify the connection structure between supernodes, shorten the time when supernodes experience abnormal operating states, and thus improve the reliability of the supernode system. The exemplary application of the model training method provided in this application is described below. The electronic device provided in this application can be implemented as a server. The exemplary application when the device is implemented as a server will be described below.

[0078] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the supernode system 100 provided in this application embodiment. To perform model training operations in the supernode system, a supernode application can be provided. For example, this supernode application can be an application specifically for managing model training within the supernode. The supernode system 100 provided in this application embodiment includes at least a supernode 210, a second rack 300, and a third rack 400 disposed in a first rack 200, and... Figure 1 The electronic device 500 (not shown) includes a second cabinet 300 comprising multiple line card groups, each line card group comprising at least one line card; a third cabinet 400 comprising at least two network boards for distributing interface configuration files to the line cards, wherein the interface configuration files are used to enable the interfaces of the line cards; and the electronic device 500 may be implemented as a server for controlling data transmission between the supernode 210, the second cabinet 300, and the third cabinet 400.

[0079] See Figure 2 Each supernode 210 includes multiple graphics processing chips 220, which are connected via a high-speed serial bus. A high-speed serial bus refers to a point-to-point serial communication link with a transmission rate of 32GT / s or higher (such as the PCIe 5.0 standard). Furthermore, the supernode 210 also includes multiple external interface cards, which connect to line cards outside the supernode 210. In a specific embodiment, this high-speed serial bus supports differential signal transmission.

[0080] See Figure 3 , Figure 3 This is a schematic diagram illustrating the connections between multiple graphics processing unit (GPU) chips 220 within a supernode provided in this embodiment. Taking supernode S0 as an example, supernode S0 includes eight GPU chips: G0, G1, G2, G3, G4, G5, G6, and G7. Each pair of GPU chips within supernode S0 is connected via a high-speed serial bus (…). Figure 3(Seen in solid lines) Connections are made, for example, graphics processor chip G0 is connected to graphics processor chips G1, G2, G3, G4, G5, G6, and G7 via a high-speed serial bus. Within supernode S0, the eight graphics processor chips G0, G1, G2, G3, G4, G5, G6, and G7 are directly connected in pairs, forming an internal mesh topology. In this structure, there is a direct point-to-point (P2P) physical connection between any two graphics processor chips (GPUs), with a data transfer bandwidth of 256 Gbps (32 GB / s) in one direction.

[0081] Here, the external expansion interface card is a specific function hardware board installed inside the supernode 120. The external expansion interface card is used to connect to the graphics processor chip inside the supernode via a high-speed serial bus, and provides a physical interface for connecting high-speed cables, thereby enabling communication with the line card.

[0082] In some embodiments, the model training method is applied to a supernode system 100, which includes: multiple supernodes, multiple line cards, and multiple network boards. The line cards and network boards constitute a fully interconnected structure, and each supernode is connected to at least one line card. The model training method includes: calculating the local model gradient corresponding to the tensor held by the first supernode through backpropagation based on the received error and the tensor it holds, wherein the tensor is obtained by partitioning the parameters of the model to be trained, and the first supernode is any one of the multiple supernodes; through multiple first supernodes, The local model gradient is sent to the connected line card; the line card slices the received local model gradient into multiple information cells and sprays the multiple information cells onto each mesh; the mesh sends the received multiple information cells to the line card connected to the second supernode; the line card connected to the second supernode reassembles the received information cells into a local model gradient and sends the reassembled local model gradient to each second supernode, where the second supernode is a supernode other than the first supernode; the second supernode updates the locally stored model parameters based on the received local model gradient and the locally stored local model gradient.

[0083] In some embodiments, the electronic device may be an electronic device, see [link to documentation]. Figure 4 , Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Figure 4 The illustrated electronic device includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components of the electronic device are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 4 The general labeled all buses as Bus System 540.

[0084] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0085] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, other input buttons, and controls.

[0086] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 510.

[0087] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0088] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0089] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0090] The network communication module 552 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0091] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0092] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 4 A model training apparatus 555 for a network switching system, stored in memory 550, is shown. This apparatus can be software in the form of programs and plug-ins, and includes the following software modules: a calculation module 5551, a first sending module 5552, a spraying module 5553, a second sending module 5554, a third sending module 5555, and an update module 5556. These modules are logically connected and can therefore be arbitrarily combined or further divided according to their implemented functions. The functions of each module will be described below.

[0093] See Figure 5 , Figure 5 This is a schematic diagram of the model training process provided in the embodiments of this application. Figure 1 , will combine Figure 5 The steps shown are explained as follows: Figure 5 As shown, the method includes steps 101 to 106. The model training method is applied to a supernode system, and data transmission between supernodes, line cards, and network boards is controlled by electronic devices such as control nodes in the supernode system.

[0094] The supernode system includes:

[0095] The first rack includes multiple supernodes.

[0096] The second rack includes multiple line cards.

[0097] The third rack includes multiple mesh panels.

[0098] Line cards and network boards form a fully interconnected structure, with each of the multiple supernodes connected to at least one line card.

[0099] Here, the electronic device can be implemented as a control node (NCC), which establishes a connection with the line card's management interface through an out-of-band management network. Using a network configuration protocol, it sends an interface configuration file containing port rates, forward error correction modes, and flow control parameters to the line card's central processing unit (CPU). The CPU parses the interface configuration file and writes the configuration parameters into the switching chip register, thereby activating the line card's physical layer and data link layer functions. For example, the control node sends a JSON file containing the instruction "Enable interface 1, rate 400Gbps" to line card A. Line card A executes this instruction, putting port 1 into forwarding state.

[0100] In addition, the control node can periodically obtain the network connection status through the link layer discovery protocol, automatically generate interface configuration files adapted to the current physical connection relationship, and distribute them in batches to newly launched line cards after verification. If the configuration application fails, it will automatically roll back to the previous available version.

[0101] Here, the configuration file contains key operating parameters such as interface rate, forward error correction mode, maximum transmission unit, and flow control strategy. The line card only performs hardware-level initialization and link negotiation based on the parameters after receiving the interface configuration file and parsing and loading it into the chip register, thereby officially enabling the interface to enter the message transmission and reception state.

[0102] In step 101, the first supernode performs backpropagation calculation based on the received error and the tensor it holds to obtain the local model gradient corresponding to the tensor held by the first supernode.

[0103] Here, the tensor is obtained by splitting the parameters of the model to be trained, and the first supernode is any one of multiple supernodes.

[0104] Here, a tensor refers to a multidimensional array data structure involved in the training of a deep learning model. It includes input data for forward propagation, activation values ​​for intermediate layers, model weights, and gradient information for backpropagation. Specifically, tensors can be categorized by their dimensionality (rank) into scalars (0-dimensional), vectors (1-dimensional), matrices (2-dimensional), and arrays with higher dimensions. In distributed training scenarios, especially large-scale model training, to overcome the limitations of single-device memory, the complete model parameters are typically divided into multiple parts and stored on different supernodes. Therefore, the tensor held by the first supernode specifically refers to the parameter tensor corresponding to the model shard it is responsible for.

[0105] Here, the supernode system can use a gating mechanism to assign tensors to designated supernodes. The gating mechanism, in a hybrid expert model or dynamic routing network (supernode system), is a routing strategy component used to determine which expert (i.e., supernode) the input data should be sent to for processing. It can be a lightweight neural network that outputs a probability distribution or a sparse index vector to indicate the destination of each data sample.

[0106] Here, backpropagation refers to a method that uses the chain rule to efficiently calculate gradients. Its core idea is to determine the direction and magnitude of parameter updates based on the partial derivative of the loss function with respect to each parameter.

[0107] Here, the Local Model Gradient refers to the gradient value calculated based on the training data batch or micro-batch processed by the current first supernode, with respect to the model parameters held by that node. The gradient itself is a vector, its direction representing the direction of the fastest growth of the loss function, and its magnitude representing the rate of growth. During backpropagation, the Local Model Gradient is calculated layer by layer using the chain rule, that is, the error signal of each layer is multiplied by the input activation value of that layer to obtain the partial derivatives of the weight parameters of that layer. Since each supernode only processes a portion of the training data (model training is usually in data parallel mode), the gradient it calculates is the local gradient. It needs to be aggregated with the gradients of other nodes through subsequent communication steps to obtain the global gradient used to update the global model parameters.

[0108] In some embodiments, the received error can be calculated based on the difference between the model's output predicted value and the true label value using a preset loss function (e.g., cross-entropy loss function or mean squared error loss function). Backpropagation calculation can be implemented by using a deep learning framework and its provided Automatic Differentiation engine to propagate the error signal layer by layer from the output layer to the input layer, and calculating the gradient vector of the weight parameters for each layer's model shard for the first supernode. For example, for the... Layer weight matrix Its gradient It can be done through formula The calculation yielded, where For the first Layer error term, This is the activation output of the previous layer.

[0109] In step 102, the local model gradient is sent to the connected line card through multiple first supernodes.

[0110] Here, the first supernode encapsulates the computationally generated memory data (local model gradients) into Ethernet frames through its network adapter. The Ethernet frames are then transmitted through the external interface card within the supernode and encoded into message data packets by the physical layer of the external interface card. Finally, they are transmitted through the first cable to the physical interface of a specific line card in the line card group. This interface receives the data stream of message data packets using a preset service access bandwidth.

[0111] In some embodiments, the first rack also includes at least one third supernode, see [link to relevant documentation]. Figure 6 , Figure 6It is shown that steps 107 to 109 can also be performed when the multiple supernodes include at least one third supernode, which is connected to at least one arbitrary line card.

[0112] Here, since the line cards and network boards have a fully connected structure, all line cards are equivalent. This means that multi-track connections are no longer needed between the supernodes in this application (i.e., each track consists of a group of supernodes and a switch; when communication is needed, the supernode selects one track for data transmission, and load balancing is achieved). Connecting a supernode to any line card (including connecting to one line card or multiple line cards) is equivalent to connecting to any supernode. Therefore, connecting a third supernode to any line card achieves redundant connection.

[0113] In step 107, the running status of the first supernode is detected.

[0114] Here, the operating status refers to a series of indicators that reflect whether the first supernode is currently working normally and whether it has the ability to perform computing. For example, it can be judged by heartbeat signals, GPU memory usage, temperature, power status, and network connectivity.

[0115] Here, electronic devices can run health monitoring services, periodically receive heartbeat messages from supernodes or query physical link status through out-of-band management networks. When no heartbeat is received for N consecutive cycles or a port physical signal loss is detected, the supernode is determined to be in an abnormal operating state. Alternatively, the supernode itself can detect its operating status.

[0116] In step 108, when the third supernode detects an abnormal running state of the first supernode, it loads the tensor held by the first supernode and calculates the corresponding local model gradient based on the tensor.

[0117] Here, the third supernode, acting as a redundant backup node in the supernode system, is typically in standby mode but possesses computing capabilities. By updating the routing table of the gating mechanism or reconfiguring the output mapping of the gating network, data streams and computing tasks originally destined for the first supernode, which is experiencing operational issues, can be redirected (Route Update) and loaded onto the third supernode.

[0118] In step 109, the third supernode sends the local model gradient to the connected line card.

[0119] In this embodiment, for the local model gradient in the third supernode, it enters the switching network (composed of line card and network board) only through the physical link of the line card connected to the third supernode. It can send the local model gradient to any supernode (multiple second supernodes) in the supernode system without the need for multi-track parallel distribution.

[0120] In related technologies, after a failure is detected in the supernode system, restoring the entire system's data to the checkpoint state before resuming calculations typically takes tens of minutes. Due to the parallel and synchronous nature of data during model training, the model parameters held by all supernodes cannot complete a single iteration during the recovery time, reducing GPU utilization efficiency, extending the time spent using GPUs for model training, and impacting the economic benefits of GPU use. When a failure is detected in the first supernode, the third supernode can quickly take over the computational tasks through logical mapping, achieving a seamless switch at the business level. This process allows the system to avoid the lengthy recovery time required to interrupt training tasks, roll back to the previous checkpoint, and reload data. Therefore, while ensuring training continuity, it significantly improves the mean time to repair (MTTR) and mean time between failures (MTBF) of the supernode system.

[0121] This application embodiment monitors the operating status of the first supernode in real time. When the first supernode's operating status is abnormal, a third supernode (redundant node) in the system is scheduled to quickly load the tensors held by the first supernode and take over its computational tasks using a gating mechanism. Specifically, this mechanism allows the third supernode to replace the first supernode in accessing the supernode network and completing the calculation and transmission of local model gradients simply by connecting to any available line card in the network via a single track, without needing to configure complex multi-track parallel connections (i.e., full connection between multiple supernodes) or reserve specific physical slots. This reduces the deployment cost of redundant system resources, network card port consumption, and cabling difficulty, and enables flexible access of backup nodes. Furthermore, since the third supernode achieves fault takeover through tensor loading at the logical level (such as gating redirection), this process does not require modifying the physical traffic allocation weights of the underlying network switching equipment or triggering complex routing protocols to reconverge, effectively avoiding latency and jitter caused by large-scale network configuration changes, and ensuring the instantaneous and seamless fault switching. Therefore, even if a single first supernode malfunctions, it will not affect the model training process of other normal supernodes, ensuring the continuity and overall stability of large-scale distributed training tasks (which are usually extremely time-consuming and costly) under hardware failure scenarios.

[0122] In some embodiments, see Figure 7 , Figure 7 It is shown that in the case where the supernode system also includes a restored first supernode, which is connected to at least one arbitrary line card, after step 109, the restored first supernode can be added to the model training task through steps 1010 to 1011.

[0123] In step 1010, if the first supernode detects that the running status is normal, the restored first supernode loads the tensor held by the third supernode and calculates the corresponding local model gradient based on the tensor.

[0124] Here, when the previously abnormal first supernode is detected to be running normally, it indicates that the first supernode has completed a restart or hardware repair, and the supernode system triggers a switchback process. At this time, the recovered first supernode reloads the tensors held by the third supernode, that is, it reloads the latest model parameter tensors and training state, and continues the model training task.

[0125] As an example, after the first supernode starts, it can proactively send a registration request or heartbeat signal to the control node. Once the control node verifies that the first supernode's hardware health (such as GPU, memory, and network) is correct, it marks the recovered first supernode as "available." Then, the control node sends a pause command to the third supernode, transmitting the latest model weight tensor and optimizer state from the third supernode's GPU memory to the recovered first supernode via Remote Direct Memory Access (RDMA) or a high-speed network. Next, using a gating mechanism, the data stream previously redirected to the third supernode is redirected back to the first supernode's physical address. Finally, the first supernode receives new batches of training data, performs forward and backward propagation using the synchronized tensors, calculates the local model gradient, and thus formally rejoins the supernode system's model training task.

[0126] In step 1011, the local model gradient is sent to the connected line card through the recovered first supernode.

[0127] The connected line card can be an existing line card connected before the first supernode failed, or any available line card in the fully interconnected structure. Data is sent through a single line card interface, and the cell spraying mechanism inside the line card ensures load-balanced data transmission within a single track.

[0128] In some embodiments, the transmission process is implemented as follows: the first supernode encapsulates the local model gradient tensor into a data packet, adds corresponding routing header information (such as the target supernode), and sends it to the connected line card via the physical link. After receiving the data packet, the line card slices and forwards it, and finally delivers the local model gradient to all the second supernodes participating in the training, completing a full round of iterative synchronization.

[0129] This embodiment of the application connects the recovered first supernode to any line card in the supernode system. By loading tensors, it can rejoin the training task and restore its original computational function without strictly restoring the physical connection topology before the failure (since each line card is equivalent, it can be connected as long as it has available bandwidth), nor does it require finding and reconnecting to the original specific network interface (re-establishing a full physical link connection). This means that after equipment repair or replacement, maintenance personnel do not need to spend time and effort restoring the complex original multi-track parallel connection or physically resetting the cables one-to-one. They only need to connect the recovered node to any currently available network port (even a single-track connection), and the system can automatically identify and restore its function through software-level configuration (such as gating mechanisms or logical mapping). This design greatly simplifies the process of bringing the failed node back online and reduces the difficulty of on-site maintenance, achieving decoupling and flexible reuse of hardware connections, thereby significantly shortening the time required for the system to recover from a degraded operating state to a fully functional state.

[0130] In some embodiments, see Figure 8 In the post-training phase, before step 102, local model gradients can also be obtained through steps 1012 to 1014.

[0131] In step 1012, multiple first supernodes perform inference calculations based on the received prompt words to obtain the response to the prompt words.

[0132] Here, multiple first supernodes act as carriers of pre-trained models (i.e. models that need to be trained later), receive prompts from the training dataset or user input, and collaboratively perform distributed inference computation to generate data (Tokens) one by one in an autoregressive manner, ultimately obtaining a complete response.

[0133] Here, prompts can refer to the input text context sequence; responses refer to the subsequent text sequence generated by the model based on the probability distribution.

[0134] In some embodiments, the inference process can be implemented by dividing the model's parameters into multiple tensor slices and distributing these tensor slices across the GPU memory of multiple first supernodes (i.e., model parallelism). Each first supernode queries its local embedding and attention layer weights based on the received prompt token index, calculates key-value pairs, and performs forward computation. At each generation step, each supernode aggregates the output probability distribution through ensemble communication and selects the next token using kernel sampling or a greedy search strategy until an end-of-line symbol is generated or the maximum length is reached.

[0135] In step 1013, the first supernode calculates the policy gradient of the model based on the score of the response.

[0136] Here, the score reflects the value of the response in terms of usefulness, safety, or human preference; the policy gradient is a vector that indicates the direction of adjustment of the model parameters, aiming to maximize the expected return.

[0137] In step 1014, the first supernode calculates the local model gradient corresponding to the tensor held by the first supernode through backpropagation based on the policy gradient and the tensor it holds.

[0138] Here, the local model gradient refers to the amount of model parameters held by the current supernode that need to be adjusted in order to optimize the strategy and obtain a higher score.

[0139] As an example, starting from the output layer, the policy gradient is propagated in the reverse direction of the network, passing through the activation function and linear transformation of each layer. Each first supernode only computes the gradient corresponding to the tensor slice held in its GPU memory. After computation, the generated local model gradient is stored in the gradient buffer of GPU memory so that it can be sent to the line card in subsequent step 102, thereby being sent to other supernodes via the line card, and synchronized and aggregated with the gradients of other supernodes to complete a full post-training parameter update iteration.

[0140] This application's embodiments effectively support the post-training stage of the model (such as human feedback-based reinforcement learning RLHF) by introducing prompt word inference, response scoring, and policy gradient calculation mechanisms, and ensure that the model parameters distributed across multiple first supernodes can be globally and synchronously updated based on policy gradients. Furthermore, this scheme avoids interruption or crashes of the entire post-training process due to single-point failures, ensuring the continuity and stability of model alignment and optimization processes, and improving the system reliability in the post-training stage.

[0141] In step 103, the line card slices the received local model gradient into multiple information cells and sprays the multiple information cells onto each screen.

[0142] Here, the slicing process can involve the line card's packet parsing engine extracting the destination IP address of the incoming packet (local model gradient), querying the local forwarding table (or the control node's forwarding table) to obtain the system-level identifier corresponding to the second supernode, and then the slicing engine dividing the variable-length packet into fixed-length data blocks and encapsulating them into a cell header containing the system-level identifier and sequence number. Finally, the scheduler uses a round-robin algorithm to evenly send the generated cells to each connected network board. For example, line card A divides a 1000-byte packet destined for supernode B into four 256-byte cells, writes the supernode B's identifier ID: 05 into the cell header, and sends the cells evenly to network boards 1, 2, 3, and 4.

[0143] In some embodiments, the line card can also slice and schedule based on service priority. The line card identifies the quality of service level of incoming packets, marks the priority field in the cell header when generating cells, and controls high-priority cells to enter the high-bandwidth network board channel first, ensuring low-latency transmission of critical control commands.

[0144] Here, spraying, also known as cell spraying, is a technique that achieves network load balancing and low latency by dividing data packets into smaller cells and sending them in parallel through multiple paths. It can maximize the use of the high-bandwidth interconnect network in the supernode system and ensure that computing units such as GPUs can efficiently exchange data when performing large-scale parallel computing (such as multiple GPUs training models in parallel).

[0145] It's important to note that in traditional networks, a data stream is typically confined to a single path, which can easily lead to network congestion. Cell spraying technology breaks a data stream into multiple cells, which are then sent simultaneously across multiple physical links within the cluster, thus avoiding single-path congestion and fully utilizing all available network bandwidth.

[0146] In some embodiments, slicing and spraying can be implemented as follows: the network processor (NPU) or FPGA logic within the line card uses a slicing algorithm to divide large gradient data packets into multiple fixed-length cells of 64 bytes or 128 bytes, and employs a cell-based dynamic load balancing algorithm (such as Round-Robin polling or a queue depth-based weighted random algorithm) to evenly distribute these cells to all available network board ports connected, thereby maximizing the utilization of switching bandwidth and avoiding single-path congestion.

[0147] In some embodiments, see Figure 9 , Figure 9 The line card in step 103 slices the received local model gradient into multiple information cells and sprays the multiple information cells onto each screen, which can be achieved through steps 1031 to 1032.

[0148] In step 1031, the line card slices the received local model gradient into multiple information cells and adds the identifier of each second supernode to each information cell.

[0149] The identifier of the second supernode is used to control the network board to send the received cell to the line card connected to the second supernode.

[0150] Here, the identifier of the second supernode refers to the network address or logical number (such as IP address, MAC address or global Rank ID) that uniquely identifies the destination node in the distributed training system. This identifier is used to indicate the final destination of the data packet in the switching network.

[0151] In some embodiments, the process of adding identifiers can be implemented as follows: the line card queries the locally maintained Group Member List or routing table to obtain the identifiers of all second supernodes that need to receive the gradient, and encodes these identifiers into the cell header in the form of unicast or multicast. For example, if multicast is used, a multicast group ID can be set in the cell header, which is associated with the port mappings of all second supernodes.

[0152] In step 1032, the line card sprays cells with the identifier of the second supernode onto each stencil.

[0153] In some embodiments, information cells can be sent to network board 1, network board 2, ..., network board N in a round-robin manner. After all network boards have been polled, the polling is repeated to ensure that each network board processes approximately the same number of information cells, thereby avoiding congestion on a single network board.

[0154] It should be noted that traditional Ethernet switching uses flow-based hash routing (ECMP). When faced with long or elephant flows in scenarios such as AI training, this can easily lead to a "hash polarization" phenomenon, where one physical link is congested while others are idle, resulting in wasted bandwidth. This application's embodiment achieves fine-grained load balancing at the cell level by slicing packets into fixed-length cells and using a round-robin mechanism to evenly distribute these cells across all connected network boards. Regardless of the characteristics of the upper-layer service flow, it ensures absolute load balance across all uplinks, eliminating hotspot congestion and maximizing the network's effective bandwidth utilization to its physical limit (close to 100%), thus constructing a non-blocking switching network. Furthermore, it employs a stateless, round-robin-based fine-grained load balancing mechanism. This mechanism eliminates the uncertainty and dependence on traffic characteristics inherent in traditional flow-based load balancing, eliminating the need for parameter tuning before operation.

[0155] This application embodiment reduces the serialization delay of data on a single link by segmenting a complete long message into multiple smaller cells and using multiple physical paths under a fully connected topology for parallel transmission. Compared to traditional single-path forwarding, which requires waiting for the entire large message to be transmitted, in this scheme, the target line card (the line card connected to the second supernode) can receive cell fragments from different network boards almost simultaneously, thereby shortening the overall end-to-end transmission time. In addition, the fully connected topology between the line card and the network board provides maximized path redundancy. When one or a few network boards (or their corresponding physical links) in the system fail, the scheduling mechanism on the line card side only needs to remove the failed network board from the available distribution list, and subsequent cells will be automatically and dynamically distributed to the remaining normally functioning network boards. Since the cells are evenly distributed, a single path failure will only cause a small linear decrease in total bandwidth without interrupting specific service flows, realizing "soft failure" of the system under hardware failure and smooth self-healing without service awareness.

[0156] In step 104, the network board sends the received cells to the line card connected to the second supernode.

[0157] Here, the network board's input interface receives cells and parses the target supernode (second supernode) identifier in the cell header. Based on its internally stored port mapping table, it determines the output port leading to the target line card (the line card connected to the second supernode) and sends the cell to that output port. This process does not involve IP layer routing resolution. For example, network board 1 reads the identifier ID:05 in the cell header, finds out that this identifier corresponds to physical port 10 connected to the target line card B, and then sends the cell from port 10.

[0158] In step 105, the line card connected to the second supernode reassembles the received information cells into local model gradients and sends the reassembled local model gradients to each second supernode.

[0159] The second supernode is a supernode other than the first supernode among multiple supernodes.

[0160] Here, reassembly refers to the process by which the receiving end reassembles multiple data units (cells) that have been received, which are fragmented and may be out of order, into a complete data packet (gradient tensor) in their original order.

[0161] It should be noted that the received cells are usually out of order because the spraying operation in step 103 causes different cells to pass through different paths in the network, and the transmission delay of each path may have slight differences.

[0162] In some embodiments, the received cells can be reassembled into local model gradients in the following way: the line card connected to the second supernode uses a reorder buffer to read the global sequence number contained in the header of each cell, and rearranges the out-of-order cells according to the size of the sequence number. After all the cells belonging to the same data packet have arrived, the cell headers are stripped and the payloads are spliced ​​together to restore the original local model gradient data.

[0163] Here, the target line card sorts the cells from different network boards according to the sequence number in the cell header within the reassembly buffer, removes the cell header, and reassembles the payload to restore the original outgoing message. It then sends the message via the outgoing port to the target supernode connected to its line card group. For example, after receiving four cells, target line card B reassembles the original 1000-byte outgoing message in sequence and sends it to the directly connected supernode B.

[0164] In some embodiments, the process of reassembling cells into outgoing messages supports out-of-order tolerance and timeout reassembly mechanisms. The target line card maintains a timer for each message being reassembled. If all cells are not collected within a preset time window, a discarding process is triggered and an error count is reported. Alternatively, when a cell sequence number change is detected, a fast buffer is started to wait for late cells to ensure data integrity.

[0165] In step 106, the second supernode updates the locally stored model parameters based on the received local model gradient and the locally stored local model gradient.

[0166] Here, model parameters refer to the variables in a neural network that can be optimized through training. They mainly include the weights and biases of neurons in each layer, and their values ​​directly determine the model's predictive ability.

[0167] In some embodiments, updating locally stored model parameters can be achieved as follows: First, a global reduction operation (All-Reduce) is performed, accumulating or averaging the gradients received from multiple first supernodes with the local gradients to obtain the global gradient; then, an optimizer algorithm (such as stochastic gradient descent SGD, Adam, or AdamW) is used to update the locally stored model parameters according to the formula... (in (where is the learning rate) uses global gradients to iteratively update the local model parameters in order to minimize the loss function.

[0168] This application embodiment connects multiple supernodes in a supernode system in a single-track manner on line cards and network boards with a fully interconnected structure. Since the line cards in this structure are completely equivalent, it enables rapid takeover without service awareness in the event of a first supernode failure, avoiding the significant time loss caused by training interruption and rollback to checkpoints in traditional solutions, thus significantly improving the system's mean time between failures (MTBF). Simultaneously, recovered nodes only need to connect to any line card to automatically rejoin the training cluster by loading tensors, without strictly restoring the original physical connection topology, achieving hardware decoupling, greatly simplifying on-site maintenance processes and shortening the mean time to repair. Furthermore, this system architecture not only supports large-scale pre-training but also effectively supports policy gradient calculation in the post-training stage through distributed inference and feedback mechanisms, exhibiting high versatility and the ability to continue operating under single-point failures even in the post-training stage. Combined with the underlying network's cell slicing and dynamic spraying transmission mechanism, this solution effectively achieves fine-grained load balancing within the switching architecture, avoiding local network congestion. This maximizes the system's computational efficiency and communication bandwidth utilization while comprehensively ensuring the continuity and stability of large-scale distributed model training. Furthermore, the centralized structure of inserting line cards and network boards into the backplane in related technologies has been changed to a distributed structure (i.e., distributed in the second and third racks and connected by cables to form a fully interconnected structure). This improves the flexibility of line card and network board deployment, thereby reducing the pressure on data center deployment and also reducing the weight and power supply cooling pressure on the data center. It also enables the construction of supernode systems using 36×400G line cards adapted to 19-inch standard chassis structures, improving adaptability to standardized server rooms.

[0169] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0170] In related technologies, to ensure the computational efficiency of supernode systems, such as when constructing a supernode system for MoE model training, it is necessary to establish connections between supernodes. Figure 10 This illustrates a high-bandwidth, low-latency fully connected relationship. Ideally, to support an all-to-all aggregate communication mode, any two supernodes in the cluster (such as...) Figure 10 All nodes (node0 to node7) on the right side should be physically directly connected to each other to ensure that data transmission does not involve intermediate hops and there is no bandwidth convergence. In other words, each supernode needs to have a physical direct connection to all other supernodes. The graphics processing unit (GPU) chip accelerator card (such as...) within each supernode... Figure 10 The accelerator cards on the left need to connect to all other accelerator cards belonging to the same supernode, as well as all external interface cards of that supernode. Figure 10(The square on the left side of the middle section). However, as the cluster size increases (e.g., reaching thousands of nodes), the number of cables and ports required to achieve a physical full-mesh interconnect will increase exponentially, making it almost infeasible in engineering. To address this, a method is proposed as follows: Figure 11 The illustrated multi-level leaf-spine topology, by deploying multiple high-density core switches and uniformly connecting the uplinks of each supernode to the left and right core switches, constructs a non-blocking switching network. This effectively simulates a fully connected structure between supernodes through the switching network.

[0171] However, in order to maximize network throughput and control construction costs, the downlink ports of the core switches in a leaf-spine topology are typically carefully planned, see [link to relevant documentation]. Figure 12 Core switches (such as) Figure 12 In this embodiment, the upper part of the switch (512×200G (102.4T) is used as an example, where 512 represents the number of ports, 200G represents the port speed, and 102.4T represents the total switching capacity) is fully occupied by the working nodes (GPUs), and redundant nodes (such as...) Figure 12 The link to the GPU512 in the supernode system cannot be connected to the network switching system. When a worker node (GPU) in the supernode system fails, the system cannot automatically switch over. Maintenance personnel must manually disconnect the connection cable of the failed node and reconnect the physical resources to a new redundant supernode, or be forced to change the original network topology. This cold backup method, which relies on physical changes, reduces the mean time to repair (MTTR) and mean time between failures (MTBF) of large-scale intelligent computing clusters, thus reducing the stability of the network switching system.

[0172] This application provides a model training method for a specific application scenario, the method including:

[0173] In the embodiments of this application, see Figure 13Taking a supernode system comprising eight 64-card supernodes as an example, each supernode has an aggregate bandwidth of 64×400G and uses line cards with a service access bandwidth of 36×400G. The total aggregate bandwidth of the eight 64-card supernodes is 8×64×400G, or 512×400G. The total service access bandwidth of the line cards needs to be greater than the total aggregate bandwidth of the eight supernodes, requiring at least 16 line cards. The service access bandwidth of 16 line cards is 576×400G. By fully interconnecting the line cards (represented by NCP, responsible for the system's control plane) with the network board (represented by NCF), all line cards are completely equivalent. Therefore, when the total service access bandwidth of the line cards is greater than the total aggregate bandwidth of the supernodes, a supernode can be connected to any line card, thus forming a connection between any two supernodes.

[0174] In this embodiment, the messages sent by each supernode can be split in two and sent to the line cards, for example, through a 1-to-2 fiber optic splitter. For example, a supernode can send messages to each line card with a bandwidth of 32×400G. For example, the first supernode connects to the first and second NCPs. If the second NCP has available access bandwidth, the second supernode can also be connected to the second and third NCPs. If the second NCP does not have available bandwidth, the second supernode connects to the third and fourth NCPs, and so on, connecting the supernodes to the line cards in sequence, until all eight supernodes are connected to the line cards.

[0175] Furthermore, as long as there are spare line cards, redundant supernodes (i.e., the aforementioned third supernode) can be added to the supernode system by connecting supernodes to any line card. By using this method of accessing redundant supernodes, the characteristic of completely equivalent line cards is fully utilized, which reduces the mean time to repair (MTTR) and increases the mean time between failures (MTBF) of the large-scale intelligent computing cluster (supernode system), thereby improving the reliability of the supernode system.

[0176] In this embodiment, the interconnect bandwidth of the line cards is 20×800G, and a network board with a total switching capacity of 128×800G is used. In this embodiment, 16 line cards are required to connect to 8 supernodes, resulting in a total interconnect bandwidth of 16×20×800G for the line cards. Since the network boards are typically used evenly during data exchange, only a total switching capacity of the network boards needs to be greater than or equal to the total interconnect bandwidth of the line cards; therefore, at least 3 network boards are required. For each line card, connecting it to each network board via cables constructs the network switching system. A comparison of the constructed network switching system with the internal architecture of a traditional chassis switch is provided. Figure 14 As shown, see Figure 14Traditional chassis switches, with their internal architecture housing interface boards, main control boards, and switching boards on a single backplane and connected via internal circuitry, suffer from low flexibility. Furthermore, the close proximity of multiple line cards (interface boards, main control boards) leads to excessive power and heat stress. The architecture described in the embodiments of this application, as... Figure 14 The internal architecture of this application's switch separates the traditional chassis switch into two parts: line cards and network boards, which are stored in different racks, reducing heat dissipation and power supply pressure. See also... Figure 15 By separating the line card from the traditional chassis switch, a line card with a service access bandwidth of 36×400G can be set up separately in a standard 19-inch chassis, and full cross-chassis connection between the line card and the network board can be achieved, which is more suitable for the standard size of server room.

[0177] After the connection is established, the model training task is distributed to each supernode. Each supernode calculates the corresponding result based on the training set data of the training task and calculates the error based on the result. By backpropagating the error to the layers responsible for each supernode, the local model gradient of the supernode can be obtained. For each supernode, it is necessary to send the locally calculated local model gradient to all supernodes in the supernode system to aggregate all local model gradients and update the complete model. That is, each supernode can act as the first supernode, and the remaining supernodes can act as the second supernodes. In addition, each supernode also needs to receive the local model gradients sent from all other supernodes (second supernodes) to obtain the global model gradient locally. And only when all supernodes have obtained the global model gradient can one iteration of model training be completed.

[0178] In this embodiment, after the supernode completes the calculation, it generates a message based on the calculated local model gradient. The message includes: the physical address of the data, the data length, the destination IP address, and the destination queue pair (QP). After generating the message, it is encapsulated layer by layer, such as by adding a message header, to obtain a standard Ethernet message. The standard Ethernet message is sent to the line card through the external interface card. After receiving the Ethernet message, the line card uses the received message as its incoming message.

[0179] The line card slices the incoming packet to obtain multiple cells, and adds the target supernode identifier to the cells according to the destination IP address in the incoming packet.

[0180] In this embodiment, when an incoming packet enters the line card, the parsing engine extracts the destination IP address of the packet and uses the destination IP address to query the local hardware forwarding table (FIB / LPM Table). In a distributed architecture such as DDC, the lookup result is a system-level destination ID (System Destination ID / Fabric ID), which represents the line card connected to the target supernode.

[0181] The complete Ethernet packet temporarily resides in the ingress buffer. The slicer then segments the packet into fixed lengths (typically 64 to 256 bytes). After segmentation into multiple cells, a fabric cell header is generated and added to the header of each cell. Additionally, a sequence number is added to each cell to indicate which slice it is; for example, SOP (Start of Packet) indicates the first slice, EOP (End of Packet) indicates the last slice, or 1 / 4, 2 / 4, etc.

[0182] The line card will evenly distribute cells with the target supernode identifier to all network boards.

[0183] In this embodiment, the process of uniformly distributing cells from the line card to the network board is also called cell spraying. By uniformly distributing cells to all network boards, non-blocking and load balancing are achieved. Compared with the method of Ethernet switches based on hash routing and sending the whole packet based on the selected path in related technologies, the bandwidth waste caused by the "hash polarization" phenomenon is reduced.

[0184] All network boards will send the received identifiable cells to the target line card.

[0185] The target line card reassembles the received cells into an outgoing message and sends the outgoing message to the corresponding supernode.

[0186] The following description continues to illustrate the exemplary structure of the model training device 555 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 4 As shown, the software modules stored in the model training device 555 in the memory 550 may include:

[0187] The calculation module 5551 is used to calculate the local model gradient corresponding to the tensor held by the first supernode by backpropagation based on the received error and the tensor held by the first supernode. The tensor is obtained by splitting the parameters of the model to be trained, and the first supernode is any one of multiple supernodes.

[0188] The first sending module 5552 is used to send the local model gradient to the connected line card through multiple first super nodes;

[0189] The spraying module 5553 is used to slice the received local model gradient into multiple information cells through the line card and spray the multiple information cells onto each screen.

[0190] The second transmitting module 5554 is used to transmit multiple received cells to the line card connected to the second supernode via the network board;

[0191] The third sending module 5555 is used to reassemble the received information cells into local model gradients through the line card connected to the second supernode, and send the reassembled local model gradients to each second supernode, wherein the second supernode is a supernode other than the first supernode among multiple supernodes.

[0192] The update module 5556 is used to update the locally stored model parameters through the second supernode based on the received local model gradient and the locally stored local model gradient.

[0193] In some embodiments, the plurality of supernodes further includes at least one third supernode, the third supernode being connected to at least one arbitrary line card, and the computing module 5551 is also used to detect the running status of the first supernode.

[0194] When the third supernode detects an abnormal running state of the first supernode, it loads the tensor held by the first supernode and calculates the corresponding local model gradient based on the tensor.

[0195] The third supernode sends the local model gradient to the connected line card.

[0196] In some embodiments, the supernode system also includes a restored first supernode, which is connected to at least one arbitrary line card. The calculation module 5551 is further configured to load the tensor held by the third supernode when the first supernode detects that the running status is normal, and calculate the corresponding local model gradient based on the tensor.

[0197] The local model gradient is sent to the connected line card via the restored first supernode.

[0198] In some embodiments, the calculation module 5551 is further configured to perform inference calculations based on the received prompt words through multiple first supernodes to obtain a response to the prompt words;

[0199] The first supernode calculates the policy gradient of the model based on the score of the response;

[0200] The first supernode calculates the local model gradient corresponding to the tensor it holds by backpropagation based on the policy gradient and the tensor it holds.

[0201] In some embodiments, the spraying module 5553 is further configured to control the line card to slice the received local model gradient into multiple information cells, and add an identifier of each second super node to each information cell, wherein the identifier of the second super node is used to control the mesh board to send the information cell to the line card connected to the second super node after receiving the information cell.

[0202] The line card will spray cells with the identifier of the second supernode onto each network panel.

[0203] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the model training method described above in this application.

[0204] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the model training method provided in this application. For example, ... Figure 5 The model training method is shown.

[0205] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0206] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0207] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A model training method, characterized in that, The invention is applied to a supernode system, which includes: multiple supernodes, multiple line cards, and multiple network boards, wherein the line cards and the network boards constitute a fully interconnected structure, and each of the multiple supernodes is connected to at least one of the line cards. The method includes: The first supernode performs backpropagation calculation based on the received error and the tensor it holds to obtain the local model gradient corresponding to the tensor held by the first supernode. The tensor is obtained by splitting the parameters of the model to be trained, and the first supernode is any one of the plurality of supernodes. The local model gradient is sent to the connected line card through multiple first supernodes; The line card slices the received local model gradient into multiple information cells, and sprays the multiple information cells onto each of the meshes; The network board sends the received multiple cells to the line card connected to the second supernode; The line card connected to the second supernode reassembles the received information cells into the local model gradient, and sends the reassembled local model gradient to each of the second supernodes, wherein the second supernode is a supernode other than the first supernode among the plurality of supernodes; The second supernode updates the locally stored model parameters based on the received local model gradient and the locally stored local model gradient.

2. The method according to claim 1, characterized in that, The plurality of supernodes also includes at least one third supernode, which is connected to at least one of the line cards; The method further includes: Detect the running status of the first supernode; When the third supernode detects an abnormal running state of the first supernode, it loads the tensor held by the first supernode and calculates the corresponding local model gradient based on the tensor. The third supernode sends the local model gradient to the connected line card.

3. The method according to claim 2, characterized in that, The supernode system also includes the restored first supernode, which is connected to at least one of the line cards; After sending the local model gradient to the connected line card via the third supernode, the method further includes: When the first supernode detects that the running status is normal, the recovered first supernode loads the tensor held by the third supernode and calculates the corresponding local model gradient based on the tensor. The local model gradient is sent to the connected line card via the restored first supernode.

4. The method according to claim 1, characterized in that, Before sending the local model gradient to the connected line card through multiple first supernodes, the method further includes: Through multiple first supernodes, inference calculations are performed based on the received prompt words to obtain the response to the prompt words; The first supernode calculates the policy gradient of the model based on the score of the response; The first supernode calculates the local model gradient corresponding to the tensor held by the first supernode through backpropagation based on the policy gradient and the tensor it holds.

5. The method according to claim 1, characterized in that, The line card slices the received local model gradient into multiple information cells and sprays the multiple information cells onto each of the meshes, including: The line card slices the received local model gradient into multiple information cells, and adds the identifier of each second super node to each information cell. The identifier of the second super node is used to control the netboard to send the information cell to the line card connected to the second super node after receiving the information cell. The line card will spray the cells, with the identifier of the second supernode added, onto each of the mesh panels.

6. A supernode system, characterized in that, Used for model training, including: The first rack includes multiple supernodes; The second cabinet includes multiple line cards; The third cabinet includes multiple mesh panels; The line card and the network board form a fully interconnected structure, and each of the plurality of super nodes is connected to at least one of the line cards; The supernode includes: The first supernode is used to perform backpropagation calculation based on the received error and the tensor it holds to obtain the local model gradient corresponding to the tensor held by the first supernode, wherein the tensor is obtained by splitting the parameters of the model to be trained, the first supernode is any one of the plurality of supernodes, and to send the local model gradient to the connected line card. The second supernode is used to update the locally stored model parameters based on the received local model gradient and the locally stored local model gradient. The line card includes: The first line card is used to slice the received local model gradient into multiple information cells and spray the multiple information cells onto each of the mesh plates; The second line card is used to reassemble the received information cells into the local model gradient, and send the reassembled local model gradient to each of the second super nodes; The network board is used to send the received plurality of cells to the second line card connected to the second supernode.

7. The system according to claim 6, characterized in that, The first cabinet also includes: At least one third supernode, the third supernode being connected to at least one of the line cards; The third supernode is used to load the tensor held by the first supernode when an abnormal running state of the first supernode is detected, and to calculate the corresponding local model gradient based on the tensor; and Used to send the local model gradient to the connected line card.

8. The system according to claim 7, characterized in that, The first cabinet also includes the restored first supernode, which is connected to at least one of the line cards; The restored first supernode is used to, when the normal operating state of the first supernode is detected, load the tensor held by the third supernode and calculate the corresponding local model gradient based on the tensor; and The local model gradient is sent to the connected line card.

9. The system according to claim 6, characterized in that, The first supernode is also configured to perform inference calculations based on the received prompt words to obtain a response to the prompt words; as well as Calculate the policy gradient of the model based on the score of the response; as well as Based on the policy gradient and the tensor held by the first supernode, the local model gradient corresponding to the tensor held by the first supernode is calculated by backpropagation.

10. The system according to claim 6, characterized in that, The first line card is further configured to slice the received local model gradient into multiple information cells, and add an identifier of each second supernode to each information cell, wherein the identifier of the second supernode is used to control the meshboard to send the information cell to the second line card connected to the second supernode after receiving the information cell; and The cells with the identifier of the second supernode added are sprayed onto each of the mesh panels.

11. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, controls the supernodes, line cards, and network boards in the supernode system to implement the method described in any one of claims 1 to 5.

12. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by the processor, they control the supernodes, line cards, and network boards in the supernode system to implement the method described in any one of claims 1 to 5.

13. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by the processor, they control the supernodes, line cards, and network boards in the supernode system to implement the method described in any one of claims 1 to 5.

Images