Model training acceleration method and system based on in-network gradient data annular full reduction

Abstract

The invention discloses a model training acceleration method and system based on in-network gradient data annular full reduction. The method comprises the following steps: acquiring cluster environment information and distributed machine learning model gradient data traffic; constructing an annular full reduction strategy based on the constraint condition; the computing node carries out message packaging according to the annular full reduction strategy, and issues the message to the switching node for aggregation according to an aggregation path set in the annular full reduction strategy; and a first data table and a second data table are constructed, and the switching node performs on-network aggregation on the distributed machine learning model gradient data on each computing node according to the first data table and the second data table, and distributes an aggregation result. According to the method, multiplexing of an existing network stack is achieved to the maximum extent through protocol-independent in-network annular aggregation of an in-network aggregation network stack, and the problem of long communication time in the model training process under the background that the parameter scale and the number of training samples of a current distributed machine learning model are increased dramatically is solved.

Description

Technical Field

[0001] This invention relates to the field of data center network technology, and in particular to a method and system for accelerating model training based on the ring full reduction of on-network gradient data. Background Technology

[0002] In the training process of distributed machine learning models, with the rapid increase in the size of model parameters and the number of training samples, the overhead of gradient data transmission and aggregation communication between computing nodes has become a core bottleneck restricting training efficiency. In existing technologies, distributed training frameworks typically rely on the computing nodes themselves to complete the full reduction operation of gradient data. This approach is limited by the serial communication mode between computing nodes and the competition for network bandwidth resources, resulting in communication latency increasing linearly with the cluster size, which severely reduces the overall training throughput.

[0003] While some studies have proposed on-network computing solutions based on dedicated hardware acceleration, existing technologies largely rely on customized network protocols or dedicated chip architectures, resulting in poor compatibility with mainstream distributed training frameworks, low network stack reusability, and high development and deployment costs. Furthermore, in large-scale cluster environments, anomalies such as network packet loss and errors in communication operator execution at computing terminals can easily lead to distorted gradient aggregation results. Existing fault-tolerance mechanisms often employ global retransmission strategies, further exacerbating communication overhead and the risk of training interruption. Therefore, optimizing the gradient data circular full reduction process through on-network computing technology while ensuring compatibility and reliability has become a key technical challenge for improving the efficiency of distributed training. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method and system for accelerating model training based on the circular full reduction of in-network gradient data.

[0005] In a first aspect, embodiments of the present invention provide a method for accelerating model training based on the circular full reduction of in-network gradient data, the method comprising the following steps: Acquire cluster environment information and gradient data flow of distributed machine learning models; and construct a ring-shaped full reduction strategy based on constraints. The computing nodes encapsulate messages according to the ring full reduction strategy and send them to the exchange nodes for aggregation according to the aggregation path set in the ring full reduction strategy; a first data table and a second data table are constructed, and the exchange nodes perform on-network aggregation of the distributed machine learning model gradient data on each computing node according to the first data table and the second data table, and distribute the aggregation results.

[0006] Furthermore, the cluster environment information includes: the interconnection topology information of computing nodes and switching nodes in the distributed task, the available bandwidth between computing nodes and switching nodes, and the size of the registers that can be allocated to the switching nodes; The gradient data flow of the distributed machine learning model includes: the batch and amount of gradient data of each computing node in the current distributed task; The constraints include: the available bandwidth between the compute node and the switching node is the minimum of the available transmit or receive bandwidth of the compute node's network card and the corresponding link bandwidth; and the compute node may continue to send messages if and only if the available memory size of the switching node is greater than the memory size occupied by the message's on-network aggregation, otherwise it must be blocked and wait.

[0007] Furthermore, the process of constructing the ring-shaped full reduction strategy includes: The number of aggregation rings for on-network gradient aggregation is obtained based on the distributed task, and a corresponding ring full reduction strategy is configured for each aggregation ring. Each ring full reduction strategy includes: the amount of data to be aggregated in each aggregation ring, the number of messages in a single message in each aggregation ring, the number of messages that a computing node in each aggregation ring can send in parallel, and the aggregation path configured for each aggregation ring. The aggregation path configured for each aggregation ring includes: a list of intermediate exchange nodes that the computing node sending the message passes through to the exchange node, the exchange node, and a list of intermediate exchange nodes that the exchange node passes through to reach the gradient result receiving node.

[0008] Furthermore, the process by which the exchange node processes the gradient data of the distributed machine learning model on each computing node according to the first data table and the second data table includes: After parsing the message and determining if the protocol identifier matches, in-network gradient aggregation is performed based on the parsed message; this includes: Based on the task identifier and destination address of the message, query the task identifier index and destination address index in the first data table. If the task identifier index and destination address index exist, update the timestamp corresponding to the task identifier; otherwise, create the task identifier index and destination address index. The second data table is queried based on the task identifier, aggregation ring identifier, and message identifier of the message. If a data item exists with the task identifier, aggregation ring identifier, and message identifier as indexes, its computation node number and message position in the bitmap are determined based on the message's computation node number and message sequence identifier. If the position is 0, the payload is aggregated at the corresponding position in the payload data area according to the message sequence identifier, and the computation node number and message position in the bitmap are set to 1. If the position is 1, the message is discarded. After each message aggregation is completed, it is checked whether all bits in the bitmap of the second data table are set to 1. If all bits are set to 1, the aggregation is complete. Aggregation; if no data item is indexed by task identifier, aggregation ring identifier, or message identifier, then determine the size of the bitmap in the second data table based on the message length and the number of compute nodes in the message, and initialize all elements in the bitmap matrix to 0; determine the size of the status code in the second data table based on the number of compute nodes in the message, and initialize all elements in the status code to 0; determine the size of the payload data area based on the message length in the message, and initialize all elements in the payload data area to 0; perform gradient aggregation operation with the corresponding data payload area according to the message sequence identifier, and overwrite the payload data area with the aggregation result.

[0009] Furthermore, the process of distributing the aggregation results includes: The message is reassembled from the destination address of the message in the first data table and the aggregation result in the second data table. The switching node forwards the reassembled message according to the aggregation path in the ring full reduction strategy; and at the start of message forwarding, it sets the timestamp value in the status area of ​​the second data table to the current system time. After receiving the reassembled message, the computing node sends a result confirmation message to the switching node; After receiving the result confirmation message sent by the computing node, the switching node sets the corresponding position of the status code in the second data table to 1; When all status codes are 1, the current message aggregation task ends and the corresponding data result in the second data table is deleted; when all status codes are not 1 before the distribution timeout, it indicates that there is packet loss in the distribution process. The message is retransmitted according to the position of the 0 element in the status code, and the timestamp value is updated to the current system time again.

[0010] Furthermore, the process of constructing the first data table includes: The first data table consists of a task identifier (as a first-level index item), a destination address (as a second-level index item), and a message header. In the process of reassembling a message based on the destination address and the aggregation results in the second data table: the data is located by the task identifier, which is used as a first-level index item, and the message header is selected by the destination address, which is used as a second-level index item, thereby completing the message reassembly.

[0011] Furthermore, the process of constructing the second data table includes: The second data table consists of an index area, a status area, and a load data area; The index area includes: task identifier, aggregation ring identifier, and message identifier; The status area includes: a bitmap, a status code, and a timestamp; wherein, the bitmap is a matrix of the number of computing nodes × message length. When a message matching the index, with computing node identifier m and message sequence identifier n, is received, the element in the m-th row and n-th column of the bitmap is set to 1. When all elements in the bitmap are 1, it indicates that the current message has been aggregated; wherein, the status code is a column vector of the number of computing nodes × 1. When an exchange node receives a result confirmation message sent by a computing node, the corresponding position of the status code is set to 1. The payload data area stores the gradient data sent by each computing node according to the message sequence.

[0012] Secondly, embodiments of the present invention provide a model training acceleration system based on in-network gradient data circular full reduction. The system is used to implement the aforementioned model training acceleration method based on in-network gradient data circular full reduction. The system includes: The management plane is used to acquire cluster environment information and gradient data traffic of distributed machine learning models; and based on constraints, it constructs a ring-shaped full reduction strategy; and distributes the ring-shaped full reduction strategy to computing nodes and exchange nodes. The compute node is used to encapsulate packets according to the ring full reduction strategy and send the packets to the switching node according to the aggregation path set in the ring full reduction strategy; An exchange node is used to construct a first data table and a second data table. The exchange node aggregates the gradient data of the distributed machine learning model on each computing node according to the first data table and the second data table, and distributes the aggregation results.

[0013] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-described model training acceleration method based on the ring full reduction of on-network gradient data.

[0014] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the above-described method for accelerating model training based on the circular full reduction of in-network gradient data.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention provides a method and system for accelerating model training based on on-network gradient data ring full reduction. Through a protocol-independent on-network ring aggregation mechanism, it maximizes the reuse of existing network stacks, reducing development and deployment costs while ensuring compatibility with mainstream distributed training frameworks. By segmenting the ring reduction process based on network topology and exchange node resources, and executing gradient reduction calculations in parallel on allocated on-network computing exchange nodes, the efficiency of gradient data aggregation can be significantly improved, effectively alleviating communication bottlenecks. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is an architecture diagram of a model training acceleration method based on on-network gradient data ring full reduction provided in an embodiment of the present invention; Figure 2 A schematic diagram of the on-network aggregation message protocol provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the first data table (meta) for online aggregation provided in an embodiment of the present invention; Figure 4 A schematic diagram of the second data table aggr table for online aggregation provided in an embodiment of the present invention; Figure 5 A flowchart of gradient data aggregation provided in an embodiment of the present invention; Figure 6 A flowchart for distributing aggregation results provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0020] Step S1: Obtain cluster environment information and gradient data flow of distributed machine learning models; and construct a ring-shaped full reduction strategy based on constraints.

[0021] Furthermore, the cluster environment information includes: the interconnection topology information of computing nodes and switching nodes in the distributed task, the available bandwidth between computing nodes and switching nodes, and the size of the registers that can be allocated to the switching nodes; Furthermore, the gradient data flow of the distributed machine learning model includes: the batch and amount of gradient data of each computing node in the current distributed task; Furthermore, the constraints include: First constraint: Available bandwidth between computation node i and switching node j To calculate the minimum value between the available transmit or receive bandwidth of the node's network interface card and the corresponding link bandwidth; and, The second constraint is that a computing node may continue to send messages if and only if the available memory size of the exchange node is greater than the memory size occupied by the on-network aggregation of messages; otherwise, it must be blocked and wait.

[0022] Specifically, the process of constructing a ring-shaped full reduction strategy includes: The number of aggregation rings for on-network gradient aggregation is obtained based on the distributed task, and a corresponding ring full reduction strategy is configured for each aggregation ring. Each ring full reduction strategy includes: the amount of data to be aggregated in each aggregation ring, the number of messages in a single message in each aggregation ring, the number of messages that a computing node in each aggregation ring can send in parallel, and the aggregation path configured for each aggregation ring. The aggregation path configured for each aggregation ring includes: a list of intermediate exchange nodes that the computing node sending the message passes through to the exchange node, the exchange node, and a list of intermediate exchange nodes that the exchange node passes through to reach the gradient result receiving node.

[0023] Furthermore, the specific configuration process of the ring-shaped full reduction strategy includes: The number of aggregation rings executing on-network gradient aggregation is obtained based on the distributed task; it should be noted that each aggregation task includes one or more aggregation rings. Get each aggregation ring The amount of data to be aggregated The exchange node corresponding to each aggregation ring Gradient message size Message size Send window size Aggregation path Therefore, the cyclic total reduction strategy can be expressed as: In the formula, This represents the nth cyclic total reduction strategy. This represents the amount of data to be aggregated in the nth aggregation ring. This indicates the number of messages in a single message of the nth aggregation ring. This represents the number of messages that a computing node in the nth aggregation ring can send in parallel. This represents the aggregation path configured in the nth aggregation ring.

[0024] Among them, in the polymerization ring In this context, if there are k compute nodes, it means that there are k aggregation paths that need to be configured. These paths can be represented as follows: In the formula, This indicates the distance from the sending compute node k-1 to the destination switching node. List of intermediate exchange nodes traversed Indicates a swapping node List of intermediate exchange nodes traversed to reach gradient result receiving node k.

[0025] It should be noted that the compute nodes are determined before the task begins. The transmission paths between compute nodes and switching nodes in each aggregation ring need to be configured. The path between a pair of compute nodes (x and y) in the aggregation ring can be represented as: in, From the computing node x that sends the message to the destination switching node The list of intermediate exchange nodes (intermediate nodes can be empty) Then it is the exchange node that performs the aggregation task in aggregation ring i. sw_ id , For exchange nodes List of intermediate exchange nodes traversed to reach the gradient result receiving node y (can be empty if there are no intermediate nodes).

[0026] It should be noted that the gradient message size Indicates a polymer ring The size of a single raw gradient data packet sent by a computing node; message size Indicates a polymer ring Number of messages per message; sending window size Indicates a polymer ring The number of messages that a single computing node can send in parallel.

[0027] In step S2, the compute node encapsulates the message according to the ring full reduction strategy and sends it to the exchange node for aggregation according to the aggregation path set in the ring full reduction strategy.

[0028] Among them, such as Figure 2As shown, the message includes: Ethernet frame header, network layer header, transport layer header, on-network aggregation header, and data payload (gradient); wherein, the on-network aggregation header includes: protocol identifier field (MagicNumber), job identifier field (Job ID), aggregation ring identifier (Ring ID), message identifier (Msg ID), message length (MsgLen), message sequence identifier (PSN), number of compute nodes (RankCnt), compute node identifier (Rank ID), and timestamp (Timestamp). The protocol identifier field (Magic Number) is 3 bytes long and contains 0x494E41, used to identify that the message is a network-aggregated message; the job identifier field (Job ID) is 2 bytes long and is a unique identifier for the gradient aggregation task to which the data message belongs; the aggregation ring identifier (Ring ID) is 1 byte long, used to identify the specific aggregation ring since a single aggregation subtask may use one or more rings to accelerate the aggregation task; the message identifier (Msg ID) is 4 bytes long and serves as the message identifier; the message length (MsgLen) is 1 byte long and is used to identify the number of messages included in the message; the message sequence identifier (PSN) is 1 byte long and is used to identify the order of messages in the message, addressing out-of-order or packet loss issues caused by network instability; the number of compute nodes (RankCnt) is 1 byte long and is used to characterize the number of compute nodes included in the current aggregation ring; the compute node identifier (RankCnt) is 1 byte long and is used to characterize the number of compute nodes included in the current aggregation ring; the number of compute nodes ... (Rank The ID is 1 byte long and identifies the computing node that sent the current message; the timestamp is 4 bytes long and records the message sending time.

[0029] The exchange node stores a first data table (denoted as the meta table) and a second data table (denoted as the aggr table). The first data table, meta, stores the original message header information according to the task dimension, and is used to recover the original message after gradient aggregation and forward it to the destination address. The second data table, aggr, aggregates and stores gradient data according to the aggregation ring dimension, and is used to complete the reception, aggregation and result distribution of gradient data.

[0030] Furthermore, such as Figure 3As shown, the first data table consists of a task identifier (job_id) as the first-level index, a destination address (dst_addr) as the second-level index, and a message header. During the process of reassembling the message based on the destination address and the aggregation results in the second data table: data is located using the task identifier (first-level index), and the message header is selected using the destination address (second-level index), thus completing the message reassembly. The message header data stores the message header information (with payload removed) from the original gradient message according to the destination address. The update time field records the data table update time with the task identifier as the dimension, used for the aging mechanism of the first data table data, such as automatically clearing data records older than one day. This parameter can be configured according to actual needs.

[0031] Furthermore, such as Figure 4 As shown, the second data table consists of an index area, a status area, and a load data area; The index area includes: job_id (task identifier), ring_id (aggregation ring identifier), and msg_id (message identifier); The status area includes: bitmap, status code, and job timestamp. The bitmap is a matrix (RankCnt×msgLen) of the number of compute nodes × message length. When a message matching the index, with compute node identifier m and message sequence identifier n is received, the element in the m-th row and n-th column of the bitmap is set to 1. When all elements in the bitmap are 1, it indicates that the current message has completed aggregation. The status code is a column vector (RankCnt×1) consisting of the number of computing nodes × 1. The aggregation results are sent sequentially to the computing nodes. Upon receiving the complete aggregation results for the message size, each computing node replies with an acknowledgment message. Upon receiving the acknowledgment message, the exchange node sets the status of the corresponding computing node to 1. When all status codes are 1, the message completes the gradient processing and releases the memory space. The timestamp is used to store key time nodes and trigger message retransmission operations. During the message reception phase, the timestamp stores the reception time of the first message. The user can set an aggregation wait time. If the current system time minus the timestamp exceeds the wait time and there are still nodes with a value of 0 in the bitmap, the corresponding compute node is notified to retransmit the message with sequence number PSN_id based on the compute node identifier rank_id and the message identifier PSN_id, and the timestamp is reset to the current system time. During the aggregation result distribution phase, the timestamp value is the start time of the distribution process. The user needs to set a distribution wait time. If the system time minus the timestamp exceeds the distribution wait time and there are still nodes with a value of 0 in the status code, all gradient aggregation results of the message will be retransmitted to the compute node corresponding to the compute node identifier rank_id, and the timestamp will be reset to the current system time. The payload data area stores the gradient data sent by each computing node according to the message sequence identifier. Specifically, it stores the gradient data in order of message PSN number. Gradient data of PSN messages with the same sequence number are directly accumulated at the corresponding positions, and the correctness of the gradient accumulation result is controlled by a bitmap.

[0032] Specifically, such as Figure 5 As shown, step S2 includes the following steps: The message is parsed to identify if its protocol identifier is 0x494E41. If no match is found, it is forwarded directly according to the destination address. If a match is found, on-network gradient aggregation is performed based on the parsed message; this includes: Based on the task identifier job_id and destination address dst_addr of the message, query the task identifier index and destination address index in the first data table meta. If the task identifier index and destination address index exist, update the timestamp corresponding to the task identifier; otherwise, create the task identifier index and destination address index. Query the second data table aggr based on the task identifier job_id, aggregation ring identifier ring_id, and message identifier msg_id of the message; If a data item is indexed by a task identifier, aggregation ring identifier, and message identifier, its compute node number and the position of its message (rank_id, pkt) in the bitmap are determined according to the compute node number (rank_id) and message sequence identifier (PSN) of the message. If the position is 0, the payload is aggregated at the corresponding position in the payload data area according to the message sequence identifier, and the position of the compute node number and its message (rank_id, pkt) in the bitmap is set to 1. If the position is 1, it indicates that it is a retransmission message, and the message can be discarded. After each message aggregation is completed, it is checked whether all the bitmaps in the second data table are set to 1. If all are set to 1, the aggregation is completed. Otherwise, it is checked whether the aggregation process has timed out. If it has timed out, a retransmission command message is sent to the corresponding compute node according to the position information of the 0 element in the bitmap. This message carries the sequence identifier number of the message to be retransmitted. If no data item is indexed by task identifier, aggregation ring identifier, or message identifier, then the size of the bitmap in the second data table aggr is determined based on the message length MsgLen and the number of compute nodes RankCnt in the message, and all elements in the bitmap matrix are initialized to 0; the size of the status code status_code in the second data table is determined based on the number of compute nodes RankCnt in the message, and all elements in the status code are initialized to 0; the size of the payload data area is determined based on the message length MsgLen in the message, and the length of a single message payload is denoted as pktSize, then the size of the payload data area is msgLen × pktSize, and all elements in the payload data area are initialized to 0; gradient aggregation is performed according to the message sequence identifier and the corresponding data payload area, and the aggregation result is overwritten to the payload data area.

[0033] Step S3: The exchange node performs on-network aggregation of the distributed machine learning model gradient data on each computing node according to the first data table and the second data table, and distributes the aggregation results.

[0034] Specifically, such as Figure 6 As shown, after all elements in the bitmap are set to 1, the distribution of the aggregation results begins. Step S3 includes the following steps: The message is reassembled from the destination address of the message in the first data table and the aggregation result in the second data table. The switching node forwards the reassembled message according to the aggregation path in the ring full reduction strategy; and at the start of message forwarding, it sets the timestamp value in the status area of ​​the second data table to the current system time. After receiving the reassembled message, the computing node sends a result confirmation message to the switching node; After receiving the result confirmation message sent by the computing node, the switching node sets the corresponding position of the status code in the second data table to 1; When all status codes are 1, the current message aggregation task ends and the corresponding data result in the second data table is deleted; when all status codes are not 1 before the distribution timeout, it indicates that there is packet loss in the distribution process. The message is retransmitted according to the position of the 0 element in the status code, and the timestamp value is updated to the current system time again.

[0035] In summary, this invention proposes a parallel on-network aggregation mechanism for ring-based full aggregation technology. Compared to the traditional parameter server architecture, it can better utilize system bandwidth. Compared to the traditional ring-based full aggregation architecture, the node transmission communication volume is reduced by nearly half. Moreover, in larger-scale clusters, the communication cost does not increase linearly with the number of communication nodes. In addition, the multi-aggregation ring mechanism proposed in this invention can fully leverage the acceleration capabilities of aggregation and exchange nodes to achieve parallel acceleration of gradient aggregation. Secondly, the on-network aggregation management platform of this invention allows users to configure different strategies, thereby providing an experimental environment to support researchers in studying various optimization algorithms for on-network aggregation scenarios. In conclusion, this invention not only provides important on-network aggregation technology support for accelerating distributed model training but also provides a powerful tool for application optimization in related fields.

[0036] Accordingly, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the model training acceleration method based on the above-described on-network gradient data ring full reduction. Figure 7 The diagram shown is a hardware structure diagram of any device with data processing capabilities for the model training acceleration method based on the on-network gradient data ring full reduction provided in an embodiment of the present invention, except for... Figure 7 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0037] Accordingly, this application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the aforementioned method for accelerating model training based on the circular full reduction of online gradient data. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0038] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A method for accelerating model training based on the circular full reduction of in-network gradient data, characterized in that, The method includes the following steps: Acquire cluster environment information and gradient data flow of distributed machine learning models; and construct a ring-shaped full reduction strategy based on constraints. The computing nodes encapsulate messages according to the ring full reduction strategy and send them to the exchange nodes for aggregation according to the aggregation path set in the ring full reduction strategy; a first data table and a second data table are constructed; the exchange nodes perform on-network aggregation of the distributed machine learning model gradient data on each computing node according to the first data table and the second data table, and distribute the aggregation results.

2. The model training acceleration method based on on-network gradient data ring full reduction according to claim 1, characterized in that, The cluster environment information includes: the interconnection topology information of computing nodes and switching nodes in the distributed task, the available bandwidth between computing nodes and switching nodes, and the size of the registers that can be allocated to the switching nodes; The gradient data flow of the distributed machine learning model includes: the batch and amount of gradient data of each computing node in the current distributed task; The constraints include: the available bandwidth between the compute node and the switching node is the minimum of the available transmit or receive bandwidth of the compute node's network card and the corresponding link bandwidth; and the compute node may continue to send messages if and only if the available memory size of the switching node is greater than the memory size occupied by the message's on-network aggregation, otherwise it must be blocked and wait.

3. The method for accelerating model training based on the circular full reduction of in-network gradient data according to claim 1, characterized in that, The process of constructing a ring-shaped full reduction strategy includes: The number of aggregation rings for on-network gradient aggregation is obtained based on the distributed task, and a corresponding ring full reduction strategy is configured for each aggregation ring. Each ring full reduction strategy includes: the amount of data to be aggregated in each aggregation ring, the number of messages in a single message in each aggregation ring, the number of messages that a computing node in each aggregation ring can send in parallel, and the aggregation path configured for each aggregation ring. The aggregation path configured for each aggregation ring includes: a list of intermediate exchange nodes that the computing node sending the message passes through to the exchange node, the exchange node, and a list of intermediate exchange nodes that the exchange node passes through to reach the gradient result receiving node.

4. The method for accelerating model training based on the circular full reduction of in-network gradient data according to claim 1, characterized in that, The process by which the exchange node processes the gradient data of the distributed machine learning model on each computing node according to the first data table and the second data table includes: After parsing the message and determining if the protocol identifier matches, in-network gradient aggregation is performed based on the parsed message; this includes: Based on the task identifier and destination address of the message, query the task identifier index and destination address index in the first data table. If the task identifier index and destination address index exist, update the timestamp corresponding to the task identifier; otherwise, create the task identifier index and destination address index. The second data table is queried based on the task identifier, aggregation ring identifier, and message identifier of the message. If a data item exists with the task identifier, aggregation ring identifier, and message identifier as indexes, its computation node number and message position in the bitmap are determined based on the message's computation node number and message sequence identifier. If the position is 0, the payload is aggregated at the corresponding position in the payload data area according to the message sequence identifier, and the computation node number and message position in the bitmap are set to 1. If the position is 1, the message is discarded. After each message aggregation is completed, it is checked whether all bits in the bitmap of the second data table are set to 1. If all bits are set to 1, the aggregation is complete. Aggregation; if no data item is indexed by task identifier, aggregation ring identifier, or message identifier, then determine the size of the bitmap in the second data table based on the message length and the number of compute nodes in the message, and initialize all elements in the bitmap matrix to 0; determine the size of the status code in the second data table based on the number of compute nodes in the message, and initialize all elements in the status code to 0; determine the size of the payload data area based on the message length in the message, and initialize all elements in the payload data area to 0; perform gradient aggregation operation with the corresponding data payload area according to the message sequence identifier, and overwrite the payload data area with the aggregation result.

5. The model training acceleration method based on the circular full reduction of in-network gradient data according to claim 4, characterized in that, The process of distributing the aggregation results includes: The message is reassembled from the destination address of the message in the first data table and the aggregation result in the second data table. The switching node forwards the reassembled message according to the aggregation path in the ring full reduction strategy; and at the start of message forwarding, it sets the timestamp value in the status area of ​​the second data table to the current system time. After receiving the reassembled message, the computing node sends a result confirmation message to the switching node; After receiving the result confirmation message sent by the computing node, the switching node sets the corresponding position of the status code in the second data table to 1; When all status codes are 1, the current message aggregation task ends and the corresponding data result in the second data table is deleted; when all status codes are not 1 before the distribution timeout, it indicates that there is packet loss in the distribution process. The message is retransmitted according to the position of the 0 element in the status code, and the timestamp value is updated to the current system time again.

6. The method for accelerating model training based on the circular full reduction of in-network gradient data according to claim 4 or 5, characterized in that, The process of constructing the first data table includes: The first data table consists of a task identifier (as a first-level index item), a destination address (as a second-level index item), and a message header. In the process of reassembling a message based on the destination address and the aggregation results in the second data table: the data is located by the task identifier, which is used as a first-level index item, and the message header is selected by the destination address, which is used as a second-level index item, thereby completing the message reassembly.

7. The method for accelerating model training based on the circular full reduction of in-network gradient data according to claim 4 or 5, characterized in that, The process of constructing the second data table includes: The second data table consists of an index area, a status area, and a load data area; The index area includes: task identifier, aggregation ring identifier, and message identifier; The status area includes: a bitmap, a status code, and a timestamp; wherein, the bitmap is a matrix of the number of computing nodes × message length. When a message matching the index, with computing node identifier m and message sequence identifier n, is received, the element in the m-th row and n-th column of the bitmap is set to 1. When all elements in the bitmap are 1, it indicates that the current message has been aggregated; wherein, the status code is a column vector of the number of computing nodes × 1. When an exchange node receives a result confirmation message sent by a computing node, the corresponding position of the status code is set to 1. The payload data area stores the gradient data sent by each computing node according to the message sequence.

8. A model training acceleration system based on on-network gradient data ring full reduction, characterized in that, The system is used to implement the model training acceleration method based on the circular full reduction of in-network gradient data as described in any one of claims 1-7, and the system includes: The management plane is used to acquire cluster environment information and gradient data flow of distributed machine learning models; and based on constraints, it constructs a ring-shaped full reduction strategy. The compute node is used to encapsulate packets according to the ring full reduction strategy and send the packets to the switching node according to the aggregation path set in the ring full reduction strategy; An exchange node is used to construct a first data table and a second data table. The exchange node performs an aggregation operation on the gradient data of the distributed machine learning model on each computing node based on the first data table and the second data table, and distributes the aggregation results.

9. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the model training acceleration method based on the on-network gradient data ring full reduction as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the model training acceleration method based on the circular full reduction of in-network gradient data as described in any one of claims 1-7.

Images