Distributed machine learning gradient synchronization method and system based on in-network computation

Abstract

This invention discloses a distributed machine learning gradient synchronization method based on intra-network computing, comprising: constructing a distributed machine learning cluster; constructing a training set and allocating a subset of the training set to each computing node; constructing a convolutional neural network at each computing node, and training the convolutional neural network at the current computing node using the subset of the training set allocated to each computing node to generate gradient data on each computing node; using a programmable switch and a server to perform data mixing and synchronization on gradient data blocks on all computing nodes to obtain complete gradient data after fusion of gradient data on all computing nodes; updating the parameters of the convolutional neural network using the complete gradient data; and iteratively training the convolutional neural network to obtain the trained convolutional neural network model. This invention can significantly reduce the amount of data transmitted in the link, alleviate congestion problems at the network card of the server node, and improve the communication efficiency of the server synchronization mode.

Description

Technical Field

[0001] This invention belongs to the field of distributed machine learning technology, specifically relating to a distributed machine learning gradient synchronization method and system based on intra-network computing, which can effectively reduce the communication overhead of distributed machine learning systems. Background Technology

[0002] In recent years, the scale of machine learning models and datasets used in image recognition, natural language processing, and other fields has increased dramatically, significantly increasing the training time and cost of neural network models. Therefore, distributed machine learning fully utilizes the computing resources in a cluster, scaling the training of neural network models from a single machine and node to multiple machines and nodes, effectively reducing training time. However, as the scale of distributed machine learning increases, the communication overhead caused by synchronization between multiple nodes intensifies, severely impacting the acceleration effect of distributed machine learning.

[0003] Classic distributed gradient synchronization models include parameter server-based and all-reduce approaches. As the amount of parameter data increases and the number of worker nodes expands, congestion can easily occur on the parameter server side. During each training round, all worker nodes need to upload their trained gradients to the parameter server node. A large backlog of gradient data can occur at the parameter server node, leading to network congestion. In the iterative process of distributed machine learning, bursts of data arriving at the parameter server within milliseconds can cause packet loss, congestion, and load imbalance, thus increasing the completion time of the entire distributed application.

[0004] In 2019, ByteDance open-sourced BytePS, a distributed training framework based on a parameter server architecture. This framework utilizes additional CPU resources as parameter servers, significantly improving communication performance. However, to achieve the theoretical communication efficiency in BytePS, the same number of parameter server nodes as worker nodes are required. Furthermore, when BytePS is scaled to a distributed multi-machine cluster, additional CPU servers need to be configured to act as parameter server nodes. Moreover, the periodic uploading of gradients from worker nodes across multiple servers to the parameter server can also create network bottlenecks. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a distributed machine learning gradient synchronization method and system based on intranet computing. The aim is to utilize programmable devices to carry out partial gradient synchronization, thereby alleviating the communication pressure caused by synchronizing a large number of gradients and improving system throughput.

[0006] The technical problem to be solved by this invention is achieved through the following technical solution:

[0007] One aspect of the present invention provides a distributed machine learning gradient synchronization method based on intra-network computation, comprising:

[0008] S1: Construct a distributed machine learning cluster, including a programmable switch and multiple servers connected to the switch, with each server configured with multiple computing nodes;

[0009] S2: Build a training set and allocate a subset of the training set to each compute node of each server;

[0010] S3: Build a convolutional neural network at each computing node, and train the convolutional neural network at the current computing node using a subset of the training set allocated to each computing node to generate a set of gradient data at each computing node.

[0011] S4: Use the programmable switch and the server to perform data mixing and synchronization on gradient data blocks on all computing nodes, so that each computing node can obtain the complete gradient data after the gradient data on all computing nodes are fused.

[0012] S5: Update the parameters of each convolutional neural network using the complete gradient data;

[0013] S6: Iteratively train the convolutional neural network to obtain the trained convolutional neural network model.

[0014] In one embodiment of the present invention, S2 includes:

[0015] S2a: Select a large number of images as image samples, each image containing a target type label, and combine the image samples and the corresponding target type labels to form a training set;

[0016] S2b: Divide the training set into a number of training set subsets equal to the total number of computing nodes, and assign an independent training set subset to each computing node.

[0017] In one embodiment of the present invention, the convolutional neural network at each computing node includes a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a third convolutional layer, a third pooling layer, a fourth convolutional layer, and a fully connected layer connected in sequence.

[0018] In one embodiment of the present invention, S4 includes:

[0019] S41: Perform reduction-dispersion on gradient data blocks on all computing nodes so that each computing node in each server obtains the fused gradient data from all nodes in the current server;

[0020] S42: Perform data slicing on the gradient block after fusion of each computing node to obtain multiple data packets and send them to the corresponding aggregation slot of the programmable switch;

[0021] S43: Utilize the aggregation slots corresponding to the programmable switch to aggregate the data packets received from the computing nodes and broadcast the aggregated data to the corresponding computing nodes of all servers;

[0022] S44: After receiving the aggregated data, the corresponding compute nodes in all servers overwrite and replace the original data in the current server.

[0023] S45: Perform full aggregation of the current data of all computing nodes within each server so that each computing node within each server obtains the complete gradient data after the fusion of gradient data from all computing nodes.

[0024] In one embodiment of the present invention, S41 includes:

[0025] S41a: Number the compute nodes within each server. s-1 G s-2 ,...,G s-n ,...,G s-N Among them, G s-n This represents the nth computing node in the s-th server, where N represents the total number of nodes in each server.

[0026] S41b: Divide the gradient data set on each computing node into N gradient blocks, and number each gradient block sequentially as chunk0, chunk1, ..., chunk N-1 ;

[0027] S41c: Set the left and right neighbors of each compute node in each server. The left neighbor is the compute node before the current compute node, the right neighbor is the compute node after the current compute node, the right neighbor of the last compute node is the first compute node, and the left neighbor of the first compute node is the last compute node.

[0028] S41d: Perform N-1 data transfers. During the i-th data transfer, the number is G. s-n The computing node receives data from its left neighbor, which is numbered chunk. (n-i-1)％8 The gradient block, and simultaneously send its numbered chunk (n-i)％8 The gradient block is drawn from the right neighbor, where the initial value of i is 1 and the range is 1≤i<N;

[0029] S41e: The number G s-nThe gradient block received after N-1 data transfers within the computing node and the chunk numbered (n+i)％8 The gradient blocks are summed.

[0030] S41f: Let i increase from 0 to 7. After each summation, repeat steps S41c-S41e. Finally, each computing node in each server obtains the gradient block of all nodes in the current server.

[0031] In one embodiment of the present invention, S42 includes:

[0032] S42a: Divide the fused gradient block at each computing node into multiple segments, with the size of each segment being the same as the vector size in each slot of the programmable switch;

[0033] S42b: Construct each fragment into a data packet, the data packet including the segmented data vector and carrying a pool index idx and a compute node number z;

[0034] S42c: Controls the first compute node of each server to send a number of data packets equal to the number of pools in the programmable switch, with each data packet sent in index order to the aggregation slot of the same index number in the programmable switch.

[0035] In one embodiment of the present invention, S43 includes:

[0036] S43a: Set a counter B for each slot of the programmable switch, with an initial value of 0;

[0037] S43b: When the programmable switch receives a data packet, it aggregates the vector p.vector of the data packet into the slot addressed by the pool index p.idx of the data packet. After each data packet from a server is received by the corresponding slot, the vector in the current data packet is added to the vector already aggregated in the slot, and the value of the counter B of the slot is incremented by 1.

[0038] S43c: When the counter value of a slot reaches the number of servers, the programmable switch uses the aggregated data in the slot to update the vector in the original data packet p and broadcasts the updated data packet to each server, while setting the aggregated vector and counter B of the current slot to zero.

[0039] In one embodiment of the present invention, S44 includes:

[0040] S44a: The first compute node among all servers receives the aggregated data packet from the programmable switch, and inserts the vector in the aggregated data packet into the position of the corresponding offset in the first gradient data segment according to the offset p.off field of the aggregated data packet;

[0041] S44b: Add x·y to the offset of the aggregated data packet, find the vector corresponding to the current offset, and construct a new data packet p(z,idx,off+x·y,vector). If the offset value of the new data packet is less than the data size of a gradient block, repeat steps S42c-S42d. Otherwise, do not process the data packet until the value of counter A becomes the number of segments contained in each gradient block. Then, all data packets in the gradient block of the current computing node have been sent, and the value of counter A is set to 0.

[0042] S44c: All servers will sequentially execute steps S42, S43, and S44a-S44d for the gradient data of the second gradient block of the second computing node, the third gradient block of the third computing node, ..., the Nth gradient block of the Nth computing node.

[0043] Another aspect of the present invention provides a distributed machine learning gradient synchronization system based on intranet computing, for executing the distributed machine learning gradient synchronization method described in any of the above embodiments. The system includes a programmable switch and multiple servers respectively connected to the switch. Each server is configured with multiple computing nodes, and each computing node is equipped with a convolutional neural network.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0045] 1. The present invention first performs reduction-distribution among the computing nodes inside the server, so that each computing node has only one complete copy of the data to be sent to the parameter server node. The data volume is only 1 / N times the size of the gradient data, which can significantly reduce the amount of data transmitted in the link, thereby alleviating the congestion problem at the network card of the parameter server node and greatly improving the communication efficiency of the parameter server synchronization mode.

[0046] 2. This invention uses a programmable switch to replace the traditional parameter server node, offloading some computation to the programmable device. The switch connects to each server in the distributed cluster with abundant uplink and downlink ports. The total amount of data transmitted does not increase, and no additional network bandwidth is required. Only the uplink and downlink bandwidth of each working node is needed. This invention can reduce data communication pressure and thus improve the overall system throughput.

[0047] 3. This invention divides the computation parameters between the end server and the switch to avoid the limitation of the limited computing power on the switch. The switch performs integer aggregation, while the end server is responsible for managing reliability and performing more complex calculations. In addition, the pool-based aggregation of the switch in this invention can avoid storage problems caused by the model size far exceeding the storage capacity of the switch.

[0048] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0049] Figure 1 This is a flowchart of a distributed machine learning gradient synchronization method based on intra-network computation provided by an embodiment of the present invention;

[0050] Figure 2 This is a detailed flowchart of a distributed machine learning gradient synchronization method based on intra-network computation provided by an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the structure of a distributed machine learning gradient synchronization system based on intranet computing provided in an embodiment of the present invention. Detailed Implementation

[0052] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following describes in detail a distributed machine learning gradient synchronization method and system based on intra-network computing proposed in accordance with the present invention, in conjunction with the accompanying drawings and specific embodiments.

[0053] The foregoing and other technical contents, features, and effects of the present invention will be clearly presented in the following detailed description of specific embodiments in conjunction with the accompanying drawings. Through the description of the specific embodiments, a more in-depth and concrete understanding can be gained of the technical means and effects adopted by the present invention to achieve its intended purpose. However, the accompanying drawings are for reference and illustration only and are not intended to limit the technical solutions of the present invention.

[0054] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that an article or apparatus comprising a list of elements includes not only those elements but also other elements not expressly listed. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or apparatus that includes said element.

[0055] Example 1

[0056] Please see Figure 1 , Figure 1 This is a flowchart illustrating a distributed machine learning gradient synchronization process based on intra-network computation, provided by an embodiment of the present invention. The method includes:

[0057] S1: Construct a distributed machine learning cluster, which includes a programmable switch and multiple servers connected to the switch, each server being configured with multiple computing nodes.

[0058] In this embodiment, the distributed machine learning cluster includes one programmable switch and four servers connected to the programmable switch, with one network port of each server connected to a port of the programmable switch. Each server is configured with eight compute nodes, therefore the distributed machine learning cluster contains a total of 32 compute nodes.

[0059] S2: Build a training set and allocate training set data to each compute node of each server.

[0060] Specifically, S2 in this embodiment includes:

[0061] S2a: Select a large number of images as an image sample. Each image contains a target type label, where the target refers to an airplane, car, ship, etc.

[0062] The images in this embodiment are largely derived from the open-source CIFAR-10 dataset, with a total of 20,000 images selected. The target type in each image is labeled to generate a label file. All label files are combined into a label set, and then the image samples and their corresponding target type labels are used to form a training set.

[0063] S2b: Divide the training set containing 20,000 pairs of image samples and labels into a subset of the same number of computing nodes. In this embodiment, it is 32 subsets. Assign an independent and ordered subset to each computing node. That is, send a subset to a computing node.

[0064] S3: Build a convolutional neural network at each computing node, and train the convolutional neural network at the current computing node using a subset of the training set allocated to each computing node to generate a set of gradient data at each computing node.

[0065] The convolutional neural network in this embodiment includes a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a third convolutional layer, a third pooling layer, a fourth convolutional layer, and a fully connected layer, arranged sequentially. The parameters of each layer of this convolutional neural network are as follows: the number of convolutional kernels in the first, second, third, and fourth convolutional layers are set to 16, 16, 8, and 8, respectively; the kernel sizes are set to 3×3, 5×5, 3×3, and 5×5, respectively; and the stride is set to 1 for all layers. The first, second, third, and fourth pooling layers all use average pooling, with pooling region sizes set to 1×4, 1×4, 1×1, and 1×1, respectively; and the stride is set to 1 for all layers. The number of neurons in the fully connected layer is set to 2.

[0066] Specifically, in a distributed machine learning cluster, each computing node inputs a subset of the received dataset into the convolutional neural network at its current computing node. The convolutional neural network computes layer by layer and obtains the predicted data; it calculates an error function to measure the difference between the true label result and the predicted result; it performs backpropagation layer by layer, using stochastic gradient descent to calculate the partial derivatives (gradients) corresponding to the parameters of the convolutional neural network model, and adjusts the corresponding weights. After each round of forward and backward propagation in machine learning is completed, a complete gradient data block will be generated on each computing node.

[0067] S4: Use the programmable switch and the server to perform data mixing and synchronization on gradient data blocks on all computing nodes, so that each computing node can obtain complete gradient data after the gradient data on all computing nodes are fused.

[0068] In this embodiment, the data mixing and synchronization process includes five steps, specifically step S4 includes the following steps:

[0069] S41: Perform a reduce-scatter process on the gradient data blocks on all computing nodes so that each computing node in each server can obtain the gradient data of all nodes in the current server.

[0070] Specifically, step S41 of this embodiment includes:

[0071] S41a: Construct a logical loop for the computing nodes within each server, and number the computing nodes within each server. The numbering of the N computing nodes within the s-th server is G. s-1 G s-2 ,...,G s-n ,...,G s-N Among them, G s-n Let G represent the nth compute node in the s-th server, and N represent the total number of compute nodes in each server. In this embodiment, N = 8, 1 ≤ s ≤ 4, and 1 ≤ n ≤ 8. For example, G 1-1 This indicates the first compute node within the first server.

[0072] S41b: Divide the gradient data of size K on each computing node into N gradient blocks, and number each gradient block sequentially as chunk0, chunk1, ..., chunk N-1 .

[0073] S41c: Set the left and right neighbors of each compute node in the server. Its left neighbor is the compute node before the current compute node, and its right neighbor is the compute node after the current compute node. The right neighbor of the last compute node is the first compute node, and the left neighbor of the first compute node is the last compute node.

[0074] Specifically, the left and right neighbors of each computing node in the server are defined as follows: G s-n The left neighbor of the compute node is G s-[(n-1)％8] The right neighbor is G s-[(n+1)％8] The % symbol represents modulo. That is, in each server, the left neighbor of a compute node is the node with the number minus 1, and its right neighbor is the compute node with the number plus 1. Special case: the left neighbor of compute node number 1 is the node with the number 8, and the right neighbor of compute node number 8 is the compute node with the number 1.

[0075] S41d: Perform N-1 data transfers. During the i-th data transfer, the number is G. s-n The computing node receives data from its left neighbor, which is numbered chunk. (n-i-1)％8 The gradient block, and simultaneously send its numbered chunk (n-i)％8 The gradient block is drawn from the right neighbor, where the initial value of i is 1 and the range is 1≤i<N.

[0076] S41e: Perform a summation operation: add the number G... s-n The gradient block received after N-1 data transfers within the computing node and the chunk numbered (n+i)％8 The gradient blocks are summed.

[0077] S41f: Let i increase from 0 to 7. Repeat steps S41c-S41e after each summation. Finally, each computing node in each server obtains the fused gradient block from all nodes in the current server, with a size of K / N. In this embodiment, N=8.

[0078] S42: The second step of the hybrid synchronization method is to slice the gradient block after fusion of each computing node into data slices, obtain multiple data packets, and send them to the corresponding aggregation slot of the programmable switch.

[0079] Specifically, step S42 of this embodiment includes:

[0080] S42a: Assume the size of the fused gradient block at each compute node is a multiple of the vector size x that can be aggregated in each slot of the programmable switch, and greater than xy, where y represents the number of pools in the programmable switch. The fused gradient block at each compute node is sliced ​​into multiple fragments, each fragment being the same size as the vector size in each slot. The number of fragments contained in each gradient block is:

[0081] NUM chunk_packet =K / (8·x).

[0082] S42b: Construct each fragment into a data packet. Each data packet p contains a segmented data vector of size x to be aggregated, along with a pool index idx and a compute node number z. Set the offset p.off of data packet p to the product of x and the data packet index. Finally, p(z,id,x of,f ve)c represents the data and index information in each data packet to be sent, where p.vector represents the vector of the data packet.

[0083] S42c: Control the first compute node of each server to send a number of data packets equal to the number y in the pool of the programmable switch. Each data packet is sent to the aggregation slot with the same index number of the programmable switch in index order. Simultaneously, a counter A is set, initially set to 0. After each data packet is sent, the value of counter A is incremented by 1. S43: Execute the third step of the hybrid synchronization method: use the aggregation slot corresponding to the programmable switch to aggregate the data packets received from the compute nodes, and broadcast the aggregated data to the corresponding compute nodes of all servers.

[0084] Step S43 in this embodiment specifically includes:

[0085] S43a: Set a counter B for each slot of the programmable switch, with an initial value of 0;

[0086] S43b: When the programmable switch receives a data packet, the programmable switch aggregates the vector p.vector of the data packet into the slot addressed by the pool index p.idx of the data packet. After each data packet from a server is received, the corresponding slot adds the vector in the current data packet to the vector already aggregated in the slot and increments the value of the counter B of that slot by 1.

[0087] For example, in this embodiment, according to the set pool index, the first fragment of the first computing node of each server is sent to the first slot in the programmable switch, and the second fragment of the first computing node of each server is sent to the second slot in the programmable switch. At the same time, the data packets that have been sent to the first slot of the switch are aggregated in the first slot, and so on.

[0088] The first compute node of each server sends a data packet of size y, and each data packet is sent to the aggregation slot of the same index number on the switch in index order.

[0089] S43c: When the counter B value of a slot reaches the number of servers s in the distributed cluster, it means that the slot has collected gradient data from a certain computing node across all servers. At this point, the programmable switch updates the vector in the original data packet p using the aggregated data in the slot; the updated data packet is then broadcast to each server. Simultaneously, the aggregated vector and counter B of that slot are set to zero.

[0090] S44: The fourth step of the hybrid synchronization method is to overwrite and replace the original data in the current server after receiving the aggregated data.

[0091] Step S44 in this embodiment specifically includes:

[0092] S44a: After the programmable switch broadcasts a data packet, the first compute node among all servers will receive an aggregated data packet from the switch. Each received aggregated data packet indicates that the switch has completed aggregation for a specific slot. Based on the offset p.off field of the received aggregated data packet, the vector in the aggregated data packet is inserted into the corresponding offset position in the first gradient data segment;

[0093] S44b: Add x·y to the offset of the aggregated data packet, find the vector corresponding to the current offset, and construct a new data packet p(z,idx,off+x·y,vector). If the offset value of the new data packet is less than the data size of a gradient block, repeat steps S42c-S42d. Otherwise, do not process the data packet until the value of counter A becomes the number of segments contained in each gradient block. Then, all data packets in the gradient block of the current computing node have been sent, and the value of counter A is set to 0.

[0094] S44c: Perform steps S42, S43 and S44a-S44d on the gradient data of the second gradient block of the second computing node, the third gradient block of the third computing node, ..., the Nth gradient block of the Nth computing node for all servers.

[0095] S45: The fifth step in executing the hybrid synchronization method is to perform all-gather on each compute node within the server.

[0096] Specifically, similar to step S41, N-1 synchronizations are performed. During the i-th synchronization, the sequence number is G. s-n The computing node receives data from its left neighbor G. s-[(n-1)％N] The Chinese number is chunk (n-i-2)％N The gradient data, and simultaneously send its chunk number. (n-i-1)％N Gradient data is given to the right neighbor G s-[(n+1+i)％N]Perform an overwrite operation: Number G s-n The computing nodes will overwrite their own chunks with the received gradient data. (n-i-2)％N The gradient data. It should be noted that the left and right neighbors of the computing nodes in each server are defined as above. Ultimately, each computing node in each server will contain fully fused gradient data of size K / N, thus completing the data synchronization of each computing node.

[0097] S5: Update the corresponding convolutional neural network parameters using the gradient data synchronously completed at each computing node.

[0098] Specifically, all computing nodes in the distributed machine learning cluster use the synchronously completed gradient data to update the network parameters in the convolutional neural network at the current computing node, thus obtaining the updated convolutional neural network for each node.

[0099] S6: Iteratively train the convolutional neural network to obtain the trained convolutional neural network model.

[0100] Specifically, the training of the convolutional neural network is repeated from S3 to S5 until the number of training iterations reaches a predetermined value, at which point the training ends and the trained convolutional neural network model is obtained.

[0101] In another embodiment of the present invention, a distributed machine learning gradient synchronization system based on intranet computing is also disclosed, characterized in that it is used to execute the distributed machine learning gradient synchronization method described in the above embodiments. The system includes a programmable switch and multiple servers respectively connected to the switch. Each server is configured with multiple computing nodes, and each computing node is equipped with a convolutional neural network.

[0102] In this embodiment of the invention, reduce-scatter is first performed between the computing nodes inside the server, so that each computing node has only one complete copy of the data to be sent to the parameter server node. The data volume is only 1 / N times the size of the gradient data, which can significantly reduce the amount of data transmitted in the link, thereby alleviating the congestion problem at the network card of the parameter server node and greatly improving the communication efficiency of the parameter server synchronous mode.

[0103] This invention uses a programmable switch instead of a traditional parameter server node, offloading some computation to a programmable device. The switch connects to each server in the distributed cluster with abundant uplink and downlink ports. The total amount of data transmitted does not increase, and no additional network bandwidth is required; only the uplink and downlink bandwidth of each worker node is needed. This invention reduces data communication pressure, thereby improving overall system throughput. This invention divides computational parameters between the end server and the switch to avoid the limitations of the switch's limited computing power. The switch performs integer aggregation, while the end server manages reliability and performs more complex calculations. Furthermore, the pool-based aggregation of the switch in this invention avoids storage problems caused by model sizes far exceeding the switch's storage capacity.

[0104] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A distributed machine learning gradient synchronization method based on intra-network computation, characterized in that, include: S1: Construct a distributed machine learning cluster, including a programmable switch and multiple servers connected to the switch, with each server configured with multiple computing nodes; S2: Build a training set and allocate a subset of the training set to each compute node of each server; S3: Build a convolutional neural network at each computing node, and train the convolutional neural network at the current computing node using a subset of the training set allocated to each computing node to generate a set of gradient data at each computing node. S4: Use the programmable switch and the server to perform data mixing and synchronization on gradient data blocks on all computing nodes, so that each computing node can obtain the complete gradient data after the gradient data on all computing nodes are fused. S5: Update the parameters of each convolutional neural network using the complete gradient data; S6: Iteratively train the convolutional neural network to obtain the trained convolutional neural network model; S4 includes: S41: Perform reduction-dispersion on gradient data blocks on all computing nodes so that each computing node in each server obtains the fused gradient data from all nodes in the current server; S42: Perform data slicing on the gradient block after fusion of each computing node to obtain multiple data packets and send them to the corresponding aggregation slot of the programmable switch; S43: Utilize the aggregation slots corresponding to the programmable switch to aggregate the data packets received from the computing nodes and broadcast the aggregated data to the corresponding computing nodes of all servers; S44: After receiving the aggregated data, the corresponding compute nodes in all servers overwrite and replace the original data in the current server. S45: Perform full aggregation of the current data of all computing nodes in each server so that each computing node in each server obtains the complete gradient data after the gradient data of all computing nodes are fused. Furthermore, S43 includes: S43a: Set a counter B for each slot of the programmable switch, with an initial value of 0; S43b: When the programmable switch receives a data packet, it will convert the data packet vector... Aggregation to the pool index of data packets In the addressed slot, after each data packet received from a server, the corresponding slot adds the vector in the current data packet to the vector that has already been aggregated in the slot, and increments the value of the slot's counter B by 1; S43c: When the counter value of a slot reaches the number of servers, the programmable switch uses the aggregated data in the slot to update the vector in the original data packet p and broadcasts the updated data packet to each server, while setting the aggregated vector and counter B of the current slot to zero.

2. The distributed machine learning gradient synchronization method based on intra-network computation according to claim 1, characterized in that, S2 includes: S2a: Select a large number of images as image samples, each image containing a target type label, and combine the image samples and the corresponding target type labels to form a training set; S2b: Divide the training set into a number of training set subsets equal to the total number of computing nodes, and assign an independent training set subset to each computing node.

3. The distributed machine learning gradient synchronization method based on intra-network computation according to claim 1, characterized in that, Each convolutional neural network at each computing node includes a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a third convolutional layer, a third pooling layer, a fourth convolutional layer, and a fully connected layer connected in sequence.

4. The distributed machine learning gradient synchronization method based on intra-network computation according to claim 1, characterized in that, S41 includes: S41a: Number the compute nodes within each server. ,in, This represents the nth computing node in the s-th server, where N represents the total number of nodes in each server. S41b: Divide the gradient data set on each computing node into N gradient blocks, and number each gradient block sequentially. ; S41c: Set the left and right neighbors of each compute node in each server. The left neighbor is the compute node before the current compute node, the right neighbor is the compute node after the current compute node, the right neighbor of the last compute node is the first compute node, and the left neighbor of the first compute node is the last compute node. S41d: Execute N -1 data transmission, in the... i During this data transmission, the number is The computing node receives data from its left neighbor, numbered as follows. The gradient block, and simultaneously send its number as The gradient block is directed to its right neighbor, where, i The initial value is 1, and the range is ; S41e: The number is... Within the computing nodes N The gradient block received after -1 data transmission and its number are... The gradient blocks are summed. S41f: Order i The gradient is added from 0 to 7. After each addition, steps S41c-S41e are repeated. Finally, each computing node in each server obtains the gradient block of all nodes in the current server.

5. The distributed machine learning gradient synchronization method based on intra-network computation according to claim 4, characterized in that, S42 includes: S42a: Divide the fused gradient block at each computing node into multiple segments, with the size of each segment being the same as the vector size in each slot of the programmable switch; S42b: Construct each fragment into a data packet, the data packet including the segmented data vector and carrying a pool index. idx and a computing node number z ; S42c: Controls the first compute node of each server to send a number of data packets equal to the number of pools in the programmable switch, with each data packet sent in index order to the aggregation slot of the same index number in the programmable switch.

6. The distributed machine learning gradient synchronization method based on intra-network computation according to claim 5, characterized in that, S44 includes: S44a: The first compute node among all servers receives the aggregated data packet from the programmable switch, based on the offset of the aggregated data packet. The field inserts the vector from the aggregated data packet into the corresponding offset position in the first gradient data segment; S44b: Add the aggregated data packet offset to... Find the vector corresponding to the current offset and construct a new data packet. If the offset value of the new data packet is less than the data size of a gradient block, repeat steps S42c-S42d; otherwise, do not process the data packet until the value of counter A becomes the number of segments contained in each gradient block. Then, all data packets in the gradient block of the current computing node have been sent, and the value of counter A is set to 0. S44c: All servers will sequentially execute steps S42, S43, and S44a-S44d for the gradient data of the second gradient block of the second computing node, the third gradient block of the third computing node, ..., the Nth gradient block of the Nth computing node.

7. A distributed machine learning gradient synchronization system based on intra-network computing, characterized in that, The system for executing the distributed machine learning gradient synchronization method according to any one of claims 1 to 6 includes a programmable switch and multiple servers respectively connected to the switch, each server being configured with multiple computing nodes, and each computing node being provided with a convolutional neural network.

Images