Communication methods, systems, and related devices for distributed training

Abstract

The application discloses a kind of distributed training communication method, system and related equipment, it is related to distributed learning field.Distributed training communication method includes: center node generates and issues data packet, wherein, the information of model updated by center node is included in the data packet issued, and the segmented routing head of the data packet issued includes the segment list corresponding to multiple computing nodes, each segment list therein includes the address of corresponding computing node and training instruction;Center node sends the data packet issued to next hop equipment.Center node can make multiple computing nodes obtain the information of model updated by once issuing process.Therefore, the embodiment of the application can reduce the occupation of bandwidth, save resources in the process of distributed training, reduce the overhead in the process of distributed training.

Description

Technical Field

[0001] This invention relates to the field of distributed learning, and in particular to a communication method, system, and related equipment for distributed training. Background Technology

[0002] Federated Learning (FL) allows multiple nodes to jointly train a global model without sharing data locally. In the federated learning process, the central node first distributes the initialized model to the compute nodes; then, the compute nodes update their models locally using a training program; next, the central node receives the locally trained models from the compute nodes and aggregates them to generate a new global model; finally, the central node distributes the new global model to the compute nodes for a new round of training until the model converges. Summary of the Invention

[0003] After analysis, the inventors discovered that distributed training processes such as federated learning often involve significant communication overhead, especially in systems with numerous non-agent nodes, where the network issues are even more severe. For example, a central node simultaneously distributing the same global model to numerous computing nodes can lead to unnecessary bandwidth consumption and resource waste.

[0004] One of the technical problems to be solved by the embodiments of the present invention is: how to reduce the overhead in the distributed training process.

[0005] According to a first aspect of some embodiments of the present invention, a communication method for distributed training is provided, comprising: a central node generating a data packet to be sent, wherein the data packet includes information of the model updated by the central node, and the segmented routing header of the data packet includes a segment list corresponding to multiple computing nodes, wherein each segment list includes the address of the corresponding computing node and a training instruction; the central node sending the data packet to the next-hop device.

[0006] In some embodiments, for each segment list corresponding to multiple computing nodes, the location field of the segment list includes the address of the corresponding computing node, and the function field of the segment list includes a training instruction.

[0007] In some embodiments, the payload field of the sent data packet includes information about the updated model from the central node.

[0008] In some embodiments, the segmented routing header of the sent data packet also includes a segment list corresponding to one or more forwarding nodes, wherein each segment list includes the network address of the corresponding forwarding node and an indication not to process it.

[0009] In some embodiments, the plurality of compute nodes include a proxy node and a plurality of non-proxy nodes, and the segment list corresponding to the proxy node is located at the first position in the segment list corresponding to the plurality of compute nodes.

[0010] In some embodiments, the data packets sent are segment-routing data packets based on the Internet Protocol version 6 (IPv6) forwarding plane.

[0011] In some embodiments, the communication method further includes: a central node receiving an upload data packet sent by one or more proxy nodes, wherein the upload data packet includes information about an intermediate model, the intermediate model being generated by the proxy node sending the upload data packet aggregating the trained models of the proxy nodes and the trained models from non-proxy nodes; and the central node aggregating the intermediate models sent by one or more proxy nodes to generate an updated model.

[0012] In some embodiments, the segmentation routing header of the uploaded data packet includes a segment list corresponding to the central node, and the segment list corresponding to the central node includes a convergence indication.

[0013] In some embodiments, the location field of the segment list corresponding to the central node includes the address of the central node, and the function field includes a convergence indicator.

[0014] In some embodiments, information about the intermediate model is located in the payload field of the uploaded data packet.

[0015] In some embodiments, the uploaded data packet is an SRv6 data packet.

[0016] In some embodiments, the central node is the central node of the federated learning, and the computing nodes are the distributed nodes of the federated learning.

[0017] According to a second aspect of some embodiments of the present invention, a communication method for distributed training is provided, comprising: a computing node receiving a data packet from a central node, wherein the data packet includes an updated model from the central node, and the segmented routing header of the data packet includes a segment list corresponding to multiple computing nodes, wherein each segment list includes the address of the corresponding computing node and a training instruction; and the computing node updating and training its local model according to the data packet.

[0018] In some embodiments, the training method further includes: the computing node determining the next segment of the segment list corresponding to the computing node; and the computing node sending a data packet to the device corresponding to the next segment.

[0019] In some embodiments, the computing node is a proxy node, and the communication method further includes: the proxy node aggregating the trained model from the proxy node and the trained model from the non-proxy node to generate an intermediate model; the proxy node generating an upload data packet including information about the intermediate model, wherein the segmented routing header of the upload data packet includes a segment list corresponding to the central node, and the segment list corresponding to the central node includes an aggregation instruction; the proxy node sending the upload data packet to the central node, so that the central node aggregates the intermediate models sent by one or more proxy nodes according to the aggregation instruction to generate an updated model.

[0020] According to a third aspect of some embodiments of the present invention, a central node is provided, comprising: a generation module configured to generate a data packet to be sent, wherein the data packet to be sent includes information about the updated model of the central node, and the segmented routing header of the data packet to be sent includes a segment list corresponding to multiple computing nodes, wherein each segment list includes the address of the corresponding computing node and a training instruction; and a sending module configured to send the data packet to a next-hop device.

[0021] According to a fourth aspect of some embodiments of the present invention, a computing node is provided, comprising: a receiving module configured to receive a data packet sent from a central node, wherein the data packet sent includes an updated model from the central node, and the segmented routing header of the data packet sent includes a segment list corresponding to multiple computing nodes, wherein each segment list includes the address of the corresponding computing node and a training instruction; and a training module configured to update and train a local model according to the data packet sent.

[0022] In some embodiments, when the computing node is a proxy node, the computing node further includes: an aggregation module configured to aggregate the trained models from the proxy node and the trained models from non-proxy nodes to generate an intermediate model; a generation module configured to generate an upload data packet including information about the intermediate model, wherein the segmented routing header of the upload data packet includes a segment list corresponding to the central node, and the segment list corresponding to the central node includes an aggregation instruction; and a sending module configured to send the upload data packet to the central node so that the central node aggregates the intermediate models sent by one or more proxy nodes according to the aggregation instruction to generate an updated model.

[0023] According to a fifth aspect of some embodiments of the present invention, a communication system for distributed training is provided, comprising: the aforementioned central node; and a plurality of the aforementioned computing nodes.

[0024] In some embodiments, the communication system further includes: a proxy node selection device configured to determine a proxy node from a plurality of computing nodes based on attribute information of the plurality of computing nodes.

[0025] In some embodiments, the attribute information includes at least one of uplink bandwidth, computing power, and distance from the central node.

[0026] In some embodiments, the proxy node selection device is further configured to determine a proxy node from a plurality of computing nodes based on a weighted result of at least one of the following: uplink bandwidth, computing power, and distance from the central node for each computing node.

[0027] According to a sixth aspect of some embodiments of the present invention, a communication apparatus for distributed training is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute any of the aforementioned communication methods for distributed training based on instructions stored in the memory.

[0028] According to a seventh aspect of some embodiments of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, wherein the program, when executed by a processor, implements any of the aforementioned distributed training communication methods.

[0029] Some embodiments of the above invention have the following advantages or beneficial effects. This invention utilizes the programmable nature of SRv6 paths to orchestrate multiple receiving objects (i.e., multiple computing nodes) of the updated model into the same path. Thus, the central node can send the updated model information to multiple computing nodes in a single distribution process. Therefore, embodiments of this invention can reduce bandwidth consumption, save resources, and lower overhead during distributed training.

[0030] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 A flowchart illustrating a model distribution method according to some embodiments of the present invention is shown.

[0033] Figure 2 A schematic flowchart of a model uploading method according to some embodiments of the present invention is shown.

[0034] Figure 3 An exemplary diagram illustrating the distributed training process is provided.

[0035] Figure 4 An exemplary diagram illustrating the upload process of distributed training is shown.

[0036] Figure 5 A schematic diagram of the structure of a central node according to some embodiments of the present invention is shown.

[0037] Figure 6 A schematic diagram of the structure of a computing node according to some embodiments of the present invention is shown.

[0038] Figure 7A A schematic diagram of the structure of a communication system for distributed training according to some embodiments of the present invention is shown.

[0039] Figure 7B A schematic diagram of the structure of a communication system for distributed training according to other embodiments of the present invention is shown.

[0040] Figure 8 A schematic diagram of the structure of a communication device for distributed training according to some embodiments of the present invention is shown.

[0041] Figure 9 A schematic diagram of the structure of a communication device for distributed training according to other embodiments of the present invention is shown. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0044] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0045] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0046] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0047] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0048] After analysis, the inventors discovered that the central node does not need to send the updated model information to all computing nodes. Instead, it can carry the addresses of multiple computing nodes in the routing protocol header, thereby orchestrating the computing nodes that need to obtain this information onto the same path. Through the data transmission process between the nodes, each node sequentially obtains the model information. See below for reference. Figure 1 An embodiment of the communication method for distributed training according to the present invention is described.

[0049] Figure 1 A flowchart illustrating a model distribution method according to some embodiments of the present invention is shown. For example... Figure 1 As shown, the communication method for distributed training in this embodiment includes steps S102 to S104.

[0050] In step S102, the central node generates a data packet to be sent. The data packet includes the updated model of the central node, and the SRH (Segment Routing Header) of the data packet includes a segment list corresponding to multiple computing nodes. Each segment list includes the address of the corresponding computing node and training instructions.

[0051] The multiple computing nodes involved in a single data packet can be all or some of the computing nodes in distributed training. When all computing nodes are involved, the central node can send only one data packet for a single model update. When only some computing nodes are involved, the central node can send multiple data packets for a single model update, but the number is still less than the number of all computing nodes. Therefore, network overhead can be saved in either case.

[0052] The SRH includes multiple segment lists, such as Segment List[0], Segment List[1], ..., Segment List[n]. Each segment list represents the address of a device, such as a 128-bit IPv6 address, used to indicate packet forwarding. By arranging the order of the segment lists, the updated model distribution path can be programmed.

[0053] Each segment list includes three fields: Locator, Function, and Args, with the Args field being optional. The Locator is mainly used to carry routing location information, such as the address of the compute node; the Function is used to bind packet forwarding operation instructions, such as carrying training instructions.

[0054] In some embodiments, the distributed data packets are SRv6 (Segment Routing over IPv6) packets. Thus, the central node can implement the updated model distribution process based on SRv6.

[0055] In some embodiments, the payload field of the sent data packet includes information about the updated model from the central node. The updated model information can be complete model information or partial model information, such as parameter values, loss values, gradient values, etc.

[0056] In step S104, the central node sends the data packet to the next-hop device, i.e., the first device in the path corresponding to the SRH, such as the device corresponding to the largest segment in the segmentation routing header. Thus, each device involved in the SRH can forward the data packet sequentially according to the SRH content and perform corresponding operations based on the instructions within it.

[0057] For the computing nodes in the path, after receiving the data packet, they update and train the local model according to the data packet.

[0058] In addition to computation nodes, the path may also include forwarding nodes. In embodiments of the present invention, forwarding nodes refer to nodes that have not deployed models and are only used for forwarding, without participating in the specific training process. For forwarding nodes in the path, after receiving the data packet, they do not perform model- and training-related processing; instead, after performing the basic routing operations necessary for forwarding, they send it to the next node.

[0059] In some embodiments, the segmented routing header of the distributed data packets also includes a list of segments corresponding to one or more forwarding nodes, where each segment list includes the network address of the corresponding forwarding node and an indication not to process it. Thus, during the distribution of the updated model, compute nodes and forwarding nodes can be orchestrated in the same path, improving the flexibility of path orchestration.

[0060] The above embodiments utilize the programmable nature of SRv6 paths to orchestrate multiple receiving objects (i.e., multiple computing nodes) of the updated model into the same path. Thus, the central node can send the updated model information to multiple computing nodes in a single distribution process. Therefore, the embodiments of the present invention can reduce bandwidth consumption, save resources, and lower overhead during distributed training.

[0061] In some embodiments, the model uploading process can be further optimized. For example, to avoid bandwidth consumption and congestion caused by the central node receiving models uploaded by all computing nodes one by one, some embodiments of the present invention further divide computing nodes into proxy nodes and non-proxy nodes. The proxy nodes first aggregate the training results of themselves and multiple non-proxy nodes, and then upload the aggregated model to the central node, thereby reducing the communication pressure on the central node. See below for reference. Figure 2 An embodiment of the model uploading method of the present invention is described.

[0062] Figure 2 A flowchart illustrating a model upload method according to some embodiments of the present invention is shown. Figure 2 As shown, the model uploading method in this embodiment includes steps S202 to S208.

[0063] In step S202, the proxy node aggregates the trained models from the proxy node and the trained models from non-proxy nodes to generate an intermediate model.

[0064] In step S204, the proxy node generates an upload data packet that includes information about the intermediate model. The segmentation header of the upload data packet includes a segment list corresponding to the central node, and the segment list corresponding to the central node includes a convergence indication.

[0065] In some embodiments, the segmentation routing header of the uploaded data packet includes a segment list corresponding to the central node, and the segment list corresponding to the central node includes a convergence indicator. For example, the Locator of the segment list corresponding to the central node includes the address of the central node, and the Function includes a convergence indicator. Thus, the central node can know that the content of the data packet is used for model convergence.

[0066] In some embodiments, the intermediate model information is located in the payload field of the uploaded data packet. The intermediate model information can be complete model information or partial model information, such as parameter values, loss values, gradient values, etc.

[0067] In some embodiments, the uploaded data packet is an SRv6 data packet. Therefore, the proxy node can implement the model upload process based on SRv6.

[0068] In step S206, the proxy node sends the upload data packet to the central node.

[0069] In step S208, the central node aggregates the intermediate models sent by one or more agent nodes according to the aggregation instruction to generate an updated model.

[0070] The above embodiments, by setting up proxy nodes, enable these nodes to pre-aggregate the training results of some computing nodes and send them to the central node. This reduces the data reception pressure on the central node, lowers bandwidth usage and network resource consumption, and thus saves the overhead of the distributed training upload phase.

[0071] The following is for reference. Figure 3 and Figure 4 The distribution and uploading processes of distributed training are described exemplarily.

[0072] Figure 3 An exemplary diagram illustrating the distributed training process is provided. Figure 3 As shown, the central node needs to send the updated model to the proxy node, edge node 1, and edge node 2 in sequence, where edge node 1 and edge node 2 are computing nodes and are not proxy nodes.

[0073] The central node generates a data packet, and its SRH includes four Segment Lists. Segment List [3] includes the address of the next node in the distribution path of the central node, i.e., the address of the forwarding node 2001::2, and includes an indication not to process it; Segment List [2] includes the address of the proxy node 2001::3, and includes a training indication; Segment List [1] includes the address of edge node 1 2001::4, and includes a training indication; Segment List [0] includes the address of edge node 2 2001::5, and includes a training indication. The updated global model information is encapsulated in the data payload of the data packet. At this time, the source address (SA) of the data packet is the address of the central node, and the destination address (DA) is the address of the next hop of the central node, i.e., the address of the forwarding node.

[0074] Then, the central node begins to execute the forwarding process. First, the central node sends the data packet to the forwarding node according to the destination address. After receiving the data packet, the forwarding node extracts the instruction from the Function of Segment List[3] and finds that the instruction does not need to be processed. Therefore, it modifies the DA of the data packet to the address of the next hop proxy node and sends it to the proxy node according to the instruction of the DA.

[0075] After receiving the data packet, the proxy node extracts the instruction from the Function in the Segment List[2] and finds that it is instructed to train. Therefore, the proxy node extracts the updated model information from the data payload to update and train the local model. The proxy node modifies the DA of the data packet to the address of the next-hop edge node 1 and sends it to edge node 1 according to the instruction of the DA.

[0076] After receiving the data packet, edge node 1 extracts the instruction from the Function in the Segment List[1] and finds that it is instructed to train. Therefore, edge node 1 extracts the updated model information from the data payload to update and train the local model. Edge node 1 modifies the DA of the data packet to the address of the next-hop edge node 2 and sends it to edge node 2 according to the instruction of the DA.

[0077] After receiving the data packet, edge node 2 extracts the instruction from the Function in Segment List[0] and finds that it is instructed to train. Therefore, edge node 2 extracts the updated model information from the data payload to update and train the local model.

[0078] At this point, the data transmission process for this path is complete. Through a single data distribution process, the central node enables the proxy node, edge node 1, and edge node 2 to obtain the updated model information. Therefore, the overhead of the model distribution process is saved.

[0079] The computation nodes in the aforementioned path include both proxy nodes and non-proxy nodes. Those skilled in the art will understand that, depending on the needs, the objects to which the central node distributes updated model information can be entirely non-proxy nodes; this will not be elaborated further here. Furthermore, the distribution path may contain one or more forwarding nodes, or it may contain none.

[0080] Figure 4 An exemplary diagram illustrating the upload process of distributed training is provided. Figure 4As shown, after aggregating the trained models sent by non-agent nodes, the proxy node sends the information of the intermediate model generated by the aggregation to the central node. In this embodiment, the upload path includes proxy nodes, forwarding nodes, and the central node.

[0081] The proxy node generates an upload data packet, whose SRH includes two Segment Lists. Segment List [1] includes the address of the next node in the upload path of the proxy node, i.e., the address of the forwarding node 2001::2, and includes an indication not to process it; Segment List [0] includes the address of the central node 2001::1, and includes an aggregation indication. The information of the aggregated model is encapsulated in the data payload of the upload data packet. At this time, the source address (SA) of the upload data packet is the address of the proxy node, and the destination address (DA) is the address of the next hop of the proxy node, i.e., the address of the forwarding node.

[0082] Then, the proxy node begins to execute the forwarding process. First, the proxy node sends the uploaded data packet to the forwarding node according to the destination address. After receiving the sent data packet, the forwarding node extracts the instruction from the Function of Segment List[1] and finds that the instruction does not need to be processed. Therefore, it modifies the DA of the sent data packet to the address of the next hop central node and sends it to the central node according to the instruction of the DA.

[0083] After receiving the data packet, the central node extracts the instruction from the Function in Segment List[0] and finds that it is to perform aggregation. Therefore, the central node extracts the intermediate model information from the data payload. After the central node obtains the intermediate model information sent by all agent nodes in the network, it aggregates these intermediate models. Of course, if there are other aggregation triggering conditions (such as collecting a certain number of intermediate model information and the waiting time has reached a threshold), it is not necessary to aggregate the intermediate models of all agent nodes, but only to aggregate the acquired intermediate models.

[0084] At this point, the data transmission process for this path is complete. Through a single data upload, the proxy node enables the central node to obtain the training results from multiple computing nodes. Therefore, the overhead of the model upload process is saved.

[0085] In some embodiments, a Segment List (SID) function table can be defined to define the identifiers corresponding to certain operations in a distributed training scenario. An exemplary SID function table is shown in Table 1.

[0086] Table 1

[0087]

[0088]

[0089] Therefore, after receiving a data packet, each node can look up the SID table based on the identifier in the Function field to confirm and execute the corresponding operation.

[0090] Some embodiments of the present invention also provide a method for determining proxy nodes.

[0091] In some embodiments, surrogate nodes are determined from multiple computing nodes based on attribute information of these nodes. Unselected computing nodes are designated as non-surrogate nodes. The surrogate nodes are used to collect and aggregate trained models from these non-surrogate nodes to generate intermediate models.

[0092] In some embodiments, the attribute information includes at least one of uplink bandwidth, computing power, and distance to the central node. Thus, it is measured from both communication status and computing power perspectives. The distance between the computing node and the central node can be determined by obtaining the locations of the computing node and the central node respectively, and then calculating the difference between them. Computing power is reflected, for example, through indicators such as memory size and central processing unit performance.

[0093] In some embodiments, a proxy node is determined from multiple computing nodes based on a weighted result of at least one of the following: uplink bandwidth, computing power, and distance from the central node.

[0094] Let N be the number of computing nodes in the local federated learning environment (e.g., a city or a large industrial park). For the i-th node (i = 1, 2, ..., N), let its uplink bandwidth be bw. i The computing power is size i The geographic coordinates are (x i ,y i ,z i Let the geographical coordinates of the central node in the federated learning process be (a, b, c). Then the distance from each computation node to the central node is...

[0095] Then, the various factors can be normalized to achieve the updated uplink bandwidth. Updated computing power Updated distance Where min(bw) and max(bw) represent the minimum and maximum uplink bandwidth of the N nodes, respectively; min(size) and max(size) represent the minimum and maximum computing power of the N nodes, respectively; and min(dis) and max(dis) represent the minimum and maximum distance of the N nodes, respectively.

[0096] Next, the updated factors are weighted. The weighted result for the i-th node is sum. i =w b ×bw ′ i +w s ×size ′ i +w d ×dis ′ i Then, for example, you can choose the computation node with the largest weighted result as the proxy node. Of course, you can also consider and select other factors as needed, which will not be elaborated here.

[0097] Through the above embodiments, the attribute information of computing nodes can be comprehensively analyzed, and computing nodes with better comprehensive capabilities can be selected as proxy nodes to improve the training and transmission performance of the entire system, thereby improving training efficiency.

[0098] The following is for reference. Figure 5 An embodiment of the central node of the present invention is described.

[0099] Figure 5 A schematic diagram of the structure of a central node according to some embodiments of the present invention is shown. For example... Figure 5 As shown, the central node 500 in this embodiment includes: a generation module 5100, configured to generate a data packet, wherein the data packet includes information about the updated model of the central node, and the segmented routing header of the data packet includes a segment list corresponding to multiple computing nodes, wherein each segment list includes the address of the corresponding computing node and a training instruction; and a sending module 5200, configured to send the data packet to the next-hop device.

[0100] In some embodiments, for each segment list corresponding to multiple computing nodes, the location field of the segment list includes the address of the corresponding computing node, and the function field of the segment list includes a training instruction.

[0101] In some embodiments, the payload field of the sent data packet includes information about the updated model from the central node.

[0102] In some embodiments, the segmented routing header of the sent data packet also includes a segment list corresponding to one or more forwarding nodes, wherein each segment list includes the network address of the corresponding forwarding node and an indication not to process it.

[0103] In some embodiments, the plurality of compute nodes include a proxy node and a plurality of non-proxy nodes, and the segment list corresponding to the proxy node is located at the first position in the segment list corresponding to the plurality of compute nodes.

[0104] In some embodiments, the data packets sent are SRv6 data packets.

[0105] In some embodiments, the central node 500 further includes: a receiving module 5300 configured to receive an upload data packet sent by one or more proxy nodes, wherein the upload data packet includes information about an intermediate model, the intermediate model being generated by the proxy node sending the upload data packet by aggregating the trained models of the proxy node and the trained models from non-proxy nodes; and an aggregation module 5400 configured to aggregate the intermediate models sent by one or more proxy nodes to generate an updated model.

[0106] In some embodiments, the segmentation routing header of the uploaded data packet includes a segment list corresponding to the central node, and the segment list corresponding to the central node includes a convergence indication.

[0107] In some embodiments, the location field of the segment list corresponding to the central node includes the address of the central node, and the function field includes a convergence indicator.

[0108] In some embodiments, information about the intermediate model is located in the payload field of the uploaded data packet.

[0109] In some embodiments, the uploaded data packet is an SRv6 data packet.

[0110] In some embodiments, the central node 500 is the central node of the federated learning, and the computing nodes are the distributed nodes of the federated learning.

[0111] The following is for reference. Figure 6 An embodiment of the computing node of the present invention is described.

[0112] Figure 6 A schematic diagram of the structure of a computing node according to some embodiments of the present invention is shown. For example... Figure 6As shown, the computing node 600 in this embodiment includes: a receiving module 6100, configured to receive a data packet sent from a central node, wherein the data packet sent includes the updated model from the central node, and the segmented routing header of the data packet sent includes a segment list corresponding to multiple computing nodes, wherein each segment list includes the address of the corresponding computing node and a training instruction; and a training module 6200, configured to update and train the local model according to the data packet sent.

[0113] In some embodiments, when the computing node 600 is a proxy node, the computing node 600 further includes: an aggregation module 6300 configured to aggregate the trained models from the proxy node and the trained models from non-proxy nodes to generate an intermediate model; a generation module 6400 configured to generate an upload data packet including information about the intermediate model, wherein the segmented routing header of the upload data packet includes a segment list corresponding to the central node, and the segment list corresponding to the central node includes an aggregation instruction; and a sending module 6500 configured to send the upload data packet to the central node so that the central node aggregates the intermediate models sent by one or more proxy nodes according to the aggregation instruction to generate an updated model.

[0114] The following is for reference. Figure 7A and 7B An embodiment of the communication system for distributed training according to the present invention is described.

[0115] Figure 7A A schematic diagram of the structure of a communication system for distributed training according to some embodiments of the present invention is shown. For example... Figure 7A As shown, the communication system 70 of this embodiment includes a central node 500 and multiple computing nodes 600.

[0116] Figure 7B A schematic diagram of the structure of a communication system for distributed training according to other embodiments of the present invention is shown. For example... Figure 7B As shown, the communication system 70 of this embodiment includes: a central node 500, a proxy node 710, and multiple non-proxy nodes 720. The central node 500 is communicatively connected to the proxy node 710, and the proxy node 710 is communicatively connected to the multiple non-proxy nodes 720. The number of proxy nodes 720 can be increased as needed, which will not be elaborated here.

[0117] In some embodiments, the communication system 70 further includes a proxy node selection device 730, configured to determine a proxy node from a plurality of computing nodes based on attribute information of the plurality of computing nodes.

[0118] In some embodiments, the attribute information includes at least one of uplink bandwidth, computing power, and distance from the central node.

[0119] In some embodiments, the proxy node selection device 730 is further configured to determine a proxy node from a plurality of computing nodes based on a weighted result of at least one of the following: uplink bandwidth, computing power, and distance from the central node for each computing node.

[0120] Figure 8 A schematic diagram of the structure of a communication device for distributed training according to some embodiments of the present invention is shown. For example... Figure 8 As shown, the distributed training communication device 80 of this embodiment includes: a memory 810 and a processor 820 coupled to the memory 810. The processor 820 is configured to execute the distributed training communication method of any of the foregoing embodiments based on instructions stored in the memory 810.

[0121] The memory 810 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory may store, for example, the operating system, application programs, boot loader, and other programs.

[0122] Figure 9 A schematic diagram of the structure of a communication device for distributed training according to other embodiments of the present invention is shown. For example... Figure 9 As shown, the distributed training communication device 90 of this embodiment includes a memory 910 and a processor 920, and may also include an input / output interface 930, a network interface 940, a storage interface 950, etc. These interfaces 930, 940, 950, and the memory 910 and processor 920 can be connected, for example, via a bus 960. The input / output interface 930 provides a connection interface for input / output devices such as a display, mouse, keyboard, and touchscreen. The network interface 940 provides a connection interface for various networked devices. The storage interface 950 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0123] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program thereon, characterized in that the program, when executed by a processor, implements any of the aforementioned distributed training communication methods.

[0124] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0125] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0126] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0127] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0128] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A communication method for distributed training, comprising: The central node generates and distributes data packets, wherein the data packets include information about the updated model from the central node, and the segmented routing header of the data packets includes a list of segments corresponding to multiple computing nodes, wherein each segment list includes the address of the corresponding computing node and a training instruction, wherein the central node is the central node of federated learning, and the computing nodes are distributed nodes of federated learning; The central node sends the data packet to the next-hop device.

2. The communication method according to claim 1, wherein, For each segment list corresponding to the plurality of computing nodes, the location field of the segment list includes the address of the corresponding computing node, and the function field of the segment list includes the training instruction.

3. The communication method according to claim 1, wherein, The payload field of the sent data packet includes information about the updated model from the central node.

4. The communication method according to claim 1, wherein, The segmented routing header of the data packet also includes a segment list corresponding to one or more forwarding nodes, where each segment list includes the network address of the corresponding forwarding node and an indication not to process it.

5. The communication method according to claim 1, wherein, The plurality of computing nodes includes one proxy node and a plurality of non-proxy nodes, and the segment list corresponding to the proxy node is located at the first position in the segment list corresponding to the plurality of computing nodes.

6. The communication method according to claim 1, wherein, The data packets being sent are segment-routing data packets based on the Internet Protocol version 6 (IPv6) forwarding plane.

7. The communication method according to any one of claims 1 to 6, further comprising: The central node receives upload data packets sent by one or more proxy nodes, wherein the upload data packets include information about intermediate models, and the intermediate models are generated by the proxy node that sent the upload data packets by aggregating the trained models of the proxy node and the trained models from non-proxy nodes. The central node aggregates the intermediate models sent by the one or more proxy nodes to generate an updated model.

8. The communication method according to claim 7, wherein, The segmented routing header of the uploaded data packet includes a segment list corresponding to the central node, and the segment list corresponding to the central node includes a convergence indicator.

9. The communication method according to claim 8, wherein, The location field of the segment list corresponding to the central node includes the address of the central node, and the function field includes the convergence indicator.

10. The communication method according to claim 7, wherein, The information of the intermediate model is located in the payload field of the uploaded data packet.

11. The communication method according to claim 7, wherein, The uploaded data packet is a segment-routing data packet based on the Internet Protocol version 6 (IPv6) forwarding plane.

12. A communication method for distributed training, comprising: The computing nodes receive data packets from the central node, wherein the data packets include the updated model from the central node, and the segmented routing header of the data packets includes a list of segments corresponding to multiple computing nodes, wherein each segment list includes the address of the corresponding computing node and a training instruction, wherein the central node is the central node of federated learning, and the computing nodes are distributed nodes of federated learning. The computing node updates and trains its local model based on the sent data packet.

13. The communication method according to claim 12, further comprising: The computing node determines the next segment in the segment list corresponding to the computing node; The computing node sends the data packet to the device corresponding to the next segment.

14. The communication method according to claim 12, wherein, The computing node is a proxy node, and the communication method further includes: The proxy node aggregates the trained model from the proxy node and the trained model from the non-proxy node to generate an intermediate model. The proxy node generates an upload data packet that includes information about the intermediate model, wherein the segmented routing header of the upload data packet includes a segment list corresponding to the central node, and the segment list corresponding to the central node includes a convergence indicator. The proxy node sends the uploaded data packet to the central node, so that the central node can aggregate the intermediate models sent by one or more proxy nodes according to the aggregation instruction to generate an updated model.

15. A central node, comprising: The generation module is configured to generate a data packet, wherein the data packet includes information about the model updated by the central node, and the segmented routing header of the data packet includes a segment list corresponding to multiple computing nodes, wherein each segment list includes the address of the corresponding computing node and a training instruction, wherein the central node is the central node of federated learning, and the computing nodes are distributed nodes of federated learning; The sending module is configured to send the downlink data packet to the next-hop device.

16. A computing node, comprising: The receiving module is configured to receive a data packet sent from the central node, wherein the data packet sent includes the updated model of the central node, and the segmented routing header of the data packet sent includes a segment list corresponding to multiple computing nodes, wherein each segment list includes the address of the corresponding computing node and a training instruction, wherein the central node is the central node of federated learning, and the computing nodes are distributed nodes of federated learning. The training module is configured to update and train the local model based on the distributed data packet.

17. The computing node according to claim 16, wherein, In the case where the computing node is a proxy node, the computing node further includes: The aggregation module is configured to aggregate the trained models from the proxy nodes and the trained models from the non-proxy nodes to generate an intermediate model. The generation module is configured to generate an upload data packet that includes information about the intermediate model, wherein the segmented routing header of the upload data packet includes a segment list corresponding to the central node, and the segment list corresponding to the central node includes a convergence indicator. The sending module is configured to send the uploaded data packet to the central node, so that the central node can aggregate the intermediate models sent by one or more proxy nodes to generate an updated model according to the aggregation instruction.

18. A communication system for distributed training, comprising: The central node as described in claim 15; as well as The computing nodes as described in claims 16 or 17.

19. The communication system according to claim 18, further comprising: The proxy node selection device is configured to determine a proxy node from a plurality of computing nodes based on attribute information of the plurality of computing nodes.

20. The communication system according to claim 19, wherein, The attribute information includes at least one of uplink bandwidth, computing power, and distance from the central node.

21. The communication system according to claim 20, wherein, The proxy node selection device is further configured to determine a proxy node from the plurality of computing nodes based on a weighted result of at least one of the following: uplink bandwidth, computing power, and distance from the central node.

22. An electronic device, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the communication method for distributed training as described in any one of claims 1 to 14 based on instructions stored in the memory.

23. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the communication method for distributed training according to any one of claims 1 to 14.

Images