Data transmission device, method, data processing node, chip and electronic device

Abstract

The present disclosure provides a data transmission device, method, data processing node, data processing chip, electronic device, and storage medium. Among them, the data transmission device includes: a main controller for generating data transmission indication information corresponding to multiple data processing nodes respectively; the data transmission indication information includes at least one of the following: identification information of data to be transmitted corresponding to at least one data transmission period, and destination node information for transmission; a data processing node for, in response to a data transmission event being triggered, in each of the at least one data transmission period, obtaining target data transmission indication information corresponding to each data transmission period from the data transmission indication information issued by the main controller, and based on the target data transmission indication information, transmitting the data to be transmitted corresponding to each data transmission period.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data transmission, and in particular, to a data transmission device, method, data processing node, data processing chip, electronic device, and storage medium. Background Art

[0002] With the wide application of Artificial Intelligence (AI) technology, neural network models are getting larger and larger, and more and more data needs to be processed and transmitted, resulting in higher and higher requirements for computing power. Limited by processes and power consumption, the computing power of a single System on Chip (SoC) chip, such as an AI dedicated chip, a Graphics Processing Unit (GPU), etc., can no longer meet the requirements; therefore, as a way to solve the problem of insufficient computing power, a distributed architecture in which multiple chips work together has the problem of low data transmission efficiency between multiple chips. Summary of the Invention

[0003] Embodiments of the present disclosure at least provide a data transmission device, method, data processing node, data processing chip, electronic device, and storage medium.

[0004] In a first aspect, an embodiment of the present disclosure provides a data transmission device, including: a main controller, and a plurality of data processing nodes;

[0005] The main controller is configured to generate data transmission indication information corresponding to the plurality of data processing nodes respectively; the data transmission indication information includes at least one of the following: identification information of data to be transmitted corresponding to at least one data transmission cycle, and destination node information of the transmission;

[0006] The data processing node is configured to, in response to a data transmission event being triggered, in each data transmission cycle of the at least one data transmission cycle, obtain target data transmission indication information corresponding to each data transmission cycle from the data transmission indication information sent by the main controller, and based on the target data transmission indication information, transmit the data to be transmitted corresponding to each data transmission cycle.

[0007] In a possible implementation manner, the data transmission indication information corresponding to each data processing node is stored in a data transmission linked list corresponding to the data processing node;

[0008] The main controller is further configured to: generate data transmission linked lists corresponding to the plurality of data processing nodes respectively, and send the corresponding data transmission linked lists to each data processing node among the plurality of data processing nodes;

[0009] When obtaining the target data transmission indication information corresponding to each data transmission cycle from the data transmission indication information sent by the master controller, the data processing node is configured to obtain the target data transmission indication information corresponding to each data transmission cycle from the data transmission linked list sent by the master controller.

[0010] In a possible implementation, the data processing node is further configured to: in response to receiving the data to be transmitted sent by another data processing node, perform a data processing task corresponding to the data to be transmitted sent by the other data processing node based on the data to be transmitted sent by the other data processing node.

[0011] In a possible implementation, when generating the data transmission linked lists respectively corresponding to multiple data processing nodes, the master controller is configured to:

[0012] Determine the data transmission path of the data to be transmitted among multiple data processing nodes, and generate the data transmission linked lists respectively corresponding to multiple data processing nodes based on the data transmission path.

[0013] In a possible implementation, when determining the data transmission path of the data to be transmitted among multiple data processing nodes, the master controller is configured to:

[0014] Determine multiple data processing tasks and the data source information respectively corresponding to the multiple data processing tasks;

[0015] Allocate corresponding data processing tasks to multiple data processing nodes respectively;

[0016] Based on the correspondence between multiple data processing nodes and the data processing tasks, and the data source information respectively corresponding to the multiple data processing tasks, determine the data to be transmitted corresponding to each data processing node and the destination node of the corresponding data to be transmitted; wherein, the data to be transmitted corresponding to each data processing node includes: the data that each data processing node needs to transmit to other data processing nodes.

[0017] For each data processing node, generate the data transmission path corresponding to each data processing node based on the data to be transmitted corresponding to each data processing node, the corresponding destination node, and the physical connection relationship among multiple data processing nodes.

[0018] In a possible implementation, when determining multiple data processing tasks and the data source information respectively corresponding to the multiple data processing tasks, the master controller is configured to:

[0019] Parse the target task to obtain multiple data processing tasks;

[0020] Based on the data source information corresponding to the target task and the dependencies among multiple data processing tasks, obtain the data source information corresponding to each of the multiple data processing tasks.

[0021] In a possible implementation, when parsing the target task to obtain multiple data processing tasks, the master controller is configured to:

[0022] Split the target task based on the type of the distributed architecture composed of multiple data processing nodes to obtain multiple data processing tasks.

[0023] In a possible implementation, when generating a data transmission linked list for the data to be transmitted among multiple data processing nodes based on the data transmission path, the master controller is configured to:

[0024] Determine the transmission method when the data to be transmitted corresponding to each data processing node is transmitted among multiple data processing nodes based on the type of the distributed architecture composed of multiple data processing nodes;

[0025] Generate a data transmission linked list for the data to be transmitted corresponding to each data processing node based on the transmission method and the data transmission path of the data to be transmitted corresponding to each data processing node.

[0026] In a possible implementation, in response to the type of the distributed architecture of the data processing nodes being a ring global reduction architecture and determining that the transmission method is a ring global reduction transmission method, the ring global reduction transmission method includes a scatter reduction transmission stage and a global aggregation transmission stage;

[0027] When generating a data transmission linked list for the data to be transmitted corresponding to each data processing node based on the transmission method and the data transmission path of the data to be transmitted corresponding to each data processing node, the master controller is configured to:

[0028] Determine, for each data processing node among multiple data processing nodes, the neighbor node located upstream in the transmission and the neighbor node located downstream in the transmission; and determine, based on the number of data processing nodes, the number of sub - data into which the corresponding data to be processed of each data processing node is divided and the labels of each sub - data.

[0029] For each data processing node, determine a first label corresponding to the sub-data to be aggregated in the scatter reduction transmission phase and a second label corresponding to the sub-data to be transmitted to other data processing nodes for the data processing node; and in multiple second transmission cycles of the global aggregation transmission phase, determine a third label of the aggregated data to be transmitted to other processing nodes; wherein the aggregated data is generated based on different sub-data with the same first label or based on different sub-data with the same second label;

[0030] For each data processing node, based on the second labels of the sub-data respectively transmitted in multiple first transmission cycles of each data processing node in the scatter reduction transmission phase, determine the storage address of the sub-data corresponding to each first transmission cycle at the current node, the storage address at the neighbor node downstream in the transmission, and the data volume of the sub-data, and generate data transmission indication information corresponding to each first transmission cycle based on the storage address of the sub-data corresponding to each first transmission cycle at the current node, the storage address at the neighbor node downstream in the transmission, and the data volume of the sub-data;

[0031] And, based on the third labels of the aggregated data respectively transmitted in multiple second transmission cycles of each data processing node in the scatter reduction transmission phase, determine the storage address of the aggregated data corresponding to each second transmission cycle at the current node, the storage address at the neighbor node downstream in the transmission, and the data volume of the aggregated data; generate data transmission indication information corresponding to each second transmission cycle based on the storage address of the aggregated data corresponding to each second transmission cycle at the current node, the storage address at the neighbor node downstream in the transmission, and the data volume of the aggregated data;

[0032] Generate the data transmission linked list based on the transmission indication information corresponding to multiple first transmission cycles and the data transmission indication information corresponding to multiple second cycles respectively.

[0033] In a possible implementation manner, in response to the type of the data processing node distributed architecture being a parameter server architecture and determining that the transmission mode is a broadcast transmission mode; the multiple data processing nodes include: a first data processing node serving as a parameter server and a second data processing node serving as a non-parameter server;

[0034] When generating the data transmission linked list of the data to be transmitted corresponding to each data processing node based on the transmission mode and the data transmission path of the data to be transmitted corresponding to each data processing node, the main controller is configured to:

[0035] Determine a neighbor node located downstream of the transmission for the first data processing node, and determine a neighbor node located downstream of the transmission and a neighbor node located upstream of the transmission for the second data processing node; and, based on the number of data processing nodes, determine the number of sub-data into which the data to be processed is divided, and the fourth label of each sub-data;

[0036] Based on the fourth label of the sub-data respectively transmitted by the first data processing node in multiple data transmission cycles, determine the storage address of the sub-data corresponding to each data transmission cycle of the first data processing node at the current node, the storage address of the sub-data at the neighbor node downstream of the transmission, and the data volume of the sub-data; based on the storage address of the sub-data corresponding to each data transmission cycle at the current node, the storage address of the sub-data at the neighbor node downstream of the transmission, and the data volume of the sub-data, generate the data transmission indication information corresponding to each of the data transmission cycles of the first data processing node; based on the data transmission indication information corresponding to each of the data transmission cycles of the first data processing node, generate the data transmission linked list of the first data processing node; and

[0037] For each of the other second data processing nodes except the second data processing node located at the farthest downstream of the transmission, based on the order in which each of the other second data processing nodes receives the sub-data from its neighbor node upstream of the transmission, determine the fifth label of the sub-data corresponding to the multiple data transmission cycles corresponding to each of the other second data processing nodes, and based on the fifth label, determine the storage address of the sub-data transmitted by each data transmission cycle of each of the other second data processing nodes at the current node, the storage address of the sub-data at the neighbor node downstream of the transmission, and the data volume of the sub-data; the storage address of the sub-data transmitted by each data transmission cycle of each of the other second data processing nodes at the current node, the storage address of the sub-data at the neighbor node downstream of the transmission, and the data volume of the sub-data, determine the data transmission indication information of each of the other second data processing nodes corresponding to each data transmission cycle; based on the data transmission indication information of each of the other second data processing nodes corresponding to each data transmission cycle, generate the data transmission linked list of each of the other second data processing nodes.

[0038] In a possible implementation manner, the identification information of the data to be transmitted includes: the starting storage address of the data to be transmitted corresponding to each data transmission cycle in the corresponding transmission starting point, and the data volume information of the data to be transmitted; the destination node information includes: the starting storage address of the data to be transmitted corresponding to each data transmission cycle in the destination node;

[0039] When the data processing node transmits the data to be transmitted based on the data transmission indication information, it is used for:

[0040] For each data transmission cycle, based on the starting storage address of the data to be transmitted corresponding to each data transmission cycle in the corresponding transmission starting point and the data volume information of the data to be transmitted, determine the data to be transmitted corresponding to each data transmission cycle;

[0041] Based on the starting storage address in the corresponding transmission end point of the data to be transmitted, send the data to be transmitted corresponding to each data transmission cycle to the corresponding transmission end point.

[0042] In a possible implementation manner, the data transmission linked list further includes: instruction identifiers respectively corresponding to each data transmission cycle; the instruction identifiers are used to indicate whether to receive an instruction signal for indicating the start of transmission sent by a data processing node upstream of the transmission before starting the corresponding data transmission cycle;

[0043] When transmitting the data to be transmitted based on the data transmission indication information, the data processing node is configured to:

[0044] For each data transmission cycle, in response to the instruction identifier corresponding to the data transmission cycle indicating that an instruction signal for indicating the start of transmission sent by a data processing node upstream of the transmission needs to be received before starting the data transmission cycle, after receiving the corresponding instruction signal sent by the data processing node upstream of the transmission, transmit the data to be transmitted corresponding to the data transmission cycle based on the data transmission indication information corresponding to the data transmission cycle.

[0045] In a possible implementation manner, the main controller is further configured to:

[0046] For each data processing node, based on the original position of the data to be transmitted in each data transmission cycle corresponding to the data processing node, determine the corresponding instruction identifier for each data transmission cycle corresponding to the data processing node;

[0047] Wherein, in response to the original position of the data to be transmitted in any data transmission cycle corresponding to the data processing node being the local node, determine that the instruction identifier corresponding to the any data transmission cycle indicates that an instruction signal for indicating the start of transmission sent by a data processing node upstream of the transmission does not need to be received before starting the any data transmission cycle;

[0048] In response to the original position of the data to be transmitted in any data transmission cycle corresponding to the data processing node including other nodes, determine that the instruction identifier corresponding to the any data transmission cycle indicates that an instruction signal for indicating the start of transmission sent by a data processing node upstream of the transmission needs to be received before starting the any data transmission cycle.

[0049] In a possible implementation manner, the data processing node includes: a processor and a transmission interface;

[0050] The processor is configured to obtain, in each data transmission cycle of at least one data transmission cycle, data transmission indication information corresponding to each data transmission cycle from a data transmission linked list issued by the master controller, and send the data transmission indication information corresponding to each data transmission cycle to the transmission interface;

[0051] The transmission interface is configured to, in response to receiving the data transmission indication information corresponding to each data transmission cycle, send the data to be transmitted corresponding to each data transmission cycle to a destination node indicated by the data transmission indication information corresponding to each data transmission cycle.

[0052] In a second aspect, an embodiment of the present disclosure further provides a data transmission method, including:

[0053] The master controller generates data transmission indication information corresponding to multiple data processing nodes respectively; the data transmission indication information includes at least one of the following: identification information of the data to be transmitted corresponding to at least one data transmission cycle respectively, and destination node information for transmission;

[0054] In response to a data transmission event being triggered, each data processing node obtains, in each data transmission cycle of the at least one data transmission cycle, target data transmission indication information corresponding to each data transmission cycle from the data transmission indication information issued by the master controller, and based on the target data transmission indication information, transmits the data to be transmitted corresponding to each data transmission cycle.

[0055] In a possible implementation, the data transmission indication information corresponding to each data processing node is stored in a data transmission linked list corresponding to the data processing node;

[0056] The method further includes: the master controller generates data transmission linked lists corresponding to multiple data processing nodes respectively, and issues the corresponding data transmission linked lists to each data processing node among the multiple data processing nodes;

[0057] When obtaining the target data transmission indication information corresponding to each data transmission cycle from the data transmission indication information issued by the master controller, the data processing node is configured to obtain the target data transmission indication information corresponding to each data transmission cycle from the data transmission linked list issued by the master controller.

[0058] In a possible implementation, the method further includes: in response to receiving the data to be transmitted transmitted by other data processing nodes, the data processing node executes a data processing task corresponding to the data to be transmitted transmitted by the other data processing nodes based on the data to be transmitted transmitted by the other data processing nodes.

[0059] In one possible implementation, the method further includes: in response to receiving the data to be transmitted sent by other data processing nodes, the data processing node performs a data processing task corresponding to the data to be transmitted sent by the other data processing nodes based on the data to be transmitted sent by the other data processing nodes.

[0060] In one possible implementation, the generating the data transmission linked lists respectively corresponding to multiple data processing nodes includes: determining the data transmission paths of the data to be transmitted among the multiple data processing nodes, and generating the data transmission linked lists respectively corresponding to the multiple data processing nodes based on the data transmission paths.

[0061] In one possible implementation, the determining the data transmission paths of the data to be transmitted among the multiple data processing nodes includes:

[0062] determining multiple data processing tasks and the data source information respectively corresponding to the multiple data processing tasks;

[0063] allocating the corresponding data processing tasks to the multiple data processing nodes respectively;

[0064] based on the corresponding relationship between the multiple data processing nodes and the data processing tasks and the data source information respectively corresponding to the multiple data processing tasks, determining the data to be transmitted corresponding to each data processing node and the destination node of the corresponding data to be transmitted; wherein, the data to be transmitted corresponding to each data processing node includes: the data that each data processing node needs to transmit to other data processing nodes.

[0065] For each data processing node, generating the data transmission path corresponding to each data processing node based on the data to be transmitted corresponding to each data processing node, the corresponding destination node, and the physical connection relationship among the multiple data processing nodes.

[0066] In one possible implementation, the determining multiple data processing tasks and the data source information respectively corresponding to the multiple data processing tasks includes:

[0067] analyzing the target task to obtain the multiple data processing tasks;

[0068] obtaining the data source information respectively corresponding to the multiple data processing tasks based on the data source information corresponding to the target task and the dependency relationship among the multiple data processing tasks.

[0069] In one possible implementation, the analyzing the target task to obtain the multiple data processing tasks includes:

[0070] Split the target task into multiple data processing tasks based on the type of distributed architecture composed of multiple data processing nodes.

[0071] In a possible implementation manner, generating a data transmission linked list for the data to be transmitted to be transmitted between multiple data processing nodes based on the data transmission path includes:

[0072] Based on the type of distributed architecture composed of multiple data processing nodes, determine the transmission method when the data to be transmitted corresponding to each data processing node is transmitted between multiple data processing nodes;

[0073] Generate a data transmission linked list for the data to be transmitted corresponding to each data processing node based on the transmission method and the data transmission path of the data to be transmitted corresponding to each data processing node.

[0074] In a possible implementation manner, in response to the type of the distributed architecture of the data processing nodes being a ring global reduction architecture and determining that the transmission method is a ring global reduction transmission method, the ring global reduction transmission method includes a scattered reduction transmission stage and a global aggregation transmission stage;

[0075] The generating a data transmission linked list for the data to be transmitted corresponding to each data processing node based on the transmission method and the data transmission path of the data to be transmitted corresponding to each data processing node includes:

[0076] Determine a neighbor node located upstream of the transmission and a neighbor node located downstream of the transmission for each data processing node among multiple data processing nodes; and based on the number of data processing nodes, determine the number of sub - data into which the corresponding data to be processed in each data processing node is divided and the label of each sub - data;

[0077] For each data processing node, determine a first label corresponding to the sub - data to be aggregated in the scattered reduction transmission stage and a second label corresponding to the sub - data to be transmitted to other data processing nodes; and in multiple second transmission cycles of the global aggregation transmission stage, determine a third label of the aggregated data to be transmitted to other processing nodes; wherein, the aggregated data is generated based on different sub - data with the same first label or based on different sub - data with the same second label;

[0078] For each data processing node, based on the second labels of the sub-data respectively transmitted in multiple first transmission cycles of each data processing node in the scatter reduction transmission phase, determine the storage address of the sub-data corresponding to each first transmission cycle in the current node, the storage address of the neighbor node downstream in the transmission, and the data volume of the sub-data. Based on the storage address of the sub-data corresponding to each first transmission cycle in the current node, the storage address of the neighbor node downstream in the transmission, and the data volume of the sub-data, generate the data transmission indication information corresponding to each first transmission cycle;

[0079] And, based on the third labels of the aggregated data respectively transmitted in multiple second transmission cycles of each data processing node in the scatter reduction transmission phase, determine the storage address of the aggregated data corresponding to each second transmission cycle in the current node, the storage address of the neighbor node downstream in the transmission, and the data volume of the aggregated data; Based on the storage address of the aggregated data corresponding to each second transmission cycle in the current node, the storage address of the neighbor node downstream in the transmission, and the data volume of the aggregated data, generate the data transmission indication information corresponding to each second transmission cycle;

[0080] Based on the transmission indication information corresponding to multiple first transmission cycles and the data transmission indication information corresponding to multiple second cycles respectively, generate the data transmission linked list.

[0081] In a possible implementation manner, in response to the type of the distributed architecture of the data processing nodes being a parameter server architecture and determining that the transmission mode is a broadcast transmission mode; the multiple data processing nodes include: a first data processing node serving as a parameter server and a second data processing node serving as a non-parameter server;

[0082] The generating the data transmission linked list corresponding to the data to be transmitted of each data processing node based on the transmission mode and the data transmission path of the data to be transmitted corresponding to each data processing node includes:

[0083] Determine the neighbor node downstream in the transmission for the first data processing node, and determine the neighbor node downstream in the transmission and the neighbor node upstream in the transmission for the second data processing node; And, based on the number of data processing nodes, determine the number of sub-data into which the data to be processed is divided and the fourth label of each sub-data;

[0084] Based on the fourth label of the sub-data respectively transmitted by the first data processing node in multiple data transmission cycles, determine the storage address of the sub-data corresponding to each data transmission cycle of the first data processing node in the current node, the storage address of the neighbor node in the downstream of the transmission, and the data volume of the sub-data; based on the storage address of the sub-data corresponding to each data transmission cycle in the current node, the storage address of the neighbor node in the downstream of the transmission, and the data volume of the sub-data, generate the data transmission indication information corresponding to each of the data transmission cycles of the first data processing node; based on the data transmission indication information corresponding to each of the data transmission cycles of the first data processing node, generate the data transmission linked list of the first data processing node; and

[0085] For each of the other second data processing nodes except the second data processing node located at the most downstream of the transmission, based on the order in which each of the other second data processing nodes receives the sub-data from its neighbor node in the upstream of the transmission, determine the fifth label of the sub-data corresponding to the respective multiple data transmission cycles of each of the other second data processing nodes, and based on the fifth label, determine the storage address of the sub-data transmitted by each of the data transmission cycles of each of the other second data processing nodes in the current node, the storage address of the neighbor node in the downstream of the transmission, and the data volume of the sub-data; the storage address of the sub-data transmitted by each of the data transmission cycles of each of the other second data processing nodes in the current node, the storage address of the neighbor node in the downstream of the transmission, and the data volume of the sub-data, determine the data transmission indication information of each of the other second data processing nodes in the corresponding each data transmission cycle; based on the data transmission indication information of each of the other second data processing nodes in the corresponding each data transmission cycle, generate the data transmission linked list of each of the other second data processing nodes.

[0086] In a possible implementation manner, the identification information of the data to be transmitted includes: the starting storage address of the data to be transmitted corresponding to each data transmission cycle in the corresponding transmission starting point, and the data volume information of the data to be transmitted; the destination node information includes: the starting storage address of the data to be transmitted corresponding to each data transmission cycle in the destination node;

[0087] The transmitting the data to be transmitted based on the data transmission indication information includes:

[0088] For each data transmission cycle, based on the starting storage address of the data to be transmitted corresponding to each data transmission cycle in the corresponding transmission starting point, and the data volume information of the data to be transmitted, determine the data to be transmitted corresponding to each data transmission cycle;

[0089] Based on the starting storage address in the corresponding transmission destination for the data to be transmitted, send the data to be transmitted corresponding to each data transmission cycle to the corresponding transmission destination.

[0090] In a possible implementation, the data transmission linked list further includes: instruction identifiers respectively corresponding to each data transmission cycle; the instruction identifiers are used to indicate whether to receive an instruction signal for indicating the start of transmission sent by the data processing node upstream of the transmission before starting the corresponding data transmission cycle;

[0091] The transmitting the data to be transmitted based on the data transmission indication information includes:

[0092] For each data transmission cycle, in response to the instruction identifier corresponding to this data transmission cycle indicating that before starting this data transmission cycle, it is necessary to receive an instruction signal for indicating the start of transmission sent by the data processing node upstream of the transmission, after receiving the corresponding instruction signal sent by the data processing node upstream of the transmission, transmit the data to be transmitted corresponding to this data transmission cycle based on the data transmission indication information corresponding to this data transmission cycle.

[0093] In a possible implementation, it further includes: for each data processing node, the main controller determines the corresponding instruction identifier for each data transmission cycle corresponding to this data processing node based on the original position of the data to be transmitted in each data transmission cycle corresponding to this data processing node;

[0094] Among them, in response to the original position of the data to be transmitted in any data transmission cycle corresponding to this data processing node being the local node, determine that the instruction identifier corresponding to this any data transmission cycle indicates that before starting this any data transmission cycle, it is not necessary to receive an instruction signal for indicating the start of transmission sent by the data processing node upstream of the transmission;

[0095] In response to the original position of the data to be transmitted in any data transmission cycle corresponding to this data processing node including other nodes, determine that the instruction identifier corresponding to this any data transmission cycle indicates that before starting this any data transmission cycle, receive an instruction signal for indicating the start of transmission sent by the data processing node upstream of the transmission.

[0096] In a possible implementation, the data processing node includes: a processor and a transmission interface;

[0097] In each data transmission cycle among at least one data transmission cycle, obtain the data transmission indication information corresponding to each data transmission cycle from the data transmission linked list sent by the main controller, and based on the data transmission indication information, transmit the data to be transmitted corresponding to each data transmission cycle, including:

[0098] In each data transfer cycle of at least one data transfer cycle, the processor obtains data transfer indication information corresponding to each data transfer cycle from the data transfer linked list issued by the master controller, and sends the data transfer indication information corresponding to each data transfer cycle to the transfer interface;

[0099] In response to receiving the data transfer indication information corresponding to each data transfer cycle, the transfer interface sends the data to be transferred corresponding to each data transfer cycle to the destination node indicated by the data transfer indication information corresponding to each data transfer cycle.

[0100] In a third aspect, an embodiment of the present disclosure further provides a data processing node, configured to, in response to a data transfer event being triggered, obtain, in each data transfer cycle of the at least one data transfer cycle, target data transfer indication information corresponding to each data transfer cycle from the data transfer indication information issued by the master controller, and based on the target data transfer indication information, transfer the data to be transferred corresponding to each data transfer cycle.

[0101] In a fourth aspect, an embodiment of the present disclosure provides a data processing chip, including: a data processing device as described in the first aspect above, or any possible implementation manner in the first aspect.

[0102] In a fifth aspect, an alternative implementation manner of the present disclosure further provides an electronic device, including: a data processing device as described in the first aspect above, or any possible implementation manner in the first aspect, or including the data processing chip as described in the third aspect above.

[0103] In the embodiments of the present disclosure, the master controller generates data transfer linked lists corresponding to multiple data processing nodes respectively and issues them to the corresponding data processing nodes; the data processing nodes use the data transfer linked lists to transfer the data to be transferred in each data transfer cycle of at least one data transfer cycle, so that there is no need for the data processing nodes to coordinate and control the data transfer between the upstream and downstream data processing nodes by themselves, but directly implement the data transfer between multiple data processing nodes based on the control of the master controller, thereby reducing the occupation of the computing resources of the data processing nodes in the data processing process and improving the data transfer efficiency.

[0104] To make the above objects, features, and advantages of the present disclosure more obvious and understandable, the following specifically enumerates preferred embodiments and, in conjunction with the accompanying drawings, makes the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] To more clearly illustrate the technical solutions of the embodiments of the present disclosure, the accompanying drawings required for the embodiments will be briefly introduced below. The accompanying drawings herein are incorporated into the specification and form a part of this specification. These drawings show embodiments consistent with the present disclosure and, together with the specification, are used to illustrate the technical solutions of the present disclosure. It should be understood that the following accompanying drawings only show some embodiments of the present disclosure and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related accompanying drawings can be obtained based on these drawings without creative efforts.

[0106] Figure 1 Fig. shows a schematic diagram of a data transmission device provided by an embodiment of the present disclosure;

[0107] Figure 2 Fig. shows a specific example of a data transmission device provided by an embodiment of the present disclosure;

[0108] Figure 3 Fig. shows a specific example of a data transmission device of a Ring All-Reduce architecture provided by an embodiment of the present disclosure;

[0109] Figure 4 Fig. shows a specific example of a data transmission device of a PS architecture provided by an embodiment of the present disclosure;

[0110] Figure 5 Fig. shows a specific example of a data transmission system of a Ring All-Reduce architecture provided by an embodiment of the present disclosure;

[0111] Figure 6 Fig. shows a specific example of the process and result of transmitting data to be transmitted by the Ring All-Reduce transmission method provided by an embodiment of the present disclosure;

[0112] Figure 7 Fig. shows a specific example of a data transmission system of a PS architecture provided by an embodiment of the present disclosure;

[0113] Figure 8 Fig. shows a specific example of the process and result of transmitting data to be transmitted by the broadcast method provided by an embodiment of the present disclosure;

[0114] Figure 9 Fig. shows a flowchart of a data transmission method provided by an embodiment of the present disclosure. Detailed implementation manners

[0115] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Apparently, the described embodiments are only some of the embodiments of the present disclosure, rather than all of them. The components of the embodiments of the present disclosure described and illustrated herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the claimed present disclosure, but merely represents selected embodiments of the present disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of the present disclosure without creative efforts belong to the scope of protection of the present disclosure.

[0116] Through research, it is found that a distributed architecture in which multiple chips work together is an important technology to solve the problem that the computing power of a single chip cannot meet the usage requirements. For example, in the training scenario of a neural network, a multi-GPU distributed framework can be used to train the neural network; during the training process, each GPU executes a part of the training task, generates gradient data, and synchronizes the gradient data to other GPUs. Therefore, each GPU needs to obtain the data of other GPUs, and thus efficient data transmission needs to be performed between multiple GPUs in the distributed architecture. Currently, the traditional solution is to control and coordinate the data transmission between upstream and downstream GPUs by writing threads (kernels) based on GPUs. This results in that while the GPU is executing specific data processing tasks, it also needs to control and coordinate the upstream and downstream GPUs to perform data transmission by writing threads, making the data transmission process not only preempt the computing resources of the GPU, but also have a relatively large data transmission delay and low efficiency.

[0117] Based on the above research, the present disclosure provides a data transmission device. The main controller generates data transmission linked lists corresponding to multiple data processing nodes respectively and distributes them to the corresponding data processing nodes; the data processing nodes use the data transmission linked lists to transmit data to be transmitted in each data transmission cycle of at least one data transmission cycle, so that there is no need for the data processing nodes to coordinate and control the data transmission between upstream and downstream data processing nodes by themselves, but directly realize the data transmission between multiple data processing nodes based on the control of the main controller, thereby reducing the occupation of the computing resources of the data processing nodes in the data processing process and improving the data transmission efficiency.

[0118] All the defects existing in the above solutions are the results obtained by the inventors through practice and careful research. Therefore, the process of discovering the above problems and the solutions proposed by the present disclosure in the following text for the above problems should be the contributions made by the inventors to the present disclosure during the process of the present disclosure.

[0119] It should be noted that like reference numerals and letters refer to like items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0120] To facilitate the understanding of this embodiment, a data transmission device disclosed in this disclosure embodiment will be introduced in detail first. The data transmission device provided in this disclosure embodiment can be deployed, for example, in an electronic device with certain computing capabilities. This data transmission device can be used, for example, to transmit data to be transmitted during data processing tasks such as training a neural network, performing an inference task using a neural network, or for other data processing tasks. In addition, the data transmission device provided in this disclosure embodiment can, in addition to being able to transmit data, also process the data to be transmitted after completing the transmission of the data to be transmitted, and perform a data processing task corresponding to the data to be transmitted.

[0121] Taking the use of the data transmission device for training a neural network as an example, each data processing node in the data transmission device is responsible for a part of the training task, obtains the data to be transmitted during the training process of the neural network, and then mutually transmits the data to be transmitted between the data processing nodes. After each data processing node obtains the data to be transmitted transmitted by other data processing nodes, it performs corresponding processing using the data to be transmitted transmitted by other data processing nodes and the data generated locally, and uses the processing result to perform the next training task.

[0122] In this case, the data to be transmitted includes, for example, at least one of the following: gradient data generated during the training of the neural network; feature data generated by the network layer of the neural network.

[0123] Among them, in machine learning, a loss function with the model loss as the dependent variable and the neural network parameters as the independent variable will be constructed for the training process of the neural network; by minimizing the model loss function, the neural network parameters are determined; the gradient can be defined as a vector composed of all partial derivatives of this loss function, and this vector can also be called gradient data.

[0124] Taking the data to be transmitted as the gradient data generated by each data processing node as an example, each data processing node undertakes a part of the training task. After obtaining the gradient data corresponding to the training task, it needs to transmit the gradient data to other data processing nodes; each data processing node summarizes or fuses the gradient data generated by the local node and the gradient data transmitted by other data processing nodes to obtain the total gradient data corresponding to the neural network currently being trained, and uses the total gradient data to adjust the parameters of the neural network to achieve the training of the neural network.

[0125] Taking the feature data generated by a certain network layer in a neural network as the data to be transmitted, and this network layer being a convolutional layer as an example, this convolutional layer corresponds to N convolutional kernels; there are M data processing nodes. The convolutional tasks corresponding to the N convolutional kernels are respectively assigned to the M data processing nodes to implement. Then each data processing node has to perform convolutional operations on N / M convolutional kernels. After the convolutional operations, feature data corresponding to N / M convolutional kernels is obtained. Each data processing node needs to transmit the feature data it obtains to other data processing nodes. And after each data processing node receives the feature data transmitted by other data processing nodes, it can aggregate or fuse the feature data transmitted by other data processing nodes and the feature data generated by the local node to obtain the total feature data of this network layer. After each GPU obtains the total feature data, it can also use the total feature data to perform the next processing task.

[0126] The above examples only show some examples of transmitting the data to be transmitted. The data transmission device provided by the embodiments of the present disclosure can also be used for the transmission of other types of data in other non-neural network scenarios.

[0127] The data transmission device provided by the embodiments of the present disclosure will be described in detail below.

[0128] See Figure 1 As shown, it is a schematic diagram of the data transmission device provided by the embodiments of the present disclosure. The data transmission device includes: a main controller 10 and multiple data processing nodes 20;

[0129] The main controller 10 is used to generate data transmission indication information corresponding to the multiple data processing nodes respectively. The data transmission indication information includes at least one of the following: identification information of the data to be transmitted corresponding to at least one data transmission cycle, and destination node information of the transmission;

[0130] The data processing node 20 is used to, in response to the triggering of a data transmission event, in each data transmission cycle of the at least one data transmission cycle, obtain the target data transmission indication information corresponding to each data transmission cycle from the data transmission indication information sent by the main controller, and based on the target data transmission indication information, transmit the data to be transmitted corresponding to each data transmission cycle.

[0131] In a specific implementation, the main controller 10 in the data transmission device may include, for example, a Central Processing Unit (CPU) in an electronic device; the data processing node 20 in the data transmission device may include, for example, data processing chips such as an Artificial Intelligence (AI) chip and a GPU. Multiple data processing nodes are interconnected with each other and connected to the main controller 10 to form a distributed data processing system. In another implementation, the data transmission device may be implemented as a data processing chip (e.g., an AI chip), the main controller 10 may be a CPU processing core in the data processing chip, and the data processing node 20 may be a data processing core in the data processing chip (e.g., an AI core).

[0132] See Figure 2 As shown, a specific example of a data transmission device is provided in an embodiment of the present disclosure. Among them, the main controller includes: CPU0 and CPU1; among them, CPU0 and CPU1 communicate with each other through the Ultra Path Interconnect (UPI) protocol. Both CPU0 and CPU1 can serve as the main controller of the data transmission device and execute corresponding actions.

[0133] The data processing node includes: a total of 8 GPUs, namely GPU0, GPU1, GPU2, GPU3, GPU, GPU5, GPU6, and GPU7. The 8 GPUs are interconnected with each other through physical links, and the connection relationship formed by the physical link connections among the 8 GPUs is as Figure 2 shown; among them, GPU0, GPU1, GPU2, and GPU3 are respectively connected to CPU0 through the Peripheral Component Interconnect Express (PCIE) and are controlled by CPU0. GPU0, GPU1, GPU2, and GPU3 can, under the control of CPU0, transfer the data to be transmitted among themselves according to the data transmission linked list generated by CPU0; GPU4, GPU5, GPU6, and GPU7 are respectively connected to CPU1 through the Peripheral Component Interconnect Express (PCIE) and are controlled by CPU1; GPU4, GPU5, GPU6, and GPU7 can, under the control of CPU1, transfer the data to be transmitted among themselves according to the data transmission linked list generated by CPU1. In addition, CPU0 and CPU1 can also control the data transmission between CPU0, GPU0, GPU1, GPU2, GPU3, GPU4, GPU5, GPU6, and GPU7.

[0134] The data transmission indication information corresponding to each of the data processing nodes is stored in the data transmission linked list corresponding to the data processing node; the main controller 10 is further configured to generate data transmission linked lists corresponding to the multiple data processing nodes respectively, and send the corresponding data transmission linked lists to each of the multiple data processing nodes; when the data processing node obtains the target data transmission indication information corresponding to each data transmission cycle from the data transmission indication information sent by the main controller, it is configured to obtain the target data transmission indication information corresponding to each data transmission cycle from the data transmission linked list sent by the main controller.

[0135] When the main controller 10 generates the data transmission linked lists corresponding to the multiple data processing nodes respectively, the following method can be specifically adopted:

[0136] Determine the data transmission path of the data to be transmitted among the multiple data processing nodes, and generate data transmission linked lists corresponding to the multiple data processing nodes respectively based on the data transmission path.

[0137] Among them, the data transmission path of the data to be transmitted among the multiple data processing nodes includes: the starting node of the data to be transmitted and the destination node of the data to be transmitted. Among them, the starting node is any one of the multiple data processing nodes. The destination node of any data to be transmitted can be some of the multiple data processing nodes, and the number of these data processing nodes can be one or more.

[0138] When the main controller 10 determines the data transmission path of the data to be transmitted among the multiple data processing nodes, the following method can be adopted:

[0139] Determine multiple data processing tasks and the data source information corresponding to the multiple data processing tasks respectively;

[0140] Allocate corresponding data processing tasks to the multiple data processing nodes respectively;

[0141] Based on the corresponding relationship between the multiple data processing nodes and the data processing tasks, and the data source information corresponding to the multiple data processing tasks respectively, determine the data to be transmitted corresponding to each data processing node and the destination node of the corresponding data to be transmitted; among them, the data to be transmitted corresponding to each data processing node includes: the data that each data processing node needs to transmit to other data processing nodes;

[0142] For each data processing node, generate the data transmission path corresponding to each data processing node based on the data to be transmitted corresponding to each data processing node, the corresponding target node, and the physical connection relationship among the multiple data processing nodes.

[0143] In a specific implementation, when determining multiple data processing tasks, the master controller 10 may parse the target task to obtain multiple data processing tasks. Then, based on the data source information corresponding to the target task and the dependency relationships among the multiple data processing tasks, the data source information corresponding to each of the multiple data processing tasks is obtained.

[0144] Among them, the target task may include, for example: the tasks required to be executed during the inference process using a neural network, or the tasks required to be executed during the training process of a neural network.

[0145] When parsing the target task to obtain data processing tasks, for example, the target task may be divided according to the specific execution steps for executing the target task and / or according to the data to be processed by the target task, etc.

[0146] Exemplarily, dividing the target task according to steps is, for example, disassembling the specific implementation process of the target task into multiple steps, and for each step, generating at least one data processing task corresponding to that step.

[0147] Dividing the target task according to the data to be processed is, for example, dividing a batch of data to be processed of the target task into multiple groups of sub-data to be processed, and taking the processing task corresponding to each group of sub-data to be processed as a data processing task.

[0148] In addition, the above-mentioned method of dividing according to steps and the method of dividing according to the data to be processed can also be combined. For example, the target task can be first divided according to steps, and then for at least some of these steps, the step can be further divided according to the data to be processed, finally obtaining multiple data processing tasks.

[0149] In addition, when parsing the target task to obtain the multiple data processing tasks, the target task can be split based on the type of the distributed architecture composed of multiple data processing nodes to obtain the multiple data processing tasks.

[0150] The distributed architecture composed of multiple data processing nodes includes, for example: a Ring All-Reduce architecture, or a Parameter Server (PS) architecture.

[0151] Among them, All-Reduce is an operation that reduces the target arrays in all processes (i.e., representing All) to a single array (i.e., representing Reduce) and returns the resulting array to all processes. When performing the training task of a neural network using a data transmission device with a Ring All-Reduce architecture, for example, the gradient data on all GPUs are represented by arrays respectively. It is necessary to perform a reduction operation on the gradient data generated by multiple GPUs, so that the gradient data generated by multiple GPUs are merged into one array, and then the obtained single array is returned to all GPUs. Therefore, this distributed architecture requires that the gradient data generated on each GPU executing the operator be transmitted to other GPUs executing the operator.

[0152] Among them, the reduction operation refers to the operation of merging multiple data, for example, including: performing operations such as accumulation and aggregation on multiple data.

[0153] If the distributed architecture composed of multiple data processing nodes is a ring global reduction architecture, then when splitting the operator, since multiple data processing nodes can execute the same operation in parallel, the operator can be divided according to the data to be processed.

[0154] See Figure 3 As shown, the embodiments of the present disclosure also provide a specific example of a data transmission device with a Ring All-Reduce architecture. In this example, the data processing nodes are GPUs, namely: GPU0 to GPU7; among them, GPU0 to GPU7 can execute multiple data processing tasks in parallel under the control of the main controller.

[0155] In this case, the target task can be parsed into 8 data processing tasks that can be executed in parallel, and each data processing node executes one of the data processing tasks. This division method divides the target task according to the data to be processed to obtain multiple data processing tasks.

[0156] See Figure 4As shown, embodiments of the present disclosure also provide a specific example of a data transmission device with a PS architecture, including: In this example, the data processing node is a GPU; which respectively includes: PS0, and GPU0 to GPU7; among which, GPU0 to GPU7 can, under the control of the main controller, execute multiple data processing tasks in parallel; PS0 can, under the control of the main controller, execute data processing tasks that are serially executed with the tasks executed by GPU0 to GPU7. Therefore, in this case, the target task can be decomposed into data processing task a and data processing task b, where a and b are serially executed; then data processing task a is decomposed into 8 parallel data processing tasks a0 to a7, a0 to a7 are respectively assigned to GPU0 to GPU7, and data processing task b is assigned to PS0. This partitioning method combines the partitioning method according to steps and the partitioning according to the data to be processed to obtain multiple data processing tasks.

[0157] In addition, after obtaining the data processing tasks, based on the data source information corresponding to the target task and the dependency relationships between the multiple data processing tasks, the data source information corresponding to each of the multiple data processing tasks can be obtained.

[0158] Among them, the data source information of each target task is, for example, the source of the input data corresponding to the target task. It can include at least one of: a memory, a data processing node. For example, in the case where the target task includes an inference task of a neural network, for the input data of the task corresponding to the first network layer of the neural network, the source is an external memory. At this time, the input data is, for example, an image to be processed; for the input data of the tasks corresponding to other network layers of the neural network, the sources are an external memory (such as the internal parameters of the network layer in the network layer, such as the convolution kernel of the convolutional layer, the fully connected weights of the fully connected layer, etc.) and the operator corresponding to the previous network layer (the feature data output by the previous operator).

[0159] The dependency relationship information between data processing tasks is used to indicate whether, when executing a certain data processing task, it depends on the output of other data processing tasks.

[0160] Exemplarily, if the target task is decomposed into a to-be-processed task A and a to-be-processed task B, where A is executed before B, and when B is executed, the result of A needs to be used as input, then the dependency relationship between A and B is: B depends on A.

[0161] At this time, the obtained data source information of B, for example, includes: the identifier of A, and the data identifier of the data that A needs to output to B.

[0162] And the execution of A depends on the results of other tasks that are not target tasks. The current storage location of this result and the data identifier of this result can be used as the data source information of A.

[0163] After obtaining multiple data processing tasks, corresponding data processing tasks can be allocated to multiple data processing nodes respectively according to the dependency relationships between the data processing tasks and the distributed architecture composed of multiple data processing nodes.

[0164] When determining the data to be transmitted corresponding to each data processing node based on the correspondence between multiple data processing nodes and data processing tasks, and the data source information corresponding to multiple data processing tasks respectively, first, it can be determined whether the input data required by each data processing task comes from other data processing nodes according to the data source information corresponding to multiple data processing tasks respectively; if it comes from other data processing nodes, that is, the input data can be used as the data to be transmitted to be transmitted by the data processing node where it is located, and the data processing node corresponding to the data processing task that requires the input data is used as the destination node for transmission.

[0165] For example, for a certain data processing task a, if the data processing node that executes the data processing task a is M1, and according to the data source information of the data processing task a, it is determined that the required input data s comes from the data processing node M2, then s is used as the data to be transmitted that the data processing node M2 needs to transmit; M1 is used as the destination node of s.

[0166] As described above Figure 3Taking the corresponding example of data to be transmitted using the Ring All-Reduce architecture as an example, if the data transmission device is applied to the training process of a neural network, GPU0 to GPU7 respectively perform the same processing on 8 groups of data to be processed batch0_0 to batch0_7, and each GPU obtains the gradient data of the data to be processed in the corresponding batch. The training task of the neural network requires that each data processing node obtains the gradient data obtained from 8 groups of data to be processed respectively, and after fusing and summarizing the gradient data, obtains the target gradient data, and then uses the obtained target gradient data to adjust the network parameters of the neural network. Then, this training task is used as the target task, and the data to be transmitted is the gradient data (Gradients) generated by each data processing node respectively; among them, the gradient data generated in each GPU needs to be transmitted to other GPUs, and each GPU needs to fuse and process the gradient data generated locally and the gradient data transmitted by other GPUs. During the fusion process, each GPU can perform the fusion process after receiving the gradient data transmitted by all other GPUs. In this case, the gradient data generated by each GPU can be used as the data to be transmitted corresponding to that GPU; in addition, the local gradient data and the gradient data transmitted by the upstream GPU can also be fused and processed first, and then transmitted to the downstream GPU. Therefore, the gradient data generated by each GPU and the gradient data (or gradient fusion data) transmitted by the upstream GPU can also be fused and processed to obtain the gradient fusion data of the current GPU, and the gradient fusion data of the current GPU is transmitted to the downstream GPU. In this transmission method, the gradient data and the gradient fusion data generated by each GPU can be used as the data to be transmitted. Among them, for each GPU, the destination nodes of the data to be transmitted corresponding to that GPU are the other GPUs.

[0167] As described above Figure 4Taking the corresponding example of using the PS architecture to transmit data to be transmitted as an example, GPU0 to GPU7; GPU0 to GPU7 can, under the control of the main controller, perform the same data processing tasks on 8 batches of data to be processed batch0_0 to batch0_7, and respectively generate gradient data (Gradients). Therefore, the gradient data generated by GPU0 to GPU7 can be used as the data to be transmitted corresponding to GPU0 to GPU7 respectively. After generating the gradient data, GPU0 to GPU7 respectively send the gradient data to the PS. The PS is used to aggregate the gradient data generated by multiple GPUs respectively after receiving the gradient data transmitted by GPU0 to GPU7 respectively, obtain the aggregated result of the gradient data, and then send the aggregated result of the gradient data back to GPU0 to GPU7, so that GPU0 to GPU7 can perform subsequent training tasks based on the aggregated result. Therefore, the aggregated result of the gradient data generated by the PS can also be used as the data to be transmitted corresponding to the PS.

[0168] Through the above process, the main controller 10 obtains the data transmission path of the data to be transmitted among multiple data processing nodes.

[0169] Then, the main controller 10 can generate a data transmission linked list for the data to be transmitted among multiple data processing nodes according to the data transmission path of the data to be transmitted corresponding to each data processing node for each data processing node.

[0170] Here, when generating a data transmission linked list for the data to be transmitted among multiple data processing nodes according to the data transmission path of the data to be transmitted corresponding to each data processing node, for example, based on the type of distributed architecture formed by multiple data processing nodes, the transmission method when the data to be transmitted corresponding to each data processing node is transmitted among multiple data processing nodes can be determined; based on the transmission method and the data transmission path of the data to be transmitted corresponding to each data processing node, a data transmission linked list for the data to be transmitted corresponding to each data processing node is generated.

[0171] In a specific implementation, taking the transmission of data to be transmitted among multiple data processing nodes constituting a Ring All-Reduce architecture as an example:

[0172] I: When the transmission method for data to be transmitted between multiple data processing nodes constituting a Ring All-Reduce architecture is the Ring All-Reduce transmission method, the Ring All-Reduce transmission method includes a scatter reduction transmission stage and a global aggregation transmission stage; there is data to be transmitted in each of the multiple data processing nodes constituting the ring, and the data to be transmitted corresponding to each of the multiple data processing nodes is synchronously transmitted in a ring. In this transmission method, for example, the following method can be used to generate a data transmission linked list:

[0173] When generating the data transmission linked list of the data to be transmitted corresponding to each data processing node based on the transmission method and the data transmission path of the data to be transmitted corresponding to each data processing node, the master controller is used for:

[0174] Determine a neighbor node located upstream of the transmission and a neighbor node located downstream of the transmission for each data processing node among the multiple data processing nodes; and based on the number of data processing nodes, determine the number of sub-data into which the corresponding data to be processed in each data processing node is divided and the label of each sub-data.

[0175] For each data processing node, determine a first label corresponding to the sub-data to be aggregated in the scatter reduction transmission stage and a second label corresponding to the sub-data to be transmitted to other data processing nodes; and in multiple second transmission cycles of the global aggregation transmission stage, determine a third label of the aggregated data to be transmitted to other processing nodes; wherein, the aggregated data is generated based on different sub-data with the same first label or different sub-data with the same second label.

[0176] For each data processing node, based on the second label of the sub-data respectively transmitted in multiple first transmission cycles of each data processing node in the scatter reduction transmission stage, determine the storage address of the sub-data corresponding to each first transmission cycle in the current node, the storage address in the neighbor node downstream of the transmission, and the data volume of the sub-data. Based on the storage address of the sub-data corresponding to each first transmission cycle in the current node, the storage address in the neighbor node downstream of the transmission, and the data volume of the sub-data, generate the data transmission indication information corresponding to each first transmission cycle.

[0177] Moreover, based on the third labels of the aggregated data respectively transmitted by each of the data processing nodes in multiple second transmission cycles during the scatter reduction transmission phase, determine the storage address of the aggregated data corresponding to each second transmission cycle at the current node, the storage address at the neighbor node downstream in the transmission, and the data volume of the aggregated data; based on the storage address of the aggregated data corresponding to each second transmission cycle at the current node, the storage address at the neighbor node downstream in the transmission, and the data volume of the aggregated data, generate data transmission indication information corresponding to each second transmission cycle;

[0178] Generate the data transmission linked list based on the transmission indication information corresponding to multiple first transmission cycles and the data transmission indication information corresponding to multiple second cycles respectively.

[0179] Exemplarily, for example, it includes the following steps (1) to (3):

[0180] (1): Each data processing node (for example, gpu_j) divides all the data to be transmitted into N sub-data blocks, and designates the neighbor node gpu j-1 upstream in its transmission and the neighbor node gpu j+1 downstream in the transmission for each data processing node. Wherein, N is the number of data processing nodes forming a ring.

[0181] Here, the Ring All-Reduce transmission method enables multiple data processing nodes to form a circular transmission. Multiple data processing nodes synchronously transmit the data to be transmitted. Each data processing node divides the data to be transmitted into N sub-data blocks, so that in each transmission cycle, the data volumes transmitted by different data processing nodes are the same, making the circular transmission the most efficient and reducing transmission congestion caused by different data transmission volumes. In one implementation, N can be the same as the number of processing nodes.

[0182] (2): Stage 0: Scatter-Reduce operation: Ensure that there is a global copy of the sub-data with a certain label in each data processing node.

[0183] For example, the data to be transmitted is the gradient data generated during the neural network training process; different data processing nodes process different batches of data to be processed, generate different gradient data, and need to transmit the gradient data generated by the current node to all other data processing nodes.

[0184] Then each data processing node divides the gradient data into N sub-data blocks, and different data processing nodes determine corresponding numbers for the N sub-data blocks. For example Figure 6In the example shown, G0 divides its data to be processed into four sub - data blocks A1 to A4, and assigns labels 1 to 4 to each sub - data block respectively; G0 divides its data to be processed into four sub - data blocks B1 to B4, and assigns labels 1 to 4 to each sub - data block respectively; G2 divides its data to be processed into four sub - data blocks C1 to C4, and assigns labels 1 to 4 to each sub - data block respectively; G3 divides its data to be processed into four sub - data blocks D1 to D4, and assigns labels 1 to 4 to each sub - data block respectively. Among G0 - G3, A1, B1, C1, and D1 are sub - data with corresponding labels; A2, B2, C2, and D2 are sub - data with corresponding labels; A3, B3, C3, and D3 are sub - data with corresponding labels; A4, B4, C4, and D4 are sub - data with corresponding labels.

[0185] A global copy means that the sub - data with corresponding labels are concentrated in one data - processing node.

[0186] As Figure 6 shown in step2, the four sub - data blocks A2, B2, C2, and D2 with label 2 are concentrated in G0, the four sub - data blocks A3, B3, C3, and D3 with label 3 are concentrated in G1, the four sub - data blocks A4, B4, C4, and D4 with label 4 are concentrated in G3, and the four sub - data blocks A1, B1, C1, and D1 with label 1 are concentrated in G0. In this process, the following N - 1 operations are performed:

[0187] In the i - th operation, gpu j sends one of its sub - data blocks to gpu j + 1 and receives one of the sub - data blocks sent by gpu j - 1; and performs a reduction operation on the sub - data received from gpu j - 1 and the local sub - data with the same label as the received sub - data.

[0188] That is: in the i - th operation, gpu j sends its ((j - i) % n)-th sub - data block to gpu j + 1 and receives the ((j - i - 1) % n)-th sub - data block from gpu j - 1; and performs a reduction operation on the ((j - i - 1) % n)-th sub - data block received from gpu j - 1.

[0189] where i, j, n = 0, 1, …, (N - 1).

[0190] where i, j, n = 0, 1, …, (N - 1).

[0191] where j refers to the position of the data - processing node in the ring formed by multiple data - processing nodes. Among them, gpu j represents the current data - processing node; gpu j + 1 represents the neighbor node downstream of its transmission; gpu j - 1 represents the neighbor node upstream of the current data - processing node's transmission. For example Figure 6In the example shown, the description of the position of a certain data processing node among multiple data processing nodes can be expressed as: [G0, G1, G2, G3] = [0, 1, 2, 3]. When j = 3, gpuj + 1 represents gpu0. Similarly, when j = 0, gpuj - 1 represents gpu3.

[0192] The (j - i) % n-th block of sub-data also refers to the position of this block of sub-data among the multiple blocks of sub-data into which the data to be processed is divided. For example Figure 6 In the corresponding example, the description of the position of a certain block of sub-data among multiple blocks of sub-data can be expressed as: [A1, A2, A3, A4] = [0, 1, 2, 3] = [-4, -3, -2, -1]. That is, the sub-data represented by (j - i) % n = 0 is A1, and the sub-data represented by (j - i) % n = -4 is the same block of sub-data A1. Similarly, the sub-data represented by (j - i) % n = 1 and (j - i) % n = -3 is A2; the sub-data represented by (j - i) % n = 2 and (j - i) % n = -2 is both A3; the sub-data represented by (j - i) % n = 3 and (j - i) % n = -1 is both A4.

[0193] Specifically: In the first operation, each data processing node sends the data block corresponding to its own number to the neighbor node downstream of it, and receives the data block corresponding to the number of the neighbor node upstream of it sent by the neighbor node upstream of it. And perform a reduction operation on the data block sent by the neighbor node upstream of it received and the data block whose local number corresponds to the received data block.

[0194] For example, in Figure 6 In the corresponding example, at the first operation (i.e., step0):

[0195] G0 sends the first data block A1 corresponding to its own number among A1 - A4 to G1 (the neighbor node downstream of it), and receives the fourth data block D4 corresponding to the number of G4 among D1 - D4 sent by G4 (the neighbor node upstream of it). Perform a reduction operation on the received D4 and the data block A4 whose local number corresponds to D4, to obtain the data block A4D4.

[0196] G1 sends the second data block B2 corresponding to its own number among B1 - B4 to G2 (the neighbor node downstream of it), and receives the first data block A1 corresponding to the number of G0 among A1 - A4 sent by G0 (the neighbor node upstream of it). Perform a reduction operation on the received A1 and the data block B1 whose local number corresponds to A1, to obtain the data block B1A1.

[0197] G2 sends the third data block C3 corresponding to its own number among C1 - C4 to G3 (the neighbor node downstream in the transmission), and receives the second data block B2 among B1 - B4 sent by G1 (the neighbor node upstream in the transmission). The received B2 and the data block C2 corresponding to B2 in the local number are subjected to a reduction operation to obtain the data block C2B2.

[0198] G3 sends the fourth data block D4 corresponding to its own number among D1 - D4 to G0 (the neighbor node downstream in the transmission), and receives the third data block C3 among C1 - C4 sent by G3 (the neighbor node upstream in the transmission). The received C3 and the data block D3 corresponding to C3 in the local number are subjected to a reduction operation to obtain the data block D3C3.

[0199] In the i-th operation except for the first operation, for each data processing node, if in the (i - 1)-th operation, it receives a certain data block transmitted by the neighbor node upstream in the transmission, after subjecting the data block and the corresponding local data to a reduction operation, in the i-th operation, the data block obtained from the reduction operation is sent to the neighbor node downstream in the transmission.

[0200] As Figure 6 shown in the example, in the second operation (i.e., step1):

[0201] In the first operation, G0 receives the data block D4 sent by its neighbor node G3 upstream in the transmission, and subjects D4 and A4 to a reduction operation to obtain the data block A4D4 (at this time, A4D4 represents a new data block generated by A4 and D4). In the second operation, G0 sends the data block A4D4 to G1. At the same time, G0 receives the data block D3C3 sent by G3, and subjects D3C3 and A3 to a reduction operation to obtain the data block A3D3C3.

[0202] Similarly, G1 sends the data block B1A1 generated in the first operation to G2, receives the data block A4D4 sent by G0, and subjects A4D4 and B4 to a reduction operation to obtain the data block B4A4D4.

[0203] G2 sends the data block C2B2 generated in the first operation to G3, receives the data block B1A1 sent by G1, and subjects B1A1 and C1 to a reduction operation to obtain the data block C1B1A1.

[0204] G3 sends the data block D3C3 generated in the first operation to G0, receives the data block C2B2 sent by G2, and subjects C2B2 and D2 to a reduction operation to obtain the data block D2C2B2.

[0205] In the third operation (i.e., step2):

[0206] At the second operation, G0 received the data block D3C3 sent by its upstream neighbor node G3, and performed a reduction operation on D3C3 and A3, obtaining the data block A3D3C3. Therefore, at the third operation, A3D3C3 was sent to G1, and G0 received the data block D2C2B2 sent by G3. Then, a reduction operation was performed on D2B2C2 and A2, obtaining the data block A2D2B2C2, completing the decentralized reduction process of G0.

[0207] Similarly, G1 sent the data block B4A4D4 generated at the second operation to G2, received A3D3C3 sent by G1, and performed a reduction operation on A3D3C and B3, obtaining the data block B3A3D3C3, completing the decentralized reduction operation of G1.

[0208] G2 sent the data block C1B1A1 generated at the second operation to G3, received the data block B4A4D4 sent by G1, and performed a reduction operation on B4A4D4 and C4, obtaining the data block B4A4D4C4, completing the decentralized reduction operation of G2.

[0209] G3 sent the data block D2C2B2 generated at the second operation to G0, received the data block C1B1A1 sent by G2, and performed a reduction operation on C1B1A1 and D1, obtaining the data block C1B1A1D1, completing the decentralized reduction operation of G3.

[0210] After the decentralized reduction operation, each data processing node obtained the data block generated by the data blocks with the same label among all data processing nodes.

[0211] When the (N - 1)th operation is completed, the ((i + 1) % n)-th block of data of the i-th data processing node has collected the ((i + 1) % n)-th block of data of all data processing nodes.

[0212] (3): Stage 1 - All-Gather operation: enables each data processing node to transfer the global copy of the sub-data with a certain label aggregated by the current data processing node to all other data processing nodes. Among them, in the example shown as Figure 6 , G0 transferred the global copy of the sub-data with label 2 aggregated by it to G1 - G3. In this process, the following (N - 1) operations are performed:

[0213] At the i-th operation, gpu j sends its ((j - i - 1) % n)-th block of data to the downstream neighbor node in the transmission, and receives the ((j - i - 2) % n)-th data sent by the upstream neighbor node in the transmission. However, the received data does not need to be reduced as in (2), but directly replaces its own data.

[0214] After determining the data to be transmitted and the corresponding data transmission path, the upstream neighbor nodes and downstream neighbor nodes corresponding to each data processing node in the above-mentioned transmission method (1) can be determined, and then a data transmission linked list can be generated according to the above method.

[0215] Take Figure 5 the shown Ring All-Reduce architecture as an example. In this example, there are 4 data processing nodes, namely G0 to G3; among them:

[0216] The data to be transmitted corresponding to G0 is A, and the destination nodes of A include: G1 to G3;

[0217] The data to be transmitted corresponding to G1 is B, and the destination nodes of B are G0, G2, and G3;

[0218] The data to be transmitted corresponding to G2 is C, and the destination nodes of C are G0, G1, and G3;

[0219] The data to be transmitted corresponding to G3 is D, and the destination nodes of D are G0 to G2.

[0220] (1): When performing data transmission, each data processing node divides the corresponding data to be transmitted into 4 copies according to the number of data processing nodes in the Ring. That is:

[0221] In G0, A is divided into A1 to A4, labeled 1 to 4 respectively;

[0222] In G1, B is divided into B1 to B4, labeled 1 to 4 respectively;

[0223] In G2, C is divided into C1 to C4, labeled 1 to 4 respectively;

[0224] In G4, D is divided into D1 to D4, labeled 1 to 4 respectively.

[0225] During the process of transmitting the data to be transmitted:

[0226] (2): Stage0: Scatter-Reduce, perform 3 transmission operations (step0 to step2). In each of these 3 transmission operations, execute:

[0227] For the i-th transmission operation, each data processing node G_j sends its m=(j - i)%n-th block of data to the downstream neighbor node G_j+1 and receives the k=(j - i - 1)%n-th block of data from the upstream neighbor node G_j-1. The process and results of the 3 transmissions are as shown in step0 to step2 in Figure 6 .

[0228] where i, j, m, n = 0, 1, 2.

[0229] j refers to the position of the data processing node in the ring formed by multiple data processing nodes. Among them, G_j represents the current data processing node; G_j+1 represents the neighbor node located downstream of its transmission; G_j-1 represents the neighbor node located upstream of the current data processing node's transmission. The description of the positions of 4 data processing nodes among multiple data processing nodes can be expressed as: [G0, G1, G2, G3] = [0, 1, 2, 3] = [-4, -3, -2, -1]. That is, when j = 0 and j = -4, it represents the same data processing node G0; when j = 1 and j = -3, it represents the same data processing node G1; when j = 2 and j = -2, it represents the same output processing node G2; when j = 3 and j = -1, it represents the same data processing node G3.

[0230] The (j - i) % n-th block of sub-data also refers to the position of this block of sub-data among the multiple blocks of sub-data into which the data to be processed is divided. For example Figure 6 In the corresponding example, the description of the position of a certain block of sub-data among multiple blocks of sub-data can be expressed as: [A1, A2, A3, A4] = [0, 1, 2, 3] = [-4, -3, -2, -1]. That is, the sub-data characterized by (j - i) % n = 0 and the same block of sub-data A1 characterized by (j - i) % n = -4 have the same label.

[0231] Through this process, G0 obtains the global copies of all sub-data A2, B2, C2, and D2 with label 2; G1 obtains the global copies of all sub-data A3, B3, C3, and D3 with label 3; G2 obtains the global copies of all sub-data A4, B4, C4, and D4 with label 4; G3 obtains the global copies of sub-data A1, B1, C1, and D1 with label 1.

[0232] (3): Stage1: All-Reduce: Perform 3 transmission operations. In each of these 3 transmission operations, execute:

[0233] For the i-th transfer, G_j sends its m = (j - i + 1) % n-th block of data to the neighbor G_j+1 downstream of the transmission and receives the k = (j - i) % n data from the neighbor node G_j-1 upstream of the transmission.

[0234] During this process, after 3 transmission operations, G0 transfers the global copies of all sub-data A2, B2, C2, and D2 with label 2 it has aggregated to G1 - G3. G1 transfers the global copies of all sub-data A3, B3, C3, and D3 with label 3 it has aggregated to G0, G2, and G3. G2 transfers the global copies of all sub-data A4, B4, C4, and D4 with label 4 it has aggregated to G0, G1, and G3. G3 transfers the global copies of all sub-data A1, B1, C1, and D1 with label 1 it has aggregated to G0 - G2.

[0235] In this way, each data processing node obtains the data to be transmitted transferred by all other data processing nodes. The process and results of the 3 transmissions are as shown in Figure 6 step3 - step5 in

[0236] In the data transmission linked list, it includes data transmission indication information corresponding to multiple at least one data transmission cycle.

[0237] The data transmission indication information corresponding to each data transmission cycle includes: identification information of the data to be transmitted corresponding to the data transmission cycle, and destination node information for transmission.

[0238] Among them, in the embodiments of the present disclosure, the identification information of the data to be transmitted corresponding to each data transmission cycle, for example, includes: the starting storage address of the data to be transmitted corresponding to each data transmission cycle in the corresponding transmission starting point, and the data volume information of the data to be transmitted; the destination node information, for example, includes: the starting storage address of the data to be transmitted corresponding to each data transmission cycle in the destination node.

[0239] In addition, other ways can also be set to represent the identification information of the data to be transmitted, which is not limited in the embodiments of the present disclosure.

[0240] The generated data transmission linked lists of each data processing node are respectively:

[0241] (1): The generated data transmission linked list corresponding to G0 includes:

[0242] a) des0: src = A1_addr, dst = B1_addr, size = K with all - reduce;

[0243] b) des1: src = A4_addr, dst = B4_addr, size = K with all - reduce;

[0244] c) des2: src = A3_addr, dst = B3_addr, size = K with all - reduce;

[0245] d) des3: src = A2_addr, dst = B2_addr, size = K;

[0246] e) des4: src = A1_addr, dst = B1_addr, size = K;

[0247] f) des5: src = A4_addr, dst = B4_addr, size = K.

[0248] Among them, des0: src = A1_addr means that the storage address of data block A1 to be transmitted in the first data transmission cycle in G0 is: A1_addr; dst = B1_addr means that the storage address of data block A1 to be transmitted in the first data transmission cycle in G1 is B1_addr; size = K means that the data volume information of data block A1 of the data to be transmitted is K. The all-reduce indicates that a reduction process is to be performed. d), e), and f) do not carry the identifier of the reduction process, indicating that no reduction process is required, but instead directly use the received data block to replace the original data block.

[0249] The other meanings are similar and will not be elaborated here.

[0250] (2): The data transmission linked list corresponding to G1 includes:

[0251] a) des0: src = B2_addr, dst = C2_addr, size = K with all-reduce;

[0252] b) des1: src = B1_addr, dst = C1_addr, size = K with all-reduce;

[0253] c) des2: src = B4_addr, dst = C4_addr, size = K with all-reduce;

[0254] d) des3: src = B3_addr, dst = C3_addr, size = K;

[0255] e) des4: src = B2_addr, dst = C2_addr, size = K;

[0256] f) des5: src = B1_addr, dst = C1_addr, size = K.

[0257] (3): The data transmission linked list corresponding to G2 includes:

[0258] a) des0: src = C3_addr, dst = D3_addr, size = K with all - reduce;

[0259] b) des1: src = C2_addr, dst = D2_addr, size = K with all - reduce;

[0260] c) des2: src = C1_addr, dst = D1_addr, size = K with all - reduce;

[0261] d) des3: src = C4_addr, dst = D4_addr, size = K;

[0262] e) des4: src = C3_addr, dst = D3_addr, size = K;

[0263] f) des5: src = C2_addr, dst = D2_addr, size = K.

[0264] (4): The data transfer linked list corresponding to G3 includes:

[0265] a) des0: src = D4_addr, dst = A4_addr, size = K with all - reduce;

[0266] b) des1: src = D3_addr, dst = A3_addr, size = K with all - reduce;

[0267] c) des2: src = D2_addr, dst = A2_addr, size = K with all - reduce;

[0268] d) des3: src = D1_addr, dst = A1_addr, size = K;

[0269] e) des4: src = D4_addr, dst = A4_addr, size = K;

[0270] f) des5: src = D3_addr, dst = A3_addr, size = K.

[0271] Taking the transmission of data to be transmitted between multiple data processing nodes constituting the PS architecture as an example:

[0272] II: When the transmission method for data to be transmitted among multiple data processing nodes constituting the PS architecture is the broadcast transmission method, the multiple data processing nodes include: a first data processing node serving as a parameter server and a second data processing node serving as a non-parameter server;

[0273] When generating a data transmission linked list for the data to be transmitted corresponding to each data processing node based on the transmission method and the data transmission path of the data to be transmitted corresponding to each data processing node, the master controller is configured to:

[0274] Determine a neighbor node located downstream of the first data processing node, and determine a neighbor node located downstream and a neighbor node located upstream of the second data processing node; and based on the number of data processing nodes, determine the number of sub-data into which the data to be processed is divided and the fourth label of each sub-data;

[0275] Based on the fourth label of the sub-data respectively transmitted by the first data processing node in multiple data transmission cycles, determine the storage address of the sub-data corresponding to each data transmission cycle of the first data processing node at the current node, the storage address of the neighbor node located downstream of the transmission, and the data volume of the sub-data; generate data transmission indication information corresponding to each data transmission cycle of the first data processing node based on the storage address of the sub-data corresponding to each data transmission cycle at the current node, the storage address of the neighbor node located downstream of the transmission, and the data volume of the sub-data; generate a data transmission linked list of the first data processing node based on the data transmission indication information corresponding to each data transmission cycle of the first data processing node; and

[0276] For each of the second data processing nodes except the second data processing node located at the far downstream of the transmission, based on the order in which each of the other second data processing nodes receives the sub-data from its upstream neighbor nodes in the transmission, determine the fifth label of the sub-data corresponding to the multiple data transmission cycles corresponding to each of the other second data processing nodes, and based on the fifth label, determine the storage address of the sub-data transmitted in each data transmission cycle of each of the other second data processing nodes at the current node, the storage address at the neighbor node in the downstream of the transmission, and the data volume of the sub-data; determine the data transmission indication information of each of the other second data processing nodes in the corresponding each data transmission cycle based on the storage address of the sub-data transmitted in each data transmission cycle of each of the other second data processing nodes at the current node, the storage address at the neighbor node in the downstream of the transmission, and the data volume of the sub-data; generate a data transmission linked list of each of the other second data processing nodes based on the data transmission indication information of each of the other second data processing nodes in the corresponding each data transmission cycle.

[0277] Exemplarily, in Figure 7 In the PS architecture shown, G0 is used as a parameter server. After receiving the gradient data generated by G1 to G3 respectively, G0 performs a reduction operation on the gradient data generated by G1 to G3 to obtain the total gradient data, and transmits the total gradient data (the data to be transmitted) to GPU1 to GPU3 through broadcast transmission. In Figure 7 In the figure, the solid arrows represent the transmission directions of the data to be transmitted; the dashed arrows represent the possible data transmission directions in the PS architecture.

[0278] The data to be transmitted A needs to be transmitted from the data processing node G0 to the other m data processing nodes G1 to Gm. The specific transmission process corresponding to the Broadcast transmission method includes the following two steps (1) to (2):

[0279] (1): Divide the data to be transmitted in G_0 into m pieces of sub-data, and specify the neighbor nodes located upstream of the transmission for G_1 to G_m; and specify the neighbor nodes located downstream of the transmission for G_0 to G_m-1.

[0280] In this way, in some data transmission cycles, the sub-data can be transmitted in parallel between different data processing nodes, saving transmission time and improving transmission efficiency.

[0281] (2): For G_0, perform the following m transmission operations:

[0282] In the i-th transmission operation, transmit the i-th piece of sub-data to the neighbor node G_1 located downstream of the transmission.

[0283] For \(G_j\) in \(G_1\sim G_{m - 1}\), perform the following \(m\) transmission operations:

[0284] In the \(i\)-th transmission operation, receive the \(i\)-th block of sub-data transmitted by the neighbor node \(G_{j - 1}\) upstream of its transmission, and / or transmit the \((i - 1)\)-th block of sub-data to \(G_{j + 1}\) downstream of its transmission.

[0285] Wherein, the \(i\)-th block of sub-data represents the position in the \(m\) blocks of sub-data obtained by dividing the data to be processed, and is the position of the sub-data currently to be transmitted in the \(m\) blocks of sub-data. \(i = 0, 1,\cdots,m\). \(j = 1, 2,\cdots,m - 1\).

[0286] Taking Figure 7 the data transmission system with the PS architecture shown as an example, in this example, as the parameter server \(G_0\) is to transmit the data \(A\) to \(G_1\sim G_3\) (broadcast) respectively, then \(A\) is divided into 4 blocks of sub-data, and the neighbor node downstream of the transmission determined for \(G_0\) is \(G_1\); the neighbor node downstream of the transmission determined for \(G_1\) is \(G_2\); the neighbor node downstream of the transmission determined for \(G_2\) is \(G_3\); the neighbor node upstream of the transmission determined for \(G_1\) is \(G_0\); the neighbor node upstream of the transmission determined for \(G_2\) is \(G_1\); the neighbor node upstream of the transmission determined for \(G_3\) is \(G_2\).

[0287] Then, for \(G_0\), the generated data transmission linked list includes:

[0288] a) des 0: src = \(A1\_0\_addr\), dst = \(A1\_1\_addr\), size = \(K\);

[0289] b) des 1: src = \(A2\_0\_addr\), dst = \(A2\_1\_addr\), size = \(K\);

[0290] c) des 2: src = \(A3\_0\_addr\), dst = \(A3\_1\_addr\), size = \(K\);

[0291] d) des 3: src = \(A4\_0\_addr\), dst = \(A4\_1\_addr\), size = \(K\).

[0292] Wherein, the transmission process and results are as shown in Figure 8 step0 to step3.

[0293] For des j: src = Ai_0_addr, it means that the storage address of data block Ai to be transmitted by G0 in its corresponding data transmission cycle des j is Ai_0_addr in G0, where 0 represents G0, and Ai represents the data block labeled i; dst = Ai_1_addr means that the storage address of data block A1 to be transmitted in data transmission cycle des j is Ai_1_addr in G1, where 0 represents G1; size = K means that the data volume information of data block Ai of the data to be transmitted is K. j = 0, 1, 2, 3; i = 1, 2, 3, 4.

[0294] For G1, the generated data transmission linked list includes:

[0295] a) des 0: src = A1_1_addr, dst = A1_2_addr, size = K;

[0296] b) des 1: src = A2_1_addr, dst = A2_2_addr, size = K;

[0297] c) des 2: src = A3_1_addr, dst = A3_2_addr, size = K;

[0298] d) des 3: src = A4_1_addr, dst = A4_2_addr, size = K.

[0299] Among them, the transmission process and results are as shown in Figure 8 steps 1 to 4.

[0300] For des j: src = Ai_1_addr, it means that the storage address of data block Ai to be transmitted by G1 in its corresponding data transmission cycle des j is Ai_1_addr in G1, where 1 represents G1, and Ai represents the data block labeled i; dst = Ai_2_addr means that the storage address of data block Ai to be transmitted in data transmission cycle des j is Ai_2_addr in G2, where 2 represents G2; size = K means that the data volume information of data block Ai of the data to be transmitted is K. j = 0, 1, 2, 3; i = 1, 2, 3, 4.

[0301] For G2, the generated data transmission linked list includes:

[0302] a) des 0: src = A1_2_addr, dst = A1_3_addr, size = K;

[0303] b) des 1: src = A2_2_addr, dst = A2_3_addr, size = K;

[0304] c) des 2: src = A3_2_addr, dst = A3_3_addr, size = K;

[0305] d) des 3: src = A4_2_addr, dst = A4_3_addr, size = K.

[0306] Among them, the process and result of transmitting the data to be transmitted in the broadcast mode are as shown in Figure 8 steps 2 to 5 in.

[0307] des j: src = Ai_2_addr means that the storage address of data block Ai in G2 where the data to be transmitted in its corresponding data transmission cycle des j is Ai_2_addr, where 2 represents G2 and Ai represents the data block numbered i; dst = Ai_3_addr means that the storage address of data block Ai to be transmitted in data transmission cycle des j in G3 is Ai_3_addr, where 3 represents G3; size = K means that the data volume information of data block Ai of the data to be transmitted is K. j = 0, 1, 2, 3; i = 1, 2, 3, 4.

[0308] In addition, there may be other transmission modes in the Ring All-Reduce architecture, such as: Scatter transmission mode, All-Reduce transmission mode, Gather transmission mode, Reduce transmission mode, All-Gather transmission mode, Reduce-Scatter transmission mode, etc. The data transmission system provided by the embodiments of the present disclosure can be used to generate a data transmission linked list corresponding to each transmission mode, and use the data transmission linked list to implement the transmission of the data to be transmitted according to the corresponding data transmission mode. The generated data transmission linked list can be determined according to the specific transmission mode, which is not limited in the embodiments of the present disclosure.

[0309] When the main controller 10 generates a data transmission linked list, for each data processing node corresponding to the data to be transmitted, a data transmission linked list corresponding to the data processing node is generated. If there is no data to be transmitted that needs to be sent to other data processing nodes in a certain data processing node, the content in the data transmission linked list generated for the data processing node is empty; or no data transmission linked list corresponding to the data processing node is generated.

[0310] For the data transmission linked list corresponding to each data processing node, it includes at least one piece of data transmission indication information corresponding to a transmission cycle. In some cases, for a certain data processing node, the data to be transmitted corresponding to a certain transmission cycle to be transmitted may not be generated by itself or read from an external memory, but is transmitted by other data processing points or depends on the data to be transmitted generated by other data processing nodes for transmission.

[0311] For example, in the above Figure 6 corresponding example, for data processing node G0, it corresponds to 6 data transmission cycles; the data to be transmitted in the first data transmission cycle is generated by itself; and except for the data to be transmitted in the first data transmission cycle, they are all obtained by performing a reduction operation on the data to be transmitted transmitted by the data processing node in the transmission upstream corresponding to it and local data. For example, in the second data transmission cycle, the data to be transmitted by G0 is obtained by performing a reduction operation on the data to be transmitted (D4) transmitted by the data processing node (G3) in the transmission upstream corresponding to it and local data (A4) (A4 + D4, where "+" represents the reduction operation). Therefore, if the transmission of the second data transmission cycle starts before G0 receives the complete D4, it will cause chaos in data transmission.

[0312] Another example is in the above Figure 8 corresponding example, for data processing node G1, it corresponds to 4 data transmission cycles, and the data to be transmitted in each data transmission cycle is transmitted by the data processing node G0 in the transmission upstream corresponding to it. Therefore, if G1 starts the transmission of any of its data transmission cycles before receiving the data to be transmitted transmitted by G0, it will also cause chaos in data transmission.

[0313] In order to avoid the occurrence of the above situation, in the data transmission device provided by the embodiments of the present disclosure, when the main controller 10 generates the data transmission linked list, for each of the data processing nodes, based on the original position of the data to be transmitted in each data transmission cycle corresponding to the data processing node, a corresponding instruction identifier can be determined for each data transmission cycle corresponding to the data processing node;

[0314] Among them, in response to the original position of the data to be transmitted in any one of the data transmission cycles corresponding to the data processing node being the local node, the instruction identifier corresponding to the any one of the data transmission cycles indicates that: before starting the any one of the data transmission cycles, it is not necessary to receive the instruction signal for indicating the start of transmission sent by the data processing node in the transmission upstream.

[0315] In response to the original position of the data to be transmitted in any data transmission cycle corresponding to the data processing node including other nodes, the instruction identifier corresponding to any data transmission cycle indicates that before starting any data transmission cycle, it is necessary to receive an instruction signal for indicating the start of transmission sent by the data processing node upstream of the transmission.

[0316] Exemplarily, for the above Figure 5 corresponding example, for G0 to G3, since the data to be transmitted by each data processing node in other data transmission cycles except the first data transmission cycle is determined based on the data to be transmitted by the data processing node upstream of the transmission, therefore, the instruction identifier corresponding to the first data transmission cycle (i.e., a) of each data processing node indicates that before starting the first data transmission cycle, there is no need to receive an instruction signal sent by the upstream of the transmission; while the instruction identifiers corresponding to other data transmission cycles (b to f) except the first data transmission cycle indicate that before starting the corresponding data transmission cycle, it is necessary to receive an instruction signal sent by the upstream of the transmission.

[0317] Taking G0 as an example, the generated data transmission linked list includes:

[0318] a) des0: src = a1_addr, dst = b1_addr, size = K with all-reduce; M = 0;

[0319] b) des1: src = a4_addr, dst = b4_addr, size = K with all-reduce; M = 1;

[0320] c) des2: src = a3_addr, dst = b3_addr, size = K with all-reduce; M = 1;

[0321] d) des3: src = a2_addr, dst = b2_addr, size = K; M = 1;

[0322] e) des4: src = a1_addr, dst = b1_addr, size = K; M = 1;

[0323] f) des5: src = a4_addr, dst = b4_addr, size = K; M = 1;

[0324] Among them, M is the instruction identifier; M = 0 indicates that it is possible to start the current data transmission cycle without receiving an instruction signal sent by G3 in advance; M = 1 indicates that it is necessary to receive an instruction signal sent by G3 in advance before starting the current data transmission cycle.

[0325] After the controller 10 generates the data transmission linked lists corresponding to the respective data processing nodes, the generated data transmission linked lists are sent to the corresponding data processing nodes.

[0326] When the data processing node 20 transmits the data to be transmitted by using the data transmission linked list, it may use whether the data transmission linked list is received as the start signal for determining whether to start data transmission; alternatively, after receiving the data transmission linked list, the data transmission linked list may be stored in a preset storage space. When a preset data transmission event is triggered, the data transmission indication information corresponding to each data transmission cycle is obtained from the data transmission linked list, and based on the data transmission indication information, the corresponding data to be transmitted is transmitted.

[0327] In a specific implementation, the data transmission event is, for example, an event that the main controller 10 sends to each data processing node to execute a data processing task corresponding to the data to be transmitted.

[0328] In this case, the main controller 10 also generates a data processing task, and after allocating the corresponding data processing tasks to each data processing node, the data processing tasks are sent to each data transmission node.

[0329] When the data processing node receives the data processing task and starts to execute the data processing task, it is determined that the data transmission event is triggered.

[0330] In addition, the data transmission event may also be a signal sent by the main controller 10 received by the data processing node to indicate the start of data transmission of the data to be transmitted. The specific data transmission event may be determined according to the actual needs of the data transmission device during data processing, and is not limited in the embodiments of the present disclosure.

[0331] The identification information of the data to be transmitted includes: the starting storage address of the data to be transmitted corresponding to each data transmission cycle in the corresponding transmission starting point, and the data volume information of the data to be transmitted; the destination node information includes: the starting storage address of the data to be transmitted corresponding to each data transmission cycle in the destination node.

[0332] When the data processing node 20 transmits the data to be transmitted based on the data transmission indication information, for example, the following method may be adopted: for each data transmission cycle, based on the starting storage address of the data to be transmitted corresponding to each data transmission cycle in the corresponding transmission starting point and the data volume information of the data to be transmitted, determine the data to be transmitted corresponding to each data transmission cycle; based on the starting storage address in the corresponding transmission end point of the data to be transmitted, send the data to be transmitted corresponding to each data transmission cycle to the corresponding transmission end point.

[0333] In another embodiment of the present disclosure, if the data transmission linked list further includes: instruction identifiers respectively corresponding to each data transmission period, the data processing node 20 may transmit the data to be transmitted based on the data transmission indication information in the following manner:

[0334] For each data transmission period, in response to the instruction identifier corresponding to this data transmission period indicating that before starting this data transmission period, it is necessary to receive an instruction signal for indicating the start of transmission sent by the data processing node upstream of the transmission, after receiving the corresponding instruction signal sent by the data processing node upstream of the transmission, based on the data transmission indication information corresponding to this data transmission period, transmit the data to be transmitted corresponding to this data transmission period.

[0335] An embodiment of the present disclosure further provides a specific structural example of a data processing node, including: a processor and a transmission interface.

[0336] Among them, the processor is used to obtain, in each data transmission period of at least one data transmission period, the data transmission indication information corresponding to each data transmission period from the data transmission linked list sent by the master controller, and send the data transmission indication information corresponding to each data transmission period to the transmission interface;

[0337] The transmission interface is used to, in response to receiving the data transmission indication information corresponding to each data transmission period, send the data to be transmitted corresponding to each data transmission period to the destination node indicated by the data transmission indication information corresponding to each data transmission period.

[0338] In the embodiment of the present disclosure, the transmission interface includes, for example: a Direct Memory Access (DMA) interface. The MDA interface includes an MDA engine, and the DMA engine can, under the control of the processor, extract the data to be transmitted corresponding to the data transmission indication information from the corresponding storage location according to the data transmission indication information, and transmit the data to be transmitted to the transmission interface in the node corresponding to the data to be transmitted.

[0339] If the instruction identifier respectively corresponding to each data transmission period is carried in the data transmission linked list, and if this instruction identifier indicates that before starting the corresponding data transmission period, it is necessary to receive an instruction signal for indicating the start of transmission sent by the data processing node upstream of the transmission, then after the controller 21 parses out this instruction identifier from the data transmission linked list, it controls the flow control switch of this node to be turned on; in the case where the flow control switch is turned on, before each data transmission period of the data processing node starts, it will determine whether it has received an interrupt signal sent by the neighbor node upstream of the transmission; this interrupt signal is the instruction signal; if it has received an interrupt signal sent by the neighbor node upstream of the transmission, then start a data transmission period.

[0340] The embodiments of the present disclosure adopt a chained DMA transmission method, and automatically perform DMA transmission in sequence according to the data transmission linked list. Since the sizes of the transmitted data blocks are equal, the transmission delays of different data transmission nodes are relatively consistent, and the synchronization efficiency of data transmission is higher.

[0341] Those skilled in the art can understand that in the above method of the specific implementation manner, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined according to its function and possible internal logic.

[0342] Based on the same inventive concept, the embodiments of the present disclosure also provide a data transmission method corresponding to the data transmission device. Since the principle of solving problems by the device in the embodiments of the present disclosure is similar to that of the above data transmission device in the embodiments of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0343] Refer to Figure 9 As shown in

[0344] S901: The main controller generates data transmission indication information corresponding to multiple data processing nodes respectively; the data transmission indication information includes at least one of the following: identification information of the data to be transmitted corresponding to at least one data transmission cycle, and destination node information of the transmission;

[0345] S902: In response to the triggering of the data transmission event, the data processing node obtains, in each data transmission cycle of the at least one data transmission cycle, the target data transmission indication information corresponding to each data transmission cycle from the data transmission indication information sent by the main controller, and based on the target data transmission indication information, transmits the data to be transmitted corresponding to each data transmission cycle.

[0346] In a possible implementation manner, the data transmission indication information corresponding to each data processing node is stored in a data transmission linked list corresponding to the data processing node;

[0347] The method shown also includes: the main controller generates data transmission linked lists corresponding to multiple data processing nodes respectively, and sends the corresponding data transmission linked lists to each data processing node among the multiple data processing nodes;

[0348] When the data processing node obtains the target data transmission indication information corresponding to each data transmission cycle from the data transmission indication information sent by the main controller, it is used to obtain the target data transmission indication information corresponding to each data transmission cycle from the data transmission linked list sent by the main controller.

[0349] In a possible implementation manner, the method further includes: in response to receiving the data to be transmitted transmitted by other data processing nodes, the data processing node executes a data processing task corresponding to the data to be transmitted transmitted by the other data processing nodes based on the data to be transmitted transmitted by the other data processing nodes.

[0350] In a possible implementation manner, the generating data transmission linked lists respectively corresponding to multiple data processing nodes includes: determining a data transmission path of the data to be transmitted among the multiple data processing nodes, and generating data transmission linked lists respectively corresponding to the multiple data processing nodes based on the data transmission path.

[0351] In a possible implementation manner, the determining the data transmission path of the data to be transmitted among the multiple data processing nodes includes:

[0352] determining multiple data processing tasks and data source information respectively corresponding to the multiple data processing tasks;

[0353] allocating corresponding data processing tasks to the multiple data processing nodes respectively;

[0354] Based on the corresponding relationship between the multiple data processing nodes and the data processing tasks, and the data source information respectively corresponding to the multiple data processing tasks, determining the data to be transmitted corresponding to each data processing node and the destination node of the corresponding data to be transmitted; wherein, the data to be transmitted corresponding to each data processing node includes: the data that each data processing node needs to transmit to other data processing nodes.

[0355] For each data processing node, generating a data transmission path corresponding to each data processing node based on the data to be transmitted corresponding to each data processing node, the corresponding destination node, and the physical connection relationship among the multiple data processing nodes.

[0356] In a possible implementation manner, the determining multiple data processing tasks and data source information respectively corresponding to the multiple data processing tasks includes:

[0357] analyzing a target task to obtain the multiple data processing tasks;

[0358] Based on the data source information corresponding to the target task and the dependency relationship among the multiple data processing tasks, obtaining the data source information respectively corresponding to the multiple data processing tasks.

[0359] In a possible implementation manner, the analyzing the target task to obtain the multiple data processing tasks includes:

[0360] Split the target task into multiple data processing tasks based on the type of distributed architecture composed of multiple data processing nodes.

[0361] In a possible implementation manner, generating a data transmission linked list for the data to be transmitted to be transmitted among multiple data processing nodes based on the data transmission path includes:

[0362] Determine the transmission method when the data to be transmitted corresponding to each data processing node is transmitted among multiple data processing nodes based on the type of distributed architecture composed of multiple data processing nodes;

[0363] Generate a data transmission linked list for the data to be transmitted corresponding to each data processing node based on the transmission method and the data transmission path of the data to be transmitted corresponding to each data processing node.

[0364] In a possible implementation manner, in response to the type of the distributed architecture of the data processing nodes being a ring global reduction architecture, and determining that the transmission method is a ring global reduction transmission method, the ring global reduction transmission method includes a scattered reduction transmission stage and a global aggregation transmission stage;

[0365] The generating a data transmission linked list for the data to be transmitted corresponding to each data processing node based on the transmission method and the data transmission path of the data to be transmitted corresponding to each data processing node includes:

[0366] Determine a neighbor node located upstream of the transmission and a neighbor node located downstream of the transmission for each data processing node among multiple data processing nodes; and determine the number of sub-data into which the corresponding data to be processed in each data processing node is divided and the label of each sub-data based on the number of data processing nodes.

[0367] For each data processing node, determine a first label corresponding to the sub-data to be aggregated in the scattered reduction transmission stage and a second label corresponding to the sub-data to be transmitted to other data processing nodes; and in multiple second transmission cycles of the global aggregation transmission stage, determine a third label of the aggregated data to be transmitted to other processing nodes; wherein, the aggregated data is generated based on different sub-data with the same first label or based on different sub-data with the same second label;

[0368] For each data processing node, based on the second labels of the sub-data respectively transmitted in multiple first transmission cycles during the scatter reduction transmission phase, determine the storage address of the sub-data corresponding to each first transmission cycle at the current node, the storage address at the neighbor node downstream in the transmission, and the data volume of the sub-data. Based on the storage address of the sub-data corresponding to each first transmission cycle at the current node, the storage address at the neighbor node downstream in the transmission, and the data volume of the sub-data, generate data transmission indication information corresponding to each first transmission cycle;

[0369] Moreover, based on the third labels of the aggregated data respectively transmitted in multiple second transmission cycles during the scatter reduction transmission phase, determine the storage address of the aggregated data corresponding to each second transmission cycle at the current node, the storage address at the neighbor node downstream in the transmission, and the data volume of the aggregated data; Based on the storage address of the aggregated data corresponding to each second transmission cycle at the current node, the storage address at the neighbor node downstream in the transmission, and the data volume of the aggregated data, generate data transmission indication information corresponding to each second transmission cycle;

[0370] Generate the data transmission linked list based on the transmission indication information corresponding to multiple first transmission cycles and the data transmission indication information corresponding to multiple second cycles respectively.

[0371] In a possible implementation manner, in response to the type of the data processing node distributed architecture being a parameter server architecture and determining that the transmission method is a broadcast transmission method; the multiple data processing nodes include: a first data processing node serving as a parameter server and a second data processing node serving as a non-parameter server;

[0372] The generating the data transmission linked list of the data to be transmitted corresponding to each data processing node based on the transmission method and the data transmission path of the data to be transmitted corresponding to each data processing node includes:

[0373] Determine the neighbor node downstream in the transmission for the first data processing node, and determine the neighbor node downstream in the transmission and the neighbor node upstream in the transmission for the second data processing node; Moreover, based on the number of data processing nodes, determine the number of sub-data into which the data to be processed is divided and the fourth label of each sub-data;

[0374] Based on the fourth label of the sub-data respectively transmitted by the first data processing node in multiple data transmission cycles, determine the storage address of the sub-data corresponding to each data transmission cycle of the first data processing node in the current node, the storage address of the neighbor node in the downstream of the transmission, and the data volume of the sub-data; based on the storage address of the sub-data corresponding to each data transmission cycle in the current node, the storage address of the neighbor node in the downstream of the transmission, and the data volume of the sub-data, generate the data transmission indication information corresponding to the first data processing node in each of the data transmission cycles; based on the data transmission indication information corresponding to the first data processing node in each of the data transmission cycles, generate the data transmission linked list of the first data processing node; and

[0375] For each of the other second data processing nodes except the second data processing node located at the most downstream of the transmission, based on the order in which each of the other second data processing nodes receives the sub-data from its upstream neighbor node in the transmission, determine the fifth label of the sub-data corresponding to each of the multiple data transmission cycles of each of the other second data processing nodes, and based on the fifth label, determine the storage address of the sub-data transmitted by each of the other second data processing nodes in each data transmission cycle in the current node, the storage address of the neighbor node in the downstream of the transmission, and the data volume of the sub-data; based on the storage address of the sub-data transmitted by each of the other second data processing nodes in each data transmission cycle in the current node, the storage address of the neighbor node in the downstream of the transmission, and the data volume of the sub-data, determine the data transmission indication information of each of the other second data processing nodes in the corresponding data transmission cycle; based on the data transmission indication information of each of the other second data processing nodes in the corresponding data transmission cycle, generate the data transmission linked list of each of the other second data processing nodes.

[0376] In a possible implementation manner, the identification information of the data to be transmitted includes: the starting storage address of the data to be transmitted corresponding to each data transmission cycle in the corresponding transmission starting point, and the data volume information of the data to be transmitted; the destination node information includes: the starting storage address of the data to be transmitted corresponding to each data transmission cycle in the destination node;

[0377] The transmitting the data to be transmitted based on the data transmission indication information includes:

[0378] For each data transmission cycle, based on the starting storage address of the data to be transmitted corresponding to each data transmission cycle in the corresponding transmission starting point, and the data volume information of the data to be transmitted, determine the data to be transmitted corresponding to each data transmission cycle;

[0379] Based on the starting storage address in the corresponding transmission destination for the data to be transmitted, send the data to be transmitted corresponding to each data transmission cycle to the corresponding transmission destination.

[0380] In a possible implementation, the data transmission linked list further includes: instruction identifiers respectively corresponding to each data transmission cycle; the instruction identifiers are used to indicate whether to receive an instruction signal for indicating the start of transmission sent by the data processing node upstream of the transmission before starting the corresponding data transmission cycle.

[0381] The transmitting the data to be transmitted based on the data transmission indication information includes:

[0382] For each data transmission cycle, in response to the instruction identifier corresponding to this data transmission cycle indicating that before starting this data transmission cycle, it is necessary to receive an instruction signal for indicating the start of transmission sent by the data processing node upstream of the transmission, after receiving the corresponding instruction signal sent by the data processing node upstream of the transmission, transmit the data to be transmitted corresponding to this data transmission cycle based on the data transmission indication information corresponding to this data transmission cycle.

[0383] In a possible implementation, it further includes: for each data processing node, the main controller determines a corresponding instruction identifier for each data transmission cycle corresponding to this data processing node based on the original position of the data to be transmitted in each data transmission cycle corresponding to this data processing node.

[0384] Among them, in response to the original position of the data to be transmitted in any data transmission cycle corresponding to this data processing node being the local node, it is determined that the instruction identifier corresponding to this any data transmission cycle indicates that before starting this any data transmission cycle, there is no need to receive an instruction signal for indicating the start of transmission sent by the data processing node upstream of the transmission.

[0385] In response to the original position of the data to be transmitted in any data transmission cycle corresponding to this data processing node including other nodes, it is determined that the instruction identifier corresponding to this any data transmission cycle indicates that before starting this any data transmission cycle, it is necessary to receive an instruction signal for indicating the start of transmission sent by the data processing node upstream of the transmission.

[0386] In a possible implementation, the data processing node includes: a processor and a transmission interface.

[0387] In each data transmission cycle among at least one data transmission cycle, obtain the data transmission indication information corresponding to each data transmission cycle from the data transmission linked list sent by the main controller, and transmit the data to be transmitted corresponding to each data transmission cycle based on the data transmission indication information, including:

[0388] In each data transfer cycle of at least one data transfer cycle, the processor obtains data transfer indication information corresponding to each data transfer cycle from the data transfer linked list issued by the master controller, and sends the data transfer indication information corresponding to each data transfer cycle to the transfer interface;

[0389] In response to receiving the data transfer indication information corresponding to each data transfer cycle, the transfer interface sends the data to be transferred corresponding to each data transfer cycle to the destination node indicated by the data transfer indication information corresponding to each data transfer cycle.

[0390] For the description of each processing flow in the method and the logic between different flows, reference can be made to the relevant descriptions in the above system embodiment, which will not be elaborated here.

[0391] The embodiment of the present disclosure further provides a data processing node, which is used to, in response to a data transfer event being triggered, obtain target data transfer indication information corresponding to each data transfer cycle from the data transfer indication information issued by the master controller in each data transfer cycle of the at least one data transfer cycle, and based on the target data transfer indication information, transfer the data to be transferred corresponding to each data transfer cycle.

[0392] The embodiment of the present disclosure further provides a data processing chip, including: a data transfer device as described in any embodiment of the present disclosure, or a data processing node as described in any embodiment of the present disclosure.

[0393] The embodiment of the present disclosure further provides an electronic device, such as a data transfer device as described in any embodiment of the present disclosure, or a data processing node as described in the embodiment of the present disclosure, or a data processing chip as described in the embodiment of the present disclosure.

[0394] The embodiment of the present disclosure further provides a computer program product, which carries program codes, and the instructions included in the program codes can be used to execute the steps of the data transfer method described in the above method embodiment. Specifically, reference can be made to the above method embodiment, which will not be elaborated here.

[0395] Among them, the above computer program product can be specifically implemented in a way of hardware, software or a combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium. In another optional embodiment, the computer program product is specifically embodied as a software product, such as a Software Development Kit (SDK), etc.

[0396] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems and devices described above can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the couplings, direct couplings, or communication connections shown or discussed with each other can be through some communication interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0397] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0398] In addition, in each embodiment of the present disclosure, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0399] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such an understanding, the technical solution of the present disclosure, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present disclosure. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs that can store program codes.

[0400] Finally, it should be noted that the above-described embodiments are only specific implementation manners of the present disclosure, used to illustrate the technical solutions of the present disclosure, rather than limiting it. The protection scope of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that any person skilled in the art within the technical scope disclosed by the present disclosure can still modify the technical solutions recorded in the foregoing embodiments, or can easily think of changes, or perform equivalent replacements on some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A data transmission device, characterized in that, Including: A main controller and multiple data processing nodes; The main controller is used to generate data transmission indication information corresponding to the multiple data processing nodes respectively; The data transmission indication information includes: identification information of the data to be transmitted corresponding to at least one data transmission period respectively, and destination node information of the transmission; The data processing node is used to, in response to the triggering of a data transmission event, in each data transmission period of the at least one data transmission period, obtain target data transmission indication information corresponding to each data transmission period from the data transmission indication information sent by the main controller, and based on the target data transmission indication information, transmit the data to be transmitted corresponding to each data transmission period; The data transmission indication information corresponding to each data processing node is stored in a data transmission linked list corresponding to the data processing node; The main controller is further used to: generate data transmission linked lists corresponding to the multiple data processing nodes respectively, and send the corresponding data transmission linked lists to each data processing node among the multiple data processing nodes; When obtaining the target data transmission indication information corresponding to each data transmission period from the data transmission indication information sent by the main controller, the data processing node is used to obtain the target data transmission indication information corresponding to each data transmission period from the data transmission linked list sent by the main controller.

2. The data transmission device according to claim 1, characterized in that, The data processing node is further used to: in response to receiving the data to be transmitted transmitted by other data processing nodes, based on the data to be transmitted transmitted by the other data processing nodes, execute a data processing task corresponding to the data to be transmitted transmitted by the other data processing nodes.

3. The data transmission device according to claim 1, wherein When generating the data transmission linked lists corresponding to the multiple data processing nodes respectively, the main controller is used to: Determine the data transmission path of the data to be transmitted among the multiple data processing nodes, and based on the data transmission path, generate data transmission linked lists corresponding to the multiple data processing nodes respectively.

4. The data transmission device according to claim 3, characterized in that, When determining the data transmission path of the data to be transmitted among the multiple data processing nodes, the main controller is used to: Determine multiple data processing tasks and data source information corresponding to the multiple data processing tasks respectively; Allocate corresponding data processing tasks to the multiple data processing nodes respectively; Based on the corresponding relationship between the multiple data processing nodes and the data processing tasks, and the data source information corresponding to the multiple data processing tasks respectively, determine the data to be transmitted corresponding to each data processing node and the destination node of the corresponding data to be transmitted; wherein, the data to be transmitted corresponding to each data processing node includes: the data that each data processing node needs to transmit to other data processing nodes; For each data processing node, based on the data to be transmitted corresponding to the each data processing node, the corresponding destination node, and the physical connection relationship among the multiple data processing nodes, generate the data transmission path corresponding to the each data processing node.

5. The data transmission device according to claim 4, characterized in that When determining multiple data processing tasks and data source information corresponding to the multiple data processing tasks respectively, the main controller is used to: Parse the target task to obtain multiple data processing tasks; Based on the data source information corresponding to the target task and the dependency relationships among the multiple data processing tasks, obtain the data source information corresponding to each of the multiple data processing tasks.

6. The data transmission device according to claim 5, wherein When the main controller parses the target task to obtain multiple data processing tasks, it is used for: Based on the type of the distributed architecture composed of multiple data processing nodes, split the target task to obtain multiple data processing tasks.

7. The data transmission device according to any one of claims 3-6, characterized in that, When the main controller generates a data transmission linked list for the data to be transmitted among multiple data processing nodes based on the data transmission path, it is used for: Based on the type of the distributed architecture composed of multiple data processing nodes, determine the transmission method when the data to be transmitted corresponding to each data processing node is transmitted among multiple data processing nodes; Based on the transmission method and the data transmission path of the data to be transmitted corresponding to each data processing node, generate a data transmission linked list for the data to be transmitted corresponding to each data processing node.

8. The data transmission device according to claim 7, characterized in that, In response to the type of the distributed architecture of the data processing nodes being a ring global reduction architecture and determining that the transmission method is a ring global reduction transmission method, the ring global reduction transmission method includes a scatter reduction transmission stage and a global aggregation transmission stage; When the main controller generates a data transmission linked list for the data to be transmitted corresponding to each data processing node based on the transmission method and the data transmission path of the data to be transmitted corresponding to each data processing node, it is used for: Determine the neighbor node located upstream in the transmission and the neighbor node located downstream in the transmission for each data processing node among the multiple data processing nodes; and based on the number of data processing nodes, determine the number of sub-data into which the corresponding data to be processed in each data processing node is divided and the labels of each sub-data; For each data processing node, determine the first label corresponding to the sub-data to be aggregated in the scatter reduction transmission stage and the second label corresponding to the sub-data to be transmitted to other data processing nodes; and in multiple second transmission cycles in the global aggregation transmission stage, determine the third label of the aggregated data to be transmitted to other processing nodes; wherein, the aggregated data is generated based on different sub-data with the same first label or based on different sub-data with the same second label; For each data processing node, based on the second label of the sub-data respectively transmitted by each data processing node in multiple first transmission cycles in the scatter reduction transmission stage, determine the storage address of the sub-data corresponding to each first transmission cycle in the current node, the storage address of the sub-data in the neighbor node located downstream in the transmission, and the data volume of the sub-data, and generate data transmission indication information corresponding to each first transmission cycle based on the storage address of the sub-data corresponding to each first transmission cycle in the current node, the storage address of the sub-data in the neighbor node located downstream in the transmission, and the data volume of the sub-data; Moreover, based on the third labels of the aggregated data respectively transmitted by each of the data processing nodes in multiple second transmission cycles during the scatter reduction transmission phase, determine the storage address of the aggregated data corresponding to each second transmission cycle at the current node, the storage address of the aggregated data at the neighbor node downstream in the transmission, and the data volume of the aggregated data; based on the storage address of the aggregated data corresponding to each second transmission cycle at the current node, the storage address of the aggregated data at the neighbor node downstream in the transmission, and the data volume of the aggregated data, generate the data transmission indication information corresponding to each second transmission cycle; Generate the data transmission linked list based on the transmission indication information corresponding to multiple first transmission cycles and the data transmission indication information corresponding to multiple second cycles respectively.

9. The data transmission device according to claim 7, wherein In response to the type of the distributed architecture of the data processing nodes being the parameter server architecture and determining that the transmission method is the broadcast transmission method; the multiple data processing nodes include: a first data processing node serving as a parameter server and a second data processing node serving as a non-parameter server; When generating the data transmission linked list of the data to be transmitted corresponding to each data processing node based on the transmission method and the data transmission path of the data to be transmitted corresponding to each data processing node, the main controller is used for: Determine the neighbor node downstream in the transmission for the first data processing node, and determine the neighbor node downstream in the transmission and the neighbor node upstream in the transmission for the second data processing node; moreover, based on the number of data processing nodes, determine the number of sub-data into which the corresponding data to be processed is divided in each data processing node and the fourth label of each sub-data; Based on the fourth labels of the sub-data respectively transmitted by the first data processing node in multiple data transmission cycles, determine the storage address of the sub-data corresponding to each data transmission cycle of the first data processing node at the current node, the storage address of the sub-data at the neighbor node downstream in the transmission, and the data volume of the sub-data; based on the storage address of the sub-data corresponding to each data transmission cycle at the current node, the storage address of the sub-data at the neighbor node downstream in the transmission, and the data volume of the sub-data, generate the data transmission indication information corresponding to each data transmission cycle of the first data processing node; based on the data transmission indication information corresponding to each data transmission cycle of the first data processing node, generate the data transmission linked list of the first data processing node; and For each second data processing node except the second data processing node located at the far downstream of the transmission, based on the order in which each of the other second data processing nodes receives the sub-data from its upstream neighbor nodes in the transmission, determine the fifth label of the sub-data corresponding to the respective multiple data transmission cycles of each of the other second data processing nodes, and based on the fifth label, determine the storage address of the sub-data transmitted in each data transmission cycle of each of the other second data processing nodes in the current node, the storage address in the neighbor node downstream in the transmission, and the data volume of the sub-data; based on the storage address of the sub-data transmitted in each data transmission cycle of each of the other second data processing nodes in the current node, the storage address in the neighbor node downstream in the transmission, and the data volume of the sub-data, determine the data transmission indication information of each of the other second data processing nodes in the corresponding each data transmission cycle; based on the data transmission indication information of each of the other second data processing nodes in the corresponding each data transmission cycle, generate the data transmission linked list of each of the other second data processing nodes.

10. The data transmission device according to any one of claims 1-6, characterized in that, The identification information of the data to be transmitted includes: the starting storage address of the data to be transmitted corresponding to each data transmission cycle in the corresponding transmission starting point, and the data volume information of the data to be transmitted; the destination node information includes: the starting storage address of the data to be transmitted corresponding to each data transmission cycle in the destination node. When the data processing node transmits the data to be transmitted based on the data transmission indication information, it is used for: For each data transmission cycle, based on the starting storage address of the data to be transmitted corresponding to each data transmission cycle in the corresponding transmission starting point, and the data volume information of the data to be transmitted, determine the data to be transmitted corresponding to each data transmission cycle. Based on the starting storage address in the corresponding transmission end point of the data to be transmitted, send the data to be transmitted corresponding to each data transmission cycle to the corresponding transmission end point.

11. The data transmission device according to any one of claims 1-6, characterized in that, The data transmission linked list further includes: an instruction identifier corresponding to each data transmission cycle; the instruction identifier is used to indicate whether to receive an instruction signal for indicating the start of transmission sent by the data processing node upstream in the transmission before starting the corresponding data transmission cycle. When the data processing node transmits the data to be transmitted based on the data transmission indication information, it is used for: For each data transmission cycle, in response to the instruction identifier corresponding to the data transmission cycle indicating that an instruction signal for indicating the start of transmission sent by the data processing node upstream in the transmission needs to be received before starting the data transmission cycle, after receiving the corresponding instruction signal sent by the data processing node upstream in the transmission, transmit the data to be transmitted corresponding to the data transmission cycle based on the data transmission indication information corresponding to the data transmission cycle.

12. The data transmission device according to claim 11, wherein The main controller is further used for: For each data processing node, based on the original position of the data to be transmitted in each data transmission cycle corresponding to the data processing node, determine the corresponding instruction identifier for each data transmission cycle corresponding to the data processing node. Wherein, in response to the original position of the data to be transmitted in any data transmission cycle corresponding to the data processing node being the local node, it is determined that the instruction identifier corresponding to any data transmission cycle indicates that before starting any data transmission cycle, it is not necessary to receive the instruction signal for indicating the start of transmission sent by the data processing node upstream of the transmission; In response to the original position of the data to be transmitted in any data transmission cycle corresponding to the data processing node including other nodes, it is determined that the instruction identifier corresponding to any data transmission cycle indicates that before starting any data transmission cycle, it is necessary to receive the instruction signal for indicating the start of transmission sent by the data processing node upstream of the transmission.

13. The data transmission device according to any one of claims 1-6, characterized in that, The data processing node includes: a processor and a transmission interface; The processor is configured to obtain, in each data transmission cycle of at least one data transmission cycle, data transmission indication information corresponding to each data transmission cycle from the data transmission linked list issued by the master controller, and send the data transmission indication information corresponding to each data transmission cycle to the transmission interface; The transmission interface is configured to, in response to receiving the data transmission indication information corresponding to each data transmission cycle, send the data to be transmitted corresponding to each data transmission cycle to the destination node indicated by the data transmission indication information corresponding to each data transmission cycle.

14. A data transmission method, characterized in that, Including: The master controller generates data transmission indication information corresponding to multiple data processing nodes respectively; The data transmission indication information includes: identification information of the data to be transmitted corresponding to at least one data transmission cycle respectively, and destination node information of the transmission; In response to the data transmission event being triggered, the data processing node obtains, in each data transmission cycle of the at least one data transmission cycle, target data transmission indication information corresponding to each data transmission cycle from the data transmission indication information issued by the master controller, and based on the target data transmission indication information, transmits the data to be transmitted corresponding to each data transmission cycle; The data transmission indication information corresponding to each data processing node is stored in the data transmission linked list corresponding to the data processing node; The master controller generates data transmission linked lists corresponding to multiple data processing nodes respectively, and issues the corresponding data transmission linked lists to each data processing node among the multiple data processing nodes; When the data processing node obtains the target data transmission indication information corresponding to each data transmission cycle from the data transmission indication information issued by the master controller, it obtains the target data transmission indication information corresponding to each data transmission cycle from the data transmission linked list issued by the master controller.

15. A data processing node, characterized in that, The data processing node is configured to, in response to a data transmission event being triggered, obtain, in each data transmission cycle of at least one data transmission cycle, target data transmission indication information corresponding to each data transmission cycle from the data transmission indication information sent by the master controller, and transmit the data to be transmitted corresponding to each data transmission cycle based on the target data transmission indication information; the data transmission indication information corresponding to each data processing node is stored in a data transmission linked list corresponding to the data processing node. When obtaining, from the data transmission indication information sent by the master controller, the target data transmission indication information corresponding to each data transmission cycle, the data processing node is configured to obtain, from the data transmission linked list sent by the master controller, the target data transmission indication information corresponding to each data transmission cycle.

16. A data processing chip, characterized in that, Comprising: The data transmission device according to any one of claims 1-13, or the data processing node according to claim 15.

17. An electronic device, characterized in that, Comprising: The data transmission device according to any one of claims 1-13, or the data processing node according to claim 15, or the data processing chip according to claim 16.

18. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium. When the computer program is run by an electronic device, the electronic device executes the steps of the data transmission method according to claim 14.

Images