Data reduction method, device and equipment and computer readable storage medium

Abstract

The invention discloses a data reduction method, device and equipment and a computer readable storage medium, and belongs to the technical field of communication. The method is applied to at least one root node in a multi-dimensional direct connection topology, the nodes in the multi-dimensional direct connection topology are used for model training, and the method comprises the steps that first data are received, the first data are partial data of transmission data, and the transmission data are data transmitted among the nodes in the model training process; second data is obtained, the second data is part of first generated data, and the first generated data is data generated by the root node in the model training process; the first data and the second data are reduced to obtain third data, and the third data are used for nodes in the multi-dimensional direct connection topology to execute subsequent model training; and sending the third data to a first node in the multi-dimensional direct connection topology, wherein the first node is a node except the root node in the multi-dimensional direct connection topology. According to the method, the communication efficiency in the data reduction process can be improved.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to data reduction methods, apparatus, devices and computer-readable storage media. Background Technology

[0002] In some scenarios within the field of communication technology, data from multiple nodes in the same network is reduced. Reduction refers to aggregating distributed data from multiple nodes into a single global computation result. By reducing data from multiple nodes in the network and distributing the reduced result to each node, the network's nodes can execute global data processing tasks in parallel based on the same reduced result, thereby achieving unified coordination and management of the network. Summary of the Invention

[0003] This application provides a data reduction method, apparatus, device, and computer-readable storage medium to improve the efficiency of data reduction. The technical solution is as follows:

[0004] Firstly, a data reduction method is provided, which is applied to at least one root node in a multidimensional directly connected topology, where the nodes in the multidimensional directly connected topology are used for model training. The method includes: receiving first data, which is a portion of transmitted data, where the transmitted data is data transmitted between nodes during model training; obtaining second data, which is a portion of first generated data, where the first generated data is data generated by the root node during model training; reducing the first data and the second data to obtain third data, where the third data is used by the nodes in the multidimensional directly connected topology to perform subsequent model training; and sending the third data to a first node in the multidimensional directly connected topology, where the first node is a node in the multidimensional directly connected topology other than the root node.

[0005] In this application, the first data received by the root node is a portion of the transmitted data, and the second data obtained is a portion of the first generated data. The first and second data are relatively small in size, resulting in high efficiency in reducing them and requiring fewer computational resources, thus achieving high data reduction efficiency. The root node sends third data to nodes in the multidimensional direct-connected topology, enabling these nodes to collaboratively perform subsequent model training based on the same third data, thereby ensuring the accuracy of the model training.

[0006] In one possible implementation, the first data is the data corresponding to the first path in the multidimensional direct connection topology, and the first node is a node on the first path. Before receiving the first data, the method further includes: receiving first path information sent by the first node, which is used by the root node to determine the first path. By receiving the first path information sent by the first node, the root node can determine the first path in the multidimensional direct connection topology, thereby determining that the first data is the data corresponding to the first path, obtaining the second data corresponding to the first path, and enabling accurate reduction of the first data and the second data.

[0007] In one possible implementation, before receiving the first data, the method further includes: obtaining second path information, which indicates the path in the multidimensional direct connection topology determined by the root node; and broadcasting a first message to the first node, which indicates the path information in the multidimensional direct connection topology determined by the root node.

[0008] The root node broadcasts the first message to the first node, enabling the first node to clearly understand the path in the multidimensional direct connection topology determined by the root node, ensuring that the first node and the root node have consistent confirmation of the path in the multidimensional direct connection topology.

[0009] In one possible implementation, before receiving the first data, the method further includes: receiving first path information sent by the first node, the first path information being used by the root node to determine the first path; obtaining second path information, the second path information indicating the path in the multidimensional direct connection topology determined by the root node; and, if the first path information and the second path information do not conflict, broadcasting a first message to the first node, the first message being used to indicate the path in the multidimensional direct connection topology determined by the root node.

[0010] The fact that the first path information and the second path information do not conflict indicates that the first path indicated by the first path information and the path in the multidimensional direct connection topology indicated by the second path information do not conflict. Therefore, the first path determined by the first node and the path in the multidimensional direct connection topology determined by the root node are both valid paths. Then, the root node can send the first message to the first node to ensure that the first node and the root node have consistent confirmation of the path in the multidimensional direct connection topology.

[0011] In one possible implementation, the first message includes at least one of the following: an identifier for a first path, which is any path in the multidimensional direct connection topology; an identifier for the root node; an identifier for the node sending the first message; an identifier for a second node, which is the node following the root node on the first path; and a message type for the first message. Through the information in the first message, the root node in the multidimensional direct connection topology and the path determined by the root node can be accurately indicated, thereby ensuring that the first node and the root node have consistent confirmation of the paths in the multidimensional direct connection topology.

[0012] In one possible implementation, the first data is carried by a second message. The second message also includes an identifier of the path corresponding to the first data and an identifier of the first data itself. The identifier of the path corresponding to the first data is used to determine the first generated data, and the identifier of the first data is used to determine the second data within the first generated data. By using the identifier of the path corresponding to the first data and the identifier of the first data, the second data corresponding to the first data can be accurately determined, thereby reducing the corresponding first and second data to obtain an accurate reduction result.

[0013] In one possible implementation, obtaining the second data includes: splitting the first generated data to obtain the second data. By splitting the first generated data, a smaller second data set can be obtained, thereby improving the efficiency of data reduction.

[0014] In one possible implementation, reducing the first and second data to obtain the third data includes: reducing the first and second data according to a first reduction order, wherein the first reduction order is determined based on the data transmission order on the paths corresponding to the first and second data. Performing data reduction according to a specified reduction order ensures the accuracy and reproducibility of the reduction results.

[0015] Secondly, a data reduction method is provided. The method is applied to a third node, which is a node in a multidimensional direct-connected topology other than the root node. The nodes in the multidimensional direct-connected topology are used for model training. The method includes: receiving fourth data, which is a portion of the transmitted data, which is the data transmitted between nodes during model training; obtaining fifth data, which is a portion of the second generated data, which is the data generated by the third node during model training; reducing the fourth and fifth data to obtain sixth data, which is used by the root node to obtain the third data, and the third data is used by the nodes in the multidimensional direct-connected topology to perform subsequent model training.

[0016] In this application, the fourth data received by the third node is part of the transmitted data, and the fifth data obtained is part of the second generated data. The fourth and fifth data are relatively small in size, and the efficiency of reducing the fourth and fifth data is relatively high, requiring less computing resources, thus making the data reduction more efficient.

[0017] In one possible implementation, the fourth data is the data corresponding to the first path in the multidimensional direct connection topology, and the third node is a node on the first path. Before receiving the fourth data, the method further includes: sending first path information to the root node, which is used by the root node to determine the first path. Sending the first path information to the root node enables the root node to determine the first path in the multidimensional direct connection topology, and makes the root node and the third node agree on the first path.

[0018] In one possible implementation, before receiving the fourth data, the method further includes: receiving a first message sent by the root node, the first message indicating information about the path in the multidimensional direct connection topology determined by the root node; and broadcasting a third message to the nodes in the multidimensional direct connection topology, the third message indicating information about the path in the multidimensional direct connection topology determined by the third node. By receiving the first message, the third node can clearly understand the path in the multidimensional direct connection topology determined by the root node, ensuring that the third node and the root node have consistent confirmation of the path in the multidimensional direct connection topology.

[0019] In one possible implementation, the third message includes at least one of the following: an identifier for a first path, which is any path in the multidimensional direct connection topology; an identifier for the root node on the first path; an identifier for a third node; an identifier for a fourth node, which is the node following the third node on the first path; an identifier for a fifth node, which is the node preceding the third node on the first path; and a message type for the third message. Through the information in the third message, the information of the root node and the path determined by the third node in the multidimensional direct connection topology can be accurately indicated, thereby ensuring that the third node and other nodes in the multidimensional direct connection topology agree on the path within the topology.

[0020] In one possible implementation, the fourth data is carried by a fourth message, which also includes an identifier of the path corresponding to the fourth data and an identifier of the fourth data itself. The identifier of the path corresponding to the fourth data is used to determine the second generated data, and the identifier of the fourth data is used to determine the fifth data in the second generated data. By using the identifier of the path corresponding to the fourth data and the identifier of the fourth data, the fifth data corresponding to the fourth data can be accurately determined, thereby reducing the corresponding fourth and fifth data to obtain an accurate reduction result.

[0021] In one possible implementation, obtaining the fifth data includes: splitting the second generated data to obtain the fifth data. By splitting the second generated data, a smaller fifth data can be obtained, thereby improving the efficiency of data reduction.

[0022] In one possible implementation, reducing the fourth and fifth data to obtain the sixth data includes: reducing the fourth and fifth data according to a second reduction order, where the second reduction order is determined based on the data transmission order on the paths corresponding to the fourth and fifth data. Performing data reduction according to the specified reduction order ensures the accuracy and reproducibility of the reduction results.

[0023] In one possible implementation, reducing the fourth and fifth data to obtain the sixth data involves: reducing the fourth and fifth data to obtain the sixth data when the computing resources of the third node meet a resource condition, where the resource condition indicates that the computing resources of the third node are greater than the resources required to reduce the fourth and fifth data. Reducing the fourth and fifth data when the computing resources of the third node meet the resource condition avoids reduction failures and improves the success rate of reduction.

[0024] In one possible implementation, after receiving the fourth data, the process further includes: if the computing resources of the third node do not meet the resource conditions, sending the fourth and fifth data to the sixth node. The resource conditions indicate that the computing resources of the third node are greater than the resources required to reduce the fourth and fifth data. The sixth node is the node preceding the third node on the path corresponding to the fourth data. By not reducing the fourth and fifth data when the computing resources of the third node do not meet the resource conditions, and directly sending the fourth and fifth data to the sixth node, reduction failure can be avoided, thus improving the reduction success rate.

[0025] Thirdly, a data reduction method is provided, which is applied to the seventh node, which is a node in the multidimensional direct connection topology other than the root node. The nodes in the multidimensional direct connection topology are used for model training. The method includes: obtaining the seventh data, which is a part of the third generated data, which is the data generated by the seventh node during model training; sending the seventh data to the root node, which is used by the root node to obtain the third data, and the third data is used by the nodes in the multidimensional direct connection topology to perform subsequent model training.

[0026] In this application, the seventh data obtained by the seventh node is a portion of the third generated data. The seventh data has a small volume, and the efficiency of reducing the seventh data is high, requiring less computing resources, thus making the data reduction more efficient.

[0027] In one possible implementation, the seventh node is a node on the first path in the multidimensional direct connection topology; before obtaining the seventh data, the method further includes: sending first path information to the root node, which is used by the root node to determine the first path. Sending the first path information to the root node enables the root node to determine the first path in the multidimensional direct connection topology, so that the root node and the seventh node agree on the first path.

[0028] In one possible implementation, before obtaining the seventh data, the method further includes: receiving a first message sent by the root node, the first message indicating information about the path in the multidimensional direct connection topology determined by the root node; and broadcasting a fifth message to the nodes in the multidimensional direct connection topology, the fifth message indicating information about the path in the multidimensional direct connection topology determined by the seventh node. By receiving the first message, the seventh node can clearly understand the path in the multidimensional direct connection topology determined by the root node, ensuring that the seventh node and the root node have consistent confirmation of the path in the multidimensional direct connection topology.

[0029] In one possible implementation, the fifth message includes at least one of the following: an identifier for a first path, which is any path in the multidimensional direct connection topology; an identifier for the root node on the first path; an identifier for the seventh node; an identifier for the eighth node, which is the node preceding the seventh node on the first path; and the message type of the fifth message. Through the information in the fifth message, the information of the root node and the path determined by the seventh node in the multidimensional direct connection topology can be accurately indicated, thereby ensuring that the seventh node and other nodes in the multidimensional direct connection topology agree on the path within the topology.

[0030] In one possible implementation, obtaining the seventh data includes: splitting the third generated data to obtain the seventh data. By splitting the third generated data, a smaller seventh data can be obtained, thereby improving the efficiency of data reduction.

[0031] In any possible implementation of the method provided in the first to third aspects above, the method further includes: obtaining the identifiers of nodes in the multidimensional direct connection topology, wherein the identifier of any node indicates the dimension and position of any node in the multidimensional direct connection topology; and determining the path and at least one root node in the multidimensional direct connection topology based on the identifiers of the nodes in the multidimensional direct connection topology.

[0032] Since the identifiers of nodes in a multidimensional direct-connected topology can indicate the dimension and position of the nodes in the multidimensional direct-connected topology, at least one root node and path in the multidimensional direct-connected topology can be determined based on the identifiers of the nodes in the multidimensional direct-connected topology.

[0033] In any possible implementation of the method provided in the first to third aspects above, at least one root node is determined based on the load information of the nodes in the multidimensional direct-connection topology. Based on the load information of the nodes in the multidimensional direct-connection topology, the load capacity of the nodes in the multidimensional direct-connection topology can be determined, thereby accurately identifying the root node with the strongest load capacity.

[0034] Fourthly, a data reduction device is provided. The device is applied to at least one root node in a multidimensional directly connected topology, where the nodes in the multidimensional directly connected topology are used for model training. The device includes: a first transceiver module for receiving first data, which is a portion of the transmitted data, and the transmitted data is the data transmitted between nodes during model training; a first processing module for obtaining second data, which is a portion of the first generated data, and the first generated data is the data generated by the root node during model training; reducing the first data and the second data to obtain third data, which is used by the nodes in the multidimensional directly connected topology to perform subsequent model training; and the first transceiver module for sending the third data to a first node in the multidimensional directly connected topology, where the first node is a node in the multidimensional directly connected topology other than the root node.

[0035] In one possible implementation, the first data is the data corresponding to the first path in the multidimensional direct connection topology, and the first node is a node on the first path; the first transceiver module is also used to receive the first path information sent by the first node, and the first path information is used by the root node to determine the first path.

[0036] In one possible implementation, the first processing module is further configured to acquire second path information, which indicates the path in the multidimensional direct-connection topology determined by the root node; the first transceiver module is further configured to broadcast a first message to the first node, which indicates the path information in the multidimensional direct-connection topology determined by the root node.

[0037] In one possible implementation, the first message includes at least one of the following: an identifier of a first path, which is any path in a multidimensional direct-connection topology; an identifier of the root node; an identifier of the node that sent the first message; an identifier of a second node, which is the node following the root node on the first path; and a message type of the first message.

[0038] In one possible implementation, the first data is carried by a second message, which also includes an identifier of the path corresponding to the first data and an identifier of the first data. The identifier of the path corresponding to the first data is used to determine the first generated data, and the identifier of the first data is used to determine the second data in the first generated data.

[0039] In one possible implementation, a first processing module is used to segment the first generated data to obtain the second data.

[0040] In one possible implementation, a first processing module is used to reduce the first data and the second data according to a first reduction order to obtain the third data, wherein the first reduction order is determined based on the data transmission order on the paths corresponding to the first data and the second data.

[0041] Fifthly, a data reduction device is provided. The device is applied to a third node, which is a node in a multidimensional directly connected topology other than the root node. The nodes in the multidimensional directly connected topology are used for model training. The device includes: a second transceiver module for receiving fourth data, which is a portion of the transmitted data, and the transmitted data is the data transmitted between nodes during model training; a second processing module for obtaining fifth data, which is a portion of the second generated data, and the second generated data is the data generated by the third node during model training; reducing the fourth and fifth data to obtain sixth data, which is used by the root node to obtain the third data, and the third data is used by the nodes in the multidimensional directly connected topology to perform subsequent model training; and the second transceiver module for sending the sixth data to the root node.

[0042] In one possible implementation, the fourth data is the data corresponding to the first path in the multidimensional direct connection topology, and the third node is a node on the first path; the second transceiver module is also used to send the first path information to the root node, and the first path information is used by the root node to determine the first path.

[0043] In one possible implementation, the second transceiver module is further configured to receive a first message sent by the root node, the first message indicating information about the path in the multidimensional direct connection topology determined by the root node; and broadcast a third message to the nodes in the multidimensional direct connection topology, the third message indicating information about the path in the multidimensional direct connection topology determined by the third node.

[0044] In one possible implementation, the third message includes at least one of the following: an identifier of the first path, which is any path in the multidimensional direct connection topology; an identifier of the root node on the first path; an identifier of the third node; an identifier of the fourth node, which is the node following the third node on the first path; an identifier of the fifth node, which is the node preceding the third node on the first path; and a message type of the third message.

[0045] In one possible implementation, the fourth data is carried by a fourth message, which also includes an identifier of the path corresponding to the fourth data and an identifier of the fourth data. The identifier of the path corresponding to the fourth data is used to determine the second generated data, and the identifier of the fourth data is used to determine the fifth data in the second generated data.

[0046] In one possible implementation, a second processing module is used to segment the second generated data to obtain the fifth data.

[0047] In one possible implementation, the second processing module is used to reduce the fourth and fifth data according to a second reduction order to obtain the sixth data, wherein the second reduction order is determined based on the data transmission order on the paths corresponding to the fourth and fifth data.

[0048] In one possible implementation, the second processing module is used to reduce the fourth and fifth data to obtain the sixth data, provided that the computing resources of the third node meet the resource conditions. The resource conditions are used to indicate that the computing resources of the third node are greater than the resources required to reduce the fourth and fifth data.

[0049] In one possible implementation, the second transceiver module is used to send fourth and fifth data to the sixth node when the computing resources of the third node do not meet the resource conditions. The resource conditions are used to indicate that the computing resources of the third node are greater than the resources required to reduce the fourth and fifth data. The sixth node is the node preceding the third node on the path corresponding to the fourth data.

[0050] In a sixth aspect, a data reduction device is provided. The device is applied to a seventh node, which is a node in a multidimensional direct-connected topology other than the root node. The nodes in the multidimensional direct-connected topology are used for model training. The device includes: a third processing module for obtaining seventh data, which is a portion of the third generated data, which is data generated by the seventh node during model training; and a third transceiver module for sending the seventh data to the root node, which is used by the root node to obtain the third data, and the third data is used by the nodes in the multidimensional direct-connected topology to perform subsequent model training.

[0051] In one possible implementation, the seventh node is a node on the first path in the multidimensional direct connection topology; the third transceiver module is also used to send the first path information to the root node, and the first path information is used by the root node to determine the first path.

[0052] In one possible implementation, the third transceiver module is further configured to receive a first message sent by the root node, the first message indicating information about the path in the multidimensional direct connection topology determined by the root node; and broadcast a fifth message to the nodes in the multidimensional direct connection topology, the fifth message indicating information about the path in the multidimensional direct connection topology determined by the seventh node.

[0053] In one possible implementation, the fifth message includes at least one of the following: an identifier of the first path, which is any path in the multidimensional direct connection topology; an identifier of the root node on the first path; an identifier of the seventh node; an identifier of the eighth node, which is the node preceding the seventh node on the first path; and a message type of the fifth message.

[0054] In one possible implementation, a third processing module is used to segment the third generated data to obtain the seventh data.

[0055] In one possible implementation, the first processing module, the second processing module, or the third processing module is further configured to obtain the identifiers of nodes in the multidimensional direct connection topology, wherein the identifier of any node indicates the dimension and position of any node in the multidimensional direct connection topology; and determine the path and at least one root node in the multidimensional direct connection topology based on the identifiers of the nodes in the multidimensional direct connection topology.

[0056] In one possible implementation, at least one root node is determined based on the load information of the nodes in the multidimensional direct-connected topology.

[0057] In a seventh aspect, a data reduction system is provided, comprising a root node, a third node, and a seventh node. The root node is used to execute the data reduction method in the first aspect and any possible implementation thereof, the third node is used to execute the data reduction method in the second aspect and any possible implementation thereof, and the seventh node is used to execute the data reduction method in the third aspect and any possible implementation thereof.

[0058] Eighthly, a data reduction device is provided, the device including a processor coupled to a memory; the memory stores at least one instruction, the at least one instruction being loaded and executed by the processor to enable the data reduction device to implement the data reduction method of the first aspect and any possible implementation thereof, or to implement the data reduction method of the second aspect and any possible implementation thereof, or to implement the data reduction method of the third aspect and any possible implementation thereof.

[0059] Ninthly, a computer program (product) is provided, comprising: computer program code, which, when executed by a computer, causes the computer to perform the methods described in the foregoing aspects.

[0060] In a tenth aspect, a computer-readable storage medium is provided that stores a program or instructions, wherein when the program or instructions are run on a computer, the methods described in the preceding aspects are executed.

[0061] Eleventhly, a chip is provided, including a processor for retrieving and executing instructions stored in memory, causing a computer equipped with the chip to perform the methods in the foregoing aspects.

[0062] In the twelfth aspect, another chip is provided, comprising: an input interface, an output interface, a processor, and a memory, wherein the input interface, the output interface, the processor, and the memory are connected via an internal connection path, the processor is used to execute code in the memory, and when the code is executed, a computer with the chip installed performs the methods in the above aspects.

[0063] It should be understood that the beneficial effects of the technical solutions and corresponding possible implementations of the fourth to twelfth aspects of this application can be found in the above-described technical effects of the first, second and third aspects and their corresponding possible implementations, and will not be repeated here. Attached Figure Description

[0064] Figure 1 A schematic diagram of a multidimensional direct-connect topology provided for an embodiment of this application;

[0065] Figure 2 A flowchart illustrating a data reduction method executed by the seventh node, provided as an embodiment of this application;

[0066] Figure 3 This application provides a 4*4 two-dimensional torus type topology.

[0067] Figure 4 A schematic diagram of a multidimensional direct-connect topology provided for an embodiment of this application;

[0068] Figure 5 A schematic diagram illustrating the composition of a path tree in a multidimensional direct-connection topology, provided as an embodiment of this application;

[0069] Figure 6 A schematic diagram illustrating the composition of a path tree in another multidimensional direct-connection topology provided in this application embodiment;

[0070] Figure 7 A flowchart illustrating a data reduction method executed by a third node, provided in an embodiment of this application;

[0071] Figure 8 A schematic diagram of a fourth message provided in an embodiment of this application;

[0072] Figure 9 A schematic diagram illustrating another fourth message provided in an embodiment of this application;

[0073] Figure 10 A schematic diagram illustrating a data reduction method provided in an embodiment of this application;

[0074] Figure 11 A schematic diagram illustrating another data reduction method provided in an embodiment of this application;

[0075] Figure 12 A flowchart illustrating a data reduction method executed by a root node, provided as an embodiment of this application;

[0076] Figure 13 This application provides a schematic diagram of a complete data reduction process in accordance with an embodiment of the present application.

[0077] Figure 14 A schematic diagram of a node provided in an embodiment of this application;

[0078] Figure 15 A schematic diagram illustrating a time delay provided in an embodiment of this application;

[0079] Figure 16 An implementation scenario diagram provided for related technologies;

[0080] Figure 17 A flowchart illustrating the operation of data reduction in related technologies;

[0081] Figure 18 This is a schematic diagram of the structure of a data reduction device provided in an embodiment of this application;

[0082] Figure 19 A schematic diagram of another data reduction device provided in an embodiment of this application;

[0083] Figure 20 This is a schematic diagram of another data reduction device provided in an embodiment of this application. Detailed Implementation

[0084] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0085] All-reduce is a collective communication operation designed to aggregate and compute data distributed across different nodes to obtain a global computation result, ensuring that each node can access this result. In large-scale parallel computing, all-reduce effectively achieves the aggregation of distributed data and the synchronization of global computation results, providing fundamental support for coordinating and managing the execution of global computational tasks across multiple nodes.

[0086] For example, full reduction can be used in the training and inference of large models (LMs). Large models refer to deep neural networks (DNNs) with a massive number of parameters (billions or even hundreds of billions) and complex computational structures. Large models are widely used in various fields, including natural language processing (NLP), computer vision (CV), and multimodal computing. The training and inference processes of large models require large datasets and significant computational and communication resources, involving massively parallel computing. Furthermore, large models require multiple nodes to collaborate on training. Therefore, full reduction allows multiple nodes to access the global computation results of the large model, enabling them to collaboratively train the model based on the same global computation results.

[0087] Direct-connect topology is one of the mainstream networking methods for artificial intelligence (AI) computing clusters used for large-scale model training and inference. In a direct-connect topology, each node is equipped with a forwarding device, which acts as both a data generator and ingress point, and a data relay station. Direct-connect topology provides efficient data communication and computing capabilities, supports large-scale data transmission and parallel computing, and has a certain degree of fault tolerance. However, during data reduction in a direct-connect topology, there is competition among nodes for memory resources such as high-bandwidth memory (HBM) and double-data-rate synchronous dynamic random-access memory (DDR SDRAM), as well as bandwidth, leading to lower communication efficiency during data reduction.

[0088] This application provides a data reduction method that improves communication efficiency and bandwidth utilization in directly connected topologies. The implementation scenario of the data reduction method provided in this application includes a multi-dimensional directly connected topology, which comprises multiple nodes, which can be nodes used for model training. These nodes are distributed across multiple dimensions, and this distribution can be uniform or non-uniform. Each node is connected to or equipped with a device that has forwarding capabilities, and each node receives and sends data through this device. In this application embodiment, each node can also perform data reduction through a device with forwarding capabilities.

[0089] A node, also known as a computing node, can include one or more combinations of processors, storage devices, network interfaces, input / output processing units (I / O units), and accelerators. The processor is the core of the node, responsible for executing computing tasks and controlling the node's operation. Examples of processors include central processing units (CPUs), graphics processing units (GPUs), and tensor processing units (TPUs). Storage devices store data and program instructions, providing data read / write capabilities. Examples of storage devices include hard drives, memory cards, optical discs, virtual memory, HBM, and DDR. Network interfaces facilitate communication and data transmission between nodes. Examples of network interfaces include fiber optic interfaces, serial interfaces, and multiplexed interfaces. Input / output processing units (I / O units) handle data input / output operations and manage data interaction between nodes. Examples of I / O units include input / output dies (I / O dies) and data processing units (DPUs). Accelerators are used to speed up specific types of computing tasks and improve computational efficiency. Accelerators can be, for example, field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). Multiple nodes can provide support for parallel computing, storage and data management, and network communication for the operational process in the implementation scenario.

[0090] The implementation scenarios can be various large-scale parallel computing scenarios requiring efficient parallel data reduction, such as: devices or device clusters deploying large models, distributed computing systems, edge computing systems, or virtual computing systems in cloud computing platforms. The types of multiple nodes differ in different implementation scenarios. For example, if the implementation scenario involves devices deploying large models, such as computers deploying large models, then multiple nodes can be one or more combinations of chips, processors, registers, or network interface cards (NICs) within the deployed large model device; if the implementation scenario involves a device cluster deploying large models, then multiple nodes can be multiple devices within the device cluster; if the implementation scenario is a distributed computing system, then multiple nodes can be multiple computer nodes within the distributed computing system; if the implementation scenario is an edge computing system, then multiple nodes can be multiple edge computing nodes within the edge computing system; if the implementation scenario is a virtual computing system within a cloud computing platform, then multiple nodes can be multiple virtual computing nodes within the virtual computing system.

[0091] Optionally, in the embodiments of this application, the types of multiple nodes can be the same or different. For example, when the implementation scenario is a device cluster with a large model deployed, node 1 among the multiple nodes can be a chip on device 1 in the device cluster, and node 2 can be a processor on device 2 in the device cluster.

[0092] Regardless of the implementation scenario, the data reduction method provided in this application can improve communication efficiency during data reduction, as well as the response efficiency and resource utilization of each node.

[0093] See Figure 1 The diagram shows a schematic of a multidimensional direct-connect topology provided in an embodiment of this application. Figure 1 The multidimensional direct-connect topology shown is a two-dimensional direct-connect topology, including 16 nodes from node 0 to node 15 represented by circles. The 16 nodes form a plane with a specification of 4*4.

[0094] It should be noted that, Figure 1 This application is merely an example of an implementation scenario, and does not limit the dimensions of the multidimensional direct-connect topology or the number of nodes in each dimension. Furthermore, it does not limit the type of multidimensional direct-connect topology or the connection relationships between nodes. For example, in a mesh-type multidimensional direct-connect topology, each node is connected only to its adjacent nodes. In a torus-type multidimensional direct-connect topology, each node is connected to its adjacent nodes, and an edge node in any dimension is connected to another edge node in the same dimension. In a HyperX (a direct-connect topology) type multidimensional direct-connect topology, each node is connected to all other nodes in the same dimension of the multidimensional direct-connect topology. In a ring-type multidimensional direct-connect topology, each node is connected to its adjacent nodes, and multiple nodes are connected in a specified direction to form a ring.

[0095] In the embodiments of this application, the multiple nodes in the multidimensional direct connection topology include a root node and leaf nodes. The root node is the destination node for the convergence of data of each node in the multidimensional direct connection topology during a full reduction process. The leaf nodes are nodes other than the root node. The data on the leaf nodes will be transmitted to the root node, and the root node will perform a full reduction on the data of the nodes in the multidimensional direct connection topology.

[0096] Optionally, the root node in the multidimensional direct-connect topology can be one or more. The root node in the multidimensional direct-connect topology can be different in different rounds of full reduction. For example, in a device deploying a large model, different rounds of model training or inference performed by multiple nodes correspond to different computational tasks. Different computational tasks have different reduction requirements, therefore the root node in the multidimensional direct-connect topology can be different in different rounds of full reduction. The root node in the multidimensional direct-connect topology can be manually specified and statically configured, or it can be determined by election from multiple nodes in the multidimensional direct-connect topology.

[0097] For example, at least one root node in a multidimensional direct-connected topology can be determined based on the load information of the nodes in the multidimensional direct-connected topology. During the election of the root node by multiple nodes in the multidimensional direct-connected topology, each node in the multidimensional direct-connected topology can send its own load information to all other nodes in the multidimensional direct-connected topology through flooding, so that all nodes in the multidimensional direct-connected topology can obtain the load information of all other nodes in the multidimensional direct-connected topology. Each node determines the root node in the multidimensional direct-connected topology based on the received load information and its own load information.

[0098] The load information of any node may include the size of the storage space within the storage unit of any node or the bandwidth of any node, and the load information of any node can indicate the load capacity of any node. Optionally, when determining the root node in the multidimensional direct connection topology, each node can determine the node with the strongest load capacity indicated by the load information as the root node.

[0099] In one possible implementation, before each node determines the root node in the multidimensional direct-connection topology, each node can obtain the identifiers of the nodes in the multidimensional direct-connection topology. The identifier of any node can be sent from the control device to any node, or it can be statically configured manually for any node. Each node can flood its own identifier to the nodes in the multidimensional direct-connection topology, ensuring that all nodes can obtain the identifiers of the nodes in the multidimensional direct-connection topology.

[0100] This application does not limit the type of identifier for any node. The identifier for any node can be one or more combinations of numbers, letters, or symbols. Different nodes in a multidimensional direct-connection topology have different identifiers. The identifier of any node indicates its dimension and position within the multidimensional direct-connection topology. The dimension of any node in the multidimensional direct-connection topology refers to its level within the topology, and its position refers to its relative position to a specified node within its respective level. The specified node can be manually designated.

[0101] After obtaining the identifiers and load information of each node in a multidimensional direct-connect topology, the root node in the topology can be determined more accurately, and its identifier can be identified. Once the root node is determined, the leaf nodes in the multidimensional direct-connect topology transmit data to it, initiating a data reduction operation within the topology.

[0102] In the data reduction process, nodes in a multidimensional direct-connection topology are not only divided into root nodes and leaf nodes, but leaf nodes can also be divided into head nodes and intermediate nodes. A head node is a leaf node that begins transmitting data to the root node without receiving data from other leaf nodes. An intermediate node is a node that, upon receiving data from other leaf nodes, transmits its own data and the received data from other leaf nodes back to the root node.

[0103] Below, using the example of the head node being the seventh node and the nodes in the multidimensional direct-connection topology being used for model training, we will provide an exemplary illustration of the data reduction method performed by the head node during the data reduction process. See [link to documentation]. Figure 2 The diagram shows a flowchart of a data reduction method executed by a seventh node according to an embodiment of this application. The data reduction method executed by the seventh node includes, but is not limited to, the following S201 and S202.

[0104] S201, the seventh node obtains the seventh data, which is a part of the third generated data. The third generated data is the data generated by the seventh node during model training.

[0105] For example, the seventh node obtains the seventh data by: segmenting the third generated data to obtain the seventh data. This application embodiment does not limit the method by which the seventh node segments the third generated data. The seventh node can segment the third generated data uniformly or according to the data characteristics of the third generated data. For example, if the third generated data includes multiple types of data, the seventh node can segment the third generated data according to the multiple types in the third generated data.

[0106] The multiple data slices obtained by splitting the third generated data are all partial data of the third generated data, and the seventh data can be any one of the multiple data slices obtained by splitting the third generated data. Optionally, the seventh data can be the data slice corresponding to the first path to which the seventh node belongs. Therefore, before the seventh node obtains the seventh data, the seventh node can determine the first path to which the seventh node belongs.

[0107] In this embodiment of the application, data reduction can be data reduction based on paths. Therefore, before the nodes in the multidimensional direct connection topology start to execute the operation corresponding to the data reduction method, the nodes in the multidimensional direct connection topology can determine the paths in the multidimensional direct connection topology so as to perform data reduction based on the paths in the multidimensional direct connection topology.

[0108] As explained above, any node in a multidimensional direct-connected topology can obtain the identifiers of all nodes in the topology and determine the root node based on these identifiers. The root node is used to perform a full reduction on the data generated by all nodes in the topology, and all data generated by leaf nodes must be transmitted to the root node. Therefore, after determining the root node, each leaf node can determine one or more paths from itself to the root node based on the identifiers of the nodes in the topology.

[0109] Taking the seventh node as an example, after determining the root node, the seventh node, based on the identifiers of the nodes in the obtained multi-dimensional directly connected topology, determines at least one path from the seventh node to the root node according to routing rules or shortest path tree rules, and determines the path information of at least one path. The path information of any path includes the identifiers of each node on that path, as well as the connection relationships between the nodes on that path. The relationship between any two nodes includes at least the direction of the ports that connect the two nodes; for example, node 1 is connected to node 2 via a west-facing port. Since the identifier of any node can indicate the location of any node, the relative positions of any two nodes can be determined by the locations of the nodes indicated by their identifiers, thus determining the connection relationships between any two nodes.

[0110] If the seventh node determines multiple paths, the number of nodes on these paths can be the same or different. For example, see... Figure 3 The diagram shows a schematic of a multidimensional direct-connect topology provided in an embodiment of this application. Figure 3 The multidimensional direct-connect topology shown is a 4x4 two-dimensional torus type topology. Figure 3 Each circle in the diagram represents a node, and the incrementing number on each node serves as its identifier.

[0111] Figure 3 In this context, node 5 is the root node, and the remaining nodes are leaf nodes. After obtaining the identifiers of the nodes in the multidimensional direct connection topology and determining that node 5 is the root node, node 15 determines the path from node 15 to node 5 based on the identifiers of each node in the multidimensional direct connection topology. For example... Figure 3The four straight lines shown in the diagram connect paths, with each node connected by a straight line forming a node on that path. For example, path 1 includes nodes 15, 11, 7, 6, and 5, and path 1 connects to the east (E) port of node 5; path 2 includes nodes 15, 14, 13, 9, and 5, and path 2 connects to the south (S) port of node 5; path 3 includes nodes 15, 12, 8, 4, and 5, and path 3 connects to the west (W) port of node 5; path 4 includes nodes 15, 3, 2, 1, and 5, and path 4 connects to the north (N) port of node 5.

[0112] In multidimensional direct-connect topologies of the same specifications but different types, the paths determined by each node are different. For example, see... Figure 4 The diagram illustrates another multidimensional direct-connect topology provided in an embodiment of this application. Figure 4 The multidimensional direct-connect topology shown is a 4*4 two-dimensional HyperX type topology. Figure 4 Each circle in the diagram represents a node, and the incrementing number on each node serves as its identifier. Figure 4 Node 5 is the root node, and the remaining nodes are leaf nodes. Figure 4 In the multidimensional direct-connection topology shown, node 15 determines 6 paths, which are as follows: Figure 4 The straight lines in the diagram are shown below, and will not be elaborated further here.

[0113] After each leaf node determines the path to which it belongs and the corresponding path information, it can send the determined path information to the root node, so that the root node can determine each path in the multidimensional direct connection topology. For example, the seventh node can send the first path information to the root node. The first path information is used by the root node to determine the first path in the multidimensional direct connection topology, and the first path is any path to which the seventh node belongs.

[0114] In one possible implementation, the paths in the multidimensional direct-connected topology may be determined by the root node, not the leaf nodes. After determining the paths, the root node sends a first message to each leaf node in the multidimensional direct-connected topology. This first message indicates information about the paths determined by the root node. Correspondingly, each leaf node can receive the first message sent by the root node. The content of the first message can be found in the description of the data reduction method performed by the root node below, and will not be repeated here.

[0115] After receiving the first message, the seventh node can broadcast a fifth message to the nodes in the multidimensional direct connection topology according to the path information of the paths determined by the root node. The fifth message is used to indicate the path information in the multidimensional direct connection topology determined by the seventh node. Optionally, the fifth message includes at least one of the following contents:

[0116] The identifier of the first path, which is any path to which the seventh node belongs in the multidimensional direct connection topology;

[0117] The identifier of the root node on the first path;

[0118] The identifier of the seventh node;

[0119] The identifier of the eighth node, which is the node preceding the seventh node on the first path;

[0120] The message type of the fifth message.

[0121] In one possible implementation, the paths in a multidimensional directly connected topology can form at least one path tree (or global tree) within the topology. A path tree includes multiple paths in the topology, connecting the nodes within the topology. For example, see... Figure 5 This illustration shows a schematic diagram of the path tree composition in a multidimensional direct-connection topology provided in an embodiment of this application. The path tree in the multidimensional direct-connection topology includes path trees in different directions, such as... Figure 5 The tree shown in the image is a north-facing tree, a south-facing tree, an east-facing tree, and a west-facing tree. Figure 5 The various path trees in Figure 3 The paths in the tree correspond to each other. In some cases, there can be multiple path trees in one direction. For example, see [link to example]. Figure 6 This illustration shows a schematic diagram of the path tree composition in another multidimensional direct-connection topology provided in an embodiment of this application. Figure 6 The various path trees in Figure 4 Each path in the diagram corresponds to a south-facing tree and an east-facing tree, with a total of 2 south-facing trees and 2 east-facing trees.

[0122] based on Figure 5 and Figure 6 As can be seen, in a path tree, a node belongs to a unique path. Therefore, the path identifier can be equivalent to the path tree identifier, the path number, or the path tree number. Thus, the identifier of the first path in the fifth message can also be the identifier of the first path tree.

[0123] This application embodiment does not limit the message format of the fifth message. For example, the fifth message may include a tree identifier (TreeID) field, a root node identifier (RootID) field, a sender node identifier (SenderID) field, a previous node identifier (PreNodeID) field, and a message type (MessageType). The TreeID field contains the identifier of the path tree, i.e., the identifier of the first path to which the seventh node belongs; the RootID field contains the identifier of the root node in the path tree, i.e., the identifier of the root node on the first path; the SenderID field contains the identifier of the node sending the fifth message, i.e., the identifier of the seventh node; the PreNodeID field contains the identifier of the node preceding the node sending the fifth message in the path tree (also called the parent node of the node sending the fifth message) (the direction is from the root node to the head node of each path), i.e., the identifier of the eighth node; the MessageType field contains the message type of the fifth message, indicating that the fifth message is a message used to indicate information about the root node and the first path.

[0124] Optionally, the fifth message may also include a NextNodeID field, the content of which is the identifier of the next node in the path tree of the node that sent the fifth message (in the direction from the root node to the head node of each path). However, since the seventh node is the root node on the first path, the content of the NextNodeID field in the fifth message can be empty (none).

[0125] In one possible implementation, the nodes in the multidimensional directly connected topology can also be determined jointly by the leaf nodes and the root node. Taking the seventh node as an example, after determining at least one path from the seventh node to the root node, the seventh node sends first path information to the root node to indicate the at least one path determined by the seventh node. Similarly, other leaf nodes can also send first path information to the root node after determining at least one path to which they belong, to indicate the at least one path determined by the other leaf nodes.

[0126] The root node can also determine the paths in the multidimensional direct connection topology based on the identifiers in the topology, and determine the second path information indicating the paths in the multidimensional direct connection topology. After receiving the first path information sent by each leaf node, the root node compares the first path information and the second path information, determines each path in the multidimensional direct connection topology based on the first path information and the second path information, and generates a first message indicating the information of each path in the multidimensional direct connection topology finally determined by the root node.

[0127] For example, the root node can receive first path information sent by the first node, which is used by the root node to determine the first path; obtain second path information, which indicates the path in the multidimensional direct connection topology determined by the root node; and broadcast a first message to the first node if the first path information and the second path information do not conflict (or are the same), which is used to indicate the path in the multidimensional direct connection topology determined by the root node.

[0128] The fact that the first path information and the second path information do not conflict indicates that the first path indicated by the first path information and the path in the multidimensional direct connection topology indicated by the second path information do not conflict. Therefore, the first path determined by the first node and the path in the multidimensional direct connection topology determined by the root node are both valid paths. Then, the root node can send the first message to the first node to ensure that the first node and the root node have consistent confirmation of the path in the multidimensional direct connection topology.

[0129] The seventh node determines the information of each path by receiving the first message. In this method, the root node adjudicates the paths indicated by the first and second path information, ensuring that there are no conflicts or overlaps between the paths in the multidimensional direct-connection topology ultimately determined by the root node.

[0130] In this embodiment of the application, by broadcasting the first and fifth messages, it is possible to ensure that each node has a consistent understanding of the path and root node in the multidimensional direct connection topology, thus ensuring the consistency and accuracy of the global path.

[0131] Regardless of how the seventh node determines its first path, it can also determine the seventh data corresponding to that first path. For example, the seventh node can segment the third-generation data based on the number of paths it belongs to, obtaining data slices with the same number of paths as the seventh node, and determine the path corresponding to each data slice. The data slice corresponding to the first path is the seventh data. Different data slices correspond to different paths, and different paths can transmit data in parallel during data reduction, thus achieving higher bandwidth utilization during data reduction in a multi-dimensional direct-connect topology.

[0132] S202, the seventh node sends the seventh data to the root node. The seventh data is used by the root node to obtain the third data. The third data is used by the nodes in the multidimensional direct connection topology to perform subsequent model training.

[0133] After obtaining the seventh data, the seventh node can send the seventh data to the root node. Optionally, if the seventh node is directly connected to the root node, it can send the seventh data directly to the root node. If the seventh node is not directly connected to the root node, it can send the seventh data to a node connected to it, which then sends the seventh data to the root node. For example, the seventh node can send the seventh data to the eighth node, allowing the seventh data to be transmitted to the root node along the first path.

[0134] The above example, using the seventh node, illustrates the data reduction method performed by the head node in a multidimensional directly connected topology. Below, using the third node as an example, we will provide an illustrative explanation of the data reduction method performed by the intermediate nodes in a multidimensional directly connected topology. See [link to documentation]. Figure 7 The diagram shows a flowchart of a data reduction method executed by a third node according to an embodiment of this application. The data reduction method executed by the third node includes, but is not limited to, the following steps S701 to S704.

[0135] S701, the third node receives the fourth data, which is part of the transmitted data. The transmitted data is the data transmitted between nodes during the model training process.

[0136] Since the third node is an intermediate node, the fourth data is the data sent by the upstream node of the third node. In this embodiment, during a data transmission process, each node transmits a portion of the generated data to the root node. Therefore, the data received by the intermediate node during a data transmission process is a portion of the transmitted data between nodes. Thus, the fourth data received by the third node is a portion of the transmitted data.

[0137] As explained above, before the nodes in a multidimensional direct-connection topology begin data reduction, each node can determine the path in the topology and perform data reduction along the path to which it belongs. In this case, before the third node receives the fourth data, the process also includes: the third node sending first path information to the root node, which is used by the root node to determine the first path.

[0138] The first path information sent by the third node to the root node is obtained based on the path in the multidimensional direct-connection topology determined by the third node. This application embodiment does not limit the method by which the third node determines the first path information; for example, the method by which the third node determines the first path information can be the same as the method by which the seventh node determines the first path information.

[0139] In one possible implementation, the path in the multidimensional direct connection topology is determined by the root node. In this implementation, a third node can determine its own path in the multidimensional direct connection topology by receiving a first message from the root node and based on the path information indicated in the first message. In this method, after determining the path in the multidimensional direct connection topology to which the third node belongs, it broadcasts a third message to the nodes in the multidimensional direct connection topology. This third message indicates the path information determined by the third node.

[0140] The third message includes at least one of the following:

[0141] The identifier of the first path is any path in the multidimensional direct connection topology to which the third node belongs.

[0142] The identifier of the root node on the first path;

[0143] The identifier of the third node;

[0144] The identifier of the fourth node, which is the node following the third node on the first path;

[0145] The identifier of the fifth node, which is the node preceding the third node on the first path;

[0146] The message type of the third message.

[0147] The format of the third message can be the same as that of the fifth message. (Using the third node as...) Figure 3 In the example of node 1, the first path is a path consisting of node 5, node 1, node 2, node 3 and node 15, and the identifier of the first path is Tree1. The content of the third message can be as follows.

[0148] TreeID:Tree1

[0149] RootID:5

[0150] SenderID:1

[0151] NextNodeID:2

[0152] PreNodeID:5

[0153] MessageType: RootBroadcast

[0154] In another possible implementation, the third node can also autonomously determine the first path information and send it to the root node. The root node compares the first path information with the second path information generated autonomously by the root node, and then sends a first message to the third node, sending the information of each path in the finally determined multidimensional direct connection topology to the third node. The third node determines the path information to which the third node belongs according to the path information indicated by the first message, and generates and sends a third message based on the path information to which the third node belongs.

[0155] Regardless of how the third node and other nodes in the multidimensional direct connection topology determine the paths within the topology, all nodes in the topology can perform data reduction according to the determined paths. When nodes in the multidimensional direct connection topology perform data reduction according to the paths, if the third node belongs to the first path and is an intermediate node on the first path, then the fourth data received by the third node is data sent by its upstream node on the first path, and the fourth data corresponds to the first path.

[0156] S702, the third node obtains the fifth data, which is a part of the second generated data. The second generated data is the data generated by the third node during model training.

[0157] For example, the third node obtains the fifth data by: splitting the second generated data to obtain the fifth data. This application embodiment does not limit the method by which the third node splits the second generated data. The third node can split the second generated data uniformly, or it can split it according to the data characteristics of the second generated data, or it can split the second generated data according to the number and characteristics of the paths to which the third node belongs.

[0158] Taking a scenario where the third node belongs to 4 paths as an example, the third node can determine the load capacity of each path based on the load information of the nodes on each path. Then, according to the load capacity of each path, it can segment the second generated data into data slices, with the fifth data slice being one of these slices. Each data slice corresponds to one path belonging to the third node, and the size of the data slice corresponding to each path is positively correlated with the load capacity of each path.

[0159] In this scenario, the third node can store each data slice in a different storage area and record the correspondence between the storage space and the data slice and path. This allows it to accurately obtain the fifth data corresponding to the fourth data from the storage space corresponding to the path when it receives the fourth data sent by the upstream node of the third node in a path to which the third node belongs.

[0160] In one possible implementation, the fourth data is carried by a fourth message, which also includes an identifier of the path corresponding to the fourth data and an identifier of the fourth data. The identifier of the path corresponding to the fourth data is used to determine the second generated data, and the identifier of the fourth data is used to determine the fifth data in the second generated data.

[0161] See Figure 8 The diagram illustrates a fourth message provided in an embodiment of this application. The fourth message is an extension of the Remote Direct Memory Access over Converged Ethernet (RoCE) message, including an Ethernet type (ETH) field, an Internet Protocol (IP) field, a User Datagram Protocol (UDP) field, an Infiniband Base Transport Header (IB BTH) field, a TreeID field, a DataID field, a Message Type field, a Data field, an Infiniband Cyclic Redundancy Check (ICRC) field, and a Frame Check Sequence (FCS) field. The TreeID, DataID, and MessageType fields are extended fields in this embodiment.

[0162] Since a node's path within a path tree is unique, the tree identifier in the TreeID field serves as the path identifier, used to identify the path corresponding to the data carried in the message. The DataID field identifies the data slice, allowing the receiving node to reduce or store the data in the correct location based on the DataID field. The MessageType field identifies the message type. The MessageType field differs between data reduction and the subsequent transmission of fully reduced data from the root node to other nodes, indicating different message types. This allows the receiving node to process and parse the message accordingly, ensuring that the data to be reduced is correctly reduced, or that the reduced data is correctly received and processed by the node.

[0163] For example, see Figure 9 The diagram shows a fourth message provided in an embodiment of this application. Figure 9 The format of the fourth message in the middle and Figure 8 The format of the fourth message is the same. Figure 9 The TreeID field in the fourth message contains Tree1, indicating that the data carried by the fourth message corresponds to the path in Tree1. The DataID field in the fourth message contains 1, meaning that the identifier of the fourth data carried by the fourth message is 1, indicating that the data corresponding to fragment 1 in the local generated data of the node that received the fourth message corresponds to the data carried by the fourth message. The MessageType field in the fourth message contains Reduce, indicating that the type of the fourth message is a message transmitted during the data reduction process, instructing the node that received the fourth message to reduce the data carried by the fourth message with the data corresponding to fragment 1 in the local generated data.

[0164] The third node, through the path identifier and the fourth data identifier carried in the fourth message, can accurately determine the second generated data corresponding to the path identifier, and can determine the storage area of ​​the fifth data corresponding to the fourth data identifier in the storage area of ​​each data slice storing the second generated data. Thus, it can read the fifth data in the storage area storing the fifth data, realize the accurate acquisition of the fifth data, and thus ensure the accuracy and reliability of the reduction based on the fourth data and the fifth data.

[0165] Optionally, the third node can determine the storage area of ​​multiple data slices corresponding to the identifier of the fourth data based on the identifier of the fourth data, determine the second generated data corresponding to the path identifier based on the path identifier of the fourth data, determine the storage area corresponding to the second generated data among the multiple data slices corresponding to the identifier of the fourth data, and read the fifth data to achieve accurate acquisition of the fifth data. That is, in this embodiment, the order of obtaining the fifth data based on the path identifier and the identifier of the fourth data is not limited.

[0166] S703, the third node reduces the fourth and fifth data to obtain the sixth data. The sixth data is used by the root node to obtain the third data. The third data is used by the nodes in the multidimensional direct connection topology to perform subsequent model training.

[0167] In the embodiments of this application, data reduction can involve processing such as aggregation, subtraction, multiplication, division, or taking the absolute value. In one possible implementation, different reduction orders when reducing two data points may result in different reduction results. The reduction order refers to the sequence of operations performed on the data during the data reduction process.

[0168] For example, if we consider reducing two data points to their summation, the node determines the result type based on the type of the data in the first operation. If the first operation's data is a floating-point number, the result will also be a floating-point number even if the second operation's data is an integer. Conversely, if the first operation's data is an integer, the result will be an integer regardless of the second operation's data type.

[0169] For example, if the first operation involves a value of 0.1 (floating-point) and the second operation involves a value of 100 (integer), the sum will be 100.1 (floating-point). Conversely, if the first operation involves a value of 100 (integer) and the second operation involves a value of 0.1, the sum will also be 100 (integer).

[0170] Optionally, since the reduction order when reducing two data points affects the reduction result, in this embodiment, the two data points can be reduced in a specified order to ensure that the reduction result meets the reduction requirements. For example, reducing the fourth and fifth data points to obtain the sixth data point includes: reducing the fourth and fifth data points according to a second reduction order, where the second reduction order is determined based on the data transmission order on the paths corresponding to the fourth and fifth data points. The second reduction order can be the same as or the reverse of the data transmission order on the paths corresponding to the fourth and fifth data points.

[0171] Taking the reduction of the fourth and fifth data points as an example of summing the fourth and fifth data points, if the second reduction order is the same as the data transmission order, since the fourth data point is transmitted before the fifth data point on the path corresponding to the fourth and fifth data points, the sixth data point is the sum of the fourth and fifth data points. However, if the second reduction order is the reverse of the data transmission order, then the sixth data point is the sum of the fifth and fourth data points.

[0172] In the embodiments of this application, each time the data is reduced, the reduction can be performed in a specified order to ensure that the same data is reduced at different times and the results are the same. This makes the reduction results reproducible, meets the requirements of the associative law for large model training or inference, and ensures the accuracy of the reduction results.

[0173] In one possible implementation, the third node can determine whether to reduce the fourth and fifth data based on its computing resources. For example, the third node can process the fourth and fifth data according to the following two cases, depending on whether its computing resources meet the resource requirements.

[0174] Case 1: If the computing resources of the third node meet the resource conditions, the fourth and fifth data are reduced to obtain the sixth data. The resource conditions are used to indicate that the computing resources of the third node are greater than the resources required to reduce the fourth and fifth data.

[0175] The computing resources of the third node can refer to various resources used for computation on the third node, including the computing space or computing power of the third node. The computing power of the third node refers to its ability to perform numerical calculations and processing tasks.

[0176] If the computing resources on the third node meet the resource conditions when the fourth and fifth data are obtained, it means that the computing power of the third node is sufficient to reduce the fourth and fifth data. Therefore, the third node can reduce the fourth and fifth data to obtain the sixth data and send the sixth data to the downstream node of the third node on the path corresponding to the sixth data.

[0177] If the computing resources on the third node do not meet the resource conditions when the fourth and fifth data are obtained, it means that the computing power of the third node is insufficient to reduce the fourth and fifth data. In this case, the third node can temporarily refrain from reducing the fourth and fifth data. Instead, it can wait until some of the computing resources on the third node are released and the computing resources of the third node meet the resource conditions before reducing the fourth and fifth data to obtain the sixth data and sending the sixth data to the downstream nodes of the third node on the path corresponding to the sixth data.

[0178] Scenario 2: If the computing resources of the third node do not meet the resource conditions, send the fourth and fifth data to the sixth node. The resource conditions are used to indicate that the computing resources of the third node are greater than the resources required to reduce the fourth and fifth data. The sixth node is the node preceding the third node on the path corresponding to the fourth data.

[0179] If the computing resources on the third node do not meet the resource conditions when the third node obtains the fourth and fifth data, the third node may not reduce the fourth and fifth data, nor wait for some of the computing resources on the third node to be released. Instead, it may directly send the fourth and fifth data to the downstream nodes on the path corresponding to the fourth and fifth data, and let the downstream nodes reduce the fourth and fifth data and some of the data of the downstream nodes.

[0180] In one possible implementation, if the third node sends the fourth and fifth data directly to the downstream node without reducing them, the third node may also send a seventh message to the downstream device before sending the fourth and fifth data, indicating that the third node has not reduced the fourth and fifth data. Upon receiving the seventh message from the third node, the downstream node determines that it needs to receive the fourth and fifth data, and after confirming that it has received them, it reduces the fourth and fifth data, as well as a portion of the data generated by the downstream node, ensuring the accuracy of the reduction.

[0181] Since the upstream nodes of the third node may also experience resource shortages, the fourth data received by the third node could be either reduced data from the upstream node or unreduced data. If the fourth data received by the third node is reduced data from the upstream node, the third node can reduce the fourth and fifth data after receiving a message carrying the fourth data, provided its computing resources meet the resource requirements. However, if the fourth data is unreduced data from the upstream node and comprises multiple sub-data, the third node can reduce the multiple sub-data and the fifth data after receiving multiple messages carrying these sub-data.

[0182] In one possible implementation, the third node can allocate storage space for the data to be received and the data generated by the third node. This application embodiment does not limit the size or number of storage spaces allocated by the third node for each piece of data. For example, the third node can determine the number of storage spaces it allocates based on the number of paths it belongs to during the full reduction process. For instance, if the third node belongs to 4 paths during the full reduction process, then the number of storage spaces allocated by the third node for each path can be 2, meaning the third node can allocate a total of 8 storage spaces. One of the two storage spaces corresponding to a path is used to store the data sent by the upstream node on that path, and the other storage space is used to store the portion of the third node's second generated data that corresponds to that path.

[0183] As explained above, the fourth data received by the third node from the upstream node may be unreduced data, potentially comprising data carried by multiple packets. In this case, after receiving the seventh packet from the upstream node, the third node can allocate more storage space for the fourth data based on the seventh packet, allowing multiple storage spaces to store various sub-data within the fourth data. Optionally, the seventh packet may include the number of packets carrying each sub-data of the fourth data, enabling the third node to accurately allocate multiple storage spaces for the fourth data based on the quantities in the seventh packet, thus avoiding resource waste.

[0184] In this embodiment, combining sequential reduction with a mechanism that pre-allocates storage space for the data to be received further ensures the accuracy of data reduction. For example, see... Figure 10 The diagram illustrates a data reduction process according to an embodiment of this application. The IO Die in the diagram is a device for data reduction on a third node. The IO Die pre-allocates storage space in a cache for the remote data to be received (i.e., data sent to the third node by other nodes, such as the fourth data) and for the local data to be reduced (such as the fifth data). When the IO Die on the third node receives the remote data, it stores the remote data b+c in the corresponding storage space, stores the local data a to be reduced in the storage space, and reduces the local data and remote data according to a second reduction order to obtain the reduced data a+b+c. The second reduction order is the order of local data + remote data, that is, the order in which local data is used as the data for the first operation and remote data is used as the data for the second operation.

[0185] Figure 10 This illustrates the data reduction process performed by the third node when the remote data is the reduced data from the upstream node, and the third node has allocated storage space for the reduced data in advance. In some cases, the fourth data received by the third node may be unreduced data, comprising multiple sub-data. In this situation, the third node can determine the number of sub-data included in the fourth data based on the received seventh message and increase the storage space allocated for the fourth data. This ensures that the third node can reduce each sub-data and the fifth data sequentially after receiving them, guaranteeing the accuracy of the reduction result.

[0186] For example, see Figure 11 The diagram illustrates a data reduction method provided in an embodiment of this application. Figure 11 and Figure 10The difference lies in the fact that since the third node received the seventh message, and the seventh message indicated that the fourth data included two sub-data, remote data 1 (data b) and remote data 2 (data c), the third node increased the amount of storage space allocated for the fourth data. This allowed the third node's IO Die to reduce the sub-data of the fourth data and the fifth data according to the second reduction order after receiving all the sub-data of the fourth data.

[0187] In this scenario, if the second reduction order is local data + remote data 1 + remote data 2, even if the third node receives remote data 2 first, the third node will not reduce the local data and remote data 2. Instead, it will wait to receive remote data 1 and then reduce the local data, remote data 1, and remote data 2 according to the reduction order to obtain the reduction result a+b+c, thus ensuring the accuracy of the reduction result.

[0188] S704, the third node sends the sixth data to the root node.

[0189] Since the third node is an intermediate node, after reducing the fourth and fifth data to obtain the sixth data, the third node also needs to send the sixth data to the root node so that the root node can obtain the global reduced data based on the sixth data.

[0190] If the third node is directly connected to the root node, it can send the sixth data directly to the root node. If the third node is not directly connected to the root node, it can send the sixth data to a node it is connected to, and that node will then send the sixth data to the root node. For example, the third node can send the sixth data to the fourth node, allowing the data to be transmitted to the root node along the first path.

[0191] The data reduction method performed by intermediate nodes in a multidimensional directly connected topology has been introduced above through S701 to S704. Below, an exemplary description of the data reduction method performed by the root node in a multidimensional directly connected topology is provided. See [link to documentation]. Figure 12 The diagram shows a flowchart of a data reduction method performed by a root node according to an embodiment of this application. The data reduction method performed by the root node includes, but is not limited to, the following steps S1201 to S1204.

[0192] S1201, the root node receives the first data, which is part of the transmitted data, which is the data transmitted between nodes during the model training process.

[0193] The first data is the data sent by the upstream node of the root node. In this embodiment of the application, during a data transmission process, each node transmits a portion of the generated data to the root node. Therefore, the data received by the root node during a data transmission process is a portion of the data transmitted between nodes. Thus, the first data received by the root node is a portion of the transmitted data.

[0194] As explained above, before data reduction begins at the nodes in the multidimensional direct-connection topology, each node can determine the path within the topology and perform data reduction along the path to which it belongs. This application embodiment does not limit the method by which the root node determines the nodes in the multidimensional direct-connection topology before receiving the first data.

[0195] For example, the root node receives first path information sent by the first node, which is used by the root node to determine a first path. The first node is a leaf node in the multidimensional direct connection topology, including but not limited to the third and seventh nodes mentioned above. After determining the first path information, the first node sends the first path information to the root node. Since the first path information indicates the path information in the multidimensional direct connection topology determined by the first node, the root node can determine the path in the multidimensional direct connection topology determined by the first node based on the first path information, and use the path in the multidimensional direct connection topology determined by the first node as the path in the multidimensional direct connection topology determined by the root node.

[0196] In one possible implementation, the root node obtains second path information, which indicates the path in the multidimensional direct-connection topology determined by the root node; and broadcasts a first message to the first node, which indicates the path information in the multidimensional direct-connection topology determined by the root node.

[0197] Since the root node can obtain the identifiers of the nodes in the multidimensional direct connection topology, and the identifiers in the multidimensional direct connection topology indicate the dimensions and positions of the nodes in the multidimensional direct connection topology, the root node can determine the path in the multidimensional direct connection topology based on the identifiers of the nodes in the multidimensional direct connection topology, thereby determining the second path information.

[0198] Alternatively, the paths in a multidimensional direct-connection topology can be manually specified, and a second path information can be configured for the root node, enabling the root node to obtain the second path information. After determining the second path information, the root node broadcasts a first message to the first node, allowing the first node to determine the path information determined by the root node based on the first message.

[0199] The first message includes at least one of the following:

[0200] The identifier of the first path is any path in the multidimensional direct connection topology;

[0201] The identifier of the root node;

[0202] The identifier of the node that sent the first message;

[0203] The identifier of the second node, which is the node following the root node on the first path;

[0204] The message type of the first message.

[0205] The format of the first message can be the same as that of the third and fifth messages. (Based on the root node...) Figure 3 In the example of node 5, the first path is a path consisting of nodes 5, 1, 2, 3 and 15, and the identifier of the first path is Tree1. The content of the first message can be as follows.

[0206] TreeID:Tree1

[0207] RootID:5

[0208] SenderID:5

[0209] NextNodeID:1

[0210] PreNodeID:None

[0211] MessageType:RootBroadcast

[0212] In another possible implementation, the root node can obtain the first path information and the second path information, compare the first path information and the second path information, and determine the path in the final multidimensional direct connection topology.

[0213] In this approach, the root node adjudicates the paths indicated by the first and second path information, ensuring that there are no conflicts or overlaps between the paths in the final multidimensional direct-connection topology determined by the root node. The root node determines the final paths in the multidimensional direct-connection topology.

[0214] Regardless of how the root node and other nodes in the multidimensional direct-connection topology determine the paths within the topology, all nodes in the topology can perform data reduction according to the determined paths. When nodes in the multidimensional direct-connection topology perform data reduction according to paths, if the root node belongs to the first path, the first data received by the root node is the data sent by the upstream node of the root node on the first path (i.e., the first node on the first path), and the first data corresponds to the first path.

[0215] S1202, the root node obtains the second data, which is a part of the first generated data. The first generated data is the data generated by the root node during model training.

[0216] For example, the root node obtains the second data by: splitting the first generated data to obtain the second data. This application embodiment does not limit the method by which the root node splits the first generated data. The root node can split the first generated data uniformly, or it can split it according to the data characteristics of the first generated data, or it can split the first generated data according to the number and characteristics of the paths to which the root node belongs.

[0217] Taking a root node belonging to 4 paths as an example, the root node can determine the load capacity of each path based on the load information of the nodes on each path. Then, according to the load capacity of each path, it divides the generated data into first data slices, and the second data slice is one of these data slices. Each data slice corresponds to one path belonging to the root node, and the size of the data slice corresponding to each path is positively correlated with the load capacity of each path.

[0218] In this scenario, the root node can store each data slice in a different storage area and record the correspondence between the storage space and the data slice and path. This allows it to accurately obtain the second data corresponding to the first data from the storage space corresponding to the path when it receives the first data sent by the upstream node of the root node in a path to which the root node belongs.

[0219] In one possible implementation, the first data is carried by a second message, which also includes an identifier of the path corresponding to the first data and an identifier of the first data. The identifier of the path corresponding to the first data is used to determine the first generated data, and the identifier of the first data is used to determine the second data in the first generated data.

[0220] The format of the second message can be the same as that of the fourth message. The process by which the root node determines the second data based on the identifier of the path corresponding to the first data and the identifier of the first data can be referred to the process in S702 by which the third node determines the fifth data based on the identifier of the path corresponding to the fourth data and the identifier of the fourth data. This will not be repeated here.

[0221] S1203, the root node reduces the first and second data to obtain the third data, which is used by the nodes in the multidimensional direct connection topology to perform subsequent model training.

[0222] In one possible implementation, reducing the first data and the second data to obtain the third data includes: reducing the first data and the second data according to a first reduction order to obtain the third data, wherein the first reduction order is determined based on the data transmission order on the paths corresponding to the first data and the second data. The first reduction order can be the same as or the reverse of the data transmission order on the paths corresponding to the first data and the second data.

[0223] S1204, the root node sends third data to the first node in the multidimensional direct connection topology. The first node is a node in the multidimensional direct connection topology other than the root node.

[0224] This application does not limit the method by which the root node sends third data to the first node in a multidimensional directly connected topology. For example, if the root node and the first node are directly connected, the root node can directly send the third data to the first node. However, if the root node and the first node are not directly connected, and the first node is a node on the first path, the root node can return the third data to the first node according to the first path. Sending third data from the root node to the first node enables the first node to perform subsequent model training based on the reduction result, ensuring successful model training.

[0225] In one possible implementation, since the nodes in the multidimensional direct connection topology perform data reduction according to multiple paths in multiple path trees, the root node can receive data sent by the upstream nodes of the root node on each path to which the root node belongs, and reduce the received data with a portion of the data in the first generated data according to the path corresponding to the data to obtain multiple reduction results. The third data is the reduction result corresponding to one of the paths.

[0226] Therefore, the root node can not only send third data to the first node, but also send reduction results corresponding to other paths to the first node. This application embodiment does not limit the method by which the root node sends reduction results corresponding to other paths to the first node. For example, the root node can send the reduction results corresponding to each path to the first node on each path separately; or, the root node can package multiple reduction results and send the packaged result to the first node.

[0227] Optionally, the root node can also reduce the multiple reduction results corresponding to multiple paths again to obtain the full reduction result, and send the full reduction result directly to or according to any path to which the first node belongs.

[0228] See Figure 13This diagram illustrates a complete data reduction process according to an embodiment of this application. Nodes 1 and 2 are examples of leaf nodes in a multidimensional direct-connection topology. In step 1, identifiers are assigned to each node (including the root node, node 1, and node 2). After obtaining their identifiers, each node exchanges identifiers with the others via flooding. In step 2, the root node broadcasts the path information in the multidimensional direct-connection topology determined by the root node to nodes 1 and 2, ensuring that each node's understanding of the paths in the multidimensional direct-connection topology is consistent with that of the root node. In step 3, reduction begins from the leaf nodes and proceeds towards the root node along the paths. The root node then broadcasts the final reduction result back to the leaf nodes.

[0229] During data reduction, two tasks—communication and computation—need to be executed in parallel. When performing the communication task, data needs to be read from the node's storage unit and transmitted, consuming the node's bandwidth designated for data transmission. When performing the computation task, data needs to be read from the storage unit. Since the computation unit and storage unit within a node are separate units, data also needs to be transmitted within the node during computation, further consuming the node's bandwidth. Therefore, data reduction involves a competition between the communication and computation tasks for the node's storage unit and bandwidth, leading to a decrease in the node's communication performance.

[0230] In this embodiment, new software or programs can be flashed onto modules with forwarding capabilities connected to or deployed on nodes through software modification. This allows the forwarding modules to not only receive and send data but also reduce the data, enabling simultaneous transmission and computation. For example, see... Figure 14 The diagram illustrates a node provided in an embodiment of this application. The node includes a DDR, a compute die, and an IO die. The IO die is a module on the node with forwarding functionality. An on-network compute engine for data reduction is flashed onto the IO die.

[0231] By making soft modifications to the IO Die, the IO Die on a node can receive data from other nodes (such as...). Figure 14 As shown in ①), the IO Die reads the data that needs to be reduced on this node from the DDR (such as...). Figure 14 As shown in ②), and performs reduction and transmission on the data that needs to be reduced on this node and the data received from other nodes (such as...). Figure 14 As shown in ③), the calculation of IO Die is offloaded, and the number of DDR calls is reduced. This makes the competition for storage units and bandwidth between the two tasks of calculation and communication less during the data reduction process, thus improving the efficiency of communication.

[0232] Furthermore, in a single data reduction process in this application embodiment, the static delay between the head node starting data transmission and the root node determining the reduction result is relatively small. For example, see... Figure 15 The diagram illustrates a time delay provided by an embodiment of this application. Figure 3 In the illustrated 2D 4x4 Torus-type topology, multi-tree along-path aggregation achieves a maximum hop count of 5. The latency that cannot be masked by the pipeline is the transmission latency of 5 packets (or data packets). For example, when node 12 transmits pkt1 to pkt5, node 5 cannot receive the pkt and cannot work. However, when node 12 transmits pkt6, node 5 can receive pkt1 and start working. In a direct-connect topology, full bandwidth is utilized, the number of transmissions is halved, and both static latency and DDR access frequency are reduced, effectively improving network communication efficiency and bandwidth utilization. This avoids the linear increase in static latency and memory access overhead with node size in a hierarchical ringall-reduce topology.

[0233] exist Figure 3 In the multidimensional direct-connect topology shown, the total delay in the data reduction process is (α+θ)+2(M / D·1 / B+L_(non-over)), where D is the degree of a node in the multidimensional direct-connect topology, that is, the number of nodes directly connected to any node. Figure 3 In the multidimensional direct-connect topology shown, D is 4. α is the static overhead, θ is the memory overhead, and B is the single-link bandwidth. L_(non-over) = (MaxHop × PktSize) / B considers the different hop counts from different nodes to the root node (e.g., ...). Figure 3 The maximum number of hops in the multidimensional direct-connect topology shown is 5. Figure 4 The maximum number of hops in the multidimensional direct-connect topology shown is 3. Assuming that the nodes in the multidimensional direct-connect topology can perform data reduction along the path in a pipeline manner, its value should be the unmaskable static delay.

[0234] Below, we will compare related technologies with the embodiments of this application to further explain and supplement the beneficial effects of the embodiments of this application.

[0235] See Figure 16 This document provides an implementation scenario diagram of the related technology. The technology is applied to a 4x4 two-dimensional HyperX topology, comprising 16 nodes. Any one of these 16 nodes can connect to any node along the same dimension. The technology achieves high throughput through pipelined embedding. Each row (e.g., the y0 to y3 direction in the diagram) has a root node, and the root nodes in each row are located in the same column. Furthermore, the column containing the root node of each row contains another root node for the entire topology. For example... Figure 16As shown, the root node in each row from x0 to x3 is located in column y2. The node with coordinates (x2, y2) in column y2 is then determined as the root node of the entire topology.

[0236] During data reduction, each node transmits its local data to the root node of its respective row. The root node of each row then sends the received data along with its local data to the root node of the entire topology. The root node of the entire topology then performs the data reduction for all nodes in the topology. However, in this data reduction process, each node sends its complete local data to the root node, utilizing only a portion of the available network bandwidth in the topology, resulting in low bandwidth utilization.

[0237] For example, see Figure 17 This diagram illustrates the operational flowchart of data reduction in related technologies. The nodes in these technologies include three interconnected parts: a computation die, a DDR (Data Transfer Layer), and an IO die. The computation die comprises a communication core and a computation core; the communication core is used for data transmission, while the computation core performs calculations such as data reduction. Figure 17 When the nodes shown perform data reduction, the IO Die on the node receives data sent by other nodes and transfers the received data from other nodes to the computation Die (e.g., ...). Figure 17 As shown in ①). The computation die reads local data stored in DDR (such as...) through the communication core. Figure 17 As shown in ②), the core is used to reduce the local data and the data received from other nodes to obtain the reduced data. Then, the Die writes the reduced data back to the DDR (e.g., ...). Figure 17 As shown in ③), this allows the DDR to store the reduced data. When a node needs to send the reduced data to other nodes, the computation Die reads the reduced data stored in the DDR through the communication core and transmits the reduced data to the IO Die. After obtaining the reduced data, the IO Die sends the reduced data to other nodes (such as...). Figure 17 (as shown in ④).

[0238] The related technology requires three calls to DDR during the data reduction process: two readings of data from DDR and one writing of data to DDR. Furthermore, data transmission within the node is also required during the DDR call process. This causes communication and computing tasks to compete for the communication bandwidth of the storage unit DDR and the node, thereby reducing communication efficiency.

[0239] In summary, the data reduction method provided in this application can improve communication efficiency and bandwidth utilization during the data reduction process.

[0240] The data reduction method provided in the embodiments of this application has been described above. Corresponding to the above method, the embodiments of this application also provide a data reduction device. This device is applied to the root node. This device is used to reduce data by... Figure 18 Each module shown performs the above... Figure 12 The data reduction method performed by the root node. For example... Figure 18 As shown in the embodiments of this application, the data reduction device includes the following modules.

[0241] A data reduction device is provided, applied to at least one root node in a multidimensional directly connected topology, where nodes in the multidimensional directly connected topology are used for model training. The device includes: a first transceiver module 1801, used to receive first data, which is a portion of transmitted data, and the transmitted data is data transmitted between nodes during model training; a first processing module 1802, used to obtain second data, which is a portion of first generated data, and the first generated data is data generated by the root node during model training; reducing the first data and the second data to obtain third data, which is used by the nodes in the multidimensional directly connected topology to perform subsequent model training; and the first transceiver module 1801, used to send the third data to a first node in the multidimensional directly connected topology, where the first node is a node in the multidimensional directly connected topology other than the root node.

[0242] In one possible implementation, the first data is the data corresponding to the first path in the multidimensional direct connection topology, and the first node is the node on the first path; the first transceiver module 1801 is also used to receive the first path information sent by the first node, and the first path information is used by the root node to determine the first path.

[0243] In one possible implementation, the first processing module 1802 is further configured to acquire second path information, which indicates the path in the multidimensional direct connection topology determined by the root node; the first transceiver module 1801 is further configured to broadcast a first message to the first node, which indicates the path information in the multidimensional direct connection topology determined by the root node.

[0244] In one possible implementation, the first message includes at least one of the following: an identifier of a first path, which is any path in a multidimensional direct-connection topology; an identifier of the root node; an identifier of the node that sent the first message; an identifier of a second node, which is the node following the root node on the first path; and a message type of the first message.

[0245] In one possible implementation, the first data is carried by a second message, which also includes an identifier of the path corresponding to the first data and an identifier of the first data. The identifier of the path corresponding to the first data is used to determine the first generated data, and the identifier of the first data is used to determine the second data in the first generated data.

[0246] In one possible implementation, the first processing module 1802 is used to segment the first generated data to obtain the second data.

[0247] In one possible implementation, the first processing module 1802 is used to reduce the first data and the second data according to a first reduction order to obtain the third data. The first reduction order is determined based on the data transmission order on the paths corresponding to the first data and the second data.

[0248] See Figure 19 A data reduction device is provided, applied to a third node, which is a node in a multidimensional directly connected topology other than the root node. Nodes in the multidimensional directly connected topology are used for model training. The device includes: a second transceiver module 1901 for receiving fourth data, which is a portion of the transmitted data (data transmitted between nodes during model training); a second processing module 1902 for obtaining fifth data, which is a portion of the second generated data (data generated by the third node during model training); reducing the fourth and fifth data to obtain sixth data, which is used by the root node to obtain the third data, and the third data is used by nodes in the multidimensional directly connected topology to perform subsequent model training; and the second transceiver module 1901 for sending the sixth data to the root node.

[0249] In one possible implementation, the fourth data is the data corresponding to the first path in the multidimensional direct connection topology, and the third node is the node on the first path; the second transceiver module 1901 is also used to send the first path information to the root node, and the first path information is used by the root node to determine the first path.

[0250] In one possible implementation, the second transceiver module 1901 is further configured to receive a first message sent by the root node, the first message indicating information about the path in the multidimensional direct connection topology determined by the root node; and broadcast a third message to the nodes in the multidimensional direct connection topology, the third message indicating information about the path in the multidimensional direct connection topology determined by the third node.

[0251] In one possible implementation, the third message includes at least one of the following: an identifier of the first path, which is any path in the multidimensional direct connection topology; an identifier of the root node on the first path; an identifier of the third node; an identifier of the fourth node, which is the node following the third node on the first path; an identifier of the fifth node, which is the node preceding the third node on the first path; and a message type of the third message.

[0252] In one possible implementation, the fourth data is carried by a fourth message, which also includes an identifier of the path corresponding to the fourth data and an identifier of the fourth data. The identifier of the path corresponding to the fourth data is used to determine the second generated data, and the identifier of the fourth data is used to determine the fifth data in the second generated data.

[0253] In one possible implementation, the second processing module 1902 is used to segment the second generated data to obtain the fifth data.

[0254] In one possible implementation, the second processing module 1902 is used to reduce the fourth data and the fifth data according to a second reduction order to obtain the sixth data, wherein the second reduction order is determined based on the data transmission order on the paths corresponding to the fourth data and the fifth data.

[0255] In one possible implementation, the second processing module 1902 is used to reduce the fourth and fifth data to obtain the sixth data, provided that the computing resources of the third node meet the resource conditions. The resource conditions are used to indicate that the computing resources of the third node are greater than the resources required to reduce the fourth and fifth data.

[0256] In one possible implementation, the second transceiver module 1901 is used to send fourth and fifth data to the sixth node when the computing resources of the third node do not meet the resource conditions. The resource conditions are used to indicate that the computing resources of the third node are greater than the resources required to reduce the fourth and fifth data. The sixth node is the node preceding the third node on the path corresponding to the fourth data.

[0257] See Figure 20 A data reduction device is provided, which is applied to a seventh node, which is a node in a multidimensional direct-connected topology other than the root node. The nodes in the multidimensional direct-connected topology are used for model training. The device includes: a third processing module 2002, which is used to obtain seventh data, which is a part of the third generated data, which is the data generated by the seventh node during model training; and a third transceiver module 2001, which is used to send the seventh data to the root node, which is used by the root node to obtain the third data, and the third data is used by the nodes in the multidimensional direct-connected topology to perform subsequent model training.

[0258] In one possible implementation, the seventh node is a node on the first path in the multidimensional direct connection topology; the third transceiver module 2001 is also used to send the first path information to the root node, and the first path information is used by the root node to determine the first path.

[0259] In one possible implementation, the third transceiver module 2001 is further configured to receive a first message sent by the root node, the first message indicating information about the path in the multidimensional direct connection topology determined by the root node; and broadcast a fifth message to the nodes in the multidimensional direct connection topology, the fifth message indicating information about the path in the multidimensional direct connection topology determined by the seventh node.

[0260] In one possible implementation, the fifth message includes at least one of the following: an identifier of the first path, which is any path in the multidimensional direct connection topology; an identifier of the root node on the first path; an identifier of the seventh node; an identifier of the eighth node, which is the node preceding the seventh node on the first path; and a message type of the fifth message.

[0261] In one possible implementation, the third processing module 2002 is used to segment the third generated data to obtain the seventh data.

[0262] In one possible implementation, the first processing module, the second processing module, or the third processing module 2002 is further configured to obtain the identifiers of nodes in the multidimensional direct connection topology, wherein the identifier of any node indicates the dimension and position of any node in the multidimensional direct connection topology; and determine the path and at least one root node in the multidimensional direct connection topology based on the identifiers of the nodes in the multidimensional direct connection topology.

[0263] In one possible implementation, at least one root node is determined based on the load information of the nodes in the multidimensional direct-connected topology.

[0264] It should be understood that the beneficial effects of the device described above in implementing its functions are the same as those of the data reduction method, and will not be repeated here. Furthermore, the implementation of the device's functions is only illustrated by the division of the functional modules described above. In practical applications, the functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Additionally, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation processes are detailed in the method embodiments, and will not be repeated here.

[0265] In an exemplary embodiment, a data reduction system is provided, comprising a root node, a third node, and a seventh node, wherein the root node is used for execution. Figure 12 The data reduction method in the code, the third node is used for execution. Figure 7 The data reduction method in the code, the seventh node is used for execution. Figure 2 Data reduction methods in [the context of data reduction].

[0266] In an exemplary embodiment, a data reduction device is provided. The device includes a processor coupled to a memory. The memory stores at least one instruction, which is loaded and executed by the processor to enable the data reduction device to perform... Figure 2 Data reduction methods or implementations in [the context of data reduction]. Figure 7 Data reduction methods or implementations in [the context of data reduction]. Figure 12 Data reduction methods in [the context of data reduction].

[0267] Ninthly, a computer program (product) is provided, comprising: computer program code, which, when executed by a computer, causes the computer to perform... Figure 2 , Figure 7 or Figure 12 The method in the middle.

[0268] In a tenth aspect, a computer-readable storage medium is provided that stores a program or instructions, which, when executed on a computer, [are described above]. Figure 2 , Figure 7 or Figure 12 The method in the code is executed.

[0269] Eleventhly, a chip is provided, including a processor for retrieving and executing instructions stored in memory, causing a computer equipped with the chip to perform the aforementioned... Figure 2 , Figure 7 or Figure 12 The method in the middle.

[0270] In a twelfth aspect, another chip is provided, comprising: an input interface, an output interface, a processor, and a memory, wherein the input interface, output interface, processor, and memory are connected via internal interconnection paths, and the processor is used to execute code in the memory. When the code is executed, a computer with the chip installed performs the aforementioned actions. Figure 2 , Figure 7 or Figure 12 The method in the middle.

[0271] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive).

[0272] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with substantially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or order of execution. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another.

[0273] It should also be understood that, in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0274] In this application, the term "at least one" means one or more, and the term "multiple" means two or more. For example, multiple second devices means two or more second devices. The terms "system" and "network" are often used interchangeably herein.

[0275] It should be understood that the terminology used in the description of the various examples herein is for the purpose of describing particular examples only and is not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0276] It should also be understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. The term "and / or" describes an association between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " in this application generally indicates that the preceding and following related objects are in an "or" relationship.

[0277] It should also be understood that the terms “if” and “if” can be interpreted as meaning “when” or “upon”, or “in response to determination” or “in response to detection”. Similarly, depending on the context, the phrases “if determination…” or “if detection [the stated condition or event]” can be interpreted as meaning “when determination…”, or “in response to determination…”, or “when detection [the stated condition or event]” or “in response to detection [the stated condition or event]”.

[0278] The above description is merely an embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method of data reduction, comprising: The method is applied to at least one root node in a multi-dimensional direct connection topology, nodes in the multi-dimensional direct connection topology are used for model training, and the method comprises: receiving first data, the first data being part of transmission data, the transmission data being data transmitted between nodes in the model training process; obtaining second data, the second data being part of first generated data, the first generated data being data generated by the root node in the model training process; reducing the first data and the second data to obtain third data, the third data being used for subsequent model training of nodes in the multi-dimensional direct connection topology; sending the third data to a first node in the multi-dimensional direct connection topology, the first node being a node other than the root node in the multi-dimensional direct connection topology.

2. The method of claim 1, wherein, The first data is data corresponding to a first path in the multi-dimensional direct connection topology, and the first node is a node on the first path; before receiving the first data, the method further comprises: receiving first path information sent by the first node, the first path information being used for the root node to determine the first path.

3. The method according to claim 1 or 2, characterized in that, Before receiving the first data, the method further comprises: obtaining second path information, the second path information indicating a path in the multi-dimensional direct connection topology determined by the root node; broadcasting a first packet to the first node, the first packet being used to indicate information of the path in the multi-dimensional direct connection topology determined by the root node.

4. The method of claim 3, wherein, The first packet comprises at least one of the following contents: an identifier of a first path, the first path being any path in the multi-dimensional direct connection topology; an identifier of the root node; an identifier of a node sending the first packet; an identifier of a second node, the second node being a node subsequent to the root node on the first path; a packet type of the first packet.

5. The method according to any of claims 1 to 4, characterized in that, The first data is carried by a second packet, the second packet further comprising an identifier of a path corresponding to the first data and an identifier of the first data, the identifier of the path corresponding to the first data being used to determine the first generated data, and the identifier of the first data being used to determine the second data in the first generated data.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: segmenting the first generated data to obtain the second data.

7. The method according to any of claims 1 to 6, characterized in that The method further comprises: reducing the first data and the second data according to a first reduction order to obtain the third data, the first reduction order being determined based on a data transmission order on a path corresponding to the first data and the second data.

8. A data reduction method characterized by, The method is applied to a third node, the third node being a node other than a root node in a multi-dimensional direct connection topology, nodes in the multi-dimensional direct connection topology being used for model training, and the method comprising: receiving fourth data, the fourth data being part of transmission data, the transmission data being data transmitted between nodes in the model training process; obtaining fifth data, the fifth data being part of second generated data, the second generated data being data generated by the third node in the model training process; reducing the fourth data and the fifth data to obtain sixth data, the sixth data being used by the root node to obtain third data, the third data being used by nodes in the multi-dimensional direct connection topology to perform subsequent model training; sending the sixth data to the root node.

9. The method of claim 8, wherein, The fourth data is data corresponding to a first path in the multi-dimensional direct connection topology, and the third node is a node on the first path. Before the fourth data is received, the method further includes: sending first path information to the root node, the first path information being used by the root node to determine the first path.

10. The method according to claim 8 or 9, characterized in that, Before the fourth data is received, the method further includes: receiving a first message sent by the root node, the first message being used to indicate information of a path in the multi-dimensional direct connection topology determined by the root node; broadcasting a third message to nodes in the multi-dimensional direct connection topology, the third message being used to indicate information of a path in the multi-dimensional direct connection topology determined by the third node.

11. The method of claim 10, wherein, The third message includes at least one of the following contents: an identifier of a first path, the first path being any path in the multi-dimensional direct connection topology; an identifier of a root node on the first path; an identifier of the third node; an identifier of a fourth node, the fourth node being a next node of the third node on the first path; an identifier of a fifth node, the fifth node being a previous node of the third node on the first path; a message type of the third message.

12. The method according to any of claims 8-11, characterized by, The fourth data is carried by a fourth message, the fourth message further including an identifier of a path corresponding to the fourth data and an identifier of the fourth data, the identifier of the path corresponding to the fourth data being used to determine the second generated data, and the identifier of the fourth data being used to determine the fifth data in the second generated data.

13. The method according to claims 8-12, characterized in that, The obtaining of the fifth data includes: segmenting the second generated data to obtain the fifth data.

14. The method of any one of claims 8-13, wherein, The reducing of the fourth data and the fifth data to obtain the sixth data includes: reducing the fourth data and the fifth data according to a second reduction order to obtain the sixth data, the second reduction order being determined based on a data transmission order on a path corresponding to the fourth data and the fifth data.

15. The method of any of claims 8-14, wherein, The reducing of the fourth data and the fifth data to obtain the sixth data includes: in a case where a computing resource of the third node meets a resource condition, reducing the fourth data and the fifth data to obtain the sixth data, the resource condition being used to indicate that the computing resource of the third node is greater than a resource required for reducing the fourth data and the fifth data.

16. The method of any one of claims 8-14, wherein, After the fourth data is received, the method further includes: In a case where the computing resource of the third node does not satisfy a resource condition, the fourth data and the fifth data are sent to a sixth node, the resource condition is used to indicate that the computing resource of the third node is greater than the resource required for reducing the fourth data and the fifth data, and the sixth node is a previous node of the third node on a path corresponding to the fourth data.

17. A method of data reduction, comprising: The method is applied to a seventh node, the seventh node is a node in a multi-dimensional direct connection topology except a root node, and the nodes in the multi-dimensional direct connection topology are used for model training. The method comprises: obtaining seventh data, the seventh data being part of third generated data, the third generated data being data generated by the seventh node in the model training process; sending the seventh data to the root node, the seventh data being used for the root node to obtain third data, and the third data being used for the nodes in the multi-dimensional direct connection topology to perform subsequent model training.

18. The method of claim 17, wherein, The seventh node is a node on a first path in the multi-dimensional direct connection topology. Before the seventh data is obtained, the method further comprises: sending first path information to the root node, the first path information being used for the root node to determine the first path.

19. The method of claim 17 or 18, wherein, Before the seventh data is obtained, the method further comprises: receiving a first message sent by the root node, the first message being used to indicate information of a path in the multi-dimensional direct connection topology determined by the root node; broadcasting a fifth message to the nodes in the multi-dimensional direct connection topology, the fifth message being used to indicate information of a path in the multi-dimensional direct connection topology determined by the seventh node.

20. The method of claim 19, wherein, The fifth message comprises at least one of the following contents: an identifier of a first path, the first path being any path in the multi-dimensional direct connection topology; an identifier of a root node on the first path; an identifier of the seventh node; an identifier of an eighth node, the eighth node being a previous node of the seventh node on the first path; a message type of the fifth message.

21. The method of any of claims 17-20, wherein, The seventh data is obtained by: segmenting the third generated data to obtain the seventh data.

22. The method of any one of claims 1-21, wherein, The method further comprises: obtaining identifiers of the nodes in the multi-dimensional direct connection topology, an identifier of any node indicating a dimension and a position of the any node in the multi-dimensional direct connection topology; determining the paths in the multi-dimensional direct connection topology and the at least one root node according to the identifiers of the nodes in the multi-dimensional direct connection topology.

23. The method of any one of claims 1-22, wherein, The at least one root node is determined based on load information of the nodes in the multi-dimensional direct connection topology.

24. A data reduction device, comprising: The device is applied to at least one root node in a multi-dimensional direct connection topology, the nodes in the multi-dimensional direct connection topology are used for model training, and the device comprises: a first transceiver module, configured to perform the receiving and / or sending related operations in the method of any one of claims 1-7, 22 or 23; a first processing module, configured to perform other operations in addition to the receiving and / or sending related operations in the method of any one of claims 1-7, 22 or 23.

25. A data reduction device, comprising: The device is applied to a third node, the third node being a node other than a root node in a multi-dimensional direct connection topology, the nodes in the multi-dimensional direct connection topology being used for model training, and the device comprising: The second transceiving module is configured to perform the receiving and / or sending related operations in the method of any one of claims 8-16, 22 or 23. The second processing module is configured to perform operations other than the receiving and / or sending related operations in the method of any one of claims 8-16, 22 or 23.

26. A data reduction device, comprising: The device is applied to a seventh node, the seventh node being a node other than a root node in a multi-dimensional direct connection topology, the nodes in the multi-dimensional direct connection topology being used for model training, and the device comprising: The third transceiving module is configured to perform the receiving and / or sending related operations in the method of any one of claims 17-23. The third processing module is configured to perform operations other than the receiving and / or sending related operations in the method of any one of claims 17-23.

27. A data reduction device, comprising: The device comprises a processor coupled with a memory, and the memory stores at least one instruction loaded and executed by the processor, so that the data reduction device implements the data reduction method of any one of claims 1-23.

28. A data reduction system, comprising: The system comprises a root node, a third node and a seventh node, the root node being configured to perform the data reduction method of any one of claims 1-7, 22 or 23, the third node being configured to perform the data reduction method of any one of claims 8-16, 22 or 23, and the seventh node being configured to perform the data reduction method of any one of claims 17-23.

29. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one instruction loaded and executed by the processor to implement the data reduction method of any one of claims 1-23.

30. A computer program product, characterised in that, The computer program product comprises computer programs / instructions executed by the processor to enable the computer to implement the data reduction method of any one of claims 1-23.

Images