Data processing method, switching node, and related system

Abstract

Provided in the present application are a data processing method, a switching node and a related system. The data processing method is applied to a distributed computing system comprising a plurality of computing nodes and at least one first switching node, where each first switching node is connected to at least one computing node among the plurality of the computing nodes, and comprises: after receiving control information of the at least one computing node connected to the first switching node, the first switching node sends to the at least one computing node a data reading request according to the control information; and, after obtaining a computing result obtained by computing the data read by the data reading request, the first switching node sends the computing result to the at least one computing node. In the distributed computing system, when the plurality of the computing nodes need to perform collective communication, the switching node acquires from the computing nodes data needing to be aggregated, aggregates the data, and then sends an aggregation result to the computing nodes, thereby reducing the size of data needing to be transmitted and the synchronization frequency during data aggregation by means of collective communication, and improving computing efficiency.

Description

A data processing method, switching node and related system

[0001] This application claims priority to the Chinese patent applications filed with the State Intellectual Property Office of China on June 27, 2023, with application number "202310776855.2" and application name "A converged computing system", and filed with the State Intellectual Property Office of China on August 14, 2023, with application number "202311023938.0" and application name "A data processing method, switching node and related system", all of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to a data processing method, a switching node, and related systems. Background Art

[0003] With the development of high-performance computing (HPC) and artificial intelligence (AI) technologies, many large-scale applications have emerged, and the scale of data that needs to be processed is also increasing. To solve the problem of computing large amounts of data, distributed computing has emerged. In distributed computing, data exchange occurs between computing nodes. Collective communication has numerous and important applications in distributed computing. For example, collective communication methods such as allreduce and allgather are common methods for enabling data exchange between multiple computing nodes in distributed computing. However, current collective communication methods suffer from problems such as large data transmission volumes and frequent data synchronization, which affect the efficiency of distributed computing.

[0004] Summary of the Invention

[0005] The present application provides a data processing method, a switching node, and a related system, which can reduce the amount of data that needs to be transmitted and the number of synchronization times when a distributed computing system aggregates data through collective communication, thereby improving the computing efficiency of distributed computing.

[0006] In a first aspect, the present application provides a data processing method, which is applied to a distributed computing system including multiple computing nodes and at least one first switching node, wherein each of the at least one first switching node is connected to at least one computing node among the multiple computing nodes. For any first switching node, after receiving first control information sent by at least one computing node to which it is connected, the first switching node sends a data read request to the at least one computing node to which it is connected, obtains a data calculation result based on the data read by the data read request, and then sends the obtained data calculation result to the at least one computing node to which it is connected. The first control information includes a communication task identifier (ID), and the communication task ID is used to indicate the communication task in which the at least one computing node participates.

[0007] In this distributed computing system, when multiple computing nodes are performing distributed computing and collective communication is required, the computing nodes send control information to the switching nodes. The switching nodes that receive the control information actively obtain the data that needs to be aggregated from the computing nodes, aggregate the data in the switching nodes, and then send the aggregated results to the computing nodes. This can reduce the amount of data that needs to be transmitted and the number of synchronization times when the distributed computing system performs data aggregation through collective communication, thereby improving the computing efficiency of distributed computing.

[0008] Furthermore, in the above-described distributed computing process, only after the computing node sends control information to the switching node will the switching node send a data read request, obtain the data to be aggregated, and perform data aggregation on the switching node. After the data aggregation is completed, the aggregated data result is sent to the node that sent the control information. Before and after this process, the switching node can be used to process other tasks. That is, the aggregation task triggered by the control information will not continuously occupy the computing resources of the switching node for a long time without releasing them. In other words, for a service, the service will only occupy the resources of the switching node for data aggregation when collective communication is required. The resources of the switching node are not occupied while the computing node is executing the service. Therefore, the data processing method provided by this application can fully utilize the resources of the switching node when a distributed computing system executes multiple services.

[0009] In one possible implementation, the distributed computing system further includes at least one second switching node, each of the at least one second switching node being connected to some of the at least one first switching node. For a second switching node connected to the first switching node that received the first control information, the first switching node sending a data read request to at least one computing node based on the first control information includes: the first switching node sending second control information to the second switching node based on the first control information, and only after receiving notification information sent by the second switching node, the first switching node sending the data read request to the at least one computing node connected to it, wherein both the second control information and the notification information include the communication task ID, and the notification information is used to instruct the first switching node to obtain data from the at least one computing node connected to it.

[0010] When the distributed computing system includes the above-mentioned at least one second switching node, the data read by the first switching node through the above-mentioned data reading request also needs to be sent to the second switching node connected to it for processing. Therefore, the first switching node also needs to send the second control information obtained based on the above-mentioned first control information to the second switching node. After the second switching node sends a notification message to the first switching node, indicating that the first switching node and the second switching node are able to process the data, the first switching node will obtain the data from the computing node. This can avoid the problem that when the second switching node is unavailable, the data cannot be processed after being read, resulting in wasted bandwidth or calculation errors.

[0011] In one possible implementation, when the distributed computing system includes the above-mentioned second switching node, after the above-mentioned first switching node sends a data reading request to at least one computing node connected to it, the first switching node reads data from the above-mentioned at least one computing node based on the data reading request, and the first switching node calculates the read data to obtain a first calculation result. There are other first switching nodes in the distributed computing system that also perform the same operation to obtain the first calculation result. Multiple first switching nodes send their respective calculated first calculation results to the second switching node, and the second switching node calculates the above-mentioned data calculation result based on the first calculation result. The first switching node obtains the above-mentioned data calculation result from the second switching node.

[0012] Since the computing and storage resources of switching nodes are limited, in a distributed computing node including multiple layers of switching nodes, each layer of switching nodes performs corresponding computing operations based on the acquired data, and can implement the aggregation calculation of large amounts of data through multiple switching nodes, thereby improving the efficiency of data aggregation in the switching nodes.

[0013] In a possible implementation, the communication task is Allreduce, and the data calculation result is the sum, average, and maximum value of the data obtained from the data read request.

[0014] In one possible implementation, the first control information further includes address information, which indicates the location where the first switching node reads data from the computing node that sends the first control information; the data read by the data read request is part or all of the data indicated by the first control information.

[0015] The data read request needs to include the location of the data to be read, so the first control information also includes address information. When generating a data read request, the first switching node needs to determine the location of the data to be read based on the address information. Since the size of the payload carried by a data packet is limited or the node receiving the data packet can only receive a limited payload of the data packet, and in a distributed computing system, the amount of data that needs to participate in collective communication in each computing node is large, the data read by a data read request is generally only the portion of the data indicated by the address information. The first switching node or the second switching node needs to determine the address of the data to be read by each of the multiple data read requests generated based on the location in the address information.

[0016] In one possible implementation, the first control information further includes computing node information, which is used to indicate the number of at least one computing node connected to the first switching node; after the first switching node confirms receipt of the first control information sent by the computing node to which it is connected based on the computing node information, it sends a data read request to the computing node based on the first control information. The at least one computing node connected to the first switching node actually represents a computing node connected to the first switching node and participating in the execution of this business, and the communication task is a task that needs to be executed for this business. Due to the limited storage resources of the switching node, the first switching node sends a data read request to the computing node only after confirming that it has received the first control information sent by all the computing nodes to which it is connected. This can avoid the problem of reading data upon receiving one first control information, resulting in the need to cache the first read data for a long time.

[0017] The computing nodes may be graphics processing units (GPUs), neural network processing units (NPUs), tensor processing units (TPUs), or dedicated AI processing chips; the switching nodes may be switching chips or switching devices with switching functions.

[0018] In a possible implementation, the distributed computing system further includes a host, which generates operator information based on topology information of the distributed computing system and the business to be executed. The operator information includes communication operators and computing operators in the business to be executed.

[0019] In one possible implementation, the method further includes: a host determining, based on topology information, the number of first switching nodes to which each computing node is connected, N; splitting each communication operator into N communication tasks, with each first switching node corresponding to one of the N communication tasks; and then dispatching the communication operator and the N communication tasks to the multiple computing nodes. N is a positive integer. It should be understood that each communication operator may also be split into m*N communication tasks, where m is a positive integer, with each first switching node corresponding to m communication tasks, or each first switching node corresponding to at least one communication task. The communication operator is an operator that performs data aggregation. Execution of the operator enables the switch node to obtain data from the computing node and perform data aggregation. By splitting the communication operator into N communication tasks, each communication task essentially involves a switch node obtaining partial data from a computing node. Each computing node is connected to multiple first switching nodes, and each first switching node is responsible for processing partial data from the connected computing nodes. This reduces the amount of data that each switch node needs to process, fully utilizes the resources of the switching nodes in the distributed computing system, and improves the data processing efficiency of the distributed computing system.

[0020] In a second aspect, the present application provides a switching node, the switching node including a communication control module and a processing module, and applied to a distributed computing system, the distributed computing system including a plurality of computing nodes and at least one switching node, each of the at least one first switching node being connected to at least one computing node of the plurality of computing nodes;

[0021] The above-mentioned communication control module is used to receive first control information sent by at least one computing node connected to the switching node, where the first control information includes a communication task identification ID, which is used to indicate a communication task in which at least one computing node participates; the processing module is used to generate a data read request based on the first control information; the communication control module is also used to send a data read request to at least one computing node connected to the above-mentioned switching node; the processing module is also used to obtain a data calculation result, where the data calculation result is obtained based on the data read from the data read request; the communication control module is also used to send the data calculation result to at least one computing node connected to the switching node.

[0022] In one possible implementation, the above-mentioned distributed computing system also includes a second switching node, which is connected to the above-mentioned switching node; the processing module is further used to generate second control information based on the first control information, and the second control information includes the above-mentioned communication task ID; the communication control module is further used to send the second control information to the second switching node; the processing module is specifically used to: after receiving the notification information sent by the second switching node, generate a data read request based on the first control information; the notification information includes the communication task ID, and the notification information is used to instruct the first switching node to obtain data from at least one computing node to which it is connected.

[0023] In one possible implementation, the processing module is further configured to calculate data read from at least one computing node to obtain a first calculation result; the communication control module is further configured to send the first calculation result to the second switching node; and receive a data calculation result calculated by the second switching node, wherein the data calculation result is a collective calculation result obtained by the second switching node based on the first calculation result and the first calculation results sent by other switching nodes in the at least one switching node.

[0024] In a possible implementation, the communication task is Allreduce, and the data calculation result is the sum, average, and maximum value of the data obtained from the data read request.

[0025] In a possible implementation, the first control information further includes address information, which indicates a location where the first switching node reads data from the computing node that sends the first control information; the data read by the data read request is part or all of the data indicated by the first control information.

[0026] In one possible implementation, the first control information also includes computing node information, which is used to indicate the number of at least one computing node connected to the first switching node; the processing module is further used to generate a data read request after confirming receipt of the first control information of at least one computing node based on the computing node information.

[0027] In one possible implementation, the computing node is a GPU, NPU, TPU or a dedicated AI processing chip, and the switching node is a switching chip or a switching device with switching function.

[0028] In one possible implementation, the communication control module and the processing module are logic circuits in a switching node, wherein the internal micro-architecture of the processing module can be implemented as a ring architecture, a dual-ring architecture, a single centralized control architecture, or a multiple distributed centralized unit architecture.

[0029] In a third aspect, the present application provides a distributed computing system, which includes multiple computing nodes and at least one first switching node, each of the at least one first switching node is connected to at least one computing node among the multiple computing nodes, each computing node is used to send first control information to the first switching node to which it is connected, and each first switching node is used to execute the operations implemented by the first switching node in the first aspect or any possible implementation of the first aspect.

[0030] In one possible implementation, the distributed computing system further includes a host, which is configured to execute the operations implemented by the host in the first aspect or any possible implementation of the first aspect.

[0031] In a fourth aspect, the present application provides a computer-readable storage medium storing instructions, which, when the instructions are executed on a computing device, enable the computing device to execute the method described in the first aspect or any possible implementation of the first aspect.

[0032] In a fifth aspect, the present application provides a computer-readable storage medium storing instructions, which, when executed on a switching node, enables a computing device to execute the method described in the second aspect or any possible implementation of the second aspect.

[0033] In a sixth aspect, the present application provides a computer program product, which, when executed on a computing device, enables the computing device to execute the method implemented by the first switching node in the first aspect or any possible implementation of the first aspect.

[0034] In a seventh aspect, the present application provides a computer program product, which, when running on a device, enables the device to execute the method implemented by the host in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the background technology, the drawings required for use in the embodiments of the present invention or the background technology will be described below.

[0036] FIG1 is a schematic diagram of a computing device cluster provided in an embodiment of the present application;

[0037] FIG2 is a schematic diagram of a distributed computing system provided in an embodiment of the present application;

[0038] FIG3 is a schematic diagram of another distributed computing system provided in an embodiment of the present application;

[0039] FIG4 is a schematic diagram of data partitioning within a computing node provided by an embodiment of the present application;

[0040] FIG5 is an interactive diagram of a data processing method provided in an embodiment of the present application;

[0041] FIG6 is a schematic diagram of a host generating task information provided by an embodiment of the present application;

[0042] FIG7 is a schematic diagram of a network initialization provided in an embodiment of the present application;

[0043] FIG8 is an interactive diagram of another data processing method provided in an embodiment of the present application;

[0044] FIG9 is a schematic diagram of a switching node provided in an embodiment of the present application;

[0045] FIG10 is a schematic diagram of a host provided in an embodiment of the present application;

[0046] FIG11 is a schematic diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] With the development of high-performance computing (HPC) and artificial intelligence (AI) technologies, many large-scale applications, such as AI model training, have emerged. The scale of data required for large-scale applications is also increasing. To address the computational challenges of large-scale data, distributed computing has emerged. Distributed computing processes data across multiple compute nodes and then completes the computation by exchanging data between them. Collective communication (such as Allreduce and Allgather) is a common method for enabling data exchange between multiple compute nodes in distributed computing and has numerous and important applications in distributed computing.

[0048] For example, in AI model training, when the data scale of the training set is large, multiple computing nodes can use data parallelism to perform model training. Each computing node includes the complete AI model that needs to be trained, and the data in the training set is divided into multiple subsets. A computing node trains the model based on one of the subsets. During the model training process of the above multiple computing nodes, after the backpropagation of each iterative process, the gradient data obtained by each computing node needs to be all-reduce to ensure that the model parameters of the models in each computing node remain consistent in the next iteration.

[0049] However, current collective communication suffers from issues such as large data volumes and frequent data synchronization. For example, when implementing Allreduce using ring-Allreduce or halving-doubling Allreduce, the actual amount of data transferred between multiple compute nodes is twice the original data. Therefore, the collective communication methods currently used in distributed computing can hinder distributed computing efficiency.

[0050] The present application provides a distributed computing system and a data processing method applied to the distributed computing system. The distributed computing system includes multiple computing nodes and at least one first switching node, each of which is connected to at least one computing node among the multiple computing nodes. In the distributed computing system, when multiple computing nodes need to perform collective communication during distributed computing, data aggregation can be achieved in the switching node, and the results of the data aggregation can then be sent to the computing nodes. This can reduce the amount of data that needs to be transmitted and the number of data synchronization times when the distributed computing system performs data aggregation through collective communication, thereby improving the computing efficiency of the distributed computing.

[0051] The distributed computing system provided by the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0052] The present application provides a computing device cluster, as shown in FIG1 , which is a schematic diagram of a computing device cluster provided in an embodiment of the present application. The computing device cluster includes one or more computing devices. When the computing device cluster includes multiple computing devices, the multiple computing devices are connected via a network. The network can be a carrier network or a network consisting of optical cables and data transmission equipment, and the present application does not impose specific limitations thereon.

[0053] The above-mentioned computing device cluster can constitute a distributed computing system, and each computing device in the cluster can also independently serve as a distributed computing system, as shown in Figure 2, which is a schematic diagram of a distributed computing system provided in an embodiment of the present application. The distributed computing system includes a host, multiple computing nodes and at least one first switching node. Among them, the multiple computing nodes are connected to the host; each of the at least one first switching node is connected to at least one computing node among the multiple computing nodes. As shown in Figure 2, Figure 2 takes a distributed computing system including 4 computing nodes (C0~C3) and 3 first switching nodes (S0~S2) as an example, and the 4 computing nodes and the 3 first switching nodes are connected in a fully connected manner.

[0054] As shown in Figure 3, Figure 3 is a schematic diagram of another distributed computing system provided by an embodiment of the present application. The distributed computing system includes a host, multiple computing nodes, at least one first switching node, and at least one second switching node. Among them, multiple computing nodes are connected to the host; each first switching node in the at least one first switching node is connected to at least one computing node in the multiple computing nodes; each second switching node in the at least one second switching node is connected to some of the first switching nodes in the at least one first switching node. As shown in Figure 3, Figure 3 takes the example of the distributed computing system including 8 computing nodes (C0~C7), 6 first switching nodes (S0~S5) and 6 second switching nodes (S6~S11). The connection relationship between multiple computing nodes and at least one first switching node and at least one first switching node and at least one second switching node is shown in Figure 3.

[0055] In an embodiment of the present application, the above-mentioned host can be a central processing unit (CPU) of a computing device; the above-mentioned computing node can be a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU) or a dedicated AI processing chip, etc.; the above-mentioned switching node can be a switching chip, or it can be a switching device with switching function such as a switch.

[0056] It should be understood that Figures 2 and 3 illustrate the logical connection relationships between hosts, computing nodes, and switching nodes. The aforementioned multiple computing nodes may be computing nodes located on the same physical device or on different physical devices. When multiple computing nodes are located on different physical devices, these multiple physical devices may be physical devices located in the same cabinet. For example, computing nodes C0-C3 shown in Figure 3 are located on the same server, while computing nodes C4-C7 are located on another server. These two servers are located in the same cabinet and connected via a switching node within the cabinet. When the aforementioned multiple computing nodes are computing nodes located on the same physical device, the switching node may be a switching chip within the physical device or a switching device separate from the physical device.

[0057] It should be understood that the above description in conjunction with FIG. 2 and FIG. 3 is merely an example of the topology of the distributed computing system provided by the present application and should not be construed as a specific limitation. The distributed computing system may also have other topologies. For example, the distributed computing system may include more or fewer computing nodes or switching nodes, or may include more layers of switching nodes. The connection relationship between the computing nodes and the switching nodes, as well as the connection relationship between at least one first switching node and at least one second switching node, is merely an example and should not be construed as a specific limitation. Other connection relationships between computing nodes and switching nodes may also be used, and are not specifically limited in the embodiments of the present application.

[0058] In an embodiment of the present application, when a task needs to be executed through the computing device cluster shown in Figure 1 or the distributed computing system shown in Figure 2 or Figure 3, the host of each computing device in the computing device cluster, after obtaining the pending business of each computing device, executes the corresponding pending business through each computing node of the distributed computing system. For a distributed computing system within a computing device, when multiple computing nodes are executing business, if data aggregation is required, each computing node will send control information to the connected switching node. After receiving the control information sent by the computing node, the switching node obtains the required data from each computing node based on the control information, performs data aggregation, and sends the aggregated data results to each computing node.

[0059] Exemplarily, if the above task is an AI model training task, the business to be executed by each computing device is model training in a data-parallel manner. If the distributed computing system shown in Figure 2 is used for data-parallel model training, the same model will be loaded in each computing node of computing nodes C0~C3, and different computing nodes train the model through different training data. After each computing node completes one iteration of back propagation, the gradient data obtained in each computing node needs to be Allreduce. Since each computing node in Figure 2 is connected to three first switching nodes S0~S2, when Allreduce is performed, the gradient data in each computing node can be divided into three parts, and each first switching node obtains a corresponding part of gradient data from a computing node, and the gradients are aggregated synchronously through the three first switching nodes to make full use of the resources of each first switching node for data aggregation, thereby improving the computing efficiency of the distributed computing system. As shown in Figure 4, Figure 4 is a schematic diagram of data partitioning within a computing node provided in an embodiment of the present application. The gradient data in the calculation node C0 can be divided into three data: A0, B0 and D0. The gradient data in the calculation node C1 can be divided into three data: A1, B1 and D1. The gradient data in the calculation node C2 can be divided into three data: A2, B2 and D2. The gradient data in the calculation node C3 can be divided into three data: A3, B3 and D3.

[0060] Each computing node sends control information to its three connected first switching nodes. Each control information includes a communication task identifier, which indicates a communication task. The control information sent by each computing node to the same first switching node contains the same communication task identifier. After receiving the control information, each first switching node obtains data from the computing node that sent the control information based on the control information. Upon receiving the control information from the four computing nodes, a first switching node obtains corresponding gradient data from the four computing nodes, calculates the obtained gradient data, obtains data calculation results, and sends the data calculation results to each computing node.

[0061] For example, after receiving control information from compute node C0, first switch node S0 sends one or more data read requests to compute node C0 to obtain data A0 from compute node C0. Using the same method, S0 can obtain data A1, data A2, and data A3 from other compute nodes. It then aggregates data A0, A1, A2, and A3 to obtain aggregated data calculation result A0123, and then sends data calculation result A0123 to each compute node. Similarly, switch node S1 can obtain data B0, B1, B2, and B3 from the aforementioned four compute nodes and calculate data calculation result B0123. Switch node S2 can obtain data D0, D1, D2, and D3 from the aforementioned four compute nodes and calculate data calculation result D0123, and then send the obtained data calculation results to the corresponding compute nodes, ultimately ensuring that the gradient data in each compute node is the same.

[0062] Since the model requires multiple rounds of iterative training, the above method can synchronize the gradient data in each computing node before the next iteration. After obtaining new gradient data through backpropagation of the next iterative training, the above method can be used again to aggregate gradient data between the computing nodes and the exchange nodes.

[0063] The above describes the data processing method provided by this application using an example of a distributed computing system performing model training through data parallelism and aggregating gradient data after backpropagation of model training. It should be understood that the above distributed computing system can also be used to aggregate other data in data parallelism, and can also be used in other scenarios that require data aggregation through collective communication, such as scenarios where model training is performed through model parallelism, which will not be described in detail here.

[0064] It should be noted that each time a distributed computing system executes a pending business, not all computing nodes in the distributed computing system necessarily participate in the execution. For example, in the distributed computing system shown in FIG2 , when executing one business, only three computing nodes are used to participate in the execution of the business. When executing another business through the distributed computing system, four computing nodes are used to participate in the execution of the business. When introducing the data processing method provided by this application, for ease of description, all computing nodes shown in the figure are used as an example for the purpose of explanation.

[0065] The following describes in detail a data processing method based on the above distributed computing system in conjunction with the accompanying drawings. As shown in Figure 5 , Figure 5 is an interactive schematic diagram of a data processing method provided by an embodiment of the present application.

[0066] S501. The host obtains topology information of the distributed computing system and generates operator information based on the topology information and the business to be executed.

[0067] In the embodiment of the present application, the data processing method provided by the embodiment of the present application is introduced by taking the distributed computing system shown in Figure 2 as an example. As shown in Figure 6. After the distributed computing system starts a business to be executed, the host can obtain the topology information of the distributed computing system from the management node. The topology information includes the connection relationship between each computing node and each first switching node in the distributed system. The host then generates operator information based on the computing node that executes the business to be executed and the topology information. The above operator information includes computing operators and communication operators. When the computing operator is executed, it is used to implement the computing operation in the business to be executed. When the communication operator is executed, it is used to implement the data sending or receiving operation between the computing node and other nodes.

[0068] Since the process of executing the above-mentioned business to be executed through a distributed computing system involves collective communication between each computing node, that is, the above-mentioned communication operator is used to realize collective communication between each computing node. In the embodiment of the present application, collective communication between each computing node is realized through a switching node. In order to make full use of the resources of the first switching node connected to each computing node, the embodiment of the present application realizes the operation corresponding to a communication operator through multiple first switching nodes connected to each computing node. Therefore, after the host obtains the above-mentioned topology information, the host determines the number N of first switching nodes connected to each computing node based on the above-mentioned topology information, and then splits the operation corresponding to each communication operator into m*N communication tasks, each first switching node corresponds to at least one of the m*N communication tasks, or each first switching node corresponds to m of the m*N communication tasks, and the operation corresponding to the communication operator is realized through the m*N communication tasks corresponding to the N first switching nodes, where m is a positive integer. In the embodiment of the present application, the data processing method provided by the present application is introduced by taking m equal to 1 as an example, that is, the operation corresponding to each communication operator is split into N communication tasks.

[0069] For example, taking the distributed computing system shown in Figure 4 above as an example, the business to be executed is AI model training using data parallelism, and this AI model training is performed by the four computing nodes in Figure 4. After each computing node completes one iteration of backpropagation, the gradient data obtained in each computing node needs to be Allreduceed, that is, the communication operator corresponding to Allreduce is executed. Since each computing node in Figure 4 is connected to three first switching nodes S0-S2, the operation performed by the communication operator corresponding to Allreduce in each computing node can be split into three communication tasks; that is, the gradient data in each computing node is divided into three parts. Each of the three first switching nodes obtains a corresponding portion of gradient data from a computing node; that is, each first switching node corresponds to one of the three communication tasks. The three first switching nodes can obtain the complete gradient data from a computing node, that is, the three communication tasks corresponding to the three first switching nodes implement the operation corresponding to the Allreduce communication operator.

[0070] By splitting the communication operator into N communication tasks, each communication task is essentially a first switching node obtaining part of the data in a computing node. Each computing node is connected to multiple first switching nodes, and each first switching node is used to process part of the data in the connected computing nodes. This can reduce the amount of data that each first switching node needs to process, make full use of the resources of the switching nodes in the distributed computing system, and improve the efficiency of the distributed computing system in processing data.

[0071] In an embodiment of the present application, before S501, the network initialization of the distributed computing system is also included. For example, before the distributed computing system is used to perform business for the first time, the network initialization of the distributed computing system needs to be performed. As shown in Figure 7, Figure 7 is a schematic diagram of a network initialization provided by an embodiment of the present application. The management node can obtain the topology information of the distributed computing system by scanning the network of the distributed computing system, and then complete the routing configuration of the network in the distributed computing system based on the topology information. Among them, the above-mentioned topology information includes the connection relationship between each computing node and each first switching node in the distributed computing system. If the distributed computing system includes multiple layers of switching nodes, for example, in Figure 3, the distributed computing system includes two layers of switching nodes, then the above-mentioned topology information also includes the connection relationship between each switching node between the switching nodes of the two adjacent layers. For example, if a second switching node is connected to multiple first switching nodes through multiple ports, then the topology information includes the corresponding connection relationship between the ports of the second switching node and the first switching node.

[0072] After scanning the network to obtain the above-mentioned topology information, the management node sends this topology information to each switching node. After receiving the topology information, each switching node completes its initial configuration. For example, as shown in Figure 7, in the case where the distributed computing system includes multiple layers of switching nodes, each switching node generates a mapping between logical ports and physical ports based on the topology information and assigns a queue and queue ID to each logical port. A logical port includes one or more physical ports, and the physical ports included in a logical port are connected to a switching node in an adjacent layer. After completing the initial configuration, the connected switching nodes exchange communication resource information. This communication resource information includes a queue ID, which indicates the queue ID used when one switching node communicates with another switching node. For example, in Figure 3, a second switching node sends the queue ID for communicating with a first switching node to the first switching node, and the first switching node similarly sends the queue ID for communicating with the second switching node to the second switching node. It should be noted that the management node can be a management device different from the computing device to which the host belongs, or a software module deployed on the computing device of the host, which is not specifically limited in the embodiments of the present application.

[0073] S502. The host generates first control information corresponding to each computing node.

[0074] In S501 above, the communication operator is split into N communication tasks. Executing the communication operator translates into executing N communication tasks. Each communication task corresponds to a first switching node obtaining a portion of data from a computing node. Therefore, when a computing node executes the communication operator, the computing node is required to send first control information to the connected first switching node, so that the first switching node that receives the first control information obtains data from the computing node that sent the first control information.

[0075] In an embodiment of the present application, after splitting each communication operator into N communication tasks, the host generates corresponding first control information for each communication task. The first control information includes a communication task ID, which is used to uniquely identify a communication task. It should be understood that each communication operator executed by each computing node is converted into N communication tasks, and the purpose of each communication task is to enable the first switching node that receives the first control information to obtain data from the computing node that sent the first control information and perform data aggregation. Therefore, the data obtained by a first switching node based on the first control information sent by different computing nodes should be able to be aggregated. In an embodiment of the present application, the communication task IDs in multiple first control information used by a first switching node to obtain aggregated data from multiple computing nodes are the same, wherein the aggregated data is data that can be aggregated and obtained by a first switching node from different computing nodes. For example, in Figure 4 above, A0, A1, A2, and A3 are aggregated data, and the communication task IDs in the four first control information used by S0 to obtain A0, A1, A2, and A3 are the same.

[0076] S503. The host sends task information corresponding to the business to be executed to each computing node.

[0077] The host converts the business to be executed into computing operators and communication operators that need to be executed by each computing node through the methods in S501 and S502 above, and generates communication tasks corresponding to each communication operator and first control information corresponding to each communication task. Then, the host obtains task information corresponding to the business to be executed, which includes operator information and first control information. Then, the host sends the above-generated operator information and the first control information corresponding to each communication task to the corresponding computing node.

[0078] S504. Each computing node sends first control information to the first connected switching node.

[0079] Taking any computing node in a distributed computing system as an example, such as a first computing node, which is connected to N first switching nodes, when the first computing node executes a communication operator, it sends first control information to each of the N first switching nodes connected to it. For example, in Figure 4 above, computing node C0 sends first control information to first switching nodes S0, S1, and S2; computing node C1 sends first control information to first switching nodes S0, S1, and S2.

[0080] S505. The first switching node sends a data read request to the computing node connected thereto.

[0081] In an embodiment of the present application, the first control information also includes address information, which includes the address of data to be transmitted in a communication task. This address information is used to indicate the location at which the first switching node should read data from the computing node that sent the first control information. After receiving the first control information from a computing node, the first switching node generates one or more data read requests based on the address information in the first control information. When multiple data read requests are generated, each data read request is used to read a portion of the data indicated by the first control information. These one or more data read requests are then sent to the corresponding computing node to read the corresponding data from the computing node. For example, in Figure 2, the first control information sent by C1 to S0 includes the starting address and data length of data A1. S0 generates one or more data read requests based on the starting address, data length, and maximum payload size (MPS) of A1. For example, if the data length of data A1 is 1000 bytes and the MPS is 125 bytes, S0 will generate eight data read requests, each of which reads only a portion of data A1. Each data read request includes a starting address and a read length, and is used to read a segment of data starting from the starting address and having a length equal to the read length. It should be noted that the first switching node may also generate one or more data read requests based on the starting address, data length, and maximum read request size (MRRS), which is not specifically limited in the embodiments of the present application. The first switching node needs to generate one or more read requests based on the first control information sent by each computing node, that is, the one or more read requests generated based on the first control information sent by a computing node are used to obtain the data indicated by the first control information to be read from the computing node. In the case where multiple data read requests need to be generated, one data read request can only read the portion of the data indicated by the first control information to be read.

[0082] In one possible implementation, the first control information also includes computing node information, which indicates the number of computing nodes connected to each first switching node. Before sending a data read request, each first switching node needs to confirm, based on the computing node information, whether it has received the first control information sent by all connected computing nodes. A first switching node will only generate the one or more data read requests after receiving the first control information sent by all connected computing nodes. For example, after receiving a first control message with a communication task ID of 01, the first switching node determines that it is connected to four computing nodes based on the computing node information in the first control message. After receiving four first control messages carrying the communication ID of 01, the first switching node confirms receipt of the first control information sent by all connected computing nodes. Due to limited storage resources of switching nodes, a first switching node only sends a data read request to a computing node after confirming that it has received the first control information from all connected computing nodes. This avoids the problem of reading data immediately after receiving a first control message, which results in the previously read data needing to be cached for a long time. It should be understood that the computing node information actually indicates the number of computing nodes connected to each first switching node and used to execute the current business to be executed. In this application, taking the example that all computing nodes in the distributed computing device participate in executing the business, the number of computing nodes connected to each first switching node and used to execute the current business to be executed is the same as the number of all computing nodes connected to the first switching node.

[0083] Optionally, the computing node information can be a numerical value, that is, a numerical value directly indicating the number of computing nodes connected to each first switching node. The computing node information can also be in other forms, for example, the computing node information includes k bits, where k is the number of ports included in the first switching node, and each bit corresponds to a port of the first switching node. When a port is connected to a computing node, the bit corresponding to the port is set to 1, and the remaining bits are set to 0. After receiving the first control information, the first switching node can determine the number of computing nodes connected to it, that is, the number of computing nodes participating in executing the to-be-executed service, based on the number of bits set to 1 in the computing node information.

[0084] The first switching node can also confirm whether it has received the first control information sent by all the computing nodes connected to it by other means, which is not specifically limited in the embodiments of the present application. For example, the computing node information includes k bits, where k is the number of ports included in the first switching node, and each bit corresponds to a port of the first switching node. When a computing node is connected to a port, the bit position corresponding to the port is set to 1, and the remaining bits are set to 0. After receiving a first control information, the first switching node can determine which ports need to receive the first control information based on the bit position set to 1. After the first switching node receives the first control information from these ports, the first switching node confirms that it has received the first control information sent by all the computing nodes connected to it.

[0085] In an embodiment of the present application, when a distributed computing system is capable of executing multiple services simultaneously, after receiving the first control information sent by any computing node, the first switching node needs to first determine whether its own computing resources are available based on the first control information. Only when the first switching node confirms that its own computing resources are available will the first switching node send a data read request to the computing node to which it is connected. For example, each first switching node is configured to support 3 concurrencies at the same time. If a first switching node S0 is already processing 3 concurrencies when it receives the first control information, and determines that the first control information corresponds to a new concurrent request based on the communication task ID, then S0 determines that its own computing resources are unavailable and returns a message of insufficient resources to the computing node. If S0 is processing 2 concurrencies at the same time when it receives the first control information, and S0 confirms that its own computing resources are available, then S0 sends a data read request to the computing node to which it is connected.

[0086] It should be noted that after receiving the first control information sent by a computing node, the first switching node can confirm whether its own computing resources are available. If a first switching node confirms that its own computing resources are unavailable, it must receive the first control information from all connected computing nodes before returning insufficient resource information to the computing nodes. This avoids the situation where the first switching node confirms that its own computing resources are unavailable upon receiving the first control information from the first computing node, but then determines that its own computing resources are available after receiving the first control information from the remaining computing nodes.

[0087] S506. Each computing node obtains corresponding data according to the received data read request, and sends the obtained data to the first switching node.

[0088] After receiving the data read request, each computing node obtains corresponding data according to the starting address and read length in each data read request, and sends the data to the corresponding first switching node.

[0089] S507. The first switching node calculates a data calculation result based on the data obtained from each computing node, and sends the data calculation result to the computing node connected to it.

[0090] After the first switching node obtains data from each computing node according to the above one or more data read requests, it aggregates the obtained data and then sends the aggregated data to each computing node. For example, if the first switching node can read the data indicated by the first control information through a data read request, such as the first switching node obtains data A0 from C0, obtains data A1 from C1, obtains data A2 from C2, obtains data A3 from C3, and then calculates the data A0, A1, A2, and A3 to obtain data calculation results, and sends the data calculation results to the four computing nodes respectively. For example, if the above communication task is Allreduce, the first switching node sums A0, A1, A2, and A3 and sends them to the four computing nodes. Among them, the first switching node can perform a summation after obtaining the data returned by the two computing nodes, or it can wait until all four data are obtained before summing. This is not specifically limited in the embodiments of the present application.

[0091] The above describes a data processing method when a distributed computing system includes a layer of switching nodes in conjunction with Figure 2. The data processing method provided by the present application can also be applied to a distributed computing system including multiple layers of switching nodes. For example, it is applied to a distributed computing system including two layers of switching nodes as shown in Figure 3, which includes a host, multiple computing nodes, at least one layer of switching nodes, and at least one layer of switching nodes. When the distributed computing system includes the above-mentioned at least one second switching node, the data read by the first switching node through the above-mentioned data read request also needs to be sent to the second switching node connected thereto for processing. As shown in Figure 8, Figure 8 is a schematic diagram of another data processing method provided by an embodiment of the present application. When the distributed computing system includes two layers of switching nodes, the operations performed by the host and computing nodes of the distributed computing system can refer to the above-mentioned S501 to S504. After each computing node sends the first control information to the connected first switching node, it also includes the following S508 to S515.

[0092] S508. The first switching node generates second control information according to the first control information.

[0093] After the first switching node confirms receipt of the first control information sent by all connected computing nodes based on the computing node information, it also needs to generate second control information based on the first control information and send the second control information to the second switching node. The second control information includes the communication task ID in the first control information.

[0094] S509. The second switching node generates notification information according to the received second control information, and sends the notification information to the first switching node.

[0095] The notification information includes the communication task ID, so that the first switching node receiving the notification information confirms the corresponding first control information and the address information and data length in the first control information according to the communication task ID, and then generates the one or more data read requests.

[0096] In one possible implementation, the second switching node must confirm receipt of the second control information sent by all connected first switching nodes before sending the notification information to the first switching node. When the second switching node needs to confirm receipt of the second control information sent by all connected first switching nodes, the first control information also includes first switching node information, which indicates the number of first switching nodes connected to the second switching node. The format of the first switching node information can refer to the computing node information in the first control information and is not further described here.

[0097] In an embodiment of the present application, when a distributed computing system is capable of executing multiple services simultaneously, after receiving the second control information sent by any first switching node, the second switching node needs to first determine whether its own computing resources are available based on the second control information. Only when the second switching node confirms that its own computing resources are available will the second switching node send a notification message to the first switching node to which it is connected. Among them, the method for the second switching node to confirm whether its own computing resources are available can refer to the method for the first switching node to confirm whether its own resources are available, which will not be repeated here. Since the storage resources of the switching node are limited, the second switching node will send a notification message to the first switching node only after confirming that it has received the first control information of all the first switching nodes to which it is connected. This can avoid the problem of notifying the first switching node to read data upon receiving a second control information, resulting in the need to cache the data read first for a long time and occupying storage resources.

[0098] It should be noted that after receiving the second control information sent by a first switching node, the second switching node can confirm whether its own computing resources are available. If a second switching node confirms that its own computing resources are unavailable, it must receive the second control information sent by all first switching nodes to which it is connected before returning a resource shortage message to the first switching node. This avoids the situation where the second switching node confirms that its own computing resources are unavailable upon receiving the second control information sent by the first first switching node, but then determines that its own computing resources are available after receiving the second control information sent by the remaining first switching nodes.

[0099] S510. The first switching node generates the data reading request according to the notification information, and sends the data reading request to the computing node connected thereto.

[0100] The notification information includes the communication task ID. The first switching node that receives the notification information confirms the corresponding first control information, the address information, and the data length in the first control information based on the communication task ID, and then generates the one or more data read requests. The method for the first switching node to generate the data read request can refer to S505 above.

[0101] In one possible implementation, the notification information includes a communication task ID, an offset, and a first data length; the first data length indicates the length of the data currently being read by the first switching node. Upon receiving the notification information, the first switching node confirms the corresponding first control information based on the communication task ID. The first switching node determines the starting address of the data to be read based on address information in the first control information, determines the starting address of the data to be read based on the starting address and the offset, and then generates a data read request based on the starting address and the first data length of the data to be read. It should be understood that the first switching node needs to read data from multiple connected computing nodes. Therefore, each time the first switching node receives a notification information, it generates a data read request based on the received notification information and the first control information sent by each connected computing node, and then sends the data read request to the corresponding computing node. It should be understood that the notification information is equivalent to a read request sent by the second switching node to the first switching node, for each read of data of the first data length from the first switching node. By having the second switching node proactively read data from the first switching node, congestion at the second switching node that could be caused by the first switching node sending data to the second switching node can be avoided.

[0102] For example, in Figure 3 above, if S0 needs to read data from C0, C1, C2, and C3, then S0 generates a data read request corresponding to C0 based on the notification information and the first control information sent by C0, for reading data from C0; S0 generates a data read request corresponding to C1 based on the notification information and the first control information sent by C1, for reading data from C1; S0 generates a data read request corresponding to C2 based on the notification information and the first control information sent by C2, for reading data from C2; and S0 generates a data read request corresponding to C3 based on the notification information and the first control information sent by C3, for reading data from C3. It should be understood that one notification message is used to read part of the data that needs to be read from the computing node, and the second switching node will send multiple notification messages to the first switching node to read all the data that needs to be read from the computing node.

[0103] When the distributed computing system includes the above-mentioned at least one second switching node, the data read by the first switching node through the above-mentioned data reading request also needs to be sent to the second switching node connected to it for processing. Therefore, the first switching node also needs to send the second control information obtained based on the above-mentioned first control information to the second switching node. After the second switching node sends a notification message to the first switching node, indicating that the first switching node and the second switching node are able to process the data, the first switching node will obtain the data from the computing node. This can avoid the problem of wasting bandwidth or causing calculation errors when the resources of the second switching node are unavailable and the first switching node reads the data but the second switching node cannot process the data.

[0104] S511. The first switching node sends a data read request to the computing node connected thereto.

[0105] S512. Each computing node obtains corresponding data according to the received data read request, and sends the obtained data to the first switching node.

[0106] After receiving the data read request, each computing node obtains corresponding data according to the starting address and read length in each data read request, and sends the data to the corresponding first switching node.

[0107] S513. The first switching node calculates a first calculation result based on the data obtained from each computing node, and sends the first calculation result to the second switching node.

[0108] After acquiring data from each computing node, the first switching node performs corresponding calculations to obtain a first calculation result, and then sends the first calculation result to the second switching node.

[0109] S514. The second switching node performs calculations based on the first calculation results sent by all the first switching nodes connected to it, obtains a data calculation result, and sends the data calculation result to the first switching node connected to it.

[0110] S515. The first switching node sends the data calculation result to the respective connected computing nodes.

[0111] The first switching node generates the one or more data read requests based on the notification information and obtains the data instructed to be read by the first control information from the computing node. After receiving the data read in the data read request, the first switching node calculates the data read from the different computing nodes connected to it to obtain a first calculation result. The first calculation result is then sent to the second switching node that sent the notification information. The second switching node then calculates based on the first calculation results sent by all the first switching nodes connected to it to obtain a data calculation result. The second switching node then sends the data calculation result to the first switching node connected to it. The first switching node then sends the data calculation result to the computing nodes connected to it. For example, S0 obtains data A0 from C0, data A1 from C1, data A2 from C2, and data A3 from C3, and then calculates the data A0, A1, A2, and A3 to obtain the intermediate calculation result A0123; S1 obtains data A4 from C4, data A5 from C5, data A6 from C6, and data A7 from C7, and then calculates the data A4, A5, A6, and A7 to obtain the intermediate calculation result A4567, and then sends the two intermediate calculation results to S6 respectively, and S6 calculates the two intermediate calculation results to obtain the above data calculation results.

[0112] Since computing resources and storage resources in the switching nodes are limited, in a distributed computing node including multiple layers of switching nodes, each layer of switching nodes performs corresponding computing operations based on the acquired data, and can realize the aggregation calculation of large amounts of data through multiple switching nodes, thereby improving the efficiency of data aggregation in the distributed computing system.

[0113] In one possible implementation, a first switching node and a second switching node can be connected via one or more physical ports. If the communication protocol between the two switching nodes supports multi-path load balancing, then the communication protocol implements load balancing during data transmission. If the communication protocol between the two switching nodes does not support multi-path load balancing, then the switching nodes implement multi-path load balancing, for example, by sending data through different physical ports in a round-robin manner.

[0114] In one possible implementation, the first control information also includes a first address, which is used to write a flag bit to the position indicated by the first address after the first switching node or the second switching node completes data aggregation to obtain the data calculation result, so as to notify the computing node that the operation corresponding to the communication operator has been completed, or to notify the computing node that the operation corresponding to the communication operator has been completed by sending a message.

[0115] In one possible implementation, the first control information also includes a second address, which is used to write a flag bit to the position indicated by the address after the first switching node and the second switching node confirm that their own computing resources are available, so as to notify the computing node that the computing resources of the switching node are available, or to notify the computing node by sending a message that the operation corresponding to the communication operator has been completed.

[0116] In the embodiments corresponding to Figures 5 and 8 above, the first switching node performs a computing operation on the data read from the multiple computing nodes, and the second switching node performs a computing operation on the data read from the multiple first switching nodes. It should be understood that in other application scenarios, the switching nodes (including the first switching node and the second switching node) in the distributed computing system may not perform computing on the acquired data, and the switching nodes are only used to synchronize data between the multiple computing nodes. The distributed computing system and data processing method provided by the present application can reduce the number of times that multiple computing nodes send data to each other, improve the efficiency of data synchronization, and thus improve computing efficiency. In addition, the first switching node actively reads data from the multiple computing nodes, which can avoid congestion at the first switching node caused by the computing node actively sending data; the second switching node actively reads data from the multiple first switching nodes, which can avoid congestion at the second switching node caused by the multiple first switching nodes actively sending data.

[0117] By implementing the distributed computing system and data processing method provided by the present application, when multiple computing nodes are performing distributed computing and collective communication is required, the computing nodes send control information to the switching nodes. The switching nodes that receive the control information actively obtain the data that needs to be aggregated from the computing nodes, aggregate the data in the switching nodes, and then send the aggregated results to the computing nodes. This can reduce the amount of data that needs to be transmitted and the number of synchronization times when the distributed computing system performs data aggregation through collective communication, thereby improving the computing efficiency of distributed computing.

[0118] Furthermore, in the above-described distributed computing process, only after the computing node sends control information to the switching node will the switching node send a data read request, obtain the data to be aggregated, and perform data aggregation on the switching node. After the data aggregation is completed, the aggregated data result is sent to the node that sent the control information. Before and after this process, the switching node can be used to process other tasks. That is, the aggregation task triggered by the control information will not continuously occupy the computing resources of the switching node for a long time without releasing them. In other words, for a service, the service will only occupy the resources of the switching node for data aggregation when collective communication is required. The resources of the switching node are not occupied while the computing node is executing the service. Therefore, the data processing method provided by this application can fully utilize the resources of the switching node when a distributed computing system executes multiple services.

[0119] For the above method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited to the order of the actions described. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the present invention. Other reasonable step combinations that can be conceived by those skilled in the art based on the above description also fall within the scope of protection of the present invention.

[0120] The present application also provides a data processing system, which includes the distributed computing system shown in FIG. 2 or FIG. 3 . The distributed computing system can refer to the introduction and related descriptions corresponding to FIG. 2 or FIG. 3 , and will not be described in detail here. The operations performed by the host, computing nodes, and switching nodes in the distributed computing system can refer to the relevant descriptions in the embodiments corresponding to FIG. 4 to FIG. 8 , and will not be described in detail here.

[0121] The present application also provides a switching node, as shown in FIG9 . The switching node 900 includes a communication control module 910, a processing module 920, and a switching fabric 930. The switching node is applied to a distributed computing system as shown in FIG2 or FIG3 . The distributed computing system includes multiple computing nodes and at least one switching node 900, each of the at least one switching node 900 being connected to at least one of the multiple computing nodes. The communication control module 910 is configured to receive first control information sent by at least one computing node connected to the switching node 900; the first control information includes a communication task identifier (ID) indicating the communication task in which the at least one computing node participates. The processing module 920 is configured to generate a data read request based on the first control information. The communication control module 910 is further configured to send the data read request to the at least one computing node connected to the switching node 900; the processing module 920 is further configured to obtain a data calculation result, wherein the data calculation result is obtained based on data read from the data read request; and the communication control module 910 is further configured to send the data calculation result to the at least one computing node connected to the first switching node.

[0122] In one possible implementation, the above-mentioned distributed computing system also includes a second switching node, which is connected to the above-mentioned switching node 900; the processing module 920 is also used to generate second control information based on the first control information, and the second control information includes the above-mentioned communication task ID; the communication control module 910 is also used to send the second control information to the second switching node; the processing module 920 is specifically used to: after receiving the notification information sent by the second switching node, generate a data read request according to the first control information; the notification information includes the communication task ID, and the notification information is used to instruct the switching node 900 to obtain data from at least one computing node to which it is connected.

[0123] In one possible implementation, the processing module 920 is further configured to perform calculations on data read from at least one computing node to obtain a first calculation result; the communication control module 910 is further configured to send the first calculation result to the second switching node; and receive a data calculation result calculated by the second switching node, wherein the data calculation result is a calculation result obtained by the second switching node based on the first calculation result and the first calculation results sent by other switching nodes in the at least one switching node 900.

[0124] In one possible implementation, the communication control module 910 and the processing module 920 are logic circuits in a switching node. The internal micro-architecture of the processing module 920 can be implemented as a ring architecture, a dual-ring architecture, a single centralized control architecture, or a multiple distributed centralized unit architecture.

[0125] The switching architecture 930 is used to implement data exchange between different ports of a switching node. For example, data received from a first port may be sent out through a second port.

[0126] The operations that the switching node 900 can perform can refer to the operations performed by the first switching node or the second switching node in the embodiments corresponding to Figures 4 to 8 . Specifically, the communication control module 910 is used to implement operations such as receiving or sending data performed by the first switching node or the second switching node, and the processing module 920 is used to implement operations other than the aforementioned receiving or sending data, which are not further described here.

[0127] An embodiment of the present application also provides a host, as shown in Figure 10, which is a schematic diagram of a host provided by an embodiment of the present application. The host 100 includes a processing module 101 and a communication module 102. The host 100 is applied to a distributed computing system as shown in Figure 2 or Figure 3, which includes the host 100, multiple computing nodes and at least one first switching node, and each of the at least one first switching node is connected to at least one computing node in the multiple computing nodes. Among them, the processing module 101 can be an AI training framework such as MindSpore or TensorFlow.

[0128] When the distributed computing system needs to process a pending service, the processing module 101 is configured to obtain and analyze topology information of the distributed system. For example, the topology information may be used to determine the number of first switching nodes connected to each computing node. The topology information includes the connection relationships between each computing node and each first switching node in the distributed computing system. If the distributed computing system includes at least one second switching node as shown in FIG. 3 , the topology information also includes the connection relationships between the at least one first switching node and the at least one second switching node.

[0129] The processing module 101 generates operator information based on the topology information of the distributed computing system and the business to be executed. The operator information includes computing operators and communication operators. The processing module 101 is also used to convert the above-mentioned communication operators into multiple communication tasks based on the topology information, configure a communication task ID for each communication task, and generate first control information corresponding to each communication task based on the communication task ID. Specifically, the operations implemented by the above-mentioned processing module 101 can refer to the operations implemented by the host in the embodiments corresponding to Figures 5-8 above, and will not be repeated here.

[0130] The communication module 102 is used to send task information to each computing node in the distributed computing system. The task information includes the computing operators and communication operators that the computing nodes need to execute, the communication tasks corresponding to each communication operator, and the first control information corresponding to each communication task.

[0131] In a possible implementation, the host 100 further includes a management module 103, which is configured to scan the network of the distributed computing system, obtain topology information of the distributed system, and complete routing configuration of the network in the distributed computing system based on the topology information.

[0132] The communication module 102 is further configured to send the topology information obtained by the management module 103 to each switching node so that each switching node can complete configuration, such as generating a mapping relationship between logical ports and physical ports and allocating a queue and queue ID to each logical port.

[0133] The operations implemented by the management module 103 may refer to the operations implemented by the management node in the embodiment shown in FIG. 5 , and will not be described in detail here.

[0134] Specifically, the operations implemented by the host 100 may refer to the operations implemented by the host in the embodiments corresponding to FIG. 5 to FIG. 8 , and will not be described in detail here.

[0135] The present application also provides a computing device, as shown in FIG11 , which is a schematic diagram of a computing device provided by the present application. The computing device 110 includes a host 111, multiple computing nodes 112, a communication interface 113, and a memory 114. The host 111, multiple computing nodes 112, the communication interface 113, and the memory 114 are interconnected via a bus 115.

[0136] The host 111 may be a CPU, and the computing node 112 may be a GPU, NPU, TPU, or dedicated AI processing chip. The operations performed by the host 111 may refer to the operations performed by the host in the embodiments corresponding to Figures 4 to 8 above, and the operations performed by the computing node 112 may refer to the operations performed by the computing node in the embodiments corresponding to Figures 4 to 8 above, and will not be repeated here.

[0137] The communication interface 113 may be a wired interface or a wireless interface for communicating with other modules or devices. The wired interface may be an Ethernet interface, a local interconnect network (LIN), etc. The wireless interface may be a cellular network interface or a wireless local area network interface.

[0138] Memory 114 may be a non-volatile memory, such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Memory 114 may also be a volatile memory, which may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0139] The memory 114 can also be used to store program instructions and data, so that the host 111 can call the program instructions stored in the memory 114 to execute the host operation steps in the above method embodiment. In addition, the computing device 110 may include more or fewer components than shown in Figure 5, or have different component configurations.

[0140] Bus 115 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. Bus 115 may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, FIG11 shows only one thick line, but this does not mean that there is only one bus or only one type of bus.

[0141] The computing device 110 may further include the at least one switching chip 116. Operations performed by the switching chip 116 may refer to the operations performed by the first switching node or the second switching node in the above embodiment, and are not described in detail here.

[0142] Specifically, the specific implementation of various operations performed by the computing device 110 may refer to the specific operations performed by the distributed computing system in the above method embodiment, which will not be described in detail here.

[0143] An embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a processor, it can implement the method steps implemented by the evaluation device in the above method embodiment. The specific implementation of the processor of the computer-readable storage medium in executing the above method steps can refer to the specific operations of the above method embodiment, which will not be repeated here.

[0144] An embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a processor, it can implement the method steps implemented by the management device in the above method embodiment. The specific implementation of the processor of the computer-readable storage medium in executing the above method steps can refer to the specific operations of the above method embodiment, which will not be repeated here.

[0145] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0146] The above embodiments can be implemented in whole or in part through software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium, or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0147] The steps in the method of the embodiment of the present application can be adjusted in order, combined or deleted according to actual needs; the modules in the device of the embodiment of the present application can be divided, combined or deleted according to actual needs.

[0148] The above is a detailed introduction to the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0149] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, characterized in that: Applied to a distributed computing system, the distributed computing system includes a plurality of computing nodes and at least one first switching node, each of the at least one first switching node is connected to at least one computing node of the plurality of computing nodes, the method includes: The first switching node receives first control information of at least one computing node connected to it, wherein the first control information includes a communication task identification ID, and the communication task ID is used to indicate a communication task in which the at least one computing node participates; The first switching node sends a data read request to the at least one computing node according to the first control information; The first switching node obtains a data calculation result, wherein the data calculation result is obtained according to the data read by the data read request; The first switching node sends the data calculation result to the at least one computing node.

2. The method according to claim 1, characterized in that The distributed computing system further includes a second switching node, the second switching node being connected to the first switching node; The first switching node sends a data read request to the at least one computing node according to the first control information, including: The first switching node sends second control information to the second switching node according to the first control information, wherein the second control information includes the communication task ID; After receiving the notification information sent by the second switching node, the first switching node sends the data reading request to the at least one computing node; the notification information includes the communication task ID, and the notification information is used to instruct the first switching node to obtain data from the at least one computing node.

3. The method according to claim 2, characterized in that The first switching node obtains a data calculation result, including: The first switching node calculates the data read from the at least one computing node to obtain a first calculation result; The first switching node sends the first calculation result to the second switching node; The first switching node obtains the data calculation result calculated by the second switching node, wherein the data calculation result is a collective calculation result obtained by the second switching node according to the first calculation result and calculation results sent by other switching nodes in the at least one first switching node.

4. The method according to any one of claims 1 to 3, characterized in that: The communication task is Allreduce, and the data calculation result is the sum, average, and maximum value result of the data obtained from the data read request.

5. The method according to any one of claims 1 to 4, characterized in that: The first control information further includes address information, wherein the address information indicates a location where the first switching node reads data from the computing node that sends the first control information; The data read by the data read request is part or all of the data instructed to be read by the first control information.

6. The method according to any one of claims 1 to 5, characterized in that: The first control information further includes computing node information, where the computing node information is used to indicate the number of the at least one computing node; The first switching node sends a data read request to the at least one computing node according to the first control information, including: After the first switching node confirms receipt of the first control information of the at least one computing node according to the computing node information, the first switching node sends a data read request to the first number of computing nodes according to the first control information.

7. The method according to any one of claims 1 to 6, characterized in that: The computing node is a graphics processor GPU, a neural network processor NPU, a tensor processor TPU or a dedicated artificial intelligence AI processing chip, and the first switching node is a switching chip or a switching device with a switching function.

8. The method according to any one of claims 1 to 7, characterized in that: The distributed computing system further includes a host, and the method further includes: The host generates operator information according to the topology information of the distributed computing system and the business to be executed, and the operator information includes the communication operator and the calculation operator in the business to be executed.

9. The method according to claim 8, characterized in that The method further comprises: The host determines, according to the topology information, that the number of first switching nodes connected to each computing node is N, where N is a positive integer; The host splits the communication operator into N communication tasks, and each first switching node corresponds to one communication task among the N communication tasks.

10. The method according to claim 8 or 9, characterized in that: The host sends the communication operator and N communication tasks to the multiple computing nodes.

11. A switching node, characterized in that: The invention comprises a communication control module and a processing module, wherein the switching node is located in a distributed computing system, the distributed computing system comprises a plurality of computing nodes and at least one switching node, and the switching node is connected to at least one computing node among the plurality of computing nodes; The communication control module is used to receive first control information sent by at least one computing node connected to the switching node, wherein the first control information includes a communication task identification ID, and the communication task ID is used to indicate a communication task in which the at least one computing node participates; The processing module is used to generate a data read request according to the first control information; The communication control module is further used to send a data read request to the at least one computing node; The processing module is further used to obtain a data calculation result, wherein the data calculation result is obtained according to the data read by the data reading request; The communication control module is further used to send the data calculation result to the at least one computing node.

12. The switching node according to claim 11, characterized in that: The distributed computing system further includes a second switching node connected to the switching node; The processing module is further configured to generate second control information according to the first control information, wherein the second control information includes the communication task ID; The communication control module is further used to send second control information to the second switching node; The processing module is specifically used to: after receiving the notification information sent by the second switching node, generate the data reading request according to the first control information; the notification information includes the communication task ID, and the notification information is used to instruct the switching node to obtain data from the at least one computing node.

13. The switching node according to claim 12, characterized in that: The processing module is further used to calculate the data read from the at least one computing node to obtain a first calculation result; The communication control module is further configured to send the first calculation result to the second switching node; The data calculation result calculated by the second switching node is received, wherein the data calculation result is a collective calculation result obtained by the second switching node according to the first calculation result and calculation results sent by other switching nodes in the distributed computing system.

14. The switching node according to any one of claims 11 to 13, characterized in that: The communication task is Allreduce, and the data calculation result is the sum, average, and maximum value result of the data obtained from the data read request.

15. The switching node according to any one of claims 11 to 14, wherein the first control information further comprises address information, wherein the address information indicates a location where the first switching node reads data from the computing node that sends the first control information; The data read by the data read request is part or all of the data instructed to be read by the first control information.

16. The switching node according to any one of claims 11 to 15, characterized in that: The first control information further includes computing node information, where the computing node information is used to indicate the number of the at least one computing node; The processing module is further configured to generate the data read request after confirming receipt of the first control information of the at least one computing node according to the computing node information.

17. The switching node according to any one of claims 11 to 16, characterized in that: The computing node is a graphics processor GPU, a neural network processor NPU, a tensor processor TPU or a dedicated artificial intelligence AI processing chip, and the switching node is a switching chip or a switching device with a switching function.

18. The switching node according to any one of claims 11 to 17, characterized in that: The communication control module and the processing module are logic circuits in the switching chip.

19. A distributed computing system, characterized in that: It includes multiple computing nodes and at least one first switching node, each of the at least one first switching node is connected to at least one computing node of the multiple computing nodes, each computing node is used to send first control information to the first switching node it is connected to, and each first switching node is used to execute the method described in the preceding claims 1-7.

20. The distributed computing system according to claim 19, characterized in that: It also includes a host, which is used to execute the method described in claims 8-10.

Images