A data processing method, switching node and related system

Abstract

The application provides a data processing method, an exchange node and a related system, and is applied to a distributed computing system comprising a plurality of computing nodes and at least one first exchange node, wherein each first exchange node is connected with at least one computing node in the plurality of computing nodes, the first exchange node sends a data reading request to the at least one computing node connected therewith according to control information of the at least one computing node after receiving the control information, and sends a calculation result obtained according to data read by the data reading request to the at least one computing node. In the distributed computing system, when the plurality of computing nodes need to perform collective communication, the exchange node obtains data needing to be aggregated from the computing nodes and performs data aggregation, and then sends the aggregated result to the computing nodes, so that the amount of data needing to be transmitted and the number of synchronizations when the data is aggregated through the collective communication can be reduced, and the computing efficiency is improved.

Description

[0001] This application is a divisional application, the original application number is 202311023938.0, the original application date is August 14, 2023, and the entire contents of the original application are incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of data processing, in particular to a data processing method, an exchange node and a related system. BACKGROUND

[0003] With the development of high performance computing (HPC) and artificial intelligence (AI) technologies, many large-scale applications have emerged, and the scale of data to be processed is also increasing. In order to solve the problem of large-scale data computing, distributed computing has emerged. In distributed computing, data interaction between various computing nodes is involved. Collective communication has a large number of important applications in distributed computing, such as allreduce, allgather and other collective communication methods, which are commonly used methods for data interaction between multiple computing nodes in distributed computing. However, the current collective communication method has problems such as large amount of data transmission and multiple data synchronization, which affects the efficiency of distributed computing. SUMMARY

[0004] The present application provides a data processing method, an exchange node and a related system, which can reduce the amount of data to be transmitted and the number of synchronizations when the distributed computing system aggregates data through collective communication, and improve the computing efficiency of distributed computing.

[0005] In a first aspect, the present application provides a data processing method applied to a distributed computing system including a plurality of computing nodes and at least one first exchange node, each of the at least one first exchange node being connected to at least one computing node of the plurality of computing nodes. For any one first exchange node, after receiving first control information sent by the at least one computing node connected thereto, the first exchange node sends a data reading request to the at least one computing node connected thereto, obtains a data calculation result based on the data read by the data reading request, and then sends the obtained data calculation result to the at least one computing node connected thereto. The first control information includes a communication task identity (ID), and the communication task ID is used to indicate a communication task in which the at least one computing node participates.

[0006] In the distributed computing system, in the process of performing distributed computing, when collective communication is needed, the computing nodes send control information to the exchange nodes, the exchange nodes receiving the control information actively acquire data needed to be aggregated from the computing nodes, data aggregation is performed in the exchange nodes, and the aggregated results are sent to the computing nodes, thereby reducing the amount of data to be transmitted and the number of synchronizations when the distributed computing system performs data aggregation through collective communication, and improving the computing efficiency of the distributed computing.

[0007] In addition, in the above distributed computing process, after the computing nodes send control information to the exchange nodes, the exchange nodes send data read requests to acquire data needed to be aggregated and perform data aggregation in the exchange nodes, and after the data aggregation is completed, the aggregated results are sent to the nodes sending the control information. The exchange nodes can be used to process other tasks before and after the process, that is, the aggregation task triggered by the control information does not exist for a long time to continuously occupy the computing resources of the exchange nodes without releasing. That is, in a service, only when collective communication is needed, the service occupies the resources of the exchange nodes for data aggregation, and the computing nodes do not occupy the resources of the exchange nodes in the process of executing the service. Therefore, by using the data processing method provided in the present application, in the case that the distributed computing system executes multiple services, the resources of the exchange nodes can be fully utilized.

[0008] In a possible implementation, the distributed computing system further includes at least one second exchange node, and each of the at least one second exchange node is connected with part of the first exchange nodes. For the second exchange node connected with the first exchange node receiving the first control information, the first exchange node sends a data read request to at least one computing node according to the first control information includes that the first exchange node sends second control information to the second exchange node according to the first control information, and after receiving the notification information sent by the second exchange node, the first exchange node sends a data read request to at least one computing node connected therewith, wherein the second control information and the notification information both include the communication task ID, and the notification information is used to instruct the first exchange node to acquire data from at least one computing node connected therewith.

[0009] When the distributed computing system includes at least one second exchange node, the data read by the first exchange node through the data read request also needs to be sent to the second exchange node connected thereto for processing, and therefore the first exchange node also needs to send the second control information obtained based on the first control information to the second exchange node. After the second exchange node sends the notification information to the first exchange node, it indicates that the first exchange node and the second exchange node can process data, and the first exchange node will obtain data from the computing node, which can avoid the problem that the data cannot be processed after being read in the case that the second exchange node is unavailable, resulting in waste of bandwidth or calculation error.

[0010] In a possible implementation, when the distributed computing system includes the second exchange node, the first exchange node sends a data read request to at least one computing node connected thereto, reads data from the at least one computing node based on the data read request, and obtains first calculation results by performing calculation on the read data. Other first exchange nodes in the distributed computing system also perform the same operation to obtain first calculation results. The first exchange nodes send the first calculation results obtained by the respective calculation to the second exchange node. The second exchange node calculates the data calculation result based on the first calculation results. The first exchange node obtains the data calculation result from the second exchange node.

[0011] Since the computing resources and storage resources of the exchange nodes are limited, in the distributed computing system including multiple layers of exchange nodes, each layer of exchange node performs corresponding calculation operation based on the obtained data, which can implement aggregation calculation of a large amount of data through multiple exchange nodes, and improve the efficiency of data aggregation implemented by the exchange nodes.

[0012] In a possible implementation, the communication task is Allreduce, and the data calculation result is a sum, average, or maximum value result of the data obtained through the data read request.

[0013] In a possible implementation, the first control information further includes address information, and the address information indicates a position of the data read by the first exchange node from the computing node sending the first control information. The data read by the data read request is part or all of the data indicated by the first control information.

[0014] The data read request needs to include the position of the data to be read, and therefore the first control information also includes address information. When generating the data read request, the first switching node needs to determine the position of the data to be read according to the address information. Since the size of the payload carried by a data packet is limited or the payload of the data packet that can be received by the node is limited, and in the distributed computing system, the amount of data that needs to participate in the collective communication in each computing node is large, therefore the data read by one data read request is generally only part of the data indicated by the address information. The first switching node or the second switching node needs to determine the address of the data read by each data read request in the generated multiple data read requests according to the position in the address information.

[0015] In a possible implementation, the first control information further includes computing node information, which is used to indicate the number of at least one computing node connected to the first switching node; after the first switching node confirms that the first control information sent by the computing node connected thereto is received, the first switching node sends a data read request to the computing node according to the first control information. The at least one computing node connected to the first switching node actually represents the computing node connected to the first switching node and participating in the execution of the current service, and the communication task belongs to the task to be executed in the current service. Since the storage resource of the switching node is limited, the first switching node sends the data read request to the computing node after determining that the first control information sent by all the computing nodes connected thereto is received, which can avoid reading data as soon as a first control information is received, thereby avoiding the problem that the data read first needs to be cached for a long time.

[0016] The computing node can be a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), or a special AI processing chip, etc. The switching node can be a switching chip or a switching device with switching function.

[0017] In a possible implementation, the distributed computing system further includes a host, which generates operator information according to the topology information of the distributed computing system and the service to be executed. The operator information includes a communication operator and a computing operator in the service to be executed.

[0018] In a possible implementation, the method further includes: determining, by the host, that the number of first switching nodes connected to each computing node is N according to the topology information, splitting each communication operator into N communication tasks, each first switching node corresponding to one of the N communication tasks, and then distributing the communication operator and the N communication tasks to the plurality of computing nodes. N is a positive integer. It should be understood that each communication operator can also be split into m*N communication tasks, where m is a positive integer, each first switching node corresponding to m communication tasks, or each first switching node corresponding to at least one communication task. The communication operator is an operator for performing data aggregation, and execution of the operator enables the switching node to obtain data from the computing node and perform data aggregation. By splitting the communication operator into N communication tasks, each communication task essentially involves one switching node obtaining part of the data in one computing node. Each computing node is connected to a plurality of first switching nodes, and each first switching node is used to process part of the data in the connected computing node, thereby reducing the amount of data that each switching node needs to process, fully utilizing the resources of the switching nodes in the distributed computing system, and improving the efficiency of the distributed computing system in processing data.

[0019] In a second aspect, the present application provides a switching node, which includes a communication control module and a processing module, and is applied to a distributed computing system including a plurality of computing nodes and at least one switching node. Each switching node in the at least one first switching node is connected to at least one computing node in the plurality of computing nodes.

[0020] The communication control module is configured to receive first control information sent by at least one computing node connected to the switching node, the first control information including a communication task ID, the communication task ID being used to indicate a communication task in which the at least one computing node participates. The processing module is configured to generate a data reading request according to the first control information. The communication control module is further configured to send the data reading request to the at least one computing node connected to the switching node. The processing module is further configured to obtain a data calculation result, where the data calculation result is obtained according to data read by the data reading request. The communication control module is further configured to send the data calculation result to the at least one computing node connected to the switching node.

[0021] In a possible implementation, the distributed computing system further includes a second switching node connected with the switching node; the processing module is further configured to generate second control information according to the first control information, the second control information including the communication task ID; the communication control module is further configured to send the second control information to the second switching node; and the processing module is specifically configured to: after receiving notification information sent by the second switching node, generate a data reading request according to the first control information; the notification information includes the communication task ID, and the notification information is used to instruct the first switching node to obtain data from at least one computing node connected with the first switching node.

[0022] In a possible implementation, the processing module is further configured to perform calculation on the data read from the at least one computing node to obtain a first calculation result; the communication control module is further configured to send the first calculation result to the second switching node; and the data calculation result is obtained by the second switching node according to the first calculation result and a set calculation result of the first calculation result sent by other switching nodes in the at least one switching node.

[0023] In a possible implementation, the communication task is Allreduce, and the data calculation result is a sum, average or maximum value result of the data obtained by the data reading request.

[0024] In a possible implementation, the first control information further includes address information, the address information indicating a position of the data read by the first switching node from the computing node sending the first control information; and the data read by the data reading request is part or all of the data indicated by the first control information.

[0025] In a possible implementation, the first control information further includes computing node information, the computing node information being used to indicate a number of the at least one computing node connected with the first switching node; and the processing module is further configured to generate the data reading request after confirming that the first control information of the at least one computing node is received according to the computing node information.

[0026] In a possible implementation, the computing node is a GPU, NPU, TPU or special AI processing chip, and the switching node is a switching chip or a switching device with a switching function.

[0027] In a possible implementation, the communication control module and the processing module are logical circuits in the switching node. The internal micro-architecture of the processing module can be implemented as a ring architecture, a double ring architecture, a single centralized control architecture or a plurality of distributed centralized unit architectures.

[0028] In a third aspect, the present application provides a distributed computing system, comprising a plurality of computing nodes and at least one first switching node, each of the at least one first switching node being connected with at least one of the plurality of computing nodes, each computing node being configured to send first control information to the first switching node connected therewith, and each first switching node being configured to perform the operations of the first switching node as described in the first aspect or any possible implementation of the first aspect.

[0029] In a possible implementation, the distributed computing system further comprises a host, the host being configured to perform the operations of the host as described in the first aspect or any possible implementation of the first aspect.

[0030] In a fourth aspect, the present application provides a computer-readable storage medium having instructions stored therein, which when executed on a computing device, cause the computing device to perform the method described in the first aspect or any possible implementation of the first aspect.

[0031] In a fifth aspect, the present application provides a computer-readable storage medium having instructions stored therein, which when executed on a switching node, cause the computing device to perform the method described in the second aspect or any possible implementation of the second aspect.

[0032] In a sixth aspect, the present application provides a computer program product, which when executed on a computing device, cause the computing device to perform the method of the first switching node as described in the first aspect or any possible implementation of the first aspect.

[0033] In a seventh aspect, the present application provides a computer program product, which when executed on a computing device, cause the computing device to perform the method of the host as described in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background art, the drawings needed to be used in the embodiments of the present application or the background art will be described below.

[0035] Figure 1 is a schematic diagram of a computing device cluster provided by an embodiment of the present application;

[0036] Figure 2 is a schematic diagram of a distributed computing system provided by an embodiment of the present application;

[0037] Figure 3 is a schematic diagram of another distributed computing system provided by an embodiment of the present application;

[0038] Figure 4 is a data partitioning schematic diagram in a computing node provided by an embodiment of the present application;

[0039] Figure 5 is an interaction schematic diagram of a data processing method provided by an embodiment of the present application;

[0040] Figure 6 is a schematic diagram of task information generated by a host provided by an embodiment of the present application;

[0041] Figure 7 is a schematic diagram of network initialization provided by an embodiment of the present application;

[0042] Figure 8 is an interaction schematic diagram of another data processing method provided by an embodiment of the present application;

[0043] Figure 9 is a schematic diagram of a switching node provided by an embodiment of the present application;

[0044] Figure 10 is a schematic diagram of a host provided by an embodiment of the present application;

[0045] Figure 11 is a schematic diagram of a computing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0046] With the development of high performance computing (HPC) and artificial intelligence (AI) technologies, there are many large-scale applications, such as AI model training. The scale of data to be processed by large-scale applications is also increasing. In order to solve the computing problem of large-scale data, distributed computing emerges as the times require. Distributed computing processes data through multiple computing nodes, and then completes the computation through the interaction of data between multiple computing nodes. Collective communication (such as Allreduce, Allgather, etc.) is a common method for data interaction between multiple computing nodes in distributed computing, and has a large number of and important applications in distributed computing.

[0047] For example, in the AI model training, when the data scale of the training set is large, multiple computing nodes can perform model training in a data parallel manner, each computing node including a complete AI model to be trained, and the data in the training set is divided into multiple subsets, and one computing node trains the model based on one subset. During the above-mentioned model training of multiple computing nodes, after each iteration process of back propagation, the gradient data obtained by each computing node needs to be Allreduced to ensure that the model parameters in each computing node remain consistent in the next iteration.

[0048] However, the current collective communication has problems such as large amount of transmission data and multiple data synchronization times. For example, when implementing Allreduce through ring-Allreduce or halving-doubling Allreduce, the actual amount of data transmitted between multiple computing nodes is twice the original data. Therefore, the collective communication method currently used in distributed computing affects the efficiency of distributed computing.

[0049] The present application provides a distributed computing system and a data processing method applied to the distributed computing system. The distributed computing system includes multiple computing nodes and at least one first exchange node, each first exchange node being connected with at least one computing node in the multiple computing nodes. In the distributed computing system, when the multiple computing nodes need to perform collective communication during distributed computing, data aggregation can be realized in the exchange node, and then the result of data aggregation is sent to the computing node, thereby reducing the amount of data to be transmitted and the number of data synchronization times when the distributed computing system performs data aggregation through collective communication, and improving the computing efficiency of distributed computing.

[0050] The distributed computing system provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0051] The present application provides a computing device cluster, as shown in Figure 1 Figure 1 is a schematic diagram of a computing device cluster provided by an embodiment of the present application. The computing device cluster includes one or more computing devices. When the computing device cluster includes multiple computing devices, the multiple computing devices are connected through a network. The network can be an operator network or a network composed of optical cables and data transmission devices, and the present application does not make specific limitations.

[0052] The above-mentioned computing device cluster can constitute a distributed computing system, and each computing device in the cluster can also be independently used as a distributed computing system, as shown in Figure 2 Figure 2 ​​is a schematic diagram of a distributed computing system provided by an embodiment of the present application. The distributed computing system includes a host, a plurality of computing nodes, and at least one first switch node. Each of the plurality of computing nodes is connected to the host, and each of the at least one first switch node is connected to at least one of the plurality of computing nodes. As shown in Figure 2 , Figure 2 , the distributed computing system includes four computing nodes (C0-C3) and three first switch nodes (S0-S2), and the four computing nodes and the three first switch nodes are connected in a full connection manner.

[0053] As shown in Figure 3 , Figure 3 is a schematic diagram of another distributed computing system provided by an embodiment of the present application. The distributed computing system includes a host, a plurality of computing nodes, at least one first switch node, and at least one second switch node. Each of the plurality of computing nodes is connected to the host, each of the at least one first switch node is connected to at least one of the plurality of computing nodes, and each of the at least one second switch node is connected to part of the at least one first switch node. As shown in Figure 3 , Figure 3 , the distributed computing system includes eight computing nodes (C0-C7), six first switch nodes (S0-S5), and six second switch nodes (S6-S11), and the connection relationship between the plurality of computing nodes and the at least one first switch node, and between the at least one first switch node and the at least one second switch node can be referred to Figure 3 .

[0054] In the embodiments of the present application, the host can be a central processing unit (CPU) of a computing device, the computing node can be a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), or a dedicated AI processing chip, and the switch node can be a switch chip or a switch device with a switching function such as a switch.

[0055] It should be understood that the above Figure 2 and Figure 3The logical connection relationship between the host, the computing node and the switching node is shown. The multiple computing nodes can be computing nodes located in the same physical device, or can be computing nodes located in different physical devices. When the multiple computing nodes are located in multiple different physical devices, the multiple physical devices can be physical devices located in the same rack, for example, the computing nodes C0-C3 shown in the above Figure 3 are located in the same server, and the computing nodes C4-C7 are located in another server, and the two servers are located in the same rack and are connected through the switching nodes in the rack. When the multiple computing nodes are computing nodes located in the same physical device, the switching node can be a switching chip in the physical device, or can be a switching device different from the physical device.

[0056] It should be understood that the above Figure 2 and Figure 3 introduced are only examples of the topology of the distributed computing system provided by the present application, and cannot be understood as specific limitations. The distributed computing system can also have other topologies. For example, the distributed computing system can also include more or fewer computing nodes or switching nodes, or include more layers of switching nodes. The connection relationship between the computing nodes and the switching nodes, and the connection relationship between the at least one first switching node and the at least one second switching node are only examples and cannot be understood as specific limitations. The computing nodes and the switching nodes can also have other connection relationships, and the embodiments of the present application do not make specific limitations.

[0057] In the embodiments of the present application, when a task needs to be executed by the Figure 1 shown computing device cluster, Figure 2 or Figure 3 the distributed computing system shown, after the host of each computing device in the computing device cluster obtains the to-be-executed business of each computing device, each computing node of the distributed computing system executes the to-be-executed business corresponding thereto. For the distributed computing system in a computing device, in the process of executing the business, when data aggregation is needed, each computing node sends control information to the connected switching node. After receiving the control information sent by the computing node, the switching node obtains the required data from each computing node based on the control information to perform data aggregation, and sends the result of data aggregation to each computing node.

[0058] For example, if the above task is an AI model training task, the to-be-executed business obtained by each computing device is to perform model training in a data parallel manner. If the Figure 2The illustrated distributed computing system performs data-parallel model training, and the same model is loaded in each of the computing nodes C0-C3. Different computing nodes train the model using different training data. After each computing node completes one iteration of back propagation, the gradient data obtained in each computing node needs to be subjected to Allreduce. Figure 2 In the illustrated example, each computing node is connected to three first switching nodes S0-S2. In the case of Allreduce, the gradient data in each computing node can be divided into three parts. Each first switching node obtains one part of the gradient data from one computing node, and the aggregation of the gradient data is synchronized through the three first switching nodes, so as to fully utilize the resources of the first switching nodes for data aggregation and improve the computing efficiency of the distributed computing system. As shown in the example of FIG. 3, Figure 4 Figure 4 is a schematic diagram of data division in a computing node provided by an embodiment of the present application. The gradient data in the computing node C0 can be divided into three parts A0, B0, and D0. The gradient data in the computing node C1 can be divided into three parts A1, B1, and D1. The gradient data in the computing node C2 can be divided into three parts A2, B2, and D2. The gradient data in the computing node C3 can be divided into three parts A3, B3, and D3.

[0059] Each computing node sends control information to the three first switching nodes connected thereto, and each control information includes a communication task identifier. Each communication task identifier is used to indicate a communication task. The communication task identifiers in the control information sent by each computing node to the same first switching node are the same. After receiving the control information, each first switching node obtains data from the computing node that sent the control information based on the control information. In the case where the control information is received from the above four computing nodes, one first switching node obtains the corresponding gradient data from the four computing nodes. After performing calculation on the obtained gradient data, the first switching node obtains a data calculation result. The data calculation result is sent to each computing node.

[0060] ​For example, the first switching node S0 sends one or more data read requests to the computing node C0 to obtain data A0 from the computing node C0 after receiving the control information of the computing node C0. The S0 can obtain data A1, data A2 and data A3 from other computing nodes respectively by the same method. Then, the data A0, data A1, data A2 and data A3 are aggregated to obtain the aggregated data calculation result A0123, and the data calculation result A0123 is sent to each computing node. Similarly, the switching node S1 can obtain data B0, data B1, data B2 and data B3 from the above four computing nodes, calculate the data calculation result B0123, and the switching node S2 can obtain data D0, data D1, data D2 and data D3 from the above four computing nodes, calculate the data calculation result D0123, and then send the obtained data calculation result to the corresponding computing node, so that the gradient data in each computing node is the same.

[0061] Since the model needs to be trained for multiple rounds of iterations, the gradient data in each computing node can be synchronized before the next iteration by the above method, and after obtaining new gradient data by back propagation of the next iteration training, the computing node and the switching node can again aggregate the gradient data by the above method.

[0062] The above is an example of a data processing method provided by the present application, which is used for data aggregation of gradient data after back propagation of model training in a distributed computing system through data parallel model training.

[0063] It should be noted that not all computing nodes in the distributed computing system participate in the execution of the business to be executed each time the distributed computing system executes the business to be executed. For example, in the distributed computing system shown in the above Figure 2 In the distributed computing system shown in the above

[0064] The data processing method based on the above distributed computing system will be described in detail below in combination with the above distributed computing system and the accompanying drawings. As shown in Figure 5 Figure 5 is an interaction diagram of a data processing method provided by an embodiment of the present application. ​

[0065] S501. The host obtains topology information of the distributed computing system, and generates operator information according to the topology information and the to-be-executed service.

[0066] In the embodiments of the present application, the distributed computing system shown in Figure 2 The data processing method provided by the embodiments of the present application is introduced by taking the distributed computing system shown in Figure 6 As shown in the figure. After the distributed computing system starts a to-be-executed service, the host can obtain the topology information of the distributed computing system from the management node, and the topology information includes the connection relationship between each computing node and each first switching node in the distributed system. Then the host generates operator information according to the computing node executing the to-be-executed service and the topology information, and the above-mentioned operator information includes computing operators and communication operators. The computing operator is executed to implement the computing operation in the to-be-executed service, and the communication operator is executed to implement the data sending or receiving operation between the computing node and other nodes.

[0067] Since in the process of executing the above-mentioned to-be-executed service through the distributed computing system, the set communication between each computing node will be involved, that is, the above-mentioned communication operator is used to implement the set communication between each computing node. In the embodiments of the present application, the set communication between each computing node is implemented through the switching node, and in order to make full use of the resources of the first switching node connected to each computing node, the embodiments of the present application implement the operation corresponding to each communication operator through multiple first switching nodes connected to each computing node. Therefore, after the host obtains the above-mentioned topology information, the host determines the number N of first switching nodes connected to each computing node based on the above-mentioned topology information, and then splits the operation corresponding to each communication operator into m*N communication tasks, each first switching node corresponds to at least one of the m*N communication tasks, or each first switching node corresponds to m of the m*N communication tasks, and the operation corresponding to the communication operator is implemented through the m*N communication tasks corresponding to the N first switching nodes, wherein m is a positive integer. In the embodiments of the present application, the data processing method provided by the present application is introduced by taking m equal to 1 as an example, that is, the operation corresponding to each communication operator is split into N communication tasks.

[0068] For example, taking the distributed computing system shown in Figure 4 As an example, the to-be-executed service is AI model training in a data parallel manner, and the AI model training is performed through the 4 computing nodes in Figure 4 Then, after each computing node completes the back propagation of one iteration, the gradient data obtained in each computing node needs to be Allreduced, that is, the communication operator corresponding to Allreduce is executed. Since Figure 4Each computing node is connected with three first exchange nodes S0-S2, and the operation performed by the communication operator corresponding to the Allreduce in each computing node can be split into three communication tasks, that is, the gradient data in each computing node is divided into three parts. Each of the three first exchange nodes obtains a part of the gradient data from one computing node, that is, each first exchange node corresponds to one of the three communication tasks. The three first exchange nodes can obtain the complete gradient data in one computing node, that is, the three communication tasks corresponding to the three first exchange nodes implement the operation corresponding to the Allreduce communication operator.

[0069] By splitting the communication operator into N communication tasks, each communication task is essentially a first exchange node obtaining part of the data in one computing node, and each computing node is connected with multiple first exchange nodes, each first exchange node is used to process part of the data in the connected computing node, thereby reducing the amount of data that each first exchange node needs to process, fully utilizing the resources of the exchange nodes in the distributed computing system, and improving the efficiency of the distributed computing system in processing data.

[0070] In the embodiment of the application, before S501, the network of the distributed computing system is also initialized, for example, before the distributed computing system is used for the first time to perform a service, the network of the distributed computing system needs to be initialized. As shown in Figure 7 Figure 7 is a schematic diagram of network initialization provided by an embodiment of the application. The management node can obtain the topology information of the distributed system by scanning the network of the distributed computing system, and then complete the routing configuration of the network in the distributed computing system based on the topology information. The topology information includes the connection relationship between each computing node and each first exchange node in the distributed system. If the distributed computing system includes multiple layers of exchange nodes, for example Figure 3 In the embodiment of the application, before S501, the network of the distributed computing system is also initialized, for example, before the distributed computing system is used for the first time to perform a service, the network of the distributed computing system needs to be initialized. As shown in

[0071] After the management node scans the network to obtain the above topology information, the management node sends the topology information to each exchange node. After each exchange node receives the topology information, the initialization configuration of each exchange node is completed. For example, as shown in Figure 7 ​In the case that the above distributed computing system includes multi-layer switching nodes, each switching node generates a mapping relationship between a logical port and a physical port according to the topology information, and allocates a queue and a queue ID for each logical port. One logical port includes one or more physical ports, and the physical ports included in one logical port are connected with one switching node of an adjacent layer. After completing the initialization configuration, each connected switching node exchanges communication resource information, which includes a queue ID, for indicating the queue ID when one switching node communicates with another switching node. For example, Figure 3 In the case that the above distributed computing system includes multi-layer switching nodes, each switching node generates a mapping relationship between a logical port and a physical port according to the topology information, and allocates a queue and a queue ID for each logical port. One logical port includes one or more physical ports, and the physical ports included in one logical port are connected with one switching node of an adjacent layer. After completing the initialization configuration, each connected switching node exchanges communication resource information, which includes a queue ID, for indicating the queue ID when one switching node communicates with another switching node. For example,

[0072] S502. The host generates first control information corresponding to each computing node.

[0073] In S501, the communication operator is split into N communication tasks, and the operation of executing the communication operator is converted into executing N communication tasks, each of which corresponds to a first switching node obtaining a part of data from a computing node. Therefore, in the case that the computing node executes the communication operator, the computing node needs to send the first control information to the connected first switching node, so that the first switching node receiving the first control information obtains data from the computing node sending the first control information.

[0074] In the embodiment of the present application, after each communication operator is split into N communication tasks, the host generates corresponding first control information for each communication task, and the first control information includes a communication task ID for uniquely identifying one communication task. It should be understood that each communication operator executed by each computing node is converted into N communication tasks, and the purpose of each communication task is to make the first switching node receiving the first control information obtain data from the computing node sending the first control information and aggregate the data. Therefore, the data obtained by a first switching node based on the first control information sent by different computing nodes should be able to be aggregated. In the embodiment of the present application, the communication task ID in the multiple first control information used by a first switching node to obtain aggregated data from multiple computing nodes is the same, wherein the aggregated data is data that can be aggregated and is obtained by a first switching node from different computing nodes. For example, in the above Figure 4In the specific embodiment, A0, A1, A2 and A3 are aggregated data, and the communication task IDs in the four first control information obtained by S0 for A0, A1, A2 and A3 are the same.

[0075] S503. The host distributes the task information corresponding to the to-be-executed service to each computing node.

[0076] After the host converts the to-be-executed service into the computing operators and communication operators required by each computing node by the method in S501 and S502, generates the communication task corresponding to each communication operator and the first control information corresponding to each communication task, and obtains the task information corresponding to the to-be-executed service, the task information includes operator information and first control information, and then the host sends the generated operator information and the first control information corresponding to each communication task to the corresponding computing node.

[0077] S504. Each computing node sends the first control information to the connected first switching node.

[0078] Taking any one computing node in the distributed computing system as an example, for example, the first computing node, the first computing node is connected with N first switching nodes, and the first computing node sends the first control information to the N first switching nodes connected with the first computing node respectively when the communication operator is executed. For example, the first control information sent by the first computing node to the first switching node S0 includes the address information of the data to be transmitted by the communication task, and the first control information sent by the first computing node to the first switching node S1 includes the address information of the data to be transmitted by the communication task. Figure 4 In the specific embodiment, the computing node C0 sends the first control information to the first switching nodes S0, S1 and S2 respectively, and the computing node C1 sends the first control information to the first switching nodes S0, S1 and S2 respectively.

[0079] S505. The first switching node sends a data reading request to the computing node connected therewith.

[0080] In the embodiment of the application, the first control information further includes address information, and the address information includes the address of the data to be transmitted by the communication task, that is, the address information is used to indicate the position of the data to be read by the first switching node from the computing node sending the first control information. After receiving the first control information sent by one computing node, the first switching node generates one or more data reading requests according to the address information in the first control information; when multiple data reading requests are generated, each data reading request is used to read part of the data indicated by the first control information. Then the one or more data reading requests are sent to the corresponding computing node to read the corresponding data from the computing node. For example, the first control information sent by the first computing node to the first switching node S0 includes the address information of the data to be transmitted by the communication task, and the first control information sent by the first computing node to the first switching node S1 includes the address information of the data to be transmitted by the communication task. Figure 2In the embodiment, the first control information sent by the C1 to the S0 includes the start address and the data length of the data A1. The S0 generates one or more data read requests according to the start address of the data A1, the data length and the maximum payload size (MPS). For example, the data length of the data A1 is 1000 bytes and the MPS is 125 bytes, then the S0 generates 8 data read requests, each of which reads only a part of the data A1. Each data read request includes a start address and a read length, which is used to read a piece of data with the length of the read length starting from the start address. It should be noted that the first switching node can also generate one or more data read requests according to the start address, the data length and the maximum read request size (MRRS), which is not limited in the embodiment. The first switching node needs to generate one or more read requests according to the first control information sent by each computing node, that is, one or more read requests generated according to the first control information sent by a computing node are used to obtain the data indicated by the first control information from the computing node. In the case of generating multiple data read requests, one data read request can only read part of the data indicated by the first control information.

[0081] In a possible implementation, the first control information further includes computing node information, which is used to indicate the number of computing nodes connected to each first switching node. Before sending the data read request, each first switching node further needs to confirm whether the first control information sent by all the computing nodes connected thereto is received according to the computing node information. The first switching node generates the one or more data read requests only when the first control information sent by all the computing nodes connected thereto is received. For example, after the first switching node receives the first control information with a communication task ID of 01, it is determined according to the computing node information in the first control information that the first switching node is connected to four computing nodes. The first switching node confirms that the first control information sent by all the computing nodes connected thereto is received after receiving the four first control information carrying the communication task ID of 01. Since the storage resource of the switching node is limited, the first switching node sends the data read request to the computing nodes only after determining that the first control information of all the computing nodes connected thereto is received, which can avoid reading data as soon as a first control information is received, and the problem that the data read first needs to be cached for a long time. It should be understood that the computing node information actually indicates the number of computing nodes connected to each first switching node and used to execute the current to-be-executed service. In this application, it is assumed that all the computing nodes in the distributed computing device participate in executing the service, and therefore the number of computing nodes connected to each first switching node and used to execute the current to-be-executed service is the same as the number of all the computing nodes connected to the first switching node.

[0082] Optionally, the computing node information can be a numerical value, that is, a numerical value is directly used to indicate the number of computing nodes connected to each first switching node. The computing node information can also be in other forms, for example, the computing node information includes k bits, where k is the number of ports included in the first switching node, and each bit corresponds to a port of the first switching node. In the case where one computing node is connected to one port, the bit corresponding to the port is set to 1, and the bits corresponding to the other ports are set to 0. Then, the first switching node can determine the number of computing nodes connected thereto, that is, the number of computing nodes participating in executing the to-be-executed service, according to the number of bits set to 1 in the computing node information after receiving the first control information.

[0083] The first switching node can also confirm whether the first control information sent by all the computing nodes connected thereto is received through other manners, which are not limited in the embodiments of the present application. For example, the computing node information includes k bits, where k is the number of ports included in the first switching node, and each bit corresponds to a port of the first switching node. In the case where one port is connected to one computing node, the bit corresponding to the port is set to 1, and the remaining bits are set to 0. After receiving one first control information, the first switching node can determine from which ports the first control information needs to be received according to the bits set to 1. After the first switching node receives the first control information from the ports, the first switching node confirms that the first control information sent by all the computing nodes connected thereto is received.

[0084] In the embodiments of the present application, in the case where the distributed computing system can simultaneously execute multiple services, after receiving the first control information sent by any one computing node, the first switching node needs to first determine whether the computing resource thereof is available based on the first control information. In the case where the first switching node confirms that the computing resource thereof is available, the first switching node sends a data reading request to the computing node connected thereto. For example, each first switching node is configured to support 3 concurrent processes simultaneously. If a first switching node S0 is processing 3 concurrent processes when receiving the first control information, and it is determined according to the communication task ID that the first control information corresponds to a new concurrent request, S0 determines that the computing resource thereof is not available, and returns information of insufficient resource to the computing node. If S0 is processing 2 concurrent processes when receiving the first control information, S0 confirms that the computing resource thereof is available, and sends a data reading request to the computing node connected thereto.

[0085] It should be noted that the first switching node can confirm whether the computing resource thereof is available after receiving the first control information sent by one computing node. In the case where the first switching node confirms that the computing resource thereof is not available, the first switching node needs to return the information of insufficient resource to the computing node after receiving the first control information sent by all the computing nodes connected thereto, so as to avoid the case where the first switching node confirms that the computing resource thereof is not available when receiving the first control information sent by the first computing node, and then determines that the computing resource thereof is available after receiving the first control information sent by the remaining computing nodes.

[0086] S506. Each computing node acquires corresponding data according to the received data reading request, and sends the acquired data to the first switching node.

[0087] Each computing node acquires corresponding data according to the start address and the reading length in each data reading request after receiving the data reading request, and sends the acquired data to the corresponding first switching node.

[0088] S507. The first switching node calculates the data calculation result based on the data obtained from each computing node, and sends the data calculation result to the computing node it is connected to.

[0089] After the first exchange node obtains data from each computing node according to one or more data read requests, it aggregates the obtained data and then sends the aggregated data to each computing node. For example, if the first exchange node can read the data indicated by the first control information through a single data read request, such as obtaining data A0 from C0, data A1 from C1, data A2 from C2, and data A3 from C3, and then performing calculations on data A0, A1, A2, and A3 to obtain the data calculation results, it sends the data calculation results to the four computing nodes respectively. For example, if the above communication task is Allreduce, the first exchange node sums A0, A1, A2, and A3 and then sends them to the four computing nodes. The first exchange node can perform the summation after obtaining data returned from two computing nodes, or it can wait until all four data points are obtained before performing the summation; this embodiment does not impose specific limitations.

[0090] The above combination Figure 2 This paper introduces a data processing method for a distributed computing system that includes a single layer of switching nodes. The data processing method provided in this application can also be applied to distributed computing systems that include multiple layers of switching nodes. For example, it can be applied to systems such as… Figure 3 The distributed computing system shown includes two layers of switching nodes. This system comprises a host, multiple computing nodes, at least one first-layer switching node, and at least one second-layer switching node. When the distributed computing system includes at least one second switching node, the data read by the first switching node through the aforementioned data read request also needs to be sent to the second switching node connected to it for processing. Figure 8 As shown, Figure 8 This is a schematic diagram of another data processing method provided in the embodiments of this application. In the case where the distributed computing system includes two layers of switching nodes, the operations performed by the host and computing nodes of the distributed computing system can refer to S501 to S504 above. After each computing node sends the first control information to the connected first switching node, the following steps are also included: S508 to S515.

[0091] S508. The first switching node generates second control information based on the first control information.

[0092] The first switching node further generates second control information according to the first control information after confirming that all the computing nodes connected thereto send the first control information, and sends the second control information to the second switching node. The second control information includes the communication task ID in the first control information.

[0093] S509. The second switching node generates notification information according to the received second control information, and sends the notification information to the first switching node.

[0094] The notification information includes the communication task ID, so that the first switching node receiving the notification information confirms the corresponding first control information, the address information and the data length in the first control information according to the communication task ID, and then generates the one or more data read requests.

[0095] In a possible implementation, the second switching node needs to confirm that all the first switching nodes connected thereto send the second control information before sending the notification information to the first switching nodes. In the case that the second switching node needs to confirm whether all the first switching nodes connected thereto send the second control information, the first control information further includes first switching node information, which is used to indicate the number of the first switching nodes connected to the second switching node. The form of the first switching node information can refer to the computing node information in the first control information, which will not be described herein again.

[0096] In the embodiment of the application, in the case that the distributed computing system can simultaneously execute multiple services, the second switching node needs to determine whether the computing resource thereof is available based on the second control information after receiving the second control information sent by any one of the first switching nodes, and the second switching node sends the notification information to the first switching nodes connected thereto only in the case that the second switching node confirms that the computing resource thereof is available. The method that the second switching node confirms whether the computing resource thereof is available can refer to the method that the first switching node confirms whether the resource thereof is available, which will not be described herein again. Since the storage resource of the switching node is limited, the second switching node sends the notification information to the first switching nodes only after confirming that the first control information of all the first switching nodes connected thereto is received, which can avoid the problem that the data read first needs to be cached for a long time and occupies the storage resource due to the fact that the first switching nodes are notified to read the data as soon as the second control information is received.

[0097] It should be noted that the second switching node can confirm whether the computing resource is available after receiving the second control information sent by a first switching node. In the case that the second switching node confirms that the computing resource is not available, the second switching node needs to return the information of insufficient resource to the first switching node after receiving the second control information sent by all the first switching nodes connected thereto, so as to avoid the case that the second switching node confirms that the computing resource is not available after receiving the second control information sent by a first switching node and confirms that the computing resource is available after receiving the second control information sent by the remaining first switching nodes.

[0098] S510. The first switching node generates the data read request according to the notification information, and sends the data read request to the computing nodes connected thereto.

[0099] The notification information includes the communication task ID, and the first switching node receiving the notification information confirms the corresponding first control information and the address information and the data length in the first control information according to the communication task ID, and then generates the one or more data read requests. The method for the first switching node to generate the data read request can refer to S505.

[0100] In a possible implementation, the notification information includes the communication task ID, the offset and the first data length, wherein the first data length is used to indicate the length of the data read by the first switching node this time. The first switching node receiving the notification information confirms the corresponding first control information according to the communication task ID. The first switching node determines the starting address of the data to be read according to the address information in the first control information, determines the starting address of the data to be read this time according to the starting address and the offset, and then generates a data read request according to the starting address of the data to be read this time and the first data length. It should be understood that the first switching node needs to read data from a plurality of computing nodes connected thereto, therefore, the first switching node generates a data read request according to the received notification information and the first control information sent by each computing node connected thereto after receiving the notification information each time, and then sends the data read to the corresponding computing node. It should be understood that the notification information is equivalent to the read request sent by the second switching node to the first switching node, and is used to read the data of the first data length from the first switching node each time. By actively reading the data from the first switching node by the second switching node, the congestion problem at the second switching node caused by the first switching node sending data to the second switching node can be avoided.

[0101] For example, the first data length is equal to the data length in the first control information. Figure 3In some embodiments, S0 needs to read data from C0, C1, C2 and C3, S0 generates a data read request corresponding to C0 according to the notification information and the first control information sent by C0, for reading data from C0; S0 generates a data read request corresponding to C1 according to the notification information and the first control information sent by C1, for reading data from C1; S0 generates a data read request corresponding to C2 according to the notification information and the first control information sent by C2, for reading data from C2; S0 generates a data read request corresponding to C3 according to the notification information and the first control information sent by C3, for reading data from C3. It should be understood that one notification information is used to read part of the data that needs to be read from the computing nodes, and the second switching node sends multiple notification information to the first switching node to read all the data that needs to be read from the computing nodes.

[0102] When the distributed computing system includes at least one second switching node as described above, the data read by the first switching node through the data read request also needs to be sent to the second switching node connected thereto for processing, and therefore the first switching node also needs to send the second control information based on the first control information to the second switching node. After the second switching node sends the notification information to the first switching node, it indicates that the first switching node and the second switching node can process the data, and the first switching node will obtain the data from the computing nodes, which can avoid the problem of wasting bandwidth or causing calculation errors due to the inability of the second switching node to process the data after the first switching node reads the data when the resources of the second switching node are not available.

[0103] S511. The first switching node sends a data read request to the computing nodes connected thereto.

[0104] S512. Each computing node obtains corresponding data according to the received data read request and sends the obtained data to the first switching node.

[0105] Each computing node obtains corresponding data according to the start address and read length in each data read request after receiving the data read request and sends the data to the corresponding first switching node.

[0106] S513. The first switching node calculates a first calculation result according to the data obtained from each computing node and sends the first calculation result to the second switching node.

[0107] The first switching node obtains data from each computing node, performs corresponding calculation to obtain a first calculation result, and then sends the first calculation result to the second switching node.

[0108] S514. The second switching node performs calculation according to the first calculation results sent by all the first switching nodes connected thereto, obtains a data calculation result, and sends the data calculation result to the first switching nodes connected thereto.

[0109] S515. The first switching nodes send the data calculation result to the respective connected calculation nodes.

[0110] The first switching nodes generate the one or more data read requests according to the notification information, and obtain the data indicated by the first control information from the calculation nodes. After obtaining the data indicated by the data read requests, one first switching node performs calculation on the data read from different calculation nodes connected thereto to obtain a first calculation result, and sends the first calculation result to the second switching node sending the notification information. The second switching node performs calculation according to the first calculation results sent by all the first switching nodes connected thereto, obtains a data calculation result, and sends the data calculation result to the first switching nodes connected thereto, which then send the data calculation result to the respective connected calculation nodes. For example, S0 obtains data A0 from C0, data A1 from C1, data A2 from C2, and data A3 from C3, performs calculation on the data A0, A1, A2 and A3 to obtain an intermediate calculation result A0123, S1 obtains data A4 from C4, data A5 from C5, data A6 from C6, and data A7 from C7, performs calculation on the data A4, A5, A6 and A7 to obtain an intermediate calculation result A4567, and sends the two intermediate calculation results to S6, which performs calculation on the two intermediate calculation results to obtain the data calculation result.

[0111] Since the computing resources and storage resources in the switching nodes are limited, in the distributed calculation nodes including multiple layers of switching nodes, each layer of switching nodes performs corresponding calculation operation according to the obtained data, and the aggregation calculation of a large amount of data can be implemented through multiple switching nodes, thereby improving the efficiency of implementing data aggregation in the distributed calculation system.

[0112] In a possible implementation, one first switching node and one second switching node can be connected through one or more physical ports. In the case where one first switching node and one second switching node are connected through multiple physical ports, if the communication protocol between the two switching nodes supports load balancing of multiple paths, the load balancing of data transmission is implemented by the communication protocol. If the communication protocol between the two switching nodes does not support load balancing of multiple paths, the load balancing of multiple paths is implemented by the switching nodes, for example, in a polling manner, data is sent through different physical ports each time.

[0113] In a possible implementation, the first control information further comprises a first address, and the first address is used for writing a flag bit to a position indicated by the first address to inform the computing node that the operation corresponding to the communication operator has been executed after the first switching node or the second switching node executes data aggregation to obtain a data calculation result, or the operation corresponding to the communication operator has been executed by sending a message to inform the computing node.

[0114] In a possible implementation, the first control information further comprises a second address, and the second address is used for writing a flag bit to a position indicated by the second address to inform the computing node that the computing resource of the switching node is available after the first switching node and the second switching node confirm that the computing resource of the switching node is available, or the operation corresponding to the communication operator has been executed by sending a message to inform the computing node.

[0115] The first control information further comprises a second address, and the second address is used for writing a flag bit to a position indicated by the second address to inform the computing node that the computing resource of the switching node is available after the first switching node and the second switching node confirm that the computing resource of the switching node is available, or the operation corresponding to the communication operator has been executed by sending a message to inform the computing node. Figure 5 and Figure 8 In the corresponding embodiment, the first switching node performs a calculation operation on the data read from the plurality of computing nodes, and the second switching node performs a calculation operation on the data read from the plurality of first switching nodes. It should be understood that in other application scenarios, the switching nodes (including the first switching node and the second switching node) in the distributed computing system can not perform calculation on the obtained data, and the switching nodes are only used to realize synchronization of data among the plurality of computing nodes. Through the distributed computing system and the data processing method provided in the present application, the number of times of sending data among the plurality of computing nodes can be reduced, and the efficiency of data synchronization and the calculation efficiency can be improved. In addition, the first switching node actively reads data from the plurality of computing nodes, which can avoid congestion at the first switching node caused by the computing nodes actively sending data; the second switching node actively reads data from the plurality of first switching nodes, which can avoid congestion at the second switching node caused by the plurality of first switching nodes actively sending data.

[0116] Through implementation of the distributed computing system and the data processing method provided in the present application, in the process of distributed computing, when collective communication is needed, the computing node sends control information to the switching node, the switching node receiving the control information actively obtains the data to be aggregated from the computing node, performs data aggregation in the switching node, and then sends the aggregated result to the computing node, so that the amount of data to be transmitted and the number of synchronization times when the distributed computing system performs data aggregation through collective communication can be reduced, and the calculation efficiency of the distributed computing can be improved.

[0117] In addition, in the above distributed computing process, the computing node sends control information to the exchange node, and then the exchange node sends a data read request to obtain data to be aggregated and perform data aggregation on the exchange node. After the data aggregation is completed, the exchange node sends the aggregated result to the node that sends the control information. Before and after the process, the exchange node can be used to process other tasks, that is, the aggregation task triggered by the control information does not exist for a long time to continuously occupy the computing resources of the exchange node without releasing. That is, in a service, only when collective communication is needed, the service occupies the resources of the exchange node for data aggregation. In the process of executing the service by the computing node, the resources of the exchange node are not occupied. Therefore, in the case that the distributed computing system executes multiple services, the resources of the exchange node can be fully utilized by using the data processing method provided in the application.

[0118] For the above method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the application is not limited by the order of the described actions, and those skilled in the art should also know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily necessary for the application. Based on the above description, other reasonable step combinations that those skilled in the art can think of also belong to the protection scope of the application.

[0119] The application also provides a data processing system, which comprises the above Figure 2 or Figure 3 distributed computing system shown in the above Figure 2 or Figure 3 corresponding description and related description, which will not be repeated here. The operations of the host, the computing node and the exchange node in the distributed computing system can refer to the related description in the above Figure 4 to Figure 8 corresponding embodiments, which will not be repeated here.

[0120] The application also provides an exchange node, as shown in the above Figure 9 The exchange node 900 comprises a communication control module 910, a processing module 920 and an exchange architecture 930, and the exchange node is applied to the above Figure 2 or Figure 3As shown in the distributed computing system, the distributed computing system comprises a plurality of computing nodes and at least one exchange node 900, each of the at least one exchange node 900 is connected with at least one computing node of the plurality of computing nodes. The communication control module 910 is configured to receive first control information sent by the at least one computing node connected with the exchange node 900, the first control information comprises a communication task ID, the communication task ID is used to indicate a communication task participated by the at least one computing node. The processing module 920 is configured to generate a data reading request according to the first control information. The communication control module 910 is further configured to send the data reading request to the at least one computing node connected with the exchange node 900. The processing module 920 is further configured to obtain a data calculation result, the data calculation result is obtained according to data read by the data reading request. The communication control module 910 is further configured to send the data calculation result to the at least one computing node connected with the first exchange node.

[0121] In a possible implementation, the distributed computing system further comprises a second exchange node, the second exchange node is connected with the exchange node 900. The processing module 920 is further configured to generate second control information according to the first control information, the second control information comprises the communication task ID. The communication control module 910 is further configured to send the second control information to the second exchange node. The processing module 920 is specifically configured to: after receiving notification information sent by the second exchange node, generate a data reading request according to the first control information. The notification information comprises the communication task ID, and the notification information is used to indicate that the exchange node 900 obtains data from the at least one computing node connected therewith.

[0122] In a possible implementation, the processing module 920 is further configured to calculate the data read from the at least one computing node to obtain a first calculation result. The communication control module 910 is further configured to send the first calculation result to the second exchange node. The data calculation result is received, the data calculation result is obtained by the second exchange node according to the first calculation result and a first calculation result sent by another exchange node of the at least one exchange node 900.

[0123] In a possible implementation, the communication control module 910 and the processing module 920 are logical circuits in the exchange node. The internal micro-architecture of the processing module 920 can be implemented as a ring architecture, a double ring architecture, a single centralized control architecture or a plurality of distributed centralized unit architectures.

[0124] The exchange architecture 930 is used to realize the exchange of data between different ports of the exchange node, for example, data received from a first port is sent out through a second port.

[0125] The operations that can be implemented by the above exchange node 900 can refer to the operations of the first exchange node or the second exchange node in the above Figure 4 to Figure 8 corresponding embodiments. Among them, the communication control module 910 is configured to implement the operations of receiving data or sending data performed by the first exchange node or the second exchange node, and the processing module 920 is configured to implement operations other than the above receiving data or sending data, which will not be repeated here.

[0126] The embodiments of the present application also provide a host computer, such as Figure 10 as shown, Figure 10 is a schematic diagram of a host computer provided by the embodiments of the present application. The host computer 100 includes a processing module 101 and a communication module 102. The host computer 100 is applied to a distributed computing system as shown in Figure 2 or Figure 3 The distributed computing system includes a host computer 100, a plurality of computing nodes, and at least one first exchange node. Each of the at least one first exchange node is connected to at least one computing node in the plurality of computing nodes. Among them, the processing module 101 can be an AI training framework such as mindspore or tensorflow.

[0127] The processing module 101 is configured to acquire and parse topology information of the distributed system in the case that the distributed computing system needs to process a to-be-executed business, for example, according to the topology information to confirm the number of first exchange nodes connected to each computing node. The topology information includes the connection relationship between each computing node and each first exchange node in the distributed system. In the case that the distributed computing system includes Figure 3 at least one second exchange node as shown, the topology information further includes the connection relationship between the at least one first exchange node and the at least one second exchange node.

[0128] The processing module 101 generates operator information according to the topology information of the distributed computing system and the to-be-executed business, and the operator information includes computing operators and communication operators. The processing module 101 is further configured to convert the above communication operators into a plurality of communication tasks according to the topology information, configure a communication task ID for each communication task, and generate first control information corresponding to each communication task according to the communication task. Specifically, the operations implemented by the above processing module 101 can refer to the operations of the host computer in the above Figure 5 to Figure 8 corresponding embodiments, which will not be repeated here.

[0129] The communication module 102 is configured to issue task information to each computing node in the distributed computing system. The task information includes computing operators and communication operators that need to be executed by the computing nodes, and the communication tasks corresponding to each communication operator and the first control information corresponding to each communication task.

[0130] In a possible implementation, the host 100 further includes a management module 103, which is configured to scan a network of the distributed computing system to obtain topology information of the distributed system, and complete routing configuration of the network in the distributed computing system based on the topology information.

[0131] The communication module 102 is further configured to send the topology information obtained by the management module 103 to each switching node, so that each switching node completes configuration, for example, generates a mapping relationship between a logical port and a physical port, and allocates a queue and a queue ID to each logical port.

[0132] The operations implemented by the management module 103 can refer to the operations implemented by the management node in the embodiments described above, which will not be repeated here. Figure 5 The operations implemented by the management node in the embodiments described above, which will not be repeated here.

[0133] Specifically, the operations implemented by the host 100 can refer to the operations implemented by the host in the embodiments described above, which will not be repeated here. Figure 5 to Figure 8 The operations implemented by the host in the embodiments described above, which will not be repeated here.

[0134] The embodiments of the present application further provide a computing device, as shown in Figure 11 Figure 11 is a schematic diagram of a computing device provided by the embodiments of the present application. The computing device 110 includes a host 111, a plurality of computing nodes 112, a communication interface 113, and a memory 114, which are connected to each other through a bus 115, wherein,

[0135] The host 111 can be a CPU, and the computing nodes 112 can be GPUs, NPUs, TPUs, or special AI processing chips. The operations performed by the host 111 can refer to the operations performed by the host in the embodiments described above, and the operations performed by the computing nodes 112 can refer to the operations performed by the computing nodes in the embodiments described above, which will not be repeated here. Figure 4 to Figure 8 The operations performed by the host in the embodiments described above, which will not be repeated here. Figure 4 to Figure 8 The operations performed by the computing nodes in the embodiments described above, which will not be repeated here.

[0136] The communication interface 113 can be a wired interface or a wireless interface, which is configured to communicate with other modules or devices. The wired interface can be an Ethernet interface, a local interconnect network (LIN), etc., and the wireless interface can be a cellular network interface or a wireless local area network interface, etc.

[0137] ​The memory 114 can be a non-volatile memory, for example, a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), or a flash memory. The memory 114 can also be a volatile memory, which can be a random access memory (RAM) used as an external cache. By way of example, and not limitation, many forms of RAM can be used, for example, a static RAM (SRAM), a dynamic RAM (DRAM), a synchronous DRAM (SDRAM), a double data rate SDRAM (DDR SDRAM), an enhanced SDRAM (ESDRAM), a synchlink DRAM (SLDRAM), and a direct rambus RAM (DR RAM).

[0138] The memory 114 can also be used to store program instructions and data for facilitating the operation of the host 111 in performing the operations of the host in the above method embodiments by invoking the program instructions stored in the memory 114. In addition, the computing device 110 can include more or less components, or different configurations of components, than those shown. Figure 11 The more or less components shown or different configurations of components can be used.

[0139] The bus 115 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 115 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 11 Only one thick line is used to represent the bus in the figure, but it does not mean that there is only one bus or only one type of bus.

[0140] The computing device 110 described above can also include the at least one switching chip 116 described above, and the operations performed by the switching chip 116 can refer to the operations performed by the first switching node or the second switching node in the above embodiments, which will not be described here.

[0141] Specifically, the specific implementation of the computing device 110 performing various operations can refer to the specific operations performed by the distributed computing system in the above method embodiments, which will not be repeated here.

[0142] The embodiment of the present application further provides a computer readable storage medium, wherein instructions are stored in the computer readable storage medium, and when the instructions are run on a processor, the method steps implemented by the evaluation device in the above method embodiments can be implemented. The specific implementation of the processor of the computer readable storage medium in performing the above method steps can refer to the specific operations of the above method embodiments, which will not be repeated here.

[0143] The embodiment of the present application further provides a computer readable storage medium, wherein instructions are stored in the computer readable storage medium, and when the instructions are run on a processor, the method steps implemented by the management device in the above method embodiments can be implemented. The specific implementation of the processor of the computer readable storage medium in performing the above method steps can refer to the specific operations of the above method embodiments, which will not be repeated here.

[0144] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0145] The above embodiments can be realized all or partially by software, hardware, firmware or other any combination. When realized by software, the above embodiments can be realized all or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the flow or function described in the embodiments of the present application is all or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center and the like containing one or more available medium sets. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium or a semiconductor medium. The semiconductor medium can be a solid state disk.

[0146] The steps in the method embodiments of the present application can be adjusted in sequence, combined or deleted according to actual needs; the modules in the device embodiments of the present application can be divided, combined or deleted according to actual needs.

[0147] The above detailed description of the embodiments of the present application is made with specific examples, and the above description of the embodiments is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation and application range can be changed, and the above description of the present application should not be understood as a limitation.

[0148] The above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized by, The method is applied to a distributed computing system comprising a plurality of computing nodes and at least one first switching node, and comprises the following steps: The first switching node receives first control information of at least one computing node connected thereto in the distributed computing system, wherein the first control information comprises a communication task ID and computing node information, the communication task ID is used to indicate a communication task in which the at least one computing node participates, and the computing node information is used to indicate the number of computing nodes connected to the first switching node; After confirming that the first control information of all the computing nodes connected thereto is received according to the computing node information, the first switching node sends a data reading request to the at least one computing node according to the first control information; The first switching node obtains a data calculation result, wherein the data calculation result is obtained according to the data read by the data reading request; The first switching node sends the data calculation result to the at least one computing node, and releases the resources of the first switching node for other task processing after sending the data calculation result.

2. The method of claim 1, wherein, The distributed computing system further comprises a second switching node connected to the first switching node; The first switching node sends a data reading request to the at least one computing node according to the first control information, comprising: The first switching node sends second control information to the second switching node according to the first control information, wherein the second control information comprises the communication task ID; After receiving notification information sent by the second switching node, the first switching node sends the data reading request to the at least one computing node, wherein the notification information comprises the communication task ID, and the notification information is used to indicate that the first switching node acquires data from the at least one computing node.

3. The method of claim 2, wherein, The first switching node obtains a data calculation result, comprising: The first switching node calculates the data read from the at least one computing node to obtain a first calculation result; The first switching node sends the first calculation result to the second switching node; The first switching node acquires the data calculation result calculated by the second switching node, wherein the data calculation result is a set calculation result obtained by the second switching node according to the first calculation result and calculation results sent by other switching nodes in the at least one first switching node.

4. The method according to any one of claims 1 to 3, characterized in that, The communication task is Allreduce, and the data calculation result is a sum, average or maximum value result of the data acquired by the data reading request.

5. The method according to any one of claims 1 to 3, characterized in that, The first control information further comprises address information, and the address information indicates the position of the data read by the first switching node from the computing node sending the first control information; The data read by the data reading request is part or all of the data indicated by the first control information.

6. The method according to any one of claims 1 to 3, characterized in that, The computing node is a graphics processing unit (GPU), a neural processing unit (NPU), a tensor processing unit (TPU), or a dedicated artificial intelligence (AI) processing chip, and the first switching node is a switching chip or a switching device with switching function.

7. The method according to any one of claims 1 to 3, characterized in that, The distributed computing system further comprises a host, and the method further comprises: The host generates operator information according to topology information of the distributed computing system and a to-be-executed service, and the operator information comprises a communication operator and a computing operator in the to-be-executed service.

8. The method of claim 7, wherein, The method further comprises: The host determines, according to the topology information, that the number of first switching nodes connected to each computing node is N, wherein N is a positive integer; The host splits the communication operator into N communication tasks, and each first switching node corresponds to one communication task in the N communication tasks.

9. The method of claim 8, wherein, The host distributes the communication operator and the N communication tasks to the plurality of computing nodes.

10. A switching node, characterized by The switching node comprises a communication control module and a processing module, and is located in a distributed computing system comprising a plurality of computing nodes and at least one switching node; The communication control module is configured to receive first control information sent by at least one computing node connected to the switching node in the distributed computing system, wherein the first control information comprises a communication task identifier (ID) and computing node information, the communication task ID is used to indicate a communication task in which the at least one computing node participates, and the computing node information is used to indicate the number of computing nodes connected to the switching node; The processing module is configured to, after confirming that the first control information of all computing nodes connected thereto is received according to the computing node information, generate a data reading request according to the first control information; The communication control module is further configured to send the data reading request to the at least one computing node; The processing module is further configured to obtain a data calculation result, wherein the data calculation result is obtained according to data read by the data reading request; The communication control module is further configured to send the data calculation result to the at least one computing node, and after sending the data calculation result, release resources of the switching node for other task processing.

11. The switching node of claim 10, wherein, The distributed computing system further comprises a second switching node connected to the switching node; The processing module is further configured to generate second control information according to the first control information, wherein the second control information comprises the communication task ID; The communication control module is further configured to send the second control information to the second switching node; The processing module is specifically configured to, after receiving notification information sent by the second switching node, generate the data reading request according to the first control information, wherein the notification information comprises the communication task ID, and the notification information is used to indicate that the switching node acquires data from the at least one computing node.

12. The switching node of claim 11, wherein The processing module is further configured to calculate data read from the at least one computing node to obtain a first calculation result. The communication control module is further configured to send the first calculation result to the second switching node; receiving the data calculation result calculated by the second switching node, wherein the data calculation result is a collective calculation result obtained by the second switching node according to the first calculation result and calculation results sent by other switching nodes in the distributed computing system.

13. The switching node according to any of claims 10-12, c h a r a c t e r i z e d b y The communication task is Allreduce, and the data calculation result is a sum, average, or maximum value result of data obtained by the data read request.

14. The switching node of any one of claims 10-12, wherein the first control information further comprises address information indicating a position of data to be read by the switching node from a computing node sending the first control information. The data read by the data read request is part or all of the data indicated by the first control information.

15. A distributed computing system, characterized in that, The system comprises a plurality of computing nodes and at least one first switching node, each of the at least one first switching node being connected to at least one computing node of the plurality of computing nodes, each computing node being configured to send first control information to the first switching node connected thereto, and each first switching node being configured to perform the method of any one of the preceding claims 1-6.

16. The distributed computing system of claim 15, wherein, The system further comprises a host configured to perform the method of any one of the preceding claims 7-9.

Images