Data processing method, switching node and related system

Abstract

The invention provides a data processing method, a switching node and a related system, which are applied to a distributed computing system comprising a plurality of computing nodes and at least one first switching node, and each first switching node is connected with at least one computing node in the plurality of computing nodes. After receiving control information of at least one computing node connected with the first switching node, the first switching node sends a data reading request to the at least one computing node according to the control information; and the first switching node sends a calculation result to the at least one calculation node after obtaining the calculation result obtained by calculation according to the data read according to the data reading request. In the distributed computing system, when a plurality of computing nodes need to carry out set communication, an exchange node obtains data needing to be aggregated from the computing nodes, carries out data aggregation, and then sends an aggregated result to the computing nodes, so that the amount of data needing to be transmitted and the number of times of synchronization when data aggregation is carried out through set communication can be reduced; the calculation efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a data processing method, a switching node and a related system. Background Art

[0002] With the development of high performance computing (HPC) and artificial intelligence (AI) technologies, many large-scale applications have emerged, and the scale of data that needs to be processed is also increasing. In order to solve the problem of large-scale data computing, distributed computing has come into being. In distributed computing, data interaction is involved between various computing nodes. Collective communication has a large number of important applications in distributed computing. For example, collective communication methods such as allreduce and allgather are common methods for realizing data interaction between multiple computing nodes in distributed computing. However, the current collective communication method has problems such as large amount of data transmission and high number of data synchronization, which affects the efficiency of distributed computing. Summary of the invention

[0003] The present application provides a data processing method, a switching node and a related system, which can reduce the amount of data that needs to be transmitted and the number of synchronization times when a distributed computing system aggregates data through collective communication, thereby improving the computing efficiency of distributed computing.

[0004] In a first aspect, the present application provides a data processing method, which is applied to a distributed computing system including multiple computing nodes and at least one first switching node, wherein each of the at least one first switching node is connected to at least one computing node among the multiple computing nodes. For any first switching node, after receiving the first control information sent by at least one computing node connected to it, the first switching node sends a data read request to at least one computing node connected to it, obtains a data calculation result based on the data read by the data read request, and then sends the obtained data calculation result to the at least one computing node connected to it. The first control information includes a communication task identification (Identity, ID), and the communication task ID is used to indicate the communication task in which the at least one computing node participates.

[0005] In this distributed computing system, when multiple computing nodes are performing distributed computing and collective communication is required, the computing nodes send control information to the switching nodes. The switching nodes that receive the control information actively obtain the data that needs to be aggregated from the computing nodes, aggregate the data in the switching nodes, and then send the aggregated results to the computing nodes. This can reduce the amount of data that needs to be transmitted and the number of synchronization times when the distributed computing system aggregates data through collective communication, thereby improving the computing efficiency of distributed computing.

[0006] In addition, in the above-mentioned distributed computing process, after the computing node sends control information to the switching node, the switching node will send a data reading request to obtain the data to be aggregated and perform data aggregation on the switching node. After the data is aggregated, the result of the data aggregation is sent to the node that sent the control information. Before and after this process, the switching node can be used to process other tasks, that is, the aggregation task triggered by the control information will not occupy the computing resources of the switching node for a long time without releasing them. That is, for a business, only when collective communication is required, the business will occupy the resources of the switching node for data aggregation, and the computing node does not occupy the resources of the switching node during the execution of the business. Therefore, through the data processing method provided by the present application, when the distributed computing system executes multiple businesses, the resources of the switching node can be fully utilized.

[0007] In a possible implementation, the distributed computing system further includes at least one second switching node, and each of the at least one second switching node is connected to some of the at least one first switching node. For the second switching node connected to the first switching node that receives the first control information, the first switching node sends a data read request to at least one computing node according to the first control information, including: the first switching node sends the second control information to the second switching node according to the first control information, and the first switching node sends the data read request to at least one computing node connected to it only after receiving the notification information sent by the second switching node, wherein both the second control information and the notification information include the communication task ID, and the notification information is used to instruct the first switching node to obtain data from at least one computing node connected to it.

[0008] When the distributed computing system includes the above-mentioned at least one second switching node, the data read by the first switching node through the above-mentioned data reading request also needs to be sent to the second switching node connected to it for processing. Therefore, the first switching node also needs to send the second control information obtained based on the above-mentioned first control information to the second switching node. After the second switching node sends a notification message to the first switching node, indicating that the first switching node and the second switching node can process the data, the first switching node will obtain the data from the computing node. This can avoid the problem that when the second switching node is unavailable, the data cannot be processed after being read, resulting in a waste of bandwidth or calculation errors.

[0009] In a possible implementation, when the distributed computing system includes the above-mentioned second switching node, after the above-mentioned first switching node sends a data reading request to at least one computing node connected to it, the above-mentioned first switching node reads data from the above-mentioned at least one computing node based on the data reading request, and the first switching node calculates the read data to obtain a first calculation result. There are other first switching nodes in the distributed computing system that also perform the same operation to obtain the first calculation result. Multiple first switching nodes send their respective calculated first calculation results to the second switching node, the second switching node calculates the above-mentioned data calculation result based on the first calculation result, and the first switching node obtains the above-mentioned data calculation result from the second switching node.

[0010] Since the computing resources and storage resources of the switching nodes are limited, in a distributed computing node including multiple layers of switching nodes, each layer of switching nodes performs corresponding computing operations based on the acquired data, and can implement aggregation calculations of large amounts of data through multiple switching nodes, thereby improving the efficiency of data aggregation in the switching nodes.

[0011] In a possible implementation, the communication task is Allreduce, and the data calculation result is the sum, average, and maximum value of the data obtained by the data read request.

[0012] In a possible implementation, the first control information further includes address information, which indicates a location where the first switching node reads data from the computing node that sends the first control information; the data read by the data read request is part or all of the data indicated by the first control information to be read.

[0013] The data read request needs to include the location of the data to be read, so the first control information also includes address information. When generating a data read request, the first switching node needs to determine the location of the data to be read based on the address information. Since the size of the effective load carried by a data packet is limited or the node receiving the data packet can receive a limited effective load of the data packet, and in a distributed computing system, the amount of data that needs to participate in collective communication in each computing node is large, the data read by a data read request is generally only the part of the data indicated by the address information. The first switching node or the second switching node needs to determine the address of the data to be read by each data read request in the generated multiple data read requests based on the location in the address information.

[0014] In a possible implementation, the first control information also includes computing node information, and the computing node information is used to indicate the number of at least one computing node connected to the first switching node; after the first switching node confirms the receipt of the first control information sent by the computing node to which it is connected according to the computing node information, it sends a data read request to the computing node according to the first control information. Among them, the at least one computing node connected to the first switching node actually represents a computing node connected to the first switching node and participating in the execution of this business, and the above communication task belongs to the task that needs to be executed for this business. Since the storage resources of the switching node are limited, the first switching node sends a data read request to the computing node only after confirming that the first control information sent by all the computing nodes to which it is connected has been received, which can avoid the problem of reading data upon receiving a first control information, resulting in the need to cache the first read data for a long time.

[0015] The computing node may be a graphics processing unit (GPU), a neural network processor (NPU), a tensor processing unit (TPU) or a dedicated AI processing chip, etc.; the switching node may be a switching chip or a switching device with a switching function.

[0016] In a possible implementation, the distributed computing system further includes a host, which generates operator information according to topology information of the distributed computing system and the business to be executed, and the operator information includes communication operators and computing operators in the business to be executed.

[0017] In a possible implementation, the method further includes: the host determines that the number of first switching nodes connected to each computing node is N according to the topology information, splits each of the above communication operators into N communication tasks, each first switching node corresponds to one of the N communication tasks, and then sends the communication operator and the N communication tasks to the multiple computing nodes. Wherein, N is a positive integer. It should be understood that each communication operator can also be split into m*N communication tasks, where m is a positive integer, each first switching node corresponds to m communication tasks, or each first switching node corresponds to at least 1 communication task. Wherein, the communication operator is an operator for performing data aggregation, and the execution of the operator can enable the switching node to obtain data from the computing node and perform data aggregation. By splitting the communication operator into N communication tasks, each communication task is essentially a switching node obtaining part of the data in a computing node, each computing node is connected to multiple first switching nodes, and each first switching node is used to process part of the data in the connected computing node, thereby reducing the amount of data that each switching node needs to process, making full use of the resources of the switching nodes in the distributed computing system, and improving the efficiency of the distributed computing system in processing data.

[0018] In a second aspect, the present application provides a switching node, the switching node including a communication control module and a processing module, applied to a distributed computing system, the distributed computing system including a plurality of computing nodes and at least one switching node, each switching node in the at least one first switching node is connected to at least one computing node in the plurality of computing nodes;

[0019] The above-mentioned communication control module is used to receive first control information sent by at least one computing node connected to the switching node, the first control information includes a communication task identification ID, and the communication task ID is used to indicate a communication task in which at least one computing node participates; the processing module is used to generate a data read request according to the first control information; the communication control module is also used to send a data read request to at least one computing node connected to the above-mentioned switching node; the processing module is also used to obtain a data calculation result, wherein the data calculation result is obtained according to the data read request; the communication control module is also used to send the data calculation result to at least one computing node connected to the switching node.

[0020] In one possible implementation, the above-mentioned distributed computing system also includes a second switching node, which is connected to the above-mentioned switching node; the processing module is also used to generate second control information based on the first control information, and the second control information includes the above-mentioned communication task ID; the communication control module is also used to send the second control information to the second switching node; the processing module is specifically used to: after receiving the notification information sent by the second switching node, generate a data read request according to the first control information; the notification information includes the communication task ID, and the notification information is used to instruct the first switching node to obtain data from at least one computing node connected to it.

[0021] In a possible implementation, the processing module is further used to calculate the data read from at least one computing node to obtain a first calculation result; the communication control module is further used to send the first calculation result to the second switching node; and receive the data calculation result calculated by the second switching node, wherein the data calculation result is a collective calculation result obtained by the second switching node based on the first calculation result and the first calculation results sent by other switching nodes in at least one switching node.

[0022] In a possible implementation, the communication task is Allreduce, and the data calculation result is the sum, average, and maximum value of the data obtained by the data read request.

[0023] In a possible implementation, the first control information further includes address information, which indicates a location where the first switching node reads data from the computing node that sends the first control information; the data read by the data read request is part or all of the data indicated by the first control information to be read.

[0024] In one possible implementation, the first control information also includes computing node information, and the computing node information is used to indicate the number of at least one computing node connected to the first switching node; the processing module is also used to generate a data read request after confirming that the first control information of at least one computing node has been received according to the computing node information.

[0025] In one possible implementation, the computing node is a GPU, an NPU, a TPU, or a dedicated AI processing chip, and the switching node is a switching chip or a switching device with a switching function.

[0026] In a possible implementation, the communication control module and the processing module are logic circuits in a switching node, wherein the internal micro-architecture of the processing module can be implemented as a ring architecture, a dual-ring architecture, a single centralized control architecture, or a plurality of distributed centralized unit architectures.

[0027] In a third aspect, the present application provides a distributed computing system, which includes multiple computing nodes and at least one first switching node, each of the at least one first switching node is connected to at least one computing node of the multiple computing nodes, each computing node is used to send first control information to the first switching node to which it is connected, and each first switching node is used to execute the operation implemented by the first switching node in the first aspect or any possible implementation of the first aspect.

[0028] In a possible implementation manner, the above-mentioned distributed computing system also includes a host, and the host is used to execute the operations implemented by the host in the first aspect or any possible implementation manner of the first aspect.

[0029] In a fourth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the instructions are executed on a computing device, the computing device executes the method described in the first aspect or any possible implementation of the first aspect.

[0030] In a fifth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the instructions are executed on a switching node, the computing device executes the method described in the second aspect or any possible implementation of the second aspect.

[0031] In a sixth aspect, the present application provides a computer program product, which, when executed on a computing device, enables the computing device to execute the method implemented by the first switching node in the first aspect or any possible implementation of the first aspect.

[0032] In a seventh aspect, the present application provides a computer program product, which, when executed on a device, enables the device to execute the method implemented by the host in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the background technology, the drawings required for use in the embodiments of the present invention or the background technology will be described below.

[0034] Figure 1 is a schematic diagram of a computing device cluster provided in an embodiment of the present application;

[0035] Figure 2 is a schematic diagram of a distributed computing system provided by an embodiment of the present application;

[0036] Figure 3 is a schematic diagram of another distributed computing system provided in an embodiment of the present application;

[0037] Figure 4 This is a schematic diagram of data partitioning within a computing node provided in an embodiment of the present application;

[0038] Figure 5 It is an interactive schematic diagram of a data processing method provided in an embodiment of the present application;

[0039] Figure 6 It is a schematic diagram of a host generating task information provided by an embodiment of the present application;

[0040] Figure 7 It is a schematic diagram of network initialization provided by an embodiment of the present application;

[0041] Figure 8 is an interactive schematic diagram of another data processing method provided in an embodiment of the present application;

[0042] Fig. 9 is a schematic diagram of a switching node provided in an embodiment of the present application;

[0043] Fig.10 is a schematic diagram of a host provided in an embodiment of the present application;

[0044] Fig.11 It is a schematic diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] With the development of high performance computing (HPC) and artificial intelligence (AI) technologies, many large-scale applications have emerged, such as AI model training. The scale of data that needs to be processed by large-scale applications is also increasing. In order to solve the computing problem of large-scale data, distributed computing came into being. Distributed computing processes data through multiple computing nodes, and then completes the calculation by exchanging data between multiple computing nodes. Collective communication (such as Allreduce, Allgather, etc.) is a common method for realizing data interaction between multiple computing nodes in distributed computing, and has a large number of important applications in distributed computing.

[0046] For example, in AI model training, when the data scale of the training set is large, multiple computing nodes can use data parallelism to perform model training. Each computing node includes a complete AI model that needs to be trained. The data in the training set is divided into multiple subsets, and a computing node trains the model based on one of the subsets. In the process of model training by the above multiple computing nodes, after the back propagation of each iterative process, the gradient data obtained by each computing node needs to be Allreduce to ensure that the model parameters of the models in each computing node remain consistent in the next iteration.

[0047] However, current collective communication has problems such as large amount of data transmission and many data synchronization times. For example, when implementing Allreduce through ring-Allreduce or halving-doubling Allreduce, the actual amount of data transmitted between multiple computing nodes is twice the original data. Therefore, the collective communication method currently used in distributed computing will affect the efficiency of distributed computing.

[0048] The present application provides a distributed computing system and a data processing method applied to the distributed computing system. The distributed computing system includes multiple computing nodes and at least one first switching node, and each first switching node is connected to at least one computing node among the multiple computing nodes. In the distributed computing system, when multiple computing nodes need to perform collective communication during distributed computing, data aggregation can be achieved in the switching node, and then the results of data aggregation are sent to the computing nodes, thereby reducing the amount of data that needs to be transmitted and the number of data synchronization times when the distributed computing system performs data aggregation through collective communication, thereby improving the computing efficiency of distributed computing.

[0049] The distributed computing system provided by the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0050] The present application provides a computing device cluster, such as Figure 1 As shown, Figure 1 : is a schematic diagram of a computing device cluster provided in an embodiment of the present application. The computing device cluster includes one or more computing devices. When the computing device cluster includes multiple computing devices, the multiple computing devices are connected through a network. The network can be an operator network or a network composed of optical cables and data transmission equipment. The embodiment of the present application does not make specific restrictions.

[0051] The above computing device cluster can form a distributed computing system, and each computing device in the cluster can also be used independently as a distributed computing system, such as Figure 2 As shown, Figure 21 is a schematic diagram of a distributed computing system provided by an embodiment of the present application. The distributed computing system includes a host, multiple computing nodes, and at least one first switch node. The multiple computing nodes are all connected to the host; each of the at least one first switch node is connected to at least one computing node among the multiple computing nodes. Figure 2 As shown in Figure 2 In the example, a distributed computing system includes four computing nodes (C0-C3) and three first switching nodes (S0-S2), and the four computing nodes and the three first switching nodes are connected in a fully connected manner.

[0052] like Figure 3 As shown, Figure 3 1 is a schematic diagram of another distributed computing system provided by an embodiment of the present application. The distributed computing system includes a host, multiple computing nodes, at least one first switching node, and at least one second switching node. The multiple computing nodes are all connected to the host; each of the at least one first switching node is connected to at least one computing node among the multiple computing nodes; each of the at least one second switching node is connected to some of the first switching nodes among the at least one first switching node. Figure 3 As shown, Figure 3 In the example, the distributed computing system includes 8 computing nodes (C0-C7), 6 first switching nodes (S0-S5) and 6 second switching nodes (S6-S11), and the connection relationship between the multiple computing nodes and at least one first switching node and at least one first switching node and at least one second switching node is shown in Figure 3 .

[0053] In an embodiment of the present application, the above-mentioned host can be a central processing unit (CPU) of a computing device; the above-mentioned computing node can be a graphics processing unit (GPU), a neural network processor (NPU), a tensor processing unit (TPU) or a dedicated AI processing chip, etc.; the above-mentioned switching node can be a switching chip, or it can be a switching device with switching function such as a switch.

[0054] It should be understood that the above Figure 2 and Figure 3The figure shows the logical connection relationship between the host, the computing node and the switching node. The above-mentioned multiple computing nodes can be computing nodes located in the same physical device, or they can be computing nodes located in different physical devices. When the multiple computing nodes are located in multiple different physical devices, the multiple physical devices can be physical devices located in the same cabinet, for example, Figure 3 The computing nodes C0-C3 shown are located in the same server, and the computing nodes C4-C7 are located in another server. The two servers are located in the same cabinet and are connected through the switching node in the cabinet. When the above-mentioned multiple computing nodes are computing nodes located in the same physical device, the above-mentioned switching node can be a switching chip in the physical device, or a switching device different from the physical device.

[0055] It should be understood that the above combination Figure 2 and Figure 3 What is introduced is only an example of the topological structure of the distributed computing system provided by the present application, which cannot be understood as a specific limitation. The distributed computing system can also be other topological structures. For example, the distributed computing system can also include more or fewer computing nodes or switching nodes, or include more layers of switching nodes. The connection relationship between the above-mentioned computing nodes and the switching nodes, and the connection relationship between at least one first switching node and at least one second switching node are only used as an example and cannot be understood as a specific limitation. There can also be other connection relationships between computing nodes and switching nodes, which are not specifically limited in the embodiments of the present application.

[0056] In the embodiment of the present application, when it is necessary to Figure 1 The computing device cluster shown, Figure 2 or Figure 3 When the distributed computing system shown executes a task, the host of each computing device in the computing device cluster obtains the services to be executed by each computing device and executes the corresponding services to be executed through each computing node of the distributed computing system. For a distributed computing system in a computing device, when multiple computing nodes are executing services and data aggregation is required, each computing node will send control information to the connected switching node. After receiving the control information sent by the computing node, the switching node obtains the required data from each computing node based on the control information for data aggregation and sends the results of data aggregation to each computing node.

[0057] For example, if the above task is an AI model training task, the business to be executed by each computing device is to perform model training in a data parallel manner. Figure 2The distributed computing system shown in the figure performs data parallel model training. The same model will be loaded in each computing node C0~C3. Different computing nodes train the model with different training data. After each computing node completes one iteration of back propagation, the gradient data obtained in each computing node needs to be Allreduce. Figure 2 Each computing node is connected to three first switching nodes S0~S2. In the case of Allreduce, the gradient data in each computing node can be divided into three parts. Each first switching node obtains a corresponding gradient data from a computing node, and the three first switching nodes are used to synchronously aggregate the gradients, so as to make full use of the resources of each first switching node for data aggregation and improve the computing efficiency of the distributed computing system. Figure 4 As shown in Figure 4 This is a schematic diagram of data partitioning within a computing node provided by an embodiment of the present application. The gradient data in computing node C0 can be divided into three data parts: A0, B0, and D0; the gradient data in computing node C1 can be divided into three data parts: A1, B1, and D1; the gradient data in computing node C2 can be divided into three data parts: A2, B2, and D2; and the gradient data in computing node C3 can be divided into three data parts: A3, B3, and D3.

[0058] Each computing node sends control information to the three first switching nodes to which it is connected. Each control information includes a communication task identifier. Each communication task identifier is used to indicate a communication task. The communication task identifiers in the control information sent by each computing node to the same first switching node are the same. After receiving the control information, each first switching node obtains data from the computing node that sent the control information based on the control information. When a first switching node receives the control information sent by the above four computing nodes, it obtains corresponding gradient data from the four computing nodes, calculates the obtained gradient data to obtain data calculation results, and sends the data calculation results to each computing node.

[0059] For example, after receiving the control information of the computing node C0, the first switching node S0 sends one or more data read requests to the computing node C0 to obtain data A0 from the computing node C0. S0 can obtain data A1, data A2 and data A3 from other computing nodes respectively by the same method. Then, data A0, data A1, data A2 and data A3 are aggregated and calculated to obtain the aggregated data calculation result A0123, and then the data calculation result A0123 is sent to each computing node. Similarly, the switching node S1 can obtain data B0, data B1, data B2 and data B3 from the above four computing nodes, and calculate the data calculation result B0123. The switching node S2 can obtain data D0, data D1, data D2 and data D3 from the above four computing nodes, and calculate the data calculation result D0123, and then send the obtained data calculation result to the corresponding computing node, so that the gradient data in each computing node are finally the same.

[0060] Since the model needs to undergo multiple rounds of iterative training, the above method can be used to synchronize the gradient data in each computing node before the next iteration. After obtaining new gradient data through back propagation of the next iterative training, the above method can be used again to aggregate the gradient data between the computing nodes and the exchange nodes.

[0061] The above is an introduction to the data processing method provided by the present application, taking the distributed computing system as an example of performing model training through data parallelism and performing data aggregation on the gradient data after back propagation of the model training. It should be understood that the above distributed computing system can also be used to aggregate other data in data parallelism, and can also be used in other scenarios that require data aggregation through collective communication, such as scenarios of model training through model parallelism, which will not be described one by one here.

[0062] It should be noted that each time the distributed computing system executes a pending business, not all computing nodes in the distributed computing system necessarily participate in the execution. Figure 2 In the distributed computing system shown, when executing a business, only three computing nodes are used to participate in the execution of the business, and when executing another business through the distributed computing system, four computing nodes are used to participate in the execution of the business. When introducing the data processing method provided by the present application, for the convenience of description, the present application takes the example that all the computing nodes shown in the figure participate in the execution as an example.

[0063] The following describes in detail the data processing method based on the above distributed computing system in combination with the above distributed computing system and the accompanying drawings. Figure 5 As shown, Figure 5 It is an interactive schematic diagram of a data processing method provided in an embodiment of the present application.

[0064] S501. The host obtains topology information of the distributed computing system, and generates operator information according to the topology information and the business to be executed.

[0065] In the present application embodiment, Figure 2 The distributed computing system shown in the figure is used as an example to introduce the data processing method provided by the embodiment of the present application. Figure 6 As shown in . After the distributed computing system starts a business to be executed, the host can obtain the topology information of the distributed computing system from the management node, and the topology information includes the connection relationship between each computing node and each first switching node in the distributed system. Then the host generates operator information according to the computing node that executes the business to be executed and the topology information, and the above operator information includes computing operators and communication operators. When the computing operator is executed, it is used to implement the computing operation in the business to be executed, and when the communication operator is executed, it is used to implement the data sending or receiving operation between the computing node and other nodes.

[0066] Since in the process of executing the above-mentioned business to be executed through a distributed computing system, collective communication between various computing nodes will be involved, that is, the above-mentioned communication operator is used to realize collective communication between various computing nodes. In the embodiment of the present application, collective communication between various computing nodes is realized by switching nodes. In order to make full use of the resources of the first switching nodes connected to each computing node, the embodiment of the present application realizes the operation corresponding to a communication operator through multiple first switching nodes connected to each computing node. Therefore, after the host obtains the above-mentioned topology information, the host determines the number N of the first switching nodes connected to each computing node based on the above-mentioned topology information, and then splits the operation corresponding to each communication operator into m*N communication tasks, each first switching node corresponds to at least one of the m*N communication tasks, or each first switching node corresponds to m of the m*N communication tasks, and the operation corresponding to the communication operator is realized by the m*N communication tasks corresponding to the N first switching nodes, where m is a positive integer. In the embodiment of the present application, the data processing method provided by the present application is introduced by taking m equal to 1 as an example, that is, the operation corresponding to each communication operator is split into N communication tasks.

[0067] For example, the above Figure 4 As an example of the distributed computing system shown in the figure, the above-mentioned business to be executed is to train the AI ​​model in a data parallel manner, and Figure 4 The AI ​​model is trained on the four computing nodes in the network. After each computing node completes one iteration of back propagation, the gradient data obtained in each computing node needs to be Allreduce, that is, the communication operator corresponding to Allreduce is executed. Figure 4Each computing node is connected to three first switching nodes S0~S2, then the operation performed by the communication operator corresponding to Allreduce in each computing node can be split into three communication tasks; that is, the gradient data in each computing node is divided into three parts. Each of the three first switching nodes obtains a corresponding gradient data from a computing node; that is, each first switching node corresponds to one of the three communication tasks. The three first switching nodes can obtain the complete gradient data in a computing node, that is, the three communication tasks corresponding to the three first switching nodes implement the operations corresponding to the Allreduce communication operator.

[0068] By splitting the communication operator into N communication tasks, each communication task is essentially a first switching node obtaining part of the data in a computing node, each computing node is connected to multiple first switching nodes, and each first switching node is used to process part of the data in the connected computing nodes, thereby reducing the amount of data that each first switching node needs to process, making full use of the resources of the switching nodes in the distributed computing system, and improving the efficiency of data processing in the distributed computing system.

[0069] In the embodiment of the present application, before S501, the network initialization of the distributed computing system is also included. For example, before the distributed computing system is used to execute a service for the first time, the network initialization of the distributed computing system needs to be performed. Figure 7 As shown, Figure 7 is a schematic diagram of network initialization provided by an embodiment of the present application. The management node can obtain the topology information of the distributed computing system by scanning the network of the distributed computing system, and then complete the routing configuration of the network in the distributed computing system based on the topology information. The above topology information includes the connection relationship between each computing node and each first switching node in the distributed computing system. If the distributed computing system includes multiple layers of switching nodes, for example Figure 3 In the distributed computing system, the distributed computing system includes two layers of switching nodes, and the above topology information also includes the connection relationship between each switching node between two adjacent layers of switching nodes. For example, a second switching node is connected to multiple first switching nodes through multiple ports, and the topology information includes the corresponding connection relationship between the ports of the second switching node and the first switching node.

[0070] After the management node scans the network to obtain the above topology information, it sends the above topology information to each switching node. After each switching node receives the topology information, it completes the initialization configuration of each switching node. For example, Figure 7As shown in , when the above-mentioned distributed computing system includes multiple layers of switching nodes, each switching node will generate a mapping relationship between logical ports and physical ports based on topology information, and allocate a queue and queue ID to each logical port. Among them, a logical port includes one or more physical ports, and the physical ports included in a logical port are connected to a switching node of an adjacent layer. After the switching node completes the initialization configuration, the communication resource information is exchanged between the connected switching nodes, and the communication resource information includes a queue ID, which is used to indicate the queue ID when a switching node communicates with another switching node. For example, Figure 3 In the embodiment, a second switching node sends a queue ID for communicating with a first switching node to the first switching node, and the first switching node also sends a queue ID for communicating with the second switching node to the second switching node. It should be noted that the above-mentioned management node can be a management device different from the computing device to which the host belongs, and the management node can also be a software module deployed on the computing device described in the host, which is not specifically limited in the embodiments of the present application.

[0071] S502. The host generates first control information corresponding to each computing node.

[0072] In the above S501, the communication operator is divided into N communication tasks, and the operation of executing the communication operator is converted into executing N communication tasks, and each communication task corresponds to a first switching node obtaining a part of data from a computing node. Therefore, when the computing node executes the communication operator, the computing node needs to send the first control information to the connected first switching node, so that the first switching node that receives the first control information obtains data from the computing node that sends the first control information.

[0073] In an embodiment of the present application, after splitting each communication operator into N communication tasks, the host generates corresponding first control information for each communication task, and the first control information includes a communication task ID for uniquely identifying a communication task. It should be understood that each communication operator executed by each computing node is converted into N communication tasks, and the purpose of each communication task is to enable the first switching node that receives the first control information to obtain data from the computing node that sends the first control information and perform data aggregation. Therefore, the data obtained by a first switching node based on the first control information sent by different computing nodes should be able to be aggregated. In an embodiment of the present application, the communication task IDs in multiple first control information used by a first switching node to obtain aggregated data from multiple computing nodes are the same, wherein the aggregated data is data that can be aggregated and obtained by a first switching node from different computing nodes. For example, the above Figure 4In the figure, A0, A1, A2 and A3 are aggregated data, and the communication task IDs used by S0 to obtain the four first control information of A0, A1, A2 and A3 are the same.

[0074] S503. The host sends task information corresponding to the business to be executed to each computing node.

[0075] The host converts the to-be-executed business into computing operators and communication operators that need to be executed by each computing node through the methods in S501 and S502 above, and generates communication tasks corresponding to each communication operator and first control information corresponding to each communication task. Then, the host obtains task information corresponding to the to-be-executed business, which includes operator information and first control information. Then, the host sends the above-generated operator information and the first control information corresponding to each communication task to the corresponding computing node.

[0076] S504. Each computing node sends first control information to the connected first switching node.

[0077] Taking any computing node in the distributed computing system as an example, for example, the first computing node, the first computing node is connected to N first switching nodes. When the first computing node executes the communication operator, the first computing node sends the first control information to the N first switching nodes connected to the first computing node. Figure 4 In the example, the computing node C0 sends the first control information to the first switching nodes S0, S1 and S2 respectively; the computing node C1 sends the first control information to the first switching nodes S0, S1 and S2 respectively.

[0078] S505. The first switching node sends a data read request to the computing node connected to it.

[0079] In an embodiment of the present application, the first control information also includes address information, which includes the address of data that needs to be transmitted by a communication task, that is, the address information is used to indicate the location where the first switching node reads data from the computing node that sends the first control information. After receiving the first control information sent by a computing node, the first switching node generates one or more data read requests according to the address information in the first control information; when multiple data read requests are generated, each data read request is used to read the part of the data indicated by the first control information. Then, the one or more data read requests are sent to the corresponding computing node to read the corresponding data from the computing node. For example, the above Figure 2In the first control information sent by C1 to S0, the starting address and data length of data A1 are included. S0 generates one or more data read requests according to the starting address, data length and maximum payload size (MPS) of A1. For example, if the data length of data A1 is 1000 bytes and the above MPS is 125 bytes, S0 will generate 8 data read requests, and the data read by each data read request is only a part of data A1. Each data read request includes a starting address and a read length, which is used to read a section of data starting from the starting address with a length of the read length. It should be noted that the first switching node can also generate one or more data read requests according to the starting address, data length and maximum read request size (MRRS), which is not specifically limited in the embodiment of the present application. The first switching node needs to generate one or more read requests according to the first control information sent by each computing node, that is, one or more read requests generated according to the first control information sent by a computing node are used to obtain the data indicated to be read in the first control information from the computing node. In the case where multiple data read requests need to be generated, a data read request can only read the part of the data indicated to be read by the first control information.

[0080] In a possible implementation, the first control information also includes computing node information, which is used to indicate the number of computing nodes connected to each first switching node. Before sending a data read request, each first switching node also needs to confirm whether it has received the first control information sent by all the computing nodes connected to it according to the computing node information. The first switching node will generate the above one or more data read requests only when it receives the first control information sent by all the computing nodes connected to it. For example, after the first switching node receives a first control information with a communication task ID of 01, it determines that it is connected to 4 computing nodes according to the computing node information in the first control information. After receiving 4 first control information with a communication representation of 01, the first switching node confirms that it has received the first control information sent by all the computing nodes connected to it. Due to the limited storage resources of the switching node, the first switching node sends a data read request to the computing node only after it determines that it has received the first control information of all the computing nodes connected to it, which can avoid the problem of reading data after receiving a first control information, resulting in the need to cache the first read data for a long time. It should be understood that the computing node information actually indicates the number of computing nodes connected to each first switching node and used to execute the current business to be executed. In the present application, it is taken as an example that the computing nodes in the distributed computing device all participate in executing the business. Therefore, the number of computing nodes connected to each first switching node and used to execute the current business to be executed is the same as the number of all computing nodes connected to the first switching node.

[0081] Optionally, the computing node information may be a numerical value, that is, a numerical value is directly used to indicate the number of computing nodes connected to each first switching node. The computing node information may also be in other forms, for example, the computing node information includes k bits, where k is the number of ports included in the first switching node, and each bit corresponds to a port of the first switching node. When a port is connected to a computing node, the bit position corresponding to the port is set to 1, and the remaining bits are set to 0. Then, after receiving the first control information, the first switching node can determine the number of computing nodes connected to it, that is, the number of computing nodes participating in the execution of the service to be executed, according to the number of bits set to 1 in the computing node information.

[0082] The first switching node can also confirm whether it has received the first control information sent by all the computing nodes connected to it in other ways, which is not specifically limited in the embodiments of the present application. For example, the computing node information includes k bits, where k is the number of ports included in the first switching node, and each bit corresponds to a port of the first switching node. When a computing node is connected to a port, the bit position corresponding to the port is set to 1, and the remaining bits are set to 0. After receiving a first control information, the first switching node can determine which ports need to receive the first control information based on the bit position set to 1. After the first switching node receives the first control information from these ports, the first switching node confirms that it has received the first control information sent by all the computing nodes connected to it.

[0083] In an embodiment of the present application, when a distributed computing system is capable of executing multiple services simultaneously, after receiving the first control information sent by any computing node, the first switching node needs to first determine whether its own computing resources are available based on the first control information. Only when the first switching node confirms that its own computing resources are available will the first switching node send a data read request to the computing node connected to it. For example, each first switching node is configured to support 3 concurrencies at the same time. If a first switching node S0 is already processing 3 concurrencies when receiving the first control information, and it is determined that the first control information corresponds to a new concurrent request based on the communication task ID, then S0 determines that its own computing resources are unavailable and returns information about insufficient resources to the computing node. If S0 is processing 2 concurrencies at the same time when receiving the first control information, and S0 confirms that its own computing resources are available, then S0 sends a data read request to the computing node connected to it.

[0084] It should be noted that after receiving the first control information sent by a computing node, the first switching node can confirm whether its own computing resources are available. When a first switching node confirms that its own computing resources are unavailable, it needs to receive the first control information sent by all the computing nodes connected to it before returning the information of insufficient resources to the computing node, so as to avoid the situation that when the first switching node receives the first control information sent by the first computing node, it confirms that its own computing resources are unavailable, and after receiving the first control information sent by the remaining computing nodes, it determines that its own computing resources are available.

[0085] S506. Each computing node obtains corresponding data according to the received data reading request, and sends the obtained data to the first switching node.

[0086] After receiving the data read request, each computing node obtains corresponding data according to the starting address and the read length in each data read request, and sends the data to the corresponding first switching node.

[0087] S507. The first switching node obtains a data calculation result based on the data obtained from each computing node, and sends the data calculation result to the computing node connected to it.

[0088] After the first switching node obtains data from each computing node according to the above one or more data read requests, it aggregates the obtained data and then sends the aggregated data to each computing node. For example, if the first switching node can read the data indicated by the first control information through a data read request, such as the first switching node obtains data A0 from C0, obtains data A1 from C1, obtains data A2 from C2, obtains data A3 from C3, and then calculates data A0, A1, A2, A3 to obtain data calculation results, and sends the data calculation results to the four computing nodes respectively. For example, if the above communication task is Allreduce, the first switching node sums A0, A1, A2, A3 and sends them to the four computing nodes. Among them, the first switching node can perform a summation after obtaining the data returned by the two computing nodes, or it can wait until all four data are obtained before summing, which is not specifically limited in the embodiments of the present application.

[0089] Combination of the above Figure 2 The present invention introduces a data processing method when a distributed computing system includes a layer of switching nodes. The data processing method provided by the present application can also be applied to a distributed computing system including multiple layers of switching nodes. For example, Figure 3 The distributed computing system including two layers of switching nodes shown in the figure includes a host, multiple computing nodes, at least one layer of switching nodes and at least one layer of switching nodes. When the distributed computing system includes the at least one second switching node, the data read by the first switching node through the data read request also needs to be sent to the second switching node connected thereto for processing. Figure 8 As shown, Figure 8 It is a schematic diagram of another data processing method provided by an embodiment of the present application. In the case where the distributed computing system includes two layers of switching nodes, the operations performed by the host and computing nodes of the distributed computing system can refer to the above S501 to S504, and after each computing node sends the first control information to the connected first switching node, it also includes the following S508 to S515.

[0090] S508. The first switching node generates second control information according to the first control information.

[0091] After the first switching node confirms receipt of the first control information sent by all connected computing nodes according to the computing node information, it also needs to generate second control information according to the first control information and send the second control information to the second switching node. The second control information includes the communication task ID in the first control information.

[0092] S509. The second switching node generates notification information according to the received second control information, and sends the notification information to the first switching node.

[0093] The notification information includes the communication task ID, so that the first switching node receiving the notification information confirms the corresponding first control information and the address information and data length in the first control information according to the communication task ID, and then generates the one or more data read requests.

[0094] In a possible implementation, the second switching node needs to confirm that it has received the second control information sent by all the first switching nodes connected to it before sending the above notification information to the first switching node. In the case where the second switching node needs to confirm whether it has received the second control information sent by all the first switching nodes connected to it, the above first control information also includes first switching node information, and the first switching node information is used to indicate the number of first switching nodes connected to the second switching node. The form of the first switching node information can refer to the computing node information in the above first control information, which will not be repeated here.

[0095] In an embodiment of the present application, when the distributed computing system is capable of executing multiple services simultaneously, after receiving the second control information sent by any first switching node, the second switching node needs to first determine whether its own computing resources are available based on the second control information. Only when the second switching node confirms that its own computing resources are available will the second switching node send a notification message to the first switching node to which it is connected. Among them, the method for the second switching node to confirm whether its own computing resources are available can refer to the method for the first switching node to confirm whether its own resources are available, which will not be repeated here. Since the storage resources of the switching node are limited, the second switching node will send a notification message to the first switching node only after determining that the first control information of all the first switching nodes to which it is connected has been received, which can avoid the problem of notifying the first switching node to read data upon receiving a second control information, resulting in the need to cache the first read data for a long time to occupy storage resources.

[0096] It should be noted that after receiving the second control information sent by a first switching node, the second switching node can confirm whether its own computing resources are available. When a second switching node confirms that its own computing resources are unavailable, it needs to receive the second control information sent by all the first switching nodes to which it is connected before returning the information that the resources are insufficient to the first switching node, so as to avoid the situation where the second switching node confirms that its own computing resources are unavailable when it receives the second control information sent by the first first switching node, and then determines that its own computing resources are available after receiving the second control information sent by the remaining first switching nodes.

[0097] S510. The first switching node generates the above data reading request according to the notification information, and sends the data reading request to the computing node connected to it.

[0098] The notification information includes the communication task ID. The first switching node that receives the notification information confirms the corresponding first control information and the address information and data length in the first control information according to the communication task ID, and then generates the one or more data read requests. The method for the first switching node to generate a data read request can refer to S505 above.

[0099] In a possible implementation, the notification information includes a communication task ID, an offset, and a first data length; wherein the first data length is used to indicate the length of the data read by the first switching node this time. The first switching node that receives the notification information confirms the corresponding first control information according to the communication task ID. The first switching node determines the starting address of the data to be read according to the address information in the first control information, determines the starting address of the data to be read this time according to the starting address and the offset, and then generates a data read request according to the starting address and the first data length of the data to be read this time. It should be understood that the first switching node needs to read data from multiple computing nodes connected to it. Therefore, each time the first switching node receives the notification information, it will generate a data read request according to the received notification information and the first control information sent by each computing node connected to it, and then send the data read to the corresponding computing node. It should be understood that the notification information is equivalent to a read request sent by the second switching node to the first switching node, which is used to read the data of the first data length from the first switching node each time. By actively reading data from the first switching node by the second switching node, the congestion problem at the second switching node that may be caused by the first switching node sending data to the second switching node can be avoided.

[0100] For example, the above Figure 3In the example, S0 needs to read data from C0, C1, C2, and C3. Then, S0 generates a data read request corresponding to C0 according to the notification information and the first control information sent by C0, for reading data from C0; S0 generates a data read request corresponding to C1 according to the notification information and the first control information sent by C1, for reading data from C1; S0 generates a data read request corresponding to C2 according to the notification information and the first control information sent by C2, for reading data from C2; S0 generates a data read request corresponding to C3 according to the notification information and the first control information sent by C3, for reading data from C3. It should be understood that one notification information is used to read part of the data that needs to be read from the computing node, and the second switching node will send multiple notification information to the first switching node to read all the data that needs to be read from the computing node.

[0101] When the distributed computing system includes the above-mentioned at least one second switching node, the data read by the first switching node through the above-mentioned data reading request also needs to be sent to the second switching node connected to it for processing. Therefore, the first switching node also needs to send the second control information obtained based on the above-mentioned first control information to the second switching node. After the second switching node sends a notification message to the first switching node, indicating that the first switching node and the second switching node can process the data, the first switching node will obtain the data from the computing node. This can avoid the problem of wasting bandwidth or causing calculation errors when the resources of the second switching node are unavailable and the second switching node cannot process the data after the first switching node reads the data.

[0102] S511. The first switching node sends a data read request to the computing node connected to it.

[0103] S512. Each computing node obtains corresponding data according to the received data reading request, and sends the obtained data to the first switching node.

[0104] After receiving the data read request, each computing node obtains corresponding data according to the starting address and the read length in each data read request, and sends the data to the corresponding first switching node.

[0105] S513. The first switching node obtains a first calculation result based on the data obtained from each computing node, and sends the first calculation result to the second switching node.

[0106] After acquiring data from each computing node, the first switching node performs corresponding calculations to obtain a first calculation result, and then sends the first calculation result to the second switching node.

[0107] S514. The second switching node performs calculations according to the first calculation results sent by all the first switching nodes connected to it, obtains a data calculation result, and sends the data calculation result to the first switching node connected to it.

[0108] S515. The first switching node sends the data calculation result to the respective connected computing nodes.

[0109] The first switching node generates the above one or more data read requests according to the notification information, and obtains the data indicated by the first control information to be read from the computing node. After obtaining the data read by the data read request, a first switching node calculates the data read from different computing nodes connected to it to obtain a first calculation result, and then sends the first calculation result to the second switching node that sent the notification information to it. The second switching node calculates according to the first calculation results sent by all the first switching nodes connected to it, obtains the data calculation result, and sends the data calculation result to the first switching node connected to it, and then the first switching node sends it to the computing nodes connected to it. For example, S0 obtains data A0 from C0, data A1 from C1, data A2 from C2, and data A3 from C3, and then calculates data A0, A1, A2, and A3 to obtain an intermediate calculation result A0123; S1 obtains data A4 from C4, data A5 from C5, data A6 from C6, and data A7 from C7, and then calculates data A4, A5, A6, and A7 to obtain an intermediate calculation result A4567, and then sends the two intermediate calculation results to S6 respectively, and S6 calculates the two intermediate calculation results to obtain the above data calculation results.

[0110] Since computing resources and storage resources in the switching nodes are limited, in a distributed computing node including multiple layers of switching nodes, each layer of switching nodes performs corresponding computing operations based on the acquired data, and can realize the aggregation calculation of a large amount of data through multiple switching nodes, thereby improving the efficiency of data aggregation in the distributed computing system.

[0111] In a possible implementation, a first switching node and a second switching node can be connected through one or more physical ports. When a first switching node and a second switching node are connected through multiple physical ports, if the communication protocol between the two switching nodes supports multi-path load balancing, the communication protocol implements load balancing during data transmission. If the communication protocol between the two switching nodes does not support multi-path load balancing, the switching node implements multi-path load balancing, for example, in a polling manner, sending data through different physical ports each time.

[0112] In a possible implementation, the first control information also includes a first address, which is used to write a flag bit to the position indicated by the first address after the first switching node or the second switching node completes data aggregation to obtain a data calculation result, so as to notify the computing node that the operation corresponding to the communication operator has been completed, or to notify the computing node that the operation corresponding to the communication operator has been completed by sending a message.

[0113] In one possible implementation, the first control information also includes a second address, which is used to write a flag bit to the position indicated by the address after the first switching node and the second switching node confirm that their own computing resources are available, so as to notify the computing node that the computing resources of the switching node are available, or to notify the computing node by sending a message that the operation corresponding to the communication operator has been completed.

[0114] Above Figure 5 and Figure 8 In the corresponding embodiment, the first switching node performs a computing operation on the data read from the multiple computing nodes, and the second switching node performs a computing operation on the data read from the multiple first switching nodes. It should be understood that in other application scenarios, the switching nodes (including the first switching node and the second switching node) in the distributed computing system may not calculate the acquired data, and the switching nodes are only used to synchronize the data between the multiple computing nodes. The distributed computing system and the data processing method provided by the present application can reduce the number of times that the multiple computing nodes send data to each other, improve the efficiency of data synchronization, and thus improve the computing efficiency. In addition, the first switching node actively reads data from the multiple computing nodes, which can avoid the congestion at the first switching node caused by the computing node actively sending data; the second switching node actively reads data from the multiple first switching nodes, which can avoid the congestion at the second switching node caused by the multiple first switching nodes actively sending data.

[0115] By implementing the distributed computing system and data processing method provided by the present application, when multiple computing nodes are performing distributed computing and collective communication is required, the computing nodes send control information to the switching nodes, and the switching nodes that receive the control information actively obtain the data that needs to be aggregated from the computing nodes, aggregate the data in the switching nodes, and then send the aggregated results to the computing nodes, thereby reducing the amount of data that needs to be transmitted and the number of synchronization times when the distributed computing system aggregates data through collective communication, thereby improving the computing efficiency of distributed computing.

[0116] In addition, in the above-mentioned distributed computing process, after the computing node sends control information to the switching node, the switching node will send a data reading request to obtain the data to be aggregated and perform data aggregation on the switching node. After the data is aggregated, the result of the data aggregation is sent to the node that sent the control information. Before and after this process, the switching node can be used to process other tasks, that is, the aggregation task triggered by the control information will not occupy the computing resources of the switching node for a long time without releasing them. That is, for a business, only when collective communication is required, the business will occupy the resources of the switching node for data aggregation, and the computing node does not occupy the resources of the switching node during the execution of the business. Therefore, through the data processing method provided by the present application, when the distributed computing system executes multiple businesses, the resources of the switching node can be fully utilized.

[0117] For the above method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described order of actions. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the present invention. Other reasonable step combinations that can be thought of by those skilled in the art based on the above description also fall within the scope of protection of the present invention.

[0118] The present application also provides a data processing system, which includes the above-mentioned Figure 2 or Figure 3 The distributed computing system shown in FIG. Figure 2 or Figure 3 The corresponding introduction and related instructions are not repeated here. The operations performed by the host, computing node and switching node in the distributed computing system can refer to the above Figures 4 to 8 The relevant introduction in the corresponding embodiment will not be repeated here.

[0119] The present application also provides a switching node, such as Fig. 9 As shown, the switching node 900 includes a communication control module 910, a processing module 920 and a switching architecture 930. The switching node is applied to Figure 2 or Figure 3The distributed computing system shown in the figure includes a plurality of computing nodes and at least one switching node 900, each of the at least one switching node 900 is connected to at least one computing node of the plurality of computing nodes. The communication control module 910 is used to receive first control information sent by at least one computing node connected to the switching node 900; the first control information includes a communication task identification ID, and the communication task ID is used to indicate the communication task in which the at least one computing node participates. The processing module 920 is used to generate a data read request according to the first control information. The communication control module 910 is also used to send a data read request to at least one computing node connected to the switching node 900; the processing module 920 is also used to obtain a data calculation result, wherein the data calculation result is obtained according to the data read in the data read request; the communication control module 910 is also used to send the data calculation result to at least one computing node connected to the first switching node.

[0120] In one possible implementation, the above-mentioned distributed computing system also includes a second switching node, and the above-mentioned second switching node is connected to the above-mentioned switching node 900; the processing module 920 is also used to generate second control information based on the first control information, and the second control information includes the above-mentioned communication task ID; the communication control module 910 is also used to send the second control information to the second switching node; the processing module 920 is specifically used to: after receiving the notification information sent by the second switching node, generate a data read request according to the first control information; the notification information includes the communication task ID, and the notification information is used to instruct the switching node 900 to obtain data from at least one computing node connected to it.

[0121] In a possible implementation, the processing module 920 is further used to calculate the data read from at least one computing node to obtain a first calculation result; the communication control module 910 is further used to send the first calculation result to the second switching node; and receive the data calculation result calculated by the second switching node, wherein the data calculation result is a calculation result obtained by the second switching node based on the first calculation result and the first calculation results sent by other switching nodes in at least one switching node 900.

[0122] In a possible implementation, the communication control module 910 and the processing module 920 are logic circuits in a switching node. The internal micro-architecture of the processing module 920 can be implemented as a ring architecture, a dual-ring architecture, a single centralized control architecture, or a plurality of distributed centralized unit architectures.

[0123] The above-mentioned switching architecture 930 is used to implement data exchange between different ports of the switching node, for example, data received from a first port is sent out through a second port.

[0124] The operations that the switching node 900 can implement can refer to the above Figure 4-Figure 8 The operations performed by the first switching node or the second switching node in the corresponding embodiment. The communication control module 910 is used to implement the operations such as receiving data or sending data performed by the first switching node or the second switching node, and the processing module 920 is used to implement operations other than the above-mentioned receiving data or sending data, which will not be repeated here.

[0125] The present application also provides a host, such as Fig.10 As shown, Fig.10 1 is a schematic diagram of a host provided in an embodiment of the present application. The host 100 includes a processing module 101 and a communication module 102. The host 100 is applied to Figure 2 or Figure 3 The distributed computing system shown includes a host 100, a plurality of computing nodes and at least one first switching node, each of the at least one first switching node is connected to at least one computing node of the plurality of computing nodes. The processing module 101 may be an AI training framework such as mindspore or tensorflow.

[0126] The processing module 101 is used to obtain and analyze the topology information of the distributed system when the distributed computing system needs to process a pending service, for example, to confirm the number of first switching nodes connected to each computing node according to the topology information. The topology information includes the connection relationship between each computing node and each first switching node in the distributed computing system. Figure 3 In the case of the at least one second switching node shown, the topology information also includes a connection relationship between the at least one first switching node and the at least one second switching node.

[0127] The processing module 101 generates operator information according to the topology information of the distributed computing system and the business to be executed, and the operator information includes a computing operator and a communication operator. The processing module 101 is also used to convert the above communication operator into multiple communication tasks according to the topology information, configure a communication task ID for each communication task, and generate first control information corresponding to each communication task according to the communication task ID. Specifically, the operations implemented by the above processing module 101 can refer to the above Figure 5-Figure 8 The operations implemented by the host in the corresponding embodiment will not be repeated here.

[0128] The communication module 102 is used to send task information to each computing node in the distributed computing system. The task information includes the computing operators and communication operators that the computing nodes need to execute, the communication tasks corresponding to each communication operator, and the first control information corresponding to each communication task.

[0129] In a possible implementation, the host 100 further includes a management module 103, which is used to scan the network of the distributed computing system, obtain topology information of the distributed system, and complete routing configuration of the network in the distributed computing system based on the topology information.

[0130] The communication module 102 is also used to send the topology information obtained by the management module 103 to each switching node so that each switching node completes configuration, such as generating a mapping relationship between a logical port and a physical port, and allocating a queue and a queue ID to each logical port.

[0131] The operations implemented by the management module 103 can refer to the above Figure 5 The operations implemented by the management node in the illustrated embodiment will not be described in detail here.

[0132] Specifically, the operations implemented by the host 100 can refer to the above Figure 5-Figure 8 The operations implemented by the host in the corresponding embodiment will not be repeated here.

[0133] The present application also provides a computing device, such as Fig.11 As shown, Fig.11 1 is a schematic diagram of a computing device provided in an embodiment of the present application. The computing device 110 includes a host 111, multiple computing nodes 112, a communication interface 113, and a memory 114, wherein the host 111, multiple computing nodes 112, the communication interface 113, and the memory 114 are interconnected via a bus 115, wherein:

[0134] The host 111 may be a CPU, and the computing node 112 may be a GPU, an NPU, a TPU, or a dedicated AI processing chip. Figure 4-Figure 8 The operations performed by the host in the corresponding embodiment and the operations performed by the computing node 112 can refer to the above Figure 4-Figure 8 The operations performed by the computing nodes in the corresponding embodiments are not described in detail here.

[0135] The communication interface 113 may be a wired interface or a wireless interface for communicating with other modules or devices. The wired interface may be an Ethernet interface, a local interconnect network (LIN), etc., and the wireless interface may be a cellular network interface or a wireless local area network interface.

[0136] The memory 114 may be a non-volatile memory, such as a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The memory 114 may also be a volatile memory, which may be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0137] The memory 114 may also be used to store program instructions and data, so that the host 111 can call the program instructions stored in the memory 114 to execute the operation steps of the host in the above method embodiment. Figure 5 Show more or fewer components, or have different component configurations.

[0138] The bus 115 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus 115 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0139] The computing device 110 may further include the at least one switching chip 116. The operations performed by the switching chip 116 may refer to the operations performed by the first switching node or the second switching node in the above embodiment, which will not be described in detail herein.

[0140] Specifically, the specific implementation of various operations performed by the computing device 110 may refer to the specific operations performed by the distributed computing system in the above method embodiment, which will not be described in detail here.

[0141] An embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is executed on a processor, the method steps implemented by the evaluation device in the above method embodiment can be implemented. The specific implementation of the processor of the computer-readable storage medium in executing the above method steps can refer to the specific operations of the above method embodiment, which will not be repeated here.

[0142] An embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is executed on a processor, the method steps implemented by the management device in the above method embodiment can be implemented. The specific implementation of the processor of the computer-readable storage medium in executing the above method steps can refer to the specific operations of the above method embodiment, which will not be repeated here.

[0143] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0144] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium, or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0145] The steps in the method of the embodiment of the present application can be adjusted in order, combined or deleted according to actual needs; the modules in the device of the embodiment of the present application can be divided, combined or deleted according to actual needs.

[0146] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0147] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, characterized in that: Applied to a distributed computing system, the distributed computing system includes a plurality of computing nodes and at least one first switching node, each of the at least one first switching node is connected to at least one computing node of the plurality of computing nodes, the method includes: The first switching node receives first control information of at least one computing node connected to it, wherein the first control information includes a communication task identification ID, and the communication task ID is used to indicate a communication task in which the at least one computing node participates; The first switching node sends a data read request to the at least one computing node according to the first control information; The first switching node obtains a data calculation result, wherein the data calculation result is obtained according to the data read by the data read request; The first switching node sends the data calculation result to the at least one computing node.

2. The method according to claim 1, characterized in that The distributed computing system further includes a second switching node, the second switching node being connected to the first switching node; The first switching node sends a data read request to the at least one computing node according to the first control information, including: The first switching node sends second control information to the second switching node according to the first control information, wherein the second control information includes the communication task ID; After receiving the notification information sent by the second switching node, the first switching node sends the data reading request to the at least one computing node; the notification information includes the communication task ID, and the notification information is used to instruct the first switching node to obtain data from the at least one computing node.

3. The method according to claim 2, characterized in that The first switching node obtains a data calculation result, including: The first switching node calculates the data read from the at least one computing node to obtain a first calculation result; The first switching node sends the first calculation result to the second switching node; The first switching node obtains the data calculation result calculated by the second switching node, wherein the data calculation result is a collective calculation result obtained by the second switching node according to the first calculation result and calculation results sent by other switching nodes in the at least one first switching node.

4. The method according to any one of claims 1 to 3, characterized in that: The communication task is Allreduce, and the data calculation result is the sum, average, and maximum value result of the data obtained from the data read request.

5. The method according to any one of claims 1 to 4, characterized in that: The first control information further includes address information, wherein the address information indicates a location where the first switching node reads data from the computing node that sends the first control information; The data read by the data read request is part or all of the data instructed to be read by the first control information.

6. The method according to any one of claims 1 to 5, characterized in that: The first control information further includes computing node information, where the computing node information is used to indicate the number of the at least one computing node; The first switching node sends a data read request to the at least one computing node according to the first control information, including: After the first switching node confirms receipt of the first control information of the at least one computing node according to the computing node information, the first switching node sends a data read request to the first number of computing nodes according to the first control information.

7. The method according to any one of claims 1 to 6, characterized in that: The computing node is a graphics processor GPU, a neural network processor NPU, a tensor processor TPU or a dedicated artificial intelligence AI processing chip, and the first switching node is a switching chip or a switching device with a switching function.

8. The method according to any one of claims 1 to 7, characterized in that: The distributed computing system further includes a host, and the method further includes: The host generates operator information according to the topology information of the distributed computing system and the business to be executed, and the operator information includes the communication operator and the calculation operator in the business to be executed.

9. The method according to claim 8, characterized in that The method further comprises: The host determines, according to the topology information, that the number of first switching nodes connected to each computing node is N, where N is a positive integer; The host splits the communication operator into N communication tasks, and each first switching node corresponds to one communication task among the N communication tasks.

10. The method according to claim 8 or 9, characterized in that: The host sends the communication operator and N communication tasks to the multiple computing nodes.

11. A switching node, characterized in that: The invention comprises a communication control module and a processing module, wherein the switching node is located in a distributed computing system, the distributed computing system comprises a plurality of computing nodes and at least one switching node, and the switching node is connected to at least one computing node among the plurality of computing nodes; The communication control module is used to receive first control information sent by at least one computing node connected to the switching node, wherein the first control information includes a communication task identification ID, and the communication task ID is used to indicate a communication task in which the at least one computing node participates; The processing module is used to generate a data read request according to the first control information; The communication control module is further used to send a data read request to the at least one computing node; The processing module is further used to obtain a data calculation result, wherein the data calculation result is obtained according to the data read by the data reading request; The communication control module is further used to send the data calculation result to the at least one computing node.

12. The switching node according to claim 11, characterized in that: The distributed computing system further includes a second switching node connected to the switching node; The processing module is further configured to generate second control information according to the first control information, wherein the second control information includes the communication task ID; The communication control module is further used to send second control information to the second switching node; The processing module is specifically used to: after receiving the notification information sent by the second switching node, generate the data reading request according to the first control information; the notification information includes the communication task ID, and the notification information is used to instruct the switching node to obtain data from the at least one computing node.

13. The switching node according to claim 12, characterized in that: The processing module is further used to calculate the data read from the at least one computing node to obtain a first calculation result; The communication control module is further configured to send the first calculation result to the second switching node; The data calculation result calculated by the second switching node is received, wherein the data calculation result is a collective calculation result obtained by the second switching node according to the first calculation result and calculation results sent by other switching nodes in the distributed computing system.

14. The switching node according to any one of claims 11 to 13, characterized in that: The communication task is Allreduce, and the data calculation result is the sum, average, and maximum value result of the data obtained from the data read request.

15. The switching node according to any one of claims 11 to 14, wherein the first control information further comprises address information, wherein the address information indicates a location where the first switching node reads data from the computing node that sends the first control information; The data read by the data read request is part or all of the data instructed to be read by the first control information.

16. The switching node according to any one of claims 11 to 15, characterized in that: The first control information further includes computing node information, where the computing node information is used to indicate the number of the at least one computing node; The processing module is further configured to generate the data read request after confirming receipt of the first control information of the at least one computing node according to the computing node information.

17. The switching node according to any one of claims 11 to 16, characterized in that: The computing node is a graphics processor GPU, a neural network processor NPU, a tensor processor TPU or a dedicated artificial intelligence AI processing chip, and the switching node is a switching chip or a switching device with a switching function.

18. The switching node according to any one of claims 11 to 17, characterized in that: The communication control module and the processing module are logic circuits in the switching chip.

19. A distributed computing system, characterized in that: It includes multiple computing nodes and at least one first switching node, each of the at least one first switching node is connected to at least one computing node of the multiple computing nodes, each computing node is used to send first control information to the first switching node it is connected to, and each first switching node is used to execute the method described in the preceding claims 1-7.

20. The distributed computing system according to claim 19, characterized in that: It also includes a host, which is used to execute the method described in claims 8-10.

Images