An all-to-all communication system, a communication method and a computer device

Abstract

The application provides an all-to-all communication system, a communication method and a computer device. The all-to-all communication system comprises: a plurality of computing nodes connected by a 2D mesh topology network, the plurality of computing nodes being divided into a plurality of first communication modules and a plurality of second communication modules; in a first communication stage, the computing nodes in the first communication modules are used for aggregating first data slices to first transition nodes; the computing nodes in the second communication modules are used for sending the aggregated first data slices to first target nodes; in a second communication stage, the computing nodes in the second communication modules are used for aggregating second data slices to second transition nodes; and each computing node in the first communication modules is used for sending the second data slices aggregated by the computing nodes to second target nodes. The all-to-all communication system, the communication method and the computer device provided by the embodiment of the application improve communication efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, in particular to an all-to-all communication system, a communication method and a computer device. BACKGROUND

[0002] All-to-all communication is a common communication primitive in a collection communication library, which allows each participating device to distribute its local data to other devices, while receiving data from other devices, thereby realizing data exchange between multiple devices.

[0003] The all-to-all communication implementation scheme in the prior art is mainly based on a spine-leaf network structure. The spine-leaf network is a non-convergent full interconnection network topology, and there is a direct connection between any two computing nodes through an independent physical link to realize the communication between any two computing nodes. In the mesh topology network structure, the computing nodes are directly connected with adjacent nodes, and the communication between non-adjacent nodes needs to be routed by other nodes. Due to the great difference in network topology, the all-to-all communication implementation in the prior art cannot be applied to the mesh topology network structure. Therefore, how to realize all-to-all communication in the mesh topology network has become an important issue to be solved in the field. SUMMARY

[0004] In view of the problems in the prior art, the embodiments of the present application provide an all-to-all communication system, a communication method and a computer device, which can at least partially solve the problems in the prior art.

[0005] In a first aspect, the present application provides an all-to-all communication system, comprising a plurality of computing nodes, wherein the plurality of computing nodes are communicatively connected through a 2D mesh topology network, and wherein:

[0006] The plurality of computing nodes are divided into a plurality of first communication modules according to a first division rule, and divided into a plurality of second communication modules according to a second division rule;

[0007] In the first communication stage, each computing node in each first communication module is configured to aggregate a first data slice of the computing node to a corresponding first transition node based on a first data aggregation rule, and each first communication module performs communication in parallel; each computing node in each second communication module is configured to send a first data slice aggregated by each computing node to a corresponding first target node based on a first data dissemination rule, and each second communication module performs communication in parallel; wherein each computing node in the first communication module serves as a first transition node.

[0008] In the second communication stage, each computing node in the second communication module is configured to aggregate the respective second data slice to a corresponding second transition node based on a second data aggregation rule, and each second communication module performs communication in parallel; each computing node in the first communication module is configured to send the second data slice aggregated by each computing node to a corresponding second target node based on a second data dissemination rule, and each first communication module performs communication in parallel; wherein each computing node in the second communication module serves as a second transition node.

[0009] Further, the first division rule comprises: dividing each computing node according to a row of a 2D mesh topology network, and the computing nodes in each row serve as a first communication module; and the second division rule comprises: dividing each computing node according to a column of the 2D mesh topology network, and the computing nodes in each column serve as a second communication module.

[0010] Alternatively, the first division rule comprises: dividing each computing node according to a column of a 2D mesh topology network, and the computing nodes in each column serve as a first communication module; and the second division rule comprises: dividing each computing node according to a row of the 2D mesh topology network, and the computing nodes in each row serve as a second communication module.

[0011] Further, the first data aggregation rule comprises: each computing node in the first communication module serves as a first transition node, and aggregates the first data slice sent by each computing node in the first communication module, wherein the first data slice takes the computing node included in the second communication module corresponding to the first transition node as a first target node.

[0012] The first data dissemination rule comprises: each computing node in the second communication module sends the aggregated first data slice to the corresponding computing node in the second communication module which serves as a first target node.

[0013] Further, the second data aggregation rule comprises: each computing node in the second communication module serves as a second transition node, and aggregates the first data slice sent by each computing node in the second communication module, wherein the first data slice takes the computing node included in the first communication module corresponding to the second transition node as a second target node.

[0014] The second data dissemination rule comprises: each computing node in the first communication module sends the aggregated second data slice to the corresponding computing node in the first communication module.

[0015] Further, the first division rule comprises: dividing each computing node according to rows of the 2D mesh topology network, and taking every two rows of computing nodes as a first communication module, and taking computing nodes with odd node numbers as a first sub-module and computing nodes with even node numbers as a second sub-module according to the parity of the node numbers of each computing node in the first communication module; wherein the node numbers of each computing node of the 2D mesh topology network are obtained in advance.

[0016] The second division rule comprises: dividing each computing node according to columns of the 2D mesh topology network, and taking every two columns of computing nodes as a second communication module, and taking computing nodes of odd rows as a third sub-module and computing nodes of even rows as a fourth sub-module according to the parity of the rows to which each computing node in the second communication module belongs in the 2D mesh topology network.

[0017] Further, the first data aggregation rule comprises: each computing node in the first sub-module as a first transition node, aggregating the first data slice sent by each computing node in the first sub-module, the first data slice taking the computing node included in the third sub-module or the fourth sub-module corresponding to the first transition node as a first target node; each computing node in the second sub-module as a first transition node, aggregating the first data slice sent by each computing node in the second sub-module, the first data slice taking the computing node included in the third sub-module or the fourth sub-module corresponding to the first transition node as a first target node.

[0018] The first data scattering rule comprises: each computing node in the third sub-module sending the aggregated first data slice to the corresponding computing node as a first target node in the third sub-module; and each computing node in the fourth sub-module sending the aggregated first data slice to the corresponding computing node as a first target node in the fourth sub-module.

[0019] Further, the second data aggregation rule comprises: each computing node in the third sub-module as a second transition node, aggregating the second data slice sent by each computing node in the third sub-module, the second data slice taking the computing node included in the first sub-module or the second sub-module corresponding to the second transition node as a second target node; and each computing node in the fourth sub-module as a second transition node, aggregating the second data slice sent by each computing node in the fourth sub-module, the second data slice taking the computing node included in the first sub-module or the second sub-module corresponding to the second transition node as a second target node.

[0020] The second data dissemination rule comprises: each computing node in the first sub-module sends the aggregated second data slice to the corresponding computing node in the first sub-module as the second target node; each computing node in the second sub-module sends the aggregated second data slice to the corresponding computing node in the second sub-module as the second target node.

[0021] Further, the first division rule comprises: dividing each computing node according to the column of the 2D mesh topology network, and taking every 2 columns of computing nodes as a first communication module, and taking the computing nodes in the odd rows as a fifth sub-module and the computing nodes in the even rows as a sixth sub-module according to the parity of the rows to which each computing node in the first communication module belongs in the 2D mesh topology network.

[0022] The second division rule comprises: dividing each computing node according to the row of the 2D mesh topology network, and taking every 2 rows of computing nodes as a second communication module, and taking the computing nodes with odd node numbers as a seventh sub-module and the computing nodes with even node numbers as an eighth sub-module according to the parity of the node numbers of each computing node in the second communication module; wherein the node numbers of each computing node of the 2D mesh topology network are obtained in advance.

[0023] Further, the first data aggregation rule comprises: each computing node in the fifth sub-module aggregates the first data slice sent by each computing node in the fifth sub-module, and the first data slice takes the computing node included in the seventh sub-module or the eighth sub-module corresponding to the first transition node as the first target node; each computing node in the sixth sub-module aggregates the first data slice sent by each computing node in the sixth sub-module, and the first data slice takes the computing node included in the seventh sub-module or the eighth sub-module corresponding to the first transition node as the first target node.

[0024] The first data dissemination rule comprises: each computing node in the seventh sub-module sends the aggregated first data slice to the corresponding computing node in the seventh sub-module as the first target node; each computing node in the eighth sub-module sends the aggregated first data slice to the corresponding computing node in the eighth sub-module as the first target node.

[0025] Further, the second data aggregation rule comprises: each computing node in the seventh submodule as a second transition node, aggregating the second data slice sent by each computing node in the seventh submodule, the second data slice taking the computing node included in the fifth submodule or the sixth submodule corresponding to the second transition node as a second target node; each computing node in the eighth submodule as a second transition node, aggregating the second data slice sent by each computing node in the eighth submodule, the second data slice taking the computing node included in the seventh submodule or the eighth submodule corresponding to the second transition node as a second target node.

[0026] The second data dissemination rule comprises: each computing node in the fifth submodule sends the aggregated second data slice to the corresponding computing node in the fifth submodule as a second target node; each computing node in the sixth submodule sends the aggregated second data slice to the corresponding computing node in the sixth submodule as a second target node.

[0027] Further, the first division rule comprises: dividing the computing nodes in the 2D mesh topology network into a plurality of first communication modules according to 2 rows and n columns, each first communication module comprising 2x n computing nodes, and the total number of columns of the 2D mesh topology network being 2n.

[0028] The second division rule comprises: dividing the computing nodes in the 2D mesh topology network by columns, the computing nodes in the i-th column and the i+n-th column as a second communication module, and according to the parity of the row to which the computing node in the second communication module belongs in the 2D mesh topology network, the computing nodes in the odd row as a ninth submodule and the computing nodes in the even row as a tenth submodule, i being greater than or equal to 1 and less than or equal to n.

[0029] Further, the first data aggregation rule comprises: each computing node in the first communication module as a first transition node, aggregating the first data slice sent by each computing node in the first communication module, the first data slice taking the computing node included in the ninth submodule or the tenth submodule corresponding to the first transition node as a first target node.

[0030] The first data dissemination rule comprises: each computing node in the ninth submodule sends the aggregated first data slice to the corresponding computing node in the ninth submodule as a first target node; each computing node in the tenth submodule sends the aggregated first data slice to the corresponding computing node in the tenth submodule as a first target node.

[0031] Further, the second data aggregation rule comprises: each computing node in the ninth sub-module as a second transition node, aggregating the second data slice sent by each computing node in the ninth sub-module, the second data slice taking the computing node included in the first communication module corresponding to the second transition node as a second target node; each computing node in the tenth sub-module as a second transition node, aggregating the second data slice sent by each computing node in the tenth sub-module, the second data slice taking the computing node included in the first communication module corresponding to the second transition node as a second target node.

[0032] The second data dissemination rule comprises: each computing node in the first communication module sending the aggregated second data slice to the corresponding computing node in the first communication module as a second target node.

[0033] Further, the first division rule comprises: dividing the computing nodes in the 2D mesh topology network by column, the computing nodes in the ith column and the ith+n column as a first communication module, and according to the parity of the row to which the computing node in the first communication module belongs in the 2D mesh topology network, the computing nodes in the odd row as an eleventh sub-module, and the computing nodes in the even row as a twelfth sub-module, i is greater than or equal to 1 and less than or equal to n, and the total number of columns of the 2D mesh topology network is 2n.

[0034] The second division rule comprises: dividing the computing nodes in the 2D mesh topology network into a plurality of second communication modules according to 2 rows and n columns, each second communication module including 2x n computing nodes.

[0035] Further, the first data aggregation rule comprises: each computing node in the eleventh sub-module as a first transition node, aggregating the first data slice sent by each computing node in the eleventh sub-module, the first data slice taking the computing node included in the second communication module corresponding to the first transition node as a first target node; each computing node in the twelfth sub-module as a first transition node, aggregating the first data slice sent by each computing node in the twelfth sub-module, the first data slice taking the computing node included in the second communication module corresponding to the first transition node as a first target node.

[0036] The first data dissemination rule comprises: each computing node in the second communication module sending the aggregated first data slice to the corresponding computing node in the second communication module as a first target node.

[0037] Further, the second data aggregation rule comprises: each computing node in the second communication module as a second transition node, aggregating the second data slice sent by each computing node in the second communication module, and the second data slice taking the computing node included in the eleventh submodule or the twelfth submodule corresponding to the second transition node as a second target node.

[0038] The second data scattering rule comprises: each computing node in the eleventh submodule sending the aggregated second data slice to the corresponding computing node in the eleventh submodule as a second target node; and each computing node in the twelfth submodule sending the aggregated second data slice to the corresponding computing node in the twelfth submodule as a second target node.

[0039] Further, the computing node adopts a computing power card or a processor core.

[0040] In a second aspect, the present application provides a communication method applied to the all-to-all communication system in any of the above embodiments, comprising:

[0041] In the first communication stage, each computing node in the first communication module aggregates the first data slice thereof to the corresponding first transition node based on a first data aggregation rule; and each computing node in the second communication module sends the first data slice aggregated by each computing node to the corresponding first target node based on a first data scattering rule; wherein each computing node in the first communication module is a first transition node; each first communication module performs communication in parallel; and each second communication module performs communication in parallel.

[0042] In the second communication stage, each computing node in the second communication module aggregates the second data slice thereof to the corresponding second transition node based on a second data aggregation rule; and each computing node in the first communication module sends the second data slice aggregated by each computing node to the corresponding second target node based on a second data scattering rule; wherein each computing node in the second communication module is a second transition node; each second communication module performs communication in parallel; and each first communication module performs communication in parallel.

[0043] In a third aspect, the present application provides a computer device comprising the all-to-all communication system in any of the above embodiments.

[0044] The all-to-all communication system, the communication method and the computer device provided by the embodiment of the application comprise a plurality of computing nodes, the plurality of computing nodes are communicatively connected through a 2D mesh topology network, the plurality of computing nodes are divided into a plurality of first communication modules according to a first division rule and are divided into a plurality of second communication modules according to a second division rule; in a first communication stage, each computing node in each first communication module is configured to aggregate a respective first data slice to a corresponding first transition node based on a first data aggregation rule, and each first communication module performs communication in parallel; each computing node in each second communication module is configured to send a first data slice aggregated by each computing node to a corresponding first target node for processing based on a first data dissemination rule, and each second communication module performs communication in parallel; wherein each computing node in the first communication module serves as a first transition node; in a second communication stage, each computing node in the second communication module is configured to aggregate a respective second data slice to a corresponding second transition node based on a second data aggregation rule, and each second communication module performs communication in parallel; each computing node in the first communication module is configured to send a second data slice aggregated by each computing node to a corresponding second target node based on a second data dissemination rule, and each first communication module performs communication in parallel; wherein each computing node in the second communication module serves as a second transition node, communication delay and congestion are reduced, and communication efficiency is improved. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor. In the drawings:

[0046] Figure 1 FIG. 1 is a structural schematic diagram of a 2D mesh topology network of an all-to-all communication system provided by an embodiment of the application.

[0047] Figure 2 FIG. 2 is a schematic diagram of a mapping relationship between data slices and computing nodes provided by an embodiment of the application.

[0048] Figure 3 FIG. 3 is a structural schematic diagram of a 2D mesh topology network of an all-to-all communication system provided by an embodiment of the application.

[0049] Figure 4 FIG. 4 is a schematic diagram of a first data slice aggregation process provided by an embodiment of the application.

[0050] Figure 5 is a first data slice dissemination process diagram provided by an embodiment of the present application.

[0051] Figure 6 is a structure diagram of a 2D mesh topology network of an all-to-all communication system provided by an embodiment of the present application.

[0052] Figure 7 is a first data slice aggregation process diagram provided by an embodiment of the present application.

[0053] Figure 8 is a first data slice dissemination process diagram provided by an embodiment of the present application.

[0054] Figure 9 is a second data slice aggregation process diagram provided by an embodiment of the present application.

[0055] Figure 10 is a second data slice dissemination process diagram provided by an embodiment of the present application.

[0056] Figure 11 is a second data slice aggregation process diagram provided by an embodiment of the present application.

[0057] Figure 12 is a second data slice dissemination process diagram provided by an embodiment of the present application.

[0058] Figure 13A is a structure diagram of a 2D mesh topology network of an all-to-all communication system provided by an embodiment of the present application.

[0059] Figure 13B is a structure diagram of a 2D mesh topology network of an all-to-all communication system provided by an embodiment of the present application.

[0060] Figure 14A is a first data slice of a computing node diagram provided by an embodiment of the present application.

[0061] Figure 14B is a first data slice of a computing node diagram provided by an embodiment of the present application.

[0062] Figure 15A is a first data slice aggregation process diagram provided by an embodiment of the present application.

[0063] Figure 15B is a first data slice dissemination process diagram provided by an embodiment of the present application.

[0064] Figure 16A is a first data slice aggregation process diagram provided by an embodiment of the present application.

[0065] Figure 16B is a schematic diagram of a first data slice dissemination process according to an embodiment of the application.

[0066] Figure 17 is a schematic diagram of a second data slice of a compute node according to an embodiment of the application.

[0067] Figure 18 is a schematic diagram of a second data slice aggregation process according to an embodiment of the application.

[0068] Figure 19 is a schematic diagram of a second data slice dissemination process according to an embodiment of the application.

[0069] Figure 20A is a schematic diagram of a 2D mesh topology network of an all-to-all communication system according to an embodiment of the application.

[0070] Figure 20B is a schematic diagram of a 2D mesh topology network of an all-to-all communication system according to an embodiment of the application.

[0071] Figure 21A is a schematic diagram of a first data slice aggregation process according to an embodiment of the application.

[0072] Figure 21B is a schematic diagram of a first data slice dissemination process according to an embodiment of the application.

[0073] Figure 22A is a schematic diagram of a first data slice dissemination process according to an embodiment of the application.

[0074] Figure 22B is a schematic diagram of a first data slice dissemination process according to an embodiment of the application.

[0075] Figure 23 is a schematic diagram of a second data slice aggregation process according to an embodiment of the application.

[0076] Figure 24 is a schematic diagram of a second data slice dissemination process according to an embodiment of the application.

[0077] Figure 25 is a schematic diagram of a 2D mesh topology network of an all-to-all communication system according to an embodiment of the application.

[0078] Figure 26 is a schematic diagram of a first data slice aggregation process according to an embodiment of the application.

[0079] Figure 27is a first data slice dissemination process schematic diagram provided by an embodiment of the present application.

[0080] Figure 28 is a first data slice dissemination process schematic diagram provided by an embodiment of the present application.

[0081] Figure 29 is a second data slice aggregation process schematic diagram provided by an embodiment of the present application.

[0082] Figure 30 is a second data slice aggregation process schematic diagram provided by an embodiment of the present application.

[0083] Figure 31 is a second data slice dissemination process schematic diagram provided by an embodiment of the present application.

[0084] Figure 32 is a structure schematic diagram of a 2D mesh topology network of an all-to-all communication system provided by an embodiment of the present application.

[0085] Figure 33 is a first data slice aggregation process schematic diagram provided by an embodiment of the present application.

[0086] Figure 34 is a first data slice aggregation process schematic diagram provided by an embodiment of the present application.

[0087] Figure 35 is a first data slice dissemination process schematic diagram provided by an embodiment of the present application.

[0088] Figure 36 is a second data slice aggregation process schematic diagram provided by an embodiment of the present application.

[0089] Figure 37 is a second data slice dissemination process schematic diagram provided by an embodiment of the present application.

[0090] Figure 38 is a second data slice dissemination process schematic diagram provided by an embodiment of the present application.

[0091] Figure 39 is a flow schematic diagram of a communication method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0092] To make the purposes, technical solutions and advantages of the embodiments of the present application clearer, further detailed descriptions of the embodiments of the present application will be given below with reference to the drawings. Here, the illustrative embodiments of the present application and their descriptions are used to explain the present application but are not intended to limit the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other at will without conflict. The acquisition, storage, use, processing, etc. of data in the technical solutions in the present application all comply with relevant provisions of laws and regulations. The user information in the embodiments of the present application is obtained through legal and compliant ways, and the acquisition, storage, use, processing, etc. of the user information are agreed by the client.

[0093] In order to facilitate understanding of the technical solutions provided by the present application, the related contents of the technical solutions of the present application will be described first.

[0094] All-to-all communication is a key technology for efficient landing of large artificial intelligence models, especially based on the Mixture of Experts (MOE) model, which needs to be optimized through software and hardware collaboration to effectively solve the communication bottleneck problem of all-to-all communication in distributed large artificial intelligence model training and inference.

[0095] The all-to-all communication system and communication method provided by the present application can be applied in a 2D mesh topology network to realize efficient all-to-all communication. In particular, in the training and inference process of a large-scale distributed AI model based on a 2D Mesh topology, using the method can effectively improve the all-to-all communication efficiency, thereby improving the model throughput and reducing the delay, and making up for the technical gap of lacking efficient all-to-all communication methods in the 2D Mesh topology network.

[0096] Figure 1 is a schematic diagram of the interconnection topology of the all-to-all communication system provided by an embodiment of the present application, as shown in Figure 1 The all-to-all communication system provided by the embodiment of the present application includes a plurality of computing nodes 1, and the plurality of computing nodes 1 are communicatively connected through a 2D mesh topology network, wherein:

[0097] The plurality of computing nodes (XPU) 1 are divided into a plurality of first communication modules 10 according to a first division rule, and divided into a plurality of second communication modules 20 according to a second division rule;

[0098] In the first communication stage, each computing node 1 in each first communication module 10 is configured to aggregate the respective first data slice into a corresponding first transition node based on a first data aggregation rule, and each first communication module 10 performs communication in parallel; each computing node in each second communication module 20 is configured to send the first data slice aggregated by each computing node to a corresponding first target node for processing based on a first data dissemination rule, and each second communication module 20 performs communication in parallel; wherein each computing node 1 in the first communication module 10 acts as a first transition node.

[0099] In the second communication stage, each computing node 1 in each second communication module 20 is configured to aggregate the respective second data slice into a corresponding second transition node based on a second data aggregation rule, and each second communication module 20 performs communication in parallel; each computing node in the first communication module 10 is configured to send the second data slice aggregated by each computing node 1 to a corresponding second target node based on a second data dissemination rule, and each first communication module 10 performs communication in parallel; wherein each computing node 1 in the second communication module 20 acts as a second transition node.

[0100] Specifically, each computing node 1 is connected in communication through a 2D mesh topology network. Each computing node 1 can be arranged in one server or in multiple servers. The computing node 1 can be a GPU, TPU, RPU, etc. according to actual needs, and the embodiments of the present application are not limited.

[0101] In the 2D mesh topology network, adjacent computing nodes 1 are directly connected in communication, and non-adjacent computing nodes 1 need to relay and route through other computing nodes to realize interconnection communication, so that there are a large number of public communication links between non-adjacent computing nodes 1. However, in all-to-all communication, each computing node 1 needs to communicate with all other computing nodes, resulting in a large number of data flows in the communication network, which causes serious network congestion in the 2D mesh topology network and reduces communication efficiency. Therefore, in order to improve the communication efficiency, the present application divides a plurality of computing nodes 1 into a plurality of first communication modules 10 according to a first division rule, and divides a plurality of second communication modules 20 according to a second division rule; and divides the all-to-all communication into a first communication stage and a second communication stage.

[0102] In the first communication stage, each computing node 1 needs to send a first data slice to other computing nodes, and the transmission process of the first data slice is divided into two processes: first data slice aggregation and first data slice dissemination. In the first data slice aggregation process, each computing node 1 in each first communication module 10 aggregates the first data slice of each computing node 1 to the corresponding first transition node based on the first data aggregation rule, and each computing node 1 in the first communication module 10 can serve as the first transition node. Each first communication module 10 communicates in parallel. In the first data slice dissemination process, each computing node 1 in each second communication module 20 sends the first data slice aggregated by each computing node 1 to the corresponding first target node based on the first data dissemination rule, and each computing node 1 in the second communication module 20 can serve as the first target node. Each second communication module 20 communicates in parallel. In the first communication stage, the data sent from one computing node to another computing node is referred to as a first data slice, and since all-to-all communication is performed, each computing node can have multiple first data slices. Each first transition node aggregates the first data slice sent to the corresponding first target node, and the correspondence between the first transition node and the first target node is predetermined, and the first target node corresponding to the first transition node can have multiple first target nodes. The first target node is a computing node 1 that receives the first data slice. The communication between each computing node 1 in the first communication module 10 can be implemented based on a Tree manner or a Ring manner, and the communication between each computing node 1 in the second communication module 20 can be implemented based on a Tree manner or a Ring manner, which is selected according to actual needs, and the embodiments of the present application are not limited.

[0103] During the second communication phase, each compute node 1 needs to send its second data slice to each of the other compute nodes. The second data slice transmission process is divided into two steps: second data slice aggregation and second data slice dissemination. During the second data slice aggregation process, each compute node 1 in the second communication module 20 aggregates its second data slice to the corresponding second transition node based on the second data aggregation rule. Each compute node 1 in the second communication module 20 serves as a second transition node. The second communication modules 20 communicate in parallel. During the second data slice dissemination process, each compute node 1 in each first communication module 10 sends the aggregated second data slice to the corresponding second destination node based on the second data dissemination rule. The first communication modules 10 communicate in parallel. In the second communication phase, the data sent from one compute node to another is called the second data slice. The second data slice can be the data obtained by the compute node 1 after processing the first data slice after receiving it. The second data slice needs to be returned to the compute node that sent the first data slice. Each second transition node aggregates the second data slices sent to the corresponding second target node. The correspondence between the second transition node and the second target node is predetermined, and there can be multiple second target nodes corresponding to the second transition node. The second target node is the computing node 1 that receives the second data slice. The communication between the various computing nodes 1 in the first communication module 10 can be implemented based on a tree method or a ring method. The communication between the various computing nodes 1 in the second communication module 20 can be implemented based on a tree method or a ring method. The selection is made based on actual needs and is not limited in the embodiments of the present invention.

[0104] The present application divides the all-to-all communication process into a first communication phase and a second communication phase. During the first communication phase and the second communication phase, each first communication module and each second communication module can communicate in parallel, thereby improving communication efficiency. In addition, the communication of each first communication module is independent of each other. The computing nodes in each first communication module perform transit routing within the first communication module, reducing the number of transit routings, thus reducing the public communication links, thereby reducing communication delays and congestion, and improving communication efficiency. The communication of each second communication module is independent of each other. The computing nodes in each second communication module perform transit routing within the second communication module, reducing the number of transit routings, thus reducing the public communication links, thereby reducing communication delays and congestion, and improving communication efficiency.

[0105] For example, Figure 1 As shown, each computing node 1 forms a communication connection with a 2D Mesh topology network, J0~J n-1Row represents a row of 2D Mesh topology network, a total of n rows, and the computing node 1 of each row is a first communication module 10; G0~G m-1 Column represents a column of 2D Mesh topology network, a total of m columns, and the computing node 1 of each column is a second communication module 20; Each computing node 1 in the 2D Mesh array adopts a computing power card. One or more rows of computing nodes 1 in the 2D Mesh topology network are arranged in one server, or one or more columns of computing nodes 1 in the 2D Mesh topology network are arranged in one server, or multiple rows and multiple columns of computing nodes 1 in the 2D Mesh topology network are arranged in one server.

[0106] According to the total number mn of the computing nodes of the all-to-all communication system, each computing node reserves m x n cache spaces to store data, denoted as m x n data slices, and the arrangement order of the data slices is one-to-one corresponding to the node number in the cluster. For standard all-to-all communication, m x n data slices in each computing node are all effective; the data contents in these data slices can be completely the same, partially the same or completely different; for non-standard all-to-all communication, usually only part of the m x n data slices in each computing node are actually effective, and the invalid data slices indicate that the node does not need to communicate with the computing node corresponding to the slice position; similarly, the data contents in the effective data slices in each computing node can also be completely the same, partially the same or completely different. As shown in FIG. 2, the mapping relationship between the data slices of each computing node and each computing node is shown, taking D Figure 2 1 2 as an example for illustration, the upper subscript represents the node number of the computing node, and the lower subscript represents the data slice number in the computing node; D 1 2 represents the data slice with the data slice number 2 in the computing node with the node number 1; D 1 2 is the data slice that the computing node with the node number 1 needs to send to the computing node with the node number 2 through all-to-all communication. In this application, the data slice is divided into a first data slice and a second data slice.

[0107] ​The all-to-all communication system provided by the embodiment of the present application comprises a plurality of computing nodes, the plurality of computing nodes are communicatively connected through a 2D mesh topology network, the plurality of computing nodes are divided into a plurality of first communication modules according to a first division rule and divided into a plurality of second communication modules according to a second division rule; in a first communication stage, each computing node in each first communication module is configured to aggregate a respective first data slice to a corresponding first transition node based on a first data aggregation rule, and each first communication module performs communication in parallel; each computing node in each second communication module is configured to send a first data slice aggregated by each computing node to a corresponding first target node for processing based on a first data dissemination rule, and each second communication module performs communication in parallel; wherein each computing node in the first communication module serves as a first transition node; in a second communication stage, each computing node in the second communication module is configured to aggregate a respective second data slice to a corresponding second transition node based on a second data aggregation rule, and each second communication module performs communication in parallel; each computing node in the first communication module is configured to send a second data slice aggregated by each computing node to a corresponding second target node based on a second data dissemination rule, and each first communication module performs communication in parallel; wherein each computing node in the second communication module serves as a second transition node, thereby reducing communication delay and congestion and improving communication efficiency.

[0108] Further, the first division rule comprises dividing each computing node according to a row of the 2D mesh topology network, and the computing node 1 in each row serves as a first communication module 10; and the second division rule comprises dividing each computing node 1 according to a column of the 2D mesh topology network, and the computing node 1 in each column serves as a second communication module 20.

[0109] Alternatively, the first division rule comprises dividing each computing node 1 according to a column of the 2D mesh topology network, and the computing node 1 in each column serves as a first communication module; and the second division rule comprises dividing each computing node 1 according to a row of the 2D mesh topology network, and the computing node in each row serves as a second communication module.

[0110] Specifically, the computing node 1 in each row of the 2D mesh topology network constitutes a first communication module 10. The computing node 1 in each column of the 2D mesh topology network constitutes a second communication module 20.

[0111] Alternatively, the computing node 1 in each column of the 2D mesh topology network constitutes a first communication module 10. The computing node 1 in each row of the 2D mesh topology network constitutes a second communication module 20.

[0112] For example, as shown in FIG. 1, a 2D mesh topology network is composed of n x m computing nodes 1 communicatively connected, and each row of computing nodes 1 constitutes a first communication module 10, and there are n first communication modules 10 in total. Each column of computing nodes 1 constitutes a second communication module 20, and there are m second communication modules 20 in total. Figure 1

[0113] For example, as shown in FIG. 2, a 2D mesh topology network is composed of n x m computing nodes 1 communicatively connected, and each column of computing nodes 1 constitutes a first communication module 10, and there are m first communication modules 10 in total. Each row of computing nodes 1 constitutes a second communication module 20, and there are n second communication modules 20 in total. Figure 3

[0114] On the basis of the above embodiments, further, the first data aggregation rule includes: each computing node 1 in the first communication module 10 as a first transition node, aggregating the first data slice sent by each computing node 1 in the first communication module 10, and the first data slice taking the computing node 1 included in the second communication module corresponding to the first transition node as a first target node;

[0115] The first data scattering rule includes: each computing node 1 in the second communication module 20 sending the aggregated first data slice to the corresponding computing node 1 in the second communication module 20 as the first target node.

[0116] Specifically, in the first data slice aggregation process, each computing node 1 in the first communication module 10 as a first transition node will aggregate the first data slice, and the first data slice aggregated by the first transition node is sent by each computing node 1 in the first communication module 10, and is the first data slice sent to the computing node 1 included in the second communication module corresponding to the first transition node as the first target node. The second communication module corresponding to the first transition node refers to the second communication module to which the computing node 1 as the first transition node belongs.

[0117] Each computing node 1 in the second communication module 20 will aggregate the first data slice as a first transition node in the first data slice aggregation process. In the first data slice scattering process, each computing node 1 in the second communication module 20 sends the aggregated first data slice to the corresponding first target node, and each computing node 1 in the second communication module 20 can receive the first data slice taking each computing node 1 as a data receiving node as the first target node.

[0118] For example, as shown in FIG. 3, a 2D mesh topology network is composed of n x m computing nodes 1 communicatively connected, and each column of computing nodes 1 constitutes a first communication module 10, and there are m first communication modules 10 in total. Each row of computing nodes 1 constitutes a second communication module 20, and there are n second communication modules 20 in total. Figure 4 n-1 ​​​The process of aggregating the first data slice by the computing nodes 1 in any row, that is, the process of aggregating the first data slice by the computing nodes 1 in any first communication module 10 as the first transition node. According to the correspondence between the first target node receiving the first data slice and the first transition node, the first data slices sent to the first target node by all computing nodes 1 in the first communication module 10 are first aggregated to the corresponding first transition node. For example, computing node J0G0, as the first transition node, will aggregate the first data slices sent by all computing nodes in row J0 to all computing nodes in column G0. All computing nodes in column G0 serve as the first target nodes. Computing node J0G0 corresponds to all computing nodes in column G0. All computing nodes in row J0 constitute a first communication module 10, and all computing nodes in column G0 constitute a second communication module 20. For example, computing node J0G1 acts as the first transition node, and aggregates the first data slices sent by all computing nodes in row J0 to all computing nodes in column G1. All computing nodes in column G0 act as the first target nodes, and computing node J0G1 corresponds to all computing nodes in column G1. All computing nodes in row J0 form a first communication module 10, and all computing nodes in column G1 form a second communication module 20. Computing nodes J0G2~J0G1 in row J0 m-1 The aggregation process of the first data slice of J0G0 and the aggregation process of the first data slice of J0G1 are similar to those of the computing nodes J0G0 and J0G1, and are not described here. n The aggregation process of the first data slice in row -1 is similar to the aggregation process of the first data slice in row J0, and is not described here in detail.

[0119] For example, Figure 5 As shown, in the 2D mesh topology network, G0~G m-1 The process of spreading the first data slice aggregated by the computing nodes 1 in any column is the process of spreading the first data slice aggregated by the computing nodes 1 in any second communication module 20. Spread the first data slice aggregated by all computing nodes 1 in the second communication module 20 to the corresponding first target node. For example, computing node J0G0 sends the aggregated first data slice to other computing nodes in column G0 according to the first target node that receives the first data slice; computing node J1G0 sends the aggregated first data slice to other computing nodes in column G0 according to the first target node that receives the first data slice. Computing nodes J2G0~J in column G0 n-1 The process of spreading the first data slice of G0 is similar to the process of spreading the first data slice of computing nodes J0G0 and J1G0, and will not be described in detail here. m The spreading process of the first data slice of each computing node in each column of the -1 column is similar to the spreading process of the first data slice in the G0 column, and is not described in detail here.

[0120] For example, Figure 6 As shown, the 2D mesh topology network consists of 8×8 computing nodes (XPUs) 1 connected in communication. The computing nodes 1 in each column form a first communication module 10, for a total of 8 first communication modules 10. The computing nodes 1 in each row form a second communication module 20, for a total of 8 second communication modules 20. Starting with computing node J0G0, the nodes are numbered sequentially from left to right and from top to bottom. That is, row J0 is numbered from left to right, row J1 is numbered from left to right, row J2 is numbered from left to right, row J4 is numbered from left to right, and so on. The node number of computing node J0G0 is 0, the node number of computing node J0G1 is 1, the node number of computing node J0G2 is 2, ..., and the node number of computing node J7G7 is 63.

[0121] by Figure 6 Taking the 2D mesh topology network shown as an example, the process of aggregating the first data slice by the computing node 1 in the first communication module 10 is explained, that is, the process of aggregating the first data slice by any computing node 1 in the first communication module 10 as the first transition node. According to the correspondence between the first target node receiving the first data slice and the first transition node, the first data slices sent to the first target node by all computing nodes 1 in the first communication module 10 are first aggregated to the corresponding first transition node. For example, as the first transition node, computing node J0G0 will aggregate the first data slices sent by all computing nodes in column G0 to all computing nodes in row J1. All computing nodes in row J1 serve as the first target nodes, and computing node J0G0 corresponds to all computing nodes in row J1. Computing nodes J1G0~J0 in column G0 n-1 The aggregation process of the first data slice of G0 is similar to that of the first data slice of computing node J0G0, and will not be described here. m-1 The aggregation process of the first data slice in the column is similar to the aggregation process of the first data slice in the G0 column and is not described here.

[0122] Furthermore, if Figure 7 As shown, Figure 6 The process of aggregating the first data slice by the computing node with the node number 0 in the G0 column in the 2D mesh topology network shown in FIG is taken as an example to illustrate the aggregation process of the first data slice. Each computing node in the first communication module composed of the computing nodes in the G0 column includes 64 first data slices, which are specifically represented as follows Figure 14AThe computing nodes numbered 0, 8, 16, 24, 32, 40, 48 and 56 constitute a first communication module. The computing node numbered 0, as a first transition node, aggregates the first data slices sent by all the computing nodes of the first communication module to the computing nodes of a second communication module constituted by the computing nodes of the J0 row, all the computing nodes of the second communication module being first target nodes, and the computing node numbered 0 corresponding to all the computing nodes of the second communication module. The computing node numbered 0 aggregates the first data slices numbered 0-7 of all the computing nodes (i.e. the computing nodes numbered 0, 8, 16, 24, 32, 40, 48 and 56) of the first communication module. The computing node numbered 8 aggregates the first data slices numbered 8-15 of all the computing nodes (i.e. the computing nodes numbered 0, 8, 16, 24, 32, 40, 48 and 56) of the first communication module. The computing node numbered 16 aggregates the first data slices numbered 16-23 of all the computing nodes (i.e. the computing nodes numbered 0, 8, 16, 24, 32, 40, 48 and 56) of the first communication module. This analogy continues.

[0123] Further, as shown in Figure 8 Figure 6 ​The process of the first data slice spreading by the computing node numbered 0 in the J0 row in the illustrated 2D mesh topology network is taken as an example to illustrate the spreading process of the first data slice. The computing nodes in the J0 row constitute the second communication module. The computing node numbered 0 in the second communication module sends the aggregated first data slice to other computing nodes in the second communication module according to the first target node receiving the first data slice. The computing node numbered 0 sends the first data slice numbered 0~7 of the aggregated first data slice of the computing nodes numbered 0, 8, 16, 24, 32, 40, 48 and 56 to the corresponding first target node, i.e. the computing nodes numbered 0~7 in the second communication module. The computing node numbered 1 sends the first data slice numbered 0~7 of the aggregated first data slice of the computing nodes numbered 1, 9, 17, 25, 33, 41, 49 and 57 to the corresponding first target node, i.e. the computing nodes numbered 0~7 in the second communication module. The computing node numbered 3 sends the first data slice numbered 0~7 of the aggregated first data slice of the computing nodes numbered 2, 10, 18, 26, 34, 42, 50 and 58 to the corresponding first target node, i.e. the computing nodes numbered 0~7 in the second communication module. By analogy, finally each computing node in the second communication module receives the first data slice sent by other computing nodes to the computing node. For example, 0~63 0 represents that the computing node numbered 0 receives 64 first data slices sent to the computing node numbered 0.

[0124] Further, on the basis of the above-mentioned embodiments, the second data aggregation rule comprises: each computing node 1 in the second communication module 20 aggregates the first data slice sent by each computing node 1 in the second communication module 20, and the first data slice is the second data slice with the computing node 1 included in the first communication module 10 corresponding to the second transition node as the second target node.

[0125] The second data spreading rule comprises: each computing node 1 in the first communication module 10 sends the aggregated second data slice to the corresponding computing node in the first communication module 10 as the second target node.

[0126] Specifically, during the second data slice aggregation process, each computing node 1 in the second communication module 20, acting as a second transition node, aggregates the second data slice. The second data slice aggregated by the second transition node is sent by each computing node 1 in the second communication module 20 and is sent to the computing node 1 included in the first communication module 10 corresponding to the second transition node as the second target node. The first communication module corresponding to the second transition node is the first communication module to which the computing node 1 serving as the second transition node belongs.

[0127] During the second data slice aggregation process, each computing node 1 in the first communication module 10 acts as a second transition node to aggregate the second data slice. During the second data slice dissemination process, each computing node 1 in the first communication module 10 sends the aggregated second data slice to the corresponding second destination node. Each computing node 1 in the first communication module 10 acts as a second destination node and receives the second data slice with each computing node 1 as the data receiving node.

[0128] For example, Figure 9 As shown, in the 2D mesh topology network, G0~G m-1 The process of computing nodes 1 in any column aggregating the first data slice is, in other words, the process of computing nodes 1 in any second communication module 20 acting as the first transition node aggregating the first data slice. According to the correspondence between the second target node receiving the second data slice and the second transition node, the second data slices sent to the second target node by all computing nodes 1 in the second communication module 20 are first aggregated to the corresponding second transition node. For example, computing node J0G0, acting as the second transition node, will aggregate the second data slices sent by all computing nodes in column G0 to all computing nodes in row J0. All computing nodes in row J0 act as second target nodes, and computing node J0G0 corresponds to all computing nodes in row J0. All computing nodes in column G0 constitute a second communication module 20, and all computing nodes in row J0 constitute a first communication module 10. For example, computing node J1G0 acts as the second transition node, and aggregates the second data slices sent by all computing nodes in column G0 to all computing nodes in row J1. All computing nodes in row J1 act as the second target nodes, and computing node J1G0 corresponds to all computing nodes in row J1. All computing nodes in column G0 form a second communication module 20, and all computing nodes in row J1 form a first communication module 10. n-1 The aggregation process of the second data slice of G0 is similar to the aggregation process of the second data slice of computing nodes J0G0 and computing nodes J1G0, and will not be repeated here. m The aggregation process of the second data slice in the -1 column is similar to the aggregation process of the second data slice in the G0 column and is not described here in detail.

[0129] For example, Figure 10 As shown, in the 2D mesh topology network, J0~J n-1 The process of spreading the second data slice aggregated by the computing nodes 1 in any row, that is, the process of spreading the second data slice aggregated by the computing nodes 1 in any first communication module 10. Spread the second data slices aggregated by all computing nodes 1 in the first communication module 10 to the corresponding second target nodes, and each computing node in the first communication module 10 serves as the second target node. For example, computing node J0G0 sends the aggregated second data slice to other computing nodes in row J0 according to the second target nodes that receive the second data slice; computing node J0G1 sends the aggregated second data slice to other computing nodes in row J0 according to the second target nodes that receive the second data slice. Computing nodes J0G2~J0G in row J0. m-1 The spreading process of the second data slice of J0G0 and the second data slice of J0G1 is similar to that of the computing nodes J0G0 and J0G1, and will not be described in detail here. n The spreading process of the second data slices of each computing node in each row of row -1 is similar to the spreading process of the second data slices of each computing node in row J0, and will not be described in detail here.

[0130] For example, Figure 6 The process of aggregating the second data slice is described by taking the computing nodes in row J0 with node number 0 in the 2D mesh topology network as an example. The computing nodes in row J0 constitute the second communication module, such as Figure 11 As shown, the compute node with node number 0 acts as the second transition node and aggregates the second data slices sent by all compute nodes in row J0 to all compute nodes in column G0. All compute nodes in column G0 serve as the second target nodes, and the compute node with node number 0 corresponds to all compute nodes in column G0. The compute node with node number 0 aggregates the second data slices numbered 0, 8, 16, 24, 32, 40, 48, and 56 from all compute nodes in the second communication module (i.e., compute nodes with node numbers 0, 1, 2, 3, 4, 5, 6, and 7). The compute node with node number 1 aggregates the second data slices numbered 1, 9, 17, 25, 33, 41, 49, and 57 from all compute nodes in the second communication module (i.e., compute nodes with node numbers 0, 1, 2, 3, 4, 5, 6, and 7). The computing node with node number 3 aggregates the second data slices with numbers 2, 10, 18, 26, 34, 42, 50, and 58 of the first data slices of all computing nodes in the second communication module (i.e., computing nodes with node numbers 0, 1, 2, 3, 4, 5, 6, and 7). The same applies.

[0131] For example, the process of spreading the second data slice is illustrated by taking the computing node numbered 0 in the G0 column in the 2D mesh topology network shown in Figure 6 Figure 12 The computing node numbered 0 in the G0 column sends the aggregated second data slices of the computing nodes numbered 0, 1, 2, 3, 4, 5, 6 and 7 to other computing nodes in the first communication module according to the second target nodes receiving the second data slices. The computing node numbered 0 sends the second data slices numbered 0, 8, 16, 24, 32, 40, 48 and 56 of the aggregated second data slices of the computing nodes numbered 0, 1, 2, 3, 4, 5, 6 and 7 to the corresponding second target nodes, i.e., the computing nodes numbered 0, 8, 16, 24, 32, 40, 48 and 56 in the first communication module. The computing node numbered 8 sends the second data slices numbered 0, 8, 16, 24, 32, 40, 48 and 56 of the aggregated second data slices of the computing nodes numbered 8, 9, 10, 11, 12, 13, 14 and 15 to the corresponding second target nodes, i.e., the computing nodes numbered 0, 8, 16, 24, 32, 40, 48 and 56 in the first communication module. The computing node numbered 16 sends the second data slices numbered 0, 8, 16, 24, 32, 40, 48 and 56 of the aggregated second data slices of the computing nodes numbered 16, 17, 18, 19, 20, 21, 22 and 23 to the corresponding second target nodes, i.e., the computing nodes numbered 0, 8, 16, 24, 32, 40, 48 and 56 in the first communication module. By analogy, finally, each computing node in the first communication module receives the second data slices sent by other computing nodes to the computing node. 0~63 0 represents that the computing node numbered 0 receives 64 second data slices sent to the computing node numbered 0.

[0132] On the basis of the above embodiments, further, the first division rule comprises: dividing each computing node 1 according to the rows of the 2D mesh topology network, and taking every 2 rows of computing nodes 1 as a first communication module 10, and taking the computing nodes 1 numbered odd numbers as a first sub-module and the computing nodes 1 numbered even numbers as a second sub-module according to the parity of the node numbers of each computing node 1 in the first communication module 10; wherein the node numbers of each computing node 1 in the 2D mesh topology network are obtained in advance.

[0133] ​The second division rule comprises: dividing each computing node 1 according to the column of the 2D mesh topology network, and taking every 2 columns of computing nodes 1 as a second communication module 20, and taking the computing nodes 1 in the odd rows as a third submodule and the computing nodes 1 in the even rows as a fourth submodule according to the parity of the rows to which the computing nodes 1 in the second communication module 20 belong in the 2D mesh topology network.

[0134] Specifically, every 2 rows of computing nodes 1 in the 2D mesh topology network form a first communication module 10. Each computing node 1 in the 2D mesh topology network is numbered according to a preset numbering mode, and the node number of each computing node 1 is obtained, and the node number of each computing node 1 is odd or even. The preset numbering mode is set according to actual needs, and the embodiment of the application is not limited. For each computing node 1 in each first communication module 10, the computing node 1 with an odd node number is taken as a first submodule, and the computing node 1 with an even node number is taken as a second submodule.

[0135] Every 2 columns of computing nodes 1 in the 2D mesh topology network form a second communication module 20. For each computing node 1 in each second communication module 20, the computing nodes 1 in the odd rows are taken as a third submodule, and the computing nodes 1 in the even rows are taken as a fourth submodule according to the parity of the rows to which the computing nodes 1 in the second communication module 20 belong in the 2D mesh topology network.

[0136] For example, the preset numbering mode is to take the computing node in the first row and the first column of the 2D mesh topology network as a starting node, and the first row is numbered from left to right, the second row is numbered from right to left, the third row is numbered from left to right, the fourth row is numbered from right to left, and so on, to obtain the node number of each computing node 1 in the 2D mesh topology network. As shown in FIG. 2, starting from the computing node J0G0, the numbering is started with the Arabic numeral 0, and the numbering is performed according to the above-mentioned preset numbering mode to obtain the numbering of each computing node. Figure 13A As shown in FIG. 2, starting from the computing node J0G0, the numbering is started with the Arabic numeral 0, and the numbering is performed according to the above-mentioned preset numbering mode to obtain the numbering of each computing node.

[0137] For example, the preset numbering mode is to take the computing node in the first row and the first column of the 2D mesh topology network as a starting node, and each row is numbered from left to right in a circular manner, that is, the first row is numbered from left to right, the second row is numbered from left to right, the third row is numbered from left to right, the fourth row is numbered from left to right, and so on, to obtain the node number of each computing node 1 in the 2D mesh topology network. As shown in FIG. 3, starting from the computing node J0G0, the numbering is started with the Arabic numeral 1, and the numbering is performed according to the above-mentioned preset numbering mode to obtain the numbering of each computing node. Figure 13B As shown in FIG. 3, starting from the computing node J0G0, the numbering is started with the Arabic numeral 1, and the numbering is performed according to the above-mentioned preset numbering mode to obtain the numbering of each computing node.

[0138] For example, Figure 13A As shown, the 2D mesh topology network consists of 8×8 connected compute nodes (XPUs) 1. Every two rows of compute nodes 1 form a first communication module 10, for a total of four first communication modules 10. Starting with compute node J0G0, the nodes are numbered sequentially from left to right and from right to left: compute node J0G0 is numbered 0, compute node J0G1 is numbered 1, compute node J0G2 is numbered 2, and so on. For each of the four first communication modules 10, compute nodes 1 with odd node numbers are designated as the first submodule, and compute nodes 1 with even node numbers are designated as the second submodule. For example, for the first communication module 10 formed by compute nodes 1 in rows J0 and J1, compute nodes 1 with node numbers 1, 3, 5, 7, 9, 11, 13, and 15 form the first submodule, and compute nodes 1 with node numbers 0, 2, 4, 6, 8, 10, 12, and 14 form the second submodule.

[0139] Every two columns of computing nodes 1 constitute a second communication module 20, for a total of four second communication modules 20. For each of the four second communication modules 20, based on the parity of the rows to which each computing node 1 in the second communication module 20 belongs in the 2D mesh topology network, the computing nodes 1 in odd-numbered rows serve as the third submodule, and the computing nodes 1 in even-numbered rows serve as the fourth submodule. For example, for a second communication module 20 composed of computing nodes 1 in columns G0 and G1, the computing nodes numbered 0 and 1 in the first row, 16 and 17 in the third row, 32 and 33 in the fifth row, and 48 and 49 in the seventh row constitute the third submodule. The computing nodes numbered 14 and 15 in the second row, 30 and 31 in the fourth row, 46 and 47 in the sixth row, and 62 and 63 in the eighth row constitute the fourth submodule.

[0140] like Figure 13BAs shown, the 2D mesh topology network is composed of 8*8 communication connections of computing nodes (XPU) 1. Every 2 rows of computing nodes 1 form a first communication module 10, and there are 4 first communication modules 10. Starting from computing node J0G0, the numbering is sequentially circulated from left to right, the node number of computing node J0G0 is 1, the node number of computing node J0G1 is 2, the node number of computing node J0G2 is 3, and so on. For each computing node 1 in the 4 first communication modules 10, the computing node 1 with an odd node number is taken as a first submodule, and the computing node 1 with an even node number is taken as a second submodule. For example, for the first communication module 10 formed by the computing nodes 1 in the J0th row and the J1th row, the computing nodes 1 with node numbers 1, 3, 5, 7, 9, 11, 13 and 15 form a first submodule, and the computing nodes 1 with node numbers 2, 4, 6, 8, 10, 12, 14 and 16 form a second submodule.

[0141] Every 2 columns of computing nodes 1 form a second communication module 20, and there are 4 second communication modules 20. For the 4 second communication modules 20, according to the parity of the row to which each computing node 1 in the second communication module 20 belongs in the 2D mesh topology network, the computing nodes 1 in the odd row are taken as a third submodule, and the computing nodes 1 in the even row are taken as a fourth submodule. For example, for the second communication module 20 formed by the computing nodes 1 in the G0th column and the G1th column, the computing nodes with node numbers 1 and 2 in the first row, the computing nodes with node numbers 17 and 18 in the third row, the computing nodes with node numbers 33 and 34 in the fifth row, and the computing nodes with node numbers 49 and 50 in the seventh row form a third submodule. The computing nodes with node numbers 9 and 10 in the second row, the computing nodes with node numbers 25 and 26 in the fourth row, the computing nodes with node numbers 41 and 42 in the sixth row, and the computing nodes with node numbers 57 and 58 in the eighth row form a fourth submodule.

[0142] Further, the first data aggregation rule includes: each computing node 1 in the first submodule as a first transition node, aggregating the first data slice sent by each computing node 1 in the first submodule, the first data slice taking the computing node 1 included in the third submodule or the fourth submodule corresponding to the first transition node as a first target node; each computing node 1 in the second submodule as a first transition node, aggregating the first data slice sent by each computing node 1 in the second submodule, the first data slice taking the computing node 1 included in the third submodule or the fourth submodule corresponding to the first transition node as a first target node.

[0143] The first data dissemination rule comprises: each computing node 1 in the third submodule aggregates the aggregated first data slice and sends it to the corresponding computing node 1 in the third submodule as the first target node; each computing node 1 in the fourth submodule aggregates the aggregated first data slice and sends it to the corresponding computing node 1 in the fourth submodule as the first target node.

[0144] Specifically, in the first data slice aggregation process, each computing node 1 in the first submodule of the first communication module 10 aggregates the first data slice as the first transition node, the first data slice aggregated by the first transition node is sent by each computing node 1 in the first submodule, and is the first data slice sent to the computing node 1 included in the third submodule or the fourth submodule corresponding to the first transition node as the first target node. Each computing node 1 in the second submodule of the first communication module 10 aggregates the first data slice as the first transition node, the first data slice aggregated by the first transition node is sent by each computing node 1 in the second submodule, and is the first data slice sent to the computing node 1 included in the third submodule or the fourth submodule corresponding to the first transition node as the first target node. Wherein, the third submodule corresponding to the first transition node refers to the third submodule to which the computing node 1 as the first transition node belongs; the fourth submodule corresponding to the first transition node refers to the fourth submodule to which the computing node 1 as the first transition node belongs. The first submodule and the second submodule communicate in parallel.

[0145] Each computing node 1 in the second communication module 20 aggregates the first data slice as the first transition node in the first data slice aggregation process. In the first data slice dissemination process, each computing node 1 in the third submodule of the second communication module 20 aggregates the aggregated first data slice and sends it to the corresponding first target node, each computing node 1 in the third submodule as the first target node receives the first data slice with each computing node 1 as the data receiving node. Each computing node 1 in the fourth submodule of the second communication module 20 aggregates the aggregated first data slice and sends it to the corresponding first target node, each computing node 1 in the fourth submodule as the first target node receives the first data slice with each computing node 1 as the data receiving node. The third submodule and the fourth submodule communicate in parallel.

[0146] Take 64 computing nodes as an example to illustrate the mapping relationship between the first data slices in the computing nodes and the computing nodes. Each computing node has 64 cache spaces, denoted as 64 first data slices. The arrangement order of the first data slices is one-to-one corresponding to the node number. For standard all-to-all communication, the 64 first data slices in each computing node are all effective; the data contents in the 64 first data slices can be completely same, partially same or completely different; for non-standard all-to-all communication, the 64 first data slices in each computing node can only have some first data slices actually effective, and the ineffective first data slices indicate that the computing node does not need to communicate with the computing node corresponding to the first data slice position; the data contents in the actually effective first data slices in each computing node can also be completely same, partially same or completely different. For example, as shown in Figure 14A , the mapping relationship between the first data slices in the computing nodes and the computing nodes is shown, taking E 3 6 as an example. The upper subscript represents the node number, the lower subscript represents the first data slice number in the computing node corresponding to the node number, and E 3 6 represents the first data slice with the first data slice number 6 in the computing node with the node number 3; meanwhile, it is indicated that the computing node with the node number 3 needs to send the first data slice E 3 6 to the computing node with the node number 6 through all-to-all communication.

[0147] For example, as shown in Figure 15A , the process of aggregating the first data slices by the first communication module composed of the computing nodes (XPU) in the J0 row and the J1 row in the 2D mesh topology network shown in Figure 13A is taken as an example to illustrate the aggregation process of the first data slices. Each computing node in the above first communication module includes 64 first data slices, and the specific representation is as shown in Figure 14AAs shown. Compute nodes numbered 0, 2, 4, 6, 8, 10, 12, and 14 constitute the second submodule, while compute nodes numbered 1, 3, 5, 7, 9, 11, 13, and 15 constitute the first submodule. Compute node numbered 0 (J0G0) acts as the first transition node and aggregates the first data slices sent by all compute nodes in the second submodule to all compute nodes in the third submodule of the second communication module 20, which is composed of compute node 1 in columns G0 and G1. All compute nodes in the third submodule serve as first target nodes, and compute node J0G0 corresponds to all compute nodes in the third submodule. Compute node J0G0 aggregates the first data slices (numbered 0-1, 16-17, 32-33, and 48-49) from compute nodes numbered 0, 2, 4, 6, 8, 10, 12, and 14 in the second submodule. Computing node J1G0, node numbered 15, acts as the first transition node and aggregates the first data slices sent by all computing nodes in the first submodule to all computing nodes in the fourth submodule of the second communication module 20, which is composed of computing node 1 in columns G0 and G1. All computing nodes in the fourth submodule serve as first target nodes, and computing node J1G0 corresponds to all computing nodes in the fourth submodule. Computing node J1G0 aggregates the first data slices of computing nodes numbered 1, 3, 5, 7, 9, 11, 13, and 15 in the first submodule, as well as the first data slices numbered 14-15, 30-31, 46-47, and 62-63. The aggregation process for the first data slices of other computing nodes in the second submodule is similar to the aggregation process for the first data slice of computing node numbered 0 and is not further described here. The aggregation process for the first data slices of other computing nodes in the first submodule is similar to the aggregation process for the first data slice of computing node numbered 15 and is not further described here.

[0148] For example, Figure 15B As shown, Figure 13AThe process of the second communication module composed of the computing nodes (XPU) of the G0 column and the G1 column in the 2D mesh topology network shown disperses the first data slice. The computing nodes with node numbers 0 and 1 in the first row, the computing nodes with node numbers 16 and 17 in the third row, the computing nodes with node numbers 32 and 33 in the fifth row, and the computing nodes with node numbers 48 and 49 in the seventh row constitute a third submodule. The computing nodes with node numbers 14 and 15 in the second row, the computing nodes with node numbers 30 and 31 in the fourth row, the computing nodes with node numbers 46 and 47 in the sixth row, and the computing nodes with node numbers 62 and 63 in the eighth row constitute a fourth submodule. The computing node with node number 0 in the third submodule sends the aggregated first data slice to other computing nodes in the third submodule according to the first target node receiving the first data slice. The computing node with node number 0 in the third submodule sends the first data slice with numbers 0~1, 16~17, 32~33, and 48~49 of the aggregated first data slice of the computing nodes with node numbers 0, 2, 4, 6, 8, 10, 12, and 14 in the second submodule to the corresponding first target node, i.e., the computing nodes with node numbers 0~1, 16~17, 32~33, and 48~49 in the third submodule. The computing node with node number 15 in the fourth submodule sends the first data slice with numbers 14~15, 30~31, 46~47, and 62~63 of the aggregated first data slice of the computing nodes with node numbers 1, 3, 5, 7, 9, 11, 13, and 15 in the first submodule to the corresponding first target node, i.e., the computing nodes with node numbers 14~15, 30~31, 46~47, and 62~63 in the fourth submodule. The dispersing process of the first data slice of other computing nodes in the third submodule is similar to the dispersing process of the first data slice of the computing node with node number 0, which is not described herein. The dispersing process of the first data slice of other computing nodes in the fourth submodule is similar to the dispersing process of the first data slice of the computing node with node number 15, which is not described herein. Finally, each computing node in the second communication module receives the first data slice sent by other computing nodes to the computing node. For example, 0~63 0 represents that the computing node with node number 0 receives 64 first data slices sent to the computing node with node number 0.

[0149] For example, as shown in FIG. 6, the first communication module composed of the computing nodes (XPU) of the J0 row and the J1 row in the 2D mesh topology network shown aggregates the first data slice. Each computing node in the first communication module includes 64 first data slices, which are specifically represented as shown in FIG. 7. Figure 16A Figure 13B For example, as shown in FIG. 6, the first communication module composed of the computing nodes (XPU) of the J0 row and the J1 row in the 2D mesh topology network shown aggregates the first data slice. Each computing node in the first communication module includes 64 first data slices, which are specifically represented as shown in FIG. 7. Figure 14B ​As shown in Figure 2, the compute nodes numbered 1, 3, 5, 7, 9, 11, 13, and 15 constitute the first submodule, and the compute nodes numbered 2, 4, 6, 8, 10, 12, 14, and 16 constitute the second submodule. Computing node numbered 1 (J0G0) acts as the first transition node and aggregates the first data slices sent by all compute nodes in the second submodule to all compute nodes in the third submodule of the second communication module 20, which is composed of compute nodes 1 in columns G0 and G1. All compute nodes in the third submodule serve as first target nodes, and compute node J0G0 corresponds to all compute nodes in the third submodule. Computing node J0G0 aggregates the first data slices (numbered 1-2, 17-18, 33-34, and 49-50) from compute nodes numbered 1, 3, 5, 7, 9, 11, 13, and 15 in the first submodule. Computing node J1G1, node numbered 10, acts as the first transition node and aggregates the first data slices sent by all computing nodes in the second submodule to all computing nodes in the fourth submodule of the second communication module 20, which is composed of computing node 1 in columns G0 and G1. All computing nodes in the fourth submodule serve as first target nodes, and computing node J1G1 corresponds to all computing nodes in the fourth submodule. Computing node J1G1 aggregates the first data slices of computing nodes numbered 2, 4, 6, 8, 10, 12, 14, and 16 in the second submodule, as well as the first data slices numbered 9-10, 25-26, 41-42, and 57-58. The aggregation process for the first data slices of other computing nodes in the second submodule is similar to that of the computing node numbered 10 and is not further described here. The aggregation process for the first data slices of other computing nodes in the first submodule is similar to that of the computing node numbered 1 and is not further described here.

[0150] For example, Figure 16B As shown, Figure 13BThe process of the second communication module composed of the computing nodes (XPU) of the G0 column and the G1 column in the illustrated 2D mesh topology network spreading the first data slice is taken as an example to illustrate the spreading process of the first data slice. The computing nodes with node numbers 1 and 2 in the first row, the computing nodes with node numbers 17 and 18 in the third row, the computing nodes with node numbers 33 and 34 in the fifth row, and the computing nodes with node numbers 49 and 50 in the seventh row constitute a third sub-module. The computing nodes with node numbers 9 and 10 in the second row, the computing nodes with node numbers 25 and 26 in the fourth row, the computing nodes with node numbers 41 and 42 in the sixth row, and the computing nodes with node numbers 57 and 58 in the eighth row constitute a fourth sub-module. The computing node with node number 1 in the third sub-module sends the aggregated first data slices of the computing nodes with node numbers 1, 3, 5, 7, 9, 11, 13 and 15 in the first sub-module to the corresponding first target nodes, i.e., the computing nodes with node numbers 1-2, 17-18, 33-34 and 49-50 in the third sub-module, respectively. The computing node with node number 2 in the third sub-module sends the aggregated first data slices of the computing nodes with node numbers 2, 4, 6, 8, 10, 12, 14 and 16 in the second sub-module to the corresponding first target nodes, i.e., the computing nodes with node numbers 1-2, 17-18, 33-34 and 49-50 in the third sub-module, respectively. The computing node with node number 10 in the fourth sub-module sends the aggregated first data slices of the computing nodes with node numbers 2, 4, 6, 8, 10, 12, 14 and 16 in the second sub-module to the corresponding first target nodes, i.e., the computing nodes with node numbers 9-10, 25-26, 41-42 and 57-58 in the fourth sub-module, respectively. The spreading processes of the first data slices of the other computing nodes in the third sub-module are similar to that of the computing node with node number 1, and are not described herein. The spreading processes of the first data slices of the other computing nodes in the fourth sub-module are similar to that of the computing node with node number 10, and are not described herein. Finally, each computing node in the second communication module receives the first data slice sent by the other computing node to the computing node.

[0151] Further, the second data aggregation rule comprises: each computing node 1 in the third sub-module as a second transition node, aggregating the second data slice sent by each computing node in the third sub-module, the second data slice taking the computing node 1 included in the first sub-module or the second sub-module corresponding to the second transition node as a second target node; each computing node in the fourth sub-module as a second transition node, aggregating the second data slice sent by each computing node in the fourth sub-module, the second data slice taking the computing node 1 included in the first sub-module or the second sub-module corresponding to the second transition node as a second target node.

[0152] The second data dissemination rule comprises: each computing node 1 in the first sub-module sending the aggregated second data slice to the corresponding computing node 1 in the first sub-module as a second target node; each computing node 1 in the second sub-module sending the aggregated second data slice to the corresponding computing node 1 in the second sub-module as a second target node.

[0153] Specifically, in the second data slice aggregation process, each computing node 1 in the third sub-module of the second communication module 20 as a second transition node will converge the second data slice, the second data slice aggregated by the second transition node is sent by each computing node 1 in the third sub-module, and is the second data slice sent to the computing node 1 included in the first sub-module or the second sub-module corresponding to the second transition node as a second target node. Each computing node 1 in the fourth sub-module of the second communication module 20 as a second transition node will converge the second data slice, the second data slice aggregated by the second transition node is sent by each computing node 1 in the fourth sub-module, and is the second data slice sent to the computing node 1 included in the first sub-module or the second sub-module corresponding to the second transition node as a second target node. Wherein, the first sub-module corresponding to the second transition node refers to the first sub-module to which the computing node 1 as the second transition node belongs; the second sub-module corresponding to the second transition node refers to the second sub-module to which the computing node 1 as the second transition node belongs. The third sub-module and the fourth sub-module perform communication in parallel.

[0154] Each computing node 1 in the first communication module 10 aggregates the second data slice as a second transition node in the second data slice aggregation process. In the second data slice dissemination process, each computing node 1 in the first submodule of the first communication module 10 sends the aggregated second data slice to the corresponding second target node, each computing node 1 in the first submodule being the second target node receiving the second data slice with each computing node 1 as the data receiving node. Each computing node 1 in the second submodule of the first communication module 10 sends the aggregated second data slice to the corresponding second target node, each computing node 1 in the second submodule being the second target node receiving the second data slice with each computing node 1 as the data receiving node. The first submodule and the second submodule communicate in parallel.

[0155] The mapping relationship between the second data slice in the computing node and the computing node is described by taking 64 computing nodes as an example. Each computing node has 64 cache spaces, denoted as 64 second data slices, and the arrangement order of the second data slices corresponds to the node number one by one. For standard all-to-all communication, the 64 second data slices in each computing node are all valid; the data contents in the 64 second data slices can be completely the same, partially the same, or completely different; for non-standard all-to-all communication, only part of the 64 second data slices in each computing node can be actually valid, and the invalid second data slices indicate that the computing node does not need to communicate with the computing node corresponding to the position of the second data slice; the data contents in the actually valid second data slices in each computing node can also be completely the same, partially the same, or completely different. As shown in Figure 17 3 6, the upper subscript represents the node number, the lower subscript represents the second data slice number in the computing node corresponding to the node number, and F 3 6 represents the second data slice with the second data slice number 6 in the computing node with the node number 3; at the same time, it is indicated that the computing node with the node number 3 needs to send the second data slice F

[0156] For example, as shown in Figure 18 Figure 13A The aggregation process of the second data slice is described by taking the process in which the computing nodes (XPU) in the G0 column and the G1 column in the 2D mesh topology network shown in Figure 17 ​​The computing nodes with node numbers 0 and 1 in the first row, the computing nodes with node numbers 16 and 17 in the third row, the computing nodes with node numbers 32 and 33 in the fifth row, and the computing nodes with node numbers 48 and 49 in the seventh row constitute a third submodule. The computing nodes with node numbers 14 and 15 in the second row, the computing nodes with node numbers 30 and 31 in the fourth row, the computing nodes with node numbers 46 and 47 in the sixth row, and the computing nodes with node numbers 62 and 63 in the eighth row constitute a fourth submodule. The computing node with node number 0 (J0G0) as a second transition node aggregates the second data slices sent by all the computing nodes in the third submodule to all the computing nodes in the second submodule constituted by the computing nodes 1 in the J0 row and the J1 row of the first communication module 10, and all the computing nodes in the second submodule are second target nodes, and the computing node with node number 0 corresponds to all the computing nodes in the second submodule. The computing node with node number 0 aggregates the second data slices with numbers 0, 2, 4, 6, 8, 10, 12, and 14 of the second data slices sent to the computing nodes with node numbers 0-1, 16-17, 32-33, and 48-49 in the second submodule. The computing node with node number 15 (J1G0) as a second transition node aggregates the second data slices sent by all the computing nodes in the first submodule to all the computing nodes in the first submodule constituted by the computing nodes 1 in the J0 row and the J1 row of the first communication module 10, and all the computing nodes in the first submodule are second target nodes, and the computing node with node number 15 corresponds to all the computing nodes in the first submodule. The computing node with node number 15 aggregates the second data slices with numbers 1, 3, 5, 7, 9, 11, 13, and 15 of the second data slices of the computing nodes with node numbers 14-15, 30-31, 46-47, and 62-63 in the fourth submodule. The aggregation process of the second data slices of other computing nodes in the third submodule is similar to the aggregation process of the first data slices of the computing node with node number 0, which is not described here. The aggregation process of the second data slices of other computing nodes in the fourth submodule is similar to the aggregation process of the second data slices of the computing node with node number 15, which is not described here.

[0157] For example, as Figure 19 shown, the first communication module 10 is divided into four submodules, and the computing nodes with node numbers 0-15 in the J0 row and the J1 row constitute a first submodule. The computing nodes with node numbers 16-31 in the J0 row and the J1 row constitute a second submodule. The computing nodes with node numbers 32-47 in the J0 row and the J1 row constitute a third submodule. The computing nodes with node numbers 48-63 in the J0 row and the J1 row constitute a fourth submodule. Figure 13AThe process of the first communication module composed of the computing nodes (XPU) of the J0 row and the J1 row in the illustrated 2D mesh topology network spreading the second data slice is taken as an example to illustrate the spreading process of the second data slice. The computing nodes numbered 0, 2, 4, 6, 8, 10, 12 and 14 constitute a second submodule, and the computing nodes numbered 1, 3, 5, 7, 9, 11, 13 and 15 constitute a first submodule. The computing node numbered 0 in the above-mentioned second submodule sends the aggregated second data slice to other computing nodes in the second submodule according to the second target node receiving the second data slice. All computing nodes in the above-mentioned second submodule serve as the second target node, and the computing node numbered 0 corresponds to all computing nodes in the above-mentioned second submodule. The computing node numbered 0 sends the second data slice numbered 0, 2, 4, 6, 8, 10, 12 and 14 of the aggregated second data slice of the computing nodes numbered 0-1, 16-17, 32-33 and 48-49 in the above-mentioned third submodule to the corresponding second target node, i.e., the computing nodes numbered 0, 2, 4, 6, 8, 10, 12 and 14 in the second submodule. The computing node numbered 15 sends the second data slice numbered 1, 3, 5, 7, 9, 11, 13 and 15 of the aggregated second data slice of the computing nodes numbered 14-15, 30-31, 46-47 and 62-63 in the above-mentioned fourth submodule to the corresponding second target node, i.e., the computing nodes numbered 1, 3, 5, 7, 9, 11, 13 and 15 in the first submodule. The spreading process of the second data slice of other computing nodes in the above-mentioned second submodule is similar to that of the computing node numbered 0, and will not be described here. The spreading process of the second data slice of other computing nodes in the above-mentioned first submodule is similar to that of the computing node numbered 15, and will not be described here. Finally, each computing node in the first communication module receives the second data slice sent by other computing nodes to the computing node. For example, 0~63 0 represents that the computing node numbered 0 receives 64 second data slices sent to the computing node numbered 0.

[0158] On the basis of the above-mentioned embodiments, further, the first division rule comprises: dividing each computing node 1 according to the column of the 2D mesh topology network, and taking every 2 columns of computing nodes 1 as a first communication module, and taking the computing nodes 1 in the odd rows as a fifth submodule and taking the computing nodes 1 in the even rows as a sixth submodule according to the parity of the row to which each computing node 1 in the first communication module 10 belongs in the 2D mesh topology network.

[0159] The second division rule comprises: dividing each computing node 1 according to the row of the 2D mesh topology network, and taking every 2 rows of computing nodes as a second communication module 20, and taking the computing nodes 1 with odd node numbers as a seventh submodule and taking the computing nodes 1 with even node numbers as an eighth submodule according to the parity of the node numbers of each computing node 1 in the second communication module 20; wherein the node numbers of each computing node 1 in the 2D mesh topology network are obtained in advance.

[0160] Specifically, every 2 columns of computing nodes 1 in the 2D mesh topology network constitute a first communication module 10. For each computing node 1 in each first communication module 10, the computing nodes 1 in the odd rows are taken as a fifth submodule and the computing nodes 1 in the even rows are taken as a sixth submodule according to the parity of the rows to which each computing node 1 belongs in the 2D mesh topology network.

[0161] Every 2 rows of computing nodes 1 in the 2D mesh topology network constitute a second communication module 20. For each computing node 1 in the 2D mesh topology network, the node numbers of each computing node 1 are obtained by numbering in Arabic numerals according to a preset numbering mode, and the node number of each computing node 1 is odd or even. The preset numbering mode is set according to actual needs, which is not limited by the embodiment of the application. For each computing node 1 in each second communication module 20, the computing nodes 1 with odd node numbers are taken as a first submodule and the computing nodes 1 with even node numbers are taken as a second submodule.

[0162] For example, the preset numbering mode is to take the computing node in the first row and the first column of the 2D mesh topology network as a starting node, number from left to right in the first row, number from right to left in the second row, number from left to right in the third row, number from right to left in the fourth row, and so on, to obtain the node numbers of each computing node 1 in the 2D mesh topology network. As shown in FIG. 2, starting from the computing node J0G0, the numbering is performed according to the above preset numbering mode with Arabic numeral 0 as a starting number, to obtain the numbers of each computing node. Figure 20A

[0163] For example, the preset numbering mode is to take the computing node in the first row and the first column of the 2D mesh topology network as a starting node, number from left to right in each row in a circular manner, that is, number from left to right in the first row, number from left to right in the second row, number from left to right in the third row, number from left to right in the fourth row, and so on, to obtain the node numbers of each computing node 1 in the 2D mesh topology network. As shown in FIG. 3, starting from the computing node J0G0, the numbering is performed according to the above preset numbering mode with Arabic numeral 0 as a starting number, to obtain the numbers of each computing node. Figure 20B ​As shown, starting from computing node J0G0, with Arabic numeral 1 as the starting number, numbering is performed according to the above preset numbering method to obtain the number of each computing node.

[0164] For example, Figure 20A As shown, every two columns of computing nodes 1 constitute a first communication module 10, for a total of four first communication modules 10. For the four second communication modules 20, based on the parity of the rows to which each computing node 1 in the first communication module 10 belongs in the 2D mesh topology network, the computing nodes 1 in odd-numbered rows serve as the third submodule, and the computing nodes 1 in even-numbered rows serve as the fourth submodule. For example, for the first communication module 10 composed of computing nodes 1 in columns G0 and G1, the computing nodes numbered 0 and 1 in the first row, 16 and 17 in the third row, 32 and 33 in the fifth row, and 48 and 49 in the seventh row constitute the fifth submodule. The computing nodes numbered 14 and 15 in the second row, 30 and 31 in the fourth row, 46 and 47 in the sixth row, and 62 and 63 in the eighth row constitute the sixth submodule.

[0165] The 2D mesh topology network consists of 8×8 compute nodes (XPUs) 1 connected in communication. Every two rows of compute nodes 1 form a second communication module 20, for a total of four second communication modules 20. Starting with compute node J0G0, the nodes are numbered sequentially from left to right and from right to left: compute node J0G0 is numbered 0, compute node J0G1 is numbered 1, compute node J0G2 is numbered 2, and so on. For each of the four second communication modules 20, compute nodes 1 with odd node numbers are designated as the seventh submodule, and compute nodes 1 with even node numbers are designated as the eighth submodule. For example, for the second communication module 20 formed by compute nodes 1 in rows J0 and J1, compute nodes 1 with node numbers 1, 3, 5, 7, 9, 11, 13, and 15 constitute the seventh submodule, and compute nodes 1 with node numbers 0, 2, 4, 6, 8, 10, 12, and 14 constitute the eighth submodule.

[0166] For example, Figure 20BAs shown, each 2 columns of the computing nodes 1 constitute a first communication module 10, and there are 4 first communication modules 10 in total. For the 4 second communication modules 20, according to the parity of the rows to which the respective computing nodes 1 in the first communication module 10 belong in the 2D mesh topology network, the computing nodes 1 in the odd rows are taken as the third sub-modules, and the computing nodes 1 in the even rows are taken as the fourth sub-modules. For example, for the first communication module 10 constituted by the computing nodes 1 in the G0 column and the G1 column, the computing nodes with the node numbers of 1 and 2 in the first row, the computing nodes with the node numbers of 17 and 18 in the third row, the computing nodes with the node numbers of 33 and 34 in the fifth row, and the computing nodes with the node numbers of 49 and 50 in the seventh row constitute the fifth sub-module. The computing nodes with the node numbers of 9 and 10 in the second row, the computing nodes with the node numbers of 25 and 26 in the fourth row, the computing nodes with the node numbers of 41 and 42 in the sixth row, and the computing nodes with the node numbers of 57 and 58 in the eighth row constitute the sixth sub-module.

[0167] The 2D mesh topology network is constituted by 8*8 computing nodes (XPU) 1 in communication connection, each 2 rows of the computing nodes 1 constitute a second communication module 20, and there are 4 second communication modules 20 in total. Starting from the computing node J0G0, the numbering is sequentially and circularly performed from left to right, the node number of the computing node J0G0 is 1, the node number of the computing node J0G1 is 2, the node number of the computing node J0G2 is 3, and so on. For the respective computing nodes 1 in the 4 second communication modules 20, the computing nodes 1 with the odd node numbers are taken as the seventh sub-modules, and the computing nodes 1 with the even node numbers are taken as the eighth sub-modules. For example, for the second communication module 20 constituted by the computing nodes 1 in the J0th row and the J1th row, the computing nodes 1 with the node numbers of 1, 3, 5, 7, 9, 11, 13 and 15 constitute the seventh sub-module, and the computing nodes 1 with the node numbers of 2, 4, 6, 8, 10, 12, 14 and 16 constitute the eighth sub-module.

[0168] Further, on the basis of the above-mentioned embodiments, the first data aggregation rule comprises: each computing node in the fifth sub-module as a first transition node, aggregating the first data slice sent by each computing node in the fifth sub-module, the first data slice taking the computing node included in the seventh sub-module or the eighth sub-module corresponding to the first transition node as a first target node; each computing node in the sixth sub-module as a first transition node, aggregating the first data slice sent by each computing node in the sixth sub-module, the first data slice taking the computing node included in the seventh sub-module or the eighth sub-module corresponding to the first transition node as a first target node.

[0169] The first data dissemination rule comprises: each computing node in the seventh submodule sends the aggregated first data slice to the corresponding computing node as the first target node in the seventh submodule; each computing node in the eighth submodule sends the aggregated first data slice to the corresponding computing node as the first target node in the eighth submodule.

[0170] Specifically, in the first data slice aggregation process, each computing node 1 in the fifth submodule of the first communication module 10 as a first transition node aggregates the first data slice, the first data slice aggregated by the first transition node is sent by each computing node 1 in the fifth submodule, and is the second data slice sent to the computing node 1 included in the seventh submodule or the eighth submodule corresponding to the first transition node as the first target node. Each computing node 1 in the sixth submodule of the second communication module 20 as a first transition node aggregates the first data slice, the first data slice aggregated by the first transition node is sent by each computing node 1 in the sixth submodule, and is the first data slice sent to the computing node 1 included in the seventh submodule or the eighth submodule corresponding to the first transition node as the first target node. Wherein, the seventh submodule corresponding to the first transition node refers to the seventh submodule to which the computing node 1 as the first transition node belongs; the eighth submodule corresponding to the first transition node refers to the eighth submodule to which the computing node 1 as the first transition node belongs. The fifth submodule and the sixth submodule perform communication in parallel.

[0171] Each computing node 1 in the second communication module 20 aggregates the first data slice as a first transition node in the first data slice aggregation process. In the first data slice dissemination process, each computing node 1 in the seventh submodule of the second communication module 20 sends the aggregated first data slice to the corresponding first target node, each computing node 1 in the seventh submodule as the first target node receives the first data slice with each computing node 1 as the data receiving node. Each computing node 1 in the eighth submodule of the second communication module 20 sends the aggregated first data slice to the corresponding first target node, each computing node 1 in the eighth submodule as the first target node receives the first data slice with each computing node 1 as the data receiving node. The seventh submodule and the eighth submodule perform communication in parallel.

[0172] For example, as shown in Figure 21A As shown in the 2D mesh topology network shown in Figure 20A The first communication module in the above first communication module aggregates the first data slice, each computing node includes 64 first data slices, and the specific representation is as shown in Figure 14AThe computing nodes with node numbers of 0 and 1 in the first row, the computing nodes with node numbers of 16 and 17 in the third row, the computing nodes with node numbers of 32 and 33 in the fifth row, and the computing nodes with node numbers of 48 and 49 in the seventh row in the 2D mesh topology network constitute a fifth sub-module. The computing nodes with node numbers of 14 and 15 in the second row, the computing nodes with node numbers of 30 and 31 in the fourth row, the computing nodes with node numbers of 46 and 47 in the sixth row, and the computing nodes with node numbers of 62 and 63 in the eighth row constitute a sixth sub-module. The computing node with node number of 0 (J0G0) as a first transition node aggregates all the first data slices sent by all the computing nodes in the fifth sub-module to all the computing nodes in an eighth sub-module in a second communication module 20 constituted by the computing nodes 1 in the J0 row and the J1 row, and all the computing nodes in the eighth sub-module are first target nodes, and the computing node with node number of 0 corresponds to all the computing nodes in the eighth sub-module. The computing node with node number of 0 aggregates the first data slices with numbers of 0, 2, 4, 6, 8, 10, 12 and 14 sent to the computing nodes with node numbers of 0-1, 16-17, 32-33 and 48-49 in the eighth sub-module. The computing node with node number of 15 (J1G0) as a first transition node aggregates all the first data slices sent by all the computing nodes in the eighth sub-module to all the computing nodes in a seventh sub-module in the second communication module 20 constituted by the computing nodes 1 in the J0 row and the J1 row, and all the computing nodes in the seventh sub-module are first target nodes, and the computing node with node number of 15 corresponds to all the computing nodes in the seventh sub-module. The computing node with node number of 15 aggregates the first data slices with numbers of 1, 3, 5, 7, 9, 11, 13 and 15 of the computing nodes with node numbers of 14-15, 30-31, 46-47 and 62-63 in the sixth sub-module. The aggregation processes of the first data slices of other computing nodes in the fifth sub-module are similar to the aggregation process of the first data slices of the computing node with node number of 0, which will not be described herein. The aggregation processes of the first data slices of other computing nodes in the sixth sub-module are similar to the aggregation process of the first data slices of the computing node with node number of 15, which will not be described herein.

[0173] For example, as Figure 21B shown, the first data slice of the computing node with node number of 0 (J0G0) is aggregated by the computing node with node number of 0 in the J0 row and the computing node with node number of 1 in the J1 row, and the first data slice of the computing node with node number of 15 (J1G0) is aggregated by the computing node with node number of 15 in the J0 row and the computing node with node number of 16 in the J1 row. Figure 20AThe process of the second communication module composed of the computing nodes (XPU) of the J0 row and the J1 row in the 2D mesh topology network shown disperses the first data slices. The computing nodes numbered 0, 2, 4, 6, 8, 10, 12, and 14 constitute an eighth submodule, and the computing nodes numbered 1, 3, 5, 7, 9, 11, 13, and 15 constitute a seventh submodule. The computing node numbered 0 in the eighth submodule (J0G0) sends the aggregated first data slices to other computing nodes in the eighth submodule according to the first target nodes that receive the first data slices. All computing nodes in the eighth submodule serve as the first target nodes, and the computing node numbered 0 corresponds to all computing nodes in the eighth submodule. The computing node numbered 0 sends the first data slices numbered 0, 2, 4, 6, 8, 10, 12, and 14 of the first data slices of the computing nodes numbered 0-1, 16-17, 32-33, and 48-49 in the eighth submodule to the corresponding second target nodes, that is, the computing nodes numbered 0, 2, 4, 6, 8, 10, 12, and 14 in the eighth submodule. The computing node numbered 15 sends the first data slices numbered 1, 3, 5, 7, 9, 11, 13, and 15 of the first data slices of the computing nodes numbered 14-15, 30-31, 46-47, and 62-63 in the sixth submodule to the corresponding second target nodes, that is, the computing nodes numbered 1, 3, 5, 7, 9, 11, 13, and 15 in the seventh submodule. The dispersing process of the first data slices of other computing nodes in the eighth submodule is similar to the dispersing process of the first data slices of the computing node numbered 0, which is not described here. The dispersing process of the first data slices of other computing nodes in the seventh submodule is similar to the dispersing process of the first data slices of the computing node numbered 15, which is not described here. Finally, each computing node in the second communication module receives the first data slices sent by other computing nodes to the computing node. For example, 0~63 0 represents that the computing node numbered 0 receives 64 first data slices sent to the computing node numbered 0.

[0174] For example, as shown in FIG. 8, the first communication module composed of the computing nodes (XPU) of the G0 column and the G1 column in the 2D mesh topology network shown aggregates the first data slices. Each computing node in the first communication module includes 64 first data slices, which are specifically represented as shown in FIG. 9. Figure 22A Figure 20B For example, as shown in FIG. 8, the first communication module composed of the computing nodes (XPU) of the G0 column and the G1 column in the 2D mesh topology network shown aggregates the first data slices. Each computing node in the first communication module includes 64 first data slices, which are specifically represented as shown in FIG. 9. Figure 14B ​The computing nodes with node numbers of 1 and 2 in the first row, the computing nodes with node numbers of 17 and 18 in the third row, the computing nodes with node numbers of 33 and 34 in the fifth row, and the computing nodes with node numbers of 49 and 50 in the seventh row in the 2D mesh topology network constitute a fifth sub-module. The computing nodes with node numbers of 9 and 10 in the second row, the computing nodes with node numbers of 25 and 26 in the fourth row, the computing nodes with node numbers of 41 and 42 in the sixth row, and the computing nodes with node numbers of 57 and 58 in the eighth row constitute a sixth sub-module. The computing node with node number of 1 (J0G0) as a first transition node aggregates all the first data slices sent by all the computing nodes in the fifth sub-module to all the computing nodes in a seventh sub-module in the second communication module 20 constituted by the computing nodes 1 in the J0 row and the J1 row, all the computing nodes in the seventh sub-module as first target nodes, and the computing node with node number of 1 corresponding to all the computing nodes in the seventh sub-module. The computing node with node number of 1 aggregates the first data slices with numbers of 1, 3, 5, 7, 9, 11, 13 and 15 sent to the computing nodes with node numbers of 1-2, 17-18, 33-34 and 49-50 in the seventh sub-module. The computing node with node number of 10 (J1G1) as a first transition node aggregates all the first data slices sent by all the computing nodes in the eighth sub-module to all the computing nodes in an eighth sub-module in the second communication module 20 constituted by the computing nodes 1 in the J0 row and the J1 row, all the computing nodes in the eighth sub-module as first target nodes, and the computing node with node number of 10 corresponding to all the computing nodes in the eighth sub-module. The computing node with node number of 10 aggregates the first data slices with numbers of 2, 4, 6, 8, 10, 12, 14 and 16 sent to the computing nodes with node numbers of 9-10, 25-26, 41-42 and 57-58 in the sixth sub-module. The aggregation processes of the first data slices of other computing nodes in the fifth sub-module are similar to that of the computing node with node number of 1, which will not be described herein. The aggregation processes of the first data slices of other computing nodes in the sixth sub-module are similar to that of the computing node with node number of 10, which will not be described herein.

[0175] For example, as Figure 22B shown, the first data slice sent by the computing node with node number of 1 in the first row in the 2D mesh topology network is sent to the computing node with node number of 1 in the J0 row and the J1 row, and the first data slice sent by the computing node with node number of 10 in the second row is sent to the computing node with node number of 10 in the J0 row and the J1 row. Figure 20BThe process of the second communication module composed of the computing nodes (XPU) of J0 row and J1 row in the illustrated 2D mesh topology network spreading the first data slice is taken as an example to illustrate the spreading process of the first data slice. The computing nodes numbered 2, 4, 6, 8, 10, 12, 14 and 16 constitute an eighth submodule, and the computing nodes numbered 1, 3, 5, 7, 9, 11, 13 and 15 constitute a seventh submodule. The computing node numbered 2 (J0G1) in the eighth submodule transmits the aggregated first data slice to other computing nodes in the eighth submodule according to the first target node receiving the first data slice. All computing nodes in the eighth submodule serve as the first target node, and the computing node numbered 2 corresponds to all computing nodes in the eighth submodule. The computing node numbered 2 transmits the first data slice numbered 2, 4, 6, 8, 10, 12, 14 and 16 of the aggregated first data slice of the computing nodes numbered 1-2, 17-18, 33-34, 49-50 in the fifth submodule to the corresponding second target node, i.e., the computing nodes numbered 2, 4, 6, 8, 10, 12, 14 and 16 in the eighth submodule. The computing node numbered 9 transmits the first data slice numbered 1, 3, 5, 7, 9, 11, 13 and 15 of the aggregated first data slice of the computing nodes numbered 9-10, 25-26, 41-42, 57-58 in the sixth submodule to the corresponding second target node, i.e., the computing nodes numbered 1, 3, 5, 7, 9, 11, 13 and 15 in the seventh submodule. The spreading process of the first data slice of other computing nodes in the eighth submodule is similar to that of the computing node numbered 2, and will not be described here. The spreading process of the first data slice of other computing nodes in the seventh submodule is similar to that of the computing node numbered 9, and will not be described here. Finally, each computing node in the second communication module receives the first data slice transmitted to the computing node by other computing nodes. For example, 1~64 1 represents that the computing node numbered 1 receives 64 first data slices transmitted to the computing node numbered 1.

[0176] Further, the second data aggregation rule comprises: each computing node 1 in the seventh submodule as a second transition node, aggregates the second data slice sent by each computing node 1 in the seventh submodule, and the second data slice takes the computing node 1 included in the fifth submodule or the sixth submodule corresponding to the second transition node as the second target node; each computing node 1 in the eighth submodule as a second transition node, aggregates the second data slice sent by each computing node 1 in the eighth submodule, and the second data slice takes the computing node 1 included in the seventh submodule or the eighth submodule corresponding to the second transition node as the second target node.

[0177] The second data dissemination rule comprises: each computing node 1 in the fifth submodule sends the aggregated second data slice to the corresponding computing node as the second target node in the fifth submodule; each computing node 1 in the sixth submodule sends the aggregated second data slice to the corresponding computing node as the second target node in the sixth submodule.

[0178] Specifically, in the second data slice aggregation process, each computing node 1 in the seventh submodule of the second communication module 20 as a second transition node will converge the second data slice, the second data slice aggregated by the second transition node is sent by each computing node 1 in the seventh submodule, and is the second data slice sent to the computing node 1 included in the fifth submodule or the sixth submodule corresponding to the second target node. Each computing node 1 in the eighth submodule of the second communication module 20 as a second transition node will converge the second data slice, the second data slice aggregated by the second transition node is sent by each computing node 1 in the eighth submodule, and is the second data slice sent to the computing node 1 included in the fifth submodule or the sixth submodule corresponding to the second target node. Wherein, the fifth submodule corresponding to the second transition node refers to the fifth submodule to which the computing node 1 as the second transition node belongs; the sixth submodule corresponding to the second transition node refers to the sixth submodule to which the computing node 1 as the second transition node belongs. The seventh submodule and the eighth submodule perform communication in parallel.

[0179] Each computing node 1 in the first communication module 10 aggregates a second data slice as a second transition node in the second data slice aggregation process. In the second data slice dissemination process, each computing node 1 in the fifth submodule of the first communication module 10 sends the aggregated second data slice to the corresponding second target node, each computing node 1 in the fifth submodule being a second target node that receives the second data slice with each computing node 1 as a data receiving node. Each computing node 1 in the sixth submodule of the first communication module 10 sends the aggregated second data slice to the corresponding second target node, each computing node 1 in the sixth submodule being a second target node that receives the second data slice with each computing node 1 as a data receiving node. The fifth submodule and the sixth submodule communicate in parallel.

[0180] For example, as shown in FIG. 2, the second communication module composed of the computing nodes (XPU) of the J0 row and the J1 row in the 2D mesh topology network shown in FIG. 1 aggregates the second data slice. Figure 23 As shown in FIG. 2, the second communication module composed of the computing nodes (XPU) of the J0 row and the J1 row in the 2D mesh topology network shown in FIG. 1 aggregates the second data slice. Figure 20A The aggregation process of the second data slice is described by taking the process of the second communication module composed of the computing nodes (XPU) of the J0 row and the J1 row in the 2D mesh topology network shown in FIG. 2 as an example. Each computing node in the above-mentioned second communication module includes 64 second data slices, which are specifically represented as shown in FIG. 3. Figure 17As shown. Computing nodes numbered 0, 2, 4, 6, 8, 10, 12, and 14 constitute the eighth submodule, while computing nodes numbered 1, 3, 5, 7, 9, 11, 13, and 15 constitute the seventh submodule. Computing node numbered 0 (J0G0) acts as the second transition node and aggregates the second data slices sent by all computing nodes in the eighth submodule to all computing nodes in the fifth submodule of the first communication module 10, which is composed of computing nodes 1 in columns G0 and G1. All computing nodes in the fifth submodule serve as second target nodes, and computing node J0G0 corresponds to all computing nodes in the fifth submodule. Computing node J0G0 aggregates the second data slices numbered 0-1, 16-17, 32-33, and 48-49 from computing nodes numbered 0, 2, 4, 6, 8, 10, 12, and 14 in the eighth submodule. Compute node (J1G0) numbered 15 acts as the second transition node and aggregates the second data slices sent by all compute nodes in the seventh submodule to all compute nodes in the sixth submodule of the first communication module 10, which is composed of compute nodes 1 in columns G0 and G1. All compute nodes in the sixth submodule serve as second destination nodes, and compute node J1G0 corresponds to all compute nodes in the sixth submodule. Compute node J1G0 aggregates the second data slices numbered 14-15, 30-31, 46-47, and 62-63 for the J1G0-th data slices of compute nodes numbered 1, 3, 5, 7, 9, 11, 13, and 15 in the seventh submodule. The aggregation process for the second data slices of the other compute nodes in the eighth submodule is similar to that for the compute node numbered 0 and is not further described here. The aggregation process for the second data slices of the other compute nodes in the seventh submodule is similar to that for the compute node numbered 15 and is not further described here.

[0181] For example, Figure 24 As shown, Figure 20AThe process of the first communication module composed of the computing nodes (XPU) of the G0 column and the G1 column in the illustrated 2D mesh topology network spreading the second data slice is taken as an example to illustrate the spreading process of the second data slice. The computing nodes with the node numbers 0 and 1 in the first row, the computing nodes with the node numbers 16 and 17 in the third row, the computing nodes with the node numbers 32 and 33 in the fifth row, and the computing nodes with the node numbers 48 and 49 in the seventh row in the 2D mesh topology network constitute a fifth sub-module. The computing nodes with the node numbers 14 and 15 in the second row, the computing nodes with the node numbers 30 and 31 in the fourth row, the computing nodes with the node numbers 46 and 47 in the sixth row, and the computing nodes with the node numbers 62 and 63 in the eighth row constitute a sixth sub-module. The computing node with the node number 0 in the fifth sub-module sends the aggregated second data slice of the computing nodes with the node numbers 0, 2, 4, 6, 8, 10, 12 and 14 in the eighth sub-module to other computing nodes in the fifth sub-module according to the second target node receiving the second data slice, i.e., the computing nodes with the node numbers 0~1, 16~17, 32~33, 48~49 in the fifth module. The computing node with the node number 15 in the seventh sub-module sends the aggregated second data slice of the computing nodes with the node numbers 1, 3, 5, 7, 9, 11, 13 and 15 in the seventh sub-module to other computing nodes in the seventh sub-module according to the second target node receiving the second data slice, i.e., the computing nodes with the node numbers 14~15, 30~31, 46~47, 62~63 in the sixth module. The spreading process of the second data slice of other computing nodes in the fifth sub-module is similar to the spreading process of the second data slice of the computing node with the node number 0, and details are not described herein. The spreading process of the second data slice of other computing nodes in the sixth sub-module is similar to the spreading process of the second data slice of the computing node with the node number 15, and details are not described herein. Finally, each computing node in the first communication module receives the second data slice sent by other computing nodes to the computing node. For example, 0~63 0 represents that the computing node with the node number 0 receives 64 second data slices sent to the computing node with the node number 0.

[0182] On the basis of the above-mentioned embodiments, further, the first division rule comprises: dividing the computing nodes 1 in the 2D mesh topology network into a plurality of first communication modules 10 according to 2 rows n columns, each first communication module 10 comprising 2×n computing nodes 1, and the total number of columns of the 2D mesh topology network being 2n;

[0183] The second division rule comprises: dividing the computing nodes 1 in the 2D mesh topology network by columns, the computing nodes 1 in the ith column and the ith + n column as a second communication module 20, and according to the parity of the rows to which the computing nodes 1 in the second communication module 20 belong in the 2D mesh topology network, the computing nodes 1 in the odd rows as a ninth submodule and the computing nodes 1 in the even rows as a tenth submodule, i is greater than or equal to 1 and less than or equal to n.

[0184] Specifically, every 2 rows and n columns of computing nodes 1 in the 2D mesh topology network form a first communication module 10, each first communication module 10 includes 2x n computing nodes 1, and the total number of columns of the 2D mesh topology network is 2n.

[0185] The computing nodes 1 in the 2D mesh topology network are divided by columns, and the computing nodes in the ith column and the ith + n column are a second communication module 20. For each computing node 1 in each second communication module 20, according to the parity of the rows to which the computing nodes 1 in the second communication module 20 belong in the 2D mesh topology network, the computing nodes 1 in the odd rows are a ninth submodule, and the computing nodes 1 in the even rows are a tenth submodule.

[0186] For example, as shown in Figure 25 The 2D mesh topology network is composed of 8x8 computing nodes (XPU) 1 communication connections, every 2 rows and 4 columns of computing nodes 1 form a first communication module 10, and there are 8 first communication modules 10. In the topology network composed of 8 computing nodes 1 in each first communication module 10, the computing node 1 at the upper left corner is the starting node, which is numbered in the clockwise direction in sequence, and each first communication module 10 is numbered continuously from left to right and from top to bottom, and the node number of each computing node is as shown in Figure 25 .

[0187] The computing nodes 1 in the first column and the fifth column in the 2D mesh topology network are divided into a second communication module 20, the computing nodes 1 in the second column and the sixth column are divided into a second communication module 20, the computing nodes 1 in the third column and the seventh column are divided into a second communication module 20, and the computing nodes 1 in the fourth column and the eighth column are divided into a second communication module 20, and there are four second communication modules 20. For the four second communication modules 20, according to the parity of the rows to which the computing nodes 1 in the second communication module 20 belong in the 2D mesh topology network, the computing nodes 1 in the odd rows are the ninth sub-module, and the computing nodes 1 in the even rows are the tenth sub-module. For example, for the second communication module 20 formed by the computing nodes 1 in the G0 column and the G4 column, the computing nodes with node numbers 0 and 8 in the first row, the computing nodes with node numbers 16 and 24 in the third row, the computing nodes with node numbers 32 and 40 in the fifth row, and the computing nodes with node numbers 48 and 56 in the seventh row form the ninth sub-module. The computing nodes with node numbers 7 and 15 in the second row, the computing nodes with node numbers 23 and 31 in the fourth row, the computing nodes with node numbers 39 and 47 in the sixth row, and the computing nodes with node numbers 55 and 63 in the eighth row form the tenth sub-module.

[0188] Further, on the basis of the above-mentioned embodiments, the first data aggregation rule includes: each computing node 1 in the first communication module 10 as a first transition node, aggregating the first data slice sent by each computing node 1 in the first communication module 10, and the first data slice taking the computing node 1 included in the ninth sub-module or the tenth sub-module corresponding to the first transition node as the first target node.

[0189] The first data scattering rule includes: each computing node 1 in the ninth sub-module sends the aggregated first data slice to the corresponding computing node as the first target node in the ninth sub-module; and each computing node 1 in the tenth sub-module sends the aggregated first data slice to the corresponding computing node as the first target node in the tenth sub-module.

[0190] Specifically, in the first data slice aggregation process, each computing node 1 in the first communication module 10 as a first transition node will aggregate the first data slice, and the first data slice aggregated by the first transition node is sent by each computing node 1 in the first sub-module and is the first data slice sent to the computing node 1 included in the ninth sub-module or the tenth sub-module corresponding to the first transition node as the first target node. Among them, the ninth sub-module corresponding to the first transition node refers to the ninth sub-module to which the computing node 1 as the first transition node belongs; and the tenth sub-module corresponding to the first transition node refers to the tenth sub-module to which the computing node 1 as the first transition node belongs.

[0191] During the first data slice aggregation process, each computing node 1 in the second communication module 20 acts as a first transition node to aggregate the first data slice. During the first data slice dissemination process, each computing node 1 in the ninth submodule of the second communication module 20 sends the aggregated first data slice to the corresponding first destination node. Each computing node 1 in the ninth submodule acts as a first destination node and receives the first data slice with each computing node 1 as the data receiving node. Each computing node 1 in the tenth submodule of the second communication module 20 sends the aggregated first data slice to the corresponding first destination node. Each computing node 1 in the tenth submodule acts as a first destination node and receives the first data slice with each computing node 1 as the data receiving node. The ninth and tenth submodules communicate in parallel.

[0192] For example, Figure 26 As shown, Figure 25 The process of aggregating the first data slices is described by taking the first communication module composed of the computing nodes (XPUs) in rows J0 and J1 and columns G0 to G3 in the 2D mesh topology network shown in FIG. Each computing node in the first communication module includes 64 first data slices, which are specifically represented as follows: Figure 14A As shown. The compute node (J0G0) with node number 0 acts as the first transition node and aggregates the first data slices sent by all compute nodes in the first communication module to all compute nodes in the ninth submodule of the second communication module 20, which is composed of compute nodes 1 in columns G0 and G4. All compute nodes in the ninth submodule serve as first destination nodes, and compute node J0G0 corresponds to all compute nodes in the ninth submodule. Computing node J0G0 aggregates the first data slices numbered 0, 8, 16, 24, 32, 40, 48, and 56 sent by all compute nodes in the first communication module. Computing node (J1G0) with node number 7 acts as the first transition node and aggregates the first data slices sent by all compute nodes in the first communication module to all compute nodes in the tenth submodule of the second communication module 20, which is composed of compute nodes 1 in columns G0 and G4. All compute nodes in the tenth submodule serve as first destination nodes, and compute node J1G0 corresponds to all compute nodes in the tenth submodule. Computing node J1G0 aggregates the first data slices numbered 7, 15, 23, 31, 39, 47, 55, and 63, sent by all computing nodes in the first communication module. There are eight computing nodes in the first communication module, numbered 0, 1, 2, 3, 4, 5, 6, and 7. The process of aggregating the first data slices by the other computing nodes in the first communication module is similar to the process of aggregating the first data slice by computing node J0G0 or computing node J1G0, and is not further described here.

[0193] For example, Figure 27 and Figure 28 As shown, Figure 25 The dissemination process of the first data slice is described by taking the second communication module composed of the computing nodes (XPUs) in columns G0 and G4 of the 2D mesh topology network as an example. The computing nodes in the first row (row J0), the third row (row J2), the fifth row (row J4), and the seventh row (row J6) of the second communication module G0 and G4 as the ninth submodule, as shown in FIG. Figure 27 The computing nodes in the second row (J1 row), the computing nodes in the fourth row (J3 row), the computing nodes in the sixth row (J5 row) and the computing nodes in the eighth row (J7 row) of the second communication module G0 and G4 columns above constitute the tenth submodule, as shown in FIG. Figure 28 shown.

[0194] The computing node J0G0 in the ninth submodule sends the aggregated first data slice to other computing nodes in the ninth submodule according to the first target node that receives the first data slice. Each computing node in the ninth submodule can serve as the first target node. That is, the computing node with node number 0 sends the aggregated first data slices of the computing nodes with node numbers 0, 1, 2, 3, 4, 5, 6 and 7 in the first communication module 10, and the first data slices with numbers 0, 8, 16, 24, 32, 40, 48 and 56 to the corresponding first target nodes, that is, the computing nodes with node numbers 8, 16, 24, 32, 40, 48 and 56 in the ninth submodule. The dissemination process of the first data slices of other computing nodes in the ninth submodule is similar to the dissemination process of the first data slice of the computing node J0G0, and will not be repeated here. Finally, each computing node in the ninth submodule receives the first data slice sent to the computing node by other computing nodes. For example, E 0~63 0 indicates the 64 first data slices received by the computing node with the node number 0 and sent to the computing node with the node number 0.

[0195] The computing node J1G0 in the tenth sub-module aggregates the first data slices, and sends the aggregated first data slices to other computing nodes in the tenth sub-module according to the first target nodes receiving the first data slices. Each computing node in the tenth sub-module can serve as the first target node. That is, the computing node numbered 7 sends the first data slices numbered 7, 15, 23, 31, 39, 47, 55 and 63 of the first data slices of the computing nodes numbered 1, 3, 5, 7, 9, 11, 13 and 15 in the first communication module 10 to the corresponding first target nodes, i.e., the computing nodes numbered 7, 15, 23, 31, 39, 47, 55 and 63 in the tenth sub-module. The spreading process of the first data slices of other computing nodes in the tenth sub-module is similar to that of the first data slices of the computing node J1G0, which is not described here. Finally, each computing node in the tenth sub-module receives the first data slices sent by other computing nodes.

[0196] On the basis of the above embodiments, further, the second data aggregation rule includes: each computing node 1 in the ninth sub-module as a second transition node aggregates the second data slices sent by each computing node 1 in the ninth sub-module, and the second data slices take the computing node 1 included in the first communication module corresponding to the second transition node as the second target node; and each computing node 1 in the tenth sub-module as a second transition node aggregates the second data slices sent by each computing node 1 in the tenth sub-module, and the second data slices take the computing node 1 included in the first communication module corresponding to the second transition node as the second target node.

[0197] The second data spreading rule includes: each computing node 1 in the first communication module sends the aggregated second data slices to the corresponding computing node in the first communication module as the second target node.

[0198] Specifically, in the second data slice aggregation process, each computing node 1 in the ninth sub-module of the second communication module 20 as a second transition node aggregates the second data slices, the second data slices aggregated by the second transition node are sent by each computing node 1 in the ninth sub-module, and are the second data slices sent to the computing node 1 included in the first communication module 10 corresponding to the second transition node as the second target node. Each computing node 1 in the tenth sub-module of the second communication module 20 as a second transition node aggregates the second data slices, the second data slices aggregated by the second transition node are sent by each computing node 1 in the tenth sub-module, and are the second data slices sent to the computing node 1 included in the first communication module 10 corresponding to the second transition node as the second target node. Wherein, the first communication module corresponding to the second transition node refers to the first communication module to which the computing node 1 as the second transition node belongs.

[0199] For example, Figure 29 and Figure 30 As shown in FIG, the process of aggregating the second data slices by the second communication module composed of the computing nodes (XPU) of the G0 column and the G4 column in the 2D mesh topology network is taken as an example to illustrate the aggregation process of the second data slices. Each computing node in the above second communication module includes 64 second data slices, which are specifically represented as follows Figure 17 The computing nodes in the first row (J0 row), the computing nodes in the third row (J2 row), the computing nodes in the fifth row (J4 row) and the computing nodes in the seventh row (J6 row) of the second communication module G0 and G4 constitute the ninth submodule, as shown in FIG. Figure 29 The computing nodes in the second row (J1 row), the computing nodes in the fourth row (J3 row), the computing nodes in the sixth row (J5 row) and the computing nodes in the eighth row (J7 row) of the second communication module G0 and G4 columns above constitute the tenth submodule, as shown in FIG. Figure 30 shown.

[0200] Computing node J0G0, acting as a second transition node, aggregates the second data slices sent by all computing nodes in the ninth submodule to all computing nodes in the first communication module to which computing node J0G0 belongs. All computing nodes in the first communication module to which computing node J0G0 belongs serve as second target nodes, and computing node J0G0 corresponds to all computing nodes in the aforementioned first communication module. That is, computing node numbered 0 aggregates the second data slices of computing nodes numbered 0, 8, 16, 24, 32, 40, 48, and 56 in the aforementioned ninth submodule, including the second data slices numbered 0, 1, 2, 3, 4, 5, 6, and 7. Computing node J0G4, acting as a second transition node, aggregates the second data slices sent by all computing nodes in the ninth submodule to all computing nodes in the first communication module to which computing node J0G4 belongs. All computing nodes in the first communication module to which computing node J0G4 belongs serve as second target nodes, and computing node J0G4 corresponds to all computing nodes in the aforementioned first communication module. That is, the computing node with node number 8 aggregates the second data slices of the computing nodes with node numbers 0, 8, 16, 24, 32, 40, 48, and 56 in the ninth submodule, which are second data slices numbered 8, 9, 10, 11, 12, 13, 14, and 15. The process of aggregating the second data slices by other computing nodes in the ninth submodule is similar to the process of aggregating the second data slices by computing nodes J0G0 and J0G4, and is not further described here.

[0201] Computing node J1G0, acting as the second transition node, aggregates the second data slices sent by all computing nodes in the tenth submodule to all computing nodes in the first communication module to which computing node J1G0 belongs. All computing nodes in the first communication module to which computing node J1G0 belongs serve as second target nodes, and computing node J1G0 corresponds to all computing nodes in the first communication module. That is, computing node 7 aggregates the second data slices (numbered 0, 1, 2, 3, 4, 5, 6, and 7) of computing nodes numbered 7, 15, 23, 31, 39, 47, 55, and 63 in the tenth submodule. Computing node J1G4, acting as the second transition node, aggregates the second data slices sent by all computing nodes in the tenth submodule to all computing nodes in the first communication module to which computing node J1G4 belongs. All computing nodes in the first communication module to which computing node J1G4 belongs serve as second target nodes, and computing node J1G4 corresponds to all computing nodes in the first communication module. That is, the computing node numbered 15 aggregates the second data slices of the computing nodes numbered 7, 15, 23, 31, 39, 47, 55, and 63 in the tenth submodule, which are the second data slices of numbers 8, 9, 10, 11, 12, 13, 14, and 15. The process of aggregating the second data slices by the other computing nodes in the tenth submodule is similar to the process of aggregating the second data slices by computing nodes J1G0 and J1G4, and is not further described here.

[0202] For example, Figure 31 As shown, the process of disseminating the second data slice in the first communication module, consisting of the compute nodes (XPUs) in rows J0 and J1 and columns G0 to G3 in a 2D mesh topology network, is used as an example to illustrate the dissemination process of the second data slice. Compute node J0G0 in the first communication module sends the aggregated second data slice to the other compute nodes in the first communication module, based on the second target nodes that received the second data slice. Each compute node in the first communication module can serve as a second target node. Specifically, compute node 0 sends the aggregated second data slices (numbered 0, 1, 2, 3, 4, 5, 6, and 7) from compute nodes numbered 0, 8, 16, 24, 32, 40, 48, and 56 in the ninth submodule to the corresponding second target nodes, namely, compute nodes numbered 1, 2, 3, 4, 5, 6, and 7 in the first communication module. The dissemination process of the second data slices for the other compute nodes in the first communication module is similar to that for compute node J0G0 and is not further described here. Finally, each computing node in the first communication module receives the second data slice sent by other computing nodes to the computing node. 0~630 represents that the computing node numbered 0 receives 64 second data slices sent to the computing node numbered 0.

[0203] Further based on the above embodiments, the first division rule comprises: dividing the computing nodes 1 in the 2D mesh topology network by columns, the computing nodes 1 in the i th column and the i + n th column being a first communication module 10, and according to the parity of the rows to which the computing nodes in the first communication module 10 belong in the 2D mesh topology network, the computing nodes 1 in the odd rows being an eleventh submodule, and the computing nodes 1 in the even rows being a twelfth submodule, i being greater than or equal to 1 and less than or equal to n, and the total number of columns of the 2D mesh topology network being 2n.

[0204] The second division rule comprises: dividing the computing nodes 1 in the 2D mesh topology network into a plurality of second communication modules 20 according to 2 rows and n columns, each second communication module 20 comprising 2x n computing nodes.

[0205] Specifically, the computing nodes 1 in the 2D mesh topology network are divided by columns, the computing nodes in the i th column and the i + n th column being a first communication module 10. For each computing node 1 in each first communication module 10, according to the parity of the rows to which the computing nodes 1 in the first communication module 10 belong in the 2D mesh topology network, the computing nodes 1 in the odd rows are an eleventh submodule, and the computing nodes 1 in the even rows are a twelfth submodule.

[0206] The computing nodes 1 in every 2 rows and n columns in the 2D mesh topology network form a second communication module 20, each second communication module 20 comprising 2x n computing nodes 1, and the total number of columns of the 2D mesh topology network being 2n.

[0207] For example, as Figure 32As shown, the computing nodes 1 in the 2D mesh topology network are divided by column, the computing nodes 1 in the first column and the fifth column are taken as a first communication module 10, the computing nodes 1 in the second column and the sixth column are taken as a first communication module 10, the computing nodes 1 in the third column and the seventh column are taken as a first communication module 10, and the computing nodes 1 in the fourth column and the eighth column are taken as a first communication module 10, a total of four first communication modules 10. For the four first communication modules 10, according to the parity of the rows to which the computing nodes 1 in the first communication module 10 belong in the 2D mesh topology network, the computing nodes 1 in the odd rows are taken as an eleventh submodule, and the computing nodes 1 in the even rows are taken as a twelfth submodule. For example, for the first communication module 10 composed of the computing nodes 1 in the G0 column and the G4 column, the computing nodes with node numbers 0 and 8 in the first row, the computing nodes with node numbers 16 and 24 in the third row, the computing nodes with node numbers 32 and 40 in the fifth row, and the computing nodes with node numbers 48 and 56 in the seventh row constitute the eleventh submodule. The computing nodes with node numbers 7 and 15 in the second row, the computing nodes with node numbers 23 and 31 in the fourth row, the computing nodes with node numbers 39 and 47 in the sixth row, and the computing nodes with node numbers 55 and 63 in the eighth row constitute the twelfth submodule.

[0208] The 2D mesh topology network is composed of 8x8 computing nodes (XPU) 1 in communication connection, and each 2 rows and 4 columns of computing nodes 1 constitute a second communication module 20, a total of 8 second communication modules 20. In each second communication module 20, the computing node 1 at the top left corner of the topology network composed of 8 computing nodes 1 is taken as a starting node, and is sequentially numbered in a clockwise direction, and each second communication module 20 is sequentially numbered from left to right and from top to bottom to obtain the node numbers of each computing node as shown in Figure 32 .

[0209] On the basis of the above embodiments, further, the first data aggregation rule includes: each computing node 1 in the eleventh submodule as a first transition node, aggregating the first data slice sent by each computing node 1 in the eleventh submodule, and the first data slice taking the computing node included in the second communication module 20 corresponding to the first transition node as a first target node; each computing node 1 in the twelfth submodule as a first transition node, aggregating the first data slice sent by each computing node 1 in the twelfth submodule, and the first data slice taking the computing node included in the second communication module 20 corresponding to the first transition node as a first target node.

[0210] The first data scattering rule includes: each computing node 1 in the second communication module 20 sends the aggregated first data slice to the corresponding computing node in the second communication module 20 as a first target node.

[0211] Specifically, in the first data slice aggregation process, each computing node 1 in the eleventh submodule of the first communication module 10 aggregates the first data slice as the first transition node, the first data slice aggregated by the first transition node is sent by each computing node 1 in the eleventh submodule, and is the first data slice sent to the first target node which is the computing node 1 included in the second communication module 20 corresponding to the first transition node. Each computing node 1 in the twelfth submodule of the first communication module 10 aggregates the first data slice as the first transition node, the first data slice aggregated by the first transition node is sent by each computing node 1 in the twelfth submodule, and is the first data slice sent to the first target node which is the computing node 1 included in the second communication module 20 corresponding to the first transition node. Wherein, the second communication module corresponding to the first transition node refers to the second communication module to which the computing node 1 as the first transition node belongs.

[0212] For example, the process of the first communication module composed of the computing nodes (XPU) of G0 column and G4 column in the 2D mesh topology network aggregating the first data slice is taken as an example to illustrate the aggregation process of the first data slice. Each computing node in the above-mentioned first communication module includes 64 first data slices, which is specifically represented as Figure 14A indicated. The computing nodes of the first row (J0 row), the third row (J2 row), the fifth row (J4 row) and the seventh row (J6 row) of the G0 column and the G4 column of the above-mentioned first communication module constitute the eleventh submodule, as shown in Figure 33 indicated. The computing nodes of the second row (J1 row), the fourth row (J3 row), the sixth row (J5 row) and the eighth row (J7 row) of the G0 column and the G4 column of the above-mentioned first communication module constitute the twelfth submodule, as shown in Figure 34 indicated.

[0213] The computing node J0G0 as the first transition node aggregates the first data slices sent by all computing nodes in the eleventh submodule to all computing nodes in the second communication module to which the computing node J0G0 belongs, all computing nodes of the second communication module to which the computing node J0G0 belongs are the first target nodes, and the computing node J0G0 corresponds to all computing nodes of the second communication module. That is, the computing node numbered 0 aggregates the first data slices numbered 0, 1, 2, 3, 4, 5, 6 and 7 of the first data slices of the computing nodes numbered 0, 8, 16, 24, 32, 40, 48 and 56 in the above eleventh submodule. The computing node J0G4 as the first transition node aggregates the first data slices sent by all computing nodes in the eleventh submodule to all computing nodes in the first communication module to which the computing node J0G4 belongs, all computing nodes of the second communication module to which the computing node J0G4 belongs are the first target nodes, and the computing node J0G4 corresponds to all computing nodes of the second communication module. That is, the computing node numbered 8 aggregates the second data slices numbered 8, 9, 10, 11, 12, 13, 14 and 15 of the first data slices of the computing nodes numbered 0, 8, 16, 24, 32, 40, 48 and 56 in the above eleventh submodule. The aggregation process of the first data slices of other computing nodes in the eleventh submodule is similar to the aggregation process of the first data slices of the computing node J0G0 and the computing node J0G4, which will not be described here.

[0214] Computing node J1G0, acting as the first transition node, aggregates the first data slices sent by all computing nodes in the twelfth submodule to all computing nodes in the second communication module to which computing node J1G0 belongs. All computing nodes in the second communication module to which computing node J1G0 belongs serve as first target nodes, and computing node J1G0 corresponds to all computing nodes in the second communication module. That is, computing node 7 aggregates the first data slices (numbered 0, 1, 2, 3, 4, 5, 6, and 7) of computing nodes numbered 7, 15, 23, 31, 39, 47, 55, and 63 in the twelfth submodule. Computing node J1G4, acting as the first transition node, aggregates the second data slices sent by all computing nodes in the twelfth submodule to all computing nodes in the second communication module to which computing node J1G4 belongs. All computing nodes in the first communication module to which computing node J1G4 belongs serve as first target nodes, and computing node J1G4 corresponds to all computing nodes in the second communication module. That is, the computing node with node number 15 aggregates the first data slices of numbers 8, 9, 10, 11, 12, 13, 14, and 15 of the second data slices of the computing nodes with node numbers 7, 15, 23, 31, 39, 47, 55, and 63 in the twelfth submodule. The process of aggregating the first data slices by other computing nodes in the twelfth submodule is similar to the process of aggregating the first data slices by computing nodes J1G0 and J1G4, and is not further described here.

[0215] For example, Figure 35 As shown, the process of disseminating the first data slice in the second communication module, consisting of the compute nodes (XPUs) in rows J0 and J1 and columns G0 to G3 in a 2D mesh topology network, is used as an example to illustrate the dissemination process of the first data slice. Compute node J0G0 in the second communication module sends the aggregated first data slice to the other compute nodes in the second communication module, based on the first target node that received the first data slice. Each compute node in the second communication module can serve as a first target node. Specifically, compute node 0 sends the aggregated first data slices (numbered 0, 1, 2, 3, 4, 5, 6, and 7) from compute nodes numbered 0, 8, 16, 24, 32, 40, 48, and 56 in the eleventh submodule, respectively, to the corresponding first target nodes, namely, compute nodes numbered 1, 2, 3, 4, 5, 6, and 7 in the second communication module. The dissemination process of the second data slices for the other compute nodes in the second communication module is similar to that for compute node J0G0 and is not further described here.

[0216] On the basis of the above embodiments, further, the second data aggregation rule includes: each computing node in the second communication module serves as a second transition node, aggregating second data slices sent by each computing node in the second communication module, and the second data slices use the computing node included in the eleventh submodule or the twelfth submodule corresponding to the second transition node as the second target node;

[0217] The second data dissemination rule includes: each computing node in the eleventh sub-module sends the aggregated second data slice to the corresponding computing node in the eleventh sub-module as the second target node; each computing node in the twelfth sub-module sends the aggregated second data slice to the corresponding computing node in the twelfth sub-module as the second target node.

[0218] Specifically, during the second data slice aggregation process, each computing node 1 in the second communication module 20, as a second transition node, aggregates the second data slice. The second data slice aggregated by the second transition node is sent by each computing node 1 in the second submodule and is sent to the computing node 1 included in the eleventh submodule or the twelfth submodule corresponding to the second transition node as the second target node. The eleventh submodule corresponding to the second transition node refers to the eleventh submodule to which the computing node 1 serving as the second transition node belongs; the twelfth submodule corresponding to the second transition node refers to the twelfth submodule to which the computing node 1 serving as the second transition node belongs.

[0219] During the second data slice aggregation process, each computing node 1 in the first communication module 10 acts as a second transition node to aggregate the second data slice. During the second data slice dissemination process, each computing node 1 in the eleventh submodule of the first communication module 10 sends the aggregated second data slice to the corresponding second destination node. Each computing node 1 in the eleventh submodule acts as a second destination node and receives the second data slice with each computing node 1 as the data receiving node. Each computing node 1 in the twelfth submodule of the first communication module 10 sends the aggregated second data slice to the corresponding second destination node. Each computing node 1 in the twelfth submodule acts as a second destination node and receives the second data slice with each computing node 1 as the data receiving node. The eleventh and twelfth submodules communicate in parallel.

[0220] For example, Figure 36 As shown, Figure 32 The process of aggregating the second data slices is described by taking the second communication module composed of the computing nodes (XPUs) in rows J0 and J1 and columns G0 to G3 in the 2D mesh topology network shown in FIG. Each computing node in the second communication module includes 64 second data slices, which are specifically represented as follows:Figure 17 The computing node numbered 0 (J0G0) as the second transition node aggregates the second data slices sent by all the computing nodes in the second communication module to all the computing nodes of the eleventh submodule of the first communication module 10 composed of the computing nodes 1 of the G0 column and the G4 column, all the computing nodes of the eleventh submodule as the second target node, and the computing node J0G0 corresponding to all the computing nodes in the eleventh submodule. The computing node J0G0 aggregates the second data slices numbered 0, 8, 16, 24, 32, 40, 48 and 56 of the second data slices sent by all the computing nodes in the second communication module. The computing node numbered 7 (J1G0) as the second transition node aggregates the second data slices sent by all the computing nodes in the second communication module to all the computing nodes of the twelfth submodule of the first communication module 10 composed of the computing nodes 1 of the G0 column and the G4 column, all the computing nodes of the twelfth submodule as the second target node, and the computing node J1G0 corresponding to all the computing nodes in the twelfth submodule. The computing node J1G0 aggregates the second data slices numbered 7, 15, 23, 31, 39, 47, 55 and 63 of the second data slices sent by all the computing nodes in the second communication module. There are 8 computing nodes in the second communication module, numbered 0, 1, 2, 3, 4, 5, 6 and 7. The aggregation of the second data slices of the other computing nodes in the second communication module is similar to the aggregation of the second data slices of the computing node J0G0 or the computing node J1G0, which will not be described here.

[0221] For example, as shown in Figure 37 and Figure 38 shown in the 2D mesh topology network shown in Figure 32 , the process of spreading the second data slices by the first communication module composed of the computing nodes (XPU) of the G0 column and the G4 column in the 2D mesh topology network shown in Figure 37 is taken as an example to illustrate the spreading process of the second data slices. The computing nodes of the first row (J0 row), the third row (J2 row), the fifth row (J4 row) and the seventh row (J6 row) of the G0 column and the G4 column of the first communication module form the eleventh submodule, as shown in Figure 38 .

[0222] The computing node J0G0 in the eleventh submodule sends the aggregated second data slices to other computing nodes in the eleventh submodule according to the second target nodes receiving the second data slices, and each computing node in the eleventh submodule can serve as the second target node. That is, the computing node numbered 0 sends the second data slices numbered 0, 8, 16, 24, 32, 40, 48, and 56 of the second data slices of the computing nodes numbered 0, 1, 2, 3, 4, 5, 6, and 7 in the eleventh submodule to the corresponding second target nodes, that is, the computing nodes numbered 8, 16, 24, 32, 40, 48, and 56 in the eleventh submodule. The spreading process of the second data slices of other computing nodes in the eleventh submodule is similar to that of the computing node J0G0, which is not described here. Finally, each computing node in the eleventh submodule receives the second data slices sent by other computing nodes to the computing node. 0~63 0 represents that the computing node numbered 0 receives 64 second data slices sent to the computing node numbered 0.

[0223] The computing node J1G0 in the twelfth submodule sends the aggregated second data slices to other computing nodes in the twelfth submodule according to the second target nodes receiving the second data slices, and each computing node in the twelfth submodule can serve as the second target node. That is, the computing node numbered 7 sends the second data slices numbered 7, 15, 23, 31, 39, 47, 55, and 63 of the second data slices of the computing nodes numbered 1, 3, 5, 7, 9, 11, 13, and 15 in the twelfth submodule to the corresponding second target nodes, that is, the computing nodes numbered 7, 15, 23, 31, 39, 47, 55, and 63 in the twelfth submodule. The spreading process of the second data slices of other computing nodes in the twelfth submodule is similar to that of the computing node J1G0, which is not described here. Finally, each computing node in the twelfth submodule receives the second data slices sent by other computing nodes to the computing node.

[0224] Further, the computing node adopts a computing card or a processor core on the basis of the above embodiments. The computing card includes but is not limited to a graphics processing unit (GPU), a tensor processing unit (TPU), a recurrent processing unit (RPU), and the like, which is selected according to actual needs, and the embodiments of the present application are not limited.

[0225] Figure 39 is a flowchart of a communication method provided by an embodiment of the present application, as shown in Figure 39 The communication method provided by the embodiment of the present application is applied to the all-to-all communication system described in any of the above embodiments, and includes the following steps.

[0226] S3901, in the first communication stage, each computing node in the first communication module aggregates the first data slice of itself to the corresponding first transition node based on a first data aggregation rule; each computing node in the second communication module sends the first data slice aggregated by each computing node to the corresponding first target node based on a first data dissemination rule; wherein each computing node in the first communication module serves as a first transition node; each first communication module communicates in parallel; each second communication module communicates in parallel.

[0227] Specifically, the all-to-all communication is divided into two communication stages. In the first communication stage, each computing node in the first communication module aggregates the first data slice of itself to the corresponding first transition node based on a first data aggregation rule. Each computing node in the first communication module can serve as a first transition node. Each computing node in the second communication module sends the first data slice aggregated by each computing node to the corresponding first target node based on a first data dissemination rule. Each computing node in the second communication module can serve as a first target node. In the first communication stage, the data sent from one computing node to another computing node is referred to as a first data slice. Each first transition node aggregates the first data slice sent to the corresponding first target node. The correspondence between the first transition node and the corresponding first target node is predetermined. The first target node can have multiple corresponding first target nodes. The first target node is a computing node that receives the first data slice. The communication between each computing node in the first communication module can be implemented based on a Tree manner or a Ring manner. The communication between each computing node in the second communication module can be implemented based on a Tree manner or a Ring manner. The selection is made according to actual needs, which is not limited by the embodiment of the present application. Each first communication module communicates in parallel. Each second communication module communicates in parallel.

[0228] S3902, in the second communication stage, each computing node in the second communication module aggregates the second data slice of itself to the corresponding second transition node based on a second data aggregation rule; each computing node in the first communication module sends the second data slice aggregated by each computing node to the corresponding second target node based on a second data dissemination rule; wherein each computing node in the second communication module serves as a second transition node; each second communication module communicates in parallel; each first communication module communicates in parallel.

[0229] Specifically, in the second communication stage, each computing node in the second communication module aggregates the respective second data slice to the corresponding second transition node based on the second data aggregation rule, and each computing node in the second communication module can serve as the second transition node. Each computing node in the first communication module sends the second data slice aggregated by each computing node to the corresponding second target node based on the second data dissemination rule, and each computing node in the first communication module can serve as the second target node. Wherein, the data sent from one computing node to another computing node in the second communication stage is referred to as the second data slice. The second data slice can be the data obtained by processing the first data slice after the computing node receives the first data slice. Each second transition node aggregates the second data slice sent to the corresponding second target node, and the correspondence between the second transition node and the second target node is predetermined, and the second target node corresponding to the second transition node can be multiple. The second target node is the computing node receiving the second data slice. The communication between each computing node in the first communication module can be implemented based on the Tree mode or the Ring mode, and the communication between each computing node in the second communication module can be implemented based on the Tree mode or the Ring mode, which is selected according to actual needs, and the embodiment of the application is not limited. Each second communication module communicates in parallel. Each first communication module communicates in parallel.

[0230] The communication method provided by the embodiment of the application, in the first communication stage, each computing node in the first communication module aggregates the respective first data slice to the corresponding first transition node based on the first data aggregation rule; each computing node in the second communication module sends the first data slice aggregated by each computing node to the corresponding first target node based on the first data dissemination rule; wherein, each computing node in the first communication module serves as the first transition node; each first communication module communicates in parallel; each second communication module communicates in parallel;

[0231] In the second communication stage, each computing node in the second communication module aggregates the respective second data slice to the corresponding second transition node based on the second data aggregation rule; each computing node in the first communication module sends the second data slice aggregated by each computing node to the corresponding second target node based on the second data dissemination rule; wherein, each computing node in the second communication module serves as the second transition node; each second communication module communicates in parallel; each first communication module communicates in parallel, which can reduce communication delay and congestion, and improve communication efficiency.

[0232] The embodiment of the communication method provided by the embodiment of the application can be specifically used for implementing the communication process of the embodiment of each all-to-all communication system, and details are not described herein again. For details, refer to the detailed description of the embodiment of each all-to-all communication system.

[0233] The computer device provided by the embodiment of the application comprises the all-to-all communication system described in any of the above embodiments.

[0234] In the description of the specification, the description of the terms "one embodiment", "one specific embodiment", "some embodiments", "for example", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0235] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the application. It should be understood that the above description is only for specific embodiments of the application and is not used to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application should be included in the protection scope of the application.

Claims

1. An all-to-all communication system, characterized by The multiple computing nodes are communicatively connected through a 2D mesh topology network, wherein: The multiple computing nodes are divided into multiple first communication modules according to a first division rule and multiple second communication modules according to a second division rule; In the first communication stage, each computing node in each first communication module is configured to aggregate a respective first data slice to a corresponding first transition node based on a first data aggregation rule, and each first communication module performs communication in parallel; each computing node in each second communication module is configured to send a first data slice aggregated by each computing node to a corresponding first target node based on a first data dissemination rule, and each second communication module performs communication in parallel; wherein each computing node in the first communication module serves as a first transition node; In the second communication stage, each computing node in the second communication module is configured to aggregate a respective second data slice to a corresponding second transition node based on a second data aggregation rule, and each second communication module performs communication in parallel; each computing node in the first communication module is configured to send a second data slice aggregated by each computing node to a corresponding second target node based on a second data dissemination rule, and each first communication module performs communication in parallel; wherein each computing node in the second communication module serves as a second transition node.

2. The all-to-all communication system of claim 1, wherein, The first division rule comprises dividing each computing node according to a row of the 2D mesh topology network, and the computing nodes in each row serve as a first communication module; and the second division rule comprises dividing each computing node according to a column of the 2D mesh topology network, and the computing nodes in each column serve as a second communication module. Alternatively, the first division rule comprises dividing each computing node according to a column of the 2D mesh topology network, and the computing nodes in each column serve as a first communication module; and the second division rule comprises dividing each computing node according to a row of the 2D mesh topology network, and the computing nodes in each row serve as a second communication module.

3. The all-to-all communication system of claim 2, wherein, The first data aggregation rule comprises that each computing node in the first communication module serves as a first transition node, and aggregates a first data slice sent by each computing node in the first communication module, wherein the first data slice takes a computing node included in a second communication module corresponding to the first transition node as a first target node; The first data dissemination rule comprises that each computing node in the second communication module sends an aggregated first data slice to a corresponding computing node in the second communication module serving as a first target node.

4. The all-to-all communication system of claim 2, wherein, The second data aggregation rule comprises that each computing node in the second communication module serves as a second transition node, and aggregates a first data slice sent by each computing node in the second communication module, wherein the first data slice takes a computing node included in a first communication module corresponding to the second transition node as a second target node; The second data dissemination rule comprises that each computing node in the first communication module sends an aggregated second data slice to a corresponding computing node in the first communication module.

5. The all-to-all communication system of claim 1, wherein, The first division rule comprises: dividing each computing node according to rows of the 2D mesh topology network, and taking every two rows of computing nodes as a first communication module, and taking computing nodes with odd node numbers as a first sub-module and computing nodes with even node numbers as a second sub-module according to the parity of the node numbers of each computing node in the first communication module; wherein the node numbers of each computing node of the 2D mesh topology network are obtained in advance. The second division rule comprises: dividing each computing node according to columns of the 2D mesh topology network, and taking every two columns of computing nodes as a second communication module, and taking computing nodes of odd rows as a third sub-module and computing nodes of even rows as a fourth sub-module according to the parity of the rows to which each computing node in the second communication module belongs in the 2D mesh topology network.

6. The all-to-all communication system of claim 5, wherein, The first data aggregation rule comprises: taking each computing node in the first sub-module as a first transition node to aggregate the first data slice sent by each computing node in the first sub-module, and taking the computing node included in the third sub-module or the fourth sub-module corresponding to the first transition node as a first target node; and taking each computing node in the second sub-module as a first transition node to aggregate the first data slice sent by each computing node in the second sub-module, and taking the computing node included in the third sub-module or the fourth sub-module corresponding to the first transition node as a first target node. The first data scattering rule comprises: taking each computing node in the third sub-module to send the aggregated first data slice to the corresponding computing node as a first target node in the third sub-module; and taking each computing node in the fourth sub-module to send the aggregated first data slice to the corresponding computing node as a first target node in the fourth sub-module.

7. The all-to-all communication system of claim 5, wherein, The second data aggregation rule comprises: taking each computing node in the third sub-module as a second transition node to aggregate the second data slice sent by each computing node in the third sub-module, and taking the computing node included in the first sub-module or the second sub-module corresponding to the second transition node as a second target node; and taking each computing node in the fourth sub-module as a second transition node to aggregate the second data slice sent by each computing node in the fourth sub-module, and taking the computing node included in the first sub-module or the second sub-module corresponding to the second transition node as a second target node. The second data scattering rule comprises: taking each computing node in the first sub-module to send the aggregated second data slice to the corresponding computing node as a second target node in the first sub-module; and taking each computing node in the second sub-module to send the aggregated second data slice to the corresponding computing node as a second target node in the second sub-module.

8. The all-to-all communication system of claim 1, wherein, The first division rule comprises: dividing each computing node according to the column of the 2D mesh topology network, and taking every 2 columns of computing nodes as a first communication module, and taking the computing nodes in the odd rows as a fifth sub-module and the computing nodes in the even rows as a sixth sub-module according to the parity of the rows to which the computing nodes in the first communication module belong in the 2D mesh topology network. The second division rule comprises: dividing each computing node according to the row of the 2D mesh topology network, and taking every 2 rows of computing nodes as a second communication module, and taking the computing nodes with odd node numbers as a seventh sub-module and the computing nodes with even node numbers as an eighth sub-module according to the parity of the node numbers of the computing nodes in the second communication module; wherein the node numbers of each computing node of the 2D mesh topology network are obtained in advance.

9. The all-to-all communication system of claim 8, wherein, The first data aggregation rule comprises: taking each computing node in the fifth sub-module as a first transition node to aggregate the first data slices sent by each computing node in the fifth sub-module, and taking the computing nodes included in the seventh sub-module or the eighth sub-module corresponding to the first transition node as a first target node; and taking each computing node in the sixth sub-module as a first transition node to aggregate the first data slices sent by each computing node in the sixth sub-module, and taking the computing nodes included in the seventh sub-module or the eighth sub-module corresponding to the first transition node as a first target node. The first data aggregation rule comprises: taking each computing node in the fifth sub-module as a first transition node to aggregate the first data slices sent by each computing node in the fifth sub-module, and taking the computing nodes included in the seventh sub-module or the eighth sub-module corresponding to the first transition node as a first target node; and taking each computing node in the sixth sub-module as a first transition node to aggregate the first data slices sent by each computing node in the sixth sub-module, and taking the computing nodes included in the seventh sub-module or the eighth sub-module corresponding to the first transition node as a first target node.

10. The all-to-all communication system of claim 8, wherein, The second data aggregation rule comprises: taking each computing node in the seventh sub-module as a second transition node to aggregate the second data slices sent by each computing node in the seventh sub-module, and taking the computing nodes included in the fifth sub-module or the sixth sub-module corresponding to the second transition node as a second target node; and taking each computing node in the eighth sub-module as a second transition node to aggregate the second data slices sent by each computing node in the eighth sub-module, and taking the computing nodes included in the seventh sub-module or the eighth sub-module corresponding to the second transition node as a second target node. The second data aggregation rule comprises: taking each computing node in the seventh sub-module as a second transition node to aggregate the second data slices sent by each computing node in the seventh sub-module, and taking the computing nodes included in the fifth sub-module or the sixth sub-module corresponding to the second transition node as a second target node; and taking each computing node in the eighth sub-module as a second transition node to aggregate the second data slices sent by each computing node in the eighth sub-module, and taking the computing nodes included in the seventh sub-module or the eighth sub-module corresponding to the second transition node as a second target node.

11. The all-to-all communication system of claim 1, wherein, The first division rule comprises: dividing the computing nodes in the 2D mesh topology network into a plurality of first communication modules according to 2 rows and n columns, each first communication module comprising 2×n computing nodes, and the total number of columns of the 2D mesh topology network being 2n. The second division rule comprises: dividing the computing nodes in the 2D mesh topology network by columns, the computing nodes in the ith column and the (i+n)th column as a second communication module, and according to the parity of the rows to which the computing nodes in the second communication module belong in the 2D mesh topology network, the computing nodes in the odd rows as a ninth submodule and the computing nodes in the even rows as a tenth submodule, i being greater than or equal to 1 and less than or equal to n.

12. The all-to-all communication system of claim 11, wherein, The first data aggregation rule comprises: each computing node in the first communication module as a first transition node, and aggregating the first data slice sent by each computing node in the first communication module, the first data slice taking the computing nodes included in the ninth submodule or the tenth submodule corresponding to the first transition node as a first target node. The first data scattering rule comprises: each computing node in the ninth submodule sending the aggregated first data slice to the corresponding computing node in the ninth submodule as the first target node; and each computing node in the tenth submodule sending the aggregated first data slice to the corresponding computing node in the tenth submodule as the first target node.

13. The all-to-all communication system of claim 11, wherein, The second data aggregation rule comprises: each computing node in the ninth submodule as a second transition node, and aggregating the second data slice sent by each computing node in the ninth submodule, the second data slice taking the computing nodes included in the first communication module corresponding to the second transition node as a second target node; and each computing node in the tenth submodule as a second transition node, and aggregating the second data slice sent by each computing node in the tenth submodule, the second data slice taking the computing nodes included in the first communication module corresponding to the second transition node as a second target node. The second data scattering rule comprises: each computing node in the first communication module sending the aggregated second data slice to the corresponding computing node in the first communication module as the second target node.

14. The all-to-all communication system of claim 1, wherein, The first division rule comprises: dividing the computing nodes in the 2D mesh topology network by columns, the computing nodes in the ith column and the (i+n)th column as a first communication module, and according to the parity of the rows to which the computing nodes in the first communication module belong in the 2D mesh topology network, the computing nodes in the odd rows as an eleventh submodule and the computing nodes in the even rows as a twelfth submodule, i being greater than or equal to 1 and less than or equal to n, and the total number of columns of the 2D mesh topology network being 2n. The second division rule comprises: dividing the computing nodes in the 2D mesh topology network into a plurality of second communication modules according to 2 rows and n columns, each second communication module comprising 2x n computing nodes.

15. The all-to-all communication system of claim 14, wherein, The first data aggregation rule comprises: each computing node in the eleventh sub-module as a first transition node, aggregating the first data slice sent by each computing node in the eleventh sub-module, the first data slice taking the computing node included in the second communication module corresponding to the first transition node as a first target node; each computing node in the twelfth sub-module as a first transition node, aggregating the first data slice sent by each computing node in the twelfth sub-module, the first data slice taking the computing node included in the second communication module corresponding to the first transition node as a first target node; The first data scattering rule comprises: each computing node in the second communication module sending the aggregated first data slice to the corresponding computing node in the second communication module as a first target node.

16. The all-to-all communication system of claim 14, wherein, The second data aggregation rule comprises: each computing node in the second communication module as a second transition node, aggregating the second data slice sent by each computing node in the second communication module, the second data slice taking the computing node included in the eleventh sub-module or the twelfth sub-module corresponding to the second transition node as a second target node; The second data scattering rule comprises: each computing node in the eleventh sub-module sending the aggregated second data slice to the corresponding computing node in the eleventh sub-module as a second target node; each computing node in the twelfth sub-module sending the aggregated second data slice to the corresponding computing node in the twelfth sub-module as a second target node.

17. The all-to-all communication system according to any one of claims 1 to 16, characterized by, The computing node adopts a computing card or a processor core.

18. A method of communication, comprising: The all-to-all communication system is applied to any one of claims 1 to 17, comprising: In the first communication stage, each computing node in the first communication module aggregates the respective first data slice to the corresponding first transition node based on the first data aggregation rule; each computing node in the second communication module sends the first data slice aggregated by each computing node to the corresponding first target node based on the first data scattering rule; wherein each computing node in the first communication module is a first transition node; each first communication module communicates in parallel; each second communication module communicates in parallel; In the second communication stage, each computing node in the second communication module aggregates the respective second data slice to the corresponding second transition node based on the second data aggregation rule; each computing node in the first communication module sends the second data slice aggregated by each computing node to the corresponding second target node based on the second data scattering rule; wherein each computing node in the second communication module is a second transition node; each second communication module communicates in parallel; each first communication module communicates in parallel.

19. A computer device, comprising: The all-to-all communication system comprises any one of claims 1 to 17.

Images