Computing system and computing method for multi-processing unit interconnect

Abstract

The present disclosure provides a computing system and a computing method for multi-processing unit interconnection, the computing system comprising: one or more parallel processing unit groups, a plurality of parallel processing units in the parallel processing unit group are organized into subsets of parallel processing units, in the same subset, each parallel processing unit is configurable coupled to two nearest neighbor parallel processing units through two communication links, and each parallel processing unit is configurable coupled to the farthest neighbor parallel processing unit through one communication link, and each parallel processing unit is configurable coupled to the corresponding parallel processing unit in the other subset through two communication links. The present disclosure processes functions such as Reduce function and All_Reduce function of inter-node communication mode by chip set, thereby improving the bandwidth utilization and efficiency of inter-chip communication.

Description

Technical Field

[0001] This disclosure relates to the field of information processing and communication in chip interconnect networks, and more particularly to computing systems and computing methods for interconnecting multiple processing units. Background Technology

[0002] Currently, methods for distributed parallel training of deep neural networks include deploying synchronously executed large-scale minibatch stochastic gradient descent (SDG) processing across multiple distributed computing nodes to explore data parallelism-based acceleration. (Reference) Figure 1 , Figure 1 An exemplary mini-batch stochastic gradient descent process running on a CPU host is shown, along with pseudocode. The mini-batch stochastic gradient descent process is constrained by the synchronization component, which creates a bottleneck in the overall parallel acceleration process. For example... Figure 2 As shown, in order to reduce bottlenecks, it is necessary to increase the bandwidth of the accelerator-side network and / or reduce the frequency of communication between the host and the accelerator.

[0003] Mini-batch stochastic gradient descent has many synchronization algorithms. Some common functions that implement inter-node communication are the `Reduce` and `All_Reduce` functions. Now refer to... Figure 3 , Figure 3 This illustrates the Reduce function. In the Reduce function, groups of values ​​from each of multiple nodes 310-340 are passed to a given node 310, which then sums the corresponding values. The given node 310 stores the sum of the value groups. For example, if node 310 receives the values ​​5, 2, 7, and 4 from multiple nodes 310-340, it will sum these values ​​together and store the sum as 18. Node 310 will also sum the values ​​1, 3, 8, and 2 together and store the sum as 14. Now refer to... Figure 4 , Figure 4This illustrates the All_Reduce function. In the All_Reduce function, a set of values ​​from each of the multiple nodes 410-440 is passed to a given node 410, which adds the corresponding values ​​together. The sum is then broadcast from the given node 410 to the multiple nodes 410-440, and the multiple nodes 410-440 store the sum. For example, node 410 adds the values ​​5, 2, 7, and 4 received from the multiple nodes 410-440. Node 410 also adds the values ​​1, 3, 8, and 2 together. Node 410 broadcasts the sums 18 and 14 to the multiple nodes 410-440, and each of the multiple nodes 410-440 stores the sums 18 and 14. As shown above, both the Reduce and All_Reduce functions are applied to a batch of variables simultaneously.

[0004] Although the simple topological implementations of the Reduce and All_Reduce functions are tree-based, ring-based implementations can achieve higher bandwidth utilization and efficiency. (See now for reference.) Figure 5 , Figure 5 This illustrates a traditional ring-based All_Reduce function implemented in a distributed computing system. In the All_Reduce function, each of the N nodes in the distributed computing system communicates with its two peer nodes 2*(N-1) times. During communication, nodes send and receive sets of values. In the first (N-1) iterations, the received value is added to the value in the corresponding node's buffer. In the second (N-1) iterations, the value stored in the corresponding node's buffer is replaced with the received value. For example, Figure 5Example 510 shows three nodes (N=3), each buffering its own set of input values. In the first iteration 520, the first node passes the first set of input values ​​to the second node. The second node adds the set of input values ​​received from the first node to the corresponding set of input values ​​held by the second node. The first node also receives a third set of input values ​​from the third node. The first node adds the set of input values ​​received from the third node to the corresponding set of input values ​​held by the first node. In the first iteration 520, the second and third nodes also pass and add their respective sets of input values. In the second iteration 530, the first node passes the third set of input values ​​to the second node, and the second node adds the third set of input values ​​to the corresponding set of input values ​​held by the second node. The first node also receives a second set of values ​​from the third node, and the first node adds the second set of values ​​to the corresponding set of input values ​​held by the first node. In the second iteration 530, the second and third nodes again pass and add their respective sets of values. In the third iteration 540, the first node passes the second set of sums to the second node, and the second node stores the second set of sums. The first node also receives the first sum value group from the third node and stores it. The second and third nodes also pass and store their respective sum value groups. In the fourth iteration (550), the first node passes the first sum value group to the second node, which stores it. The first node also receives the third sum value group from the third node and stores it. The second and third nodes also pass and store their respective sum value groups. After the fourth iteration, each node has a sum value group. If the buffer is large enough, then... Figure 5 The ring-based All_Reduce function shown can optimally utilize the available network of a distributed computing system.

[0005] However, improvements are needed in high-speed chip-to-chip serial / deserialization (SerDes) interconnects so that such distributed systems used to compute the All_Reduce function can be implemented within a chipset (cluster), rather than on distributed computers connected via slower Ethernet, infinite bandwidth, or similar communication links. Summary of the Invention

[0006] In view of this, embodiments of the present disclosure provide a computing system and computing method for interconnecting multiple processing units.

[0007] In one embodiment, the computing system may include one or more groups of parallel processing units (PPUs). Multiple parallel processing units within a group of PPUs may be organized into subsets of parallel processing units. Within the same subset, each parallel processing unit may be configurably coupled to its two nearest neighbor parallel processing units via two communication links, and each parallel processing unit may be configurably coupled to its farthest neighbor parallel processing unit via one communication link. Furthermore, each parallel processing unit may be configurably coupled to corresponding parallel processing units in other subsets via two communication links.

[0008] In another embodiment, the computation method may include configuring the communication links of a group of parallel processing units into one or more computation groups comprising corresponding multiple communication rings based on specified computation parameters. The one or more computation groups may use a parallel communication ring algorithm to compute a function on the input data. The function may be, but is not limited to, a Reduce function or an All_Reduce function.

[0009] The above scheme enables the computation of communication modes between computing nodes, such as Reduce and All_Reduce functions, within the chip cluster, thereby improving the bandwidth utilization and efficiency of inter-chip communication.

[0010] To introduce the selection of concepts in a simplified form, a summary section is provided, which are further described in the detailed embodiments section below. This summary section is neither intended to identify key or essential features of the claimed subject matter nor to limit the scope of the claimed subject matter. Attached Figure Description

[0011] Embodiments of this disclosure are shown in the accompanying drawings by way of example rather than by way of limitation, wherein the same reference numerals denote similar elements, wherein:

[0012] Figure 1 An exemplary mini-batch stochastic gradient descent process according to conventional techniques is shown;

[0013] Figure 2 This illustrates another exemplary mini-batch stochastic gradient descent process based on conventional techniques;

[0014] Figure 3 The calculation process of the Reduce function according to conventional techniques is shown;

[0015] Figure 4 The computation process of the All_Reduce function according to conventional techniques is shown;

[0016] Figure 5The computation process of the ring-based All_Reduce algorithm based on conventional techniques is shown;

[0017] Figure 6 Multiple parallel processing units providing hierarchical scaling according to embodiments of the present disclosure are shown;

[0018] Figure 7 A hierarchical scaling configuration of multiple parallel processing units according to embodiments of the present disclosure is illustrated;

[0019] Figure 8 A hierarchical scaling method for multiple parallel processing units according to embodiments of the present disclosure is illustrated;

[0020] Figure 9A A hierarchical scaling configuration of multiple parallel processing units according to embodiments of the present disclosure is illustrated;

[0021] Figure 9B A hierarchical scaling configuration of multiple parallel processing units according to embodiments of the present disclosure is illustrated;

[0022] Figure 9C A hierarchical scaling configuration of multiple parallel processing units according to embodiments of the present disclosure is illustrated;

[0023] Figure 10 A hierarchical scaling configuration of multiple parallel processing units according to embodiments of the present disclosure is illustrated;

[0024] Figure 11 A hierarchical scaling configuration of multiple parallel processing units according to embodiments of the present disclosure is illustrated;

[0025] Figure 12 An example computing system including multiple parallel processing units according to an embodiment of the present disclosure is shown;

[0026] Figure 13 An example parallel processing unit according to an embodiment of this disclosure is shown. Specific Implementation

[0027] This disclosure is best understood by referring to the following description and accompanying drawings, which illustrate embodiments of the accelerator system and its configuration techniques for multiprocessor unit interconnects disclosed herein. Embodiments of this disclosure will now be described in detail, with examples shown in the following drawings. Although this disclosure will be described in conjunction with these embodiments, it will be understood that they are not intended to limit the disclosure to these embodiments. Rather, this disclosure is intended to cover alternatives, modifications, and equivalents that may be included within the scope of this disclosure as defined in the appended claims. Furthermore, numerous specific details are set forth in the following detailed description of this disclosure in order to provide a thorough understanding of the disclosure. However, it should be understood that this disclosure may be practiced without these specific details. In other instances, well-known methods, processes, components, and circuits have not been described in detail in order to avoid unnecessarily obscuring various aspects of this disclosure.

[0028] The following embodiments of this disclosure are presented in the form of routines, modules, logic blocks, and other notations representing data operations in one or more electronic devices. These descriptions and representations are means used by those skilled in the art to most effectively convey the substance of their work to others skilled in the art. In this disclosure, routines, modules, logic blocks, and / or the like are generally considered as a self-consistent sequence of steps or instructions that lead to a desired result. These steps are those that involve physical operations on physical quantities. Typically, although not essential, these physical operations take the form of electrical or magnetic signals that can be stored, transmitted, compared, and otherwise manipulated in an electronic device. For convenience and with reference to common usage, according to embodiments of this disclosure, these signals are referred to as data, bits, values, elements, symbols, characters, terms, numbers, strings, and / or the like.

[0029] However, it should be remembered that these terms should be interpreted as referring to physical operations and physical quantities, and are merely convenient labels, and should be further interpreted according to the terminology commonly used in the art. Unless explicitly indicated in the description below, it will be understood that throughout the description of this disclosure, the use of terms such as receiving and / or the like refers to the actions and processes of operating and converting data in electronic devices, such as electronic computing devices. In the logic circuits, registers, memories, and / or the like of electronic devices, data is represented as physical quantities (e.g., electronic physical quantities), and in electronic devices, data is converted into other data similarly represented as physical quantities.

[0030] In this application, the separate description of each feature also implies that these features can be combined with each other. The use of definite or indefinite articles is not intended to indicate cardinality. In particular, references to "the" object or "an" object are intended to indicate that it may be one of a plurality of such objects. Words such as "comprising," "including," and "having" are used to specify the presence of the declared element, but not to exclude the presence or addition of one or more other elements and / or combinations thereof. It should also be understood that although words such as "first," "second," etc., may be used herein to describe various elements, these elements should not be limited by these words. These words are used in this disclosure to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without exceeding the scope of embodiments of this disclosure. It should also be understood that when an element is described as "coupled" to another element, the element may be directly or indirectly connected to the other element, or there may be an intermediate element between the element and the other element. In contrast, when an element is described as "directly connected" to another element, there is no intermediate element between them. It should also be understood that the term "and / or" includes any and all combinations of one or more associated elements. It should also be understood that the phrases and terms used in this disclosure are for descriptive purposes and should not be construed as limiting the disclosure.

[0031] Now for reference Figure 6 , Figure 6 A plurality of parallel processing units (PPUs) with hierarchical scaling provided according to embodiments of the present disclosure are illustrated. The plurality of parallel processing units may include one or more groups of eight parallel processing units each. Each parallel processing unit may include seven communication ports. The eight parallel processing units in a group of parallel processing units may be organized into a first subset and a second subset, the first subset comprising four parallel processing units and the second subset comprising four parallel processing units. Within the same subset, each parallel processing unit may be configurably coupled to two nearest neighbor parallel processing units via two communication links. Within the same subset, each parallel processing unit may also be configurably coupled to the farthest neighbor parallel processing unit via one communication link. Each parallel processing unit may also be configurably coupled to corresponding parallel processing units in other subsets via two communication links. In one embodiment, parallel processing units may be coupled via configurable bidirectional communication links. Figure 7As shown, the configurably coupled communication links can be configured as three communication rings 710-730, which couple eight parallel processing units together. For example, the first bidirectional ring, shown by the dashed line 710, can communicatively link the first parallel processing unit 305 to the fourth parallel processing unit 320, the fourth parallel processing unit 320 to the seventh parallel processing unit 330, the seventh parallel processing unit 330 to the third parallel processing unit 315, the third parallel processing unit 315 to the eighth parallel processing unit 325, the eighth parallel processing unit 325 to the fifth parallel processing unit 340, the fifth parallel processing unit 340 to the second parallel processing unit 310, the second parallel processing unit 310 to the sixth parallel processing unit 335, and the sixth parallel processing unit 335 back to the first parallel processing unit 305. In addition to the three communication rings 710-730, there are also some communication links 740, represented by solid lines. It is understood that communication rings 710-730 are merely exemplary communication ring groups of three communication rings, configured based on multiple communication links between the nearest neighbor parallel processing units, one bidirectional communication link between the farthest neighbor parallel processing units, and two bidirectional communication links between the corresponding parallel processing units in the two subsets.

[0032] Reference Figure 8 Further explanation of the hierarchical scaling of parallel processing units. In step 810, multiple communication links of a parallel processing unit group can be configured into one or more computing groups based on specified computation parameters. A parallel processing unit group includes eight parallel processing units, and a computing group includes corresponding multiple communication rings. In one embodiment, the computation parameter is the number of parallel processing units in a given computing group, for example, 8, 4, or 2 parallel processing units in a given computing group. In another embodiment, the computation parameter is a quantity of computational processing bandwidth. The computational processing bandwidth is mapped to a given number of parallel processing units. In one embodiment, such as... Figure 7 As shown, eight parallel processing units can be configured as a computational group of eight parallel processing units communicatively coupled through three bidirectional communication rings. In other cases, the application may not require a computational group of eight parallel processing units to compute the Reduce function, All_Reduce function, or other similar functions. In other cases, such as cloud computing services, customers may want the option to choose whether to pay for the computational processing bandwidth of eight, four, or two parallel processing units.

[0033] Therefore, in another embodiment, such as Figure 9AAs shown, the eight parallel processing units can be configured into two computing groups 905 and 910. Computing group 905 includes four parallel processing units (i.e., parallel processing units 305-320), and computing group 910 includes four parallel processing units (i.e., parallel processing units 325-340). Communication links can be configured by enabling communication links for a given combination and disabling other communication links, such that the parallel processing units in each computing group 905 and 910 are communicatively coupled via two bidirectional communication rings 915-920 and 925-930. For example, the first ring 915 and the second ring 920 in the bidirectional rings can couple the first parallel processing unit 305 to the fourth parallel processing unit 320, the fourth parallel processing unit 320 to the third parallel processing unit 315, the third parallel processing unit 315 to the second parallel processing unit 310, and the second parallel processing unit 310 to the first parallel processing unit 305. Similarly, the third ring 925 and the fourth ring 930 in the bidirectional ring can couple the fifth parallel processing unit 340 to the sixth parallel processing unit 335, the sixth parallel processing unit 335 to the seventh parallel processing unit 330, the seventh parallel processing unit 330 to the eighth parallel processing unit 325, and the eighth parallel processing unit 325 to the fifth parallel processing unit 340. Other communication links 935 can be disabled or used for other purposes. Through the two bidirectional communication rings, each group of four parallel processing units can be configured to compute different Reduce functions, All_Reduce functions, or similar functions. Figure 9A The exemplary configuration shown is just one possible configuration for dividing eight parallel processing units into two computing groups consisting of four parallel processing units each. Figure 9B and 9C Other possible configurations for dividing eight parallel processing units into two computation groups consisting of four parallel processing units are shown.

[0034] In yet another embodiment, such as Figure 10As shown, the eight parallel processing units can be configured into four computing groups 1005, 1010, 1015, and 1020. Computing group 1005 includes two parallel processing units (i.e., parallel processing units 305-310), computing group 1010 includes two parallel processing units (i.e., parallel processing units 315-320), computing group 1015 includes two parallel processing units (i.e., parallel processing units 325-330), and computing group 1020 includes two parallel processing units (i.e., parallel processing units 335-340). The parallel processing units in each computing group 1005, 1010, 1015, and 1020 can be coupled via their respective bidirectional communication rings. For example, a first parallel processing unit 305 can be coupled to a second parallel processing unit 310 via a first bidirectional communication link and a second bidirectional communication link. Other communication links can be disabled or used for other purposes. Each computation group of 1050, 1010, 1015, and 1020, consisting of two parallel processing units, can be configured to compute different Reduce functions, All_Reduce functions, or similar functions. Similarly, Figure 10 The exemplary configuration shown is only one possible configuration for dividing eight parallel processing units into four computing groups consisting of two parallel processing units each.

[0035] In yet another embodiment, such as Figure 11 As shown, the eight parallel processing units can be configured as a combination of a computation group 1105 consisting of four parallel processing units and two computation groups 1110 and 1115 consisting of two parallel processing units each. Computation group 1105 includes parallel processing units 305-320, computation group 1110 includes parallel processing units 325-330, and computation group 1115 includes parallel processing units 335-340. Similarly, each computation group can be configured to compute different Reduce functions, All_Reduce functions, or similar functions. Furthermore, Figure 11 The exemplary configuration shown is only one possible configuration for dividing eight parallel processing units into one computing group consisting of four parallel processing units and two computing groups consisting of two parallel processing units.

[0036] Refer again Figure 8In step 820, the input data can be partitioned for computation on a given computation group and loaded onto the respective parallel processing units of the given computation group. For a computation group consisting of eight parallel processing units coupled through three bidirectional communication rings, the input data for the Reduce function, All_Reduce function, or similar functions can be divided into six data groups: three data groups are used to propagate along a first direction on the three parallel rings of the bidirectional communication link, and the other three data groups are used to propagate along a second direction on the three parallel rings of the bidirectional communication link. For a computation group consisting of four parallel processing units coupled through two bidirectional communication rings, the input data for the Reduce function, All_Reduce function, or similar functions can be divided into four data groups: two data groups are used to propagate along a first direction on the two parallel rings of the bidirectional communication link, and the other two data groups are used to propagate along a second direction on the two parallel rings of the bidirectional communication link. For a computation group consisting of two parallel processing units coupled through two bidirectional communication links, the input data for the Reduce function, All_Reduce function, or similar functions can be divided into two data groups: one data group is used to propagate along the first direction on the two bidirectional communication links, and the other data group is used to propagate along the second direction on the two bidirectional communication links.

[0037] In step 830, a given computation group may compute a Reduce function, All_Reduce function, or similar function on the input data using a parallel ring Reduce, All_Reduce, or similar parallel ring algorithm. In a parallel ring algorithm, each of the multiple parallel processing units (e.g., N nodes) communicates with its two nearest neighbor parallel processing units 2*(N-1) times, exchanging corresponding data sets on the corresponding ring in the corresponding direction. In the first (N-1) iterations, the given parallel processing unit sends the corresponding value on the corresponding ring to its nearest neighbor parallel processing unit. In the first (N-1) iterations, the given parallel processing unit also receives the corresponding value on the corresponding ring from its nearest neighbor parallel processing unit and adds the received value to the corresponding value in the given parallel processing unit's buffer. In the second (N-1) iterations, the given parallel processing unit sends the corresponding value on the corresponding ring to its nearest neighbor parallel processing unit. In the second (N-1)th iteration, the given parallel processing unit also receives the corresponding value on the corresponding ring from its nearest neighbor parallel processing unit and replaces the corresponding value in the buffer of the given parallel processing unit with the received corresponding value.

[0038] Now for reference Figure 12 , Figure 12An example computing system comprising multiple parallel processing units according to various aspects of this disclosure is illustrated. The example computer system 1200 may include multiple parallel processing units 1210-1220 coupled together via one or more high-bandwidth inter-chip networks 1230. The multiple parallel processing units 1210-1220 may be, but are not limited to, multiple neural processing accelerators. The parallel processing units 1210-1220 may also be coupled to multiple host processing units 1240-1250 via one or more communication buses 1260 and 1270. The one or more communication buses 1260 and 1270 may be, but are not limited to, one or more peripheral component interface express (PCIe) buses. The one or more host processing units 1240-1250 may be coupled to one or more host-side networks 1280 via one or more network interface cards (NICs) 1290-1295.

[0039] Now for reference Figure 13 , Figure 13 An example parallel processing unit (PPU) according to various aspects of this disclosure is shown. The parallel processing unit 1300 may include multiple computing cores 1305-1310, multiple inter-chip links (ICLs) 1315-1320, one or more high-bandwidth memory interfaces (HBM I / Fs) 1325-1330, one or more communication processors 1335, one or more direct memory access (DMA) controllers 1340-1345, one or more command processors (CPs) 1350, one or more network-on-chip (NoC) devices 1355, shared memory 1360, and one or more high-bandwidth memory (HBM) devices 1365-1370. The parallel processing unit 1300 may also include one or more joint test action group (JTAG) engines 1375, one or more inter-integrated circuit (I / O) controllers, and communication processors 1335-1340-1345. 2C) Interfaces and / or Serial Peripheral Interface (SPI) 1380, one or more Peripheral Component Interface Express (PCIe) interfaces 1385, one or more codecs (CoDec) 1390, etc. In one embodiment, multiple computing cores 1305-1310, multiple inter-chip links (ICL) 1315-1320, one or more high-bandwidth memory interfaces (HBM I / F) 1325-1330, one or more communication processors 1335, one or more direct memory access (DMA) controllers 1340-1345, one or more command processors (CP) 1350, one or more on-chip networks (NoC) 1355, shared memory 1360, one or more high-bandwidth memory (HBM) 1365-1370, one or more Joint Test Action Group (JTAG) engines 1375, and one or more inter-integrated circuit (I) interfaces can be fabricated within a single integrated circuit (IC). 2 C) Interfaces and / or Serial Peripheral Interface (SPI) 1380, one or more Peripheral Component Interfaces High Speed ​​(PCIe) 1385, one or more Codecs (CoDec) 1390, etc.

[0040] Inter-chip links (ICLs) 1315-1320 can be configured for chip-to-chip communication between multiple parallel processing units. In one embodiment, parallel processing unit 1300 may include seven inter-chip links (ICLs) 1315-1320. Communication processor 1335 and direct memory access engines 1340-1345 can be configured to coordinate data sent and received via the inter-chip links (ICLs) 1315-1320. On-chip network (NoC) 1355 can be configured to coordinate data movement between compute cores 1305-1310 and shared memory 1360. Communication processor 1335, direct memory access engines 1340-1345, on-chip network 1355, and high-bandwidth memory interface (HBM I / F) 1325-1330 can be configured to coordinate data movement between high-bandwidth memory 1365-1370, shared memory 1360, and inter-chip links (ICLs) 1315-1320. Command processor 1350 can be configured as an interface between parallel processing unit 1300 and one or more main processing units. Multiple parallel processing units 1300 can be advantageously used for computation as described above (reference provided). Figure 7 , 8 Reduce functions in 9A-9C, 10 and 11, All_Reduce function, or other similar functions.

[0041] According to various aspects of this disclosure, tiering enables multiple parallel processing units to be configured as one or more compute clusters coupled by corresponding multiple parallel communication rings. Tiered scaling of multiple parallel processing units is advantageous when the computational resources required by an application are less than the portion that a compute cluster can serve as a subset of the multiple parallel processing units. Similarly, tiered scaling can be advantageously employed in cloud computing platforms to easily enable clients to purchase the computational bandwidth of one compute cluster of multiple parallel processing units, rather than purchasing the entire computational bandwidth of those multiple parallel processing units.

[0042] For purposes of illustration and description, the foregoing description has presented specific embodiments of the present disclosure. The foregoing description is not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed, and it will be apparent that many modifications and variations can be made in accordance with the foregoing teachings. These embodiments were chosen and described in order to best explain the principles of the present disclosure and its practical application, thereby enabling others skilled in the art to best utilize the present disclosure and its various embodiments with various modifications suitable for the particular intended use. The scope of the present disclosure is defined by the appended claims and their equivalents.

Claims

1. A computing system, comprising: One or more parallel processing unit groups, wherein each parallel processing unit group comprises eight parallel processing units, wherein the eight parallel processing units in the parallel processing unit group are organized into two subsets, each subset comprising four parallel processing units, wherein in the same subset each parallel processing unit is configurably coupled to two nearest neighbor parallel processing units via two communication links, and each parallel processing unit is configurably coupled to the farthest neighbor parallel processing unit via one communication link, and each parallel processing unit is configurably coupled to corresponding parallel processing units in other subsets via two communication links; The eight parallel processing units in the parallel processing unit group are configured according to one of the following methods: It is configured as a computing group, with the eight parallel processing units of the parallel processing unit group coupled together through three communication rings; It is configured into two computing groups, each of which includes four parallel processing units, and the four parallel processing units of each computing group are coupled together through two communication rings; It is configured into four computing groups, each computing group comprising two parallel processing units, and the two parallel processing units of each computing group are coupled together via a communication ring; It is configured as a first computing group and two second computing groups. The first computing group includes four parallel processing units, and the second computing group includes two parallel processing units. The four parallel processing units of the first computing group are coupled together through two communication rings, and the two parallel processing units of each second computing group are coupled together through their respective communication rings. The communication link includes a bidirectional communication link.

2. The computing system according to claim 1, wherein, The communication links of a parallel processing unit group are configured into one or more computing groups based on computing parameters, and the computing groups include corresponding multiple communication rings.

3. The computing system according to claim 2, wherein, Each of the one or more computation groups is configured to compute the corresponding Reduce or All_Reduce function on the corresponding input data using a parallel ring Reduce or All_Reduce algorithm.

4. The computing system according to claim 2, wherein, The calculation parameters include the number of parallel processing units in the calculation group and the amount of calculation bandwidth.

5. A calculation method, comprising: Based on the computational parameters, the communication links of the parallel processing unit group are configured into one or more computational groups, and the computational group includes multiple corresponding communication rings; as well as The computation group uses a parallel communication loop algorithm to compute functions on the input data; The parallel processing unit group includes eight parallel processing units, which are organized into two subsets, each subset including four parallel processing units. The parallel processing unit group also includes, in each subset, two bidirectional communication links between each parallel processing unit and its nearest neighbor set, one bidirectional communication link between each parallel processing unit and its farthest neighbor set, and two bidirectional communication links between the corresponding parallel processing units in the two subsets. Configuring the communication links of the parallel processing unit groups into one or more computing groups includes: The bidirectional communication link is configured as three parallel communication rings, which couple the eight parallel processing units into a computing group. The calculation method further includes: The input data is divided into six data groups, and the corresponding data group pairs are loaded into the corresponding parallel processing units; or Configuring the communication links of the parallel processing unit groups into one or more computing groups includes: The bidirectional communication link is configured as a parallel communication ring group, which includes two parallel communication rings. The parallel communication ring group couples the corresponding subsets into one or more corresponding computing groups. The calculation method further includes: The input data is divided into four data groups, and the corresponding data group pairs are loaded into the corresponding parallel processing units of the computing group consisting of four parallel processing units.

6. The calculation method according to claim 5, wherein, The calculation parameters include the number of parallel processing units in the calculation group and the amount of calculation bandwidth.

Images