A GPU network vertical scaling system oriented towards model parallelism strategy
By adopting a fully interconnected approach between optical switching modules and electrical switches in the GPU cluster, the parallel transmission path of the model is optimized, which solves the problems of multiple jumps and extended communication time in the parallel transmission of the model in the GPU cluster, and improves the efficiency and flexibility of global reduction transmission.
Patent Information
- Application Number
- CN202611360059.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-03
- Publication Date
- 2026-11-10
AI Technical Summary
In GPU clusters, model parallel transmission involves numerous jumps, prolonged communication time, and low global reduction transmission efficiency, making it difficult to achieve efficient model parallel strategy expansion.
The GPU network vertical scaling system adopts a model-parallel strategy, which utilizes the first optical switching module to fully interconnect with the electrical switch to replace the multi-layer electrical switch, and combines the second optical switching module to provide optical direct connection path and reconfigurable characteristics to optimize the communication path.
It reduces the number of jumps and average communication latency in parallel model transmission, improves the efficiency and flexibility of global protocol transmission, and meets the requirements of high bandwidth utilization.
Smart Images

Figure CN123075005A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of integrated circuit design technology, and in particular to a GPU network vertical scaling system oriented towards a model-parallel strategy. Background Technology
[0002] In recent years, with the rapid growth of new datasets and model parameters, how to achieve linear scaling of computing power in GPU-accelerated intelligent computing clusters through high-performance networks and efficient communication has become a critical issue that the industry urgently needs to address. To fully utilize hardware resources and improve the training efficiency of large models, GPU clusters typically combine model parallelism (including tensor parallelism, pipeline parallelism, and expert parallelism) with data parallelism to form a hybrid parallelism strategy. In the tensor parallelism, pipeline parallelism, and expert parallelism dimensions, the GPU network constructs a high-bandwidth computing domain through vertical scaling (scale-up) and forms a supernode with tight coupling between computation and communication, thereby overcoming the limitation that a single GPU chip's memory cannot accommodate all model parameters.
[0003] Scale-Up interconnects primarily handle model-parallel traffic, requiring support not only for AllReduce transmission but also for traffic patterns such as global switching, one-to-many broadcasting, and many-to-one aggregation. During each iteration of large model training, AllReduce traffic within the Scale-Up domain significantly exceeds model-parallel traffic, while the connectivity of model-parallel transmission is higher than that of AllReduce transmission. On one hand, as the growth rate of model parameters exceeds the growth rate of GPU memory capacity, the scale of GPU accelerator interconnects in the Scale-Up domain is increasing from hundreds to thousands, thus providing better support for strong scaling based on model-parallel strategies (i.e., reducing the overall training iteration time by increasing the number of GPUs within a fixed batch size). In this scenario, model-parallel traffic, characterized by a large number of GPUs and low load, needs to hop across multiple Scale-Up interconnects, which is detrimental to reducing average communication latency and improving training performance. On the other hand, AllReduce traffic, which accounts for the majority of the data transmitted within the Scale-Up domain, has low-entropy communication characteristics of small quantity and large load. Its traffic pattern is variable when the parallel strategy remains unchanged, which will exacerbate hash collisions in multi-layer electrical switching networks and thus reduce link utilization.
[0004] Therefore, how to reduce the number of jumps and average communication latency in model parallel transmission, and improve the efficiency and flexibility of global reduction transmission caused by model parallelism, has become an urgent problem to be solved. Summary of the Invention
[0005] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows:
[0006] A GPU network vertical scaling system for model parallelism strategy, the system comprising: a first optical switching module, M GPU computing modular units and a second optical switching module, wherein the first optical switching module comprises K first optical switching units, the second optical switching module comprises 4 second optical switching units, and a single GPU computing modular unit comprises K GPU chips and N electrical switches;
[0007] Each electrical switch is connected to K first optical switching units;
[0008] Each GPU chip is connected to four second optical switch units;
[0009] For a single GPU computing modular unit, each electrical switch contained in the GPU computing modular unit is connected to K GPU chips contained in the GPU computing modular unit.
[0010] Compared with existing technologies, this invention has significant advantages. Through the above technical solution, the GPU network vertical scaling system for model parallelism strategies provided by this invention achieves considerable technological advancement and practicality, and has broad industrial application value. It has at least the following advantages:
[0011] In this invention, based on a single-layer vertically extended electrical switch structure, for model parallel transmission in a static network structure, a single-layer first optical switching module replaces the second-layer and above electrical switches in the prior art, fully interconnecting each first optical switching unit with each electrical switch. This avoids complex routing mechanisms and load balancing designs, shortens the number of hops and average communication latency in model parallel transmission, and simultaneously uses a second optical switching module to provide a direct optical connection for any pair of GPU chips within the vertically extended domain, meeting the high bandwidth utilization requirements of global protocol transmission. Furthermore, the reconfigurable characteristics of the second optical switching module can be used to change the global protocol transmission order within the vertically extended domain to adapt to the variability of global protocol traffic patterns, thereby improving the efficiency and flexibility of global protocol transmission caused by model parallelism. Attached Figure Description
[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0013] Figure 1This is a schematic diagram of the structure of a GPU network vertical scaling system based on a model parallelism strategy provided in Embodiment 1 of the present invention;
[0014] Figure 2 This is a schematic diagram of the first ring global reduction order in a GPU network vertical scaling system oriented towards model parallelism, provided in Embodiment 1 of the present invention.
[0015] Figure 3 This is a schematic diagram of the second ring global reduction order in a GPU network vertical scaling system oriented towards model parallelism, provided in Embodiment 1 of the present invention.
[0016] Figure 4 This is a schematic diagram of the global reduction order of the first bibinary tree in a GPU network vertical scaling system oriented towards model parallelism, provided in Embodiment 1 of the present invention.
[0017] Figure 5 This is a schematic diagram of the global reduction order of the second bibinary tree in a GPU network vertical scaling system oriented towards model parallelism strategy provided in Embodiment 1 of the present invention.
[0018] Figure 6 This is a schematic diagram of a GPU network vertical scaling system based on topology reconstruction, provided in Embodiment 2 of the present invention. Detailed Implementation
[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0020] This embodiment provides a GPU network vertical scaling system for a model parallel strategy. The system includes: a first optical switching module, M GPU computing modular units and a second optical switching module. The first optical switching module includes K first optical switching units, the second optical switching module includes 4 second optical switching units, and a single GPU computing modular unit includes K GPU chips and N electrical switches.
[0021] Each electrical switch is connected to K first optical switching units;
[0022] Each GPU chip is connected to four second optical switch units;
[0023] For a single GPU computing modular unit, each electrical switch contained in the GPU computing modular unit is connected to K GPU chips contained in the GPU computing modular unit.
[0024] The first optical switching module contains K first optical switching units, each of which is connected to all electrical switches (M×N in total) in all GPU computing modular units. The first optical switching module is used to carry communication traffic under the model parallel strategy.
[0025] The second optical switching module contains four second optical switching units. Each GPU chip is connected to all four second optical switching units to carry the traffic of the All-Reduce global protocol set communication.
[0026] The GPU computing modular unit contains K GPU chips and N electrical switches. The GPU computing modular unit is a computing unit that can be deployed and expanded independently.
[0027] It should be noted that the number of second optical switching units in this embodiment is 4, which is intended to support different global protocol set communication algorithms at the same time. The number of second optical switching units is actually the maximum number of connections of a single GPU chip in each global protocol set communication algorithm. Implementers should know that the number of second optical switching units used under a specific global protocol set communication algorithm is still within the protection scope of this invention. For example, when the global protocol set communication algorithm adopts the ring global protocol set communication algorithm, the number of second optical switching units can be set to 2.
[0028] In one specific implementation, the GPU chip includes at least N+4 GPU ports, wherein N GPU ports are used to connect to N electrical switches in the GPU computing modular unit, and 4 GPU ports are used to connect to 4 second optical switching units.
[0029] Among them, N GPU ports are allocated to carry the parallel transmission traffic of the model and are connected to N electrical switches respectively. The remaining 4 GPU ports are allocated to carry the global protocol transmission traffic and are connected to 4 second optical switching units for topology reconstruction respectively.
[0030] In one specific implementation, the electrical switch includes at least 2K electrical switch ports, wherein K electrical switch ports are used to connect K GPU chips in the GPU computing modular unit, and K electrical switch ports are used to connect K first optical switching units.
[0031] In this configuration, K electrical switch ports of a single electrical switch are connected as uplink ports to K first optical switching units used for static networks, and K electrical switch ports are connected as downlink ports to K GPU chips.
[0032] In one specific implementation, the first optical switching unit includes at least MN×MN first optical switching ports.
[0033] The system contains M×N electrical switches, each occupying one first optical switching port, for a total of M×N first optical switching ports. To achieve full connectivity between the electrical switches, each first optical switching port needs to establish an optical path connection with all other first optical switching ports, which requires MN×MN first optical switching ports.
[0034] Specifically, MN×MN should be understood as the port matrix size, that is, the first optical switching unit needs to support MN input ports and MN output ports (a total of 2MN physical ports, or MN×MN cross-connection capability) in order to realize arbitrary optical path connection between MN optical switches.
[0035] Through K first optical switching units, all M×N electrical switches can achieve arbitrary pairwise connections at the optical layer, providing sufficient optical path resources for model parallel communication.
[0036] See Figure 1 This is a schematic diagram of the structure of a GPU network vertical scaling system based on a model parallel strategy, provided in Embodiment 1 of the present invention.
[0037] In one specific implementation, the second optical switching unit includes at least MK×MK second optical switching ports.
[0038] The system contains M×K GPU chips, each occupying one second optical switching port, for a total of M×K second optical switching ports. To achieve full connectivity between all GPU chips, each second optical switching port needs to establish an optical path connection with all other second optical switching ports, which requires MK×MK first optical switching ports.
[0039] Similarly, MK×MK should be understood as the port matrix scale, that is, the second optical switching unit needs to support MK input ports and MK output ports (a total of 2MK physical ports, or MK×MK cross-connection capability) in order to realize arbitrary optical path connection between MK GPU chips.
[0040] In one specific implementation, the first optical switching module is used to support static connections between MN electrical switches to carry model-parallel traffic; correspondingly, the first optical switching module does not support link reconfiguration.
[0041] Static connection refers to the first optical switching module completing the optical path configuration once during system deployment, without changing any optical connection relationships during the training task.
[0042] Link reconfiguration refers to dynamically changing the optical path connection in the optical switching module. The first optical switching module is statically connected and does not need to support link reconfiguration capability.
[0043] Model parallel traffic refers to the communication traffic generated by model parallel strategies such as tensor parallelism and expert parallelism. For example, model parallel traffic can include activation value transfer in forward propagation and gradient transfer in back propagation.
[0044] In one specific implementation, the second optical switching module is used to support direct connections between MK GPU chips to carry the traffic of global protocol set communication. Accordingly, the second optical switching module supports link reconfiguration.
[0045] AllReduce is a set communication operation that reduces (e.g., sums, averages) the data (such as gradients) on all GPU chips and broadcasts the result to all GPU chips. AllReduce is the core operation for gradient synchronization in data parallelism and model parallelism.
[0046] In one specific implementation, the global reduction set communication includes ring global reduction set communication and bibinary tree global reduction set communication.
[0047] In this context, the ring-based global reduction set communication refers to organizing GPU chips into a logical ring. Each GPU chip sends data to the next GPU chip in the ring while simultaneously receiving data from the previous GPU chip. Global reduction is achieved through multiple rounds of Scatter-Reduce and All-Gather operations.
[0048] Bi-binary tree global reduction set communication refers to organizing the GPU into two complementary binary trees, one for reduction and the other for broadcasting. The two phases are executed in parallel, reducing the number of communication rounds and lowering latency.
[0049] See Figure 2 This diagram illustrates the first ring-shaped global reduction order in a GPU network vertical scaling system based on a model-parallel strategy, as provided in Embodiment 1 of the present invention. The diagram uses a vertical scaling domain of 16 GPU chips with K=8 and M=2 as an example. It can be seen that each GPU chip... Figure 2 The ring connection is shown. Since there are four second optical switching units in this embodiment, each GPU chip is connected to other GPU chips through four ports. Therefore, for a single GPU chip, it is connected to the previous GPU chip through two ports and to the next GPU chip through two ports.
[0050] See Figure 3This diagram illustrates the second ring-shaped global reduction order in a GPU network vertical scaling system based on a model-parallel strategy, as provided in Embodiment 1 of the present invention. The diagram also uses a vertical scaling domain of 16 GPU chips with K=8 and M=2 as an example. It can be seen that each GPU chip... Figure 3 The ring connection shown is used, although Figure 2 and Figure 3 Although the ring connection methods differ, they can all be achieved through link reconfiguration of the second optical switching module to support different ring global protocol order requirements.
[0051] See Figure 4 This diagram illustrates the global reduction order of the first bibinary tree in a GPU network vertical scaling system oriented towards model parallelism, as provided in Embodiment 1 of the present invention. The diagram also uses the vertical scaling domain of 16 GPU chips with K=8 and M=2 as an example. It can be seen that each GPU chip... Figure 4 The method shown uses a dual binary tree connection. For a single GPU chip, a maximum of 4 ports are needed to connect to other GPU chips. For example, for GPU3,1, GPU1,1, GPU5,1 and GPU7,1 need to be connected in the binary tree on the left side of the diagram, and GPU2,2 needs to be connected in the binary tree on the right side of the diagram.
[0052] See Figure 5 This diagram illustrates the global reduction order of the second bibinary tree in a GPU network vertical scaling system oriented towards model parallelism, as provided in Embodiment 1 of the present invention. The diagram also uses the vertical scaling domain of 16 GPU chips with K=8 and M=2 as an example. It can be seen that each GPU chip... Figure 5 The method shown demonstrates the connection of a binary tree, although Figure 4 and Figure 5 The binary tree connection methods are different, but they can all be achieved through link reconstruction of the second optical switching module to support different binary tree global reduction order requirements.
[0053] In this first embodiment, based on the single-layer vertically extended electrical switch structure, for model parallel transmission in the static network structure, the first optical switching module of the single layer replaces the electrical switches of the second layer and above in the prior art. Each first optical switching unit is fully interconnected with each electrical switch, thereby avoiding complex routing mechanisms and load balancing designs, shortening the number of hops and average communication latency in model parallel transmission. At the same time, the second optical switching module provides an optical direct connection path for any pair of GPU chips in the vertically extended domain, meeting the high bandwidth utilization requirements of global protocol transmission. Furthermore, the reconfigurable characteristics of the second optical switching module can be used to change the order of global protocol transmission in the vertically extended domain to adapt to the variability of global protocol traffic patterns, thereby improving the efficiency and flexibility of global protocol transmission caused by model parallelism.
[0054] This embodiment 2 provides a GPU network vertical scaling system based on topology reconstruction. The system includes: a first optical switching module, M GPU computing modular units, and a second optical switching module. The first optical switching module includes K first optical switching units, the second optical switching module includes a second optical switching submodule and a third optical switching submodule. The second optical switching submodule includes 4 second optical switching units, and the third optical switching submodule includes 4 third optical switching units. A single GPU computing modular unit includes K GPU chips and N electrical switches.
[0055] Each electrical switch is connected to K first optical switching units;
[0056] Each GPU chip is connected to the input ports of four high-speed optical switches;
[0057] The first output port of each high-speed optical switch is connected to four second optical switch units, and the second output port of each high-speed optical switch is connected to four third optical switch units.
[0058] For a single GPU computing modular unit, each electrical switch contained in the GPU computing modular unit is connected to K GPU chips contained in the GPU computing modular unit.
[0059] The GPU computing modular unit contains K GPU chips and N electrical switches, and is a computing unit that can be deployed and expanded independently.
[0060] The first optical switching module contains K first optical switching units, each of which is connected to all electrical switches (M×N in total) in all GPU computing modular units. The first optical switching module is used to carry communication traffic under the model parallel strategy.
[0061] The second optical switching module includes a second optical switching submodule (4 second optical switching units) and a third optical switching submodule (4 third optical switching units), which are used to carry the traffic of global protocol set communication.
[0062] A high-speed optical switch is a fast optical path switching device with one input port and at least two output ports, which can switch the input optical signal to different output channels at high speed. In this embodiment, the high-speed optical switch is a 1×2 high-speed optical switch.
[0063] Specifically, a single GPU chip includes at least N+4 GPU ports, of which N GPU ports are used to connect to N electrical switches in the GPU computing modular unit, and 4 GPU ports are used to connect to 4 high-speed optical switches. The 4 high-speed optical switches determine whether the GPU chip is connected to the second optical switching submodule or the third optical switching submodule.
[0064] Most current network technologies typically require multiple channels at the physical layer for their port speeds. For example, under the 100Gb / s EPS serializer / deserializer standard, 400Gb / s, 800Gb / s, and 1.6Tb / s Ethernet require 4, 8, and 16 channels, respectively; under the 200Gb / s EPS serializer / deserializer standard, 1.6Tb / s and 3.2Tb / s Ethernet require 8 and 16 channels, respectively. Therefore, to ensure that all physical channels in the same logical link are multiplexed and transmitted on the same optical fiber and at the same optical switching port, the calculation of the required number of optical switching ports in this invention is based on the following premises: the ports of the OCS or optical distribution frame must use bidirectional technology, and single-mode or multi-mode optical transceivers must use wavelength division multiplexing (WDM) technology.
[0065] See Figure 6 This is a schematic diagram of a GPU network vertical expansion system based on topology reconstruction provided in Embodiment 2 of the present invention.
[0066] In one specific implementation, the first optical switching unit is implemented using an optical circuit switch.
[0067] Among them, the Optical Circuit Switch (OCS) is a network device that directly switches and reconstructs optical signals in the optical domain.
[0068] A single optical switching unit for implementing the first optical switching unit includes at least MN×MN optical switching ports.
[0069] In one specific implementation, the first optical switching module is implemented using a first optical distribution frame, which is divided into K first optical distribution subframes. Correspondingly, the kth first optical switching unit is implemented using the kth first optical distribution subframe, where k is an integer in the range [1, K].
[0070] Among them, an optical distribution frame refers to a passive optical distribution device used for connecting, distributing and scheduling optical fiber lines. An optical distribution subframe refers to a logical or physical grouping unit inside an optical distribution frame. Each subframe corresponds to an independent optical switching domain and can be configured with an optical switching matrix independently. Functionally, it is equivalent to an independent optical switching unit.
[0071] In one specific implementation, the first optical distribution frame includes at least KMN×KMN optical switching ports.
[0072] Wherein, KMN×KMN refers to the port matrix size of the first optical distribution frame. The first optical distribution frame needs to support KMN input ports and KMN output ports to realize arbitrary optical path connections between KMN electrical switches.
[0073] In one specific implementation, the second optical switching unit and the third optical switching unit are implemented using optical circuit switches.
[0074] Each optical circuit switch used to implement the second or third optical switching unit includes at least MK×MK optical switching ports.
[0075] In one specific implementation, the second optical switching module is implemented using a second optical distribution frame, which is divided into four second optical distribution subframes and four third optical distribution subframes. Correspondingly, the i-th second optical switching unit is implemented using the i-th second optical distribution subframe, and the i-th third optical switching unit is implemented using the i-th third optical distribution subframe, where i is an integer in the range [1,4].
[0076] The second optical distribution frame contains four optical distribution subframes corresponding to the second optical switching submodule, each subframe being equivalent to a second optical switching unit. The third optical distribution frame contains four optical distribution subframes corresponding to the third optical switching submodule, each subframe being equivalent to a third optical switching unit.
[0077] Specifically, since the model parallelism strategy and physical layer topology in large model training scenarios can be obtained offline, the GPU interconnection order participating in the global reduction algorithm and the configuration of the optical switching matrix can be pre-calculated before executing a task, without dynamically switching the optical switching matrix during task execution. Regarding the electro-optical hybrid switching scale-up network architecture proposed in this invention, the scheme based on optical distribution frames and 1×2 high-speed optical switches significantly outperforms the OCS-based scheme in terms of port quantity, insertion loss, single-mode and multi-mode connectivity, and cost per port, and can be considered the preferred option.
[0078] In one specific implementation, the second optical distribution frame includes at least 8MK×8MK optical switching ports.
[0079] Here, 8MK×8MK represents the port matrix size of the second optical distribution frame, meaning the second optical distribution frame needs to support 8MK input ports and 8MK output ports.
[0080] In one specific implementation, for any training task, if the training task uses four second optical switching units to implement the global protocol transmission topology, then the four third optical switching units are configured as the global protocol transmission topology required for the next training task.
[0081] If the training task uses four third optical switching units to implement the global protocol transmission topology, then the four second optical switching units are configured as the global protocol transmission topology required for the next training task.
[0082] When the current task uses one set of optical switching units (such as the second optical switching submodule) to carry global protocol traffic, another set of optical switching units (the third optical switching submodule) can be pre-configured with the optical switching matrix required for the next task. During task switching, only high-speed optical switches need to switch the output ports, without waiting for optical path reconstruction, achieving zero-interruption switching.
[0083] Global reduction transmission topology refers to the optical path connection topology required to realize global reduction set communication. Different global reduction algorithms (ring, binary tree, etc.) correspond to different optical switching matrix configurations.
[0084] In this second embodiment, each GPU chip is connected to the second and third optical switching submodules via the input port of a high-speed optical switch. The two optical switching submodules can serve as backups for each other or work alternately. When the current training task uses the second optical switching submodule to carry global reduction traffic, the third optical switching submodule can pre-configure the optical switching matrix required for the next training task, and vice versa. This "ping-pong" operation realizes seamless switching of global reduction transmission topology, eliminates training interruptions during optical path reconstruction, can adapt to the variability of global reduction traffic mode and improve the execution efficiency of training tasks, thereby improving the efficiency and flexibility of global reduction transmission caused by model parallelism.
[0085] While specific embodiments of the invention have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of the invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the invention. The scope of this invention is defined by the appended claims.
Claims
1. A GPU network vertical scaling system oriented towards model parallelism strategy, characterized in that, The system includes: a first optical switching module, M GPU computing modular units, and a second optical switching module. The first optical switching module includes K first optical switching units, the second optical switching module includes 4 second optical switching units, and a single GPU computing modular unit includes K GPU chips and N electrical switches. Each electrical switch is connected to K first optical switching units; Each GPU chip is connected to four second optical switch units; For a single GPU computing modular unit, each electrical switch contained in the GPU computing modular unit is connected to K GPU chips contained in the GPU computing modular unit.
2. The GPU network vertical scaling system based on model-parallel strategy according to claim 1, characterized in that, The GPU chip includes at least N+4 GPU ports, wherein N GPU ports are used to connect to N electrical switches in the GPU computing modular unit, and 4 GPU ports are used to connect to 4 second optical switching units.
3. The GPU network vertical scaling system based on model-parallel strategy according to claim 1, characterized in that, The electrical switch includes at least 2K electrical switch ports, wherein K electrical switch ports are used to connect to K GPU chips in the GPU computing modular unit, and K electrical switch ports are used to connect to K first optical switching units.
4. The GPU network vertical scaling system for model-parallel strategies according to claim 1, characterized in that, The first optical switching unit includes at least MN×MN first optical switching ports.
5. The GPU network vertical scaling system for model-parallel strategies according to claim 1, characterized in that, The second optical switching unit includes at least MK×MK second optical switching ports.
6. The GPU network vertical scaling system based on model-parallel strategy according to claim 1, characterized in that, The first optical switching module is used to support static connections between MN electrical switches to carry parallel traffic. Accordingly, the first optical switching module does not support link reconfiguration.
7. The GPU network vertical scaling system for model-parallel strategies according to claim 1, characterized in that, The second optical switching module is used to support direct connections between MK GPU chips to carry the traffic of global protocol set communication. Accordingly, the second optical switching module supports link reconfiguration.
8. The GPU network vertical scaling system for model-parallel strategies according to claim 7, characterized in that, The global reduction set communication includes ring global reduction set communication and binary tree global reduction set communication.
