Dual-loop chip structure, set communication method, device, equipment and storage medium

Abstract

Embodiments of the present application provide a double-loop chip structure, a set communication method and device, equipment and a storage medium, and relate to the technical field of chip communication. The double-loop chip structure comprises a plurality of chip groups, each chip group comprising a first chip and a second chip, and a direct communication connection between the corresponding first chip and the second chip; the first chips in different chip groups are sequentially connected in the order of the chip groups to form a first loop; and the second chips in different chip groups are sequentially connected in the order of the chip groups to form a second loop. The first chip and the second chip of each chip group in the plurality of chip groups can realize information interaction between chips in the group through the direct communication connection, without needing to pass through the loop transfer, and the direct link can realize efficient interaction. The first loop and the second loop are independent and parallel to each other, and can reduce the data blocking probability of cross-group communication. The structure of "in-group direct connection + cross-group double-loop parallel" can significantly improve the execution efficiency of set communication.

Description

Technical Field

[0001] This application relates to the field of chip communication technology, and in particular to dual-loop chip structures, integrated communication methods, devices, equipment and storage media. Background Technology

[0002] With the rapid development of large-scale model training techniques, artificial intelligence models are becoming increasingly complex. However, this increased model complexity makes ensemble communication a key bottleneck for improving training efficiency. Ensemble communication overhead, especially the overhead from the AllReduce operation, remains a major performance constraint. Even in environments with high-bandwidth interconnects, ensemble communication can still account for approximately 10% to 50% of the total execution time.

[0003] Related technologies utilize chipplet-based processors to improve the computational efficiency of cascade communication algorithms. Specifically, chipplet-based processors introduce explicit die-to-die direct interconnects between two compute dies within the chip to provide higher bandwidth. However, the overall execution efficiency of cascade communication remains relatively poor. Summary of the Invention

[0004] The main objective of this application is to propose a dual-loop chip structure, a collection communication method, apparatus, device, and storage medium to improve the execution efficiency of collection communication.

[0005] To achieve the above objectives, a first aspect of this application provides a dual-loop chip structure, comprising:

[0006] Multiple chipsets, each chipset including a first chip and a second chip, with a direct communication connection between the corresponding first chip and second chip;

[0007] The first chips within different chipsets are connected sequentially according to the order of the chipsets to form a first loop;

[0008] The second chips within different chipsets are connected sequentially according to the order of the chipsets to form a second loop.

[0009] To achieve the above objectives, a second aspect of this application provides a collection communication method, comprising:

[0010] Obtain a preset number of initial data components contained in all chips in the dual-loop chip structure. The initial data components include corresponding component identifiers. The component identifiers are grouped into target identifier groups in pairs. The dual-loop chip structure is the dual-loop chip structure described in the first aspect.

[0011] The number of first iterations is determined based on the number of chips, and the target identifier group corresponding to the chip group is determined for each iteration. During continuous iterations, the target identifier group in the same chip group is different for each iteration, and the chip group corresponding to the same target identifier group is updated along the order of the chip groups.

[0012] Based on the first iteration number, the iteration is performed. In each iteration, for two chips in the same chipset, the two initial data components corresponding to the target identifier group are alternately transmitted based on the corresponding direct communication connection, and / or, between the chipsets, the two initial data components corresponding to the target identifier group are transmitted based on the first loop and the second loop, respectively.

[0013] Until the iteration ends, the initial data components corresponding to the same component identifier are aggregated to obtain the preset number of initial aggregated data, and each chip obtains one initial aggregated data.

[0014] In one embodiment, the step of alternately transmitting two initial data components corresponding to the target identifier group to two chips in the same chipset based on the corresponding direct communication connection includes:

[0015] The two component identifiers in the target identifier group are assigned to the first chip and the second chip respectively in sequence;

[0016] The initial data component in the first chip that matches the component identifier of the second chip, and the initial data component in the second chip that matches the component identifier of the first chip, are exchanged and transmitted through the direct communication connection.

[0017] In one embodiment, transmitting the two initial data components corresponding to the target identifier group between the chipsets based on the first loop and the second loop, respectively, includes:

[0018] In the iterations following the second iteration, each of the chipsets is selected as the current chipset, and the chipset corresponding to the target identifier group in the previous iteration is taken as the target chipset in the current iteration. The correspondence between the two chips in the target chipset and the component identifiers in the target identifier group is determined.

[0019] The target data component sequence is defined as at least one initial data component corresponding to the two component identifiers of the target identifier group in the previous iteration in the two chips of the current chip group.

[0020] Based on the correspondence, the target data component sequence is transmitted to the corresponding chip of the target chipset through the first loop or the second loop.

[0021] In one embodiment, the method further includes:

[0022] The number of the second iteration is determined based on the number of chips;

[0023] Based on the second iteration number, the iteration is performed. In each iteration, for two chips in the same chipset, the initial aggregated data missing from one chip relative to the other chip is determined as the first direct transmission aggregated data. The first direct transmission aggregated data is transmitted alternately based on the corresponding direct communication connection. In two adjacent chipsets, based on the direction of the chipset, the initial aggregated data missing from the latter chip relative to the former chip is determined as the inter-group aggregated data. The inter-group aggregated data is transmitted based on the directions of the first loop and the second loop, respectively.

[0024] After the iteration is completed, for two chips in the same chipset, the missing initial aggregated data of each chip relative to the other chip is determined as the second direct-transmission aggregated data. The first direct-transmission aggregated data is transmitted alternately based on the corresponding direct communication connection until each chip contains all the initial aggregated data.

[0025] To achieve the above objectives, a third aspect of this application proposes a collective communication method, comprising:

[0026] The source chip is determined from all the chips included in the dual-loop chip structure. The initial block data of the source chip is divided equally by a preset number to obtain a first data group and a second data group. The number of chips is not less than the preset number. The dual-loop chip structure is the dual-loop chip structure described in the first aspect.

[0027] The second data group is transmitted to another chip corresponding to the source chip via a corresponding direct communication connection;

[0028] The first data group and the second data group are broadcast in the first loop, the second loop, and each of the direct communication connections until each of the chips contains all of the initial block data.

[0029] In one embodiment, the broadcast transmission of the first data group and the second data group in the first loop, the second loop, and each of the direct communication connections until each of the chips contains all of the initial block data includes:

[0030] The third iteration number is determined based on the number of chips and the preset number;

[0031] Based on the third iteration number, in each iteration, for two chips in the same chipset, the missing initial block data of the chip relative to the other chip is determined as the third direct transmission aggregate data, and the third direct transmission aggregate data is transmitted alternately based on the corresponding direct communication connection. In two adjacent chipsets, based on the direction of the chipset, the missing initial block data of the latter chip relative to the former chip is determined as the inter-group block data, and the inter-group block data is transmitted based on the directions of the first loop and the second loop, respectively.

[0032] After the iteration is completed, for two chips in the same chipset, the missing initial aggregated data of each chip relative to the other chip is determined as the fourth direct-transmission aggregated data. The fourth direct-transmission aggregated data is transmitted alternately based on the corresponding direct communication connection until each chip contains all the initial block data.

[0033] To achieve the above objectives, a fourth aspect of this application provides a collection communication device, comprising:

[0034] Data acquisition module: used to acquire a preset number of initial data components contained in all chips in the dual-loop chip structure, wherein the initial data components include component identifiers, the component identifiers are grouped into target identifier groups in pairs, and the dual-loop chip structure is the dual-loop chip structure described in the first aspect;

[0035] Communication preparation module: used to determine the first iteration number based on the number of chips, and to determine the target identifier group corresponding to the chipset in each iteration. In the continuous iteration process, the target identifier group in the same chipset is different in each iteration, and the chipset corresponding to the same target identifier group is updated along the order of the chipsets.

[0036] Data transmission module: used to perform iteration based on the first iteration number, and in each iteration, for two chips in the same chipset, to alternately transmit the two initial data components corresponding to the target identifier group based on the corresponding direct communication connection, and / or, between the chipsets, to transmit the two initial data components corresponding to the target identifier group based on the first loop and the second loop respectively;

[0037] Aggregation module: used to aggregate the initial data components corresponding to the same component identifier until the iteration ends, to obtain the preset number of initial aggregated data, and each chip obtains one initial aggregated data.

[0038] To achieve the above objectives, a fifth aspect of this application provides a collective communication device, comprising:

[0039] Initial data allocation module: used to determine the source chip from all the chips included in the dual-loop chip structure, and to evenly divide the preset number of initial block data in the source chip to obtain a first data group and a second data group. The number of chips is not less than the preset number. The dual-loop chip structure is the dual-loop chip structure described in the first aspect.

[0040] Intra-group transmission module: used to transmit the second data group to another chip corresponding to the source chip through a corresponding direct communication connection;

[0041] Broadcast transmission module: used to broadcast the first data group and the second data group in the first loop, the second loop and each of the direct communication connections until each of the chips contains all the initial block data.

[0042] To achieve the above objectives, a sixth aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the second or third aspect above.

[0043] To achieve the above objectives, a seventh aspect of the present application provides a storage medium that stores a computer program, which, when executed by a processor, implements the method described in the second or third aspect above.

[0044] The dual-loop chip structure, aggregated communication method, apparatus, device, and storage medium proposed in this application embodiment include: a dual-loop chip structure comprising multiple chipsets, each chipset including a first chip and a second chip, with a direct communication connection between the corresponding first and second chips; first chips within different chipsets connected sequentially according to the chipset order to form a first loop; and second chips within different chipsets connected sequentially according to the chipset order to form a second loop. When the dual-loop chip structure provided in this application embodiment is applied to aggregated communication scenarios where data needs to be broadcast, aggregated, or synchronized among multiple chips, the first and second chips in each of the multiple chipsets can achieve information exchange between chips within the group through direct communication connections, eliminating the need for loop relays and enabling efficient interaction via direct links, reducing latency and path loss in intra-group communication. Simultaneously, the first and second loops formed by the first and second chips of different chipsets are independent and parallel, reducing the probability of data blocking in cross-group communication and making the communication path more flexible. This "direct connection within the group + parallel dual-loop across groups" structure can shorten the communication path and improve data transmission efficiency, thereby significantly improving the execution efficiency of aggregated communication. Attached Figure Description

[0045] Figure 1 This is a connection diagram of a chiplet-packaged processor based on related technologies.

[0046] Figure 2 This is a schematic diagram of the dual-loop chip structure provided in the embodiments of this application.

[0047] Figure 3 This is a flowchart of the collection communication method provided in the embodiments of this application.

[0048] Figure 4 This is a schematic diagram of the data flow of the Reduce Scatter function provided in the embodiments of this application.

[0049] Figure 5 This is a schematic diagram of the target identifier group provided in the embodiments of this application.

[0050] Figure 6 This is a flowchart illustrating how two chips in the same chipset alternately transmit two initial data components corresponding to a target identifier group based on their corresponding direct communication connections, as provided in an embodiment of this application.

[0051] Figure 7 yes Figure 4 A detailed diagram illustrating data transmission.

[0052] Figure 8 This is a flowchart provided in this application embodiment of the process of transmitting two initial data components corresponding to a target identifier group between chipsets based on a first loop and a second loop, respectively.

[0053] Figure 9 This is a flowchart of the All Gather function provided in the embodiments of this application.

[0054] Figure 10 This is a schematic diagram illustrating the principle of the All Gather function provided in the embodiments of this application.

[0055] Figure 11 This is another optional flowchart of the collection communication method provided in the embodiments of this application.

[0056] Figures 12a to 12c This is a schematic diagram illustrating the principle of the Broadcast function provided in the embodiments of this application.

[0057] Figure 13 This is a flowchart provided in an embodiment of the present application, showing the broadcast transmission of the first data group and the second data group in the first loop, the second loop, and each direct communication connection until each chip contains all the initial block data.

[0058] Figure 14 This is a structural block diagram of a collection communication device provided in another embodiment of this application.

[0059] Figure 15 This is yet another structural block diagram of a collection communication device provided in another embodiment of this application.

[0060] Figure 16 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0062] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0064] In the field of parallel computing, especially in high-performance computing (HPC) and large-scale distributed training, Collective Communication (CC) is a key communication mode. It refers to global communication operations involving all processes within a group of processes (such as a set of chips or CPU cores). Unlike point-to-point communication between two processes, collective communication involves the coordination and data exchange of a group of processes.

[0065] To simplify parallel programming and efficiently utilize underlying hardware, these common and repetitive global communication patterns are abstracted into a set of standard interfaces, known as Collective Communication Primitives (CCPs). These primitives are the foundational modules for building parallel algorithms, implemented by specialized communication libraries (such as MPI, NCCL, and CCL), and optimized for specific hardware and network topologies to achieve high bandwidth and low latency.

[0066] Here are some of the most common collection communication primitives and their functions:

[0067] The Reduce Scatter function first performs a reduction operation (such as summation) on the data of all processes, then divides the reduced result into blocks and distributes them to all processes. Each process receives only a portion of the final result.

[0068] The All Gather function: Each process in the communication group sends its own data to all other processes. After the operation is complete, each process has a complete dataset composed of the original data from all processes in the group.

[0069] The Broadcast function allows a designated "root" process to send (broadcast) its data to all other processes in a process group. After the operation is complete, all processes in the group have the same copy of the root process's original data.

[0070] The All Reduce function collects data from all processes, performs a reduction operation, and then broadcasts the final reduction result back to all processes. After the operation is complete, all processes within the group have the same final result.

[0071] With the rapid development of large-scale model training techniques, artificial intelligence models are becoming increasingly complex. However, this increased model complexity makes ensemble communication a key bottleneck for improving training efficiency. Ensemble communication overhead, especially the overhead from the AllReduce operation, remains a major performance constraint. Even in environments with high-bandwidth interconnects, ensemble communication can still account for approximately 10% to 50% of the total execution time.

[0072] Related technologies utilize chipplet-based processors to improve the computational efficiency of ensemble communication algorithms. Specifically, chipplet-based processors introduce explicit die-to-die direct interconnects between two compute dies within the chip to provide higher bandwidth. (See reference...) Figure 1 , Figure 1 This is a schematic diagram of a chiplet-packaged processor based on related technology. As shown in the diagram, this processor utilizes chiplet packaging technology to package two NPU (Neural Network Processing Unit) chips, NPU0 and NPU1 in the diagram, onto a single substrate. A high-speed communication link (Super Input / Output, SIO) connects the two chips, and both chips also have a channel to exchange data with the system-level package (SLP) via the HCCS link. While this connection method can provide higher bandwidth and improve the communication efficiency of distributed systems to some extent, its potential for aggregated communication is insufficiently developed, resulting in relatively poor overall execution efficiency.

[0073] Based on this, embodiments of this application provide a dual-loop chip structure, a unified communication method, apparatus, device, and storage medium. When the dual-loop chip structure is applied to unified communication scenarios where data needs to be broadcast, aggregated, or synchronized among multiple chips, the first and second chips in each of the multiple chipsets can directly communicate to achieve information exchange between chips within the group. This eliminates the need for loop relays, enabling efficient interaction via direct links and reducing latency and path loss in intra-group communication. Simultaneously, the first and second loops formed by the first and second chips of different chipsets are independent and parallel, reducing the probability of data blocking in cross-group communication and making the communication path more flexible. This "direct intra-group connection + cross-group dual-loop parallel" structure shortens the communication path and improves data transmission efficiency, thereby significantly improving the execution efficiency of unified communication.

[0074] This application provides a dual-loop chip structure, a collection communication method, an apparatus, a device, and a storage medium, which are specifically described through the following embodiments. First, the collection communication method in the embodiments of this application is described.

[0075] The aggregate communication method provided in this application relates to the field of chip communication technology. This aggregate communication method can be applied to a terminal, a server, or a computer program running on either the terminal or the server. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run, such as a client supporting dual-ring aggregate communication, i.e., a program that only needs to be downloaded to a browser environment to run; it can also be a small program that can be embedded into any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin. The terminal communicates with the server through a network. This aggregate communication method can be executed by the terminal or the server, or by the terminal and the server working together.

[0076] In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc. The server can be a standalone server, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; it can also be a service node in a blockchain system, where the service nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). The terminal and server can connect via Bluetooth, Universal Serial Bus (USB), or a network, etc., and this embodiment does not impose any limitations.

[0077] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0078] First, the dual-loop chip structure provided in the embodiments of this application is described.

[0079] Reference Figure 2 , Figure 2 This is a schematic diagram of the dual-loop chip structure provided in the embodiments of this application. Figure 2 The example uses 8 chips, numbered 0-8. These chips form 4 chip groups. For example, chip 0 and chip 1 form chip group 1, chip 2 and chip 3 form chip group 2, chip 4 and chip 5 form chip group 3, and chip 6 and chip 7 form chip group 4.

[0080] Specifically, each chipset includes a first chip and a second chip, and there is a direct communication connection between the first chip and the second chip. (See reference...) Figure 2 Chips with even-numbered designations can be designated as the first chip, and chips with odd-numbered designations can be designated as the second chip. Therefore, in chipset 1, the first chip is chip 0, and the second chip is chip 1. There is a direct communication connection between chip 0 and chip 1, represented by a high-speed communication link SIO. The direct communication connection between the first chip and the second chip is represented by a double-headed dashed line in the diagram.

[0081] Furthermore, the first chips within different chipsets are connected sequentially according to the chipset's order to form a first loop, and the second chips within different chipsets are connected sequentially according to the chipset's order to form a second loop. (See reference...) Figure 2In other words, the four first chips corresponding to chipsets 1 through 4 form the first loop, and the four second chips corresponding to chipsets 1 through 4 form the second loop. The first loop is represented as: Chip 0 -> Chip 2 -> Chip 4 -> Chip 6 -> Chip 0, and the second loop is represented as: Chip 1 -> Chip 3 -> Chip 5 -> Chip 7 -> Chip 1. Since Chip 0 in chipset 1 points to Chip 2 in chipset 2, and Chip 1 points to Chip 3 in chipset 2, it can be considered that chipset 1 points to chipset 2. The order between chipsets is: Chipset 1 -> Chipset 2 -> Chipset 3 -> Chipset 4 -> Chipset 1. Therefore, the directions of the first and second loops can be seen as being obtained by connecting the corresponding chips in the order of the chipsets.

[0082] It is understood that the above chipset order is only for illustration. It can also be the reverse order, such as chipset 4->chipset 3->chipset 2->chipset 1->chipset 4. This embodiment does not limit this and can be determined according to the actual situation.

[0083] Depend on Figure 2 As can be seen from the dual-loop chip structure, the first and second chips in each chipset can directly communicate to achieve information exchange between chips within the group. This eliminates the need for inter-loop relays, enabling efficient interaction and reducing latency and path loss in intra-group communication. Simultaneously, the first and second loops formed by the first and second chips of different chipsets are independent and parallel, reducing the probability of data congestion in cross-group communication and making the communication path more flexible. Specifically, when the dual-loop chip structure is applied to aggregated communication algorithms, the first and second loops handle communication between chipsets, enabling simultaneous data transmission, while the direct communication connection handles data synchronization between the two loops and high-speed communication between the two chips within the same chipset. This dual-loop chip structure achieves higher bandwidth utilization and scalability for distributed training tasks of large-scale AI models.

[0084] The following is combined with Figure 2 The dual-loop chip structure is described, and the data flow processing logic of the aggregate communication method in the embodiment of this application is described in the dual-loop chip structure. Here, the Broadcast function and the All Reduce function are introduced for aggregate communication.

[0085] Figure 3 This is an optional flowchart of the collection communication method provided in the embodiments of this application. Figure 3 The method described herein is the implementation logic for the All Reduce function, and may include, but is not limited to, steps 310 to 340. It is also understood that this embodiment... Figure 3The order of steps 310 to 340 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0086] Step 310: Obtain the preset number of initial data components contained in all chips in the dual-loop chip structure.

[0087] In one embodiment, the All Reduce function is broken down into a Reduce Scatter function and an All Gather function. The Reduce Scatter function performs a reduction operation (such as summation) on the data from all chips, then divides the reduced result into blocks and distributes them to all chips. Ultimately, each chip receives only a portion of the final result. Therefore, the data in each chip before the reduction operation is performed is called the initial data component. Each chip contains a preset number of initial data components. The preset number can be set according to the actual scenario, but it must be less than or equal to the number of chips.

[0088] In one embodiment, the ensemble communication scenario corresponding to distributed deep learning training includes data parallelism or model parallelism.

[0089] In data-parallel scenarios, model parameters are fully copied to each chip, but training data is divided into different batches. Each chip processes a different subset of data and calculates the corresponding local gradients. Assuming the model has multiple network layers, different chips use their own subsets of data to calculate the gradient components of the weights for each network layer. For example, if the model has 8 layers, 8 chips process the corresponding network layers A, B, ..., H. Chip 0 processes the 0th batch of data, calculating the local gradients for each layer as A0, B0, ..., H0. Similarly, chip 1 processes the 1st batch of data, calculating the local gradients for each layer as A1, B1, ..., H1, ..., and chip 7 processes the 0th batch of data, calculating the local gradients for each layer as A7, B7, ..., H7. Here, the local gradients in each chip are used as the initial data components for that chip, and they are distinguished within the same chip by component identifiers, such as A, B, etc. Next, the Reduce Scatter function is used to sum the initial data components corresponding to the same component identifier on different chips. For example, the sum of component A across all chips is: A = A0 + A1 + ... + A7. Then, the aggregated A is distributed to chip 0, B is distributed to chip 1, and so on. Chip 0 obtains the total gradient A of the weights of all batches at layer 0, which is used to update the parameters of layer 0. Chip 1 obtains the total gradient B of the weights of all batches at layer 1, which is used to update the parameters of layer 1, and so on, until the last chip.

[0090] In a model-parallel scenario, assume each component identifier (A, B, ..., H) represents a different parameter block of the model. A could be the weight matrix of layer 0 of the neural network, B could be the weight matrix of layer 1, and so on. Chip 0 stores A0, B0, ..., H0, where A0 is the 0th row of matrix A. Chip 1 stores A1, B1, ..., H1, where A1 is the 1st row of matrix A, and so on. The Reduce Scatter function is used to sum the initial data components corresponding to the same component identifier. Then, the corresponding components are distributed, for example, A is distributed to chip 0, B to chip 1, and so on. Finally, chip 0 obtains the complete A for subsequent calculations or updates, and chip 1 obtains the complete B, etc.

[0091] It is understandable that the above is only an illustration of a collective communication scenario and does not imply any limitation on its details.

[0092] In one embodiment, reference is made to Figure 4 , Figure 4 This is a data flow diagram of the Reduce Scatter function provided in this application embodiment. Here, we take eight chips as an example for illustration. The chips are NPU0, NPU1, NPU2, NPU3, NPU4, NPU5, NPU6, and NPU7. NPU0 and NPU1, NPU2 and NPU3, NPU4 and NPU5, and NPU6 and NPU7 each constitute four chipsets. The order of the chipsets is: Chipset 4 -> Chipset 3 -> Chipset 2 -> Chipset 1 -> Chipset 4. Correspondingly, the first loop can be represented as: NPU6 -> NPU4 -> NPU2 -> NPU0 -> NPU6, and the second loop can be represented as: NPU7 -> NPU5 -> NPU3 -> NPU1 -> NPU7. Each chip contains eight initial data components: A0, B0, C0, ..., H0; NPU0 contains A1, B1, C1, ..., H1; and so on, with NPU7 containing A7, B7, C7, ..., H7.

[0093] In one embodiment, component identifiers are grouped into target identifier groups in pairs according to actual needs or in a random manner. (See also...) Figure 5 , Figure 5 This is a schematic diagram of the target identifier group provided in the embodiments of this application. Figure 5 The component identifiers of AH are divided into four groups: AB, CD, EF, and GH, resulting in four target identifier groups. These target identifier groups are then assigned to different chipsets.

[0094] Step 320: Determine the number of the first iteration based on the number of chips, and determine the target identifier group corresponding to the chipset in each iteration.

[0095] In one embodiment, assuming the number of chips is n, the first iteration count is (n / 2). When n=8, the first iteration count is 4. In each iteration, each chipset needs to obtain its currently assigned target identifier group. (Refer to...) Figure 5 The process iterates a total of 4 times. In the first iteration, the target identifier group corresponding to chipset 1 is CD, the target identifier group corresponding to chipset 2 is EF, the target identifier group corresponding to chipset 3 is GH, and the target identifier group corresponding to chipset 4 is AB. In the second iteration, the target identifier group corresponding to chipset 1 is EF, the target identifier group corresponding to chipset 2 is GH, the target identifier group corresponding to chipset 3 is AB, and the target identifier group corresponding to chipset 4 is CD. In the third iteration, the target identifier group corresponding to chipset 1 is GH, the target identifier group corresponding to chipset 2 is AB, the target identifier group corresponding to chipset 3 is CD, and the target identifier group corresponding to chipset 4 is EF. In the fourth iteration, the target identifier group corresponding to chipset 1 is AB, the target identifier group corresponding to chipset 2 is CD, the target identifier group corresponding to chipset 3 is EF, and the target identifier group corresponding to chipset 4 is GH. This shows that during continuous iteration, the target identifier group in the same chipset is different in each iteration. For example, in the four iterations of chipset 1, the target identifier groups are CD, GH, EF, and AB. At the same time, the chipset corresponding to the same target identifier group is updated along the order of the chipsets. Taking the target identifier group CD as an example, its update order is: chipset 1, chipset 4, chipset 3, and chipset 2. The corresponding order of the chipsets is: chipset 4 -> chipset 3 -> chipset 2 -> chipset 1 -> chipset 4 in a circular direction.

[0096] Step 330: Iterate based on the first iteration number. In each iteration, for two chips in the same chipset, transmit the two initial data components corresponding to the target identifier group alternately based on the corresponding direct communication connection, and / or, between chipsets, transmit the two initial data components corresponding to the target identifier group based on the first loop and the second loop, respectively.

[0097] In one embodiment, iterations are performed according to a first iteration number. During each iteration, for two chips within the same chipset, two initial data components corresponding to the target identifier group are alternately transmitted based on their corresponding direct communication connections. Alternatively, between chipsets, the two initial data components corresponding to the target identifier group are transmitted based on a first loop and a second loop, respectively. Specifically, in the first iteration, only data transmission via the direct communication connection is performed; in subsequent iterations, data transmission via both the direct communication connection and the first and second loops is performed simultaneously. It is understood that if both intra-chipset and inter-chipset transmissions are performed concurrently, dual-loop data transmission must be performed only after the intra-chipset direct transmission link has completed.

[0098] In one embodiment, the data transmission process of a direct communication connection is first described. (Refer to...) Figure 6 , Figure 6 The flowchart provided in this application embodiment, which describes the alternating transmission of two initial data components corresponding to a target identifier group based on a corresponding direct communication connection between two chips in the same chipset, specifically includes the following steps:

[0099] Step 610: Assign the two component identifiers in the target identifier group to the first chip and the second chip in sequence.

[0100] In one embodiment, reference is made to Figure 4 and Figure 5 Taking the first iteration as an example, the target identifier group corresponding to chipset 1 is CD. Component identifier C is assigned to the first chip NPU0, and component identifier D is assigned to the second chip NPU1. This assignment can be set according to the actual situation. NPU2 is assigned to component identifier E, NPU3 to component identifier F, NPU4 to component identifier G, NPU5 to component identifier H, NPU6 to component identifier A, and NPU7 to component identifier B. This process continues, with the corresponding component identifiers assigned in the same way and order during each iteration.

[0101] Step 620: The initial data component in the first chip that has the same component identifier as the second chip and the initial data component in the second chip that has the same component identifier as the first chip are exchanged and transmitted through a direct communication connection.

[0102] In one embodiment, reference is made to Figure 7 , Figure 7 yes Figure 4 A detailed data transmission diagram is provided. The diagram uses dashed lines to represent direct communication connections and solid lines to represent data transmission in the first and second loops. The data within the dashed boxes corresponding to the chip represents data acquired during iteration via the direct communication connection, while the data within the thick solid boxes represents relevant data acquired during iteration via the first or second loop. (Refer to...) Figure 4 and Figure 7 In the first iteration, only data transmission occurs through direct communication connections. Figure 4 Although the loop data transmission was marked with a solid line in the first iteration, at this time, combined with Figure 7 There is no relevant data transmission in the loop.

[0103] In one embodiment, for data transmission via a direct communication connection, it is sufficient to exchange and transmit the initial data component in the first chip whose component identifier matches that of the second chip, and the initial data component in the second chip whose component identifier matches that of the first chip, through the direct communication connection. (Refer to...) Figure 7For chipset 1, the component identifier of the first chip NPU0 is C, and the component identifier of the second chip NPU1 is D. Therefore, the initial data component in the first chip NPU0 with the same component identifier D as the second chip NPU1 is D0, while the initial data component in the second chip NPU1 with the same component identifier C as the first chip NPU0 is C0. Therefore, they can be exchanged and transmitted through a direct communication connection. At this point, after the first iteration, Figure 7 NPU0 obtains C0 through a direct communication connection, and NPU1 obtains D0 through a direct communication connection. The initial data components contained in NPU0 at this time are: A0, B0, C0, C1, E0, F0, G0, and H0, while the initial data components contained in NPU1 at this time are: A1, B1, D0, D1, E1, F1, G1, and H1. This process continues, with each pair of chips within each chipset performing a direct transmission of initial data components.

[0104] The following describes the data exchange process between the first and second loops of the chipset. (Refer to...) Figure 8 , Figure 8 This application provides a flowchart illustrating the transmission of two initial data components corresponding to a target identifier group between chipsets based on a first loop and a second loop, respectively. The flowchart specifically includes the following steps:

[0105] Step 810: In the iterations after the second iteration, select each chipset as the current chipset, take the chipset corresponding to the target identifier group in the previous iteration as the target chipset in the current iteration, and determine the correspondence between the two chips in the target chipset and the component identifiers in the target identifier group.

[0106] In one embodiment, in each iteration after the second iteration, data transmission via direct communication connections within the chipset and loop data transmission between chipsets are required. In this case, in each iteration, each chipset is selected as the current chipset, and the chipset corresponding to the target identifier group in the previous iteration is selected as the target chipset. The correspondence between the two chips in the target chipset and the component identifiers in the target identifier group is then determined.

[0107] Reference Figure 7In the second iteration, chipsets are selected one by one as the current chipset, following the order from chipset 1 to chipset 4. Assuming the current chipset is chipset 1, its target identifier group in the previous iteration (i.e., the first iteration) is CD, and the chipset corresponding to target identifier group CD in the current iteration (i.e., the second iteration) is chipset 4, is designated as the target chipset for the current chipset. The target chipset contains two chips: NPU6 and NPU7. NPU6 corresponds to component identifier C in target identifier group CD, and NPU7 corresponds to component identifier D in target identifier group CD. This process continues, and in each iteration after the second iteration, the correspondence between the current chipset, the target chipset, and the component identifiers is determined sequentially.

[0108] Step 820: Take at least one initial data component corresponding to the two components of the target identifier group in the previous iteration in the two chips of the current chipset as the target data component sequence.

[0109] In one embodiment, reference is made to Figure 7 Taking the second iteration as an example, with chipset 1 as the current chipset, in its previous iteration, the two component identifiers of the target identifier group were component identifier C and component identifier D. At this time, in NPU0 of chipset 1, component identifier C corresponds to two initial data components C0 and C1. Therefore, C0 and C1 are taken as the target data component sequence of NPU0. Correspondingly, in NPU1 of chipset 1, component identifier D corresponds to two initial data components D0 and D1. Therefore, D0 and D1 are taken as the target data component sequence of NPU1. It can be understood that as the number of iterations increases, the number of initial data components contained in the relevant target data component sequence also increases.

[0110] Step 830: Based on the correspondence, transmit the target data component sequence to the corresponding chip of the target chipset through the first loop or the second loop.

[0111] In one embodiment, reference is made to Figure 7 In the target chipset, NPU6 corresponds to component identifier C in target identifier group CD, and NPU7 corresponds to component identifier D in target identifier group CD. NPU6 is located in the first loop, and NPU7 is located in the second loop. Therefore, the target data component sequences C0 and C1 of NPU0 are transmitted to NPU6 through the first loop, and the target data component sequences D0 and D1 of NPU1 are transmitted to NPU7 through the second loop.

[0112] The following reference Figure 4 and Figure 7Let's take chipset 3 as the current chipset in the third iteration as an example to illustrate the above-mentioned data transmission process on the ring. In the previous iteration (i.e., the second iteration) of chipset 3, the target identifier group was AB. In the current iteration (i.e., the third iteration), the chipset corresponding to the target identifier group AB is chipset 2. Chipset 2 is taken as the target chipset of the current chipset. The two chips in the target chipset are NPU2 and NPU3. Among them, NPU2 corresponds to component identifier A in the target identifier group AB, and NPU3 corresponds to component identifier B in the target identifier group AB.

[0113] Meanwhile, in the NPU4 of chipset 3, component identifier A corresponds to four initial data components A4, A5, A6, and A7. Therefore, A4, A5, A6, and A7 are used as the target data component sequence of NPU4. Correspondingly, in the NPU5 of chipset 3, component identifier B corresponds to four initial data components B4, B5, B6, and B7. Therefore, B4, B5, B6, and B7 are used as the target data component sequence of NPU5.

[0114] Since NPU2 in the target chipset corresponds to component identifier A in target identifier group AB, and NPU3 corresponds to component identifier B in target identifier group AB, and NPU2 is located in the first loop and NPU3 is located in the second loop, the target data component sequence A4, A5, A6, and A7 of NPU4 in chipset 3 is transmitted to NPU6 through the first loop, and the target data component sequence B4, B5, B6, and B7 of NPU5 in chipset 3 is transmitted to NPU3 through the second loop.

[0115] The above method is used to perform multiple iterations, referring to... Figure 7 Finally, multiple initial data components corresponding to the same component identifier are aggregated in each chip. For example, in NPU0, the aggregated component identifier A corresponds to A0, A1, A2, A3, A4, A5, A6, and A7; in NPU1, the aggregated component identifier B corresponds to B0, B1, B2, B3, B4, B5, B6, and B7; and in NPU7, the aggregated component identifier H corresponds to H0, H1, H2, H3, H4, H5, H6, and H7.

[0116] Step 340: The iteration ends. The initial data components corresponding to the same component identifier are aggregated to obtain a preset number of initial aggregated data.

[0117] In one embodiment, initial data components corresponding to the same component identifier are aggregated to obtain a preset number of initial aggregated data. For example, in NPU0, 8 initial data components related to component identifier A are aggregated to obtain the corresponding initial aggregated data A. The aggregation process of the initial data components can be described as: A = A0 + A1 + A2 + A3 + A4 + A5 + A6 + A7. In this way, each chip can obtain one initial aggregated data.

[0118] In one embodiment, reference is made to Figure 4 After the dual-ring data transmission, aggregation can also be performed once after each iteration. Taking target identifier group AB as an example, it serves as the target identifier group of chipset 4 during the first iteration. During the first iteration, NPU6 and NPU7 transmit data to each other through direct communication connection. Subsequently, the target data component sequence in NPU6 contains A6 and A7, and the target data component sequence in NPU7 contains B6 and B7.

[0119] In the second iteration, each chipset first performs a direct communication connection data transmission. During this process, chipset 3 directly exchanges data between NPU4 and NPU5, swapping B4 in NPU4 and A5 in NPU5. Next, chipset 3 is designated as the target chipset. NPU4 corresponds to component identifier A in target identifier group AB, and NPU5 corresponds to component identifier B in target identifier group AB. A6 and A7 from the target data component sequence in NPU6 are sent to NPU4 via the first loop, and B6 and B7 from the target data component sequence in NPU7 are sent to NPU5 via the second loop. At this point, the target data component sequence in NPU4 contains A4, A5, A6, and A7, and the target data component sequence in NPU5 contains B4, B5, B6, and B7. If a reduction calculation is performed before the dual-loop transmission, the target data component sequence in NPU4 will then contain A... 45 =A4+A5, A6, A7, the target data component sequence in NPU5 contains B. 45 B6, B7.

[0120] In the third iteration, a direct communication connection data transmission is first performed within each chipset. At this time, direct data exchange occurs between NPU2 and NPU3 in chipset 2, swapping B2 in NPU2 and A3 in NPU3. Next, chipset 2 is used as the target chipset, and A3 in the target data component sequence of NPU4 is transferred through the first loop. 45 A6 and A7 are sent to NPU2, and B in the target data component sequence of NPU5 is sent through the second loop. 45 B6 and B7 are sent to NPU3. At this point, the target data component sequence in NPU2 will subsequently contain A2, A3, and A7. 4567The target data component sequence in NPU3 contains B2, B3, and B 4567 .

[0121] In the fourth iteration, a direct communication connection data transmission is first performed within each chipset. At this time, direct data exchange occurs between NPU0 and NPU1 in chipset 1, swapping B0 in NPU0 and A1 in NPU1. Next, using chipset 1 as the target chipset, A2, A3, and A4 from the target data component sequence in NPU2 are transferred through the first loop. 4567 Send to NPU0, and through the second loop, retrieve B2, B3, and B6 from the target data component sequence in NPU3. 4567 The data is sent to NPU1. At this point, the target data component sequence in NPU0 will subsequently contain A0, A1, and A... 234567 The target data component sequence in NPU1 contains B0, B1, and B 234567 After one final reduction and aggregation, the data in NPU0 is A=A. 01234567 The data in NPU1 is B=B 01234567 .

[0122] As can be seen from the above process, the Reduce Scatter function implemented using a dual-loop chip structure in this application embodiment can be viewed as three stages: the initial reduction stage, the dual-loop transmission stage, and the final reduction stage. The initial reduction stage is the first iteration process, where data transmission is performed within the chipset using direct communication connections. This stage uses SIO and HCCS to achieve load balancing across the two links. Next, the dual-loop transmission stage, which encompasses all iterations after the second iteration, involves data transmission via HCCS in two independent loops, the first and second loops, while data synchronization occurs between the two loops via direct communication connections. The dual-loop transmission stage requires a total of (n / 2-1) communication steps. In the final reduction stage, since in-place reduction operations cannot be performed when data is simultaneously received from both the SIO and HCCS links, an additional reduction operation is needed to merge intermediate results from the SIO and HCCS buffers. This stage incurs no communication overhead and has a very short processing time.

[0123] In one embodiment, after the Reduce Scatter function is executed, the All Gather function phase begins. (See also...) Figure 9 , Figure 9 This is a flowchart of the All Gather function provided in the embodiments of this application, which specifically includes the following steps:

[0124] Step 910: Determine the number of the second iteration based on the number of chips.

[0125] In one embodiment, the second iteration of the All Gather function is (n / 2-1), where n is the number of chips. If n=8, then the second iteration is 3 times.

[0126] Step 920: Iterate based on the second iteration number. In each iteration, for two chips in the same chipset, determine the initial aggregated data that is missing from the chip relative to the other chip as the first direct transmission aggregated data. Transmit the first direct transmission aggregated data alternately based on the corresponding direct communication connection. In two adjacent chipsets, based on the orientation of the chipset, determine the initial aggregated data that is missing from the chip relative to the previous chip as the inter-group aggregated data. Transmit the inter-group aggregated data based on the orientation of the first loop and the second loop, respectively.

[0127] In one embodiment, similar to the Reduce Scatter function, it can be divided into a dual-loop transmission phase and a final synchronization phase. In the dual-loop transmission phase, data is transmitted via HCCS in two independent loops, a first loop and a second loop, while data synchronization between the two loops is achieved through a direct communication connection. Each iteration includes both direct communication connection transmission within the chipset and dual-loop data transmission.

[0128] In one embodiment, reference is made to Figure 10 , Figure 10 This is a schematic diagram illustrating the principle of the All Gather function provided in the embodiments of this application. (Continued here...) Figure 4 or Figure 7 The results are processed. Before processing begins, NPU0 includes initial aggregated data A, NPU1 includes initial aggregated data B, NPU2 includes initial aggregated data C, NPU3 includes initial aggregated data D, NPU4 includes initial aggregated data E, NPU5 includes initial aggregated data F, NPU6 includes initial aggregated data G, and NPU7 includes initial aggregated data H.

[0129] In each iteration, for two chips in the same chipset, the initial aggregated data missing from one chip relative to the other is determined as the first direct-transmission aggregated data. This first direct-transmission aggregated data is then transmitted alternately based on the corresponding direct communication connection. (Refer to...) Figure 10 Taking the first iteration as an example, for chipset 1, the initial aggregated data missing from NPU0 relative to NPU1 is B contained in NPU1, and the initial aggregated data missing from NPU1 relative to NPU0 is A contained in NPU0. Therefore, the first direct-transmission aggregated data includes A and B. At this time, A contained in NPU0 is transmitted to NPU1, and B contained in NPU1 is transmitted to NPU0. Similarly, the corresponding first direct-transmission aggregated data is also clearly defined in other chipsets, and alternating transmission of it is performed through direct communication connections.

[0130] Simultaneously, during iteration, adjacent chipsets determine the initial aggregated data missing from the previous chip relative to the next chip, based on the chipset orientation, as inter-group aggregated data, and transmit the inter-group aggregated data based on the orientations of the first and second loops, respectively. (Refer to...) Figure 10 Chipset 1 and Chipset 2 are adjacent chipsets, and the orientation of the chipsets is: Chipset 2 -> Chipset 1. Therefore, the orientations of the first and second loops are also determined. For the first loop, the initial aggregated data missing from the preceding chip NPU2 relative to the next chip NPU0 is C. C is used as the inter-group aggregated data, and the inter-group aggregated data C is transferred from NPU2 to NPU0 based on the orientation of the first loop. Correspondingly, for the second loop, the initial aggregated data missing from the preceding chip NPU3 relative to the next chip NPU1 is D. D is used as the inter-group aggregated data, and the inter-group aggregated data D is transferred from NPU3 to NPU1 based on the orientation of the second loop. The same operation is performed between other adjacent chipsets.

[0131] After the first iteration, NPU0 contains initial aggregated data A, B, and C; NPU1 contains initial aggregated data A, B, and D; NPU2 contains initial aggregated data C, D, and E; NPU3 contains initial aggregated data C, D, and F; NPU4 contains initial aggregated data E, F, and G; NPU5 contains initial aggregated data E, F, and H; NPU6 contains initial aggregated data A, G, and H; and NPU7 contains initial aggregated data B, G, and H.

[0132] Next, we proceed to the second iteration. (Refer to...) Figure 10 For chipset 1, the missing initial aggregated data for NPU0 relative to NPU1 is D contained in NPU1, and the missing initial aggregated data for NPU1 relative to NPU0 is C contained in NPU0. Therefore, the first direct-transmission aggregated data includes D and C. At this time, D contained in NPU0 is transmitted to NPU1, and C contained in NPU1 is transmitted to NPU0. Similarly, the corresponding first direct-transmission aggregated data is also clearly defined in other chipsets, and alternating transmission of it is performed through direct communication connections.

[0133] Simultaneously, chipset 1 and chipset 2, being two adjacent chipsets, for the first loop, the missing initial aggregated data D and E for the subsequent chip NPU0 relative to the preceding chip NPU2 are used as inter-group aggregated data, and the inter-group aggregated data D and E are transferred from NPU2 to NPU0 based on the direction of the first loop. Correspondingly, for the second loop, the missing initial aggregated data C and F for the subsequent chip NPU1 relative to the preceding chip NPU3 are used as inter-group aggregated data, and the inter-group aggregated data C and F are transferred from NPU3 to NPU1 based on the direction of the second loop. The same operation is performed between other adjacent chipsets.

[0134] After the second iteration, NPU0 contains initial aggregated data A, B, C, D, and E; NPU1 contains initial aggregated data A, B, C, D, and F; NPU2 contains initial aggregated data C, D, E, F, and G; NPU3 contains initial aggregated data C, D, E, F, and H; NPU4 contains initial aggregated data A, E, F, G, and H; NPU5 contains initial aggregated data B, E, F, G, and H; NPU6 contains initial aggregated data A, B, C, G, and H; and NPU7 contains initial aggregated data A, B, D, G, and H.

[0135] The third iteration was performed following the above process. After the third iteration, each chip contained seven different initial aggregate data.

[0136] Step 930: After the iteration is completed, for two chips in the same chipset, the initial aggregated data that is missing from the chip relative to the other chip is determined as the second direct-transmission aggregated data. The first direct-transmission aggregated data is transmitted alternately based on the corresponding direct communication connection until each chip contains all the initial aggregated data.

[0137] In one embodiment, after the iteration is completed, the dual-loop transmission phase is finished, at which point each chip is missing a piece of initial aggregated data. (Refer to...) Figure 10NPU0 is missing initial aggregated data H, NPU1 is missing initial aggregated data G, NPU2 is missing initial aggregated data B, and so on. Each NPU chip in the dual-ring transmission stage is missing a piece of data. Therefore, in this embodiment, the final synchronization stage uses a dual-link HCCS / SIO connection to transmit the last piece of initial aggregated data. Direct communication connections within the chipset are used for transmission. For two chips in the same chipset, the initial aggregated data missing from one chip relative to the other is determined as the second direct-transmission aggregated data. The first direct-transmission aggregated data is transmitted alternately based on the corresponding direct communication connection until each chip contains all the initial aggregated data. For example, for chipset 1, the initial aggregated data missing from NPU0 relative to NPU1 is H contained in NPU1, and the initial aggregated data missing from NPU1 relative to NPU0 is G contained in NPU0. Therefore, the second direct-transmission aggregated data includes H and G. At this point, G contained in NPU0 is transmitted to NPU1, and H contained in NPU1 is transmitted to NPU0. This process continues until all data transmissions within the direct communication connections of the chipset are completed.

[0138] This completes the All Gather function. As can be seen, this embodiment utilizes a dual-loop chip structure, combining the ReduceScatter and All Gather functions to achieve the All Reduce operation. During operation, the first and second chips in each of multiple chipsets can directly communicate to exchange information within the group, eliminating the need for loop relays and enabling efficient interaction via direct links, reducing latency and path loss in intra-group communication. Simultaneously, the first and second loops formed by the first and second chips of different chipsets are independent and parallel, reducing the probability of data blocking in cross-group communication and making the communication path more flexible. This "direct intra-group connection + cross-group dual-loop parallel" structure shortens the communication path and improves data transmission efficiency, thereby significantly improving the execution efficiency of aggregated communication.

[0139] In one embodiment, a Broadcast function in aggregated communication is described based on a dual-loop chip architecture. (Refer to...) Figure 11 , Figure 11 This is another optional flowchart of the collection communication method provided in the embodiments of this application. Figure 11 The method described herein is the implementation logic for the Broadcast function, and may include, but is not limited to, steps 1110 to 1130. It is also understood that this embodiment... Figure 11 The order of steps 1110 to 1130 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0140] Step 1110: Determine the source chip from all the chips contained in the dual-loop chip structure, and divide the preset number of initial block data in the source chip equally to obtain the first data group and the second data group.

[0141] In one embodiment, a source chip is determined from all chips included in the dual-loop chip structure. The source chip can be set according to actual conditions. The source chip contains a preset number of initial block data. Then, the initial block data is divided equally to obtain a first data group and a second data group. The first data group and the second data group contain the same number of initial block data.

[0142] In one embodiment, reference is made to Figures 12a to 12c , Figures 12a to 12c This is a schematic diagram illustrating the principle of the Broadcast function provided in this application embodiment. Assume NPU0 is the source chip, containing eight initial data blocks A'-H'. First, the eight initial data blocks are evenly divided, separating them into two parts according to actual needs, such as... Figure 12a In the diagram, A', B', C', and D represent the first data group, while E', F', G', and H' represent the second data group.

[0143] Step 1120: Transmit the second data group to another chip corresponding to the source chip via the corresponding direct communication connection.

[0144] In one embodiment, the Broadcast function can also be divided into an initial phase, a dual-ring transmission phase, and a final synchronization phase. In the initial phase, the second data group is transmitted to another chip corresponding to the source chip via a corresponding direct communication connection. (Refer to...) Figure 12a The source chip NPU0 is located in chipset 1, and its corresponding chip is NPU1. At this time, the second data group E', F', G', H' is transmitted to the corresponding chip NPU1 through the direct communication connection in chipset 1.

[0145] Step 1130: Broadcast the first data group and the second data group through the first loop, the second loop, and each direct communication connection until each chip contains all the initial block data.

[0146] In one embodiment, reference is made to Figure 13 , Figure 13 This application provides a flowchart illustrating the broadcast transmission of the first and second data groups in the first loop, the second loop, and each direct communication connection until each chip contains all the initial block data. The flowchart specifically includes the following steps:

[0147] Step 1310: Determine the number of the third iteration based on the number of chips and the preset number.

[0148] In one embodiment, the third iteration number of the dual-ring transmission stage is determined based on the number of chips and a preset number, expressed as (m+n) / 2-2, where m is the preset number. Assuming m=n=8, the third iteration number is 6.

[0149] Step 1320: Iterate based on the third iteration number. In each iteration, for two chips in the same chipset, determine the missing initial block data of the chip relative to the other chip as the third direct transmission aggregate data. Transmit the third direct transmission aggregate data alternately based on the corresponding direct communication connection. In two adjacent chipsets, based on the orientation of the chipset, determine the missing initial block data of the latter chip relative to the former chip as the inter-group block data. Transmit the inter-group block data based on the orientation of the first loop and the second loop respectively.

[0150] In one embodiment, during each iteration, for two chips in the same chipset, an initial data block missing from one chip relative to the other is determined as the third direct-transmission aggregated data, and the third direct-transmission aggregated data is alternately transmitted based on the corresponding direct communication connection. (Refer to...) Figure 12b Taking the first iteration as an example, for chipset 1, NPU0 and NPU1 communicate with each other. It is determined that chip NPU1 is missing an initial block of data relative to chip NPU0, which is A'. However, NPU0 does not have any missing initial block of data relative to chip NPU1. Therefore, only A' is used as the third direct-transfer aggregated data. Then, based on the corresponding direct communication connection in chipset 1 (indicated by the dotted line in the figure), the third direct-transfer aggregated data is transmitted alternately, transferring the initial block of data A' from NPU0 to NPU1. At this time, since there is no initial block of data in other chipsets, the corresponding third direct-transfer aggregated data can be regarded as empty.

[0151] Next, in two adjacent chipsets, based on the orientation of the chipset, the initial block data missing from the previous chip relative to the subsequent chip is determined as inter-group block data, and the inter-group block data is transmitted based on the orientation of the first loop and the second loop, respectively. (Refer to...) Figure 12bIn the first iteration, chipset 1 and chipset 2 are adjacent, with the direction being chipset 1 -> chipset 2. Therefore, in the first loop, the missing initial block data A' for the latter chip NPU2 relative to the former chip NPU0 is used as the inter-group block data corresponding to the first loop. Based on the direction of the first loop, the inter-group block data is transmitted, and A' in NPU0 is transmitted along the first loop to NPU2. Similarly, in the second loop, the missing initial block data H' for the latter chip NPU3 relative to the former chip NPU1 is used as the inter-group block data corresponding to the second loop. Based on the direction of the second loop, the inter-group block data is transmitted, and H' in NPU1 is transmitted along the second loop to NPU3.

[0152] Next, refer to Figure 12b Let's take the second iteration as an example to continue the explanation.

[0153] For chipset 1, it is determined that chip NPU1 is missing an initial block of data relative to chip NPU0, which is B'. NPU0 has no missing initial block of data relative to NPU1. Therefore, only B' is used as the third direct-transfer aggregated data. Then, based on the corresponding direct communication connection in chipset 1, the initial block of data B' in NPU0 is transmitted to NPU1. For chipset 2, it is determined that chip NPU2 is missing an initial block of data relative to chip NPU3, which is H'. NPU3 is missing an initial block of data relative to NPU2, which is A'. Therefore, A' and H' are used as the third direct-transfer aggregated data. Then, based on the corresponding direct communication connection in chipset 2, the initial block of data A' in NPU2 is transmitted to NPU3, and the initial block of data H' in NPU3 is transmitted to NPU2. Since chipsets 3 and 4 do not yet have initial block of data, their corresponding third direct-transfer aggregated data can be considered empty.

[0154] Next, chipset 1 and chipset 2 are adjacent. Therefore, in the first loop, the missing initial block data of the latter chip NPU2 relative to the former chip NPU0 is B'. Thus, B' is used as the inter-group block data corresponding to the first loop, and the inter-group block data is transmitted according to the direction of the first loop, transmitting B' from NPU0 to NPU2 along the first loop. Similarly, in the second loop, the missing initial block data of the latter chip NPU3 relative to the former chip NPU1 is G'. Therefore, G' is used as the inter-group block data corresponding to the second loop, and the inter-group block data is transmitted according to the direction of the second loop, transmitting G' from NPU1 to NPU3 along the second loop. Meanwhile, chipset 3 and chipset 2 are adjacent. Therefore, in the first loop, the missing initial block data of the latter chip NPU4 relative to the former chip NPU2 is A'. Therefore, A' is used as the inter-group block data corresponding to the first loop, and the inter-group block data is transmitted according to the direction of the first loop, transmitting A' from NPU2 to NPU4 along the first loop. Similarly, in the second loop, the initial block data that the next chip NPU5 is missing from the previous chip NPU3 is H'. Therefore, H' is used as the inter-group block data corresponding to the second loop. Based on the direction of the second loop, the inter-group block data is transmitted, and H' in NPU3 is transmitted to NPU5 along the second loop.

[0155] This process continues until the third iteration corresponds to the end of the iteration process.

[0156] Step 1330: After the iteration is completed, for two chips in the same chipset, the initial aggregated data that is missing from the chip relative to the other chip is determined as the fourth direct-transmission aggregated data. The fourth direct-transmission aggregated data is transmitted alternately based on the corresponding direct communication connection until each chip contains all the initial block data.

[0157] In one embodiment, reference is made to Figure 12b Except for the last chipset, the two chips in the other chipsets contain all the initial block data. However, the two chips in the last chipset are each missing one initial block data. Therefore, in the final synchronization phase, data completion is performed using the direct communication connection within the chipset. For two chips in the same chipset, the initial block data missing by one chip relative to the other is determined as the fourth direct transmission aggregate data. The fourth direct transmission aggregate data is transmitted alternately based on the corresponding direct communication connection until each chip contains all the initial block data.

[0158] Reference Figure 12b After the final iteration, NPU6 in chipset 4 was missing E', and NPU7 was missing D'. Therefore, the direct communication connection within chipset 4 was used to alternately transmit D' from NPU6 and E' from NPU7. (See reference...) Figure 12cAfter alternating transmission within chipset 4, NPU6 and NPU7 also contain all the initial block data.

[0159] At this point, all chips contain all the initial block data, and the Broadcast function is implemented.

[0160] As can be seen, the embodiments of this application are based on a dual-loop chip structure for aggregated communication. The two computing dies (such as NPUs) within the chipplet package are placed in two independent logical ring networks. Odd- and even-numbered NPUs are connected via traditional links such as HCCS to form two parallel ring loops, resulting in the first and second loops. Simultaneously, a high-bandwidth SIO link connects the corresponding odd- and even-numbered NPU pairs within the same chip package as a chipset, responsible for data synchronization between the two loops and high-speed communication within the chip. This effectively utilizes the high-speed inter-chip SIO resources of the chipplet package, allowing simultaneous data transmission in both the first and second loops. Data exchange and synchronization are achieved through direct communication connections within the chipset, realizing load balancing of the dual links and higher bandwidth utilization, fully leveraging hardware potential, and further improving communication bandwidth and efficiency.

[0161] In one embodiment, experiments show that on an Ascend NPU cluster, the ensemble communication method of this application embodiment achieves a significant improvement in algorithm bandwidth, for example, an average improvement of 15.81% in a 4-NPU configuration and an average improvement of 9.15% in a 128-NPU configuration. Secondly, by decoupling the communication mode into a coordinated dual-ring operation, the cardinality of the rings is reduced (by 50%) and the communication volume of the HCCS link is decreased, while retaining the bandwidth advantage of SIO. Finally, it exhibits good scalability across different cluster sizes from 4 to 128 NPUs, significantly improving the overall communication efficiency of distributed training of large-scale AI models.

[0162] The technical solution provided in this application embodiment includes a dual-loop chip structure comprising: multiple chipsets, each chipset including a first chip and a second chip, with a direct communication connection between the corresponding first chip and second chip; the first chips in different chipsets are connected sequentially according to the order of the chipsets to form a first loop; the second chips in different chipsets are connected sequentially according to the order of the chipsets to form a second loop. When the dual-loop chip structure provided in this application embodiment is applied to scenarios such as aggregated communication where data needs to be broadcast, aggregated, or synchronized among multiple chips, the first chip and second chip in each of the multiple chipsets can achieve information exchange between chips within the group through direct communication connections, without needing to go through a loop relay. Direct links can achieve efficient interaction, reducing latency and path loss in intra-group communication. Simultaneously, the first and second loops formed by the first and second chips of different chipsets are independent and parallel, reducing the probability of data blocking in cross-group communication and making the communication path more flexible. This "direct connection within the group + parallel dual-loop across groups" structure can shorten the communication path and improve data transmission efficiency, thereby significantly improving the execution efficiency of aggregated communication.

[0163] This application also provides a collection communication device and a collection communication method that can implement the above-mentioned All Reduce function, see reference. Figure 14 The device includes:

[0164] Data acquisition module 1410: used to acquire a preset number of initial data components contained in all chips in the dual-loop chip structure. The initial data components include component identifiers, and the component identifiers are grouped into target identifier groups in pairs. The dual-loop chip structure is the dual-loop chip structure of claim 1.

[0165] Communication preparation module 1420: used to determine the first iteration number based on the number of chips, and to determine the target identifier group corresponding to the chipset in each iteration. In the continuous iteration process, the target identifier group in the same chipset is different in each iteration, and the chipset corresponding to the same target identifier group is updated along the order of the chipsets.

[0166] Data transmission module 1430: is used to obtain the data based on the first iteration number, and in each iteration, for two chips in the same chipset, to alternately transmit the two initial data components corresponding to the target identifier group based on the corresponding direct communication connection, and / or, between chipsets, to transmit the two initial data components corresponding to the target identifier group based on the first loop and the second loop respectively.

[0167] Aggregation module 1440: Used to aggregate the initial data components corresponding to the same component identifier until the end of the iteration, to obtain a preset number of initial aggregated data, with each chip obtaining one initial aggregated data.

[0168] The specific implementation of the collective communication device in this embodiment is basically the same as the specific implementation of the collective communication method described above, and will not be repeated here.

[0169] This application also provides a collective communication device and a collective communication method that can implement the above-mentioned Broadcast function, see reference. Figure 15 The device includes:

[0170] Initial data allocation module 1510: used to determine the source chip from all the chips included in the dual-loop chip structure, and to divide the preset number of initial block data in the source chip equally to obtain a first data group and a second data group. The number of chips is not less than the preset number. The dual-loop chip structure is the dual-loop chip structure of claim 1.

[0171] Intra-group transmission module 1520: used to transmit the second data group to another chip corresponding to the source chip via the corresponding direct communication connection.

[0172] Broadcast transmission module 1530: used to broadcast the first data group and the second data group in the first loop, the second loop and each direct communication connection until each chip contains all the initial block data.

[0173] The specific implementation of the collective communication device in this embodiment is basically the same as the specific implementation of the collective communication method described above, and will not be repeated here.

[0174] This application also provides an electronic device, including:

[0175] At least one memory;

[0176] At least one processor;

[0177] At least one program;

[0178] The program is stored in a memory, and the processor executes the at least one program to implement the above-described collection communication method of this application. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), in-vehicle computers, etc.

[0179] Please see Figure 16 , Figure 16 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0180] The processor 1601 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0181] The memory 1602 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1602 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1602 and is called and executed by the processor 1601 using the collection communication method of the embodiments of this application.

[0182] The input / output interface 1603 is used to implement information input and output;

[0183] The communication interface 1604 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0184] Bus 1605 transmits information between various components of the device (e.g., processor 1601, memory 1602, input / output interface 1603, and communication interface 1604);

[0185] The processor 1601, memory 1602, input / output interface 1603 and communication interface 1604 are connected to each other within the device via bus 1605.

[0186] This application embodiment also provides a storage medium that stores a computer program, which, when executed by a processor, implements the above-described collection communication method.

[0187] Memory, as a non-transitory storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0188] The dual-loop chip structure, integrated communication method, device, equipment, and storage medium proposed in the embodiments of this application provide technical solutions and beneficial effects.

[0189] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0190] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0191] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0192] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0193] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0194] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0195] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0196] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0197] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0198] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0199] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. An aggregation communication method characterized by, The method comprises: acquiring a preset number of initial data components contained in all chips in a double-loop chip structure, the initial data components corresponding to component identifiers, the component identifiers being grouped into target identifier groups two by two, the double-loop chip structure comprising a plurality of chip groups, each chip group comprising a first chip and a second chip, and a direct communication connection existing between the corresponding first chip and second chip; the first chips in different chip groups are sequentially connected in the order of the chip groups to form a first loop; the second chips in different chip groups are sequentially connected in the order of the chip groups to form a second loop; determining a first iteration number according to the number of chips, and determining the target identifier group corresponding to the chip group at each iteration, wherein the target identifier group in the same chip group is different at each iteration in the continuous iteration process, and the chip group corresponding to the same target identifier group is updated in the order of the chip groups; iterating based on the first iteration number, wherein at each iteration, for the two chips in the same chip group, two initial data components corresponding to the target identifier group are alternately transmitted based on the corresponding direct communication connection, and / or, for the chip groups, two initial data components corresponding to the target identifier group are transmitted based on the first loop and the second loop, respectively; until the iteration ends, the initial data components corresponding to the same component identifier are aggregated to obtain a preset number of initial aggregated data, and each chip obtains one initial aggregated data.

2. The collective communication method according to claim 1, wherein The method further comprises: determining a second iteration number according to the number of chips. ​ 3. The collective communication method according to claim 2, wherein ​ ​ ​ ​ 4. The group communication method of claim 1, wherein, ​ ​ iterating based on the second iteration number, at each iteration, for two chips in the same chip group, determining the initial aggregated data missing from one chip relative to the other chip as first direct transmission aggregated data, alternately transmitting the first direct transmission aggregated data based on the corresponding direct communication connection, for two adjacent chip groups, based on the direction of the chip group, determining the initial aggregated data missing from one chip relative to the other chip as inter-group aggregated data, transmitting the inter-group aggregated data based on the direction of the first loop and the second loop respectively; after the iteration, for two chips in the same chip group, determining the initial aggregated data missing from one chip relative to the other chip as second direct transmission aggregated data, alternately transmitting the second direct transmission aggregated data based on the corresponding direct communication connection, until each chip contains all the initial aggregated data.

5. An aggregation communication method characterized by comprising: Comprising: determining a source chip from all chips included in a double-loop chip structure, dividing a preset number of initial block data in the source chip equally to obtain a first data group and a second data group, the number of chips is not less than the preset number, the double-loop chip structure includes a plurality of chip groups, each chip group includes a first chip and a second chip, and the corresponding first chip and the second chip have a direct communication connection; the first chips in different chip groups are connected in turn according to the order of the chip groups, forming a first loop; the second chips in different chip groups are connected in turn according to the order of the chip groups, forming a second loop; transmitting the second data group to the corresponding another chip of the source chip through the corresponding direct communication connection; determining a third iteration number according to the number of chips and the preset number; iterating based on the third iteration number, at each iteration, for two chips in the same chip group, determining one initial block data missing from one chip relative to the other chip as third direct transmission aggregated data, alternately transmitting the third direct transmission aggregated data based on the corresponding direct communication connection, for two adjacent chip groups, based on the direction of the chip group, determining one initial block data missing from one chip relative to the other chip as inter-group block data, transmitting the inter-group block data based on the direction of the first loop and the second loop respectively; after the iteration, for two chips in the same chip group, determining the initial block data missing from one chip relative to the other chip as fourth direct transmission aggregated data, alternately transmitting the fourth direct transmission aggregated data based on the corresponding direct communication connection, until each chip contains all the initial block data.

6. An aggregation communication device, comprising: Comprising: The data acquisition module is configured to acquire a preset number of initial data components contained in all chips in a double-loop chip structure, the initial data components correspond to component identifiers, the component identifiers are grouped into target identifier groups two by two, the double-loop chip structure includes a plurality of chip groups, each chip group includes a first chip and a second chip, and the corresponding first chip and the second chip are directly connected in communication; the first chips in different chip groups are sequentially connected in the order of the chip groups to form a first loop; and the second chips in different chip groups are sequentially connected in the order of the chip groups to form a second loop. The communication preparation module is configured to determine a first iteration number according to the number of chips, and determine the target identifier groups corresponding to the chip groups at each iteration; in a continuous iteration process, the target identifier groups in the same chip group are different at each iteration, and the chip groups corresponding to the same target identifier group are updated in the order of the chip groups. The data transmission module is configured to perform iteration based on the first iteration number, and at each iteration, alternately transmit two initial data components corresponding to the target identifier groups based on the direct communication connection corresponding to the two chips in the same chip group, and / or transmit the two initial data components corresponding to the target identifier groups based on the first loop and the second loop between the chip groups, respectively. The aggregation module is configured to aggregate the initial data components corresponding to the same component identifier until the iteration ends, to obtain a preset number of initial aggregated data, and each chip obtains one initial aggregated data.

7. An aggregation communication device, comprising: The initial data allocation module is configured to determine a source chip from all chips in a double-loop chip structure, and equally divide a preset number of initial block data in the source chip to obtain a first data group and a second data group, the number of chips is not less than the preset number, the double-loop chip structure includes a plurality of chip groups, each chip group includes a first chip and a second chip, and the corresponding first chip and the second chip are directly connected in communication; the first chips in different chip groups are sequentially connected in the order of the chip groups to form a first loop; and the second chips in different chip groups are sequentially connected in the order of the chip groups to form a second loop. The intra-group transmission module is configured to transmit the second data group to another chip corresponding to the source chip through the corresponding direct communication connection. ​ The broadcast transmission module is configured to determine a third iteration number according to the number of the chips and the preset number; perform iteration based on the third iteration number, determine, at each iteration, one of the initial block data missing from one of the chips relative to the other chip as third direct transmission aggregated data, alternately transmit the third direct transmission aggregated data based on the corresponding direct communication connection, determine, based on the direction of the chip group, one of the initial block data missing from the latter chip relative to the former chip as inter-group block data, and transmit the inter-group block data based on the direction of the first loop and the second loop, respectively; and after the iteration, determine the initial block data missing from one of the chips relative to the other chip as fourth direct transmission aggregated data, alternately transmit the fourth direct transmission aggregated data based on the corresponding direct communication connection, until all the initial block data are contained in each chip.

8. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the set communication method in any one of claims 1 to 5 when executing the computer program.

9. A storage medium storing a computer program, characterized by The computer program is executed by the processor to implement the set communication method in any one of claims 1 to 5.

Images