Optimization methods for distributed operators, artificial intelligence chips, computer equipment, readable storage media, and program products

Abstract

This application relates to an optimization method for distributed operators, an artificial intelligence chip, a computer device, a readable storage medium, and a program product. The method includes: each streaming processor cluster (SPC) of the artificial intelligence chip has a global address mapping unit internally configured; the SPC of the artificial intelligence chip reads operation data from global memory into shared memory to perform corresponding matrix multiplication calculations, and the results of the matrix multiplication calculations are stored in the shared memory; after the matrix multiplication calculations are completed, the global address mapping unit, based on the locally recorded shared memory address information of each target SPC, retrieves the results of the matrix multiplication calculations from the shared memory of each target SPC and writes the retrieved results into the shared memory of the SPC; based on the results of each matrix multiplication calculation stored in the shared memory of the SPC, the next matrix multiplication calculation is performed. This method can improve the overall execution efficiency of distributed operators.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence chip technology, and in particular to an optimization method for a distributed operator, an artificial intelligence chip, a computer device, a readable storage medium, and a program product. Background Technology

[0002] With the rapid development of artificial intelligence technology, the demand for computational efficiency in large-scale model training continues to rise. Operator fusion technology for AI chips has become a core means to improve computing performance, and distributed operator fusion is a key path to overcome the limitations of single-card computing power and memory. Distributed operator fusion technology requires multiple AI chip devices to collaboratively execute the fused computational logic, and its core requirement is to achieve efficient data communication between devices.

[0003] In traditional solutions, cross-device communication for distributed operator fusion is achieved using global memory, which includes: the source AI chip copies the data stored in the shared memory of the SPC (Streaming Processor Cluster) to the global memory; then, the data is sent from the global memory of the source AI chip to the global memory of the target AI chip; finally, before computation, the SPC of the target AI chip copies the data from the global memory to the shared memory of the SPC.

[0004] As a result, cross-device communication requires two additional data copies (i.e., the source AI chip copies data from shared memory to global memory, and the target AI chip copies data from global memory to shared memory). The cumulative effect of multiple copy operations between memory at different speeds will significantly increase communication latency and significantly reduce the overall execution efficiency of distributed operators, making it difficult to meet the high throughput and low latency requirements of large model training. Summary of the Invention

[0005] Therefore, it is necessary to provide an optimization method for distributed operators, an artificial intelligence chip, a computer device, a readable storage medium, and a program product that can meet the requirements of high throughput and low latency for large model training, in order to address the above-mentioned technical problems.

[0006] In a first aspect, this application provides an optimization method for a distributed operator, applied to an artificial intelligence chip. Each streaming processor cluster of the artificial intelligence chip is internally equipped with a global address mapping unit. The global address mapping unit records shared memory address information of the streaming processor clusters of the artificial intelligence chip involved in the distributed operator. The method includes:

[0007] After the target task is started, during the calculation of the current data, the streaming processor cluster of the artificial intelligence chip reads the operation data from the global memory into the shared memory to perform the corresponding matrix multiplication calculation. The result of the matrix multiplication calculation is stored in the shared memory of the streaming processor cluster. The operation data is the data block corresponding to the current data.

[0008] After the matrix multiplication calculation is completed, a communication process is triggered. The communication process includes the global address mapping unit of the streaming processor cluster obtaining the matrix multiplication calculation result from the shared memory of each target streaming processor cluster based on the shared memory address information of each target streaming processor cluster recorded locally, and writing the obtained result into the shared memory of the streaming processor cluster.

[0009] Based on the results of each matrix multiplication calculation stored in the shared memory of the streaming processor cluster, the matrix multiplication calculation for the next calculation process is executed.

[0010] In one embodiment, obtaining the matrix multiplication result from the shared memory of each target streaming processor cluster based on the locally recorded shared memory address information of each target streaming processor cluster includes:

[0011] From the shared memory address information of the streaming processor clusters of each artificial intelligence chip recorded locally, the shared memory address information of each target streaming processor cluster is determined. The target streaming processor cluster is the streaming processor cluster of other artificial intelligence chips participating in the same distributed operator computing task.

[0012] Access the shared memory of each target streaming processor cluster based on the shared memory address information of each target streaming processor cluster, and read the matrix multiplication calculation results from the shared memory of each target streaming processor cluster.

[0013] In one embodiment, obtaining the matrix multiplication result from the shared memory of each target streaming processor cluster based on the locally recorded shared memory address information of each target streaming processor cluster includes:

[0014] From the shared memory address information of the streaming processor clusters of each artificial intelligence chip recorded locally, the shared memory address information of each target streaming processor cluster is determined. The target streaming processor cluster is the streaming processor cluster of other artificial intelligence chips participating in the same distributed operator computing task.

[0015] Based on the shared memory address information of each target streaming processor cluster, the results of matrix multiplication calculations stored locally are sent to each target streaming processor cluster, and the results of matrix multiplication calculations sent by each target streaming processor cluster are received.

[0016] In one embodiment, the shared memory address information includes a streaming processor cluster identifier and a base shared memory address. The streaming processor cluster identifier includes the identifier of the AI ​​chip to which the streaming processor cluster belongs and the identifier of the streaming processor cluster itself. The base shared memory address in the shared memory address information of the streaming processor clusters of each AI chip is consistent.

[0017] In one embodiment, the distributed operator includes a fusion operator for performing matrix multiplication calculations and inter-chip communication, and the method further includes:

[0018] Receive the target task issued by the central processing unit;

[0019] The fusion operator is scheduled to execute the target task.

[0020] Secondly, this application also provides an artificial intelligence chip, including at least one streaming processor cluster and global memory, wherein the streaming processor cluster includes computing units, shared memory, and a global address mapping unit, and the global address mapping unit records the shared memory address information of the streaming processor clusters of each artificial intelligence chip involved in the distributed operator, wherein:

[0021] The streaming processor cluster is used to read operation data from the global memory and write the read operation data into the shared memory. The operation data is a data block of operation data corresponding to the currently executed target task.

[0022] The computing unit is configured to perform matrix multiplication calculations based on the operation data written to the shared memory, and store the results of the matrix multiplication calculations in the shared memory;

[0023] The global address mapping unit is used to trigger a communication process after the computing unit completes the matrix multiplication calculation. The communication process includes the global address mapping unit of the streaming processor cluster obtaining the matrix multiplication calculation result from the shared memory of each target streaming processor cluster based on the shared memory address information of each target streaming processor cluster recorded locally, and writing the obtained result into the shared memory of the streaming processor cluster.

[0024] The computing unit is also used to perform matrix multiplication calculations for the next calculation process based on the results of each matrix multiplication calculation stored in the shared memory of the streaming processor cluster.

[0025] In one embodiment, obtaining the matrix multiplication result from the shared memory of each target streaming processor cluster based on the locally recorded shared memory address information of each target streaming processor cluster includes:

[0026] From the shared memory address information of the streaming processor clusters of each artificial intelligence chip recorded locally, the shared memory address information of each target streaming processor cluster is determined. The target streaming processor cluster is the streaming processor cluster of other artificial intelligence chips participating in the same distributed operator computing task.

[0027] Access the shared memory of each target streaming processor cluster based on the shared memory address information of each target streaming processor cluster, and read the matrix multiplication calculation results from the shared memory of each target streaming processor cluster.

[0028] In one embodiment, obtaining the matrix multiplication result from the shared memory of each target streaming processor cluster based on the locally recorded shared memory address information of each target streaming processor cluster includes:

[0029] From the shared memory address information of the streaming processor clusters of each artificial intelligence chip recorded locally, the shared memory address information of each target streaming processor cluster is determined. The target streaming processor cluster is the streaming processor cluster of other artificial intelligence chips participating in the same distributed operator computing task.

[0030] Based on the shared memory address information of each target streaming processor cluster, the results of matrix multiplication calculations stored locally are sent to each target streaming processor cluster, and the results of matrix multiplication calculations sent by each target streaming processor cluster are received.

[0031] Thirdly, this application also provides an optimization device for a distributed operator, applied to an artificial intelligence chip. Each streaming processor cluster of the artificial intelligence chip is internally equipped with a global address mapping unit. The global address mapping unit records shared memory address information of the streaming processor clusters of the artificial intelligence chip involved in the distributed operator. The device includes:

[0032] The first computing module is used to, after starting the target task, read operation data from global memory into shared memory during the calculation process for the current data in the streaming processor cluster of the artificial intelligence chip to perform corresponding matrix multiplication calculations. The result of the matrix multiplication calculation is stored in the shared memory of the streaming processor cluster. The operation data is the data block corresponding to the current data.

[0033] The communication module is used to trigger a communication process after the matrix multiplication calculation is completed. The communication process includes the global address mapping unit of the streaming processor cluster obtaining the matrix multiplication calculation result from the shared memory of each target streaming processor cluster based on the shared memory address information of each target streaming processor cluster recorded locally, and writing the obtained result into the shared memory of the streaming processor cluster.

[0034] The second calculation module is used to perform matrix multiplication calculations for the next calculation process based on the results of each matrix multiplication calculation stored in the shared memory of the streaming processor cluster.

[0035] In one embodiment, obtaining the matrix multiplication result from the shared memory of each target streaming processor cluster based on the locally recorded shared memory address information of each target streaming processor cluster includes:

[0036] From the shared memory address information of the streaming processor clusters of each artificial intelligence chip recorded locally, the shared memory address information of each target streaming processor cluster is determined. The target streaming processor cluster is the streaming processor cluster of other artificial intelligence chips participating in the same distributed operator computing task.

[0037] Access the shared memory of each target streaming processor cluster based on the shared memory address information of each target streaming processor cluster, and read the matrix multiplication calculation results from the shared memory of each target streaming processor cluster.

[0038] In one embodiment, obtaining the matrix multiplication result from the shared memory of each target streaming processor cluster based on the locally recorded shared memory address information of each target streaming processor cluster includes:

[0039] From the shared memory address information of the streaming processor clusters of each artificial intelligence chip recorded locally, the shared memory address information of each target streaming processor cluster is determined. The target streaming processor cluster is the streaming processor cluster of other artificial intelligence chips participating in the same distributed operator computing task.

[0040] Based on the shared memory address information of each target streaming processor cluster, the results of matrix multiplication calculations stored locally are sent to each target streaming processor cluster, and the results of matrix multiplication calculations sent by each target streaming processor cluster are received.

[0041] In one embodiment, the shared memory address information includes a streaming processor cluster identifier and a base shared memory address. The streaming processor cluster identifier includes the identifier of the AI ​​chip to which the streaming processor cluster belongs and the identifier of the streaming processor cluster itself. The base shared memory address in the shared memory address information of the streaming processor clusters of each AI chip is consistent.

[0042] In one embodiment, the distributed operator includes a fusion operator for performing matrix multiplication calculations and inter-chip communication, and the apparatus further includes:

[0043] A receiving module is used to receive the target task issued by the central processing unit;

[0044] The scheduling module is used to schedule the fusion operator to execute the target task.

[0045] Fourthly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the optimization method of the distributed operator of any of the above.

[0046] Fifthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements an optimization method for the distributed operator of any of the above.

[0047] Sixthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements an optimization method for the distributed operator of any of the above.

[0048] The aforementioned optimization method for distributed operators, artificial intelligence chips, computer equipment, readable storage media, and program products are applied to artificial intelligence chips. Each stream processor cluster of the artificial intelligence chip has a global address mapping unit, which records the shared memory address information of the stream processor clusters of the artificial intelligence chips involved in the distributed operator. When the target task is started, during the calculation of the current data, the stream processor clusters of the artificial intelligence chip read the operation data from the global memory into the shared memory to perform the corresponding matrix multiplication calculation. The result of the matrix multiplication calculation is stored in the shared memory of the stream processor cluster, and the operation data is the data block corresponding to the current data. After the matrix multiplication calculation is completed, a communication process is triggered. The communication process includes the global address mapping unit of the stream processor cluster retrieving the matrix multiplication calculation result from the shared memory of each target stream processor cluster based on the locally recorded shared memory address information of each target stream processor cluster, and writing the retrieved result into the shared memory of the stream processor cluster. Finally, based on the matrix multiplication calculation results stored in the shared memory of the stream processor cluster, the matrix multiplication calculation of the next calculation process is executed. The distributed operator optimization method, artificial intelligence chip, computer device, readable storage medium, and program product provided in this application embodiment enable direct access to shared memory between different artificial intelligence chips by setting a global address mapping unit within the streaming processor cluster to record the shared memory address information of all artificial intelligence chips in the cluster. In this way, when optimizing the Tiling (matrix block computation technique) of the distributed operator, after the divided data blocks are computed on a single streaming processor cluster, the streaming processor clusters of different artificial intelligence chips can immediately trigger communication operations, directly accessing the shared memory between different artificial intelligence chips through the global address mapping unit to directly obtain the computation results of each data block. Therefore, the overall execution efficiency of the distributed operator can be significantly improved, meeting the high throughput and low latency requirements of large model training. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating an optimization method for a distributed operator in one embodiment;

[0051] Figure 2 This is a schematic diagram of the GPGPU structure in one embodiment;

[0052] Figure 3 This is a flowchart illustrating step 104 in one embodiment;

[0053] Figure 4 This is a flowchart illustrating step 104 in another embodiment;

[0054] Figure 5 This is a flowchart illustrating the optimization method for distributed operators in another embodiment;

[0055] Figure 6 This is a schematic diagram of the structure of an artificial intelligence chip in one embodiment;

[0056] Figure 7 This is a schematic diagram of the SPC structure in one embodiment;

[0057] Figure 8 This is a schematic diagram illustrating the fusion of matrix multiplication operators and communication operators in one embodiment;

[0058] Figure 9 This is a schematic diagram of the solution before optimization in one embodiment;

[0059] Figure 10 This is a schematic diagram of the SPC structure before optimization in one embodiment;

[0060] Figure 11 This is a schematic diagram of the optimized solution in one embodiment;

[0061] Figure 12 This is a structural block diagram of the optimization device for a distributed operator in one embodiment;

[0062] Figure 13 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0065] like Figure 1 As shown, an optimization method for distributed operators is provided and applied to artificial intelligence chips. In this embodiment, the artificial intelligence chip is any one of GPU (Graphics Processing Unit), TPU (Tensor Processing Unit), NPU (Neural Network Processing Unit), DPU (Deep Learning Processing Unit), APU (Accelerated Processing Unit), and GPGPU (General-Purpose Graphics Processing Unit). This embodiment does not specifically limit the specific type of chip, and the following description uses GPGPU as an example.

[0066] Reference Figure 2 The diagram shows a schematic of a GPGPU. A GPGPU is actually an array of Streaming Processor Clusters (SPCs), including, for example,... Figure 2 The diagram shows streaming processor clusters 1, ..., M, where M is a positive integer greater than 1. In a graphics processing unit (GPU), one streaming processor cluster processes one computational task, or multiple streaming processor clusters process one computational task. Multiple streaming processor clusters share data through a global cache or global memory.

[0067] like Figure 2 As shown, taking streaming processor cluster 1 as an example, one streaming processor cluster includes multiple computing units, such as... Figure 2 The system is structured as Computation Unit 1, Computation Unit 2, ..., Computation Unit N, where N is a positive integer. Each Computation Unit (CU) performs arithmetic and logical operations other than matrix calculations such as matrix multiplication and convolution, including operations like accumulation, reduction, and standard addition, subtraction, multiplication, and division. A Computation Unit contains multiple cores (also called computational kernels), each including an Arithmetic Logic Unit (ALU), a floating-point unit, etc., which are used to execute specific computational tasks. Furthermore, the Computation Unit also includes registers (e.g., ...). Figure 2 The register file and shared cache in a computing unit are used to store source and destination data related to computing tasks in a hierarchical manner. The shared cache in a computing unit is used to share data between the cores of that computing unit.

[0068] In parallel computing, computational tasks are typically executed by multiple threads. These threads are divided into multiple thread blocks before execution in a general-purpose graphics processor (or parallel computing processor), and then dispatched via a thread block distribution module. Figure 2 (Not shown in the image) Multiple thread blocks are distributed to various computation units. All threads in a thread block must be assigned to the same computation unit for execution. Simultaneously, thread blocks are broken down into minimum execution thread bundles (or simply warps), each containing a fixed number (or less than this fixed number) of threads, for example, 32 threads. Multiple thread blocks can execute in the same computation unit or in different computation units.

[0069] In each computing unit, the thread beam scheduling / distribution module ( Figure 2 (Not shown in the diagram) Thread bundles are scheduled and allocated so that multiple computing cores within the computing unit can run thread bundles. Depending on the number of computing cores in the computing unit, multiple thread bundles within a thread block can be executed concurrently or in a time-sharing manner. Multiple threads within each thread bundle execute the same instructions. Memory execution instructions are issued to the shared cache within the computing unit or further issued to intermediate-level caches, global caches, or global memory for read and write operations, etc.

[0070] like Figure 2 As shown, the streaming processor cluster 1 also includes a tensor operation unit, which is used to perform tensor calculations, such as matrix multiplication, convolution operations, etc.

[0071] Reference Figure 1 As shown, this application provides an optimization method for distributed operators, applied to artificial intelligence chips. Each streaming processor cluster of the artificial intelligence chip has a global address mapping unit, which records the shared memory address information of the streaming processor clusters of the artificial intelligence chip involved in the distributed operator. The method may include the following steps, wherein:

[0072] Step 102: After starting the target task, during the calculation of the current data, the streaming processor cluster of the artificial intelligence chip reads the operation data from the global memory into the shared memory to perform the corresponding matrix multiplication calculation. The result of the matrix multiplication calculation is stored in the shared memory of the streaming processor cluster. The operation data is the data block corresponding to the current data.

[0073] In this embodiment, after starting the target task (such as the forward computation task of the Transformer layer), during the computation process for the current data, the SPC (Streaming Processor Cluster, hereinafter referred to as SPC) of the AI ​​chip first reads the data blocks, i.e., the operation data, corresponding to the current data from the global memory. This operation data is the result of splitting the original large tensor (i.e., the current data) based on the Tiling strategy. The splitting is based on the upper limit of the SPC's shared memory capacity and the computational parallelism requirements, ensuring that the block data can be completely loaded into the SPC's shared memory and efficiently executed. After the SPC stores the read operation data in its local shared memory, it starts matrix multiplication calculation, and the result of the matrix multiplication calculation is directly stored in the SPC's shared memory.

[0074] It should be noted that the embodiments of this application do not limit the specific size and splitting dimension of the data blocks. They can be dynamically adjusted according to the SPC shared memory capacity, the calculation dimension of matrix multiplication, or the scale of the hardware cluster. As long as efficient local calculation of block data and subsequent cross-device aggregation can be achieved, this method is applicable.

[0075] Step 104: After the matrix multiplication calculation is completed, a communication process is triggered. The communication process includes the global address mapping unit of the streaming processor cluster obtaining the matrix multiplication calculation result from the shared memory of each target streaming processor cluster based on the shared memory address information of each target streaming processor cluster recorded locally, and writing the obtained result into the shared memory of the streaming processor cluster.

[0076] In this embodiment, the communication process may include an allgather communication process, a reduce-scatter communication process, etc. This embodiment does not specifically limit the type of communication process. The communication process can be automatically triggered after the local matrix multiplication calculation is completed. The execution entity of the communication process is the global address mapping unit (also called GSRAM (Global SRAM Address Mapper)) within the SPC. The global address mapping unit pre-stores the shared memory address information of all AI chip SPCs involved in the distributed operator, forming a cluster global shared memory address table.

[0077] During the communication process, the global address mapping unit first determines the shared memory address information of other AI chip SPCs (i.e. target SPCs) participating in the same distributed operator computing task from the locally pre-stored global shared memory address table, and directly accesses the shared memory of each target SPC through a cross-device direct connection communication channel (such as PCIe (Peripheral Component Interconnect Express) channel) to read the matrix multiplication calculation results stored in the target SPC (i.e., the calculation results corresponding to the data blocks written by the target SPC).

[0078] During data reading, the distributed memory consistency protocol built into the global address mapping unit ensures the atomicity and order of data access. For example, it ensures that reading is only initiated after the target SPC's calculation result has been written, avoiding the reading of incompletely written data. Simultaneously, a hardware signal handshake mechanism prevents data conflicts caused by concurrent access from multiple devices.

[0079] The global address mapping unit writes the calculation results of the data blocks read from each target SPC into the shared memory of the local SPC, and concatenates them with the calculation results of the local data blocks to form a complete calculation result. The entire communication process does not require transit through global memory, completely eliminating the two data copy operations of shared memory to global memory and global memory to shared memory in traditional solutions.

[0080] Step 106: Based on the results of each matrix multiplication calculation stored in the shared memory of the streaming processor cluster, perform the matrix multiplication calculation for the next calculation process.

[0081] In this embodiment, after communication is completed, the complete calculation result is stored in the shared memory of the local SPC. Since the calculation result is directly stored in the high-speed shared memory, the SPC does not need to read data from global memory and can immediately perform matrix multiplication calculations for the next calculation process based on the result, such as the second GEMM (General Matrix Multiplication) calculation of the Feed-Forward module of the Transformer layer or the matrix multiplication calculation of the next layer network.

[0082] In the aforementioned optimization method for distributed operators, each streaming processor cluster (SPC) of the AI ​​chip is equipped with a global address mapping unit. This unit records the shared memory address information of the SPCs of each AI chip involved in the distributed operator. When a target task is initiated, during the computation of the current data, the AI ​​chip's SPC reads the operation data from global memory into shared memory to perform the corresponding matrix multiplication calculation. The result of the matrix multiplication calculation is stored in the SPC's shared memory, and the operation data is the data block corresponding to the current data. After the matrix multiplication calculation is completed, a communication process is triggered. This process includes the SPC's global address mapping unit retrieving the matrix multiplication results from the shared memory of each target SPC based on the locally recorded shared memory address information, and writing the retrieved results into the SPC's shared memory. Finally, based on the matrix multiplication results stored in the SPC's shared memory, the next matrix multiplication calculation is executed. The optimization method for distributed operators provided in this application's embodiments supports direct access to shared memory between different AI chips by setting up a global address mapping unit within the SPC that records the shared memory address information of all AI chips in the cluster. In this way, when optimizing the Tiling (matrix block computation technique) of the distributed operator, after the divided data blocks are computed on a single SPC, the SPCs of different AI chips can immediately trigger communication operations, directly accessing the shared memory between different AI chips through the global address mapping unit to directly obtain the computation results of each data block. Therefore, the overall execution efficiency of the distributed operator can be significantly improved, meeting the high throughput and low latency requirements of large model training.

[0083] In one exemplary embodiment, refer to Figure 3 As shown, in step 104, based on the shared memory address information of each target streaming processor cluster recorded locally, the result of the matrix multiplication calculation is obtained from the shared memory of each target streaming processor cluster. This may include the following steps:

[0084] Step 302: Determine the shared memory address information of each target streaming processor cluster from the shared memory address information of each AI chip streaming processor cluster recorded locally. The target streaming processor cluster is the streaming processor cluster of other AI chips participating in the same distributed operator computing task.

[0085] Step 304: Access the shared memory of each target streaming processor cluster based on the shared memory address information of each target streaming processor cluster, and read the matrix multiplication calculation results from the shared memory of each target streaming processor cluster.

[0086] In this embodiment, the global address mapping unit of the local SPC pre-stores the shared memory address information of all AI chip SPCs participating in the task in the entire distributed cluster, forming a cluster global shared memory address table. After the local matrix multiplication calculation is completed, the global address mapping unit receives a communication trigger signal and then filters out the target SPC shared memory address information participating in the same distributed operator calculation task from the global shared memory address table.

[0087] For example, the global address mapping unit can generate the physical access address of each target SPC based on the shared memory address information of each target SPC, and send a data read request to each target SPC through a cross-device direct communication channel. The request may include the physical access address, data block identifier, and read length. Since the target SPC and the local SPC belong to the same distributed task, and the result of matrix multiplication calculation has been retained in the shared memory, the global address mapping unit of the target SPC can directly locate the data in the shared memory based on the physical access address without CPU intervention after receiving the request.

[0088] During data reading, the distributed memory consistency protocol built into the global address mapping unit ensures the accuracy of data access. The global address mapping unit of the target SPC first verifies the status of the local data, such as whether it has been calculated or is being accessed by other devices, to ensure that the data is in a readable state. If the data has not been calculated, the global address mapping unit of the target SPC sends a wait signal to the local SPC, and responds to the read request only after the calculation is completed. Finally, the global address mapping unit of the local SPC reads the matrix multiplication calculation results from the shared memory of each target SPC through the cross-device direct connection channel, and writes these read results into the shared memory of the local SPC, concatenating them with the local matrix multiplication calculation results to form a complete calculation result, preparing for the next stage of matrix multiplication calculation.

[0089] In one exemplary embodiment, refer to Figure 4 As shown, in step 104, based on the shared memory address information of each target streaming processor cluster recorded locally, the result of the matrix multiplication calculation is obtained from the shared memory of each target streaming processor cluster. This may include the following steps:

[0090] Step 402: Determine the shared memory address information of each target streaming processor cluster from the shared memory address information of each AI chip's streaming processor cluster recorded locally. The target streaming processor cluster is the streaming processor cluster of other AI chips participating in the same distributed operator computing task.

[0091] Step 404: Send the locally stored matrix multiplication calculation results to each target streaming processor cluster based on the shared memory address information of each target streaming processor cluster, and receive the matrix multiplication calculation results sent by each target streaming processor cluster.

[0092] In this embodiment of the application, the local SPC can also actively push the calculation results of the local SPC to the target SPC through the global address mapping unit, and at the same time receive the calculation results sent by each target SPC, and complete the full aggregation of calculation results through bidirectional data interaction.

[0093] After filtering each target SPC, the global address mapping unit generates corresponding send and receive addresses based on the shared memory address information of the target SPCs. The send address is the physical address where the local computation result is written to the shared memory of the target SPC, and the receive address is the physical address where the local SPC receives the computation result sent by the target SPC. After generating the send address, the global address mapping unit simultaneously initiates data write requests to each target SPC through the cross-device direct communication channel, directly pushing the computation result in the local SPC's shared memory to the corresponding send address of each target SPC via the direct communication channel. This process does not require passing through the global memory.

[0094] After receiving a data write request, the global address mapping unit of the target SPC writes the received data into its local shared memory. Simultaneously, while the local SPC sends data to the target SPC, the global address mapping unit synchronously listens for data write requests from the target SPC on the cross-device direct connection channel. When the target SPC sends a calculation result to the local SPC, the local global address mapping unit receives the data write request, locates the pre-generated receiving address, and writes the calculation result sent by the target SPC into that receiving address. After the local SPC completes sending its own calculation result and receiving calculation results from all the target SPCs, it concatenates and integrates the local calculation result with the calculation results sent by all SPCs to form a complete calculation result, which is then written into the local SPC's shared memory, providing complete input for the next stage of matrix multiplication calculation.

[0095] In an exemplary embodiment, the shared memory address information includes the serial number identifier of the streaming processor cluster and a base shared memory address. The serial number identifier of the streaming processor cluster includes the identifier of the AI ​​chip to which the streaming processor cluster belongs and the identifier of the streaming processor cluster. The base shared memory address in the shared memory address information of the streaming processor cluster of each AI chip is consistent.

[0096] In this embodiment, the shared memory address information is the basis for realizing direct access to shared memory across devices' SPCs. By combining the SPC's identification number with the base shared memory address, it can be ensured that the global address mapping unit can accurately locate the shared memory location of the target SPC, while simplifying the address translation logic and avoiding data access conflicts.

[0097] The SPC identifier is a globally unique identifier used to distinguish different SPCs within a cluster. It's a hierarchical identifier that avoids conflicts caused by single-level identifiers (e.g., SPCs on different chips might have the same local identifier). The SPC identifier can consist of two levels. The first level is the identifier of the AI ​​chip to which the SPC belongs, used to uniquely distinguish different AI chips in the cluster. This can use globally unique encoding, such as a numerical number or physical address segment. For example, multiple chips can be identified as Chip0, Chip1, Chip2, etc. The second level is the SPC identifier, used to distinguish multiple SPCs on the same chip. This uses a unique encoding within the chip. For example, a single SPC chip can be identified as SPC0, and multiple SPC chips can be identified as SPC0, SPC1, SPC2, etc. Ultimately, the SPC identifier is formed by combining the chip identifier and the SPC identifier, in formats such as Chip0-SPC0, Chip1-SPC0, Chip2-SPC0, etc.

[0098] The baseline shared memory address serves as a unified coordinate for cross-device access. All SPCs share the same baseline shared memory address, such as 0x7F00_0000. This address refers to the starting logical address of the shared memory of each AI chip's SPC, used to unify the address naming rules for all SPCs in the cluster. It does not mean that all SPCs have identical physical memory starting addresses. Maintaining a consistent baseline shared memory address simplifies the hardware-level address translation logic of the global address mapping unit. It eliminates the need for complex cross-device address mapping table lookups; it only requires hardware-level calculation of the baseline address and the offset of the identifier.

[0099] For example, when a local SPC (identified as Chip0-SPC0) needs to access the shared memory of a target SPC (identified as Chip1-SPC0), the global address mapping unit automatically retrieves the preset chip offset 0x0000_1000 based on the target SPC's chip identifier Chip1-SPC0, and translates it to generate a physical address of 0x7F00_0000+0x0000_1000=0x7F00_1000. When accessing a target SPC (identified as Chip2-SPC0), the offset is 0x0000_2000, and the physical address is 0x7F00_0000+0x0000_2000=0x7F00_2000.

[0100] Meanwhile, maintaining a consistent base shared memory address resolves data access conflicts when multiple senders are involved. For example, in active sending mode, multiple SPCs need to send data to the same target SPC simultaneously. Because the base address is consistent, the target SPC's global address mapping unit can allocate a dedicated storage partition for each sender based on their identifier, avoiding data overwriting. For instance, if the base address of the target SPC (identified as Chip0-SPC0) is 0x7F00_0000, when receiving data from Chip1-SPC0, the allocated partition address is 0x7F00_0000 + 0x0000_1000 (sender offset). When receiving data from Chip2-SPC0, the allocated partition address is 0x7F00_0000 + 0x0000_2000. The write addresses of each sender do not overlap, eliminating the risk of overwriting at the hardware level.

[0101] In one exemplary embodiment, refer to Figure 5 As shown, the distributed operator includes a fusion operator, which is used to perform matrix multiplication calculations and inter-chip communication. The method may also include the following steps, wherein:

[0102] Step 502: Receive the target task issued by the central processing unit;

[0103] Step 504: Schedule the fusion operator to execute the target task.

[0104] In this embodiment, the distributed operator includes a fusion operator, which is an operator obtained by fusing matrix multiplication operators and communication operators for inter-chip communication. For example, the fusion operator may include a fusion operator of the GEMM operator and the Allgather communication operator, or a fusion operator of the GEMM operator and the Reduce_Scatter communication operator, or a fusion operator of other matrix operations and other communication operators. This embodiment does not specifically limit the fusion operator. That is, the fusion operator deeply integrates matrix multiplication calculation logic and inter-chip communication logic, avoiding the multiple scheduling overhead of matrix multiplication calculation operators and independent communication operators in traditional solutions, and simplifying the interaction process between the CPU and the artificial intelligence chip.

[0105] For example, the CPU, acting as the cluster's global controller, first broadcasts information including the core task instructions, distributed parallel configuration, Tiling block parameters, and weights required by the fusion operator to all participating AI chips. After receiving this information, each chip control unit verifies the task's ownership and parameter integrity, sends a confirmation signal back to the CPU, and synchronizes the configuration to the global address mapping unit and fusion operator parameter buffer of the local SPC. After the input data is copied from the CPU's main memory to the AI ​​chip's global memory and the weights are loaded into the SPC's shared memory, the chip control unit sends a single fusion operator start command to the SPC, triggering the fusion operator to autonomously execute the target task (such as calculation by the Transformer layer's Feed-Forward module). The fusion operator then reads data blocks from the global memory into the shared memory according to preset logic and performs the first GEMM calculation. Finally, it triggers the global address mapping unit via hardware signals to perform cross-device shared memory direct access to aggregate the complete calculation results.

[0106] Finally, based on the complete calculation results, a second GEMM calculation is performed and written to global memory. All Tiling blocks are processed in a loop. No additional CPU intervention is required throughout the process. The CPU only sends a completion signal to the CPU after all tasks are completed. Through the CPU's one-time dispatch and single scheduling of fusion operators and autonomous collaborative calculation, the latency caused by the CPU's multiple scheduling of operators and frequent interactions in traditional schemes is completely eliminated. At the same time, relying on the hardware-level communication capabilities of the global address mapping unit, seamless connection and pipeline parallelism of calculation and communication are achieved, which greatly improves the efficiency of distributed training.

[0107] In one exemplary embodiment, refer to Figure 6 As shown, this application also provides an artificial intelligence chip, including at least one streaming processor cluster and global memory, wherein, referring to Figure 7 As shown, the streaming processor cluster includes computing units, shared memory, and a global address mapping unit. The global address mapping unit records the shared memory address information of the streaming processor cluster of each AI chip involved in the distributed operator, wherein:

[0108] The streaming processor cluster is used to read operation data from global memory and write the read operation data into shared memory. The operation data is a data block corresponding to the operation data of the currently executed target task.

[0109] The computing unit is used to perform matrix multiplication calculations based on the operation data written to shared memory, and to store the results of the matrix multiplication calculations in shared memory;

[0110] The global address mapping unit is used to trigger the communication process after the computing unit completes the matrix multiplication calculation. The communication process includes the global address mapping unit of the streaming processor cluster obtaining the matrix multiplication calculation result from the shared memory of each target streaming processor cluster based on the shared memory address information of each target streaming processor cluster recorded locally, and writing the obtained result into the shared memory of the streaming processor cluster.

[0111] The computing unit is also used to perform matrix multiplication calculations for the next computation process based on the matrix multiplication results stored in the shared memory of the streaming processor cluster.

[0112] In one exemplary embodiment, obtaining the matrix multiplication result from the shared memory of each target streaming processor cluster based on the locally recorded shared memory address information of each target streaming processor cluster may include:

[0113] From the shared memory address information of the streaming processor clusters of each AI chip recorded locally, determine the shared memory address information of each target streaming processor cluster. The target streaming processor clusters are the streaming processor clusters of other AI chips participating in the same distributed operator computing task. Based on the shared memory address information of each target streaming processor cluster, access the shared memory of each target streaming processor cluster and read the matrix multiplication calculation results from the shared memory of each target streaming processor cluster.

[0114] In one exemplary embodiment, based on the locally recorded shared memory address information of each target streaming processor cluster, the result of matrix multiplication calculation is obtained from the shared memory of each target streaming processor cluster, including:

[0115] From the shared memory address information of the streaming processor clusters of each AI chip recorded locally, determine the shared memory address information of each target streaming processor cluster. The target streaming processor clusters are the streaming processor clusters of other AI chips participating in the same distributed operator computing task. Based on the shared memory address information of each target streaming processor cluster, send the locally stored matrix multiplication calculation results to each target streaming processor cluster, and receive the matrix multiplication calculation results sent by each target streaming processor cluster.

[0116] In this embodiment, the artificial intelligence chip is a hardware carrier specifically designed for distributed operator fusion scenarios. It constructs an integrated computing and communication hardware architecture through a global address mapping unit built into a streaming processor cluster (SPC) to achieve direct access to shared memory across devices, thus solving the communication latency caused by the two data copies between shared memory and global memory in traditional chips. (Refer to...) Figure 6As shown, an artificial intelligence chip mainly includes at least one streaming processor cluster (SPC) and global memory. Global memory is a chip-level storage unit primarily used to store input data for the target task, model weights, and final calculation results.

[0117] Reference Figure 7 As shown, a Streaming Processor Cluster (SPC) comprises compute units, shared memory, and a global address mapping unit. It serves as the computation and communication unit of an AI chip, and multiple SPCs can be configured according to the chip's computing power requirements. Each SPC independently performs matrix multiplication calculations of block data and cross-device data aggregation, with all SPCs having identical hardware structures, supporting flexible parallel strategies. The compute unit is the core of the SPC's computing power, consisting of multiple arithmetic logic units and floating-point units. It is specifically optimized for parallel efficiency in matrix multiplication calculations, primarily responsible for reading operation data and corresponding weighted slices from shared memory, performing matrix multiplication calculations, and writing the calculation results to the shared memory's result temporary storage area.

[0118] Simultaneously, the computing unit can also be used to execute the matrix multiplication calculations of the next calculation process based on the matrix multiplication results stored in the shared memory of the SPC. The global address mapping unit is hardware that integrates address management, communication control, and consistency guarantee. The pre-stored shared memory address information is a cluster global shared memory address table. It is mainly responsible for address information management, triggering communication processes, hardware-level address translation, cross-device direct communication, and consistency guarantee. The communication process includes the SPC's global address mapping unit obtaining the matrix multiplication calculation results from the shared memory of each target SPC based on the locally recorded shared memory address information of each target SPC, and writing the obtained results into the shared memory of the SPC. The specific process can be referred to the relevant description in the foregoing embodiments, and will not be repeated here in the embodiments of this application.

[0119] The artificial intelligence chip provided in this application supports direct access to shared memory between different artificial intelligence chips by setting a global address mapping unit inside the SPC that records the shared memory address information of all artificial intelligence chips in the cluster. In this way, when optimizing the Tiling (matrix block computation technique) of distributed operators, after the divided data blocks are computed on a single SPC, the SPCs of different artificial intelligence chips can immediately trigger communication operations, directly accessing the shared memory between different artificial intelligence chips through the global address mapping unit to directly obtain the computation results of each data block. Therefore, the overall execution efficiency of distributed operators can be significantly improved, meeting the high throughput and low latency requirements of large model training.

[0120] To enable those skilled in the art to better understand the embodiments of this application, the embodiments of this application are described below through specific examples.

[0121] Before the application of distributed operator fusion technology, in large model training, computational operators and communication operators were generally separate, requiring tasks to be distributed via the CPU. For example, when executing a DP2TP4 distributed strategy on a single 8-chip machine, the tensors involved in matrix operations in the model would be divided into equal parts according to the TP size (tensor parallel size), with each part computed on its respective chip. Then, an allgather communication operator would be executed to collect the results from all chips. Currently, the conventional optimization method is to divide large tensors and implement an overlap strategy for computational and communication operators, referring to... Figure 8 As shown, since the computation after GEMM computation is equivalent to the overall computation, the large tensor is divided into blocks. Once the communication operator collects a complete small block, the general matrix multiplication (GEMM) operation is immediately performed. In this way, computation and communication are overlapped, thereby improving efficiency.

[0122] Reference Figure 9 As shown, during the execution of the Allgather communication operator in this optimization strategy, the data results after being divided and executed on SPC need to be copied from shared memory to global memory. When continuing to execute the next GEMM operator, the results obtained from Allgather need to be copied from global memory to shared memory again, resulting in two additional copies. Furthermore, the copying process involves the slower global memory, leading to inefficiency. (Refer to...) Figure 10 As shown, in the current optimization strategy, SPC mainly includes computing units and shared memory.

[0123] In this embodiment of the application, in view of the above-mentioned problems, refer to Figure 7 As shown, a global address mapping unit is added to the SPC, which can record the shared memory address information of all SPCs globally. Based on the global address mapping unit, the shared memory on the SPC chips of different chips can be directly accessed. (Refer to...) Figure 11 As shown, the CPU initiates the fusion operator and enables the Tiling strategy. Large tensors are divided into blocks of data suitable for SPC computation. When Allgather communication is executed, when the data collection of a block is completed, the computation result of the previous operator is still in the shared memory of the SPC. The data of SPCs on different chips can be directly communicated through the global address mapping unit to obtain the complete block computation result. Finally, based on the complete block computation data of the SPC shared memory on the card, the next GEMM operator is executed, and the above steps are repeated to complete the training of the entire model.

[0124] By adding a global address mapping unit to the existing GPU architecture to record the address information of the shared memory of all SPC computing units in the cluster, direct access to shared memory between different chips is supported. When implementing the Tiling optimization of the distributed fusion operator, the small data blocks that are divided can be computed entirely on a single SPC. When the SPC computation is complete, the SPCs between different chips can immediately trigger communication operations, enabling direct access to shared memory between different cards through the global address mapping unit. This mechanism reduces two copies: one is the copy operation from the shared memory on the SPC to the global memory of the source AI chip (e.g., when accessing data between devices, the computation result needs to be copied from shared memory to global memory); the other is the copy from the global memory of the target AI chip to the shared memory of the SPC on the target AI chip (e.g., when accessing data between devices, the computation result of the SPC on the source AI chip is copied from global memory to the shared memory of the SPC on the target AI chip).

[0125] In other words, this embodiment only requires two data copies: one from the global memory of the source AI chip to the SPC shared memory on the source AI chip; and the other from the global memory of the target AI chip to the SPC shared memory on the target AI chip. Furthermore, since global memory is no longer used for communication within the fusion operator, only high-speed shared memory resources are used, improving communication performance and ultimately significantly enhancing the overall operator performance.

[0126] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0127] Based on the same inventive concept, this application also provides an optimization apparatus for a distributed operator to implement the optimization method for the distributed operator described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more embodiments of the distributed operator optimization apparatus provided below can be found in the limitations of the distributed operator optimization method described above, and will not be repeated here.

[0128] In one exemplary embodiment, such as Figure 12 As shown, an optimization device 1200 for distributed operators is provided, applied to an artificial intelligence chip. Each stream processor cluster of the artificial intelligence chip is equipped with a global address mapping unit, which records the shared memory address information of the stream processor clusters of the artificial intelligence chip involved in the distributed operator. The device includes a first computing module 1202, a communication module 1204, and a second computing module 1206, wherein:

[0129] The first computing module 1202 is used to, after starting the target task, read operation data from global memory into shared memory to perform corresponding matrix multiplication calculations during the calculation process for the current data in the AI ​​chip's streaming processor cluster. The result of the matrix multiplication calculation is stored in the shared memory of the streaming processor cluster. The operation data is the data block corresponding to the current data.

[0130] The communication module 1204 is used to trigger a communication process after the matrix multiplication calculation is completed. The communication process includes the global address mapping unit of the streaming processor cluster obtaining the matrix multiplication calculation result from the shared memory of each target streaming processor cluster based on the shared memory address information of each target streaming processor cluster recorded locally, and writing the obtained result into the shared memory of the streaming processor cluster.

[0131] The second calculation module 1206 is used to perform matrix multiplication calculations in the next calculation process based on the matrix multiplication results stored in the shared memory of the streaming processor cluster.

[0132] In the aforementioned optimization device for the distributed operator, each streaming processor cluster (SPC) of the artificial intelligence chip is equipped with a global address mapping unit. This unit records the shared memory address information of the SPCs of each artificial intelligence chip involved in the distributed operator. When a target task is initiated, during the computation of the current data, the SPCs of the artificial intelligence chip read operation data from global memory into shared memory to perform corresponding matrix multiplication calculations. The results of the matrix multiplication calculations are stored in the shared memory of the SPCs, and the operation data consists of the data blocks corresponding to the current data. After the matrix multiplication calculations are completed, a communication process is triggered. This process includes the global address mapping unit of the SPC retrieving the matrix multiplication results from the shared memory of each target SPC based on the locally recorded shared memory address information, and writing the retrieved results into the shared memory of the SPCs. Finally, based on the matrix multiplication results stored in the shared memory of the SPCs, the next matrix multiplication calculation is performed. The optimization device for distributed operators provided in this application supports direct access to shared memory between different AI chips by setting a global address mapping unit within the SPC that records the shared memory address information of all AI chips in the cluster. In this way, when optimizing the Tiling (matrix block computation technique) of distributed operators, after the divided data blocks are computed on a single SPC, the SPCs of different AI chips can immediately trigger communication operations, directly accessing the shared memory between different AI chips through the global address mapping unit to directly obtain the computation results of each data block. Therefore, the overall execution efficiency of distributed operators can be significantly improved, meeting the high throughput and low latency requirements of large model training.

[0133] In an exemplary embodiment, obtaining the matrix multiplication result from the shared memory of each target streaming processor cluster based on the locally recorded shared memory address information of each target streaming processor cluster includes:

[0134] From the shared memory address information of the streaming processor clusters of each artificial intelligence chip recorded locally, the shared memory address information of each target streaming processor cluster is determined. The target streaming processor cluster is the streaming processor cluster of other artificial intelligence chips participating in the same distributed operator computing task.

[0135] Access the shared memory of each target streaming processor cluster based on the shared memory address information of each target streaming processor cluster, and read the matrix multiplication calculation results from the shared memory of each target streaming processor cluster.

[0136] In an exemplary embodiment, obtaining the matrix multiplication result from the shared memory of each target streaming processor cluster based on the locally recorded shared memory address information of each target streaming processor cluster includes:

[0137] From the shared memory address information of the streaming processor clusters of each artificial intelligence chip recorded locally, the shared memory address information of each target streaming processor cluster is determined. The target streaming processor cluster is the streaming processor cluster of other artificial intelligence chips participating in the same distributed operator computing task.

[0138] Based on the shared memory address information of each target streaming processor cluster, the results of matrix multiplication calculations stored locally are sent to each target streaming processor cluster, and the results of matrix multiplication calculations sent by each target streaming processor cluster are received.

[0139] In an exemplary embodiment, the shared memory address information includes a streaming processor cluster identifier and a base shared memory address. The streaming processor cluster identifier includes the identifier of the AI ​​chip to which the streaming processor cluster belongs and the identifier of the streaming processor cluster itself. The base shared memory address in the shared memory address information of the streaming processor clusters of each AI chip is consistent.

[0140] In one exemplary embodiment, the distributed operator includes a fusion operator for performing matrix multiplication calculations and inter-chip communication, and the apparatus further includes:

[0141] A receiving module is used to receive the target task issued by the central processing unit;

[0142] The scheduling module is used to schedule the fusion operator to execute the target task.

[0143] Each module in the aforementioned distributed operator optimization device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0144] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 13As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements an optimization method using distributed operators. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0145] Those skilled in the art will understand that Figure 13 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0146] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0147] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0148] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0149] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0150] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0151] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0152] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for optimization of a distributed operator, characterized in that, The method is applied to an artificial intelligence chip, each stream processor cluster of the artificial intelligence chip is internally provided with a global address mapping unit, and shared memory address information of each stream processor cluster of the artificial intelligence chip involved in a distributed operator is recorded in the global address mapping unit, and the method comprises the following steps: After starting the target task, in a calculation process for current data, the stream processor cluster of the artificial intelligence chip reads operation data from the global memory into the shared memory to perform corresponding matrix multiplication calculation, and the result of the matrix multiplication calculation is stored in the shared memory of the stream processor cluster; the operation data is a data block corresponding to the current data; After the matrix multiplication calculation is completed, a communication process is triggered, which comprises the following steps: Based on the shared memory address information of each target stream processor cluster recorded locally, the global address mapping unit of the stream processor cluster acquires the result of the matrix multiplication calculation from the shared memory of each target stream processor cluster, and writes the acquired result into the shared memory of the stream processor cluster; 2. The method of claim 1, wherein, Based on the result of each matrix multiplication calculation stored in the shared memory of the stream processor cluster, matrix multiplication calculation of a next calculation process is performed. The method comprises the following steps: From the shared memory address information of each stream processor cluster of the artificial intelligence chip recorded locally, shared memory address information of each target stream processor cluster is determined, the target stream processor cluster being a stream processor cluster of another artificial intelligence chip participating in the same distributed operator calculation task; 3. The method of claim 1, wherein, Based on the shared memory address information of each target stream processor cluster, the shared memory of each target stream processor cluster is accessed, and the result of the matrix multiplication calculation is read from the shared memory of each target stream processor cluster. The method comprises the following steps: From the shared memory address information of each stream processor cluster of the artificial intelligence chip recorded locally, shared memory address information of each target stream processor cluster is determined, the target stream processor cluster being a stream processor cluster of another artificial intelligence chip participating in the same distributed operator calculation task; 4. The method according to any one of claims 1 to 3, characterized in that, Based on the shared memory address information of each target stream processor cluster, the shared memory of each target stream processor cluster is accessed, and the result of the matrix multiplication calculation is read from the shared memory of each target stream processor cluster. The shared memory address information comprises a number identifier of a stream processor cluster and a reference shared memory address, the number identifier of the stream processor cluster comprising an identifier of an artificial intelligence chip to which the stream processor cluster belongs and an identifier of the stream processor cluster, and the reference shared memory addresses in the shared memory address information of each stream processor cluster of the artificial intelligence chip are consistent.

5. The method of claim 1, wherein, The distributed operators include a fusion operator used to perform matrix multiplication calculation and inter-chip communication, and the method further comprises: receiving the target task issued by the central processor; scheduling the fusion operator to perform the target task.

6. An artificial intelligence chip, characterized by, The system comprises at least one stream processor cluster and a global memory, wherein the stream processor cluster comprises a computing unit, a shared memory and a global address mapping unit, and the global address mapping unit records shared memory address information of the stream processor cluster of each artificial intelligence chip involved in a distributed operator; The stream processor cluster is configured to read operation data from the global memory and write the read operation data into the shared memory, wherein the operation data is a data block of operation data corresponding to a target task currently executed; The computing unit is configured to perform matrix multiplication calculation based on the operation data written into the shared memory and store the result of the matrix multiplication calculation in the shared memory; The global address mapping unit is configured to trigger a communication process after the computing unit completes the matrix multiplication calculation, wherein the communication process comprises that the global address mapping unit of the stream processor cluster acquires the result of the matrix multiplication calculation from the shared memory of each target stream processor cluster based on the locally recorded shared memory address information of each target stream processor cluster and writes the acquired result into the shared memory of the stream processor cluster; The computing unit is further configured to perform matrix multiplication calculation of a next calculation process based on the result of each matrix multiplication calculation stored in the shared memory of the stream processor cluster.

7. The chip of claim 6, wherein The acquiring the result of the matrix multiplication calculation from the shared memory of each target stream processor cluster based on the locally recorded shared memory address information of each target stream processor cluster comprises: determining the shared memory address information of each target stream processor cluster from the locally recorded shared memory address information of each stream processor cluster of the artificial intelligence chip, wherein the target stream processor cluster is a stream processor cluster of another artificial intelligence chip participating in the same distributed operator calculation task; accessing the shared memory of each target stream processor cluster based on the shared memory address information of each target stream processor cluster and reading the result of the matrix multiplication calculation from the shared memory of each target stream processor cluster.

8. The chip of claim 6, wherein, The acquiring the result of the matrix multiplication calculation from the shared memory of each target stream processor cluster based on the locally recorded shared memory address information of each target stream processor cluster comprises: determining the shared memory address information of each target stream processor cluster from the locally recorded shared memory address information of each stream processor cluster of the artificial intelligence chip, wherein the target stream processor cluster is a stream processor cluster of another artificial intelligence chip participating in the same distributed operator calculation task; sending the locally stored result of the matrix multiplication calculation to each target stream processor cluster based on the shared memory address information of each target stream processor cluster and receiving the result of the matrix multiplication calculation sent by each target stream processor cluster. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 5.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 5.

11. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 5.

Images