Data processing method, processor, chip, display card and electronic equipment

Abstract

The invention relates to a data processing method, a processor, a chip, a display card and electronic equipment, and relates to the field of intelligent computing, and the method comprises the following steps: reading a part of a first matrix from a local memory or a register of a computing unit in response to M tensor computing engines in a tensor computing engine cluster, reading the second matrix from the tensor memory, and executing matrix operation by the M tensor calculation engines according to part of the first matrix and the second matrix respectively to obtain operation results corresponding to the M tensor calculation engines; and the M tensor calculation engines respectively write operation results corresponding to the M tensor calculation engines into registers of the M tensor calculation engines. According to the embodiment of the invention, the computing power can be increased under the condition that the sizes and the bandwidths of the local memory and the register in the computing unit are not increased.

Description

Technical Field

[0001] This disclosure relates to the field of intelligent computing, and more particularly to a data processing method, processor, chip, graphics card, and electronic device. Background Technology

[0002] With the development of artificial intelligence (AI) and the rapid application of large-scale models in various fields, the demand for hardware computing power and bandwidth is increasing. Enhancing computing power can be achieved by adding more computing cores to run more tasks simultaneously. However, for scenarios requiring improved performance of a single task, this approach is insufficient due to the limited capabilities of a single computing core. Summary of the Invention

[0003] This disclosure presents a data processing method, a processor, a chip, a graphics card, and an electronic device.

[0004] According to one aspect of this disclosure, a data processing method is provided, comprising: the data processing method being applied to a tensor computation engine cluster in a processor, the processor including multiple computation cores, each computation core including multiple computation units, the tensor computation engine cluster being composed of interconnected tensor computation engines of M computation units, each of the M computation units having a local memory or register for storing a portion of a first matrix, and a tensor memory shared by the M computation units for storing a second matrix, where M is an integer greater than 1; the method comprising: in response to the M tensor computation engines in the tensor computation engine cluster completing reading the portion of the first matrix from the local memory or register of their respective computation units, and reading the second matrix from the tensor memory, the M tensor computation engines respectively performing matrix operations based on the portion of the first matrix and the second matrix to obtain their corresponding operation results; the M tensor computation engines respectively writing their corresponding operation results into their own registers.

[0005] In one possible implementation, the processor further includes a task distribution module, and the method further includes: the task distribution module distributing M thread bundles to the M computing units; the M computing units, based on the thread bundles they receive, moving the second matrix from a shared cache to the tensor memory, and moving their respective portions of the first matrix to their local memory; the M computing units respectively sending matrix operation instructions to their internal tensor computing engines, so that the M tensor computing engines read the portions of the first matrix from the local memory of their respective computing units and read the second matrix from the tensor memory according to the matrix operation instructions.

[0006] In one possible implementation, the M tensor computation engines read a portion of the first matrix from the local memory of their respective computation units and read the second matrix from the tensor memory according to the matrix operation instructions, including: in response to each tensor computation engine completing the matrix operation instructions, the tensor computation engine marks a first synchronization signal in a synchronization counter; in response to detecting that the synchronization counter has marked M first synchronization signals, a second synchronization signal is generated, the second synchronization signal indicating that the M tensor computation engines have completed reading the portion of the first matrix from the local memory of their respective computation units and reading the second matrix from the tensor memory.

[0007] In one possible implementation, the M tensor computation engines each write their corresponding computation results into their own registers, including: in response to the M tensor computation engines writing their corresponding computation results into their own registers, clearing the first synchronization signal and the second synchronization signal in the synchronization counter.

[0008] In one possible implementation, the tensor computation engine includes multiple arithmetic logic units, each of which is a multi-element multiply-accumulate unit array; the bandwidth of the register is the same as the number of elements in the first dimension of the partial first matrix and the number of elements in the second dimension of the second matrix.

[0009] In one possible implementation, each tensor computation engine is further configured to read a third matrix from the register of its respective computation unit. The M tensor computation engines perform matrix operations based on the partial first matrix and the second matrix to obtain their respective computation results, including: the M tensor computation engines perform matrix multiplication operations based on their respective partial first matrix and the second matrix to obtain multiple product results; the M tensor computation engines perform matrix sum operations based on their respective product results and the third matrix to obtain multiple computation results.

[0010] In one possible implementation, the M computing units, based on the thread bundles they receive, move the second matrix from the shared peer cache to the tensor memory, and each move its corresponding portion of the first matrix to its respective local memory. This includes: the computing unit distributing the thread groups in the received thread bundles to at least one execution unit; and the execution unit generating an access request based on the thread groups it receives, the access request being used to move the second matrix from the shared peer cache to the tensor memory, and to move the portion of the first matrix to the local memory.

[0011] According to one aspect of this disclosure, a processor is provided, comprising: a processor including a plurality of computing cores, each computing core including a plurality of computing units, wherein tensor computing engines of M computing units are interconnected to form a tensor computing engine cluster, wherein a local memory or register of each of the M computing units is used to store a portion of a first matrix, and a tensor memory shared by the M computing units is used to store a second matrix, wherein M is an integer greater than 1; in response to the M tensor computing engines in the tensor computing engine cluster completing reading the portion of the first matrix from the local memory or register of their respective computing units and reading the second matrix from the tensor memory, the M tensor computing engines are configured to perform matrix operations based on the portion of the first matrix and the second matrix respectively to obtain their corresponding operation results; and the M tensor computing engines are configured to write their corresponding operation results into their own registers respectively.

[0012] In one possible implementation, the processor further includes a task distribution module, which distributes M thread bundles to the M computing units; the M computing units, based on the thread bundles they receive, move the second matrix from a shared cache to the tensor memory, and move their respective portions of the first matrix to their local memory; the M computing units send matrix operation instructions to their internal tensor computing engines, so that the M tensor computing engines read the portions of the first matrix from the local memory of their respective computing units and read the second matrix from the tensor memory according to the matrix operation instructions.

[0013] In one possible implementation, the M tensor computation engines read a portion of the first matrix from the local memory of their respective computation units and read the second matrix from the tensor memory according to the matrix operation instructions, including: in response to each tensor computation engine completing the matrix operation instructions, the tensor computation engine marks a first synchronization signal in a synchronization counter; in response to detecting that the synchronization counter has marked M first synchronization signals, a second synchronization signal is generated, the second synchronization signal indicating that the M tensor computation engines have completed reading the portion of the first matrix from the local memory of their respective computation units and reading the second matrix from the tensor memory.

[0014] In one possible implementation, the M tensor computation engines are specifically used to: in response to the M tensor computation engines writing their corresponding computation results into their own registers, clear the first synchronization signal and the second synchronization signal in the synchronization counter.

[0015] In one possible implementation, the tensor computation engine includes multiple arithmetic logic units, each of which is a multi-element multiply-accumulate unit array; the bandwidth of the register is the same as the number of elements in the first dimension of the partial first matrix and the number of elements in the second dimension of the second matrix.

[0016] In one possible implementation, each tensor computation engine is further configured to read a third matrix from the register of its respective computation unit. The M tensor computation engines are configured to: perform matrix multiplication operations based on their respective partial first matrices and second matrices to obtain multiple product results; and perform matrix sum operations based on their respective product results and the third matrix to obtain multiple operation results.

[0017] In one possible implementation, the M computing units, based on the thread bundles they receive, move the second matrix from the shared peer cache to the tensor memory, and each move its corresponding portion of the first matrix to its respective local memory. This includes: the computing unit distributing the thread groups in the received thread bundles to at least one execution unit; and the execution unit generating an access request based on the thread groups it receives, the access request being used to move the second matrix from the shared peer cache to the tensor memory, and to move the portion of the first matrix to the local memory.

[0018] According to one aspect of this disclosure, a chip is provided that includes the processor described above.

[0019] According to one aspect of this disclosure, a graphics card is provided, the graphics card including the processor described above.

[0020] According to one aspect of this disclosure, an electronic device is provided, the electronic device including the chip described above.

[0021] The data processing method of this disclosure can be applied to a tensor computation engine cluster in a processor. The processor includes multiple computation cores, each of which includes multiple computation units. The tensor computation engine cluster is composed of interconnected tensor computation engines of M computation units. The local memory or register of each of the M computation units is used to store a portion of a first matrix, and the tensor memory shared by the M computation units is used to store a second matrix, where M is an integer greater than 1. The method includes: in response to the M tensor computation engines in the tensor computation engine cluster completing reading the portion of the first matrix from the local memory or register of their respective computation units and reading the second matrix from the tensor memory, the M tensor computation engines respectively perform matrix operations based on the portion of the first matrix and the second matrix to obtain their corresponding operation results; and the M tensor computation engines respectively write their corresponding operation results into their own registers.

[0022] In this way, existing tensor computing engines can be utilized. Without changing the structure of the tensor computing engines, tensor computing engines of multiple computing units can be interconnected to form a tensor computing engine cluster. Multiple tensor computing engines can be used to perform calculations simultaneously, making full use of the parallel computing capabilities of the tensor computing engine cluster. Each tensor computing engine handles a part of the matrix operations, realizing the decomposition and parallelization of the computing task, which significantly improves the computing efficiency. Furthermore, by adding a shared tensor memory to multiple computing units, each computing unit can reuse the second matrix stored in the tensor memory, reducing the bandwidth requirements. Since each computing unit's own local memory or register does not need to store the second matrix, it can support storing larger-sized blocks of data (such as parts of the first matrix), which is beneficial to increase computing power without increasing the size and bandwidth of the local memory and registers inside the computing unit.

[0023] Furthermore, the data processing method of this disclosure embodiment can also adjust the number of tensor computing engines in the tensor computing engine cluster according to different task loads, reducing resource idleness or overload, and making it suitable for more scenarios. For example, for scenarios that require performance improvement of a single task, the tensor computing engine cluster is composed of interconnected tensor computing engines of more or even all computing units to provide sufficient computing power. For application scenarios without enough tasks, the tensor computing engine cluster is composed of interconnected tensor computing engines of fewer computing units, so that other idle computing units can perform other tasks, reducing the waste of hardware resources.

[0024] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0026] Figure 1 A schematic diagram of the structure of a multilayer sensor in related technologies is shown.

[0027] Figure 2 A schematic diagram of a fully connected layer in a related technology is shown.

[0028] Figure 3 A schematic diagram of a processor in the related technology is shown.

[0029] Figure 4 A schematic diagram of the matrix multiplication calculation process in related technologies is shown.

[0030] Figure 5 A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown.

[0031] Figure 6 A schematic diagram of the structure of a tensor computation engine according to an embodiment of the present disclosure is shown.

[0032] Figure 7 A schematic diagram of a tensor computation engine cluster according to an embodiment of the present disclosure is shown.

[0033] Figure 8 A schematic diagram illustrating the mapping of matrices and computing units according to an embodiment of the present disclosure is shown.

[0034] Figure 9 A schematic diagram showing the arrangement of matrices in registers according to an embodiment of the present disclosure is shown.

[0035] Figure 10 A schematic diagram showing the arrangement of matrices in local memory according to an embodiment of the present disclosure is provided.

[0036] Figure 11 A schematic diagram is shown illustrating the loading of block data by a tensor computation engine according to an embodiment of the present disclosure.

[0037] Figure 12 A schematic diagram illustrating the connection relationship between local memory and cache according to an embodiment of the present disclosure is shown.

[0038] Figure 13 A schematic diagram illustrating thread synchronization according to an embodiment of the present disclosure is shown.

[0039] Figure 14 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0040] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0041] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0042] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0043] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0044] For most artificial intelligence models, including neural network models involving multilayer perceptrons, matrix multiplication is one of the most commonly used operations, playing a particularly important role in the computation of various network layers. Figure 1 The diagram shows a schematic of the structure of a multilayer sensor in the related technology, such as... Figure 1 As shown, a multilayer perceptron is a feedforward neural network commonly used for tasks such as classification and regression. It typically includes an input layer 11, hidden layers 12, and an output layer 13. The input layer 11 receives the raw feature data, with each input feature corresponding to one neuron. The hidden layers 12 can consist of one or more fully connected layers, where each neuron is connected to all neurons in the previous fully connected layer. The number of hidden layers 12 and the number of neurons in each layer can be used as hyperparameters of the multilayer perceptron. The output layer 13 generates the final prediction result. For example, for classification tasks, the number of neurons in the output layer 13 can be the same as the number of categories; for regression tasks, the number of neurons in the output layer 13 is 1.

[0045] Figure 2 This diagram illustrates a fully connected layer in related technologies, such as... Figure 2As shown, performing a matrix multiplication operation between the input tensor 21 and the weight value 22 yields the output tensor 23, which can then be used as the input tensor 21 for the next layer. In related technologies, the operation at each layer divides the input tensor 21 into tiles of equal size and distributes them across the entire processor, including graphics processing units (GPUs), general-purpose graphics processing units (GPGPUs), and neural network processing units (NPUs). The size of the tile is related to the resources of the computing units (e.g., kernels, which can correspond to neurons) in the processor.

[0046] Optionally, the processor may include multiple computing cores, each computing core including multiple computing unit clusters, each computing unit cluster including multiple computing units, and each computing unit including registers, local memory, level 1 cache, floating-point processing unit (FP), integer processing unit (INT), special function unit (SFU), load store unit (LSU), tensor memory engine (TME), and tensor computing engine composed of multiple arithmetic logic units (ALU) and control logic. The multiple computing cores may correspond to a level 3 cache, the multiple computing unit clusters may correspond to a level 2 cache, and each computing unit may have a corresponding level 1 cache.

[0047] The floating-point processing unit (FP) is used for single-precision or double-precision floating-point arithmetic; the integer processing unit (INT) is used for integer operations; the special function unit (SFU) can calculate various functions, such as reciprocal functions, square root functions, exponential functions, logarithmic functions, and activation functions, through numerical approximation methods, which can be used to improve the computational performance of the computing unit; the load-store unit (LSU) is used to execute load or store instructions; and the tensor storage unit (TME) can be used for data transfer between memory and registers.

[0048] Figure 3 A schematic diagram of a processor in the related art is shown, such as... Figure 3As shown, the processor can be a GPU, and may include four computing cores: computing core 0 to computing core 3. Each computing core may include four computing unit clusters: computing unit cluster 0 to computing unit cluster 3. Each computing unit cluster may include two computing units: computing unit 0 and computing unit 1. Each computing unit includes a tensor computation engine composed of four arithmetic logic units (e.g., arithmetic logic units 0 to 3) and control logic. Each computing unit also includes local memory, a level 1 cache, four sets of registers, a floating-point number processing unit (FP), an integer processing unit (INT), a special function unit (SFU), a load-store unit (LSU), and a tensor storage unit (TME).

[0049] like Figure 3 As shown, a GPU can process multiple threads simultaneously, and the threads can be organized into a thread bundle and distributed to each computing unit. The register size of each computing unit is 128KB × 4 = 512KB, and the local memory size is 256KB.

[0050] Taking a matrix multiplication of size 64×64×64 as an example, the matrix calculation can be expressed as D=A×B+C, where matrices A and B can come from local memory, matrix C can come from a register, and the result of the multiplication and addition, D, can be written to a register. Assuming the computing power of the tensor calculation engine in each computing unit is 2048, 4096, and 8192 respectively, the bandwidth requirements for registers and local memory are shown in Table 1.

[0051] Table 1:

[0052]

[0053] Referring to Table 1, when the computing power of the tensor computation engine in each computing unit increases to 8192, the bandwidth requirements of the computing unit for local memory can no longer be met. Simultaneously, to reduce the bandwidth requirements for global memory, larger data blocks, such as 256×256×64, can be used. Table 2 shows the latency hiding of a single computing unit under different computing powers. Latency hiding is used to conceal delays caused by communication, memory access, conditional transition decisions, etc. For example, the parallel processing capabilities of the system can be utilized to execute other operations while waiting for certain operations to complete, thereby hiding the latency of these operations.

[0054] Table 2:

[0055]

[0056] As shown in Table 2, assuming the data format is 16-bit floating-point numbers (FP16), matrices A and B require 64KB of local memory space. With a local memory size of 256KB, a maximum of three levels of cache for matrix A (or matrix B) can be accommodated. When the computing power reaches 4096, the bandwidth between the computing unit and the L2 cache is insufficient; therefore, this bandwidth can be increased to 128 bytes / cycle. When the computing power reaches 8192, it is impossible to further increase the bandwidth to hide memory latency, and the complexity of the interconnection between the computing unit and the L2 cache becomes unsatisfactory.

[0057] Figure 4 A schematic diagram of the matrix multiplication calculation process in related technologies is shown, such as... Figure 4 As shown, assuming the computing unit has a 3-level cache, it can first read two blocks of data from matrix A (or matrix B) consecutively, then compute the first block of data from matrix A (or matrix B), while simultaneously reading the third block of data from matrix A (or matrix B). Once the first block of data has been computed, the first-level cache for matrix A (or matrix B) can be released, and the remaining blocks of data from matrix A (or matrix B) can be read.

[0058] In related technologies, solutions to increase computing power include adding more computing cores and more chips. This is a weak-scale approach to increase the system's speedup; a higher speedup indicates better parallel computing capabilities. Weak scaling refers to maintaining relatively constant efficiency when both the system size (e.g., the number of computing cores) and the task size increase simultaneously. In other words, this weak-scale approach achieves speedup by increasing the number of processors when the task size and the number of processors increase proportionally. For tasks with high workloads and high parallelism, this can improve performance, such as by increasing training batches, or by requiring users to rewrite code to achieve higher parallelism to adapt to new hardware.

[0059] Another way to improve performance through strong scaling is to increase system performance by improving the performance of individual computing cores. However, this approach requires greater bandwidth and larger local memory size. The data processing method of this disclosure interconnects the tensor computing engines of M computing units to form a tensor computing engine cluster, creating a larger array and achieving greater data reuse. The larger the value of M, the stronger the data processing capability of the tensor computing engine cluster, which serves as the computing core. Furthermore, the M computing units share tensor memory, reducing bandwidth requirements. Simultaneously, the local memory of each computing unit can be used, supporting the computation of larger blocks of data (e.g., portions of the first matrix), and, under the same blocks of data (e.g., portions of the first matrix), supporting more levels of caching to mask data transfer latency.

[0060] Figure 5 A flowchart illustrating a data processing method according to an embodiment of this disclosure is shown, such as... Figure 5 As shown, the data processing method is applied to a tensor computation engine cluster in a processor. The processor includes multiple computation cores, each of which includes multiple computation units. The tensor computation engine cluster is composed of interconnected tensor computation engines of M computation units. Each of the M computation units has a local memory or register for storing a portion of a first matrix, and the tensor memory shared by the M computation units is used to store a second matrix. M is an integer greater than 1. The data processing method includes:

[0061] In step S11, in response to the M tensor calculation engines in the tensor calculation engine cluster completing the reading of a portion of the first matrix from the local memory or register of their respective calculation units and the reading of the second matrix from the tensor memory, the M tensor calculation engines respectively perform matrix operations based on the portion of the first matrix and the second matrix to obtain their corresponding calculation results;

[0062] In step S12, each of the M tensor calculation engines writes its corresponding calculation result into its own register.

[0063] In one possible implementation, the processor includes multiple computing cores, and each computing core includes multiple computing units. The embodiments of this disclosure do not limit the number of computing cores in the processor, nor the number of computing units within each computing core; the number can be set according to the actual application scenario. The processor in the embodiments of this disclosure can be a completely new design or an improvement upon an existing processor chip. The type of processor chip may include, but is not limited to: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a General-Purpose Computing on Graphics Processing Units (GPGPU), a Neural Network Processing Unit (NPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Tensor Processing Unit (TPU), a Field Programmable Gate Array (FPGA), or other programmable logic devices. It may also include microprocessors or other conventional processors.

[0064] In one possible implementation, it is assumed that each computational core may include multiple computational unit clusters, and each computational unit cluster includes multiple computational units. Tensor computation engines of all or part of the computational units within a certain computational unit cluster can be interconnected to form a tensor computation engine cluster; or, tensor computation engines of all or part of the computational units within different computational unit clusters can be interconnected to form a tensor computation engine cluster.

[0065] Optionally, the tensor computation engine of each computation unit constituting the tensor computation engine cluster can work independently, and the tensor computation engine cluster consisting of M computation units can also work together. The embodiments of this disclosure do not limit this.

[0066] like Figure 3 As shown, the processor includes four computing cores, each computing core may include four computing unit clusters, and each computing unit cluster may include two computing units. In this case, the tensor computing engines of computing units 0 and 1 (M=2) in computing unit cluster 0 can be interconnected to form a tensor computing engine cluster; or, the tensor computing engines of computing units 0 and 1 in computing unit cluster 0, and computing units 0 and 1 in computing unit cluster 1 (M=4) can be interconnected to form a tensor computing engine cluster.

[0067] It should be understood that the embodiments of this disclosure do not limit the value of M, which may be 2, 4, or other values. The following description uses M=4 as an example.

[0068] In one possible implementation, the tensor computation engine includes multiple arithmetic logic units (ALUs), each of which is a multi-element multiply-accumulate unit array;

[0069] Figure 6 A schematic diagram of the structure of a tensor computation engine according to an embodiment of the present disclosure is shown, such as... Figure 6 As shown, the tensor computation engine of each computation unit consists of 4 arithmetic logic units (e.g., arithmetic logic unit 0 to arithmetic logic unit 3), and each arithmetic logic unit is an m×n×K multiply-accumulate unit array. Figure 6 The multiply-accumulate unit array shown in the figure has a size of 8×8×16. The specific values ​​of m, n and K are not limited in the embodiments disclosed herein.

[0070] like Figure 6 As shown, the tensor computation engine can read the data of matrix A and matrix B from the local memory inside its computation unit, read the information of matrix C from the register, and write the matrix D, which is the result of the computation, back to the register after the calculation is completed. Figure 6The structure of the medium tensor computation engine consists of a control logic unit, reuse cache A, reuse cache B, scaling factor unit, and four arithmetic logic units. Each arithmetic logic unit may include an 8×8 data processing unit DP, which is a multi-element multiply-accumulate unit array that can be used to perform multiply-accumulate-accumulate operations, such as A0×B0 + A1×B1 + A2×B2 + ..., where A0, A1, and A2 represent data elements in matrix A, and B0, B1, and B2 represent data elements in matrix B.

[0071] The control logic unit can be connected to each data processing unit (DP) within each arithmetic logic unit. In some embodiments, the control logic unit can number each data processing unit (DP) in the form of a two-dimensional matrix or a multi-dimensional matrix, so that multiple data processing units (DP) can be logically arranged in the form of a two-dimensional matrix or a multi-dimensional matrix, thereby better accommodating matrix logical operations. The control logic unit can also be connected to the local memory of the computing unit. The control logic unit can be used for address calculation to load data from the local memory into the arithmetic logic unit and control the arithmetic logic unit to process the data to be processed.

[0072] In one possible implementation, the local memory can store the data to be processed and the executable program. The executable program can include control instructions. The control logic unit can execute the control instructions to perform logical operations on the data to be processed, such as matrix multiplication, convolution, and other operations related to multiplication and accumulation.

[0073] To reduce redundant data reads, lower processor power consumption, and increase processor communication bandwidth, the tensor computation engine can also configure a reuse buffer and a scaling factor unit. A loader and decoder can be configured within the control logic unit. The loader loads data or a portion of data to be processed from local memory into the reuse buffer or scaling factor unit. The decoder decodes the control instructions for accessing data in the executable program based on the change in the storage address of the data to be processed after loading. For example, for a control instruction to access data X in local memory, since data X is cached in the reuse buffer (or scaling factor unit), the address of data X stored in the reuse buffer (or scaling factor unit) can be obtained through decoding. The decoder can then translate the control instruction for accessing data X in local memory into a control instruction for accessing data X in the reuse buffer (or scaling factor unit). This allows the subsequent control logic unit to directly send the cached data in the reuse buffer (or scaling factor unit) to the arithmetic logic unit, where the arithmetic logic unit performs multiplication, accumulation, and addition operations.

[0074] In one possible implementation, to facilitate the processor's handling of large-scale data, such as Figure 6 As shown, the tensor computation engine can also communicate with registers within the computation unit. This significantly reduces instruction execution time when the computation unit processes large amounts of data. These registers may include address registers, data registers, instruction registers, opcode word registers, flag registers, etc., and this disclosure does not limit their scope. In the example, the control logic unit can be used for address calculation to facilitate data transfer between the various arithmetic logic units and the registers.

[0075] In one possible implementation, the control logic unit can obtain descriptive information in constant memory through control instructions. The descriptive information includes, for example, the dimension of the data to be processed, the size of the data to be processed, and the data type of the elements in the data to be processed (e.g., integer type, single-precision floating-point type, double-precision floating-point type, character type, etc.), which are used to describe the data to be processed. The embodiments of this disclosure do not limit this.

[0076] Figure 7 A schematic diagram of a tensor computation engine cluster according to embodiments of the present disclosure is shown, such as Figure 7 As shown, assuming each computational unit cluster has four computational units, namely computational units 0 to 3, the tensor computation engines of computational units 0 to 3 can be arranged into a 4x1 array, namely: a tensor computation engine cluster. Here, matrix A represents the first matrix, matrix B represents the second matrix, and the local memory of each computational unit can be used to store a portion of the first matrix. For example, the local memory of computational unit 0 can be used to store the first part of the first matrix A, the local memory of computational unit 1 can be used to store the second part of the first matrix A, the local memory of computational unit 2 can be used to store the third part of the first matrix A, and the local memory of computational unit 3 can be used to store the fourth part of the first matrix A; the tensor memory shared by computational units 0 to 3 is used to store the second matrix B.

[0077] like Figure 7 As shown, the first matrix A and the second matrix B can be transferred between tensor computation engines in different computation units. A tensor memory shared by computation units 0 to 3, added in each computation unit cluster, can store the second matrix B. Each tensor computation engine reads the corresponding portion of the first matrix A from the local memory of its corresponding computation unit. Then, the tensor computation engine cluster reads the second matrix B from the tensor memory and broadcasts it to each computation unit. The tensor computation engine of each computation unit is connected to the local memory within the computation unit, avoiding cross-connections. Compared with related technologies, the newly added tensor memory shared by computation units 0 to 3 increases bandwidth and simplifies the architecture.

[0078] The table below shows the bandwidth requirements for different matrix sizes under the above structure. For example, referring to the last row of Table 3, the bandwidth of matrix A (256×64×64) is 1024 bytes, which averages to 256 bytes per computational unit. The bandwidth of matrix B is 256 bytes, so the bandwidth of the tensor memory configuration is 256 bytes.

[0079] Table 3:

[0080]

[0081] In step S11, the M tensor calculation engines can simultaneously read a portion of the first matrix from the local memory or register of their respective computing units and read a second matrix from the tensor memory. In response to the M tensor calculation engines in the tensor calculation engine cluster completing the reading of a portion of the first matrix from the local memory or register of their respective computing units and the reading of the second matrix from the tensor memory, the M tensor calculation engines perform matrix operations based on the portion of the first matrix and the second matrix respectively to obtain multiple calculation results.

[0082] In one possible implementation, each tensor computation engine is also used to read the third matrix from the register of its respective computation unit. Step S11 may include: M tensor computation engines performing matrix multiplication operations based on a portion of the first matrix and the second matrix to obtain multiple product results; M tensor computation engines performing matrix sum operations based on the corresponding product results and the third matrix to obtain multiple operation results.

[0083] In this way, the tensor computation engine can perform operations on fully connected layers. The first matrix corresponds to the input data of the fully connected layer, a partial first matrix corresponds to a portion of the input data of the fully connected layer, the second matrix corresponds to the weights of the fully connected layer, and the third matrix corresponds to the biases of the fully connected layer. Furthermore, using M tensor computation engines simultaneously fully utilizes the parallel computing capabilities of the tensor computation engine cluster, significantly improving computational efficiency. Each tensor computation engine handles a portion of the matrix operations, achieving the decomposition and parallelization of computational tasks (such as fully connected layer computation), reducing the burden on a single tensor computation engine.

[0084] Figure 8 A schematic diagram illustrating the mapping between matrices and computational units according to embodiments of the present disclosure is shown, such as... Figure 8 As shown, each computation unit in computation units 0 through 3 computes a 64×64×K matrix, and the four computation units together perform a matrix multiplication of size 256×64×K. The first matrix A can come from local memory or from a register, the second matrix comes from tensor memory, and the third matrix C can come from a register, with the result D written back to the register.

[0085] In step S11, the M tensor computation engines each obtain their respective computation results. Assuming matrix A is the first matrix, matrix B is the second matrix, and matrix C is the third matrix, the computation result obtained in step S11 can be either A×B or A×B+C. In step S12, each of the M tensor computation engines writes its corresponding computation result into its own register. Since the neural network model is trained iteratively, the computation result of the current training iteration can be used as the third matrix for the next training iteration.

[0086] Through steps S11 to S12, existing tensor computing engines can be utilized. Without changing the structure of the tensor computing engine, computing power can be increased by interconnecting tensor computing engines of multiple computing units and adding shared tensor memory to multiple computing units, without increasing the size and bandwidth of local memory and registers within the computing units.

[0087] In one possible implementation, the processor further includes a task distribution module, and the method further includes: the task distribution module distributes M thread bundles to M computing units; the M computing units, according to the thread bundles they receive, move a second matrix from a shared cache to the tensor memory, and move a portion of their corresponding first matrix to their respective local memory; the M computing units send matrix operation instructions to their internal tensor computing engines, so that the M tensor computing engines read a portion of the first matrix from the local memory of their respective computing units and read the second matrix from the tensor memory according to the matrix operation instructions.

[0088] For example, suppose the tensor computation engine is composed of interconnected tensor computation engines of computation units 0 to 3. The task distribution module distributes thread bundle 0 to computation unit 0, thread bundle 1 to computation unit 1, thread bundle 2 to computation unit 2, and thread bundle 3 to computation unit 3 in parallel. A computation unit can be selected from computation units 0 to 3 to move the second matrix from the peer cache shared by computation units 0 to 3 to the tensor memory according to the received thread bundle. At the same time, computation unit 0 moves the first part of the first matrix from the peer cache to its own local memory according to the received thread bundle 0, computation unit 1 moves the second part of the first matrix from the peer cache to its own local memory according to the received thread bundle 1, computation unit 2 moves the third part of the first matrix from the peer cache to its own local memory according to the received thread bundle 2, and computation unit 3 moves the fourth part of the first matrix from the peer cache to its own local memory according to the received thread bundle 3. Then, computation units 0 to 3 can send matrix operation instructions to the four corresponding tensor computation engines, so that the four tensor computation engines can read a portion of the first matrix from the local memory of their respective computation units and read the second matrix from the tensor memory according to the matrix operation instructions they receive.

[0089] In this way, the task distribution module distributes the M thread bundles of the tasks to be processed to M computing units within the same or different computing unit clusters. By distributing the M thread bundles to the M computing units, the M computing units can process the M threads in parallel. Each computing unit can work independently, processing its own received thread bundle, thus enabling the simultaneous processing of different data segments and significantly improving computational efficiency. Moving the second matrix from the shared peer cache to the tensor memory and moving a portion of the first matrix to their respective local memories helps reduce memory access conflicts and latency. Furthermore, setting up a shared tensor memory for the M computing units increases computational power without increasing the size and bandwidth of their local memories.

[0090] In one possible implementation, M computing units, based on the received thread bundles, move a second matrix from a shared cache among the M computing units to the tensor memory, and each move a portion of the first matrix to its respective local memory, including: the computing units distributing the thread groups in the received thread bundles to at least one execution unit; the execution unit generating an access request based on the thread groups it receives, the access request being used to move the second matrix from the shared cache among the M computing units to the tensor memory, and to move a portion of the first matrix to its local memory.

[0091] Each computing unit may include multiple execution units, see [link to relevant documentation]. Figure 3Each execution unit may include a floating-point processing unit (FP), an integer processing unit (INT), a special function unit (SFU), a load storage unit (LSU), a tensor storage unit (TME), etc. The embodiments of this disclosure do not impose specific limitations on the specific composition of the execution unit.

[0092] In this way, multiple execution units within each computational unit can process multiple thread groups in the thread bundle in parallel, further improving the parallelism of the tensor computation engine cluster.

[0093] For example, the task distribution module can be used to provide a distribution mechanism. When the task distribution module is enabled, for example, if the user can use a preset application programming interface (API SquadWGMMA), the task distribution mode will enter an enabled state. The task distribution module will distribute four consecutive (M=4) thread bundles to four computing units of a certain computing unit cluster in a round-robin mode. Each thread bundle may include four thread groups, and each thread group may consist of 32 threads, that is, each thread bundle may include 4×32 threads. For each thread bundle, the four thread groups in the thread bundle can be distributed to four execution units of each computing unit. The execution units can generate access requests based on the received thread groups. Access requests generated by different execution units can simultaneously update the local memory and tensor memory in their respective computing units. It should be understood that the embodiments of this disclosure do not impose specific limitations on the number of thread bundles, the number of thread groups included in each thread bundle, and the number of threads included in each thread group, and can be set according to the actual application scenario.

[0094] Figure 9 A schematic diagram showing the matrix arrangement in the register according to an embodiment of the present disclosure is shown, as follows: Figure 9 As shown, in a thread group, the size of each thread's register is 32 bits. For example, thread group 0 can correspond to 32×2 threads, and each thread can correspond to one register. The size of each register in T0~T31 is 32 bits. Ra represents an 8×4 structure composed of the registers of the 32 threads in thread group 0, and Ra+1 represents another 8×4 structure composed of the registers of the other 32 threads in thread group 0.

[0095] The register can be used to store a third matrix, which may be the result of operations on the first and second matrices from the previous training round. The bandwidth of the register is the same as the number of elements in the first dimension of the first matrix and the number of elements in the second dimension of the second matrix. In this way, the bandwidth of the register (i.e., the number of elements that can be transmitted at one time) can be reasonably set, reducing the transmission latency caused by the register.

[0096] Figure 10A schematic diagram showing the arrangement of a matrix in local memory according to an embodiment of the present disclosure is shown, such as... Figure 10 As shown, to avoid bandwidth contention, data needs to be swizzled. Swizzling uses coordinate changes to remap physical addresses to logical addresses, causing data from the same matrix to be staggered across different rows. It should be understood that... Figure 10 The numbers 0 to 15 are used to represent different matrices.

[0097] Optionally, the first matrix corresponding to each computational unit can come from local memory or from registers, and different thread bundles within the computational unit cluster can be arranged according to... Figure 9 , Figure 10 The diagram shows the arrangement of local memory or registers in different computing units, allowing data to be read from video memory or peer cache. Each thread bundle carries first and second matrix coordinates, and the data coordinates accessed by each thread group within the thread bundle can be configured. Data is accessed through the load-store component (LSU) of each execution unit in the computing unit. Alternatively, a tensor storage component (TME) can be used, specifying the starting coordinates and the size of the data blocks to be accessed (e.g., a portion of the first matrix A0 or a portion of the first matrix A1).

[0098] Optionally, data synchronization can be achieved using the Tensor Memory Component (TME) of each execution unit within the computation unit via an asynchronous barrier. The TME adds support for tensor memory, allowing direct data transfer from the processor's global memory to tensor memory. Furthermore, for accessing tensor memory, new read / write instructions between tensor memory and registers can be added, such as instructions for transferring data from tensor memory to registers (Tensor Memory to Register, T2R) and transferring data from registers to tensor memory (Register to Tensor Memory, R2T), enabling data transfer between tensor memory and registers.

[0099] Furthermore, each execution unit in the computing unit can select a thread group to generate executable instructions for the tensor storage component (TME), specifying the starting coordinates and the size of the block data, such as part of the first matrix A0, part of the first matrix A1, part of the first matrix A2, part of the first matrix A3, etc.

[0100] Figure 11 This diagram illustrates the loading of block data by a tensor computation engine according to an embodiment of the present disclosure, such as... Figure 11As shown, the description information of the first matrix may include the horizontal dimension dimension dim0, the vertical dimension dimension dim1, and the stride0 required to jump from one element to the next element in the specified dimension. The description information of the portion of the first matrix to be accessed by the current tensor computation engine or tensor computation engine cluster may include the horizontal dimension dimension block_dim, the vertical dimension dimension block_dim1, and the position coordinates pos of the portion of the first matrix within the first matrix.

[0101] In one possible implementation, M tensor computation engines read a portion of a first matrix from the local memory of their respective computation units and read a second matrix from the tensor memory according to the matrix operation instructions. This includes: in response to all execution units of each tensor computation engine completing the matrix operation instructions, the tensor computation engine marks a first synchronization signal in a synchronization counter; in response to detecting that the synchronization counter has marked M first synchronization signals, a second synchronization signal is generated, the second synchronization signal indicating that the M tensor computation engines have completed reading a portion of the first matrix from the local memory of their respective computation units and reading the second matrix from the tensor memory.

[0102] Figure 12 A schematic diagram illustrating the connection relationship between local memory and cache according to an embodiment of the present disclosure is shown, such as... Figure 12 As shown, for computation units 0 to 3, the local memory of each computation unit can be connected to a cache at the same level (e.g., a L2 cache) via a bus. The four thread groups in each computation unit send matrix operation instructions to the tensor computation engine. These matrix operation instructions can carry the encoding of the thread bundle to which the thread group to be executed belongs. Based on the thread bundle encoding, the tensor computation engine reads the address information of a portion of the first matrix from the local memory of the four execution units, and the address information of the second matrix from the tensor memory.

[0103] Figure 13 A schematic diagram illustrating thread synchronization according to an embodiment of the present disclosure is shown, such as... Figure 13 As shown, the issuance of this matrix operation instruction requires waiting for the corresponding first and second matrix data to be ready, which is generally ensured by data dependency mechanisms such as synchronization counters. When the same thread bundle on the four execution units of each computing unit (computing units 0 to 3) executes the matrix operation instruction, the synchronization completion of one computing unit is marked with a first synchronization signal in the synchronization counter. When all four computing units of a computing unit cluster have completed synchronization, and multiple thread bundles of the entire computing unit cluster have also completed synchronization, the synchronization counter can generate a second synchronization signal.

[0104] In this way, thread bundle synchronization across computing units can be achieved in a hierarchical manner, that is, thread synchronization across computing units. Specifically, the first synchronization signal is used for synchronization between different thread groups within the same thread bundle, and the second synchronization signal is used for synchronization between different thread bundles.

[0105] In one possible implementation, each of the M tensor computation engines writes its corresponding computation result to its own register, including: in response to all M tensor computation engines writing their corresponding computation results to their own registers, clearing the first synchronization signal and the second synchronization signal in the synchronization counter. By clearing the synchronization signals, it can be ensured that the tensor computation engine cluster can process the matrix operation task of the next training round in a timely manner, avoiding mutual interference between matrix operation tasks between adjacent training rounds.

[0106] For example, the tensor computation engine cluster can be configured according to... Figure 9 , Figure 10 The described data arrangement is read, then calculations are performed. Once the calculations are complete and the corresponding results are obtained, the results are written back to the registers of each MP (Multi-Processing Unit). Then, the first and second synchronization signals in the synchronization counters are released. This allows subsequent steps relying on matrix operation instructions to begin execution.

[0107] In summary, the data processing method of this disclosure can interconnect the tensor computation engines of M computation units to form a tensor computation engine cluster, creating a larger array and achieving greater data reuse. Furthermore, the M computation units share tensor memory, reducing bandwidth requirements. Simultaneously, the local memory or registers of each computation unit can be used to support the computation of larger-sized data blocks (e.g., portions of the first matrix), and under the same data blocks (e.g., portions of the first matrix), more levels of caching can be supported to mask data transfer latency. In addition, a hierarchical approach can be used to achieve thread bundle synchronization across computation units.

[0108] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0109] This disclosure also proposes a processor, which includes multiple computing cores, each computing core including multiple computing units, and M computing units interconnected to form a tensor computing engine cluster. Each of the M computing units has a local memory or register for storing a portion of a first matrix, and a shared tensor memory for storing a second matrix, where M is an integer greater than 1. In response to the M tensor computing engines in the cluster reading the portion of the first matrix from the local memory or register of their respective computing units and reading the second matrix from the tensor memory, the M tensor computing engines perform matrix operations based on the portion of the first matrix and the second matrix respectively to obtain their corresponding operation results. The M tensor computing engines also write their corresponding operation results into their own registers.

[0110] In one possible implementation, the processor further includes a task distribution module, which distributes M thread bundles to the M computing units; the M computing units, based on the thread bundles they receive, move the second matrix from a shared cache to the tensor memory, and move their respective portions of the first matrix to their local memory; the M computing units send matrix operation instructions to their internal tensor computing engines, so that the M tensor computing engines read the portions of the first matrix from the local memory of their respective computing units and read the second matrix from the tensor memory according to the matrix operation instructions.

[0111] In one possible implementation, the M tensor computation engines read a portion of the first matrix from the local memory of their respective computation units and read the second matrix from the tensor memory according to the matrix operation instructions, including: in response to each tensor computation engine completing the matrix operation instructions, the tensor computation engine marks a first synchronization signal in a synchronization counter; in response to detecting that the synchronization counter has marked M first synchronization signals, a second synchronization signal is generated, the second synchronization signal indicating that the M tensor computation engines have completed reading the portion of the first matrix from the local memory of their respective computation units and reading the second matrix from the tensor memory.

[0112] In one possible implementation, the M tensor computation engines are specifically used to: in response to the M tensor computation engines writing their corresponding computation results into their own registers, clear the first synchronization signal and the second synchronization signal in the synchronization counter.

[0113] In one possible implementation, the tensor computation engine includes multiple arithmetic logic units, each of which is a multi-element multiply-accumulate unit array; the bandwidth of the register is the same as the number of elements in the first dimension of the partial first matrix and the number of elements in the second dimension of the second matrix.

[0114] In one possible implementation, each tensor computation engine is further configured to read a third matrix from the register of its respective computation unit. The M tensor computation engines are configured to: perform matrix multiplication operations based on their respective partial first matrices and second matrices to obtain multiple product results; and perform matrix sum operations based on their respective product results and the third matrix to obtain multiple operation results.

[0115] In one possible implementation, the M computing units, based on the thread bundles they receive, move the second matrix from the shared peer cache to the tensor memory, and each move its corresponding portion of the first matrix to its respective local memory. This includes: the computing unit distributing the thread groups in the received thread bundles to at least one execution unit; and the execution unit generating an access request based on the thread groups it receives, the access request being used to move the second matrix from the shared peer cache to the tensor memory, and to move the portion of the first matrix to the local memory.

[0116] According to one aspect of this disclosure, a chip is provided that includes the processor described above.

[0117] According to one aspect of this disclosure, a graphics card is provided, the graphics card including the processor described above.

[0118] In addition, this disclosure also provides a data processing apparatus, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any of the data processing methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding records in the method section and will not be repeated here.

[0119] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0120] This disclosure also proposes an artificial intelligence chip, which includes the processor described above.

[0121] This disclosure also proposes a graphics card, which includes the processor described above. The graphics card, also known as a display adapter, graphics processing card, or video card, is a hardware component in a computer responsible for graphics rendering and display output. The graphics card may include discrete graphics cards, integrated graphics cards, mobile graphics cards, professional graphics cards, cluster graphics cards, etc. This disclosure does not limit the type of graphics card.

[0122] This disclosure also proposes an intelligent computing center (also known as a server cluster), which includes the processor described above.

[0123] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the above-described method.

[0124] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0125] Electronic devices can be provided as terminals, servers, or other forms of devices. Among them, terminal devices can be user equipment (UE), mobile devices, user terminals, terminals, cellular phones, cordless phones, personal digital assistants (PDAs), handheld devices, computing devices, in-vehicle devices, wearable devices, etc., and the embodiments disclosed herein are not limited thereto.

[0126] Figure 14 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a server or a terminal device. (Refer to...) Figure 14 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0127] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958. Electronic device 1900 can operate on an operating system stored in memory 1932, such as a Microsoft Server operating system (Windows Server). TM Apple's graphical user interface-based operating system (Mac OS X) TM ), a multi-user, multi-process computer operating system (Unix) TM Linux is a free and open-source Unix-like operating system. TM ), the open-source Unix-like operating system (FreeBSD) TM (or similar.)

[0128] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0129] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0130] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, (but not limited to) electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0131] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0132] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0133] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0134] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0135] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0136] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0137] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0138] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0139] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0140] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0141] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A data processing method, characterized in that, The data processing method is applied to a tensor computation engine cluster in a processor. The processor includes multiple computation cores, each of which includes multiple computation units. The tensor computation engine cluster is composed of interconnected tensor computation engines of M computation units. The local memory or register of each of the M computation units is used to store a portion of the first matrix, and the tensor memory shared by the M computation units is used to store the second matrix, where M is an integer greater than 1. The method includes: In response to the completion of reading the partial first matrix from the local memory or register of the computing unit and reading the second matrix from the tensor memory by M tensor computing engines in the tensor computing engine cluster, the M tensor computing engines respectively perform matrix operations based on the partial first matrix and the second matrix to obtain their corresponding operation results; Each of the M tensor computation engines writes its corresponding computation result into its own register.

2. The method according to claim 1, characterized in that, The processor further includes a task distribution module, and the method further includes: The task distribution module distributes M thread bundles to the M computing units; The M computing units, based on the thread bundles they receive, move the second matrix from the shared cache of the M computing units to the tensor memory, and each move its corresponding portion of the first matrix to its respective local memory. The M computing units respectively send matrix operation instructions to their internal tensor computing engines, so that the M tensor computing engines can read the partial first matrix from the local memory of their respective computing units and read the second matrix from the tensor memory according to the matrix operation instructions.

3. The method according to claim 2, characterized in that, The M tensor computation engines read the partial first matrix from the local memory of their respective computation units and the second matrix from the tensor memory according to the matrix operation instructions, including: In response to each tensor computation engine completing the matrix operation instruction, the tensor computation engine marks a first synchronization signal in a synchronization counter; In response to the detection that the synchronization counter has marked M first synchronization signals, a second synchronization signal is generated. The second synchronization signal is used to indicate that the M tensor calculation engines have completed reading the partial first matrix from the local memory of their respective calculation units and reading the second matrix from the tensor memory.

4. The method according to claim 3, characterized in that, The M tensor computation engines each write their corresponding computation results into their own registers, including: In response to the M tensor calculation engines writing their corresponding calculation results into their own registers, the first synchronization signal and the second synchronization signal in the synchronization counter are cleared.

5. The method according to claim 1, characterized in that, The tensor computation engine includes multiple arithmetic logic units, each of which is a multi-element multiply-accumulate unit array; The bandwidth of the register is the same as the number of elements in the first dimension of the first matrix and the number of elements in the second dimension of the second matrix.

6. The method according to any one of claims 1 to 5, characterized in that, Each tensor computation engine is also used to read a third matrix from the register of its respective computation unit. The M tensor computation engines perform matrix operations based on the partial first matrix and the second matrix to obtain their own corresponding operation results, including: The M tensor computation engines each perform matrix multiplication operations based on their respective first and second matrices to obtain multiple product results. The M tensor computation engines perform matrix sum operations based on the corresponding product results and the third matrix, respectively, to obtain multiple computation results.

7. The method according to any one of claims 2 to 5, characterized in that, The M computing units, based on the thread bundles they receive, move the second matrix from the shared peer cache to the tensor memory, and each move its corresponding portion of the first matrix to its respective local memory, including: The computing unit distributes the thread components from the received thread bundle to at least one execution unit; The execution unit generates an access request based on the thread group it receives. The access request is used to move the second matrix from the peer cache shared by the M computing units to the tensor memory, and to move a portion of the first matrix to the local memory.

8. A processor, characterized in that, The processor includes multiple computing cores, each computing core includes multiple computing units, and the tensor computing engines of M computing units are interconnected to form a tensor computing engine cluster. The local memory or register of each of the M computing units is used to store a portion of the first matrix, and the tensor memory shared by the M computing units is used to store the second matrix, where M is an integer greater than 1. In response to the completion of reading the partial first matrix from the local memory or register of the computing unit and reading the second matrix from the tensor memory by M tensor computing engines in the tensor computing engine cluster, the M tensor computing engines are used to perform matrix operations based on the partial first matrix and the second matrix respectively to obtain their corresponding operation results; The M tensor computation engines are used to write their respective computation results into their own registers.

9. A chip or graphics card, characterized in that, The chip or graphics card includes the processor as described in claim 8.

10. An electronic device, characterized in that, The electronic device includes the chip or graphics card as described in claim 9.

Images