Data processing method, processor, chip, graphics card and electronic device

Abstract

The present disclosure relates to a data processing method, a processor, a chip, a graphics card and an electronic device, and relates to the technical field of intelligent computing. The method comprises the following steps: in response to M tensor computing engines in a tensor computing engine cluster completing reading of a partial first matrix from a local memory or a register of a computing unit to which the M tensor computing engines belong and reading of a second matrix from a tensor memory, the M tensor computing engines respectively perform matrix operation according to the partial first matrix and the second matrix to obtain corresponding operation results of the M tensor computing engines; and the M tensor computing engines respectively write the corresponding operation results into the registers of the M tensor computing engines. The embodiment of the present disclosure can increase computing power without increasing the size and bandwidth of the local memory and the register inside the computing unit.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of intelligent computing, and in particular, to a data processing method, a processor, a chip, a graphics card and an electronic device. BACKGROUND

[0002] With the development of artificial intelligence (AI) and the rapid application of large models in various fields, the demand for hardware computing power and bandwidth is increasing. Increasing computing power can use more computing cores to run more tasks simultaneously. However, for scenarios that require performance improvement for a single task, this solution cannot meet the requirements due to the weak capabilities of individual computing cores. SUMMARY

[0003] The present disclosure provides a data processing method, a processor, a chip, a graphics card and an electronic device.

[0004] According to an aspect of the present disclosure, a data processing method is provided, which is applied to a tensor computing engine cluster in a processor, the processor comprising a plurality of computing cores, each computing core comprising a plurality of computing units, the tensor computing engine cluster being formed by a tensor computing engine interconnection of M computing units, a local memory or a register of each computing unit in the M computing units being used to store a partial first matrix, and a tensor memory shared by the M computing units being used to store a second matrix, M being an integer greater than 1; the method comprising: in response to M tensor computing engines in the tensor computing engine cluster completing reading of the partial first matrix from the local memory or the register of the computing unit to which the M tensor computing engines belong, and reading of the second matrix from the tensor memory, the M tensor computing engines respectively performing matrix operations according to the partial first matrix and the second matrix to obtain respective operation results; and the M tensor computing engines respectively writing the respective operation results into the respective registers.

[0005] In a possible implementation, the processor further comprises a task distribution module, and the method further comprises: the task distribution module distributing M thread bundles to the M computing units; the M computing units respectively carrying the second matrix from a same-level cache shared by the M computing units to the tensor memory and carrying the partial first matrix corresponding to each of the M computing units to the local memory of each of the M computing units according to the thread bundle received by each of the M computing units; and the M computing units respectively sending matrix operation instructions to the internal tensor computing engines, so that the M tensor computing engines read the partial first matrix from the local memory of the computing unit to which the M tensor computing engines belong and read the second matrix from the tensor memory according to the matrix operation instructions.

[0006] In a possible implementation, the M tensor computing engines read the partial first matrix from the local memory of the computing unit to which the tensor computing engine belongs and read the second matrix from the tensor memory according to the matrix operation instruction, including: in response to each tensor computing engine completing the matrix operation instruction, the tensor computing engine marks a first synchronization signal in a synchronization counter; in response to detecting that the synchronization counter marks M first synchronization signals, a second synchronization signal is generated, and the second synchronization signal is used to indicate that the M tensor computing engines complete reading the partial first matrix from the local memory of the computing unit to which the tensor computing engine belongs and reading the second matrix from the tensor memory.

[0007] In a possible implementation, the M tensor computing engines write the operation results corresponding to the tensor computing engines respectively into the registers of the tensor computing engines, including: in response to the M tensor computing engines writing the operation results corresponding to the tensor computing engines respectively into the registers of the tensor computing engines, the first synchronization signal and the second synchronization signal in the synchronization counter are cleared.

[0008] In a possible implementation, the tensor computing engine includes a plurality of arithmetic logic units, and each arithmetic logic unit is a multiply-add unit array of multiple elements; and the bandwidth of the register is the same as the number of elements of the first dimension of the partial first matrix and the number of elements of the second dimension of the second matrix.

[0009] In a possible implementation, each tensor computing engine is further configured to read a third matrix from the register of the computing unit to which the tensor computing engine belongs, and the M tensor computing engines perform matrix operations respectively according to the partial first matrix and the second matrix to obtain operation results corresponding to the tensor computing engines, including: the M tensor computing engines perform matrix multiplication respectively according to the partial first matrix and the second matrix corresponding to the tensor computing engines to obtain a plurality of product results; and the M tensor computing engines perform matrix addition respectively according to the corresponding product results and the third matrix to obtain a plurality of operation results.

[0010] In a possible implementation, the M computing units transfer the second matrix from the same-level cache shared by the M computing units to the tensor memory and transfer the partial first matrix corresponding to the computing unit respectively to the local memory of the computing unit according to the thread bundle received by the computing unit, including: the computing unit distributes the thread group in the thread bundle received by the computing unit to at least one execution unit; and the execution unit generates an access request according to the thread group received by the execution unit, and the access request is used to transfer the second matrix from the same-level cache shared by the M computing units to the tensor memory and transfer the partial first matrix to the local memory.

[0011] According to an aspect of the present disclosure, a processor is provided, comprising: a processor comprising a plurality of computing cores, each computing core comprising a plurality of computing units, a tensor computing engine in M computing units of the plurality of computing units being interconnected to form a tensor computing engine cluster, a local memory or a register of each computing unit of the M computing units being configured to store a partial first matrix, a tensor memory shared by the M computing units being configured to store a second matrix, M being an integer greater than 1; in response to M tensor computing engines in the tensor computing engine cluster completing reading the partial first matrix from the local memory or the register of the computing unit to which the M tensor computing engines belong and reading the second matrix from the tensor memory, the M tensor computing engines are configured to respectively perform a matrix operation according to the partial first matrix and the second matrix to obtain an operation result corresponding to the M tensor computing engines respectively; and the M tensor computing engines are configured to respectively write the operation result corresponding to the M tensor computing engines into the register of the M tensor computing engines.

[0012] In a possible implementation, the processor further comprises a task distribution module, the task distribution module is configured to distribute M thread bundles to the M computing units; the M computing units are configured to respectively carry the second matrix from a same-level cache shared by the M computing units to the tensor memory and carry the partial first matrix corresponding to the M computing units respectively to the local memory according to the thread bundle received by the M computing units; and the M computing units are configured to respectively send a matrix operation instruction to the internal tensor computing engine, so that the M tensor computing engines read the partial first matrix from the local memory of the computing unit to which the M tensor computing engines belong and read the second matrix from the tensor memory according to the matrix operation instruction.

[0013] In a possible implementation, the M tensor computing engines reading the partial first matrix from the local memory of the computing unit to which the M tensor computing engines belong and reading the second matrix from the tensor memory according to the matrix operation instruction comprises: in response to each tensor computing engine completing the matrix operation instruction, the tensor computing engine marks a first synchronization signal in a synchronization counter; and in response to detecting that the synchronization counter marks M first synchronization signals, a second synchronization signal is generated, the second synchronization signal being configured to indicate that the M tensor computing engines complete reading the partial first matrix from the local memory of the computing unit to which the M tensor computing engines belong and reading the second matrix from the tensor memory.

[0014] In a possible implementation, the M tensor computing engines are specifically configured to: in response to the M tensor computing engines writing the operation result corresponding to the M tensor computing engines into the register of the M tensor computing engines, clear the first synchronization signal and the second synchronization signal in the synchronization counter.

[0015] In a possible implementation, the tensor computing engine includes a plurality of arithmetic logic units, each of which is a multiply-add unit array of multiple elements; and the bandwidth of the register is the same as the number of elements of the first dimension of the partial first matrix and the number of elements of the second dimension of the second matrix.

[0016] In a possible implementation, each tensor computing engine is further configured to read a third matrix from the register of the computing unit to which the tensor computing engine belongs, and the M tensor computing engines are configured to: perform matrix multiplication according to the partial first matrix and the second matrix corresponding to each of the M tensor computing engines respectively, to obtain a plurality of product results; and perform matrix summation according to the corresponding product result and the third matrix, to obtain a plurality of operation results.

[0017] In a possible implementation, the M computing units carry the second matrix from the same-level cache shared by the M computing units to the tensor memory and carry the partial first matrix corresponding to each of the M computing units to the local memory of the computing unit according to the thread bundle received by the computing unit, including: the computing unit distributes the thread groups in the thread bundle received by the computing unit to at least one execution unit; and the execution unit generates an access request according to the thread group received by the execution unit, and the access request is used to carry the second matrix from the same-level cache shared by the M computing units to the tensor memory and carry the partial first matrix to the local memory.

[0018] According to an aspect of the present disclosure, a chip is provided, which includes the processor as described above.

[0019] According to an aspect of the present disclosure, a graphics card is provided, which includes the processor as described above.

[0020] According to an aspect of the present disclosure, an electronic device is provided, which includes the chip as described above.

[0021] The data processing method of the embodiments of the present disclosure can be applied to a tensor computing engine cluster in a processor, the processor comprising a plurality of computing cores, each computing core comprising a plurality of computing units, the tensor computing engine cluster being formed by tensor computing engine interconnection of M computing units, a local memory or a register of each of the M computing units being used to store a partial first matrix, and a tensor memory shared by the M computing units being used to store a second matrix, M being an integer greater than 1; the method comprising: in response to M tensor computing engines in the tensor computing engine cluster completing reading of the partial first matrix from the local memory or the register of the computing unit to which the M tensor computing engines belong, and reading of the second matrix from the tensor memory, the M tensor computing engines respectively performing matrix operations according to the partial first matrix and the second matrix to obtain respective operation results; and the M tensor computing engines respectively writing the respective operation results into the registers thereof.

[0022] In this way, the existing tensor computing engine can be used, the tensor computing engine cluster can be formed by tensor computing engine interconnection of multiple computing units without changing the structure of the tensor computing engine, multiple tensor computing engines can be used to perform calculations simultaneously, the parallel computing capability of the tensor computing engine cluster can be fully utilized, each tensor computing engine processes partial matrix operations, the decomposition and parallelization of the calculation task are realized, and the calculation efficiency is significantly improved; further, the shared tensor memory is added for the multiple computing units, the second matrix stored in the tensor memory can be reused by each computing unit, the demand for bandwidth is reduced, and the local memory or the register of each computing unit itself can support the storage of larger size of block data (for example, the partial first matrix) without increasing the size and bandwidth of the local memory and the register in the computing unit.

[0023] In addition, the data processing method of the embodiments of the present disclosure can also adjust the number of tensor computing engines in the tensor computing engine cluster according to different task quantities, reduce the idle or overload of resources, and be suitable for more scenarios; for example, for a scenario in which the performance of a single task needs to be improved, the tensor computing engine cluster is formed by tensor computing engine interconnection of more or even all computing units, so as to provide sufficient computing power; for an application scenario in which there are not enough tasks, the tensor computing engine cluster is formed by tensor computing engine interconnection of fewer computing units, so that the idle other computing units can perform other tasks, and the waste of hardware resources is reduced.

[0024] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, rather than limiting the present disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0025] The drawings incorporated in and forming a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the principles of the present disclosure.

[0026] Figure 1 A structure diagram of a multi-layer perceptron in the related art is shown.

[0027] Figure 2 A diagram of a fully connected layer in the related art is shown.

[0028] Figure 3 A diagram of a processor in the related art is shown.

[0029] Figure 4 A diagram of a matrix multiplication calculation flow in the related art is shown.

[0030] Figure 5 A flowchart of a data processing method according to an embodiment of the present disclosure is shown.

[0031] Figure 6 A structure diagram of a tensor calculation engine according to an embodiment of the present disclosure is shown.

[0032] Figure 7 A diagram of a tensor calculation engine cluster according to an embodiment of the present disclosure is shown.

[0033] Figure 8 A diagram of a matrix and a calculation unit mapping according to an embodiment of the present disclosure is shown.

[0034] Figure 9 A diagram of a matrix in a register arrangement according to an embodiment of the present disclosure is shown.

[0035] Figure 10 A diagram of a matrix in a local memory arrangement according to an embodiment of the present disclosure is shown.

[0036] Figure 11 A diagram of a tensor calculation engine loading chunked data according to an embodiment of the present disclosure is shown.

[0037] Figure 12 A diagram of a local memory and a cache connection relationship according to an embodiment of the present disclosure is shown.

[0038] Figure 13 A diagram of thread synchronization according to an embodiment of the present disclosure is shown.

[0039] Figure 14 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0040] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numbers in different drawings represent the same or similar elements. Although various aspects of embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically noted.

[0041] The term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0042] The term "and / or", used herein only to describe the connection relationship of associated objects, means that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist together, and B exists alone. In addition, the term "at least one" herein means any one of the plurality or any combination of at least two of the plurality, for example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0043] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the specific embodiments below. Those skilled in the art should understand that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, elements and circuits well known to those skilled in the art are not described in detail in order to highlight the main idea of the present disclosure.

[0044] For most artificial intelligence models, such as neural network models involving multi-layer perceptron, matrix multiplication is one of the commonly used operations, especially in the calculation of various network layers. Figure 1 A structure diagram of a multi-layer perceptron in the related art is shown as follows: Figure 1 As shown, the multi-layer perceptron is a feedforward neural network commonly used for classification, regression, etc., which can include an input layer 11, a hidden layer 12, and an output layer 13. The input layer 11 is used to receive original feature data, and each input feature can correspond to a neuron. The hidden layer 12 can be composed of one or more dense layers (DenseLayer), and the neurons in each dense layer are connected to all neurons of the previous dense layer. The number of hidden layers 12 and the number of neurons in each layer can be used as hyperparameters of the multi-layer perceptron. The output layer 13 can be used to generate the final prediction result. For example, for a classification task, the number of neurons in the output layer 13 can be the same as the number of categories; for a regression task, the number of neurons in the output layer 13 is 1.

[0045] Figure 2 A schematic diagram of a dense layer in the related art is shown as follows: Figure 2As shown, the input tensor 21 is subjected to a matrix multiplication operation with the weight values 22 to obtain an output tensor 23, which can serve as the input tensor 21 of the next layer. In the related art, the operation of each layer divides the input tensor 21 into tile data of the same size and distributes the tile data to the entire processor, such as a graphics processing unit (GPU), a general purpose computing on graphics processing units (GPGPU), a neural network processing unit (NPU), etc. The size of the tile data is related to the resources of the computing units (e.g., kernels, which can correspond to neurons) in the processor.

[0046] Optionally, the processor can include a plurality of computing cores, each computing core including a plurality of computing unit clusters, each computing unit cluster including a plurality of computing units, each computing unit including a register, a local memory, a level one cache, a float process (FP), an integer processing unit (INT), a special function unit (SFU), a load store unit (LSU), a tensor memory engine (TME), and a tensor computing engine composed of a plurality of arithmetic logic units (ALUs) and control logic. The plurality of computing cores can correspond to one level three cache, the plurality of computing unit clusters can correspond to one level two cache, and each computing unit can have one corresponding level one cache.

[0047] The float process (FP) is used for single-precision or double-precision floating-point arithmetic; the integer processing unit (INT) is used for processing integer operations; the special function unit (SFU) can calculate various functions, such as inverse functions, square root functions, exponential functions, logarithmic functions, and activation functions, through numerical approximation methods, and can be used to improve the computing performance of the computing unit; the load store unit (LSU) is used to execute load instructions or store instructions; and the tensor memory engine (TME) can be used for data transfer between the memory and the register.

[0048] Figure 3 A schematic diagram of a processor in the related art is shown as follows: Figure 3As shown, the processor can be a GPU, which can include 4 computing cores, i.e., computing core 0~computing core 3, each of which can include 4 computing unit clusters, i.e., computing unit cluster 0~computing unit cluster 3, each of which can include 2 computing units, i.e., computing unit 0 and computing unit 1, each of which includes a tensor computing engine that can be composed of 4 arithmetic logic units (e.g., arithmetic logic unit 0~arithmetic logic unit 3) and control logic. Each computing unit further includes a local memory, a level 1 cache, and four sets of registers, a floating point processing component FP, an integer processing unit INT, a special function unit SFU, a load store component LSU, and a tensor memory component TME.

[0049] As shown, Figure 3 The GPU can synchronously process multiple threads, and a thread bundle formed by the threads can be distributed to each computing unit. The register size of each computing unit is 128KBx4=512KB, and the size of the local memory is 256KB.

[0050] Taking a matrix multiplication with a size of 64x64x64 as an example, the form of matrix calculation can be represented as D=AxB+C, where matrix A and matrix B can come from the local memory, matrix C can come from the register, and the multiplication result D can be written to the register. Assuming that the computing power of the tensor computing engine in each computing unit is 2048, 4096, and 8192 respectively, the bandwidth requirements of the computing unit for the register and the local memory are shown in Table 1.

[0051] Table 1:

[0052]

[0053] Referring to Table 1, when the computing power of the tensor computing engine in each computing unit is increased to 8192, the bandwidth requirement of the computing unit for the local memory cannot be met. At the same time, in order to reduce the bandwidth requirement for the global memory, a larger size of block data can be used, for example, 256x256x64. The latency hiding of a single computing unit under different computing powers is shown in Table 2. Latency hiding is used to hide the latency caused by communication, access to storage, conditional transfer judgment, etc. For example, the parallel processing capability of the system can be used to perform other operations while waiting for some operations to complete, thereby hiding the latency time of these operations.

[0054] Table 2:

[0055]

[0056] As shown in Table 2, assuming the data format is 16-bit floating point (FP16), the matrix A and the matrix B require 64 KB of local memory space, and for a local memory size of 256 KB, the maximum can accommodate three levels of cache of the matrix A (or the matrix B). When the computing power reaches 4096, the bandwidth between the computing unit and the second-level cache cannot meet the requirements, and therefore, the bandwidth can be increased to 128B / c (byte / cycle). When the computing power reaches 8192, the bandwidth cannot be continuously increased to hide the memory delay, and the complexity of the connection between the computing unit and the second-level cache cannot meet the requirements.

[0057] Figure 4 A schematic diagram of a matrix multiplication calculation process in the related art is shown as follows. Figure 4 As shown in the figure, assuming that the computing unit has three levels of cache, the data of two blocks of the matrix A (or the matrix B) can be continuously read first, and then the data of the first block of the matrix A (or the matrix B) is calculated while the data of the third block of the matrix A (or the matrix B) is read. When the calculation of the first block of data is completed, the first-level cache of the matrix A (or the matrix B) can be released, and the remaining data of the matrix A (or the matrix B) is read.

[0058] The scheme for increasing the computing power in the related art is to increase more computing cores and more chips, which is a weak expansion (weak scale) way to increase the speed-up ratio of the system. The higher the speed-up ratio, the better the parallel computing capability of the system. The weak expansion refers to keeping the efficiency of the system relatively constant when the system size (for example, the number of computing cores) and the task size increase at the same time. It can be seen that, in the case of proportional increase of the task size and the number of processors, the speed-up ratio is obtained by increasing the number of processors. For a large amount of tasks and high parallelism, the performance can be improved, such as increasing the training batch, or the user needs to re-write the code to achieve higher parallelism to adapt to the new hardware.

[0059] Another way to improve performance is to increase the performance of a single computing core to increase the performance of the system, but this scheme requires a larger bandwidth and a larger local memory size. The data processing method of the embodiment of the disclosure interconnects the tensor computing engines of M computing units to form a cluster of tensor computing engines, forms a larger array, and realizes larger data reuse. The larger the value of M, the stronger the data processing capability of the cluster of tensor computing engines as the computing core. Moreover, the M computing units share the tensor memory, reducing the demand for bandwidth, and at the same time, the local memory of each computing unit can be used to support the calculation of larger size block data (such as part of the first matrix), and under the same block data (such as part of the first matrix), more levels of cache can be supported to mask the delay of data transfer.

[0060] Figure 5 A flowchart of a data processing method according to an embodiment of the present disclosure is shown in FIG. 1. Figure 5 As shown in FIG. 1, the data processing method is applied to a tensor computing engine cluster in a processor, the processor includes a plurality of computing cores, each computing core includes a plurality of computing units, the tensor computing engine cluster is composed of a tensor computing engine interconnection of M computing units, a local memory or a register of each computing unit in the M computing units is used to store a partial first matrix, a tensor memory shared by the M computing units is used to store a second matrix, M is an integer greater than 1, and the data processing method includes the following steps.

[0061] In step S11, in response to that M tensor computing engines in the tensor computing engine cluster complete reading of the partial first matrix from the local memory or the register of the computing unit to which the tensor computing engine belongs and reading of the second matrix from the tensor memory, the M tensor computing engines respectively perform a matrix operation according to the partial first matrix and the second matrix to obtain an operation result corresponding to the tensor computing engine itself.

[0062] In step S12, the M tensor computing engines respectively write the operation result corresponding to the tensor computing engine itself into the register of the tensor computing engine.

[0063] In a possible implementation, the processor includes a plurality of computing cores, each computing core includes a plurality of computing units, and embodiments of the present disclosure do not limit the number of computing cores in the processor or the number of computing units in the computing core, which can be set according to actual application scenarios. The processor of the embodiments of the present disclosure can be newly designed, or can be obtained by improving an existing processor chip. The type of processor chip can include, but is not limited to, a central processing unit (CPU), a graphic processing unit (GPU), a general-purpose computing on graphics processing units (GPGPU), a neural network processing unit (NPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a tensor processing unit (TPU), a field programmable gate array (FPGA) or other programmable logic devices, and can also include a microprocessor or other conventional processor.

[0064] In one possible implementation, it is assumed that each computational core may include multiple computational unit clusters, and each computational unit cluster includes multiple computational units. Tensor computation engines of all or part of the computational units within a certain computational unit cluster can be interconnected to form a tensor computation engine cluster; or, tensor computation engines of all or part of the computational units within different computational unit clusters can be interconnected to form a tensor computation engine cluster.

[0065] Optionally, the tensor computation engine of each computation unit constituting the tensor computation engine cluster can work independently, and the tensor computation engine cluster consisting of M computation units can also work together. The embodiments of this disclosure do not limit this.

[0066] like Figure 3 As shown, the processor includes four computing cores, each computing core may include four computing unit clusters, and each computing unit cluster may include two computing units. In this case, the tensor computing engines of computing units 0 and 1 (M=2) in computing unit cluster 0 can be interconnected to form a tensor computing engine cluster; or, the tensor computing engines of computing units 0 and 1 in computing unit cluster 0, and computing units 0 and 1 in computing unit cluster 1 (M=4) can be interconnected to form a tensor computing engine cluster.

[0067] It should be understood that the embodiments of this disclosure do not limit the value of M, which may be 2, 4, or other values. The following description uses M=4 as an example.

[0068] In one possible implementation, the tensor computation engine includes multiple arithmetic logic units (ALUs), each of which is a multi-element multiply-accumulate unit array;

[0069] Figure 6 A schematic diagram of the structure of a tensor computation engine according to an embodiment of the present disclosure is shown, such as... Figure 6 As shown, the tensor computation engine of each computation unit consists of 4 arithmetic logic units (e.g., arithmetic logic unit 0 to arithmetic logic unit 3), and each arithmetic logic unit is an m×n×K multiply-accumulate unit array. Figure 6 The multiply-accumulate unit array shown in the figure has a size of 8×8×16. The specific values ​​of m, n and K are not limited in the embodiments disclosed herein.

[0070] like Figure 6 As shown, the tensor computation engine can read the data of matrix A and matrix B from the local memory inside its computation unit, read the information of matrix C from the register, and write the matrix D, which is the result of the computation, back to the register after the calculation is completed. Figure 6The structure of the middle tensor calculation engine is composed of a control logic unit, a reuse buffer A, a reuse buffer B, a scale coefficient unit, and four arithmetic logic units. Each arithmetic logic unit can include 8x8 data processing units DP, each data processing unit DP is an array of multiple element multiplication-addition units, which can be used to perform a product-sum operation, for example, A0x B0+ A1x B1+ A2x B2+…, wherein A0, A1, A2 represent data elements in matrix A, and B0, B1, B2 represent data elements in matrix B.

[0071] The control logic unit can connect each data processing unit DP in each arithmetic logic unit. In some embodiments, the control logic unit can number each data processing unit DP in the form of a two-dimensional matrix or a multi-dimensional matrix, so that the plurality of data processing units DP can be logically arranged in the form of a two-dimensional matrix or a multi-dimensional matrix, thereby better adapting to the logical operation of the matrix. The control logic unit can also connect the local memory of the calculation unit, and the control logic unit can be used for address calculation to load data from the local memory to the arithmetic logic unit and control the arithmetic logic unit to process the data to be processed.

[0072] In a possible implementation, the local memory can store the data to be processed and an executable program, and the executable program can include control instructions. The control logic unit can perform the control instructions to implement the logical operation of the data to be processed, such as matrix multiplication, convolution operation, and other multiplication-sum related operations.

[0073] In order to reduce the reading of repeated data, reduce the power consumption of the processor, and increase the communication bandwidth of the processor, the tensor calculation engine can also be provided with a reuse buffer (Reuse Buffer) and a scale coefficient unit. A loader and a decoder can be provided in the control logic unit. The loader can be used to load the data to be processed or part of the data to be processed in the local memory to the reuse buffer or the scale coefficient unit. The decoder can decode the control instructions for accessing data in the executable program according to the change of the storage address of the data to be processed after loading. For example, for a control instruction for accessing data X in the local memory, since the data X is cached in the reuse buffer (or the scale coefficient unit), the address of the data X stored in the reuse buffer (or the scale coefficient unit) can be obtained by decoding. The decoder can convert the control instruction for accessing the data X in the local memory into a control instruction for accessing the data X in the reuse buffer (or the scale coefficient unit), which is conducive to the subsequent control logic unit directly sending the cached data in the reuse buffer (or the scale coefficient unit) to the arithmetic logic unit for performing the product-sum operation.

[0074] In a possible implementation, in order to facilitate the processor to process large-scale data, such asFigure 6 As shown, the tensor computing engine can also communicate data with the registers inside the computing unit. In this way, the instruction execution time can be greatly reduced when the computing unit processes a large amount of data. The registers can include address registers, data registers, instruction registers, opcode word registers, flag registers, etc., and the present disclosure is not limited thereto. In an example, the control logic unit can be used for address calculation to facilitate data transfer between the arithmetic logic units and the registers.

[0075] In a possible implementation, the control logic unit can obtain the description information in the constant memory through the control instruction, and the description information can include, for example, the dimension of the data to be processed, the size of the data to be processed, the data type of the elements in the data to be processed (for example, integer type, single-precision floating-point type, double-precision floating-point type, character type, etc.), and information for describing the data to be processed, and the embodiments of the present disclosure are not limited thereto.

[0076] Figure 7 A schematic diagram of a tensor computing engine cluster according to an embodiment of the present disclosure is shown as follows: Figure 7 As shown, it is assumed that each computing unit cluster has four computing units, i.e., computing unit 0~computing unit 3, and the tensor computing engines of the computing unit 0~computing unit 3 can be arranged into a 4x1 array, i.e., a tensor computing engine cluster. The matrix A represents the first matrix, and the matrix B represents the second matrix. The local memory of each computing unit can be used to store part of the first matrix, for example, the local memory of the computing unit 0 can be used to store the first part of the first matrix A, the local memory of the computing unit 1 can be used to store the second part of the first matrix A, the local memory of the computing unit 2 can be used to store the third part of the first matrix A, and the local memory of the computing unit 3 can be used to store the fourth part of the first matrix A. The tensor memory shared by the computing unit 0~computing unit 3 in each computing unit cluster is used to store the second matrix B.

[0077] As shown in Figure 7 The first matrix A and the second matrix B can be transmitted between the tensor computing engines of different computing units, and the tensor memory shared by the computing unit 0~computing unit 3 in each computing unit cluster can store the second matrix B. Each tensor computing engine reads the corresponding part of the first matrix A from the local memory of the corresponding computing unit, and then the tensor computing engine cluster reads the second matrix B from the tensor memory and broadcasts it to each computing unit. The tensor computing engine of each computing unit is connected with the local memory inside the computing unit, avoiding cross connection. Compared with the related art, the newly added tensor memory shared by the computing unit 0~computing unit 3 not only increases the additional bandwidth, but also simplifies the architecture.

[0078] The following table shows the bandwidth requirement of different matrix sizes under the above structure. For example, referring to the last row of Table 3, the matrix A bandwidth of 256x64x64 is 1024 bytes, the average to each computing unit is 256 bytes, the matrix B bandwidth is 256 bytes, so the bandwidth of the configured tensor memory is 256 bytes.

[0079] Table 3:

[0080]

[0081] In step S11, the M tensor computing engines can synchronously read a partial first matrix from the local memory or the register of the computing unit to which the tensor computing engine belongs, read a second matrix from the tensor memory, in response to the M tensor computing engines in the tensor computing engine cluster completing reading the partial first matrix from the local memory or the register of the computing unit to which the tensor computing engine belongs, and reading the second matrix from the tensor memory, the M tensor computing engines respectively perform a matrix operation according to the partial first matrix and the second matrix to obtain a plurality of operation results.

[0082] In a possible implementation, each tensor computing engine is further configured to read a third matrix from the register of the computing unit to which the tensor computing engine belongs, and step S11 can include: the M tensor computing engines respectively perform a matrix multiplication operation according to the partial first matrix and the second matrix to obtain a plurality of product results; and the M tensor computing engines respectively perform a matrix addition operation according to the corresponding product result and the third matrix to obtain a plurality of operation results.

[0083] In this way, the tensor computing engine can implement a full connection layer operation, the first matrix can correspond to input data of the full connection layer, the partial first matrix can correspond to partial input data of the full connection layer, the second matrix can correspond to weights of the full connection layer, and the third matrix can correspond to bias of the full connection layer. Moreover, using the M tensor computing engines to perform the calculation simultaneously can make full use of the parallel computing capability of the tensor computing engine cluster, and significantly improve the calculation efficiency. Each tensor computing engine processes a partial matrix operation, which realizes decomposition and parallelization of the calculation task (for example, full connection layer calculation), and reduces the burden of a single tensor computing engine.

[0084] Figure 8 A schematic diagram of matrix and computing unit mapping according to an embodiment of the present disclosure is shown as follows. Figure 8 As shown in the figure, each of the computing units 0 to 3 calculates a 64x64xK size matrix, and the four computing units jointly complete a 256x64xK size matrix multiplication. The first matrix A can come from the local memory or the register, and the second matrix comes from the tensor memory. The third matrix C can come from the register, and the operation result D is written back to the register.

[0085] In step S11, the M tensor calculation engines can obtain respective operation results, where it is assumed that the matrix A is a first matrix, the matrix B is a second matrix, and the matrix C is a third matrix. The operation result obtained in step S11 can be A x B or A x B + C. In step S11, the M tensor calculation engines obtain respective operation results, and in step S12, the M tensor calculation engines write the respective operation results to the registers of the M tensor calculation engines, respectively. Since the neural network model is iteratively trained, the operation result of the current round of training can be used as the third matrix of the next round of training.

[0086] Through steps S11 to S12, the existing tensor calculation engine can be used, the structure of the tensor calculation engine is not changed, the tensor calculation engines of the multiple calculation units are interconnected, the shared tensor memory is added for the multiple calculation units, and the computing power can be increased without increasing the size and bandwidth of the local memory and the register in the calculation unit.

[0087] In a possible implementation, the processor further includes a task distribution module, and the method further includes: the task distribution module distributes the M thread bundles to the M calculation units; the M calculation units carry the second matrix from the same-level cache shared by the M calculation units to the tensor memory and carry respective portions of the first matrix from the local memory to the respective local memories according to the thread bundle received by the M calculation units; and the M calculation units respectively send the matrix operation instruction to the internal tensor calculation engine, so that the M tensor calculation engines read the portion of the first matrix from the local memory of the calculation unit and read the second matrix from the tensor memory according to the matrix operation instruction.

[0088] For example, assuming that the tensor computing engine is composed of tensor computing engine interconnections of the computing unit 0~computing unit 3, the task distribution module parallelly distributes the thread bundle 0 to the computing unit 0, the thread bundle 1 to the computing unit 1, the thread bundle 2 to the computing unit 2, and the thread bundle 3 to the computing unit 3; one of the computing unit 0~computing unit 3 can carry the second matrix from the shared same-level cache to the tensor memory according to the received thread bundle, while the computing unit 0 carries the first part of the first matrix from the shared same-level cache to the local memory of the computing unit 0 according to the received thread bundle 0, the computing unit 1 carries the second part of the first matrix from the shared same-level cache to the local memory of the computing unit 1 according to the received thread bundle 1, the computing unit 2 carries the third part of the first matrix from the shared same-level cache to the local memory of the computing unit 2 according to the received thread bundle 2, and the computing unit 3 carries the fourth part of the first matrix from the shared same-level cache to the local memory of the computing unit 3 according to the received thread bundle 3. Then, the computing unit 0~computing unit 3 can respectively send the matrix operation instructions to the corresponding four tensor computing engines, so that the four tensor computing engines read the partial first matrix from the local memory of the computing unit according to the respective received matrix operation instructions, and read the second matrix from the tensor memory.

[0089] In this way, the task distribution module distributes the M thread bundles as to-be-processed tasks to the M computing units in the same or different computing unit clusters. By distributing the M thread bundles to the M computing units, the M computing units can process the M threads in parallel, each computing unit can work independently, process the received thread bundle, simultaneously process different data segments, and thus significantly improve the computing efficiency. The M computing units carry the second matrix from the shared same-level cache to the tensor memory and carry the partial first matrix to the respective local memories, which helps to reduce memory access conflicts and delays, and at the same time, the shared tensor memory for the M computing units can increase the computing power without increasing the size and bandwidth of the local memory.

[0090] In a possible implementation, the M computing units carry the second matrix from the shared same-level cache to the tensor memory and respectively carry the partial first matrix to the respective local memories according to the received thread bundles, including: the computing unit distributes the thread groups in the received thread bundle to at least one execution unit; the execution unit generates an access request according to the received thread group, and the access request is used to carry the second matrix from the shared same-level cache to the tensor memory and carry the partial first matrix to the local memory.

[0091] Each computing unit can include a plurality of execution units, see Figure 3Each execution unit can include a floating point processing component (FP), an integer processing unit (INT), a special function unit (SFU), a load store component (LSU), a tensor memory component (TME), etc. Embodiments of the present disclosure do not make specific limitations on the specific components of the execution unit.

[0092] In this way, in each computing unit, multiple execution units can process multiple thread groups in a thread bundle in parallel, further improving the parallelism of the tensor computing engine cluster.

[0093] For example, the task distribution module can be used to provide a distribution mechanism. When the task distribution module is enabled, for example, a user can use a preset application programming interface (api SquadWGMMA), the task distribution mode will enter an enabled state, and the task distribution module will distribute the next 4 (M=4) thread bundles to the 4 computing units of a certain computing unit cluster in a round robin mode. Each thread bundle can include four thread groups, and each thread group can be composed of 32 threads, that is, each thread bundle can include 4x32 threads. For each thread bundle, the four thread groups in the thread bundle can be distributed to the four execution units of each computing unit, and the execution units can generate access requests according to the received thread groups. The access requests generated by different execution units can update the local memory and the tensor memory in the corresponding computing unit at the same time. It should be understood that embodiments of the present disclosure do not make specific limitations on the number of thread bundles, the number of thread groups included in each thread bundle, and the number of threads included in each thread group, which can be set according to actual application scenarios.

[0094] Figure 9 A schematic diagram of the arrangement of a matrix in a register according to an embodiment of the present disclosure is shown as follows: Figure 9 As shown, in a thread group, the size of a register of each thread is 32 bits, for example, thread group 0 can correspond to 32x2 threads, each thread can correspond to a register, and the size of each register in T0~T31 is 32 bits. Ra represents an 8x4 structure composed of registers of 32 threads in thread group 0, and Ra+1 represents another 8x4 structure composed of registers of another 32 threads in thread group 0.

[0095] The register can be used to store a third matrix, the third matrix can be an operation result of the first matrix and the second matrix corresponding to the previous round of training, and the bandwidth of the register is the same as the number of elements of the first dimension of the part of the first matrix and the number of elements of the second dimension of the second matrix. In this way, the bandwidth of the register (i.e., the number of elements that can be transmitted at a time) can be reasonably set, and the transmission delay caused by the register can be reduced.

[0096] Figure 10A schematic diagram of the matrix in the local memory arrangement is shown in FIG. 1, according to an embodiment of the present disclosure. Figure 10 As shown, in order to avoid bandwidth competition, the data needs to be swizzled, which is a remapping from physical address to logical address, and makes the data of the same matrix scattered in different rows by coordinate change. It should be understood that, Figure 10 The numbers 0-15 in the figure represent different matrices.

[0097] Optionally, the partial first matrix corresponding to each computing unit can come from the local memory or from the register. Different thread bundles in the computing unit cluster can read data from the display memory or the same level cache according to the arrangement of the local memory or the register of different computing units as shown in FIG. 2. Figure 9 、 Figure 10 Each thread bundle carries the first matrix coordinates and the second matrix coordinates, and the data coordinates accessed by each thread group in the thread bundle can be configured. The data is accessed by the load store unit (LSU) of each execution unit in the computing unit. Alternatively, a tensor memory unit (TME) is used to specify the start coordinates and the size of the block data to be accessed (for example, the partial first matrix A0 or the partial first matrix A1) for processing.

[0098] Optionally, the data synchronization can be performed by the tensor memory unit (TME) of each execution unit in the computing unit through an asynchronous barrier. The tensor memory unit (TME) adds support for the tensor memory, and the tensor memory unit (TME) can directly transfer data from the global memory of the processor to the tensor memory. In addition, for the access to the tensor memory, read-write instructions between the tensor memory and the register can be added, such as the instruction for transferring data from the tensor memory to the register (Tensor Memory to Register, T2R) and the instruction for transferring data from the register to the tensor memory (Register to Tensor Memory, R2T), which can transfer data between the tensor memory and the register.

[0099] Further, each execution unit in the computing unit can select a certain thread group to generate an instruction executable by the tensor memory unit (TME), specify the start coordinates and the size of the block data, such as the partial first matrix A0, the partial first matrix A1, the partial first matrix A2, the partial first matrix A3, and the like.

[0100] Figure 11 A schematic diagram of the tensor computing engine loading the block data is shown in FIG. 3, according to an embodiment of the present disclosure. Figure 11As shown, the description information of the first matrix can include a size dim0 of a horizontal dimension of the first matrix, a size dim1 of a vertical dimension of the first matrix, and a stride stride0 required for jumping from one element to the next element immediately in the specified dimension. The description information of the partial first matrix to be accessed by the current tensor computing engine or the cluster of tensor computing engines can include a size block_dim of a horizontal dimension of the partial first matrix, a size block_dim1 of a vertical dimension of the partial first matrix, and a position coordinate pos of the partial first matrix in the first matrix.

[0101] In a possible implementation, the M tensor computing engines read the partial first matrix from the local memory of the computing unit and read the second matrix from the tensor memory according to the matrix operation instruction, including: in response to the execution of the matrix operation instruction by all execution units of each tensor computing engine, the tensor computing engine marks a first synchronization signal in a synchronization counter; and in response to detecting that the synchronization counter marks M first synchronization signals, a second synchronization signal is generated, the second synchronization signal being used to indicate that the M tensor computing engines complete reading the partial first matrix from the local memory of the computing unit and reading the second matrix from the tensor memory.

[0102] Figure 12 A schematic diagram of a connection relationship between a local memory and a cache according to an embodiment of the present disclosure is shown in FIG. 3. Figure 12 As shown, for the computing unit 0 to the computing unit 3, the local memory of each computing unit can be connected to a same-level cache (for example, a second-level cache) through a bus. Four thread groups in each computing unit respectively send a matrix operation instruction to a tensor computing engine. The matrix operation instruction can carry an encoding of a thread bundle to which the thread group belongs, and the tensor computing engine reads address information of a partial first matrix in a local memory and address information of a second matrix in a tensor memory from four execution units according to the encoding of the thread bundle.

[0103] Figure 13 A schematic diagram of thread synchronization according to an embodiment of the present disclosure is shown in FIG. 4. Figure 13 As shown, the sending of the matrix operation instruction needs to wait for the corresponding first matrix and second matrix data to be ready, and is generally guaranteed by a data dependency mechanism such as a synchronization counter. When the same thread bundle on the four execution units of each computing unit in the computing unit 0 to the computing unit 3 all execute the matrix operation instruction, a first synchronization signal is marked in the synchronization counter for the synchronization completion of one computing unit. When the four computing units of one computing unit cluster all synchronize, the multiple groups of thread bundles of the entire computing unit cluster also synchronize, and the synchronization counter can generate a second synchronization signal.

[0104] In this way, the thread bundle synchronization across the computing units can be implemented in a hierarchical manner, that is, the thread synchronization across the computing units is implemented. The first synchronization signal is used for the synchronization of different thread groups in the same thread bundle, and the second synchronization signal is used for the synchronization between different thread bundles.

[0105] In a possible implementation, the M tensor computing engines write the corresponding operation results into their own registers respectively, including: in response to the M tensor computing engines all writing the corresponding operation results into their own registers, clearing the first synchronization signal and the second synchronization signal in the synchronization counter. By clearing the synchronization signal, it can be ensured that the tensor computing engine cluster can process the matrix operation task of the next round of training in time, avoiding mutual interference between the matrix operation tasks between adjacent training rounds.

[0106] Exemplarily, the tensor computing engine cluster can read according to the data arrangement described above, then perform the calculation, write the corresponding operation results into the registers of the respective MPs when the calculation is completed, and then release the first synchronization signal and the second synchronization signal in the synchronization counter. In this way, the subsequent steps dependent on the matrix operation instruction can start to be executed. Figure 9 、 Figure 10 Exemplarily, the tensor computing engine cluster can read according to the data arrangement described above, then perform the calculation, write the corresponding operation results into the registers of the respective MPs when the calculation is completed, and then release the first synchronization signal and the second synchronization signal in the synchronization counter. In this way, the subsequent steps dependent on the matrix operation instruction can start to be executed.

[0107] To sum up, the data processing method of the embodiment of the present disclosure can interconnect the tensor computing engines of the M computing units to form a tensor computing engine cluster, form a larger array, and realize larger data multiplexing. Moreover, the M computing units share the tensor memory, reducing the demand for bandwidth, and at the same time, the local memory or register of each computing unit can be used, which can support the calculation of larger size of block data (for example, part of the first matrix), and under the same block data (for example, part of the first matrix), support more levels of cache to mask the delay of data transfer. In addition, the thread bundle synchronization across the computing units can also be implemented in a hierarchical manner.

[0108] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other without violating the principle logic to form combined embodiments. Limited by the length of the present disclosure, the present disclosure will not be described again. Those skilled in the art can understand that in the above-mentioned method of the specific implementation, the specific execution order of each step should be determined according to its function and possible internal logic.

[0109] The embodiment of the present disclosure further provides a processor, which comprises a plurality of computing cores, each of which comprises a plurality of computing units, tensor computing engines of M computing units in the plurality of computing units are interconnected to form a tensor computing engine cluster, a local memory or a register of each of the M computing units is used to store a partial first matrix, a tensor memory shared by the M computing units is used to store a second matrix, and M is an integer greater than 1; in response to M tensor computing engines in the tensor computing engine cluster completing reading of the partial first matrix from the local memory or the register of the computing unit to which the M tensor computing engines belong and reading of the second matrix from the tensor memory, the M tensor computing engines are configured to respectively perform matrix operations according to the partial first matrix and the second matrix to obtain corresponding operation results of the M tensor computing engines; and the M tensor computing engines are configured to respectively write the corresponding operation results into the registers of the M tensor computing engines.

[0110] In a possible implementation, the processor further comprises a task distribution module, the task distribution module distributes M thread bundles to the M computing units; the M computing units carry the second matrix from a same-level cache shared by the M computing units to the tensor memory and respectively carry the partial first matrix corresponding to the M computing units to the local memories of the M computing units according to the thread bundles received by the M computing units; and the M computing units respectively send matrix operation instructions to the internal tensor computing engines, so that the M tensor computing engines read the partial first matrix from the local memory of the computing unit to which the M tensor computing engines belong and read the second matrix from the tensor memory according to the matrix operation instructions.

[0111] In a possible implementation, the M tensor computing engines reading the partial first matrix from the local memory of the computing unit to which the M tensor computing engines belong and reading the second matrix from the tensor memory according to the matrix operation instructions comprises: in response to each tensor computing engine completing the matrix operation instruction, the tensor computing engine marks a first synchronization signal in a synchronization counter; and in response to detecting that the synchronization counter marks M first synchronization signals, a second synchronization signal is generated, and the second synchronization signal is used to indicate that the M tensor computing engines complete reading of the partial first matrix from the local memory of the computing unit to which the M tensor computing engines belong and reading of the second matrix from the tensor memory.

[0112] In a possible implementation, the M tensor computing engines are specifically configured to: in response to the M tensor computing engines writing the corresponding operation results into the registers of the M tensor computing engines, clear the first synchronization signal and the second synchronization signal in the synchronization counter.

[0113] In a possible implementation, the tensor computing engine includes a plurality of arithmetic logic units, each of which is a multiply-add unit array of multiple elements; and the bandwidth of the register is the same as the number of elements of the first dimension of the partial first matrix and the number of elements of the second dimension of the second matrix.

[0114] In a possible implementation, each tensor computing engine is further configured to read a third matrix from the register of the computing unit to which the tensor computing engine belongs, and the M tensor computing engines are configured to: perform matrix multiplication according to the partial first matrix corresponding to each of the M tensor computing engines and the second matrix, to obtain a plurality of product results; and perform matrix addition according to the corresponding product result and the third matrix, to obtain a plurality of operation results.

[0115] In a possible implementation, the M computing units are configured to, according to the thread bundle received by each of the M computing units, transfer the second matrix from the same-level cache shared by the M computing units to the tensor memory, and transfer the partial first matrix corresponding to each of the M computing units to the local memory corresponding to each of the M computing units, including: the computing unit is configured to distribute the thread group in the thread bundle received by the computing unit to at least one execution unit; and the execution unit is configured to, according to the thread group received by the execution unit, generate an access request, the access request being used to transfer the second matrix from the same-level cache shared by the M computing units to the tensor memory, and transfer the partial first matrix to the local memory.

[0116] According to an aspect of the present disclosure, a chip is provided, which includes the processor as described above.

[0117] According to an aspect of the present disclosure, a graphics card is provided, which includes the processor as described above.

[0118] In addition, the present disclosure also provides a data processing apparatus, an electronic device, a computer readable storage medium, and a program, which can be used to implement any of the data processing methods provided by the present disclosure, and the corresponding technical solutions and descriptions are referred to the method part and will not be repeated here.

[0119] The present disclosure also provides a computer readable storage medium having computer program instructions stored thereon, which are executed by a processor to implement the above method. The computer readable storage medium can be a volatile or non-volatile computer readable storage medium.

[0120] The present disclosure also provides an artificial intelligence chip, which includes the processor as described above.

[0121] The embodiments of the present disclosure further provide a graphics card, which comprises the processor as described above. The graphics card, also known as display adapter, graphics processing card or video card, is a hardware component in a computer responsible for graphics rendering and display output. The graphics card can comprise a discrete graphics card, an integrated graphics card, a mobile graphics card, a professional graphics card, a cluster graphics card, etc., and the embodiments of the present disclosure do not limit the category of the graphics card.

[0122] The embodiments of the present disclosure further provide an intelligent computing center (also known as a server cluster), which comprises the processor as described above.

[0123] The embodiments of the present disclosure further provide an electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the above method.

[0124] The embodiments of the present disclosure further provide a computer program product, comprising computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, when the computer-readable code is run in the processor of an electronic device, the processor in the electronic device performs the above method.

[0125] The electronic device can be provided as a terminal, a server or other forms of devices. Among them, the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc., and the embodiments of the present disclosure do not limit this.

[0126] Figure 14 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 can be provided as a server or a terminal device. Referring to Figure 14 , the electronic device 1900 comprises a processing component 1922, which further comprises one or more processors, and a memory resource represented by a memory 1932, for storing instructions executable by the processing component 1922, such as an application program. The application program stored in the memory 1932 can comprise one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above method.

[0127] The electronic device 1900 can further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958. The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as Microsoft Windows Server TM , Apple's graphical user interface-based operating system (Mac OS X TM ), a multi-user multi-processing computer operating system (Unix TM ), a free and open-source Unix-like operating system (Linux TM ), an open-source Unix-like operating system (FreeBSD TM ), or the like.

[0128] In an exemplary embodiment, there is also provided a non-transitory computer readable storage medium, such as the memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above-described method.

[0129] The present disclosure can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0130] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a

[0131] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0132] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computing / processing device, partly on the user's computing / processing device, as a stand-alone software package, partly on the user's computing / processing device and partly on a remote computing / processing device or entirely on the remote computing / processing device or server. In the latter scenario, the remote computing / processing device can be connected to the user's computing / processing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing / processing device, for example, through the Internet using an Internet Service Provider. In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0133] The computer readable program instructions can also be loaded onto a computing / processing device, other programmable data processing apparatus, or other device to cause a series of operations to be performed on the computing / processing device, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computing / processing device, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0134] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer readable storage medium having no data signals on it. The instructions can be executed by one or more processors to produce a computer-implemented process such that the instructions, which execute via one or more processors of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0135] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable data processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0136] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical functions (‘instructions’). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0137] The computer program product can be embodied by a hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied by a computer storage medium, and in another optional embodiment, the computer program product is embodied by a software product, such as a software development kit (SDK) and the like.

[0138] The above description of the various embodiments is directed to highlighting the differences between the various embodiments, and the same or similar parts can be mutually referred to for brevity, which will not be repeated herein.

[0139] Those skilled in the art can understand that, in the above method of the specific embodiment, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined according to its function and possible internal logic.

[0140] If the technical solution of the present application involves personal information, the product applying the technical solution of the present application has been explicitly informed of the personal information processing rules before processing the personal information, and has obtained the personal independent consent. If the technical solution of the present application involves sensitive personal information, the product applying the technical solution of the present application has obtained the personal independent consent before processing the sensitive personal information, and at the same time meets the requirement of "explicit consent". For example, at the personal information collection device such as camera, a clear and prominent mark is set to inform that the personal information collection range has been entered and the personal information will be collected. If the person voluntarily enters the collection range, it is regarded as agreeing to collect the personal information. Or, on the device for processing personal information, the personal information processing rules are informed by using obvious marks / information, and the personal authorization is obtained by means of pop-up information or asking the person to upload his / her personal information. The personal information processing rules can include personal information processor, personal information processing purpose, processing method, and personal information type, etc.

[0141] The above has described various embodiments of the present disclosure, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical application, or improvement of technology in the market, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

1. A data processing method, characterized by, The data processing method is applied to a tensor computing engine cluster in a processor, the processor comprising a plurality of computing cores, each computing core comprising a plurality of computing units, the tensor computing engine cluster being formed by a tensor computing engine interconnection of M computing units, a local memory or a register of each of the M computing units being used to store a partial first matrix, and a tensor memory shared by the M computing units being used to store a second matrix, M being an integer greater than 1; The method comprises: In response to the M tensor computing engines in the tensor computing engine cluster completing reading of the partial first matrix from the local memory or the register of the computing unit, and reading of the second matrix from the tensor memory, the M tensor computing engines respectively perform matrix operations according to the partial first matrix and the second matrix to obtain respective operation results. The M tensor computing engines respectively write the respective operation results into the registers thereof.

2. The method of claim 1, wherein, The processor further comprises a task distribution module, and the method further comprises: The task distribution module distributes M thread bundles to the M computing units; The M computing units respectively send matrix operation instructions to the internal tensor computing engines, so that the M tensor computing engines read the partial first matrix from the local memory of the computing unit and read the second matrix from the tensor memory according to the matrix operation instructions. The M tensor computing engines reading the partial first matrix from the local memory of the computing unit and reading the second matrix from the tensor memory according to the matrix operation instructions comprises:

3. The method of claim 2, wherein, In response to each tensor computing engine completing execution of the matrix operation instruction, the tensor computing engine marks a first synchronization signal in a synchronization counter; In response to detecting that the synchronization counter marks M first synchronization signals, a second synchronization signal is generated, the second synchronization signal being used to indicate that the M tensor computing engines have completed reading of the partial first matrix from the local memory of the computing unit and reading of the second matrix from the tensor memory. The M tensor computing engines respectively writing the respective operation results into the registers thereof comprises:

4. The method of claim 3, wherein, In response to the M tensor computing engines writing the respective operation results into the registers thereof, the first synchronization signal and the second synchronization signal in the synchronization counter are cleared. The tensor computing engine comprises a plurality of arithmetic logic units, each arithmetic logic unit being a multiply-add unit array of multiple elements; 5. The method of claim 1, wherein, The bandwidth of the register is the same as the number of elements of the first dimension of the partial first matrix and the number of elements of the second dimension of the second matrix. ​ 6. The method according to any one of claims 1 to 5, characterized in that, Each tensor computation engine is further configured to read a third matrix from a register of the corresponding compute unit, and perform a matrix operation according to the partial first matrix and the second matrix to obtain a corresponding operation result. The M tensor computation engines perform a matrix multiplication operation according to the partial first matrix and the second matrix to obtain a plurality of product results. The M tensor computation engines perform a matrix addition operation according to the product result and the third matrix to obtain a plurality of operation results.

7. The method according to any one of claims 2 to 5, characterized in that, The M compute units are configured to, according to a thread bundle received by the M compute units, transfer the second matrix from a same-level cache shared by the M compute units to the tensor memory, and transfer the partial first matrix corresponding to each of the M compute units to a local memory of the corresponding compute unit, including: The compute unit is configured to distribute a thread group in the received thread bundle to at least one execution unit. The execution unit is configured to, according to a thread group received by the execution unit, generate an access request, and the access request is configured to transfer the second matrix from the same-level cache shared by the M compute units to the tensor memory, and transfer the partial first matrix to the local memory.

8. A processor, comprising: The processor includes a plurality of compute cores, each compute core including a plurality of compute units, and tensor computation engines of M compute units in the plurality of compute units are interconnected to form a tensor computation engine cluster, a local memory or a register of each compute unit in the M compute units is configured to store a partial first matrix, and a tensor memory shared by the M compute units is configured to store a second matrix, and M is an integer greater than 1. In response to M tensor computation engines in the tensor computation engine cluster completing reading of the partial first matrix from a local memory or a register of a corresponding compute unit and reading of the second matrix from the tensor memory, the M tensor computation engines are configured to perform a matrix operation according to the partial first matrix and the second matrix to obtain a corresponding operation result. The M tensor computation engines are configured to write the corresponding operation result to a register of the corresponding compute unit.

9. A chip or graphics card, characterized by The chip or graphics card includes the processor of claim 8.

10. An electronic device, comprising: The electronic device includes the chip or graphics card of claim 9.

Images