A data processing method, apparatus and device, and a storage medium

Abstract

The application discloses a data processing method and device, equipment and a storage medium, and relates to the technical field of data processing. The method comprises the following steps: acquiring matrix multiplication input data; acquiring a plurality of block matrix data based on the matrix multiplication input data; storing each block matrix data in the buffer; acquiring a loading instruction; transmitting each block matrix data from the buffer to the memory based on the loading instruction; and calculating a block multiplication result matrix corresponding to each block matrix data through the tensor processor based on each block matrix data in the memory. The application splits the matrix multiplication input data to obtain block matrix data that is adapted to the hardware processing characteristics of the tensor processor, thereby avoiding the problem of increased operation time caused by the mismatch between the data dimension and the characteristics of the tensor processor due to the excessive size of the matrix multiplication input data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a data processing method, device, equipment and storage medium. BACKGROUND

[0002] The traditional central processing unit (CPU) implements matrix multiplication usually in a three-layer loop nesting manner, and sequentially traverses the rows and columns of the matrix to perform dot product calculation. However, the CPU has a limited number of cores and mainly uses serial calculation, which results in extremely slow speed when processing large-scale matrices. Taking a simple example, when multiplying two matrices with a dimension of 1000x1000, the CPU needs to perform millions of loop iterations, and the calculation time is difficult to meet the real-time requirement. The graphics processing unit (GPU) can simultaneously process multiple threads for matrix multiplication due to its parallel computing capability, and each thread is responsible for calculating an element of the result matrix. However, the GPU implementation also has problems such as low memory access efficiency and large thread synchronization overhead. In actual applications, complex synchronization mechanisms are needed among multiple thread blocks to coordinate the calculation process, which increases the programming complexity, and in large-scale data processing, the communication and data transmission among thread blocks will significantly affect the performance. These existing technical solutions have their own characteristics, and directly applying these technical solutions to the tensor processing unit (TPU) will have obvious limitations. For example, the instruction set, memory hierarchy and parallel computing model of the TPU are different from those of the CPU and GPU, and directly porting these technical solutions to the TPU will not be able to take full advantage of the hardware advantages of the TPU, and may even lead to a decrease in performance. SUMMARY

[0003] The present application aims to provide a data processing method, device, equipment and storage medium to solve the technical problem of low operation efficiency when performing large-scale matrix multiplication.

[0004] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0005] In a first aspect, the present application provides a technical solution of a data processing method, which is applied to a data processing device, the data processing device comprising a tensor processing unit, a memory and a cache, and the method comprising:

[0006] Obtaining matrix multiplication input data;

[0007] Based on the matrix multiplication input data, a plurality of block matrix data are obtained; each block matrix data comprises a left-hand matrix and a right-hand matrix, and the size of each block matrix data is a positive integer multiple of the systolic array size of the tensor processor;

[0008] Each block matrix data is stored in the cache;

[0009] A load instruction is obtained; the load instruction is used to indicate that at least one block matrix data in the cache is transmitted to the memory;

[0010] Based on the load instruction, each block matrix data is transmitted from the cache to the memory;

[0011] Based on each block matrix data in the memory, a block multiplication result matrix corresponding to each block matrix data is calculated by the tensor processor; the block multiplication result matrix is equal to the product of the left-hand matrix and the right-hand matrix in the corresponding block matrix data;

[0012] If all block matrix data in the cache are transmitted to the memory, and each block matrix data obtains a corresponding block multiplication result matrix, then based on each block multiplication result matrix, matrix multiplication output data are obtained.

[0013] In a second aspect, a technical scheme of a data processing apparatus is provided, the data processing apparatus comprises a tensor processor, a memory and a cache, and the data processing apparatus further comprises:

[0014] A reader is used to obtain matrix multiplication input data;

[0015] The tensor processor is used to obtain a plurality of block matrix data based on the matrix multiplication input data; each block matrix data comprises a left-hand matrix and a right-hand matrix, and the size of each block matrix data is a positive integer multiple of the systolic array size of the tensor processor;

[0016] Each block matrix data is stored in the cache;

[0017] A load instruction is obtained; the load instruction is used to indicate that at least one block matrix data in the cache is transmitted to the memory;

[0018] Based on the load instruction, each block matrix data is transmitted from the cache to the memory;

[0019] and the tensor processor calculates a block multiplication result matrix corresponding to each block matrix data based on each block matrix data in the memory.

[0020] If all the block matrix data in the buffer are transmitted to the memory and each block matrix data obtains a corresponding block multiplication result matrix, the matrix multiplication output data is obtained based on each block multiplication result matrix.

[0021] In a third aspect, a computer readable storage medium is provided, and a computer program is stored in the computer readable storage medium. When the computer program is executed by a processor, the data processing method in the first aspect is implemented.

[0022] Compared with the prior art, the beneficial effects of the present application are:

[0023] The present application splits the matrix multiplication input data to obtain block matrix data that adapts to the hardware processing characteristics of the tensor processor, avoids the problem of increased operation time caused by the mismatch between the data dimension and the characteristics of the tensor processor due to the large size of the matrix multiplication input data. Moreover, the buffer is used to store the block matrix data in advance, reducing the dependence on low-speed external storage and improving the data access efficiency. Relying on the parallel computing advantage of the tensor processor for small matrices, the large-scale matrix multiplication operation efficiency is greatly improved, and the core problem that the prior art cannot fully utilize the advantages of the tensor processor is solved. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 A flowchart of a data processing method according to an embodiment of the present application is shown.

[0025] Figure 2 A structural diagram of a data processing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] The terms "first", "second", and the like in the description and in the claims of the present application and the above drawings merely mean different ones of similar objects (for example, a first buffer and a second buffer represent different buffers respectively, other similar ones), and do not necessarily mean a particular order or sequence. It should be understood that the names thus used can be interchanged under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "comprise" and "have" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or modules does not have to be limited to those steps or modules clearly listed, but can include other steps or modules not clearly listed or inherent to such processes, methods, products, or devices. The division of modules in the embodiments of the present application is only a logical division, and in actual application, other division manners can be used, for example, a plurality of modules can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed modules can be through some interfaces, indirect coupling between the modules, and the communication connection can be electrical or other similar forms, which are not limited in the embodiments of the present application. In addition, the modules or sub-modules described as separate components can or can not be physically separated, can or can not be physical modules, or can be distributed into a plurality of circuit modules, and some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0027] In order for those skilled in the art to understand the embodiments of the present application, it is also necessary to clearly understand the definitions of the following professional terms:

[0028] Tensor Processing Unit (TPU): Tensor Processing Unit is a customized application-specific integrated circuit chip specially designed for machine learning tasks. The core goal of TPU is to optimize the hardware architecture for matrix operations to provide energy efficiency and computing density far exceeding Central Processing Unit (CPU) and Graphics Processing Unit (GPU). It adopts a systolic array architecture, driven by data flow, to achieve efficient pipeline calculation of matrix multiplication and reduce memory access delay. At the same time, TPU supports low-precision calculations such as BF16, FP16, and INT8, which can reduce the number of transistors required for each operation without significantly sacrificing accuracy, thereby reducing energy consumption.

[0029] Random Access Memory (RAM): Random Access Memory is the core hardware in a computer that temporarily stores data and programs, and is the key bridge between the TPU and external storage devices (e.g., mechanical hard drives or solid state drives). Its core feature is random access, which means that the TPU can directly read or write data at any address without the need to retrieve it in order, making it much faster than external storage devices.

[0030] Cache: Cache is a high-speed temporary storage module in a computer system, and its core role is to solve the speed mismatch between high-speed processors (e.g., CPU, TPU, or GPU, etc.) and low-speed memory or storage devices, avoiding the processor from being idle due to waiting for data, and improving overall efficiency. The working logic of the cache is based on the principle of locality, which means that when a computer is running, the data or instructions that the processor frequently accesses in the near future are often concentrated in a small area (e.g., frequently called program code or recently processed file fragments, etc.). The cache will copy these high-frequency data from the low-speed storage device in advance, and when the processor needs the high-frequency data again, it does not need to be slowly read from the low-speed storage device, but directly obtained from the cache which is much faster, greatly shortening the data waiting time.

[0031] Matrix multiplication refers to the multiplication of two matrices (e.g., matrix multiplication input data in the following) to generate a third matrix (e.g., matrix multiplication output data in the following), and its core is the dot product operation of the rows of the first matrix (e.g., left-hand matrix in the following) and the columns of the second matrix (e.g., right-hand matrix in the following), and the operation result needs to meet the strict dimension matching rule. Matrix multiplication is not element-by-element multiplication, but through the "row-by-column" dot product rule to generate a new matrix, and there are explicit requirements for the dimensions of the matrices involved in the operation (i.e., only when the number of columns of the left-hand matrix equals the number of rows of the right-hand matrix, can the left-hand matrix and the right-hand matrix perform matrix multiplication). For example, matrix A is an m x n dimensional matrix (i.e., matrix A is an m row, n column matrix), matrix B is an n x p dimensional matrix (i.e., matrix B is an n row, p column matrix), and m, n and p are positive integers greater than or equal to 1, then matrix A and matrix B can perform matrix multiplication. Assuming that the result of matrix multiplication of matrix A and matrix B is matrix C, then matrix C is an m x p dimensional matrix (i.e., matrix C is an m row, p column matrix). Specifically, the calculation formula of each element in matrix C is as follows:

[0032]

[0033] wherein, represents the element value of the i-th row and j-th column in matrix C; i is greater than or equal to 1 and less than or equal to m; j is greater than or equal to 1 and less than or equal to p. represents the value of the element in the i-th row, k-th column of matrix A; represents the value of the element in the k-th row, i-th column of matrix B.

[0034] The left-hand matrix refers to the matrix located on the left side of the multiplication sign in matrix multiplication (for example, matrix A in the foregoing); the right-hand matrix refers to the matrix located on the right side of the multiplication sign in matrix multiplication (for example, matrix B in the foregoing).

[0035] To solve the technical problem that it is difficult to fully exert the advantages of a tensor processor and the operation efficiency is not high when performing large-scale matrix multiplication operation, an embodiment of a data processing method is provided. The data processing method is applied to a data processing device, and the data processing device includes a tensor processor, a memory, and a cache.

[0036] In this embodiment, the memory is a storage unit for the TPU to store key data (for example, the block matrix data in the following) of matrix multiplication and temporary calculation results (for example, the first result matrix or the second result matrix in the following). That is, the memory is a transit source of core data (that is, the key data or the temporary calculation results in the foregoing) of matrix multiplication calculation. Its core role is to receive data strongly related to matrix multiplication calculation transmitted from the cache, and to provide stable data support for subsequent calculation. In this embodiment, the type of the memory is not limited, for example, the memory can be a static random access memory (SRAM), a dynamic random access memory (DRAM), or a virtual memory (VMEM), and the like. In order to adapt to the storage of the matrix, the memory can also be a vector memory.

[0037] In this embodiment, the cache is a high-speed storage unit for the TPU to temporarily store matrix multiplication input data (for example, each block matrix data in the following). Its core role is to cache all the block matrix data in advance, and then efficiently transmit the required block matrix data to the memory according to the calculation requirement, so as to avoid the high delay problem of directly reading data from the external storage. In this embodiment, the type of the cache is not limited, for example, the cache can be a level one cache, a level two cache in the tensor processor, or a solid state disk or a high bandwidth memory (HBM) in the computer, and the like.

[0038] In this embodiment, as shown in Figure 1 The data processing method includes steps 100 to 700.

[0039] Step 100: obtaining matrix multiplication input data.

[0040] In the embodiment, the matrix multiplication input data refers to a raw data set participating in a matrix multiplication operation, which is the initial input of the entire data processing flow. Each matrix multiplication operation is composed of two matrices that need to be multiplied. That is, in the embodiment, the matrix multiplication input data at least includes the raw data set of two matrices that need to be multiplied. In other words, in the embodiment, the raw data set of the two matrices that need to be multiplied in the matrix multiplication input data is not limited. For example, the matrix multiplication input data can be the raw data set formed by the matrix A and the matrix B; or the matrix multiplication input data can be the raw data set formed by the transpose matrix of the matrix A and the matrix B; or the matrix multiplication input data can be the raw data set formed by the transpose matrix of the matrix A and the matrix B; or the matrix multiplication input data can be the raw data set formed by the transpose matrix of the matrix A and the transpose matrix of the matrix B. In the following embodiments, in order to avoid redundancy, only the raw data set formed by the matrix A and the matrix B is taken as an example for description, which does not mean that the matrix multiplication input data can only be the raw data set formed by the matrix A and the matrix B, and the following will not be described in detail.

[0041] Step 200: obtaining a plurality of block matrix data based on the matrix multiplication input data.

[0042] In the embodiment, the purpose of obtaining a plurality of block matrix data based on the matrix multiplication input data is to divide a large matrix (for example, the matrix A or the matrix B in the above) into a plurality of small matrices (i.e., the left-hand matrix and the right-hand matrix in the following), and to replace the direct multiplication of the large matrix with the multiplication and addition between the small matrices, which is essentially a "block level" reuse of the matrix multiplication rule. When dividing the large matrix into small matrices, as long as the "dimension matching" condition of the matrix multiplication is met. For example, taking the matrix A and the matrix B in the above as an example, the matrix A and the matrix B are divided into four blocks.

[0043] In an embodiment of the application, the expression of the division of the matrix A can be as follows:

[0044]

[0045] wherein, represents the matrix A; to respectively represent four small matrices that constitute the matrix A.

[0046] Based on the expression of the division of the matrix A, the expression of the division of the matrix B is as follows:

[0047]

[0048] in, Represent matrix B; to These represent the four smaller matrices that make up matrix B.

[0049] Based on the expressions for the partitioning of matrices A and B, the expression for matrix C is as follows:

[0050]

[0051] In this embodiment, the matrix multiplication of two large matrices (i.e., matrix A and matrix B) is divided into matrix multiplication of multiple smaller matrices. To ensure that the smaller matrices also satisfy the "dimensionality matching" requirement for matrix multiplication, when partitioning matrix A and matrix B, the matrix... The number of rows needs to be equal to the matrix The number of columns in a matrix; The number of rows also needs to be equal to the matrix. The number of columns in a matrix; The number of rows needs to be equal to the matrix The number of columns in a matrix; The number of rows also needs to be equal to the matrix. The number of columns; ..., and so on. In this embodiment, based on the matrix multiplication input data, eight smaller matrices can be obtained, which are respectively matrix ... To matrix ,matrix To matrix These 8 smaller matrices yield 8 block matrix data, which are as follows:

[0052] The left-hand matrix is ​​a matrix The right-hand matrix is ​​a matrix The resulting block matrix data;

[0053] The left-hand matrix is ​​a matrix The right-hand matrix is ​​a matrix The resulting block matrix data;

[0054] The left-hand matrix is ​​a matrix The right-hand matrix is ​​a matrix The resulting block matrix data;

[0055] The left-hand matrix is ​​a matrix The right-hand matrix is ​​a matrix The resulting block matrix data;

[0056] The left-hand matrix is ​​a matrix The right-hand matrix is ​​a matrix The resulting block matrix data;

[0057] The left-hand matrix is matrix , and the right-hand matrix is matrix The block matrix data formed thereby;

[0058] The left-hand matrix is matrix , and the right-hand matrix is matrix The block matrix data formed thereby;

[0059] The left-hand matrix is matrix , and the right-hand matrix is matrix The block matrix data formed thereby.

[0060] In another embodiment of the present application, the expression of the matrix A division can be as follows:

[0061]

[0062] wherein, represents the matrix A; to respectively represent the four block small matrices constituting the matrix A.

[0063] Based on the expression of the matrix A division, the expression of the matrix B division is as follows:

[0064]

[0065] wherein, represents the matrix B; to respectively represent the four block small matrices constituting the matrix B.

[0066] Based on the expression of the matrix A and the matrix B division, the expression of the matrix C is as follows:

[0067]

[0068] In the present embodiment, the matrix multiplication calculation of two large matrices (i.e. the matrix A and the matrix B) is divided into the matrix multiplication calculation of a plurality of small matrices. In order to enable the small matrices to also satisfy the "dimension matching" of the matrix multiplication, when the matrix A and the matrix B are segmented, the number of rows of the matrix needs to be equal to the number of columns of the matrix ; the number of rows of the matrix needs to be equal to the number of columns of the matrix ; and so on. In the present embodiment, based on the matrix multiplication input data, eight small matrices can be obtained, which are the matrix to the matrix , the matrix to the matrix ; four block matrix data can be obtained through the eight small matrices, which are respectively:

[0069] The left-hand matrix is matrix , and the right-hand matrix is matrix The block matrix data formed by the left-hand matrix and the right-hand matrix are as follows:

[0070] The left-hand matrix is matrix , and the right-hand matrix is matrix The block matrix data formed by the left-hand matrix and the right-hand matrix are as follows:

[0071] The left-hand matrix is matrix , and the right-hand matrix is matrix The block matrix data formed by the left-hand matrix and the right-hand matrix are as follows:

[0072] The left-hand matrix is matrix , and the right-hand matrix is matrix The block matrix data formed by the left-hand matrix and the right-hand matrix are as follows.

[0073] As can be seen from the above, in the embodiment, each block matrix data includes a left-hand matrix and a right-hand matrix.

[0074] It should be noted that, in the embodiment, the division of the matrix A and the matrix B into four blocks is only illustrative, and it does not mean that the matrix A and the matrix B can only be divided into four blocks in the embodiment of the application. It should be understood that the matrix A and the matrix B can be divided into the required number of blocks according to the requirements. For example, the matrix A and the matrix B can also be divided into 5 blocks, 15 blocks, 128 blocks, or 256 blocks, etc. As long as the matrix A and the matrix B are divided into blocks, the size of each block matrix data is a positive integer multiple of the systolic array size of the tensor processor. In the embodiment, the core purpose of dividing the size of each block matrix data into a positive integer multiple of the systolic array size of the tensor processor is to completely adapt the specifications of the block matrix data to the hardware architecture of the systolic array of the tensor processor, fully exert the advantages of the systolic array pipeline calculation, specifically, on the one hand, it can avoid the idle of hardware resources caused by the mismatch between the size of the block matrix data and the size of the systolic array, prevent the situation of the idling of part of the operation units in the systolic array, on the other hand, it can reduce the waiting and adaptation overhead of data transmission in the array, avoid the additional time consumption caused by data splitting or padding operation, at the same time, guarantee the continuity and efficiency of data flow in the matrix multiplication operation process, thereby maximizing the parallel computing efficiency of the tensor processor on the block matrix data, and helping to reduce the overall time consumption of large-scale matrix multiplication operation.

[0075] In this embodiment, the time length consumed by the data processing apparatus in performing matrix multiplication operation on each of the plurality of block matrix data to obtain the matrix multiplication output data is defined as a first operation time consumption. The time length consumed by the data processing apparatus in directly performing matrix multiplication operation on the matrix multiplication input data to obtain the matrix multiplication output data is defined as a second operation time consumption. That is, in this embodiment, the plurality of block matrix data can be obtained based on the matrix multiplication input data in any reasonable manner, as long as the first operation time consumption is less than the second operation time consumption. For example, the data amount of each block matrix data can be equal to a positive integer multiple of a second capacity (which can be equal to a positive integer multiple of the size of the systolic array of the tensor processor) in the following description; or each block matrix data can be equal to the capacity of the first buffer or the second buffer (which can be equal to a positive integer multiple of the size of the systolic array of the tensor processor), and the like.

[0076] It is found through a large number of studies that as long as the matrix multiplication input data with a large data amount is divided into a plurality of block matrix data, and then the TPU is used to perform matrix multiplication operation, the first operation time consumption can be less than the second operation time consumption. Compared with the block matrix data with a size that is not equal to a positive integer multiple of the size of the systolic array of the tensor processor, the block matrix data with a size equal to a positive integer multiple of the size of the systolic array of the tensor processor can be fully adapted to the hardware architecture of the systolic array of the tensor processor, further making the first operation time consumption less than the second operation time consumption.

[0077] It should be noted that in a large-scale matrix multiplication operation application scenario, the data amount of the matrix multiplication input data is often greater than the capacity of the memory device, and the block matrix data needs to simultaneously meet the core requirements of “adaptation to the storage capability of the memory device” and “adaptation to the parallel computing characteristics of the tensor processor”. If the matrix multiplication input data is directly divided to obtain a plurality of block matrix data, the block division scheme may not meet the hardware cooperation requirements. For example, if the maximum capacity of the memory device, the minimum storage capacity of the memory device and other hardware parameters are not fully considered, the directly generated block matrix data may cause data overflow due to the data amount exceeding the capacity of the memory device, or the tensor processor hardware resources may be idle due to the excessive number of blocks (for example, the data amount of the block matrix data is less than the capacity of the minimum storage unit of the memory device), and the parallel computing advantage cannot be efficiently utilized. In order to ensure that the obtained block matrix data can match the upper limit of the memory device storage to avoid data overflow, and can adapt to the tensor processor to improve the operation efficiency, and avoid problems such as increased operation time consumption or increased data transmission time consumption caused by defects in the block division scheme, in the embodiments of the present application, step 200 of obtaining a plurality of block matrix data based on the matrix multiplication input data can include steps 210 to 250.

[0078] Step 210: Obtain a first capacity, a second capacity and a first data amount.

[0079] As can be seen from the following, in the application scenario of large-scale matrix multiplication operation, since the total data amount of each block matrix data is greater than the capacity of the memory device, that is, the memory device cannot load all the block matrix data at one time, in order to avoid data overflow, each block matrix data can only be transmitted from the cache device to the memory device in batches. In this embodiment, the first capacity refers to the maximum storage space capacity of the memory device for storing each block matrix data from the cache device. For example, in the embodiment in which the data processing apparatus in the following performs batch processing logic of "transmitting a batch, calculating a batch, and releasing a batch" to gradually complete the transmission and calculation of all block matrix data, the first capacity can be the maximum capacity of the memory device; in the embodiment in which the memory device is divided into a first buffer area and a second buffer area (the capacities of the first buffer area and the second buffer area are equal) to realize parallel processing of the transmission and calculation of the block matrix data, the first capacity can be the capacity of the first buffer area or the second buffer area.

[0080] In this embodiment, the second capacity is the capacity of the smallest storage unit in the memory device. The first data amount is the size of the storage space required when storing the matrix multiplication input data.

[0081] Step 220: Obtain a first quantity based on the first capacity and the second capacity.

[0082] In this embodiment, the first quantity is equal to the integer part of the ratio of the first capacity to the second capacity. In the field of data processing, calculating the ratio of two values (i.e., the first capacity and the second capacity) and taking the integer part is a mature technology, which will not be described here.

[0083] It should be noted that in this embodiment, the ratio of the first capacity to the second capacity is taken to obtain the first quantity, and the core purpose is to adapt to the hardware storage limit of the memory device and ensure that the block matrix data can be stably stored in the memory device. The first capacity is the capacity of a single buffer area (i.e., the first buffer area or the second buffer area) in the memory device, and the second capacity is the smallest storage capacity of the memory device. Taking the integer part can avoid the number of smallest storage units required by the block matrix data exceeding the actual carrying capacity of a single buffer area, preventing data storage overflow caused by the mismatch between the data amount of the block matrix data and the carrying capacity of a single buffer area. For example, in one specific embodiment of the present application, if the smallest storage capacity (i.e., the second capacity) of the memory device is 64K and the capacity (i.e., the first capacity) of a single buffer area is 1000k, the first quantity is 15, that is, a single buffer area can store block matrix data with a data amount of not greater than 15 smallest storage capacities. In other words, if the data amount of the block matrix data is greater than 15 smallest storage capacities, data overflow will occur when storing the block matrix data in the buffer area.

[0084] Step 230: obtaining a second quantity based on the first data quantity and the second capacity.

[0085] In the embodiment, the second quantity is equal to the ratio of the first data quantity and the second capacity rounded up. In the field of data processing, calculating the ratio of two values (i.e., the first data quantity and the second capacity) and rounding up is a mature technology, which will not be described here.

[0086] It should be noted that in the embodiment, the ratio of the first data quantity and the second capacity is rounded up to obtain the second quantity, and the core purpose is to adapt to the integrity storage requirement of the matrix multiplication input data, to ensure that each block matrix data can fully cover the matrix multiplication input data, and to avoid missing part of the matrix multiplication input data due to the limitation of the minimum storage unit of the memory, so as to provide complete and accurate data basis for subsequent block processing and tensor processor operation. For example, in a specific embodiment of the present application, if the minimum storage capacity (second capacity) of the memory is 64K, and the total storage requirement (first data quantity) of the matrix multiplication input data is 2000K. At this time, the ratio of the first data quantity and the second capacity is 2000÷64=31.25, and if it is directly rounded or rounded down, only 31 minimum storage units can be obtained, and the corresponding total storage capacity is 31x64=1984K, which cannot accommodate 2000K of matrix multiplication input data, and 16K of data will be lost due to the lack of storage units; and after rounding up, the second quantity is 32, and the corresponding total storage capacity is 32x64=2048K, which can completely accommodate 2000K of input data and adapt to the hardware limitation of the minimum storage unit of the memory, avoiding the error of subsequent block matrix multiplication operation result caused by data loss. In other words, if the second quantity is not obtained by rounding up, but by rounding down or other ways, there may be a problem that part of the input data cannot be stored, which further leads to the damage of the accuracy of the entire matrix multiplication operation.

[0087] Step 240: obtaining a target quantity based on the first quantity and the second quantity.

[0088] In the embodiment, the target quantity can be the ratio of the second quantity and the first quantity rounded up. When the matrix multiplication input data is blocked in step 250, as long as the number of blocks into which matrix A and matrix B in the matrix multiplication input data are divided is greater than or equal to the target quantity, each block matrix data transmitted from the buffer to the memory will not overflow (i.e., the data quantity of the block matrix data is less than the maximum capacity of the memory).

[0089] It needs to be clear that in the present embodiment, the ratio of the second quantity to the first quantity is rounded up to obtain the target quantity, the core purpose is to adapt the upper limit of the single buffer of the memory device (i.e. the first buffer or the second buffer) to the block transmission demand of the matrix multiplication input data, to ensure that the block matrix data can be stably transmitted to the memory device in batches, avoid data overflow or hardware resource idle problem caused by unreasonable planning of matrix multiplication input data block batches, and provide scientific batch basis for subsequent parallel transmission and calculation of block matrix data. In the present embodiment, the first quantity is the upper limit of the number of minimum storage units that a single buffer can carry, which determines the maximum storage size of each batch of block matrix data; the second quantity is the total number of minimum storage units required to store the complete matrix multiplication input data, which clearly defines the total storage demand of the overall data. Rounding up the ratio of the two, can ensure that the number of block batches is sufficient to cover the overall storage demand of the matrix multiplication input data, and then all block matrix data can adapt to the storage capacity of a single buffer. For example, if the first quantity is 15 (i.e. a single buffer can hold at most 15 block data of minimum storage units), the second quantity is 32 (i.e. a total of 32 minimum storage units are required to store the matrix multiplication input data), the ratio of the two is 32 ÷ 15 ≈ 2.13. If directly rounding or rounding down can only get 2 batches, at most can carry 2 × 15 = 30 minimum storage units of block data, cannot accommodate all 32 minimum storage units of matrix multiplication input data, will cause 2 minimum storage units corresponding data cannot be transmitted, and then cause the matrix multiplication input data missing; and the target quantity obtained by rounding up is 3 batches, 3 × 15 = 45 minimum storage units of carrying capacity, which can completely accommodate all block matrix data corresponding to 32 minimum storage units, and can adapt to the upper limit of the storage capacity of a single buffer, avoiding data overflow caused by exceeding the buffer capacity when transmitting block data. In other words, if the target quantity is not obtained by rounding up, but by rounding down or other ways, there may be problems that part of the block matrix data cannot adapt to the buffer storage capacity and cannot be normally transmitted, which will hinder the continuity of the entire matrix multiplication operation.

[0090] As can be seen from the foregoing, the memory device is a storage unit for storing key data (i.e., block matrix data) of matrix multiplication and temporary calculation results (i.e., block multiplication result matrices) of the TPU. In the embodiment, if the number of blocks into which the matrix A and the matrix B in the matrix multiplication input data are divided is equal to the target number, only one block matrix data can be transmitted to the memory device each time, and the data amount of the block matrix data exactly matches the upper limit of the storage capacity that the memory device can currently bear. Although the block matrix data will not overflow due to the data amount exceeding the limit, the memory device cannot store temporary calculation results any more. That is, the temporary calculation results obtained in the calculation process can only be stored in other storage units (such as a cache or an external hard disk) different from the memory device. When the temporary calculation results need to be used in the calculation process, the temporary calculation results need to be read from the storage units, and the read / write speed of such storage units is much lower than that of the memory device, which will additionally increase the data transmission time. Although this way can avoid data overflow, it will increase the transmission link between the temporary calculation results inside the tensor processor and the external storage, and thus the first operation time is relatively long. That is, in the process of determining the target number based on the first number and the second number, not only the storage and processing of the matrix multiplication input data (i.e., each block matrix data) need to be considered, but also the storage and processing of the matrix multiplication output data (i.e., each block multiplication result matrix and other temporary calculation results) need to be considered at the same time, so as to effectively reduce the first operation time. For example, if the storage space required by the matrix multiplication output data is not fully considered and the target number is derived based only on the matrix multiplication input data, the total data amount of the block multiplication result matrices generated subsequently may exceed the upper limit of the memory device storage, causing the matrix multiplication output data storage overflow; or the tensor processor needs to frequently read, splice and convert data when integrating multiple block multiplication result matrices, which increases the additional time consumption. In order to obtain a target number that can not only adapt to the block processing of the matrix multiplication input data to avoid overflow of the block matrix data during transmission, but also be compatible with the storage and integration of the matrix multiplication output data to reduce the redundant overhead of subsequent block multiplication result matrix operation and data processing, so as to reduce the first operation time, in the embodiment of the present application, step 240 of obtaining a target number based on the first number and the second number can include steps 241 to 245.

[0091] Step 241: Obtain a second data amount.

[0092] In the embodiment, the second data amount is the storage space required when storing the matrix multiplication output data. In matrix calculation, the data amount (i.e., the second data amount) of the matrix multiplication output data can be calculated according to the data amount (i.e., the first data amount) of the matrix multiplication input data, and the calculation process is a mature technology, which will not be described here.

[0093] Step 242: obtaining a third quantity based on the second data quantity and the second capacity.

[0094] In the embodiment, the third quantity is equal to the ratio of the second data quantity and the second capacity rounded up. In the field of data processing, calculating the ratio of two values (i.e., the second data quantity and the second capacity) and rounding up is a mature technology, which will not be described herein.

[0095] Step 243: obtaining a fourth quantity and a fifth quantity based on the first quantity, the second quantity and the third quantity.

[0096] In the embodiment, the fourth quantity is equal to the ratio of the second quantity and the first quantity rounded up. The fifth quantity is equal to the ratio of the third quantity and the first quantity rounded up. In the field of data processing, calculating the ratio of two values (i.e., the second quantity and the first quantity; or, the third quantity and the first quantity) rounded up is a mature technology, which will not be described herein.

[0097] In the embodiment, the fourth quantity is equal to the second quantity and the first quantity, and the core purpose is to accurately adapt the storage upper limit of the single buffer of the memory device to the block transmission requirement of the matrix multiplication input data, to provide scientific batch planning basis for the block of the matrix multiplication input data, and to avoid data transmission abnormalities caused by unreasonable block batch. In the embodiment, the first quantity determines the upper limit of the number of minimum storage units that a single buffer (i.e., the first buffer or the second buffer) can carry, and determines the maximum storage size of each batch of block matrix data; the second quantity represents the minimum total number of storage units required to store the complete matrix multiplication input data, and defines the overall storage requirement of the input data. The ratio of the two is rounded up, which can ensure that the number of planned block batches is sufficient to completely cover the overall storage requirement of the input data, so that each batch of block matrix data can adapt to the storage capacity of the single buffer. For example, if the first quantity is 15 (i.e., the single buffer can accommodate at most 15 block data of minimum storage units), the second quantity is 32 (i.e., a total of 32 minimum storage units are required to store the matrix multiplication input data), and the ratio of the two is 32 ÷ 15 ≈ 2.13. If direct rounding or rounding down is used, only 2 batches can be obtained, which can carry at most 2 × 15 = 30 block data of minimum storage units, and cannot accommodate all 32 minimum storage units of the matrix multiplication input data, which will cause 2 minimum storage units of data to be omitted, and further cause the subsequent calculation data to be missing; and the fourth quantity obtained by rounding up is 3 batches, and the carrying capacity of 3 × 15 = 45 minimum storage units can completely accommodate all block matrix data corresponding to the input data, and can strictly adapt to the storage upper limit of the single buffer, avoiding data overflow caused by exceeding the buffer capacity during block data transmission. Conversely, if the fourth quantity is not obtained by rounding up, some block matrix data may not be able to adapt to the buffer storage capacity and cannot be normally transmitted, which will hinder the continuity of the entire matrix multiplication operation.

[0098] In the embodiment, the fifth quantity is equal to the upward rounding of the ratio of the third quantity and the first quantity, and the core purpose is to adapt the storage upper limit of the memory single buffer (i.e. the first buffer or the second buffer) to the storage requirement of the matrix multiplication output data, to provide a reasonable batch basis for the storage planning of the output data, and to avoid the problems of storage overflow or data loss caused by improper adaptation of the matrix multiplication output data block. Among them, the first quantity is still the upper limit of the number of minimum storage units that a single buffer can carry, which determines the maximum storage size of a single batch of data; the third quantity is the total number of minimum storage units required to store the complete matrix multiplication output data, which clearly defines the overall storage requirement of the matrix multiplication output data. The upward rounding of the ratio of the two can ensure that the planned batch quantity can completely cover the storage requirement of the output data, so that the block multiplication result matrix always adapts to the storage capacity of the single buffer in the storage process. For example, if the first quantity is 15 (i.e. a single buffer can accommodate at most 15 block data of minimum storage units), the third quantity is 64 (i.e. a total of 64 minimum storage units are required to store the matrix multiplication output data), and the ratio of the two is 64 ÷ 15 ≈ 4.27. If downward rounding is used, only 4 batches can be obtained, which can carry at most 4 × 15 = 60 minimum storage units of output data, and cannot accommodate all 64 minimum storage units of block multiplication result matrix, which will cause the output data corresponding to 4 minimum storage units to be unable to be stored, and further cause the final matrix multiplication output data to be incomplete; and the fifth quantity obtained by upward rounding is 5 batches, and the carrying capacity of 5 × 15 = 75 minimum storage units can completely accommodate all block multiplication result matrices and adapt to the storage upper limit of the single buffer, avoiding the storage overflow caused by the matrix multiplication output data exceeding the buffer capacity. Conversely, if the fifth quantity is not obtained by upward rounding, the block multiplication result matrix may not be able to adapt to the buffer storage capacity and cannot be stably stored, which affects the integrity of the matrix multiplication output data and the subsequent integration efficiency.

[0099] Step 244: based on the fourth quantity and the fifth quantity, obtaining a sixth quantity.

[0100] In the embodiment, the sixth quantity is equal to the maximum value of the fourth quantity and the fifth quantity.

[0101] It should be noted that the sixth quantity is the maximum of the fourth quantity and the fifth quantity, and the core purpose is to enable the target quantity obtained in step 245 to simultaneously adapt to the storage and processing requirements of the matrix multiplication input data and output data, and to guarantee the overall compatibility and hardware adaptability of the blocking scheme. The fourth quantity is a lower limit of the blocking batch derived based on the input data storage requirement, which determines the upper limit of the storage of the input data after blocking to adapt to the single buffer area of the memory, avoiding overflow of the blocked matrix data during transmission; the fifth quantity is a lower limit of the blocking batch derived based on the output data storage requirement, which ensures that the blocked multiplication result matrix can be completely stored in the memory and adapted to subsequent integration. Taking the maximum of the two as the sixth quantity can simultaneously meet the dual requirements of input data transmission safety and output data storage integrity, and eliminate the problems of input data omission or output data overflow caused by insufficient blocking batches, providing a benchmark for subsequent determination of the target quantity that takes into account the bidirectional requirements of input and output, and laying a foundation for the number of blocks for efficient promotion of large-scale matrix multiplication operations.

[0102] Step 245: Based on the sixth quantity, the target quantity is obtained.

[0103] In this embodiment, the target quantity can be an integer multiple of the sixth quantity. For example, the target quantity can be twice or three times the sixth quantity, etc. If the target quantity is twice the sixth quantity, each batch can transmit one blocked matrix data from the cache to the memory, and the intermediate results obtained when the blocked matrix data is subjected to matrix multiplication operation can all be stored in the memory without overflow.

[0104] This embodiment optimizes the blocking logic from the dimensions of matrix multiplication input data and output data dual storage requirements through steps 241 to 245, avoids the problem of insufficient output data storage adaptability caused by determining the number of blocks only according to the input data from the source, and finally obtains a target quantity that can adapt the storage capacity of the memory for the blocked matrix data, and the blocked multiplication result matrix can also match the storage upper limit of the memory and the integration requirement of the tensor processor. That is, the obtained target quantity can not only eliminate the risk of storage overflow caused by the blocked multiplication result matrix exceeding the capacity of the memory due to not considering the output data, but also avoid the additional splicing and conversion time caused by the tensor processor when integrating the result matrix due to the insufficient adaptability of the output data and the number of blocks, providing a precise blocking quantity basis for subsequent efficient completion of matrix multiplication operations.

[0105] It should be noted that the memory (for example, the vector memory) stores the matrix data according to the data of each row or each column in the matrix data. Taking the matrix data stored according to each row of data as an example, if the capacity of the minimum storage unit of the memory (that is, the second capacity in the foregoing) is x bytes, each row of the matrix data has y elements, and each element occupies z bytes, the calculation formula (hereinafter referred to as the first calculation formula) of the number of minimum storage units required by each element in each row of the matrix data to store is as follows:

[0106]

[0107] wherein, represents the number of minimum storage units of the memory required by each row of data in the matrix (for example, the left-hand matrix in the blocked matrix data) to store; y represents the number of elements of each row of data in the matrix; z represents the number of bytes occupied by each element in the matrix to store; and x represents the capacity (in bytes) of the minimum storage unit of the memory. represents the upward rounding of the value in the parentheses. It can be known from the first calculation formula that the more the elements of each row in the matrix data, the more the minimum storage units required to store all the elements of each row of the matrix data. In the matrix multiplication calculation, the tensor processor needs to continuously read the elements of each row of the left-hand matrix and the elements of the corresponding column of the right-hand matrix. The more the minimum storage units occupied by each row of the left-hand matrix to store, the more scattered the elements of each row of the left-hand matrix are stored. The more scattered the elements of each row of the left-hand matrix are stored, the more likely it is that the memory access of each row of the left-hand matrix is not continuous in the subsequent calculation of the blocked multiplication result matrix, thereby increasing the number of cache misses and reducing the calculation efficiency.

[0108] It should be noted that, due to the characteristics that the memory device usually stores the matrix data in a row or column manner, and the minimum storage unit has a fixed capacity limit, if only the sixth quantity is used to determine the target block quantity, the distribution of row elements of the left-hand matrix and the distribution of column elements of the right-hand matrix after blocking are not fully considered, and the problem of scattered storage of data in the same row or column is easy to occur. For example, if the elements of a single row of the left-hand matrix are too many, it will cause each row of elements to occupy more minimum storage units of the memory device, and the storage location is scattered, and when the subsequent tensor processor reads the data, it is easy to cause the cache miss number to increase due to the discontinuous memory access, and the problem of reducing the calculation efficiency. If the elements of a single column of the right-hand matrix are too many, similar data reading delay problems will also occur. In order to ensure that the finally determined target quantity can adapt to the storage requirements of the matrix multiplication input data and output data, optimize the storage continuity of the blocked matrix data in the memory device, improve the access hit rate and calculation efficiency of the tensor processor, and avoid the increase of operation time caused by unreasonable blocking dimension, in the embodiments of the present application, step 245, based on the sixth quantity, obtaining the target quantity, can include steps 245a and 245b.

[0109] Step 245a: based on the sixth quantity, obtaining a first value and a second value.

[0110] In the present embodiment, the first value is the matrix row value corresponding to the left-hand matrix after the sixth quantity is used as a candidate target quantity. The second value is the matrix column value corresponding to the right-hand matrix after the sixth quantity is used as a candidate target quantity.

[0111] Step 245b: if the first value and the second value are both less than or equal to a fifth preset value, the sixth quantity is used as the target quantity; otherwise, the sixth quantity, the first value and the second value are updated; until the first value and the second value obtained by updating are both less than or equal to the fifth preset value, the sixth quantity obtained by the last update is used as the target quantity.

[0112] It should be noted that in the embodiment, the smaller the sixth quantity is, the larger the single block matrix data obtained based on the matrix multiplication input data is (i.e., the smaller the number of each block matrix data is), although in the subsequent data transmission process, the number of each block matrix data to be transmitted is smaller, and thus the data transmission time is shorter, but in the subsequent calculation process, since the block matrix data is larger, the elements of each row in the block matrix data are more scattered in storage, and the cache miss times are more likely to increase, and the calculation efficiency is reduced. The larger the sixth quantity is, the smaller the single block matrix data obtained based on the matrix multiplication input data is (i.e., the larger the number of each block matrix data is), although in the subsequent calculation process, since the block matrix data is smaller, the elements of each row in the block matrix data are more concentrated in storage, and the cache miss is less likely to occur, and thus the calculation efficiency is improved, but in the subsequent data transmission process, the number of each block matrix data to be transmitted is larger, and thus the data transmission time is longer. In the embodiment of the present application, the fifth preset value can be set according to actual needs. For example, in the embodiment in which the data calculation and the data transmission are parallel, if the time consumed by the data calculation and the time consumed by the data transmission are greatly different, the reasonable fifth preset value can be set to make the time consumed by the data calculation and the time consumed by the data transmission consistent. For example, if the time consumed by the data calculation is much greater than the time consumed by the data transmission, the fifth preset value can be reduced, so that the data amount of each block matrix data obtained is reduced, and the number of each block matrix data is increased. If the time consumed by the data calculation is much less than the time consumed by the data transmission, the fifth preset value can be increased, so that the data amount of each block matrix data obtained is increased, and the number of each block matrix data is reduced.

[0113] In order to make the size of the block matrix data obtained subsequently reasonable, so that the data transmission time and the data calculation time are approximately equal, in an embodiment of the present application, the calculation formula of the fifth preset value can be as follows:

[0114]

[0115] wherein, the fifth preset value; z represents the number of bytes occupied by the storage of each element in the matrix data to be stored; x represents the capacity (in bytes) of the smallest storage unit of the memory; represents the upward rounding of the data in the parentheses. In the embodiment, if z is 2 bytes and x is 512 bytes, the fifth preset value is 15360.

[0116] It is found through a large number of verifications that, generally, when the data amount of the block matrix data is equal to the capacity of 60 minimum storage units, the data transmission time and the data calculation time are approximately equal. If the data transmission time and the data calculation time are approximately equal, the parallel synergy effect of data transmission and data calculation can be maximized, and the idle waiting time of the tensor processor can be reduced. Under the double buffer architecture, when the block matrix data in one buffer is being calculated by the tensor processor, the other buffer can complete the transmission of the next batch of block matrix data, and the data calculation process and the data transmission process seamlessly connect and advance without gaps. In this case, the overall time consumption of the large-scale matrix multiplication operation is not limited by any single link, avoiding the problem of calculation core idling due to data transmission lag or the problem of data transmission backlog due to slow data calculation, so as to optimize the time complexity of the entire data processing process and improve the overall efficiency of the large-scale matrix multiplication operation.

[0117] The embodiment further calibrates the target number from the data calculation and data transmission angles through the settings of steps 245a and 245b, and the finally determined target number can ensure that the row value of the left-hand matrix after blocking and the column value of the right-hand matrix are adapted to the memory storage characteristics and the tensor processor calculation requirements. That is, the obtained target number can not only avoid the problem of discontinuous memory access caused by the dispersion of left-hand matrix row elements and the dispersion of right-hand matrix column elements, reduce the number of cache misses to improve data reading efficiency, but also can prevent the problems of reduced calculation efficiency or increased data transmission time caused by unreasonable blocking dimensions, and provide accurate protection for the subsequent efficient storage of block matrix data and the full play of the parallel calculation advantages of the tensor processor.

[0118] Step 250: Block the matrix multiplication input data based on the target number to obtain a plurality of block matrix data.

[0119] In the embodiment, the blocking of the matrix multiplication input data based on the target number to obtain a plurality of block matrix data can refer to the division of the matrix A and the matrix B in the above, which is not repeated here.

[0120] The embodiment sets up through steps 210 to 250, from hardware parameter adaptation, block quantity optimization to block execution landing, constructs a complete matrix multiplication input data block logic, excludes the incompatibility problem of block scheme and data processing device (including tensor processor, memory, cache) from the source, and finally obtains a plurality of block matrix data which can meet the core needs of adapting memory storage capacity and adapting tensor processor parallel computing characteristics, and is compatible with the subsequent storage and integration requirements of matrix multiplication output data. That is, steps 210 to 250 can not only avoid the data overflow risk caused by block matrix data exceeding the memory capacity, but also can eliminate the problem that the number of cache misses of the tensor processor increases and the parallel computing advantage cannot be played due to unreasonable block dimension, and can also be compatible with the storage and integration requirements of the subsequent block multiplication result matrix, and provides full-process block scheme guarantee for efficient promotion of large-scale matrix multiplication operation.

[0121] Step 300: store each block matrix data in the cache.

[0122] It needs to be clear that storing data (i.e. each block matrix data) into a storage module (i.e. cache) is a mature technology, which is not described here.

[0123] Step 400: obtain a loading instruction.

[0124] In the embodiment, the loading instruction is used to indicate that at least one block matrix data in the cache is transmitted to the memory.

[0125] It should be noted that in the application scenario of large-scale matrix multiplication operation, since the total data amount of each block matrix data is greater than the capacity of the memory device, that is, the memory device cannot load all the block matrix data at one time, in order to avoid data overflow, in an embodiment of the present application, the block matrix data can be transmitted from the cache device to the memory device one by one. That is, in the present embodiment, after a batch (that is, at least one) of block matrix data is transmitted from the cache device to the memory device, the tensor processor calculates the corresponding matrix multiplication result (that is, the block multiplication result matrix) based on the batch of block matrix data, and then releases the batch of block matrix data from the memory device to free up the storage space of the memory device; the next batch of block matrix data is transmitted from the cache device to the memory device, and the same matrix multiplication operation process as the previous batch of block matrix data is performed based on the batch of block matrix data through the tensor processor to obtain the corresponding block multiplication result matrix, and then the memory space occupied by the batch of block matrix data is also released. Such a cycle is repeated, and the batch processing logic of "transmitting a batch, calculating a batch, and releasing a batch" is followed to gradually complete the transmission and calculation of all block matrix data, which not only ensures that the memory device will not overflow due to the data amount exceeding the capacity, but also relies on the cache device to pre-cache the full amount of block matrix data, reduces the dependence on low-speed external storage during data transmission, ensures the efficiency of each batch of data transmission, and further cooperates with the parallel calculation advantage of the tensor processor for small matrices to realize the orderly and efficient execution of large-scale matrix multiplication operation.

[0126] As known from the foregoing, in the application scenario of transmitting a plurality of block matrix data in batches, only after the block multiplication result matrix corresponding to the previous batch of block matrix data is calculated and the previous batch of block matrix data is released from the memory device, the transmission of the next batch of block matrix data can be performed (that is, data calculation and data transmission are serial). In order to reduce the time consumption of data processing, in an embodiment of the present application, the memory device includes a first buffer and a second buffer. The loading instruction includes a first loading instruction and a second loading instruction. The first loading instruction is used to indicate that at least one block matrix data in the cache device is transmitted to the first buffer. The second loading instruction is used to indicate that at least one block matrix data in the cache device is transmitted to the second buffer.

[0127] In the present embodiment, step 400 of obtaining a loading instruction includes step 410 and step 420.

[0128] Step 410: obtaining a current first count value and a third buffer.

[0129] In the embodiment, the current first count value is updated by the historical first count value, and the initial first count value can be equal to the total number of the respective block matrix data. Specifically, each time a batch of block matrix data is successfully transmitted from the cache to the memory according to the loading instruction (i.e., the storage step in step 500 is completed), the data processing device can subtract the number of block matrix data transmitted this time from the historical first count value to obtain a new current first count value. For example, if the historical first count value is 100, 5 block matrix data are transmitted this time, and then the updated current first count value is 95, and so on. The updating logic ensures that the current first count value is always consistent with the number of remaining block matrix data to be transmitted, providing real-time basis for subsequent determination of whether the transmission task needs to continue.

[0130] In the embodiment, the third buffer is any buffer with remaining storage space greater than or equal to a first preset value in the first buffer and the second buffer. That is, the third buffer is a target buffer with block matrix data storage capability selected from the first buffer and the second buffer of the memory, and the selection criteria and logic directly determine the safety of block matrix data transmission. The specific selection criteria of the third buffer are as follows: the third buffer must be a buffer with remaining storage space greater than or equal to the first preset value (i.e., the first buffer or the second buffer). The first preset value is a threshold value preset according to the single batch transmission amount of block matrix data. Usually, it is equal to the total data amount of block matrix data in single planned transmission (for example, if the data amount of each block matrix data is 2 MB, and 2 block matrix data are planned to be transmitted at a time, then the total data amount of block matrix data in single planned transmission is 2*2 MB=4 MB). The core purpose of this criterion is to ensure that the selected buffer has enough space to accommodate the block matrix data in single transmission, thereby avoiding overflow problems caused by insufficient space when data is transmitted to the memory.

[0131] In the embodiment, the data processing device can collect information indicating the sizes of the remaining storage spaces of the first buffer and the second buffer in the memory in real time; then compare the sizes of the remaining spaces of the two buffers with the first preset value respectively, and select the buffer with remaining space greater than or equal to the first preset value as the third buffer. If the remaining spaces of both buffers (i.e., the first buffer and the second buffer) meet the condition of being greater than or equal to the first preset value, either one of them can be selected as the third buffer; if only one buffer meets the condition, then the buffer is the third buffer; if neither of the two buffers meets the condition, the third buffer is not determined temporarily, and it needs to wait until the block matrix data stored in the memory is released after being calculated by the tensor processor, and then select the buffer with remaining space meeting the condition.

[0132] In the embodiment, the first preset value can be set according to requirements, as long as the remaining space in the third buffer can accommodate the block matrix data transmitted by the cache. For example, the calculation formula of the first preset value can be as follows:

[0133]

[0134] wherein, the first preset value is represented by V1; the number of block matrix data transmitted by the cache to the memory in each batch is represented by N; the data amount of each block matrix data is represented by S. Of course, in other embodiments of the application, the first preset value can also be greater than V1 in the above formula. The setting of the third preset value is similar to the setting principle of the first preset value, and will not be described in detail hereinafter.

[0135] Step 420: If the current first count value is greater than or equal to the second preset value, the loading instruction is obtained based on the third buffer. If the third buffer is the first buffer, the loading instruction is the first loading instruction. If the third buffer is the second buffer, the loading instruction is the second loading instruction.

[0136] In the embodiment, the purpose of setting the second preset value is to provide a necessary judgment threshold for batch transmission of block matrix data. On the one hand, by comparing with the current first count value, the invalid operation of generating a loading instruction for data transmission after all block matrix data are transmitted is avoided, and resource waste is reduced. On the other hand, it can match the single storage capacity of the memory double buffer and the single calculation batch of the tensor processor, so as to ensure that the batch data transmitted each time can adapt to the buffer storage capacity and the processor calculation demand, avoid the idle of buffer space or the frequent switching of processor tasks, guarantee the continuity of data transmission and data calculation in parallel, and further reduce the overall time consumption of large-scale matrix multiplication.

[0137] In the embodiment, the setting of the second preset value depends on the numerical value of the initial first count value. For example, if the initial first count value is equal to the total number of each block matrix data, the second preset value is 0; if the initial first count value is equal to the total number of each block matrix data + 1, the second preset value is 1, and so on.

[0138] The embodiment constructs the dynamic loading instruction generation logic of the adaptive memory double buffer (i.e., the first buffer and the second buffer) through the cooperation of the step 410 and the step 420, can realize the parallel promotion of the block matrix data transmission and the data calculation, and does not need to wait for the completion of the calculation of the previous batch of data before starting the transmission of the next batch of data. For example, when the data processing apparatus screens the first buffer as the third buffer through the step 410, and the current first count value meets the transmission threshold (i.e., the block matrix data is not all transmitted), the first loading instruction can be generated through the step 420 to transmit a batch of block matrix data to the first buffer; in the process of the calculation and processing of the tensor processor based on each block matrix data in the first buffer, the data processing apparatus can execute the step 410 again, if the second buffer remaining storage space meets the standard and there is still block matrix data to be transmitted, the second loading instruction can be generated through the step 420 to transmit the next batch of block matrix data to the second buffer; after the calculation of the block matrix data in the first buffer is completed and the space is released, the tensor processor can directly start the calculation based on the second buffer data, while the data processing apparatus continues to transmit the next batch of block matrix data to the idle first buffer through the step 410 to the step 420, and so on, until all the block matrix data are transmitted to the memory by the cache, and all participate in the calculation to obtain the corresponding block multiplication result matrix. That is to say, the embodiment breaks the serial dependence relationship between the data transmission and the data calculation relying on the cooperation of the double buffer and the dynamic loading instruction (i.e., the first loading instruction and the second loading instruction), realizes the parallelization of the data transmission and the data calculation. This design guarantees the safety of the block matrix data transmission, greatly reduces the idle time of the tensor processor due to the waiting for the block matrix data transmission, further reduces the overall time consumption of the large-scale matrix multiplication operation, and fully plays the hardware cooperation advantage of the data processing apparatus.

[0139] Step 500: transmitting each block matrix data from the cache to the memory based on the loading instruction.

[0140] It needs to be clear that the transmission of data from one storage module (i.e., the cache) to another storage module (i.e., the memory) is a mature technology in the computer field, which is not described here.

[0141] It should be noted that the storage format of the blocked matrix data in the memory directly affects the continuity and computing efficiency of the subsequent data reading of the tensor processor. The memory usually stores matrix data in a fixed manner by row or by column, while the row and column element distribution of the left-hand matrix and the right-hand matrix in the blocked matrix data does not match the memory storage logic in the original format. For example, if the memory stores data by row, since the right-hand matrix needs to be read by column to participate in matrix multiplication, directly storing the right-hand matrix in the original format of the blocked matrix data will cause the right-hand matrix array element to be stored dispersedly, and the subsequent tensor processor needs to access data across multiple storage units when reading, increasing the number of cache misses and reducing the computing efficiency; if the memory stores data by column, since the left-hand matrix needs to be read by row to participate in matrix multiplication, directly storing the left-hand matrix in the original format of the blocked matrix data will cause the left-hand matrix array element to be stored dispersedly, and the subsequent tensor processor needs to access data across multiple storage units when reading, also increasing the number of cache misses and reducing the computing efficiency. In order to optimize the storage layout of the blocked matrix data in the memory, ensure that the tensor processor can read the blocked matrix data in a continuous and efficient manner, and avoid the increase in computing time caused by the mismatch of the storage format, in an embodiment of the present application, step 500, based on the loading instruction, transmitting each blocked matrix data from the cache to the memory, can include steps 510-540.

[0142] Step 510: Traverse each blocked matrix data to obtain a first blocked matrix data.

[0143] In the present embodiment, the first blocked matrix data is the blocked matrix data currently needed to be stored in the memory among the blocked matrix data. That is, in the present embodiment, the storage manner of each blocked matrix data in the memory can refer to the storage manner of the first blocked matrix data in the memory.

[0144] Step 520: Obtain the transposition information based on the first blocked matrix data.

[0145] In the present embodiment, the transposition information is a core parameter set for guiding the subsequent transposition processing of the first blocked matrix data, and its generation logic needs to be determined comprehensively in combination with the storage characteristics (for example, storage by row or by column) of the memory and the computing requirements (for example, the rule that the left-hand matrix is read by row and the right-hand matrix is read by column in matrix multiplication) of the tensor processor. That is, based on the first blocked matrix data, the transposition information is obtained, which specifically includes the following contents:

[0146] Firstly, the sub-matrix (i.e. left-hand matrix or right-hand matrix) in the first block matrix data which needs to be transposed is determined. For example, if the memory adopts the row storage mode, the left-hand matrix needs to be read continuously by row in the matrix multiplication calculation, and its original row storage format matches the reading requirement, so no transposition is needed. The right-hand matrix needs to be read by column, and the original row storage mode causes the column elements to be stored dispersedly, so the right-hand matrix is determined as the sub-matrix which needs to be transposed. If the memory adopts the column storage mode, the right-hand matrix needs to be read continuously by column in the calculation, and its original column storage format matches the reading requirement, so no transposition is needed. The left-hand matrix needs to be read by row, and the original column storage mode causes the row elements to be stored dispersedly, so the left-hand matrix is determined as the sub-matrix which needs to be transposed.

[0147] Then, binary identification bits (for example, "1" represents that transposition is needed, and "0" represents that no transposition is needed) are generated, which are used to quickly determine whether the transposition operation is performed in the subsequent steps (for example, for the above row storage scenario, the transposition identification of the right-hand matrix is "1", and that of the left-hand matrix is "0"). In this embodiment, the generated binary identification bits can be regarded as the transposition information.

[0148] Step 530: Transposition processing is performed on the first block matrix data based on the transposition information, to obtain transposed block matrix data.

[0149] It should be noted that the matrix transposition refers to the operation of interchanging the rows and columns of a matrix, and the new matrix after transposition is called the transposed matrix of the original matrix. That is, the matrix (i.e. left-hand matrix or right-hand matrix) in the first block matrix data is transposed based on the transposition information, to obtain the corresponding transposed block matrix data, which is a mature technology and will not be described herein.

[0150] Step 540: The transposed block matrix data is stored in the memory as the first block matrix data.

[0151] In an embodiment of the present application, if the memory adopts the row storage mode, the left-hand matrix is matrix G, and the right-hand matrix is matrix H. The expressions of matrix G and matrix H are as follows:

[0152]

[0153]

[0154] In this embodiment, matrix G does not need to be transposed, and the transposed matrix of matrix H (i.e. right-hand matrix) is The expression of the transposed matrix is as follows:

[0155]

[0156] In the embodiment, the matrix G and the matrix H are multiplied to obtain the following matrix:

[0157]

[0158] That is, in the embodiment, the matrix G and the transposed matrix are directly stored in the memory, and when the multiplication of the matrix G and the matrix H is performed, only the matrix G and the transposed matrix are read by row, and no transposition is required when reading, and only continuous reading is required.

[0159] The embodiment is optimized from the data storage and reading adaptability by the steps 510 to 540, and the transposed block matrix data finally stored in the memory can meet the requirement of continuous reading of the tensor processor in the matrix multiplication calculation, and is adapted to the fixed storage logic (for example, storage by row or by column) of the memory. That is, each transposed block matrix data obtained can avoid the element dispersion problem caused by the mismatch between the storage format of the original block matrix data and the storage logic of the memory, reduce the cache miss number when the tensor processor reads data, and eliminate the problems of reduced calculation efficiency and increased additional time caused by discontinuous data reading, thereby laying a data storage foundation for the subsequent efficient execution of the block matrix multiplication operation of the tensor processor.

[0160] Step 600: Based on each block matrix data in the memory, a block multiplication result matrix corresponding to each block matrix data is calculated by the tensor processor.

[0161] In the embodiment, the block multiplication result matrix is equal to the product of the left-hand matrix and the right-hand matrix in the corresponding block matrix data (that is, the matrix multiplication operation is performed on the left-hand matrix and the right-hand matrix in the corresponding block matrix data).

[0162] It should be noted that the matrix multiplication operation of the two matrices (that is, the left-hand matrix and the right-hand matrix) in the block matrix data by the processor (for example, the tensor processor) to obtain the corresponding result (that is, the block multiplication result matrix) is a mature technology, and will not be described here.

[0163] It should be noted that in the scenario of realizing parallel data transmission and data calculation based on double buffers (i.e., the first buffer and the second buffer), the core requirements of buffer data not overflowing and calculation task not idling need to be met at the same time: on the one hand, if the calculation is directly started without accurately judging the buffer storage state, the data may be overwritten or overflowed in the calculation process due to the buffer still receiving data; on the other hand, if the total amount of the to-be-calculated block matrix data is not tracked synchronously, the problem of calculation task termination in advance (i.e., missing calculation of part of the block matrix data) or continuous waiting (i.e., no block matrix data to calculate but occupying hardware resources) may occur. For example, if the first buffer only transmits part of the block matrix data, starting calculation at this time will cause the block multiplication result matrix to be incorrect due to incomplete data; if the first buffer has no data, but still attempts to read the data in the first buffer, it will cause the tensor processor to idle and waste computing power. In order to ensure that the calculation process of the tensor processor on the block matrix data is safe, efficient and without redundancy, and to avoid the problem of result error or hardware resource waste caused by improper calculation triggering time, in the embodiments of the present application, step 600, based on each block matrix data in the memory, calculating a block multiplication result matrix corresponding to each block matrix data through the tensor processor, can include step 610 and step 620.

[0164] Step 610: Obtain the current second count value and the fourth buffer.

[0165] In the embodiment, the current second count value is updated by the historical second count value, and the initial second count value can be equal to the total number of the respective piecewise matrix data. Specifically, after a batch of piecewise matrix data is calculated by the tensor processor and the corresponding piecewise multiplication result matrix is obtained (i.e., the calculation process in step 600 is completed), the data processing apparatus will subtract the number of the piecewise matrix data completed in this calculation from the historical second count value (the historical second count value refers to the second count value corresponding to the buffer before the batch of piecewise matrix data in the buffer is not involved in the calculation to obtain the corresponding piecewise multiplication result matrix) to obtain the current second count value (the current second count value refers to the second count value corresponding to the buffer after the batch of piecewise matrix data in the buffer is involved in the calculation to obtain the corresponding piecewise multiplication result matrix). For example, if the historical second count value is 80, 6 piecewise matrix data are successfully calculated by the tensor processor in this calculation and the corresponding piecewise multiplication result matrix is generated, and then the updated current second count value is 74, and so on. The updating logic ensures that the current second count value is always consistent with the number of the remaining piecewise matrix data to be calculated, provides real-time and accurate quantity basis for subsequent determination of whether the calculation task needs to continue or switch to other buffers (e.g., the first buffer or the second buffer) to read the piecewise matrix data, avoids the problems of missing piecewise matrix data or repeated calculation, and also provides data quantity level synchronization support for the data calculation and data transmission in the double buffer parallel processing mode.

[0166] In the embodiment, the fourth buffer is any buffer with remaining storage space less than or equal to the third preset value among the first buffer and the second buffer. That is, the fourth buffer is the target buffer with sufficient data storage and calculation conditions selected from the first buffer and the second buffer of the memory, and the selection criteria and logic directly determine the accuracy and safety of the tensor processor starting the calculation task. The specific selection criteria of the fourth buffer are as follows: the fourth buffer must be a buffer with remaining storage space less than or equal to the third preset value (i.e., the first buffer or the second buffer). The third preset value is a threshold value preset according to the total storage requirement of a single batch of piecewise matrix data to be calculated, and is usually equal to the difference between the total capacity of a single buffer of the memory and the total data amount of the piecewise matrix data calculated at a time (for example, if the total capacity of a single buffer is 10 MB and the total data amount of the piecewise matrix data calculated at a time is 8 MB, then the third preset value = 10 MB-8 MB = 2 MB). The core purpose of the criteria is to ensure that the selected buffer has sufficient piecewise matrix data (insufficient remaining space means that the used space meets the standard) to meet the data amount requirement of the tensor processor for single calculation, and to avoid the problems of calculation interruption or result loss caused by insufficient data storage and data amount in the buffer from the source.

[0167] In the embodiment, the data processing apparatus can collect the remaining storage space information of the first buffer and the second buffer in the memory in real time; then compare the remaining space sizes of the two buffers with a third preset value respectively, and screen out the buffer with a remaining space ≤ the third preset value as the fourth buffer. If the remaining spaces of the two buffers (i.e. the first buffer and the second buffer) both satisfy ≤ the third preset value, the buffer with a longer data storage time can be selected as the fourth buffer (to avoid that the data stays in the buffer for too long, affecting the overall processing efficiency); if only one buffer satisfies the condition, the buffer is the fourth buffer; if neither of the two buffers satisfies the condition, the fourth buffer is not determined temporarily, and the cache needs to continue to transmit the block matrix data to the buffer until a certain buffer meets the standard, and then the screening is performed again.

[0168] In the embodiment, the third preset value can be set according to the demand, as long as the comparison result of the remaining storage space of the buffer and the third preset value can accurately judge whether the buffer has stored enough block matrix data. For example, if the total data amount of the block matrix data calculated by a single batch plan is 5 MB, and the total capacity of a single buffer is 12 MB, the third preset value can be set to 7 MB (i.e. equal to 12 MB-5 MB), at this time, the remaining storage space ≤ 7 MB means that the used space of the buffer ≥ 5 MB, which meets the data amount requirement. If part of the space needs to be reserved to adjust the data format temporarily, the third preset value can also be set to 6 MB to ensure that the buffer still has 1 MB of redundant space after storing the target block data. This setting logic can not only flexibly adapt to the data amount requirements of different batches of calculations, but also can avoid the problems of starting calculation before the data is full or wasting space redundancy due to unreasonable preset value, and can ensure the accuracy of data preparation before calculation and efficient use of hardware resources. In the preferred embodiment of the present application, the remaining storage space of the buffer after storing enough block matrix data can be 0 MB, i.e. the third preset value is 0 MB in the embodiment.

[0169] Step 620: If the current second count value is greater than or equal to the fourth preset value, the tensor processor is used to calculate a block multiplication result matrix corresponding to each block matrix data in the fourth buffer.

[0170] In this embodiment, the purpose of setting the fourth preset value is to provide a necessary threshold for batch computation of block matrix data. On the one hand, by comparing it with the current second count value, it avoids the invalid operation of attempting to start tensor processor computation after all block matrix data has been computed, reducing the waste of hardware resources. On the other hand, it can match the single-time data storage volume of the memory double buffer and the single-time parallel computation capability of the tensor processor, ensuring that the batch data of each computation start is exactly matched with the computation throughput of the tensor processor and the amount of data already stored in the buffer. This avoids frequent processor start-stop and task switching due to too small a computation batch, or computation interruption caused by too large a computation batch exceeding the data storage capacity of the buffer, ensuring the stability of parallel data transmission and computation, and further shortening the overall computation time of large-scale matrix multiplication.

[0171] In this embodiment, the setting of the fourth preset value depends on the magnitude of the initial second count value. For example, if the initial second count value is equal to the total number of data in each block matrix, then the fourth preset value is 0; if the initial second count value is equal to the total number of data in each block matrix plus 2, then the fourth preset value is 2, and so on.

[0172] It is important to understand that after calculating the block multiplication result matrix corresponding to each block matrix data in the fourth buffer, the block matrix data in the fourth buffer needs to be released. In this embodiment, there are no restrictions on the method of releasing the block matrix data in the fourth buffer. For example, after calculating the block multiplication result matrix corresponding to each block matrix data, that block matrix data can be released from the fourth buffer; or, all the block matrix data in the fourth buffer can be released together after all the block multiplication result matrices corresponding to all the block matrix data in the fourth buffer have been calculated.

[0173] The embodiment provides safe and efficient trigger logic for the tensor processor to calculate the block multiplication result matrix by dynamically tracking the current second count value and the fourth buffer of accurate screening through steps 610 and 620, and has significant technical advantages. The current second count value reflects the number of remaining block matrix data to be calculated in real time, and in combination with the fourth preset value, invalid calculation can be avoided after all block matrix data is calculated, and hardware resource idling is reduced. Meanwhile, the fourth buffer is screened out according to the condition that the remaining storage space is less than or equal to the third preset value, so that sufficient block matrix data is stored in the fourth buffer, and calculation interruption or result error caused by insufficient data is avoided. In the application scenario in which the memory device contains a double buffer, the embodiment can be coordinated with the data transmission process to realize parallel processing of “one buffer participating in calculation and the other buffer participating in transmission”, greatly reducing the idle time of the tensor processor waiting for data, ensuring the safety and accuracy of the calculation process, further compressing the overall operation time of large-scale matrix multiplication, and fully exerting the parallel computing advantage of the tensor processor and the hardware potential of the double buffer.

[0174] It should be noted that although the tensor processor has parallel computing advantages, when processing a single block matrix data, if the whole block matrix data is directly calculated, it is easy to be affected by the size of the block matrix data. For example, if the data amount of the block matrix data is too large, a single data processing core needs to bear a large amount of operation tasks, which may cause an increase in calculation time and make it difficult to fully exert the hardware potential of the multiple cores of the tensor processor. If the block matrix data is not evenly distributed, resource waste may occur due to overload of some cores and idling of some cores. In order to ensure that the multiple core capability of the tensor processor can be efficiently utilized and avoid the decrease in calculation efficiency caused by excessive operation pressure of a single core or uneven core load, in an embodiment of the present application, the block multiplication result matrix corresponding to each block matrix data calculated by the tensor processor can include steps 630 to 670.

[0175] Step 630: Each block matrix data is traversed to obtain a second block matrix data.

[0176] In the embodiment, the second block matrix data is a block matrix data in each block matrix data that needs to calculate a corresponding block multiplication result matrix. That is, in the embodiment, the acquisition method of the block multiplication result matrix corresponding to any block matrix data in each block matrix data can refer to the second block matrix data.

[0177] Step 640: Based on the second block matrix data, a third block matrix data and a fourth block matrix data are obtained.

[0178] In this embodiment, the third block matrix data is a part of the second block matrix data, and the fourth block matrix data is another part of the second block matrix data.

[0179] In this embodiment, the second block matrix data is divided into the third block matrix data and the fourth block matrix data, similar to the division of the matrix multiplication input data into multiple block matrix data in the above, as long as the third block matrix data and the fourth block matrix data meet the "dimension matching" condition of matrix multiplication.

[0180] In order to make the calculation amount of the two data processing cores of the tensor processor approximately equal, in this embodiment, the data amount of the third block matrix data and the fourth block matrix data can be equal.

[0181] Step 650: based on one data processing core in the tensor processor, processing the third block matrix data to obtain a first result matrix.

[0182] It should be clear that the data processing core of the tensor processor is a hardware unit in the tensor processor specially used for performing core data processing tasks such as matrix operation and tensor calculation, and is a core component of the tensor processor to realize efficient parallel calculation and play its hardware advantage for machine learning / large-scale data processing, which is mature prior art. It is mature technology to perform matrix multiplication operation on matrix data (i.e. third block matrix data, which includes left-hand matrix and right-hand matrix) by a certain data processing core of the tensor processor, and the corresponding calculation result (i.e. first result matrix) is obtained, which is not described here.

[0183] Step 660: based on another data processing core in the tensor processor, processing the fourth block matrix data to obtain a second result matrix.

[0184] It should be clear that it is mature technology to perform matrix multiplication operation on matrix data (i.e. fourth block matrix data) by a certain data processing core of the tensor processor, and the corresponding calculation result (i.e. second result matrix) is obtained, which is not described here.

[0185] Step 670: based on the first result matrix and the second result matrix, obtaining a block multiplication result matrix corresponding to the second block matrix data.

[0186] It needs to be clear that since the third block matrix data and the fourth block matrix data are two parts obtained by splitting the second block matrix data according to the matrix multiplication dimension matching principle, the first result matrix and the second result matrix corresponding to the two are completely matched with the block integration requirement in the row and column dimensions, that is, the number of rows and columns of the first result matrix and the number of rows and columns of the second result matrix correspond to each other and are equal, and the sub-matrix dimensions at the corresponding positions are completely matched. Therefore, by performing a matrix addition operation of adding corresponding position elements of the first result matrix and the second result matrix, a complete block multiplication result matrix corresponding to the second block matrix data can be obtained, which is completely consistent with the matrix multiplication result obtained by directly calculating the second block matrix data as a whole.

[0187] In this embodiment, the second block matrix data to be calculated is first determined, and then it is split into third block matrix data and fourth block matrix data with approximately equal data amounts, which are allocated to two data processing cores of the tensor processor for synchronous operation, and finally the block multiplication result matrix is obtained by integrating the results. That is, the present application can avoid the overload problem of the tensor processor when processing large-size block matrix data with a single core, fully utilize the hardware potential of the tensor processor multi-core, and reduce the waste of idle cores. At the same time, the balanced split of data amount can make the operation time of the two data processing cores tend to be consistent, avoid the prolongation of the operation time of single block data due to uneven load, further compress the overall time consumption of large-scale matrix multiplication, and ensure the operation efficiency. Through the split of block matrix data and the multi-core parallel calculation of the tensor processor, the operation efficiency of single block matrix data is significantly improved.

[0188] Step 700: If all the block matrix data in the buffer are transmitted to the memory, and each block matrix data obtains a corresponding block multiplication result matrix, then based on each block multiplication result matrix, the matrix multiplication output data is obtained.

[0189] The embodiment of the data processing method proposed in the present application adapts the tensor processor hardware processing characteristics by splitting the block matrix data obtained from the matrix multiplication input data, avoiding the problem of increased operation time caused by the mismatch between the data dimensions and the tensor processor characteristics due to the too large matrix multiplication input data. And using the buffer to store the block matrix data in advance, the dependence on low-speed external storage is reduced, the data access efficiency is improved, relying on the parallel calculation advantage of the tensor processor for small matrices, the operation efficiency of large-scale matrix multiplication is greatly improved, and the core problem that the existing technology cannot fully utilize the advantages of the tensor processor is solved.

[0190] After introducing the data processing method proposed in the embodiment of the present application, the embodiment of a data processing device proposed in the present application will be introduced below. As shown in Figure 2 The data processing device includes a tensor processor 11, a memory 12, a buffer 13, and a reader 14. Among them,

[0191] a reader 14, configured to acquire matrix multiplication input data;

[0192] a tensor processor 11, configured to acquire a plurality of block matrix data based on the matrix multiplication input data; each block matrix data comprises a left-hand matrix and a right-hand matrix, and a size of each block matrix data is a positive integer multiple of a systolic array size of the tensor processor 11;

[0193] and store each block matrix data in the cache 13;

[0194] and acquire a load instruction; the load instruction is used to indicate that at least one block matrix data in the cache 13 is transmitted to the memory 12;

[0195] and transmit each block matrix data from the cache 13 to the memory 12 based on the load instruction;

[0196] and calculate a block multiplication result matrix corresponding to each block matrix data by the tensor processor 11 based on each block matrix data in the memory 12; the block multiplication result matrix is equal to a product of the left-hand matrix and the right-hand matrix in the corresponding block matrix data;

[0197] and if all block matrix data in the cache 13 are transmitted to the memory 12, and each block matrix data obtains the corresponding block multiplication result matrix, acquire matrix multiplication output data based on each block multiplication result matrix.

[0198] As a specific embodiment in the present application, the memory 12 comprises a first buffer and a second buffer; the load instruction comprises a first load instruction and a second load instruction; the first load instruction is used to indicate that at least one block matrix data in the cache 13 is transmitted to the first buffer; the second load instruction is used to indicate that at least one block matrix data in the cache 13 is transmitted to the second buffer;

[0199] The tensor processor 11 is further configured to acquire a current first count value and a third buffer; the current first count value is updated from a historical first count value; an initial first count value is equal to a total number of each block matrix data; and the third buffer is any buffer with a remaining storage space greater than or equal to a first preset value in the first buffer and the second buffer;

[0200] And, if the current first count value is greater than or equal to a second preset value, the loading instruction is obtained based on the third buffer; if the third buffer is the first buffer, the loading instruction is a first loading instruction; and if the third buffer is the second buffer, the loading instruction is a second loading instruction.

[0201] As a specific embodiment in the present application, the tensor processor 11 is further configured to obtain a current second count value and a fourth buffer; the current second count value is updated from a historical second count value; an initial second count value is equal to a total number of the plurality of block matrix data; and the fourth buffer is any buffer with a remaining storage space less than or equal to a third preset value in the first buffer and the second buffer.

[0202] And, if the current second count value is greater than or equal to a fourth preset value, a block multiplication result matrix corresponding to each block matrix data in the fourth buffer is calculated by the tensor processor 11.

[0203] As a specific embodiment in the present application, the first buffer and the second buffer have equal capacities; the tensor processor 11 is further configured to obtain a first capacity, a second capacity and a first data amount; the first capacity is a capacity of the first buffer or the second buffer; the second capacity is a minimum storage capacity of the memory 12; and the first data amount is a storage space size required to store the matrix multiplication input data.

[0204] Based on the first capacity and the second capacity, a first number is obtained; the first number is equal to a floor value of a ratio of the first capacity to the second capacity.

[0205] And, based on the first data amount and the second capacity, a second number is obtained; the second number is equal to a ceiling value of a ratio of the first data amount to the second capacity.

[0206] And, based on the first number and the second number, a target number is obtained.

[0207] And, based on the target number, the matrix multiplication input data is divided into a plurality of block matrix data.

[0208] As a specific embodiment in the present application, the tensor processor 11 is further configured to obtain a second data amount; the second data amount is a storage space required to store the matrix multiplication output data.

[0209] Based on the second data amount and the second capacity, a third number is obtained; the third number is equal to a ceiling value of a ratio of the second data amount to the second capacity.

[0210] and, based on the first number, the second number and the third number, obtaining a fourth number and a fifth number; the fourth number is equal to an upward rounding of a ratio of the second number and the first number; and the fifth number is equal to an upward rounding of a ratio of the third number and the first number;

[0211] and, based on the fourth number and the fifth number, obtaining a sixth number; the sixth number is equal to a maximum value of the fourth number and the fifth number;

[0212] and, based on the sixth number, obtaining the target number.

[0213] As one specific embodiment in the present application, the tensor processor 11 is further configured to, based on the sixth number, obtain a first value and a second value; the first value is a matrix row value corresponding to a left-hand matrix obtained after taking the sixth number as a candidate target number; and the second value is a matrix column value corresponding to a right-hand matrix obtained after taking the sixth number as a candidate target number;

[0214] and, if the first value and the second value are both less than or equal to a fifth preset value, taking the sixth number as the target number; otherwise, updating the sixth number, the first value and the second value; until the first value and the second value obtained after the updating are both less than or equal to the fifth preset value, taking the sixth number obtained after the last updating as the target number.

[0215] As one specific embodiment in the present application, the tensor processor 11 is further configured to, traverse each piece of block matrix data to obtain first block matrix data; the first block matrix data is a piece of block matrix data in each piece of block matrix data which currently needs to be stored into the memory 12;

[0216] and, based on the first block matrix data, obtain transposition information;

[0217] and, based on the transposition information, perform transposition processing on the first block matrix data to obtain transposed block matrix data;

[0218] and, store the transposed block matrix data into the memory 12 as the first block matrix data.

[0219] As one specific embodiment in the present application, the tensor processor 11 is further configured to, traverse each piece of block matrix data to obtain second block matrix data; the second block matrix data is a piece of block matrix data in each piece of block matrix data which currently needs to be calculated to obtain a corresponding block multiplication result matrix;

[0220] and, based on the second block matrix data, obtain third block matrix data and fourth block matrix data; the third block matrix data is a part of the second block matrix data; and the fourth block matrix data is another part of the second block matrix data;

[0221] and, based on processing the third block matrix data by one data processing core in the tensor processor 11, obtain a first result matrix;

[0222] and, based on processing the fourth block matrix data by another data processing core in the tensor processor 11, obtain a second result matrix;

[0223] and, based on the first result matrix and the second result matrix, obtain a block multiplication result matrix corresponding to the second block matrix data.

[0224] The embodiment of the data processing apparatus proposed in the present application avoids the problem of increasing operation time caused by mismatching between data dimensions and tensor processor characteristics due to too large matrix multiplication input data by splitting the matrix multiplication input data to obtain block matrix data that adapts to the processing characteristics of the tensor processor. Moreover, the use of the buffer to store the block matrix data in advance reduces the dependence on low-speed external storage, improves data access efficiency, relies on the parallel computing advantage of the tensor processor for small matrices, greatly improves the efficiency of large-scale matrix multiplication operation, and solves the core problem that the existing technology cannot fully utilize the advantages of the tensor processor.

[0225] After introducing the data processing apparatus proposed in the embodiments of the present application, the following introduces an embodiment of a computer readable storage medium proposed in the present application, which stores a computer program. When the computer program is executed by a processor, it realizes the data processing method as described in any one of the above embodiments.

[0226] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0227] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, apparatus and module described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0228] In several embodiments provided in the embodiments of the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the embodiments of the device described above are merely schematic; for example, the division of the modules is merely a logical function division; an actual implementation can be another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different modules can be indirect couplings or communication connections through some interfaces, devices or modules, and can be electrical, mechanical or in other forms.

[0229] The modules described as separated components can or can not be physically separated, and the components displayed as modules can or can not be physical modules, i.e., can be located in one place, or can be distributed on a plurality of network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0230] In addition, each functional module in each of the embodiments of the present application can be integrated in one processing module, or each module can be physically present alone, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can be stored in a computer readable storage medium.

[0231] In the above embodiments, all or part of the embodiments can be realized by software, hardware, firmware or any combination thereof. When realized by software, all or part of the embodiments can be realized in the form of a computer program product.

[0232] The computer program product includes one or more computer instructions. When the computer program is loaded and executed on the computer, the flow or function described in the embodiments of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that the computer can store or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0233] The above describes the technical solutions provided by the embodiments of the present application in detail. The principles and implementation manners of the embodiments of the present application are described by applying specific examples. The above examples are only used to help understand the method and core idea of the embodiments of the present application; at the same time, for those skilled in the art, according to the idea of the embodiments of the present application, the specific implementation manner and application range will be changed, and the above description should not be understood as a limitation of the embodiments of the present application.

Claims

1. A data processing method applied to a data processing apparatus, the data processing apparatus comprising a tensor processor, a memory, and a cache, characterized in that, The method includes: Obtain the input data for matrix multiplication; Based on the matrix multiplication input data, multiple block matrix data are obtained; each block matrix data includes a left-hand matrix and a right-hand matrix, and the size of each block matrix data is a positive integer multiple of the systolic array size of the tensor processor; The data of each block matrix is ​​stored in the cache. Obtain a load instruction; the load instruction is used to instruct at least one block matrix data in the cache to be transferred to the memory. Based on the loading instruction, the data of each block matrix is ​​transferred from the cache to the memory. Based on the block matrix data in the memory, the tensor processor calculates a block multiplication result matrix that corresponds one-to-one with each block matrix data; the block multiplication result matrix is ​​equal to the product of the left-hand matrix and the right-hand matrix in the corresponding block matrix data; If all the block matrix data in the buffer is transferred to the memory, and each block matrix data obtains a corresponding block multiplication result matrix, then the matrix multiplication output data is obtained based on each block multiplication result matrix. The memory includes a first buffer and a second buffer; The first buffer and the second buffer have equal capacities; the step of obtaining multiple block matrix data based on the matrix multiplication input data includes: Obtain a first capacity, a second capacity, and a first data volume; the first capacity is the capacity of the first buffer or the second buffer; the second capacity is the minimum storage capacity of the memory; the first data volume is the storage space required when storing the matrix multiplication input data; Based on the first capacity and the second capacity, a first quantity is obtained; the first quantity is equal to the ratio of the first capacity and the second capacity rounded down; Based on the first data volume and the second capacity, a second quantity is obtained; the second quantity is equal to the ratio of the first data volume to the second capacity rounded up. Based on the first quantity and the second quantity, obtain the target quantity; The matrix multiplication input data is divided into blocks based on the target quantity to obtain multiple block matrix data.

2. The data processing method according to claim 1, characterized in that, The loading instruction includes a first loading instruction and a second loading instruction; the first loading instruction is used to instruct at least one block matrix data in the buffer to be transferred to the first buffer. The second load instruction is used to instruct at least one block matrix data in the buffer to be transferred to the second buffer; The process of obtaining the loading instruction includes: Obtain the current first count value and the third buffer; the current first count value is updated from the historical first count value; the initial first count value is equal to the total number of data in each block matrix; the third buffer is any buffer in the first buffer and the second buffer whose remaining storage space is greater than or equal to a first preset value. If the current first count value is greater than or equal to the second preset value, then the loading instruction is obtained based on the third buffer; if the third buffer is the first buffer, then the loading instruction is the first loading instruction; if the third buffer is the second buffer, then the loading instruction is the second loading instruction.

3. The data processing method according to claim 2, characterized in that, The step of calculating a block multiplication result matrix corresponding one-to-one with each block matrix data based on the block matrix data in the memory using the tensor processor includes: Obtain the current second count value and the fourth buffer; the current second count value is updated from the historical second count value; the initial second count value is equal to the total number of data in each block matrix; the fourth buffer is any buffer in the first buffer and the second buffer whose remaining storage space is less than or equal to a third preset value. If the current second count value is greater than or equal to the fourth preset value, then the tensor processor calculates the block multiplication result matrix that corresponds one-to-one with the block matrix data in the fourth buffer.

4. The data processing method according to claim 1, characterized in that, Obtaining the target quantity based on the first quantity and the second quantity includes: Obtain the second data volume; the second data volume is the storage space required when storing the matrix multiplication output data; Based on the second data volume and the second capacity, a third quantity is obtained; the third quantity is equal to the ratio of the second data volume to the second capacity rounded up. Based on the first quantity, the second quantity, and the third quantity, a fourth quantity and a fifth quantity are obtained; the fourth quantity is equal to the ratio of the second quantity to the first quantity rounded up; the fifth quantity is equal to the ratio of the third quantity to the first quantity rounded up. Based on the fourth quantity and the fifth quantity, a sixth quantity is obtained; the sixth quantity is equal to the maximum value among the fourth quantity and the fifth quantity. Based on the sixth quantity, the target quantity is obtained.

5. The data processing method according to claim 4, characterized in that, The step of obtaining the target quantity based on the sixth quantity includes: Based on the sixth quantity, a first value and a second value are obtained; the first value is the matrix row value corresponding to the left-hand matrix obtained after taking the sixth quantity as the candidate target quantity; the second value is the matrix column value corresponding to the right-hand matrix obtained after taking the sixth quantity as the candidate target quantity. If both the first value and the second value are less than or equal to the fifth preset value, then the sixth quantity is taken as the target quantity; otherwise, the sixth quantity, the first value, and the second value are updated; until both the updated first value and the second value are less than or equal to the fifth preset value, the sixth quantity obtained from the last update is taken as the target quantity.

6. The data processing method according to any one of claims 1 to 5, characterized in that, The step of transferring each block matrix data from the cache to the memory based on the loading instruction includes: Traverse each block matrix data to obtain the first block matrix data; the first block matrix data is the block matrix data that needs to be stored in the memory from each block matrix data; Based on the first block matrix data, obtain the transpose information; Based on the transpose information, the first block matrix data is transposed to obtain transposed block matrix data. The transposed block matrix data is stored in the memory as the first block matrix data.

7. The data processing method according to any one of claims 1 to 5, characterized in that, The step of calculating the block multiplication result matrix corresponding one-to-one with each block matrix data through the tensor processor includes: Traverse each block matrix data to obtain the second block matrix data; the second block matrix data is the block matrix data in each block matrix data that needs to be calculated to obtain the corresponding block multiplication result matrix; Based on the second block matrix data, third block matrix data and fourth block matrix data are obtained; the third block matrix data is a part of the second block matrix data; the fourth block matrix data is another part of the second block matrix data. The third block matrix data is processed using a data processing core in the tensor processor to obtain a first result matrix; The fourth block matrix data is processed based on another data processing core in the tensor processor to obtain the second result matrix; Based on the first result matrix and the second result matrix, obtain the block multiplication result matrix corresponding to the second block matrix data.

8. A data processing apparatus, the data processing apparatus comprising a tensor processor, a memory, and a cache, characterized in that, The data processing device further includes: A reader is used to obtain input data for matrix multiplication. A tensor processor is used to obtain multiple block matrix data based on the matrix multiplication input data; each block matrix data includes a left-hand matrix and a right-hand matrix, and the size of each block matrix data is a positive integer multiple of the systolic array size of the tensor processor; And, store the data of each block matrix into the cache; And, obtain a load instruction; the load instruction is used to instruct at least one block matrix data in the cache to be transferred to the memory. And, based on the loading instruction, the data of each block matrix is ​​transferred from the cache to the memory; Furthermore, based on the block matrix data in the memory, the tensor processor calculates a block multiplication result matrix that corresponds one-to-one with each block matrix data; the block multiplication result matrix is ​​equal to the product of the left-hand matrix and the right-hand matrix in the corresponding block matrix data; Furthermore, if all the block matrix data in the buffer is transferred to the memory, and each block matrix data obtains a corresponding block multiplication result matrix, then the matrix multiplication output data is obtained based on each block multiplication result matrix. The memory includes a first buffer and a second buffer; The first buffer and the second buffer have equal capacities; the tensor processor is further configured to obtain a first capacity, a second capacity, and a first data volume; the first capacity is the capacity of the first buffer or the second buffer; the second capacity is the minimum storage capacity of the memory; the first data volume is the storage space required when storing the matrix multiplication input data; Based on the first capacity and the second capacity, a first quantity is obtained; the first quantity is equal to the ratio of the first capacity and the second capacity rounded down; Furthermore, based on the first data volume and the second capacity, a second quantity is obtained; the second quantity is equal to the ratio of the first data volume to the second capacity rounded up. And, based on the first quantity and the second quantity, obtain the target quantity; Furthermore, the matrix multiplication input data is divided into blocks based on the target quantity to obtain multiple block matrix data.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the data processing method as described in any one of claims 1 to 7.

Images