Tensor calculation coprocessor

Abstract

The invention provides a tensor calculation coprocessor which comprises a preprocessing remapping module, a scheduling calculation module and an output mapping module. The preprocessing remapping module is used for receiving task parameters of a tensor calculation task transmitted based on a preset bus and matrix data of each to-be-processed matrix and mapping the task parameters and the matrix data to the input cache unit; the scheduling calculation module is used for scheduling the task parameters from the input cache unit to the target calculation array and scheduling the matrix data to the target calculation array in a blocking manner for tensor calculation to obtain a calculation result and a block index; and the output mapping module is used for mapping the calculation result to the output cache unit according to the block index. The coprocessor supports hardware data preprocessing remapping, an AXI universal interface and flexible parameter configuration of the coprocessor and supports various numerical value precisions and matrix calculation sizes, the overall calculation efficiency of the tensor calculation coprocessor is improved, and the comprehensive advantages of high flexibility and high adaptability are achieved.

Description

Technical Field

[0001] This application relates to the field of digital integrated circuit design, and more specifically, to a tensor computing coprocessor. Background Technology

[0002] Tensor computing coprocessors, primarily used to accelerate complex tensor calculations, play a crucial role in artificial intelligence, machine learning, and deep learning. Especially when dealing with complex matrix operations, they can improve computing performance by designing dedicated hardware and employing techniques such as parallel computing and pipelined processing. This reduces the computational burden on central processing units (CPUs), microcontroller units (MCUs), and field-programmable gate arrays (FPGAs), while simultaneously accelerating matrix operations.

[0003] For matrix operations handled by tensor computation coprocessors, this typically refers to tensor operations performed through matrix multiplication, the most typical form of which is matrix multiplication-addition (A*B+C). In this operation, matrices A and B are multiplied to obtain a result matrix, which is then added to another matrix C to obtain the final result.

[0004] However, existing tensor computation coprocessors typically suffer from insufficient versatility and adaptability, limited support for matrix sizes, and a limited range of supported data types, resulting in significant drawbacks. Summary of the Invention

[0005] In view of this, this application provides a tensor computing coprocessor designed to improve versatility and adaptability, and to support a variety of matrix sizes and data types.

[0006] Specifically, this application is implemented through the following technical solution:

[0007] In a first aspect, embodiments of this disclosure provide a tensor computation coprocessor.

[0008] The tensor computation coprocessor includes a preprocessing remapping module, a scheduling computation module, and an output mapping module; the scheduling computation module is connected to the preprocessing remapping module and the output mapping module, respectively; wherein:

[0009] The preprocessing remapping module is used to receive the task parameters of the tensor calculation task transmitted based on a preset bus and the matrix data of each matrix to be processed corresponding to the tensor calculation task, and to map the task parameters and matrix data to different input buffer units.

[0010] The scheduling and calculation module is used to schedule the task parameters from different input cache units to the target computing array that matches the task parameters for the same tensor computing task, and to schedule the matrix data to the target computing array in blocks for tensor computing, so as to obtain the calculation results and block indexes corresponding to each block scheduling.

[0011] The output mapping module is used to map the calculation results corresponding to each block scheduling to the output cache unit according to the block index; the number of input cache units, the size of the computing array and the number of output cache units are defined according to preset hardware parameters.

[0012] The tensor computation coprocessor provided in this disclosure supports receiving matrix data via a preset bus protocol, thus adapting to various computing platforms that support such protocols, significantly improving its versatility and adaptability. Since the number of input buffer units, the size of the computation array, and the number of output buffer units can all be flexibly defined using preset hardware parameters, flexible parameter configuration based on the platform resources of the computing platform to which the coprocessor is deployed can be achieved, effectively improving the coprocessor's flexibility. Task parameters can be used to indicate various task information for tensor computation tasks, such as matrix size and matrix data type. Therefore, by transmitting task parameters that indicate various matrix sizes and data types, and by mapping, scheduling, and computing the matrices to be processed based on these parameters, tensor computation tasks with multiple matrix sizes and data types can be processed, increasing the diversity of supported matrix sizes and data types.

[0013] Furthermore, the tensor computing coprocessor provided in this embodiment can also utilize the matrix data mapping module inside the tensor computing coprocessor to perform zero-padding, mapping, and recombination of matrix data in matrices of various data types and matrix sizes. This enables data processing to be completed within the hardware. Compared to the prior art method of using software to reconstruct data before inputting the matrix data of the matrix to be processed into the coprocessor, this method can improve the efficiency of data recombination, thereby improving the efficiency of matrix operations.

[0014] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the architecture of a coprocessor shown in an exemplary embodiment of this application;

[0016] Figure 2This is a schematic diagram illustrating an exemplary embodiment of the present application of configuring a task parameter register based on the AXI-Lite bus;

[0017] Figure 3a This is a schematic diagram illustrating the data mapping of a 16×16 matrix A under a certain data type, as shown in an exemplary embodiment of this application;

[0018] Figure 3b This is a schematic diagram illustrating the data mapping of a 16×16 matrix A under another data type, as shown in an exemplary embodiment of this application;

[0019] Figure 3c This is a schematic diagram illustrating the data mapping of a 16×16 matrix A under another data type, as shown in an exemplary embodiment of this application;

[0020] Figure 3d This is a schematic diagram illustrating the data mapping of a 16×16 matrix A under another data type, as shown in an exemplary embodiment of this application;

[0021] Figure 4a This is a schematic diagram illustrating the data mapping of a 32×32 matrix A under a certain data type, as shown in an exemplary embodiment of this application;

[0022] Figure 4b This is a schematic diagram illustrating the data mapping of a matrix A with a size of 32×64 under a certain data type, as shown in an exemplary embodiment of this application;

[0023] Figure 5a This is a schematic diagram illustrating the data mapping of a B matrix with a size of 16×16 under a certain data type, as shown in an exemplary embodiment of this application;

[0024] Figure 5b This is a schematic diagram illustrating the data mapping of a B matrix with a size of 16×16 under another data type, as shown in an exemplary embodiment of this application;

[0025] Figure 5c This is a schematic diagram illustrating the data mapping of a B matrix with a size of 16×16 under another data type, as shown in an exemplary embodiment of this application;

[0026] Figure 5d This is a schematic diagram illustrating the data mapping of a B matrix with a size of 16×16 under another data type, as shown in an exemplary embodiment of this application;

[0027] Figure 6 This is a schematic diagram illustrating the data mapping of a C matrix under different data types with varying matrix sizes, as shown in an exemplary embodiment of this application.

[0028] Figure 7 This is a schematic diagram illustrating the data mapping of a C matrix with a size of 16×16 under a 32-bit data width, as shown in an exemplary embodiment of this application.

[0029] Figure 8 This is a schematic diagram of the architecture of a preprocessing remapping module shown in an exemplary embodiment of this application;

[0030] Figure 9 This is a schematic diagram illustrating the caching of matrix data of an A / B matrix in a temporary storage module, as shown in an exemplary embodiment of this application.

[0031] Figure 10 This is an exemplary embodiment of the present application illustrating a method for caching matrix data of a C matrix in a temporary storage module;

[0032] Figure 11 This is a schematic diagram illustrating a scheduling process according to an exemplary embodiment of this application;

[0033] Figure 12 This is a schematic diagram of a scheduling process under a matrix shape of M16N16K16, FP32 / TF32 / BF16, as shown in an exemplary embodiment of this application;

[0034] Figure 13 This is a schematic diagram of a scheduling process with matrix shapes of M16N16K16, FP16 / INT8 / FP8 / FP6 / INT4 / FP4, as shown in an exemplary embodiment of this application.

[0035] Figure 14 This is a schematic diagram of a scheduling process under a matrix shape of M32N8K16, FP32 / TF32 / BF16, as shown in an exemplary embodiment of this application;

[0036] Figure 15 This is a schematic diagram of a scheduling process under a matrix shape of M32N8K16, FP16 / INT8 / FP8 / FP6 / INT4 / FP4, as shown in an exemplary embodiment of this application.

[0037] Figure 16 This is a schematic diagram of a scheduling process under a matrix shape of M8N32K16, FP32 / TF32 / BF16, as shown in an exemplary embodiment of this application;

[0038] Figure 17 This is a schematic diagram of a scheduling process under a matrix shape of M8N32K16, FP16 / INT8 / FP8 / FP6 / INT4 / FP4, as shown in an exemplary embodiment of this application.

[0039] Figure 18This is a schematic diagram illustrating an output mapping process of a calculation result according to an exemplary embodiment of this application;

[0040] Figure 19 This is a schematic diagram illustrating another calculation result output mapping process according to an exemplary embodiment of this application;

[0041] Figure 20 This is a schematic diagram of the structure of a D data storage module shown in an exemplary embodiment of this application;

[0042] Figure 21 This is a schematic diagram of the overall architecture of a tensor computing coprocessor shown in an exemplary embodiment of this application. Detailed Implementation

[0043] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0044] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0045] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0046] Research has shown that tensor computation coprocessors typically use the three parameters MNK to represent the matrix sizes of the three matrices (matrices A, B, and C) involved in the tensor computation when performing matrix multiplication and addition. Matrix A has a size of M×K, matrix B has a size of K×N, and matrix C has a size of M×N. The matrix multiplication and addition result in a single matrix (matrice D). For example, matrix D can be obtained using the following formula:

[0047]

[0048] Where D[i][j] represents the data in the i-th row and j-th column of matrix D, A[i][k] represents the k data in the i-th row of matrix A, B[k][j] represents the k data in the j-th column of matrix B, and C[i][j] represents the data in the i-th row and j-th column of matrix C.

[0049] The elements of this matrix multiplication-addition (A*B+C) form the basis for a large number of neural network computations in deep learning, especially convolutional computations in convolutional neural networks and matrix multiplications in fully connected layers, which can be accelerated using this form. Tensor computing coprocessors can accelerate matrix operations by designing dedicated hardware and using techniques such as parallel computing and pipelined processing to improve computational performance. In tensor computing, the processor calculates the inner product value in parallel through multiple multiplication units, and then completes the matrix addition through addition units. Common acceleration techniques include: parallel processing: using multiple computing units to process different matrix multiplication operations simultaneously, significantly improving computational efficiency; data reuse and cache mapping: reducing software data intervention and memory access bottlenecks through efficient data access strategies and cache mapping mechanisms; matrix partitioning: decomposing large matrices into smaller blocks and performing multiplication operations separately, reducing computational complexity. However, existing tensor computing coprocessors generally suffer from bottlenecks in computing power and memory bandwidth. How to improve memory bandwidth utilization while ensuring high computational density is a key challenge in current design. To address this challenge, several tensor computation methods have been proposed in existing technologies, such as those in patent applications with application numbers "202411874953.0", "202410317105.3", "201911158714.4", "202211469483.0", and "202110256529.X". However, these existing technologies often suffer from the following problems: 1. Using software data rearrangement techniques requires thousands or even tens of thousands of cycles for data access and rearrangement mapping, resulting in low efficiency of software data rearrangement; 2. Tensor computation cannot fully cover... 1. Supports various low-bit numerical precision calculations such as FP32, TF32, BF16, FP16, INT8, INT4, FP8FP6FP4, etc.; 2. Tensor computing coprocessors lack adaptability, only adapting to specific data loading and memory access for specific computing platforms, and cannot flexibly adapt to multiple computing platforms; 3. Supports only a single input matrix size, i.e., only supports specific input matrix size types; 4. Does not fully consider the computing resources of different computing platforms and the differences in on-chip storage and memory access bandwidth between different computing platforms, and cannot adapt to various platforms. For example, tensor computing coprocessors designed for high-computing and resource-rich computing platforms such as GPUs, Neural Network Processing Units (NPUs), System-on-Chips (SoCs), and Application Specific Integrated Circuits (ASICs) will be subject to various limitations when implemented on resource-constrained platforms such as FPGAs and edge platforms.Therefore, designing a tensor computing coprocessor that can address the issues of limited numerical precision, single input matrix size, slow software data rearrangement, and insufficient flexibility in processor parameter configuration has become a noteworthy technical problem.

[0051] Based on the above research, this disclosure provides a tensor computing coprocessor. Because it supports receiving matrix data via a preset bus protocol, it can be adapted to various computing platforms that support such protocols, significantly improving its versatility and adaptability. Since the number of input buffer units, the size of the computing array, and the number of output buffer units can all be flexibly defined using preset hardware parameters, flexible parameter configuration based on the platform resources of the computing platform where the coprocessor is deployed can be achieved, effectively improving the coprocessor's flexibility. Task parameters can be used to indicate various task information for tensor computing tasks, such as matrix size and data type. Therefore, by transmitting task parameters indicating various matrix sizes and data types, and implementing mapping, scheduling, and computation of the matrices to be processed based on these parameters, tensor computing tasks with multiple matrix sizes and data types can be processed, increasing the diversity of supported matrix sizes and data types. Furthermore, the tensor computing coprocessor provided in this application can significantly improve the adaptability of edge platforms and meet the needs of future multi-precision, low-latency, and configurable intelligent tensor computing scenarios.

[0052] The shortcomings of the above solutions are the result of the inventor's practical experience and careful research. Therefore, the discovery process of the above problems and the solutions proposed in this disclosure below are all contributions made by the inventor to this disclosure.

[0053] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0054] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0055] It should be noted that the specific terms mentioned in the embodiments of this disclosure include:

[0056] AXI4: Advanced eXtensible Interface 4, also known as AXI-Full, is a bus protocol primarily used for data transfer between high-performance components within a System-on-a-Chip (SoC).

[0057] AXI4-Lite: It is a lightweight version of the AXI4 protocol family, designed for low-bandwidth, simple control-type peripheral interaction. It is a common protocol used in SoC to connect simple peripherals and the main controller.

[0058] Chisel: Constructing Hardware In a Scala Embedded Language, is a hardware construction language based on Scala. It allows developers to describe digital circuits in the style of high-level programming languages, ultimately generating synthesizable Verilog or VHDL code for FPGA or ASIC design.

[0059] To facilitate understanding of this embodiment, a tensor computation coprocessor disclosed in this disclosure will first be described in detail. The coprocessor provided in this disclosure can be used to perform tensor computation tasks, such as matrix multiplication and addition tasks corresponding to various matrices to be processed, and can be adapted to various computing platforms, supporting input matrices of various sizes and data types. The coprocessor can be adapted to different computing platforms, such as CPUs, GPUs, SoCs, NPUs, FPGAs, edge platforms, etc. The matrices to be processed can be stored in external storage modules (such as Double Data Rate (DDR), Secure Digital Cards (SD cards), cloud storage, etc.).

[0060] like Figure 1 The diagram shown illustrates the architecture of a tensor computation coprocessor according to an embodiment of this disclosure. The tensor computation coprocessor may include a preprocessing remapping module 10, a scheduling computation module 11, and an output mapping module 12. The scheduling computation module 11 is connected to both the preprocessing remapping module 10 and the output mapping module 12. The preprocessing remapping module 10 receives task parameters and matrix data of the matrix to be processed from an external source, and performs mapping of the task parameters and mapping of the matrix data to an input buffer unit based on the task parameters. The scheduling computation module 11 schedules the task parameters and matrix data from the input buffer unit for tensor computation to obtain the computation result; the output mapping module 12 maps the computation result to an output buffer unit for caching.

[0061] Specifically, the preprocessing remapping module 10 is used to receive the task parameters of the tensor calculation task transmitted based on a preset bus, as well as the matrix data of each matrix to be processed corresponding to the tensor calculation task, and to map the task parameters and matrix data to different input buffer units.

[0062] Here, the tensor computation task can specifically be a matrix multiplication and addition task. The matrices to be processed in each tensor computation task can include three matrices, specifically the first matrix, the second matrix, and the third matrix (described below). The first and second matrices undergo matrix multiplication, and the result is then added to the third matrix to obtain the computation result of the tensor computation task. For ease of description, the first matrix can be referred to as matrix A above, the second matrix as matrix B above, and the third matrix as matrix C above. Task parameters are used to indicate task information related to the tensor computation task, such as the matrix size and data type of the matrices to be processed.

[0063] The default bus can be an Advanced eXtensible Interface (AXI) bus. This application can use the AXI4 bus for transmitting matrix data of the matrix to be processed and AXI-Lite for transmitting task parameters. Optionally, when transmitting task parameters to the preprocessing remapping module 10, the default bus protocol can also be an Advanced Peripheral Bus (APB) or an Advanced High-performance Bus (AHB).

[0064] The AXI4 bus supports multiple data transmission modes, one of the most important being burst transfer. Burst transfer refers to requesting the transmission of multiple consecutive data units in a single transaction. In the AXI bus, burst transfer is implemented based on an "address increment" method; that is, during a single transfer, the address increments in a specific manner to achieve the transmission of multiple data units. When using the AXI4 bus for burst transfers, the AXI4 bus protocol transmits data and control signals through five independent channels. Each channel has its own signal lines and does not interfere with each other. The five independent channels will be explained below:

[0065] For the Write Address Channel (AW), the master device can notify the slave device of the address and related information (such as burst length, data size, etc.) to be written. When writing data using the AW channel, the following signals are typically included: AWVALID signal to indicate that the write address is valid; AWREADY signal to indicate that the slave device is ready to receive the write address; AWADDR signal to indicate the target address of the write operation; AAWEN signal to indicate the burst length (e.g., 0 for 1 transmission, 1 for 2 transmissions, and so on, up to a maximum of 255 for 256 transmissions); AWSIZE signal to indicate the number of bytes per transmission (e.g., 2 for 4 bytes, where the number of bytes can be less than the data width of the write data content (WDATA)); and AWBURST signal to indicate the burst type (e.g., fixed type, incrementing type, wraparound type, etc.).

[0066] For the Write Data Channel (W), the master device can send data to the slave device through this channel. When writing data using the W channel, the following signals are typically included: a WVALID signal to indicate that the master device is writing valid data; a WREADY signal to indicate that the slave device is ready to receive data; a WDATA signal to indicate the content of the data being written (typically supporting writing 8 / 16 / 32 / 64 / 128 / 256 / 512 / 1024 bits of data); a WSTRB signal to indicate which bytes are valid; and a WLAST signal to indicate that the currently transmitted data is the last piece of data.

[0067] For the Write Response Channel (B), the slave device can notify whether the write operation was successful. When using the B channel for a write response, it typically includes the following signals: a BVALID signal to indicate that the write response is valid to the slave device, a BREADY signal to indicate that the master device is ready to receive the write response, and a BRESP signal to indicate the response status of the write response (such as OKAU for success and ERROR for failure).

[0068] For the Read Address Channel (AR), the master device can notify the slave device of the address and related information of the data to be read. When using the AR channel for read address notification, the following signals are typically included: the ARVALID signal indicating that the read address is valid; the ARREADY signal indicating that the slave device is ready to receive the address; the ARADDR signal indicating the target address of the read operation; the ARLEN signal indicating the burst length; the ARSIZE signal indicating the number of bytes read per burst; and the ARBURST signal indicating the burst type.

[0069] For the Read Data Channel (R), the slave device can send back the data it has read to the master device. When using the R channel for read data feedback, the following signals are typically included: RVALID signal to indicate that the slave device has read valid data; RREADY signal to indicate that the master device is ready to receive read data; RDATA signal to indicate the content of the read data; RRESP signal to indicate the response status of the transmitted read data; and RLAST signal to indicate that the current read data is the last in a burst transmission.

[0070] The burst transfer mechanism based on the AXI4 bus can move large amounts of data (such as matrix data of matrices to be processed). When the transfer addresses are contiguous, burst transfer in auto-increment mode can be used to transfer each matrix to be processed in the tensor computation task (such as matrix A, matrix B, and matrix C) from external storage to the tensor computation coprocessor via AXI4 burst transfer. During the transfer process, the complete and correct transfer of matrix data can be achieved mainly through three channels: write address, write data, and write response.

[0071] Regarding the AXI4-LITE bus, as a simplified version of the AXI4 bus, it retains the basic functionality of five independent channels but eliminates burst transfers, allowing only one data item to be transferred at a time. Specifically, for the write address / write data / write response channels, the AxBURST burst transfer mode, AxSIZE burst size, and AxLEN burst length related to burst transfers have been removed. This application can utilize the AXI-Lite bus to transfer task parameters of tensor computation tasks to the preprocessing remapping module 10 of the tensor computation coprocessor.

[0072] In one embodiment, the task parameters may include at least the matrix size and data type of each matrix to be processed, the rounding method of the computation results of the computation array, and the storage address of the external memory.

[0073] Here, various task parameters can be transferred by reading and writing configuration parameter registers through the AXI-Lite bus interface. Each task parameter corresponds to a specific register, as shown in Table 1 below, which illustrates the register configuration for each task parameter.

[0074]

[0075]

[0076] (Table 1)

[0077] In Table 1, non-mixed precision calculations for FP32 / FP16 mean that matrices A, B, and C are all of the same data type. Mixed precision calculations for BF16 / TF32 / FP16 / INT8 / INT4 / FP8 / FP6 / FP4 mean that matrices A and B are of the same type, and matrix C is of the corresponding 32-bit integer type INT32 or FP32. For the FP8 data type, there are two specific data formats to choose from: E5M2 and E4M3. In these two formats, E and M represent the number of bits in the exponent and mantissa, respectively. Similarly, for the FP6 data type, there are two specific data formats to choose from: E3M2 and E2M3. In these two formats, E and M represent the number of bits in the exponent and mantissa, respectively.

[0078] For example, an external computing platform (such as a CPU, SOC, etc.) can read and write configuration parameter registers through the AXI-Lite bus interface to complete the reading and writing of task parameters related to tensor computation tasks. Optionally, the external core controller can also use buses such as AHB or APB to complete basic task parameter configuration register reading and writing, realizing the transmission of task parameters. However, considering that the AXI bus is more widely used and can be adapted to more computing platforms, and that AXI-Lite is more suitable for lightweight register reading and writing implementations, in order to improve the versatility of the tensor computation coprocessor of this application, the tensor computation coprocessor of this application preferably uses the AXI-Lite bus to complete the task parameter configuration of the tensor computation coprocessor by the external computing platform. Figure 2 The diagram shown is a schematic of configuring a task parameter register based on the AXI-Lite bus according to an embodiment of this application.

[0079] For the matrix data of each matrix to be processed, this application can use the AXI4 bus to transfer it to the preprocessing remapping module 10 of the tensor computation coprocessor. The following will explain the use of the AXI4 bus to transfer matrix data:

[0080] The matrix data of matrices A, B, and C can all be stored in external memory, and the storage format in external memory is row-major compact storage. That is, the matrix data of each matrix is ​​arranged row-majorly in external memory. Meanwhile, it can be assumed that the matrix data of matrices A, B, and C are not contiguous in storage space and need to be transferred separately to the preprocessing remapping module 10 of the tensor computation coprocessor via burst transfer on the AXI4 bus.

[0081] Furthermore, regarding the data types of the matrices to be processed, when the data type is FP32, a single matrix data occupies 32 bits when stored in the external storage module. When the data type is TF32, because the original data width of TF32 is 19 bits (not a power of 2), which is not conducive to efficient data access, we assume that TF32 data is padded with 13 bits of 0 at the lower end when stored externally, i.e., a 32-bit data width, thus a single matrix data occupies 32 bits. When the data type is BF16, a single matrix data occupies 16 bits, and the same applies when storing matrix data in the external storage module. When the data type is FP16, a single matrix data occupies 16 bits when stored in the external storage module. When the data types are INT8 and FP8, a single matrix data occupies 8 bits when stored in the external storage module. When the data type is FP6, because its data width is not a power of 2, we default to storing it in external storage with low-order zeros padded with 8 bits. This means that when storing matrix data in the external storage module, a single matrix data item occupies 8 bits. When the data types are INT4 and FP4, when storing matrix data in the external storage module, a single matrix data item occupies 4 bits.

[0082] For example, the data types of matrices A, B, and C, and the bit widths of each data after array calculation, corresponding to different task parameter configuration modes, are shown in Table 2 below:

[0083]

[0084]

[0085] (Table 2)

[0086] In practical implementation, the data types of the three matrices to be processed can be any of the data types indicated in Table 2 above. Based on Table 2, this application supports at least FP32, TF32, BF16, FP16, INT8, INT4, FP8, FP6, and FP4 data types, and supports tensor computation of three matrices to be processed with the same data type as well as tensor computation of three matrices to be processed with different data types. Since different data types have different data precisions, the tensor computation coprocessor of this application supports uniform data precision (non-mixed precision) tensor computation of FP32 / FP16, and also supports mixed precision computation of BF16 / TF32 / FP16 / INT8 / INT4 / FP8 / FP6 / FP4.

[0087] In practical implementation, for any tensor computation task, after determining the matrix to be processed for tensor computation, the external computing platform deployed by the tensor computation coprocessor can use the AXI-Lite bus to transmit various task parameters of the tensor computation task to the preprocessing remapping module 10 of the tensor computation coprocessor. Using burst transmission on the AXI4 bus, the matrix data of each matrix to be processed is periodically transmitted to the preprocessing remapping module 10 of the tensor computation coprocessor in row-major order. Thus, the preprocessing remapping module 10 can receive the task parameters and the matrix data of each matrix to be processed.

[0088] Furthermore, since the tensor computation coprocessor of this application supports various parameters such as numerical precision and matrix size, and can handle various types of tensor computation tasks according to the configuration of the external computing platform, in order to process tensor computation tasks of each numerical precision and matrix size, the task parameters also need to be cached together with the matrix data in the input cache unit, and then transmitted to the computing array to form a pipelined computation scheduling process, making full use of the internal computing resources of the tensor computation coprocessor. The input cache unit is a cache unit used to store various information input to the tensor computation coprocessor, and can be located inside the tensor computation coprocessor.

[0089] For example, in the section on dynamically configuring the task parameters of the tensor computation coprocessor, the read / write configuration of task parameters via the AXI-Lite bus and the transmission of matrix data via the AXI4 bus have been described. Therefore, after the preprocessing remapping module 10 receives the task parameters and matrix data, it can map the task parameters and matrix data to different input buffer units. For example, the task parameters are mapped to one input buffer unit, and the matrix data of different matrices to be processed are mapped to different input buffer units respectively. Then, after the matrix data of each matrix to be processed has been mapped, the task parameters and matrix data are passed together from the input buffer units to the computation array for tensor computation. When mapping matrix data, the preprocessing remapping module 10 can determine whether to pad the matrix data with zeros for each matrix to be processed based on the matrix size and data type of the matrix to be processed as indicated by the task parameters. If so, the zero-padded matrix data is mapped to the input buffer unit; otherwise, the matrix data can be directly mapped to the input buffer unit.

[0090] After the preprocessing remapping module 10 completes the mapping cache, the scheduling calculation module 11 can be used to schedule task parameters from different input cache units to the target calculation array that matches the task parameters, and to schedule block scheduling matrix data to the target calculation array for tensor calculation for the same calculation tensor task, so as to obtain the calculation results and block indexes corresponding to each block scheduling.

[0091] Here, the computation array refers to the array used for matrix data computation within the tensor computation coprocessor. The tensor computation coprocessor can include different computation arrays, each adapted to different task parameters. That is, the tensor computation coprocessor can deploy computation arrays suitable for different matrix sizes and data precisions. For the same tensor computation task, the scheduling computation module 11 can select a target computation array from the deployed arrays that matches the task parameters stored in the input buffer unit. Then, the scheduling computation module 11 can schedule the task parameters to the target computation array and perform block scheduling on the matrix data stored in different input buffer units. Each block scheduling schedules one block of data from the input buffer units corresponding to matrices A, B, and C, respectively, and inputs it to the target computation array. A computation is performed using each block of data and the task parameters to obtain the computation result and block index corresponding to each block scheduling. The block index indicates the order of each block scheduling and can also indicate the address in the output buffer unit where the computation result needs to be located. Simultaneously, the block index is both the index for each block scheduling and the index for the corresponding computation result. The output buffer unit is used to cache the tensor calculation results of the three matrices to be processed.

[0092] The scheduling and computation module 11 can determine the block data size corresponding to each matrix to be processed in each block scheduling based on the size of the selected target computation array and the matrix size and data type of the three matrices to be processed. Then, for each scheduling, according to the block data size corresponding to each matrix to be processed, one block of data for tensor computation can be scheduled from the input buffer unit corresponding to each matrix to be processed. The scheduled block data corresponding to the three matrices to be processed is then scheduled to the target computation array for matrix multiplication and addition calculations. The calculation results are rounded according to the rounding method indicated by the task parameters, thereby obtaining a calculation result and block index corresponding to each block scheduling. After the last scheduling is completed, all matrices of each matrix to be processed have completed the corresponding matrix multiplication and addition processing.

[0093] Furthermore, the output mapping module 12 is used to map the calculation results corresponding to each block scheduling to the output cache unit according to the block index; the number of input cache units, the size of the computing array and the number of output cache units are defined according to preset hardware parameters.

[0094] Here, the preset hardware parameters can be the hardware parameters related to the tensor computing coprocessor defined in the Chisel language. In this embodiment, various configuration parameters of the tensor computing coprocessor can be pre-configured using Chisel parameters. These configuration parameters may include the number of input cache units, the size of each computing array, the number of output cache units, the AXI bus transmission bandwidth, etc. After the Chisel parameters are configured, the architecture of the tensor computing coprocessor is determined accordingly. That is, in this embodiment, all configuration parameters of the tensor computing coprocessor can be configurably defined using Chisel to generate the corresponding tensor computing coprocessor hardware. This not only allows the scheduling computing module to flexibly adapt to matrix computing requirements with different data precisions and matrix sizes, but also eliminates the need for complex redesign and modification of the tensor computing coprocessor.

[0095] In specific implementation, the output mapping module 12 can determine the address of each calculation result in the output cache unit according to the block index of the calculation result corresponding to each block scheduling, and map each calculation result to the output cache unit for storage according to the address.

[0096] In one embodiment, the preprocessing remapping module 10 includes a task parameter mapping module and a matrix data mapping module; the input cache unit includes a task parameter cache unit and a matrix data cache unit; the number of task parameter cache units and the number of matrix data cache units are defined using preset hardware parameters based on the resource information of the computing platform deployed by the tensor computing coprocessor.

[0097] Here, the tensor computing coprocessor can be deployed on any computing platform supporting the AXI bus, such as FPGA, CPU, MCU, etc. The task parameter mapping module, also known as the parameter data mapping module, is used to map the task parameters received by the tensor computing coprocessor to task parameter buffers for scheduling by the scheduling computing module 11. The matrix data mapping module can be used to map matrix data in the input buffer to matrix data buffers. Different matrices to be processed can correspond to different matrix data mapping modules, and different matrices to be processed can also correspond to different matrix data buffers. The number of matrix data buffers corresponding to each matrix to be processed can be set according to the resource information (such as storage space, computing requirements, etc.) of the computing platform on which the tensor computing coprocessor is deployed, and the number of task parameter buffers can also be set according to the resource information (such as on-chip storage space, computing requirements, etc.) of the computing platform on which the tensor computing coprocessor is deployed. The task parameter buffer can be located inside the task parameter mapping module, and the matrix data buffer can be located inside the matrix data mapping module.

[0098] The number of task parameter cache units can be the same as the number of matrix data cache units corresponding to each matrix to be processed. For example, if the computing resources meet the preset resource requirements, two task cache units and two matrix data cache units can be set for each matrix to be processed. If the computing resources do not meet the preset resource requirements, one task cache unit and one matrix data cache unit can be set for each matrix to be processed. This application does not impose specific restrictions on the number of task parameter cache units and matrix data cache units, but emphasizes the trade-off between resource constraints and computational pipelined operation between single and multiple cache units, which can be flexibly configured through the Chisel parameter.

[0099] The task parameter mapping module can be used to receive task parameters transmitted via a preset bus and map the task parameters to the idle task parameter cache unit when there is an available idle task parameter cache unit.

[0100] For example, regardless of the number of configured task parameter cache units, the task parameter mapping module can only receive task parameters transmitted via the AXI-Lite bus and map the task parameters to the idle task parameter cache units if there are currently idle task parameter cache units available.

[0101] If there are no available task parameter cache units, it is necessary to wait for an available task parameter cache unit to be released before using the task parameter mapping module to receive task parameters transmitted by the external computing platform via the AXI-Lite bus.

[0102] The matrix data mapping module can be used to receive matrix data of each matrix to be processed based on a preset bus transmission when there are currently idle matrix data buffer units, and map the matrix data to different matrix data buffer units.

[0103] For example, regardless of the number of matrix data cache units configured for each matrix to be processed, the matrix data mapping module can only receive the matrix data of each matrix to be processed transmitted via the AXI bus and map the matrix data of each matrix to be processed to the matrix data cache unit corresponding to the matrix to be processed if there are currently idle matrix data cache units for each matrix to be processed.

[0104] The scheduling and computation module 12 can be used to schedule task parameters from the task parameter cache unit to the target computation array for the same computation tensor task, and to schedule matrix data in the matrix data cache unit to the target computation array in blocks for tensor computation, so as to obtain the computation results and block indexes corresponding to each block scheduling.

[0105] In specific implementation, for the same tensor computation task, the scheduling computation module 12 can be used to obtain task parameters from the task parameter cache unit and schedule them to the target computation array that matches the task parameters, and to divide the matrix data in the matrix data cache unit corresponding to each matrix to be processed into blocks and schedule them to the target computation array that matches the task parameters for tensor computation, so as to obtain the computation result and block index corresponding to each block scheduling.

[0106] For example, a cache unit can be called a buffer. When deploying the tensor computation coprocessor on different computing platforms that support the AXI bus, considering the different limitations of on-chip storage space on different computing platforms, the tensor computation coprocessor of this application can be configured to enable single-buffer caching mode, dual-buffer caching mode, or multi-buffer caching mode to adapt to different platform resources. For example, when the number of task parameter cache units is one, it can be determined that the task parameter mapping module of this application is in single-buffer caching mode. That is, only one task parameter cache unit is used to store task parameters. In single-buffer caching mode, it is not possible to process new external tensor computation tasks simultaneously during computation. Only after the previous tensor computation task is completed will the task parameter cache unit be released, thus allowing the task parameters of the new tensor computation task to be written. Similarly, when the number of matrix data cache units corresponding to each matrix to be processed is also one, it can be determined that the matrix data mapping module of this application is in single-buffer caching mode. That is, for each matrix to be processed, only one corresponding matrix data cache unit is used for matrix data storage. In the single buffer cache mode, it is not possible to process new external tensor calculation tasks at the same time during the calculation. Only after the previous tensor calculation task is completed can the matrix data cache unit corresponding to each matrix to be processed be released, and then the matrix data of each data to be processed corresponding to the new tensor calculation task can be written.

[0107] In one embodiment, if both the number of task parameter cache units and the number of matrix data cache units are multiple, then after the scheduling calculation module outputs the calculation results and block indices corresponding to each block scheduling, the preprocessing remapping module can switch the task parameter cache units and the new matrix data cache units corresponding to the new tensor calculation task. The scheduling calculation module 12 is used to schedule new task parameters from the switched task parameter cache units to the new target calculation array, and to schedule new matrix data from the switched matrix data cache units to the new target calculation array for tensor calculation. The new matrix data refers to the matrix data of each matrix to be processed corresponding to the new tensor calculation task, and the new task parameters refer to the task parameters corresponding to the new tensor calculation task. The new target calculation array is the calculation array that matches the new task parameters.

[0108] For example, when both the task parameter cache unit and the matrix data cache unit are two, it can be determined that both the task parameter mapping module and the matrix data mapping module enable dual-buffer caching mode (i.e., the two cache units are stored alternately). For the task parameter mapping module, the writing and output of task parameters can be updated according to the ping-pong read-write mechanism. In dual-buffer caching mode, the writing and reading of task parameters are isolated, so tensor calculation and input data processing can be executed in parallel. When the computing array completes the reading of input data (including task parameters and matrix data) and completes the corresponding tensor calculation task, it sends a finish signal to the AXI bus task parameter configuration and preprocessing remapping module 10, thereby instructing the control to switch the selection of a new output buffer (i.e., a new task parameter cache unit), so that the scheduling computing module 12 can obtain the task parameters corresponding to the new tensor calculation task from the switched output buffer and schedule it to the new target computing array for tensor calculation.

[0109] Similarly, in the dual-buffer caching mode, the matrix data mapping module can also update the writing and output of matrix data according to the ping-pong read-write mechanism. In the dual-buffer caching mode, the writing and reading of new matrix data are isolated, allowing tensor computation and input data processing to be executed in parallel. After the computing array completes the reading of input data (including task parameters and matrix data) and completes the corresponding tensor computation task, it sends a finish signal to the AXI bus task parameter configuration and preprocessing remapping module 10, thereby instructing the control to switch the selection of a new output buffer (i.e., a new matrix data cache unit). This allows the scheduling computing module 12 to obtain the matrix data corresponding to the new tensor computation task from the switched output buffer and schedule it in blocks to the new target computing array for tensor computation. It should be noted that when integrating the tensor computation coprocessor of this application into the required computing platform, the number of cache units can be flexibly configured according to processor configuration requirements and computing resources. The number of cache units does not affect the correctness of the computation, but only relates to the computational throughput, whether data writing and computation are isolated, and whether they can be executed in parallel.

[0110] Understandably, when both the number of task parameter cache units and the number of matrix data cache units are greater than two, it can be determined that both the task parameter mapping module and the matrix data mapping module enable multi-buffer caching mode (i.e., multiple cache units are stored alternately). The writing and reading of task parameters and matrix data in multi-buffer caching mode are similar to those in dual-buffer caching mode and will not be elaborated here. For example, multiple task parameter cache units may include task parameter buffer0, task parameter buffer1, and task parameter buffer2, and the matrix data cache unit corresponding to each matrix to be processed may include matrix data buffer0, matrix data buffer1, and matrix data buffer2. After the computing array finishes reading the input data from task parameter buffer0 and matrix data buffer0 and completes the tensor calculation task, it sends a finish signal to the AXI bus task parameter configuration and preprocessing remapping module 10, thereby instructing the control to switch to task parameter buffer1 and matrix data buffer1; at this time, task parameter buffer0 and matrix data buffer0 can be released and used to receive task parameters and matrix data corresponding to new tensor calculation tasks. After the computing array reads the input data for a new tensor computation task from task parameter buffer1 and matrix data buffer1 and completes the tensor computation task, it sends a "finish" signal to the AXI bus task parameter configuration and preprocessing remapping module 10, thereby instructing control to switch to task parameter buffer2 and matrix data buffer2. At this time, task parameter buffer1 and matrix data buffer1 can be released and used to receive the task parameters and matrix data corresponding to the new tensor computation task. After the computing array reads the input data for a new tensor computation task from task parameter buffer2 and matrix data buffer2 and completes the tensor computation task, it sends a "finish" signal to the AXI bus task parameter configuration and preprocessing remapping module 10, thereby instructing control to switch to task parameter buffer0 and matrix data buffer0. At this time, task parameter buffer2 and matrix data buffer2 can be released and used to receive the task parameters and matrix data corresponding to the new tensor computation task.

[0111] Here, this application further emphasizes that the appropriate number of cache buffers can be selected according to the on-chip resource limitations of different computing platforms. When there is sufficient cache for input data, the input data and computation data can be better isolated, so that the tensor computation coprocessor can avoid blocking the writing of input data due to incomplete computation.

[0112] In one embodiment, the matrices to be processed include a first matrix, a second matrix, and a third matrix. The matrix data mapping module includes a temporary storage module and a mapping module corresponding to each matrix to be processed. The first and second matrices can be two matrices used to perform matrix multiplication, and the third matrix can be a matrix added to the result of the matrix multiplication. For example, the first matrix is ​​matrix A mentioned above, the second matrix is ​​matrix B mentioned above, and the third matrix is ​​matrix C mentioned above. The data mapping module can include a temporary storage module and a mapping module corresponding to each matrix to be processed. The temporary storage module is used to temporarily store the matrix data passed in periodically via the AXI4 bus, and the mapping module maps the matrix data in the temporary storage module to the input buffer unit. The mapping module can be connected to both the temporary storage module and the input buffer unit.

[0113] Specifically, the matrix data mapping module is used to receive the matrix to be processed and matrix category indication information transmitted in row-major format via a preset bus, and to store the received matrix data in the corresponding temporary storage module of the matrix to be processed according to the matrix category indication information.

[0114] Here, the external computing platform can use the AXI4 bus to transfer the various matrices to be processed stored in external memory to the matrix data mapping module in row-major format. Matrix category indication information is used to indicate the matrix type corresponding to the transferred matrix data. The matrix type can include three types: first matrix type, second matrix type, and third matrix type. Specifically, the matrix category indication information can be the address range or user identifier of the matrix data.

[0115] For example, in the tensor computation coprocessor design of this application, the matrix data mapping module can receive input matrix data through the AXI4 bus interface, and distinguish between matrix A, matrix B, and matrix C by the source address range of the written matrix data (e.g., 0x00000000 represents matrix A, 0x00010000 represents matrix B, and 0x00020000 represents matrix C) or by the user bit in the AXI4 bus (e.g., user bit 0 represents matrix A, user bit 1 represents matrix B, and user bit 2 represents matrix C). After distinguishing the matrix type corresponding to the matrix data, the input matrix data can be temporarily stored in the data temporary storage registers of the corresponding temporary storage modules, thereby isolating the input timing path.

[0116] The mapping module is used to map each row of matrix data of the matrix to be processed in the temporary storage module to the matrix data cache unit corresponding to the matrix category indication information of the matrix to be processed, according to the data type indicated by the task parameters.

[0117] In practical implementation, the external computing platform can periodically transmit each row of matrix data from each matrix to be processed to the matrix data mapping module via the AXI4 bus, following a row-major format. The matrix data mapping module can temporarily store each row of matrix data for each matrix to be processed in its corresponding temporary storage module. Then, for each mapping module corresponding to a matrix to be processed, the module can map each row of matrix data in the temporary storage module to a matrix data cache unit corresponding to the matrix category indication information of the matrix to be processed, based on the data type of the matrix indicated by the task parameters. For example, the mapping module for matrix A can map each row of matrix data in the temporary storage module to the matrix data cache unit corresponding to matrix A.

[0118] In one embodiment, the tensor computation coprocessor in this application uses an array computation form of inner product. It uses a 32-bit floating-point high-precision computation array for 32-bit data such as FP32, BF16, and TF32, while using a mixed-precision computation array (low-precision scheduling array) for FP16, INT8, INT4, FP8, FP6, and FP4 data types. All other data types are aligned with the data width of FP16, thus reusing the mixed-precision computation array. Therefore, the mixed-precision computation array can also be considered an FP16 computation array. When calculating A*B+C, both computation arrays require matrix A to be in row-major order, matrix B to be in column-major order, and matrix C to be in row-major order. Specifically, the tensor computation process is implemented by the scheduling computation module 11, which inputs the matrix data of matrices A, B, and C into the computation array according to the state machine, thus obtaining the final computation result in cycles. The computation result is also a matrix, specifically matrix D as described above; the computation result is the matrix data in matrix D.

[0119] Table 3 below illustrates the matrix size and numerical precision supported by the two computing arrays provided in the embodiments of this application:

[0120]

[0121] (Table 3)

[0122] In Table 3 above, computation array 1 can be a high-precision computation array, and computation array 2 can be a low-precision computation array. It is understood that the various matrix sizes and data precision values ​​described above are merely examples in this application. In specific implementations, the tensor computation coprocessor provided in this application can also be applied to tensor computations with other matrix sizes and data precision values. Only the computation array needs to be modified accordingly based on the actual matrix size and data precision used.

[0123] Since the data width of BF16 is 16 bits, when transmitting BF16 type matrix data to the tensor computation coprocessor, if each data element in the matrix data is not padded with zeros to 32 bits before transmission, then in order to ensure that the BF16 type matrix data can reuse the FP32 computation array in subsequent tensor computations, the matrix data mapping module can also be used for:

[0124] After storing the received matrix data of each row into the corresponding temporary storage module of the matrix to be processed according to the matrix category indication information, for any matrix to be processed, if the data type of the matrix to be processed is the first data type, then each row of matrix data of the matrix to be processed is padded with zeros to the first target bit width according to each data element; the bit width corresponding to the first data type is less than the first target bit width.

[0125] Here, the first data type can specifically be BF16, and the first target bit width is 32 bits. Since the matrix to be processed is transmitted to the matrix data mapping module row by row, each transmitted matrix data is a row of data. This row of data can include multiple data elements, and the number of multiple data elements can specifically be the number of columns of the matrix to be processed.

[0126] For example, after the matrix data mapping module stores the row matrix data corresponding to matrix A into the temporary storage module corresponding to matrix A, if the data type of matrix A is BF16, it can pad each data element in the row matrix data with 16 bits of zeros to obtain each data element of 32 bits.

[0127] Understandably, if the data type of the matrix to be processed is not a primary data type, the mapping method for the matrix data can be determined based on the specific data type and size of the matrix. The mapping process for each of the three matrices to be processed will be described below:

[0128] In one embodiment, the temporary storage module includes a first temporary storage module corresponding to the first matrix, the mapping module includes a first mapping module corresponding to the first matrix, and the matrix data cache unit includes a first matrix data cache unit corresponding to the first matrix. The first temporary storage module can be connected to both the first mapping module and the first matrix data cache unit.

[0129] The first mapping module is used to, when the task parameters indicate that the column size of the first matrix is ​​a first preset size and the bit width corresponding to the data type of the first matrix is ​​less than the first preset bit width, pad each row of matrix data stored in the first temporary storage module with zeros to obtain matrix data with a column size of a second preset size and map it to the corresponding row in the first matrix data cache unit; or, when the task parameters indicate that the column size of the first matrix is ​​a first preset size and the bit width corresponding to the data type of the first matrix is ​​not less than the first preset bit width, or when the task parameters indicate that the column size of the first matrix is ​​not a first preset size, map each row of matrix data stored in the first temporary storage module to the corresponding row in the first matrix data cache unit.

[0130] Here, the first matrix is ​​matrix A, the first temporary storage module can be an A data store module, the first mapping module can be an A matrix data mapping module, and the first matrix data buffer unit can be an A matrix data buffer. The first matrix has a size of M×K, so the column size is K. The first preset size is 16. The first preset bit width is 16 bits. If the bit width of the matrix data is less than 16 bits, it indicates that the matrix data is low-precision data. Row-wise zero padding means padding with zeros at the end of each row of matrix data. The second preset size is related to the data type of the first matrix. If the bit width of the first matrix's data type is 8 bits, then the second preset size is 32; if the bit width of the first matrix's data type is 4 bits, then the second preset size is 64. By expanding the column size, matrix data with a bit width of less than 16 bits can be aligned to a 16-bit bit width, thus facilitating the reuse of the computational array corresponding to FP16.

[0131] For example, the function of A Matrix data mapping is to convert the row-matrix data input by AXI4 each cycle into the format required by the computation array and store it in the internal A Matrix data buffer. That is, data mapping can be performed for each row of A matrix data input. The storage mapping of the A matrix data needs to retain the row-matrix data format, i.e., the same as the input A matrix data format, but for the column size K=16 of the A matrix, it needs to perform K-dimensional zero-padding expansion on the low-bit width data. This allows the operation of zero-padding the A matrix, which is currently performed outside the tensor computation coprocessor at the software level in existing technologies, to be implemented inside the tensor computation coprocessor. The specific zero-padding operation is as follows:

[0132] When the configuration parameter K is 16 (i.e., the column size K of matrix A is 16), and the bit width of the data type of matrix A is less than 16 bits (8 bits / 4 bits):

[0133] If the data type of matrix A has a bit width of 8 bits, then K needs to be expanded from 16 to 32 by padding zeros in each row (that is, each row of matrix data is padded with 16*8=128 bits of zeros) before being stored in A Matrix data Buffer.

[0134] If the data type of matrix A has a bit width of 4 bits, then K needs to be expanded from 16 to 64 by padding with zeros in each row (that is, 16 * 12 = 192 bits of zeros are added to each row) before being stored in A Matrix data Buffer.

[0135] When storing matrix data into a Matrix data Buffer, the entire data is stored in row-major order.

[0136] Thus, when K=16, if the bit width of matrix A is less than 16 bits, by padding each row with 0s to align it to 16 bits for data processing, i.e. aligning the data processing size under the FP16 data type, the internal computation scheduling and dot product array can be reused without adding a separate custom computation array module or changing the internal computation array scheduling of the tensor computation coprocessor.

[0137] For the case where K=16, if the data type of matrix A is BF16, then each data element in the matrix data can be padded with zeros to 32 bits before being mapped to A Matrix data Buffer.

[0138] like Figures 3a-3d The diagram shown illustrates the data mapping of a 16×16 matrix A provided in this application under different data types. Specifically, in... Figure 3a In this context, the data type of matrix A has a bit width of 32 bits, with one square representing a 32-bit data element. During data mapping, the matrix data of each row of matrix A can be directly mapped to the corresponding row of the AMatrix data buffer. Figure 3b In this context, the data type of matrix A has a bit width of 16 bits, with one square representing a 16-bit data element. During data mapping, the matrix data of each row of matrix A can be directly mapped to the corresponding row of the AMatrix data buffer. Figure 3cIn this context, the data type of matrix A has a bit width of 8 bits, with one square representing an 8-bit data element. After obtaining the matrix data transmitted via the AXI4 bus, it is necessary to perform zero-padding in the K dimension to obtain matrix data of matrix A in the shape of 16×32, and then map it to the corresponding row of the A Matrix data Buffer. Figure 3c The green portion represents the 8-bit input matrix data, the gray portion represents the zero-padded matrix data, and the blue portion represents the remaining space in the A Matrix data buffer. Figure 3d In this context, the data type of matrix A has a bit width of 4 bits, with one square representing a 4-bit data element. After obtaining the matrix data transmitted via the AXI4 bus, it is necessary to perform zero-padding in the K dimension to obtain the matrix data of matrix A in the shape of 16×64, and then map it to the corresponding row of the A Matrix data Buffer. Figure 3d The green part represents the 4-bit wide matrix data input, the gray part represents the zero-padded matrix data, and the blue part represents the remaining space in the AMatrix data Buffer.

[0139] Regarding the data mapping process of matrix A under different data types when the matrix size is 8×16 / 32×16, the above is the same. Figures 3a-3d The process is similar, therefore, the data mapping of the A matrix under the 8×16 / 32×16 size will not be described in detail in this embodiment. The specific mapping process can be referred to Figures 3a-3d .

[0140] like Figure 4a The diagram shown illustrates a data mapping of a 32×32 matrix A under a specific data type, as provided in an embodiment of this application. Wherein... Figure 4a In this example, matrix A has a size of 32×32 and a data type bit width of 8 bits. One square represents an 8-bit data element. During data mapping, the matrix data of each row of matrix A can be directly mapped to the corresponding row of the A Matrix data Buffer. Regarding the process of mapping the matrix data of each row of matrix A to the corresponding row of the A Matrix data Buffer when the size of matrix A is 16×32 / 8×32 and the data type bit width is 8 bits, [the following is unclear and requires further context]. Figure 4a The process shown is similar and will not be repeated here. Figure 4b The diagram shown illustrates a data mapping of a 32×64 matrix A under a specific data type, as provided in an embodiment of this application. Figure 4bIn this example, matrix A has a size of 32×64 and a data type bit width of 4 bits. Each square represents a 4-bit data element. During data mapping, the matrix data of each row of matrix A can be directly mapped to the corresponding row of the A Matrix data Buffer. Regarding the process of mapping the matrix data of each row of matrix A to the corresponding row of the A Matrix data Buffer when the size of matrix A is 16×64 / 8×64 and the data type bit width is 4 bits, [the following is unclear and requires further context]. Figure 4b The process shown is similar, so it will not be repeated here.

[0141] In one embodiment, the first temporary storage module is provided with a first buffer area of ​​a second preset bit width; the matrix data mapping module is used to store the received matrix data of each row of the first matrix into the first buffer area when the data transmission bandwidth of the preset bus is less than the preset bandwidth, if the column size of the first matrix is ​​a first preset size and the bit width corresponding to the data type of the first matrix is ​​less than the first preset bit width; the first mapping module is used to retrieve each row of matrix data from the first buffer area after the first buffer area is full, pad it with zeros row by row to obtain matrix data with a column size of the second preset size and map it to the corresponding row in the first matrix data cache unit.

[0142] Here, the second preset bit width can be the row buffer size of matrix A. Its size can be set according to the size of a single row of matrix data in matrix A. For example, when K=16 and the bit width of matrix A is 8 bits, the size of a single row of matrix data is 128 bits, and the row buffer size can be 128 bits. The data transfer bandwidth can be predefined using the Chisel parameter, and the data transfer bandwidth is related to the transmission capability of the computing platform. Once the Chisel parameter is configured, the data transfer bandwidth is determined. The preset bandwidth can be 64 bits. If a row of matrix data stored in the A data store is the first row of matrix A, then after padding with zeros, this row of matrix data can be mapped to the first row of the A Matrix data Buffer. If a row of matrix data stored in the A data store is the Kth row of matrix A, then after padding with zeros, this row of matrix data can be mapped to the Kth row of the A Matrix data Buffer.

[0143] For example, since this application uses the AXI4 bus to transmit matrix data, and the data transmission bandwidth of the AXI4 bus can be configured to 32 / 64 / 128 / 256 / 512 / 1024, when the data transmission bandwidth is less than 64 bits (e.g., when the data transmission bandwidth is 32 / 64 bits), there may be situations where one cycle of transmission is insufficient for one row of matrix A for 4-bit and 8-bit matrix data (e.g., when K=16 of matrix A and the data width is 8 bits, one row of matrix A data is 8*16=128 bits; when K=16 of matrix A and the data width is 4 bits, one row of matrix A data is 4*16=64 bits). In this case, multiple cycles are required to transmit a complete row of matrix data. To ensure that the A Matrix data Buffer stores a complete row of matrix data, it is necessary to wait until at least one row of matrix data has been transmitted before mapping the matrix data to the A Matrix data Buffer. Therefore, a first buffer of size 128 bits can be set in data store A to collect matrix data. When the data transmission bandwidth of the AXI4 bus is 32 / 64, if K=16 of matrix A, zeros can be padded row by row after caching one row of 8-bit data in the first buffer, or zeros can be padded row by row after caching two rows of 4-bit data in the first buffer.

[0144] A Matrix data Mapping module can retrieve each row of matrix data from the first cache after the first cache is full, pad each row of matrix data with zeros, obtain matrix data with column size of the second preset size, and map it to the corresponding row in the first matrix data cache unit.

[0145] If the data transmission capacity of the preset bus exceeds the preset bandwidth, the matrix data mapping module can directly store the received entire row of matrix data into data store A, and perform a one-clock delay within data store A to buffer the matrix data and isolate the input timing path. Then, the first mapping module can directly retrieve the entire row of data from data store A, pad it with zeros row by row, and then map it to the corresponding row in matrix data buffer A.

[0146] For example, when the data transmission bandwidth of the AXI4 bus is 128 / 256 / 512 / 1024, the input matrix data is integer row data, which can be directly padded with zeros row by row. Therefore, the matrix data mapping module can store the received matrix data inside a data store (A data store), and only delay the data by one clock cycle within the A data store to buffer the data, isolate the input timing path, and then the A Matrix data Mapping module maps it to the corresponding row in the A Matrix data Buffer.

[0147] When the column size of the first matrix indicated by the task parameters belongs to the first preset size and the bit width corresponding to the data type of the first matrix is ​​not less than the first preset bit width (i.e., when K of matrix A is 16 and the bit width of the data type is 16bit / 32bit), or when the column size of the first matrix indicated by the task parameters does not belong to the first preset size (i.e., when K of matrix A is 32 and the bit width of the data type is 4bit, or K is 64 and the bit width of the data type is 8bit), no additional zero-padding operation is required on the matrix data input to matrix A inside the tensor computation coprocessor. The first mapping module can directly write the valid matrix data of matrix A output by the A data store module into the corresponding row position in the A Matrix data Buffer according to the input row-major format.

[0148] In one embodiment, the temporary storage module includes a second temporary storage module corresponding to the second matrix, the mapping module includes a second mapping module corresponding to the second matrix, and the matrix data cache unit includes a second matrix data cache unit corresponding to the second matrix. The second temporary storage module can be connected to both the second mapping module and the second matrix data cache unit.

[0149] The second mapping module is used to calculate the mapping address of each data element in the second matrix data cache unit for each row of matrix data stored in the second temporary storage module, based on the matrix size and data type of the second matrix indicated by the task parameters, and to update the preset value at the mapping address using each data element.

[0150] Here, the second matrix is ​​matrix A, the second temporary storage module can be the B data storage module, the second mapping module can be the B matrix data mapping module, and the second matrix data buffer unit can be the B matrix data buffer.

[0151] For example, the function of the B Matrix data Mapping module is to convert the row-major data input by AXI4 each cycle into the format required by the computing array and map it into the internal B Matrix data Buffer. Since the B matrix data needs to be mapped to a column-major data format, and the B matrix is ​​passed to the second temporary storage module row by row, the matrix data needs to be transposed during mapping. Therefore, the B Matrix data Mapping module can obtain each row of matrix data from the B data store, then determine each data element in each row of matrix data according to the matrix size and data type of the B matrix indicated by the task parameters, and transpose each row of matrix data according to the row order. Specifically, during transposition, transposition and mapping can be performed as soon as each row of matrix data is transmitted by the AXI bus, without waiting for all rows of matrix data to be transmitted before transposition and mapping. For each row of matrix data, the number of data elements can be expanded, and the mapping address of each data element in that row of matrix data in the B Matrix data Buffer can be calculated in parallel. Each data element is then written to the corresponding mapping address, thereby realizing the transposition and mapping of the B matrix data. Here, the mapped address is the row and column address of the data element within the B Matrix data Buffer. The B Matrix data Buffer can be a space of a specific size, capable of storing matrix data of various data types supported by the tensor computation coprocessor, with each storage address defaulting to a preset value (e.g., 0). Whenever a data element of the B matrix is ​​mapped, the preset value at the corresponding storage address can be updated using the data element, thus writing the data element. Storage addresses that are not updated retain their initial values. After the computation array completes a tensor computation task, the values ​​at all storage locations in the B Matrix data Buffer can be restored to their preset values.

[0152] In one embodiment, the second temporary storage module is provided with a second buffer of a third preset bit width. The matrix data mapping module is used to store the received row matrix data of the second matrix into the second buffer if the task parameters indicate that the row size of the second matrix is ​​a first preset size and the bit width corresponding to the data type of the second matrix is ​​less than the first preset bit width, when the data transmission bandwidth of the preset bus is less than the preset bandwidth. The second mapping module is used to retrieve each row of matrix data from the second buffer after the second buffer is full, calculate the mapping address of each data element in that row of matrix data in the second matrix data buffer unit, and update the preset value at the mapping address using each data element.

[0153] Here, the size of the second matrix is ​​K×N, and the row size of the first matrix is ​​K. The third preset bit width can be the row buffer size of the B matrix, which can be the same as or different from the second preset bit width mentioned above. The row buffer size of the B matrix can be determined based on the size of a row of matrix data in the B matrix, as well as the transpose and mapping algorithm of the matrix data in the B matrix. For example, the row buffer size of the B matrix can be 128 bits, 256 bits, etc. The data transmission bandwidth can be predefined using the Chisel parameter, and the data transmission bandwidth is related to the transmission capability of the computing platform. The preset bandwidth can be 64 bits. If a row of matrix data stored in the B data store is the first row of the B matrix, then the matrix data of that row is transposed and mapped to the first column of the BMatrix data Buffer. If a row of matrix data stored in the B data store is the Kth row of the B matrix, then the matrix data of that row is transposed and mapped to the Kth column of the BMatrix data Buffer. The first preset size is 16, and the first preset bit width is 16 bits. If the bit width of the matrix data is less than 16 bits, it indicates that the matrix data is low-bit precision data.

[0154] For example, when the task parameter K is 16 and the bit width of the data type of matrix B is less than 16 bits (e.g., 8 bits / 4 bits), since the data transmission bandwidth of AXI4 can be configured to 32 / 64 / 128 / 256 / 512 / 1024, there may still be cases where one cycle of data transmission for 4-bit and 8-bit matrix data is insufficient to transmit one row of matrix B (for example, when K=16 and the data bit width is 8 bits, one column of matrix B data is 8*16=128 bits; when K=16 and the data bit width is 4 bits, one column of matrix B data is 4*16=64 bits). In this case, multiple cycles are required to transmit an entire column of matrix data. To ensure that the B Matrix data Buffer stores a complete row of matrix data, it is necessary to wait until at least one row of matrix data has been transmitted before mapping the matrix data to the B Matrix data Buffer. Therefore, a second buffer of row buffer size (e.g., 128 bits) can be set in the B data store to collect matrix data. When the data transmission bandwidth of the AXI4 bus is 32 / 64, if K=16 of the B matrix, the entire row of B matrix data received in the B data store can be transposed and mapped after one row of 8-bit data is cached in the second buffer, or the entire row of B matrix data received in the B data store can be transposed and mapped after two rows of 4-bit data are cached in the second buffer.

[0155] The B Matrix data Mapping module can retrieve each row of matrix data from the second buffer after the second buffer is full, and then perform transpose mapping on each row of matrix data.

[0156] During transpose mapping, for each row of matrix B, the matrix data is expanded according to the number of elements, and its mapping address in the BMatrix data Buffer is calculated in parallel. Simultaneously, the initial value of the BMatrix data Buffer is set to all 0s. When a valid input data element is received, the transpose mapping address is calculated, and only the data at the mapping address of the valid input data element is updated. After each use of the BMatrix data Buffer, the used buffer is cleared to 0, restoring the initial value to 0, thus completing the "zero-padding" operation.

[0157] If the data transmission capacity of the preset bus exceeds the preset bandwidth, the matrix data mapping module can directly store the received entire row of matrix data into the B data store, and perform a one-clock delay within the B data store to buffer the matrix data and isolate the input timing path. Then, the second mapping module can directly retrieve the entire row of data from the B data store and calculate the mapping address of each data element in that row of matrix data in the second matrix data buffer unit, and then map it to the corresponding column in the B Matrix data Buffer.

[0158] For example, when the data transmission bandwidth of the AXI4 bus is 128 / 256 / 512 / 1024, the input matrix data is integer data. The matrix data mapping module can store the received matrix data in the B data store, and only delay for one clock cycle within the B data store to buffer the data and isolate the input timing path. Then, the B Matrixdata Mapping module expands each row of B matrix data according to the number of data elements, calculates its transpose mapping address in the B Matrixdata Buffer in parallel, and writes it into the B Matrix data Buffer according to the mapping address. Thus, both the transpose of the B matrix data and the zero-padding mapping of the B matrix data are realized.

[0159] When the task parameters indicate that the row size of the second matrix belongs to the first preset size and the bit width corresponding to the data type of the second matrix is ​​not less than the first preset bit width (i.e., when K of the B matrix is ​​16 and the bit width of the data type is 16bit / 32bit), or when the task parameters indicate that the row size of the second matrix does not belong to the first preset size (i.e., when K of the B matrix is ​​32 and the bit width of the data type is 4bit, or K is 64 and the bit width of the data type is 8bit), the second mapping module can directly expand the matrix data of the valid B matrix output by the B data store module according to the number of data elements, calculate the transpose mapping address in the B Matrix data Buffer in parallel, and then write it to the corresponding column in the B Matrix data Buffer.

[0160] Understandably, after the matrix data mapping module stores the data of a row corresponding to matrix B into the corresponding temporary storage module, if the data type of matrix B is BF16, it can pad each data element in that row with 16 bits of zeros to obtain 32-bit data elements. Then, for each row of matrix B, the second mapping module can expand each 32-bit data element according to the number of data elements, calculate its transpose mapping address in the BMatrix data Buffer in parallel, and then write it to the corresponding column in the BMatrix data Buffer.

[0161] like Figures 5a-5d The diagram shown illustrates the data mapping of a 16×16 B matrix under different data types, as provided in an embodiment of this application. Wherein... Figure 5a In this context, the data type of the B matrix has a bit width of 32 bits, with one square representing a 32-bit data element. During data mapping, the matrix data of each row of the B matrix can be directly transposed and mapped to the corresponding column of the B Matrix data Buffer. Figure 5b In this context, the data type of the B matrix has a bit width of 16 bits, with one square representing a 16-bit data element. During data mapping, the matrix data of each row of the B matrix can be directly mapped to the corresponding column of the BMatrix data buffer. Figure 5c In this context, the data type of the B matrix has a bit width of 8 bits. One square represents the corresponding column of an 8-bit data element. After obtaining the matrix data transmitted via the AXI4 bus, it is necessary to perform zero-padding in the K dimension to obtain the matrix data of the B matrix with a shape of 32×16, and then map it to the corresponding column of the B Matrix data Buffer. Figure 5c The green portion represents the input 8-bit matrix data, the gray portion represents the zero-padded matrix data, the blue portion represents the remaining space in the B Matrix data buffer, and the red row represents a row of mapped matrix data. Figure 5d In this context, the data type of the B matrix has a bit width of 4 bits, with one square representing a 4-bit data element. After obtaining the matrix data transmitted via the AXI4 bus, it is necessary to perform zero-padding in the K dimension to obtain the matrix data of the B matrix with a shape of 64×16, and then map it to the corresponding column of the B Matrix data Buffer. Figure 5d The green portion represents the 4-bit input matrix data, the gray portion represents the zero-padded matrix data, the blue portion represents the remaining space in the B Matrix dataBuffer, and the red row represents a row of mapped matrix data. It should be noted that... Figure 5c and Figure 5d The zero-padding result is for illustrative purposes only. In the actual mapping process, it is implemented directly when mapping to the B Matrix data Buffer.

[0162] When the matrix size of matrix B is 16×8 / 16×32 and the bit width corresponding to the data type is 32bit / 16bit, the mapping process for the case where the bit width corresponding to the data type is 32bit / 16bit is as follows: Figure 5a Similar to / 5b, it will not be repeated here. For the case where the data type corresponds to an 8-bit width, the mapping process is the same as... Figure 5c Similarly, the K dimension needs to be expanded to 32. For the case where the bit width corresponding to the data type is 4 bits, the mapping process is the same as... Figure 5d Similarly, the K dimension needs to be expanded to 64.

[0163] For B matrix sizes of 32×8 / 32×16 / 32×32 with an 8-bit bit width corresponding to the data type, or B matrix sizes of 64×8 / 64×16 / 64×32 with a 4-bit bit width corresponding to the data type, since zero padding is not required during data mapping and transpose mapping can be performed directly, the mapping process is similar to... Figure 5a Similar to / 5b, it will not be elaborated on here.

[0164] In one embodiment, the temporary storage module includes a third temporary storage module corresponding to the third matrix, the mapping module includes a third mapping module corresponding to the third matrix, and the matrix data cache unit includes a third matrix data cache unit corresponding to the third matrix. The third temporary storage module can be connected to the third mapping module and the third matrix data cache unit respectively. The third mapping module is used to, when the task parameters indicate that the bit width corresponding to the data type of the third matrix is ​​a first preset bit width, pad each row of matrix data stored in the third temporary storage module with zeros for each data element to obtain matrix data with a bit width of the first target bit width and map it to the corresponding row in the third matrix data cache unit; or, when the task parameters indicate that the bit width corresponding to the data type of the third matrix is ​​the first target bit width, map each row of matrix data stored in the third temporary storage module to the corresponding row in the third matrix data cache unit.

[0165] Here, the third matrix is ​​a C matrix, the third temporary storage module can be a C data store module, the third mapping module can be a C matrix data mapping module, and the third matrix data buffer unit can be a C matrix data buffer. The first preset bit width can be 16 bits, and the first target bit width can be 32 bits.

[0166] As shown in Table 2 above, in the tensor computation coprocessor design of this application, for matrix data of C matrix, there are only two bit widths: 16-bit and 32-bit. The 16-bit case only corresponds to non-mixed-precision computation under FP16 data type, while the other cases are all 32-bit.

[0167] In practical implementation, using the same data format, the C Data store module, due to the relatively simple mapping relationship and data width of the C matrix, only needs to handle the special case of a 16-bit width C matrix for non-mixed precision calculations. To align the internal computation array and facilitate tensor computation unit scheduling, the C Matrix data Mapping module can, when the task parameters indicate that the data type of the C matrix corresponds to a 16-bit width (i.e., the data type of the C matrix is ​​FP16), pad each row of matrix data stored in the C Data store module with zeros, resulting in 32-bit matrix data, and map it to the corresponding row in the CMatrix data Buffer; alternatively, when the task parameters indicate that the data type of the C matrix corresponds to a 32-bit width, the C Matrix data Mapping module can directly map each row of matrix data stored in the C Data store module to the corresponding row in the CMatrix data Buffer. Specifically, for the C matrix, for each row of C matrix data input, if zero padding is required, it is done first and then mapped; otherwise, the row of matrix data can be mapped directly.

[0168] Understandably, if the data transmission bandwidth of the AXI4 bus is between 32-bit and 1024-bit, the matrix data mapping module can directly store each transmitted matrix data into the C Data store module, with only a one-clock internal delay for data caching, requiring no additional processing and isolating the input timing path. Then, the C Matrix data Mapping module can use the same row-major storage format as matrix A, padding each row of matrix data in the C Data store module with zeros and mapping it to the corresponding row in the C Matrix data Buffer.

[0169] like Figure 6 The diagram shown illustrates a data mapping of a C matrix of a given size under different data types, as provided in an embodiment of this application. Wherein... Figure 6 In this example, the C matrix has a size of 16×16 and a data type of FP16. Each square represents a 16-bit data element. After receiving each row of C matrix data, each 16-bit data element in each row is padded with zeros to a total of 32 bits. Then, each padded matrix data is mapped to the corresponding row in the CMatrix data buffer. For C matrices with a size of 8×32 / 32×8 and a data type of FP16, the mapping and padding processes are similar to... Figure 6 Similarly, this will not be repeated here. It should be noted that... Figure 6 The green part represents the original input matrix data, and the white part represents the zero-padding data corresponding to the data elements.

[0170] like Figure 7 The diagram shown illustrates the data mapping of a 16×16 C matrix with a 32-bit data width, according to an embodiment of this application. Wherein... Figure 7 In this model, each square represents a 32-bit data element. After receiving each row of matrix data from a C matrix, no zero padding is needed; each row of matrix data can be directly mapped to the corresponding row in the C Matrixdata Buffer. Regarding the case where the C matrix size is 8×32 / 32×8 and the data type corresponds to a 32-bit bit width, the mapping process is similar to... Figure 7 Similarly, this will not be repeated here. It should be noted that... Figure 7 The green part represents the original input matrix data, and the red row represents a row of matrix data for matrix mapping.

[0171] Furthermore, after the matrix data of matrices A, B, and C have all been stored and mapped, the AXI parameter configuration and preprocessing remapping module enables the data valid signal and initiates a handshake with the computation scheduling module. When the computation scheduling module enables the data usage complete signal, it indicates that the use of the input data for this period is complete. The AXI parameter configuration and preprocessing remapping module then updates the data output selection based on the dual-buffer buffer mode. When the dual-buffer buffer mode is selected, the current output buffer is switched (e.g., from buffer0 to buffer1, and from buffer1 to buffer0). After the matrix data of matrices A, B, and C have all been stored and mapped, the data valid signal is enabled again.

[0172] When the single-buffer mode is selected, the tensor computation coprocessor is enabled to read external data, waits for the task parameters and matrix data of the next tensor computation task to be written from the AXI4 interface, and then performs a new storage mapping of the matrix data of matrices A, B, and C. After the matrix data of matrices A, B, and C have all been stored and mapped, the data valid signal is enabled again.

[0173] Thus, the multi-precision, multi-size tensor computation coprocessor based on the AXI bus provided in this application, for input data (i.e., matrix data of matrices A, B, and C), completes zero-padding, mapping, and reorganization of the data using a preprocessing remapping module within the tensor computation coprocessor, achieving hardware-level data processing. Compared to the existing technology that uses software to preprocess the input data before feeding it into the tensor computation coprocessor, this effectively improves the computational efficiency of the computing array and reduces the latency of data preprocessing. It also supports tensor computation with multiple data precisions (FP32, TF32, BF16, FP16, INT8, INT4, FP8, FP6, FP4) and mixed precisions, as well as multiple input matrix shapes, such as three input matrix sizes (M16N16K16, M32N8K16, M8N32K16) under FP32, TF32, BF16, FP16, INT8, and INT4 data precisions, and K-dimensional size extensions for low-bit data precision, improving the versatility of the tensor computation coprocessor. Supporting multiple platforms such as SoC / FPGA / MCU, the tensor computing coprocessor with the AXI interface offers enhanced versatility. It allows for flexible configuration of AXI4 data bandwidth, the size of the internal computing array of the tensor computing coprocessor, and the number of matrix data buffers, addressing limitations in data bus bandwidth, computing resources, and storage resources. Optional resource configurations can be tailored to different needs.

[0174] like Figure 8 The diagram shown is an architectural schematic of a preprocessing remapping module provided in an embodiment of this application. The parameter mapping module receives input task parameters (input param) from AXI-Lite bus configuration and maps them to a task parameter buffer (paramdata buffer) so that the computation scheduling module can retrieve task parameters (param_ctrl) from the param data buffer. Matrix data (i.e., data widths of matrices A, B, and C) are transmitted based on the AXI4 bus (data width can be 32, 64, 128, 256, 512, or 1024 bits). Figure 10The A / B / C Matrix data mapping module maps matrix data from the A / B / C data stores to the A, B, and C data stores within the A, B, and C matrix data mapping modules. The matrix type can be distinguished by the user bit or address (Addr) in the A / B / C data stores. The A / B / C Matrix data mapping maps matrix data from the A / B / C data stores to the A / B / C matrix data buffers, allowing the computation scheduling module to retrieve task parameters (A / B / C matrix data) from these buffers. Specifically, during the B matrix data mapping, valid B matrix data (B_valid) is mapped to the B matrix data buffer, achieving zero padding and mapping. After the computing array finishes reading the input data (including task parameters and matrix data) and completing the corresponding tensor calculation task, it can send a "finish" signal to the AXI bus task parameter configuration and preprocessing remapping module, thereby instructing the control to switch to a new output buffer (i.e., a new task parameter buffer unit). Figure 8 There are two `param` buffers: `databuffer` (A), `Matrix data Buffer` (B), and `Matrix data Buffer` (C). A dual-buffer module can be used to map task parameters to matrix data.

[0175] like Figure 9 The diagram illustrates the buffering of matrix data in a temporary storage module for an A / B matrix, as provided in this application. When transmitting A / B matrix data via an AXI4 bus, an A / B data store can be used for data collection, buffering, and valid signal generation. Specifically, during data collection and buffering, if the data width is 32 bits / 64 bits, the matrix data can be stored in a buffer area of ​​line-store size (i.e., ...). Figure 9The Line Data Collection Scratchpad retrieves a full row of matrix data (e.g., 128 bits) from the buffer after the buffer is full, and maps it to Matrix Data Buffer A / B. If the data width is 128 bits / 256 bits / 512 bits / 1024 bits, the matrix data can be delayed by one clock cycle within the A / B data store for one data buffering operation (i.e., datawidth Data Scratchpad), and then the full row of matrix data with the datawidth width can be mapped to Matrix Data Buffer A / B. For valid signal generation, an internal period counter is used. If the data width is 32 bits or 64 bits, a valid signal is output when the counting period reaches the count value ((line buffer size / datawidth) - 1), and data mapping is performed. If the data width is 128bit / 256bit / 512bit / 1024bit, then a valid signal can be output without the need for a periodic counting interval. Instead, a valid signal can be stored in each cycle, and then a valid signal can be output to perform data mapping.

[0176] like Figure 10The diagram illustrates a method for buffering C matrix data in a temporary storage module, as provided in this application. When transmitting C matrix data via an AXI4 bus, a C data store can be used for data collection, buffering, and valid signal generation. Specifically, during data collection and buffering, for any data width, if the C matrix data width (C datawidth) is 16 bits, each 16-bit data element stored in the C Data store module can be padded with zeros to obtain 32-bit matrix data. This achieves temporary storage of matrix data with twice the data width (i.e., 2*datawidth Scratchpad), and then the entire row of matrix data is mapped to the C Matrixdata Buffer. If the C datawidth of the C matrix is ​​another width, such as 32 bits, the matrix data can be delayed by one clock cycle within the C data store for one data buffering operation (i.e., datawidth Scratchpad), and then the entire row of matrix data (datawidth bits) can be mapped to the C Matrix data Buffer. For valid signal generation, for any data width, a valid signal can be stored directly every cycle, followed by the output of a valid signal from the data buffer, thereby performing data mapping.

[0177] In one embodiment, the scheduling computation module 11 includes schedulers and computation arrays corresponding to different data types.

[0178] The scheduling and computation module 11 is used to extract task parameters from the input buffer unit for the same tensor computation task, and select the target scheduler and target computation array from the schedulers and computation arrays corresponding to different data types according to the data type of the matrix to be processed indicated by the task parameters. It also determines the hardwired connection between the input buffer unit and the target computation array according to the matrix size of the matrix to be processed indicated by the task parameters. The target scheduler is used to schedule the task parameters and matrix data in the input buffer unit to the target computation array for tensor computation according to the hardwired connection, and obtain the computation result and block index corresponding to each block scheduling.

[0179] To achieve efficient support for computations of various precisions, this application designs a scheduler adapted to the characteristics of multiple precisions. This ensures that data under different precision modes (i.e., different data types) can be processed correctly and efficiently in the computation array. For FP16, INT8, FP8, FP6, INT4, and FP4 dot product units, as the data precision decreases, the computation scale is correspondingly expanded in the K-dimensional dimension (as described above, zero-padding is performed in the K-dimensional dimension) to improve parallelism and computational throughput under low precision. This expansion strategy ensures that instructions of various precisions can use a unified scheduling method, reducing hardware control complexity.

[0180] Specifically, FP8, FP6, and FP4 data types have lower data precision, which can correspond to a low-precision computing array design. Furthermore, the computing array size for FP8 and FP6 data types can be consistent with that for INT8 data types, and the computing array size for INT8 data types can be consistent with that for FP16 data types, to achieve unified instruction scheduling and simplify hardware design. Since FP6 matrix data is padded to 8 bits when stored in external memory, the INT8 computing array can be directly reused. The computing array size for FP4 data types can directly reuse the INT4 computing array size, and the INT4 computing array size can reuse the INT8 computing array size. Therefore, the computing array size for FP6 and FP4 data types can remain consistent with that for FP16 data types. Thus, this application provides a mixed-precision scheduler and a low-precision computing array corresponding to the FP16, INT8, INT4, FP8, FP6, and FP4 data types, which can support multi-way tensor computation of various data types such as FP16, INT8, INT4, FP8, FP6, and FP4 through resource reuse and other means.

[0181] When calculating A*B+C, both computational arrays require data in row-major order for A, column-major order for B, and row-major order for C. The specific tensor computation process is implemented by a tensor computation scheduling module that transmits the data to be computed (A, B, and C) to the computational arrays via a state machine, resulting in the final tensor computation result matrix D, which is obtained periodically. The size design of the computational arrays for FP32, TF32, and BF16 data types is independent of the low-precision (i.e., the precision corresponding to FP16, INT8, INT4, FP8, FP6, and FP4 data types) computational arrays, and their corresponding high-precision schedulers are also controlled by independent modules. This is because FP32, TF32, and BF16 data type computations have higher requirements for precision and numerical range, and the scheduling strategy differs significantly from that of low-precision computations. Therefore, this application provides a high-precision scheduler and a high-precision computational array corresponding to FP32, TF32, and BF16 data types, supporting tensor multiplication and addition computations of FP32, TF32, and BF16 types.

[0182] To efficiently schedule matrix data, the Tile Scheduler module employs a hard-wired scheme. Different hard-wired schemes can be used between the input buffer unit and the computation array when the matrix size and data type of the matrix to be processed differ. Internally, the scheduler maintains a state machine. The number of states is determined by the ratio of the computation instruction size (i.e., the product of M, N, and K in the task parameters) to the computation array size (i.e., the product of m, n, and k configured via Chisel). Each time the state machine enters a new state, it triggers an operation to transfer matrix data from the input buffer unit to the computation array, completing the loading of the corresponding data block within the instruction cycle. The maximum number of states in the state machine depends on the result of dividing the maximum computation instruction size by the computation array size, thus ensuring that the scheduling requirements of the largest-sized computation tasks can be supported.

[0183] For example, the tensor computation coprocessor of this application may include two schedulers and two computation arrays. The two schedulers are a mixed-precision scheduler corresponding to FP16, INT8, INT4, FP8, FP6, and FP4 data types, and a high-precision scheduler corresponding to FP32, TF32, and BF16 data types. The two computation arrays are a low-precision computation array corresponding to the mixed scheduler and a high-precision computation array corresponding to the high-precision scheduler. Typically, the sizes of the two computation arrays can be configured separately using Chisel. The sizes of the two computation arrays can be the same or different. The size of the computation array can be m×n×k, where k is usually K in the task parameters.

[0184] In specific implementation, the scheduling and computation module 11 can extract task parameters from the param databuffer for the same tensor computation task, and select the target scheduler and target computation array from the schedulers and computation arrays corresponding to different data types according to the data type of the matrix to be processed indicated by the task parameters. For example, when the data type of the matrix to be processed is FP32, a high-precision scheduler and a high-precision computation array are selected; when the data type of the matrix to be processed is INT8, a mixed-precision scheduler and a low-precision computation array are selected. Then, according to the matrix size of each matrix to be processed indicated by the task parameters, the matrix shape (i.e., M×N×K) of the three matrices to be processed can be determined, and the hardwiring between the input buffer unit and the target computation array can be determined. Here, the hardwiring schemes between various matrix shapes and various computation arrays can be pre-generated. After determining the matrix shape of the matrix to be processed in the current tensor computation task, a corresponding hardwiring scheme can be selected as the hardwiring currently needed between the input buffer unit and the target computation array. Understandably, this application does not specifically limit the values ​​of M, N, and K in the matrix shape. Generally, the values ​​of M, N, and K can be 8, 16, or 32, but other values ​​should also fall within the protection scope of this application.

[0185] The target scheduler is used to schedule the task parameters and matrix data in the input buffer unit to the target computing array for tensor computation according to hardwired routing, and obtain the computation result and block index corresponding to each block scheduling.

[0186] For example, the target scheduler can acquire the block data of matrices A, B, and C for matrix multiplication operations according to different computation cycles and input them into the corresponding computation array for tensor computation, obtaining the computation result and block index corresponding to each block scheduling. The task parameters can be scheduled to the target computation array before scheduling the block data of matrices A, B, and C, and the calculation result is rounded according to the rounding method indicated by the task parameters during matrix multiplication and addition.

[0187] To verify and demonstrate the scheduler's operation under different data precisions and matrix sizes, specific scheduling schemes were designed for various typical matrix shapes (M×N×K) in FP32, BF16, TF32 modes, and low-precision modes (such as FP16, INT8, FP8, FP6, INT4, FP4). For example, specific scheduling schemes were designed for matrix sizes M16N16K16 / M32N8K16 / M8N32K16, and K-extended matrix sizes with low bit widths (such as M16N16K32 / M32N8K32 / M8N32K32, M16N16K64 / M32N8K64 / M8N32K64).

[0188] In FP32, BF16, and TF32 modes: The high-precision scheduler loads the data required by the computation array in each state, based on the actual input data size of the high-precision computing array. The state machine scheduling ensures that the order of the input data blocks in the eight states is consistent with the order of the computing array.

[0189] Low-precision modes FP16, INT8, INT4, FP8, FP6, FP4: Similar to FP32 mode, but the low-precision scheduler simultaneously loads larger data blocks to match the scale of the low-precision computing array after the low-precision expansion, thus achieving high parallelism computing.

[0190] By processing matrices with different matrix shapes and data precision combinations, the scheduler's consistency, efficiency, and high adaptability across various scenarios were comprehensively verified. This design ensures that matrix data is continuously and optimally supplied to their respective computing arrays in scenarios with multiple precisions and matrix shapes, effectively improving the overall computing power utilization of the tensor computation coprocessor.

[0191] like Figure 11 The diagram shown is a schematic representation of a scheduling process provided in an embodiment of this application. Figure 11 The corresponding matrix shape is M16N16K16, meaning the matrix size of matrices A / B / C is 16×16, the data type is FP16, and the size of the computation array is m4n4k4, containing 4 rows and 4 columns of dot product units (DP). During the block scheduling process according to the scheduling order, for each block scheduling, the block data size corresponding to matrix A is 4×16, the block data size corresponding to matrix B is 16×4, and the block data size corresponding to matrix C is 4×4.

[0192] like Figure 12 The diagram shown is a schematic of a scheduling process under a matrix shape of M16N16K16 and FP32 / TF32 / BF16 provided in an embodiment of this application. The number of state machines is 8, meaning that 8 block scheduling operations will be performed. Figure 12 The diagram shows sets 1 through 8, illustrating the scheduling process in eight chunks. For each set, each green data block in matrix A represents the scheduled chunk data, specifically an 8x16 matrix in matrix A; each green data block in matrix B represents the scheduled chunk data, specifically a 16x4 matrix after the transpose of matrix B; and each green data block in matrix C represents the scheduled chunk data, specifically an 8x4 matrix in matrix C. Each row in matrix A (rows 0 through 7) represents a 2x16 matrix in matrix A; each column in matrix B (rows 0 through 7) represents a 16x2 matrix after the transpose of matrix B; and each data block in matrix C (blocks 0 through 7) represents an 8x4 matrix in matrix C.

[0193] like Figure 13 The diagram shown is a schematic of the scheduling process under a matrix shape of M16N16K16, FP16 / INT8 / FP8 / FP6 / INT4 / FP4, provided in an embodiment of this application. The number of state machines is 8, meaning that 8 block scheduling operations will be performed. Figure 13 The diagram shows sets 1 through 8, illustrating the scheduling process in eight chunks. For each set, each green data block in matrix A represents the scheduled chunk data, specifically an 8x16 matrix in matrix A; each green data block in matrix B represents the scheduled chunk data, specifically a 16x4 matrix after the transpose of matrix B; and each green data block in matrix C represents the scheduled chunk data, specifically an 8x4 matrix in matrix C. Each row of data in matrix A (rows 0-3) represents the 4x16 data in matrix A; each column of data in matrix B (rows 0-3) represents the 16x4 data in the transpose of matrix B; and each data block in matrix C (blocks 0-7) represents the 8x4 data in matrix C.

[0194] like Figure 14 The diagram shown is a schematic of a scheduling process under a matrix shape of M32N8K16, FP32 / TF32 / BF16 provided in an embodiment of this application. The number of state machines is 8, meaning that 8 block scheduling operations will be performed. Figure 14 The diagram shows sets 1 through 8, illustrating the scheduling process in eight blocks. For each set, each green data block in matrix A represents the scheduled block data, specifically an 8x16 matrix in matrix A; each green data block in matrix B represents the scheduled block data, specifically a 16x4 matrix after the transpose of matrix B; and each green data block in matrix C represents the scheduled block data, specifically an 8x4 matrix in matrix C. Each row of data in matrix A (rows 0 through 16) represents a 2x16 matrix in matrix A; each column of data in matrix B (columns 0 through 3) represents a 16x2 matrix after the transpose of matrix B; and each data block in matrix C (blocks 0 through 7) represents an 8x4 matrix in matrix C.

[0195] like Figure 15 The diagram shown is a schematic of the scheduling process under a matrix shape of M32N8K16, FP16 / INT8 / FP8 / FP6 / INT4 / FP4, provided in an embodiment of this application. The number of state machines is 8, meaning that 8 block scheduling operations will be performed. Figure 15The diagram shows sets 1 through 8, illustrating the scheduling process in eight blocks. For each set, each green data block in matrix A represents the scheduled block data, specifically an 8x16 matrix in matrix A; each green data block in matrix B represents the scheduled block data, specifically a 16x4 matrix after the transpose of matrix B; and each green data block in matrix C represents the scheduled block data, specifically an 8x4 matrix in matrix C. Each row of data in matrix A (rows 0 through 7) represents the 4x16 data in matrix A; each column of data in matrix B (columns 0 through 1) represents the 16x8 data in the transpose of matrix B; and each data block in matrix C (blocks 0 through 7) represents the 8x4 data in matrix C.

[0196] like Figure 16 The diagram shown is a schematic of a scheduling process under a matrix shape of M8N32K16, FP32 / TF32 / BF16 provided in an embodiment of this application. The number of state machines is 8, meaning that 8 block scheduling operations will be performed. Figure 16 The diagram shows sets 1 through 8, illustrating the scheduling process in eight blocks. For each set, each green data block in matrix A represents the scheduled block data, specifically an 8x16 matrix in matrix A; each green data block in matrix B represents the scheduled block data, specifically a 16x4 matrix after the transpose of matrix B; and each green data block in matrix C represents the scheduled block data, specifically an 8x4 matrix in matrix C. Each row of data in matrix A (rows 0-4) represents a 2x16 matrix in matrix A; each column of data in matrix B (columns 0-15) represents a 16x2 matrix after the transpose of matrix B; and each data block in matrix C (blocks 0-7) represents an 8x4 matrix in matrix C.

[0197] like Figure 17 The diagram shown is a schematic of the scheduling process under a matrix shape of M8N32K16, FP16 / INT8 / FP8 / FP6 / INT4 / FP4, provided in an embodiment of this application. The number of state machines is 8, meaning that 8 block scheduling operations will be performed. Figure 17The diagram shows sets 1 through 8, illustrating the scheduling process in eight chunks. For each set, each green data block in matrix A represents the scheduled chunk data, specifically an 8x16 matrix in matrix A; each green data block in matrix B represents the scheduled chunk data, specifically a 16x4 matrix after the transpose of matrix B; and each green data block in matrix C represents the scheduled chunk data, specifically an 8x4 matrix in matrix C. Each row in matrix A (rows 0-1) represents the 4x16 data in matrix A; each column in matrix B (rows 0-7) represents the 16x4 data in the transpose of matrix B; and each data block in matrix C (rows 0-7) represents the 8x4 data in matrix C.

[0198] In one embodiment, the scheduler also maintains a state machine; the state machine has multiple states for triggering different sub-matrix data scheduling; the number of states is related to the matrix size and the size of the target computing array.

[0199] Here, each scheduler can maintain a state machine, which can have multiple states. Each state triggers the scheduling of a data block from A Matrix data Buffer, C Matrix data Buffer, and C Matrix data Buffer. The number of states in the state machine can be M×N×K / m×n×k. The state machine can communicate with the scheduler, and the scheduler can communicate with the computing array.

[0200] The target scheduler is used to schedule task parameters to the target computing array, and determine the block data size and block scheduling order of each matrix to be processed in each state of the state machine according to the matrix size of the matrix to be processed indicated by the task parameters and the scale of the target computing array. When the state machine enters a state, according to the hardwired and block scheduling order, the block data with the block data size in the input buffer unit is scheduled to the target computing array for tensor calculation, and the calculation result and block index corresponding to each block scheduling are obtained.

[0201] Here, the block data size is used to indicate the size of the matrix data blocks separately scheduled from matrices A, B, and C each time block scheduling is performed. The block data size can be determined according to the number of states of the state machine and the matrix sizes of each matrix to be processed. The block scheduling order is used to indicate the scheduling order of the block data in each matrix when scheduling the block data separately from matrices A, B, and C. For example, the number of scheduling times = (M / m)*(N / n). In terms of the scheduling granularity, when m > n, the block data of the (m, k) shape tries to stay resident, and the block data of the (n, k) shape is preferentially moved. When m < n, the block data of the (n, k) shape tries to stay resident, and the block data of the (m, k) shape is preferentially moved. When m = n, for the block data of the (n, k) shape and the block data of the (m, k) shape, either one can be used to try to stay resident, and the other can be preferentially moved. The specific scheduling process can be organized according to the above block scheduling order. After scheduling all (M / m)*(N / n) times in sequence, a tensor calculation task is completed.

[0202] In specific implementation, the target scheduler can schedule the task parameters to the target computing array, and determine the block data size corresponding to each matrix to be processed in each state of the state machine and the block scheduling order of each matrix to be processed according to the matrix size (M×N×K) corresponding to the matrix size of the matrix to be processed indicated by the task parameters and the scale (m×n×k) of the target computing array. Then, in the case where the state machine enters each state, according to the hardwiring between the input buffer unit and the target computing array and the block scheduling order, the block data with the corresponding block data size in each input buffer unit is scheduled to the target computing array for tensor calculation, and the calculation result and block index corresponding to each block scheduling are obtained.

[0203] Moreover, for the design of the state machine in this application, to adapt to different matrix shapes (M16N16K16, M8N32K16, M32N8K16, and the matrix shapes after K expansion under low bit widths (such as M16N16K32 / 64, M8N32K32 / 64, M32N8K32 / 64)) and multiple data precisions, this application designs a unified state machine architecture that supports multiple precisions and multiple shapes and has the ability to automatically expand the k dimension. Among them, the design of the state machine needs to meet the following requirements: <了

[0204] Support for multiple matrix shapes: The state machine should be able to perform efficient scheduling in three typical shapes (M16N16K16, M8N32K16, M32N8K16) and the matrix shapes after K expansion under low bit widths; for the matrix shapes after K expansion under low bit widths, among which, the scheduling of K after expansion under low bit precision is the same as the scheduling of K16 under FP16).

[0205] It should be noted that there seems to be a typo in the translation of where "了0000502" is likely incorrect in the original text. Please check and correct it if needed.Multiple data precision support: The state machine needs to support five main data precisions, including FP32 precision, FP16 precision, FP16 mixed precision, INT8 precision, and INT4 precision; it should also support FP8 precision, FP6 precision, and FP4 precision modes, and maintain the same scheduling scale as the INT8 precision mode to maximize the utilization of hardware resources.

[0206] Unified number of states: To reduce hardware design complexity and improve predictability, the state machine in FP32, FP16, FP16 mixed precision, INT8, and INT4 precision modes all uses 8 states. Each state represents a scheduling process of loading matrix data from the input buffer unit to the dot-product (DP) unit in the computation array. A complete cycle of the state machine completes the loading of all matrix data specified for the current tensor computation task.

[0207] In one embodiment, the computation array includes multiple dot product units (DP units) matching the size of the computation array; the size of the input block data supported by the dot product units is dynamically adjusted according to the data type of each matrix to be processed. Here, the size of the computation array is m×n×k, and the number of DP units in the computation array can be m×n. That is, a computation array of size m×n×k includes m rows and n columns of DP units. The DP units can have variable K-dimensional parallel expansion capability according to the data precision. For example, for FP32, FP16, and FP16 mixed precision modes, the DP units support matrix block computation of the form [1, 16]×[16, 1]. For INT8 mode, the DP units are extended to support computation blocks of [1, 32]×[32, 1] to improve throughput at low precision. For INT4 mode, the DP units are further extended to computation blocks of [1, 64]×[64, 1] to achieve ultra-high parallelism operation. Furthermore, in the FP8, FP6, and FP4 data precision modes, the DP unit shape and size remain consistent with the INT8 mode. By padding or aligning the FP6 and FP4 precision with zeros, compatibility with the INT8 scheduling strategy is ensured, avoiding additional state machine complexity. It should be noted that this application does not specifically limit the dot product unit, as long as it supports reuse calculations for various data types.

[0208] The state machine implementation also adopts a hardwired design and uses hardware-configurable parameters (generated using Chisel) to flexibly support different matrix shapes and data precision combinations without redesigning the hardware circuit, effectively balancing computing unit utilization, scheduling consistency and scalability.

[0209] The target scheduler can be used to schedule block data with block size in the input buffer unit to different dot product units in the target computing array for tensor calculation according to hardwired and block scheduling order when the state machine enters a state, so as to obtain the calculation result and block index corresponding to each block scheduling.

[0210] For example, when the state machine enters a state, the target scheduler can, according to hardwired and block scheduling order, schedule the block data with corresponding block data sizes from the input buffer units corresponding to matrices A, B, and C to different dot product units in the target computing array for tensor computation. For instance, the block data of matrix A can be broadcast row-wise to the DP units of each row of the target computing array, the block data of matrix B can be broadcast column-wise to the DP units of each column of the target computing array, and the block data of matrix C can be broadcast row-wise to the DP units of each row of the target computing array. Suppose that during a certain scheduling operation, a 4-row data block is retrieved from matrix A, a 2-column data block is retrieved from matrix B, and a 4-row data block is retrieved from matrix C. Then, the first row of the data block from matrix A is broadcast to each DP in the first row of the target computing array, the second row to each DP in the second row, and so on, up to the fourth row. Similarly, the first column of the data block from matrix B is broadcast to each DP in the first column, the second column to each DP in the second column, and so on, up to the fourth row. After broadcasting, each DP can use the received matrix data to perform matrix multiplication and addition operations to obtain the computation result and block index corresponding to the current scheduling operation.

[0211] In one embodiment, the tensor computation coprocessor is further configured to transmit the computation results in the output cache unit to the external memory based on a preset bus and the storage address of the external memory, and according to a preset data format and storage address.

[0212] Here, the preset data format can be row-major format, and the storage address can be the storage address of the external memory indicated by the task parameters (D_Address), which is used to store the final calculation result of the tensor calculation task.

[0213] In practice, after writing the calculation results corresponding to each block scheduling into the output buffer unit, the calculation results form a result matrix (i.e., D matrix). The tensor calculation coprocessor can also use the AXI4 bus to transfer each row of matrix data in the result matrix to the storage address of the external memory for storage in the row-matrix format.

[0214] In one embodiment, the output mapping module 12 may include a fourth temporary storage module and a fourth mapping module. The fourth temporary storage module may be connected to the fourth mapping module. The output mapping module is used to store the calculation results corresponding to each block scheduling in the fourth temporary storage module; the fourth mapping module is used to determine the write address of the calculation results corresponding to each block scheduling in the fourth temporary storage module according to the block index of the block data and the preset data format, and map the calculation results in the fourth temporary storage module to the write address in the output cache unit.

[0215] Here, the fourth temporary storage module can be a D data store module, and the fourth mapping module can be a D matrix data mapping module. The output buffer unit can be a D matrix data buffer.

[0216] The output interface of the computing array includes the array's computation results and the block index of the computation results. Since the internal computing array of this application can be flexibly configured in size, for matrices to be processed, such as M32N8K16, M16N16K16, and M8N32K16, the input computing array size is used to divide the data into blocks, and the results of the block matrices are calculated sequentially. In the AXI master part, the computation results of the output block matrices need to be stored in the output D Martix Buffer, and finally written out to external storage in row-major order. The array size and AXI master data output bandwidth are determined by the Chisel parameter when the tensor computation coprocessor is generated; this application can flexibly adjust the Chisel parameter according to the limitations of the computing platform. Once the computing array size and AXI transmission bandwidth are determined, the logical mapping relationship of the content and the output logic are determined.

[0217] In specific implementation, after the scheduling calculation module 11 completes the calculation result and block index for each block scheduling operation, the output mapping module can store the calculation result corresponding to that block scheduling operation in the D data store. Then, the fourth mapping module can retrieve the calculation result of that block scheduling operation from the D data store, determine the write address of the calculation result in the D Matrix data Buffer module based on the block index and row-major format of the calculation result, and write the calculation result to the D Matrix data Buffer module according to the write address. At this time, the storage format inside the D Matrix data Buffer is row-major format.

[0218] For example, if the target computing array is m×n, the output matrix D, which is of shape M×N, is divided into M / m rows and N / n columns of sub-block matrices. The target computing array can output the computation result of each block scheduling, along with the corresponding n_index and m_index indices. Here, n_index and m_index indicate the column and row indices of the block scheduling, respectively, and indicate the write address of the computation result written to the D Matrix data buffer. When n_index = N / n-1 and m_index = M / m-1, it means that the target computing array has completed all A*B+C block scheduling operations and obtained the computation results of each block scheduling operation. The tensor computation coprocessor can then write the computed D matrix data to the external storage (such as DDR, SD card, etc.) corresponding to D_Addr via the AXI4 bus, completing a complete tensor computation.

[0219] Thus, this application achieves the rearrangement and remapping of matrix A, B, and C data through a specially designed hardware module within the tensor computation coprocessor, eliminating the need for pre-processing in a specific software format. Furthermore, the transpose of matrix B is performed internally within the tensor computation coprocessor, and the output matrix is ​​restored to row-major format, ensuring the accuracy of tensor computation. Moreover, by utilizing the AXI dynamic effective bit width to transmit low-bit numerical precision non-K extended data, and cooperating with hardware to perform zero-padding within the hardware module, data processing efficiency is improved, avoiding the inefficiency and long processing time associated with software zero-padding.

[0220] like Figure 18 The diagram illustrates an output mapping process for computation results provided in this embodiment. Here, `mx_array` is the low-precision computation array mentioned above, and `fix_result` represents the computation result of each block scheduling operation output by the low-precision computation array; `fp_32array` is the high-precision computation array mentioned above, and `fp_32result` represents the computation result of each block scheduling operation output by the high-precision computation array. `mx_array` and `fp_32array` can be selected for different tensor computation tasks. The output mapping module can store `fix_result`, `fp_32result`, and task parameters (`param_ctrl`) in the D data store. Simultaneously, when the computation array outputs the computation result of each block scheduling operation, it also outputs the block index corresponding to the computation result (i.e.,...). Figure 20The data contains row indices (data m_index) and column indices (data n_index), which are used by the D Matrix data Mapping module when mapping computation results. The D Matrix data Mapping module can map the computation results of each block scheduling to the D Matrix data buffer based on the block index. Figure 18 The diagram shows two D Matrix data buffers, indicating that the D Matrix data buffer can also use the dual-buffer mode described above. The tensor computation coprocessor can then transmit the computation results from the D Matrix data buffer to the AXI master via a burst send on the AXI4 bus. The data bit width supports 32bit / 64bit / 128bit / 256bit / 512bit / 1024bit when transmitting computation results.

[0221] like Figure 19 The diagram shown illustrates another output mapping process for calculation results provided in an embodiment of this application. Figure 19 Taking a matrix of shape M16N16K16 and a computation array size of m4n4k4 as an example, the process of writing the computation results output by the computation array into the D Matrix data Buffer module is explained. During block scheduling, the matrix data of the C matrix is ​​divided into 16 blocks, and the values ​​of m_index and n_index are both in the range of [0, 4]. During output mapping, each computation result is mapped according to the block index, thereby mapping 16 data blocks (i.e., (0, 0), (0, 1), (0, 2), (0, 3), (1, 0), (1, 1), (1, 2), (1, 3), (2, 0), (2, 1), (2, 2), (2, 3), (3, 0), (3, 1), (3, 2), and (3, 3)) in the D Matrix data Buffer module. Each data block includes 4 rows and 4 columns of data. The tensor computation coprocessor can transfer each row of matrix data in the D Matrix dataBuffer module to external memory via the AXI4 bus through the AWVALID, AWREADY, AWADDR, AWLEN, AWSIZE and AWBURST signals under the write address channel, the WVALID, WREADY, WDATA, WSTRB and WLAST signals under the write data channel, and the BVALID, BREADY and BRESP signals under the write response channel.

[0222] In one embodiment, the output mapping module is further configured to, after storing the calculation result corresponding to each block scheduling to the fourth temporary storage module, restore the bit width of the calculation result to the first preset bit width if the bit width corresponding to the data type of the third matrix in the matrix to be processed is the first preset bit width.

[0223] Here, the computation result output from the computation array first passes through the D Data store module. Because zero-padding was performed during the input data preprocessing within the tensor computation coprocessor during FP16 non-mixed-precision computation, extending the FP16 type C matrix data from 16 bits to 32 bits, the output D matrix data is also 32 bits. Therefore, the D Data store module needs to remove the extra 16 bits of zeros to restore it to its original 16 bits. In other cases, the C matrix data is originally 32 bits, so no additional processing is needed. The write address corresponding to the output D matrix data can be directly calculated and written to the D Matrix dataBuffer module. In this case, the storage format inside the D Matrix dataBuffer is row-major.

[0224] For example, the output mapping module can be used to remove the extra 16 bits added during the mapping process after storing the computation results corresponding to each block scheduling to the DData store. Since the data width of the computation results corresponding to each block scheduling is 32 bits, if the data width corresponding to the data type of the C matrix is ​​determined to be 16 bits (i.e., the data type of the C matrix is ​​FP16), then the data is written to the D Matrix data Buffer module. If the data width corresponding to the data type of the C matrix is ​​determined to be 32 bits (i.e., the data type of the C matrix is ​​FP32), then the data can be written directly to the D Matrix data Buffer module.

[0225] Thus, this application first configures the task parameters for the tensor computation task, enabling the data type, matrix MNK size, rounding method, mixed precision calculation, and result address required for a single tensor computation to be written into the corresponding registers via the AXI-Lite bus. Then, complete burst transfers of matrix data A, B, and C are performed via the AXI4 bus, transferring the matrix data corresponding to the MNK size from external storage to the tensor computation coprocessor. Finally, after the tensor computation coprocessor completes a single tensor computation task, the computation result matrix D is written into the corresponding address space of external storage via a burst transfer using the AXI4 bus, achieving accurate execution of the tensor computation task.

[0226] like Figure 20 The diagram shows a schematic of a D data storage module provided in an embodiment of this application. After the computation array outputs the calculation results, the output mapping module can collect and cache the calculation results in the D data store. Specifically, after temporarily storing the calculation results in the D data store, if it is an FP16 non-mixed precision calculation (!mix_precision) and the data type corresponding to the C matrix is ​​FP16 (datawidth = 16bit), since the 16-bit input data of the C matrix is ​​padded to 32 bits within the tensor computation coprocessor, it is necessary to restore the 32-bit calculation results to 16 bits and perform a data cache (i.e., Scratchpad) before mapping the calculation results to the D Matrix dataBuffer.

[0227] like Figure 21 The diagram shows the overall architecture of a tensor computation coprocessor provided in this embodiment. Task parameters and matrix data of each matrix to be processed are transmitted to the preprocessing and mapping module via the AXI-Lite and AXI4 buses, and the task parameters and matrix data are mapped to different input buffer units. Then, the scheduling computation module uses a crossbar to select a target scheduler and target computation array from the multi-precision tile scheduler and low-precision computation array (FP16 / INT8 / INT4 / FP8 / FP6 / FP4 Array) corresponding to FP16 / INT8 / INT4 / FP8 / FP6 / FP4, and the high-precision tile scheduler and high-precision computation array (FP32 / TF32 / BF16 Array) corresponding to FP32 / TF32 / BF16. The target scheduler is used to schedule the task parameters and matrix data to the target computation array to obtain the computation results. The calculation results are mapped to the output buffer unit using the output mapping module. Finally, the result matrix in the output buffer unit is output to external memory via the AXI4 bus.

[0228] In one embodiment, the preprocessing remapping module 10 is used to receive the initial matrix data of each matrix to be processed transmitted based on a preset bus, and when the bit width corresponding to the data type of the matrix to be processed is less than the first preset bit width, it selects the effective data from the initial matrix data as the matrix data of the matrix to be processed according to the effective data bit width signal transmitted based on the preset bus.

[0229] Here, the initial matrix data is one row of matrix data transmitted by the AXI bus in one transmission cycle. To avoid software zero-padding for low-bit data precision (i.e., 4-bit / 8-bit) and K=16 non-extended data, while also reusing the data processing and computation array within the tensor computation coprocessor, this application also designs a unique dynamic AXI data bit width. For example, although the AXI data bit width is 1024 bits, the AWSIZE signal can be used to indicate the truly effective data bit width.

[0230] Based on this AXI bus mechanism, when performing A / B matrix data transmission of less than 16 bits using M32N8K16, M16N16K16, and M8N32K16, the number of data elements transmitted in a single data transmission can be aligned to the FP16 data type:

[0231] For example, when the matrix data of an A / B matrix is ​​8-bit wide (e.g., INT8, FP8), the bandwidth of one row of matrix data transmitted in one cycle is half that of the full effective transmission bandwidth of FP16 type matrix data. When the matrix data of an A / B matrix is ​​4-bit wide (e.g., INT4, FP4), the bandwidth of one row of matrix data transmitted in one cycle is one-quarter that of the full effective transmission bandwidth of FP16 type matrix data. Therefore, within the tensor computation coprocessor, only the effective low-bit width data can be processed based on the effective data bit width signal, thereby aligning the FP16 transmission data size.

[0232] For example, after receiving the initial matrix data of a row of A / B matrix transmitted via the AXI4 bus, the preprocessing remapping module 10, if the bit width corresponding to the data type of the A / B matrix is ​​less than 16 bits when K of the A / B matrix is ​​16, can select the low-width data matching the AWSIZE signal as valid data for each data element in the initial matrix data according to the AWSIZE signal transmitted via the AXI4 bus. Based on the valid data of each data element, a row of valid matrix data of the A / B matrix is ​​obtained.

[0233] This setup is designed to reuse the preprocessing and mapping logic within the preprocessing remapping module and align data processing for FP16 data types, simplifying the complexity of hardware zero-padding. With this low data bit width, zero padding is first performed internally in hardware for K-expansion, and then the data is passed to a unified input buffer unit for processing according to the input data requirements of the computing array.

[0234] When transmitting A / B matrix data and all C matrix data under other input matrix shapes (such as FP32, TF32, BF16, FP16, etc. under M32N8K16, M16N16K16, M8N32K16, etc. with at least 16 bits of data; INT8, FP8, etc. data transmission under M32N8K32, M16N16K32, M8N32K32, etc. with 8 bits of data transmission; INT4, FP4, etc. data transmission under M32N8K64, M16N16K64, M8N32K64 with 4 bits of data transmission), no special processing for AXI4 data transmission is required.

[0235] In one embodiment, the tensor computing coprocessor is deployed on any computing platform that supports a preset bus, and the data transmission bandwidth, the number of input cache units, the size of the computing array, and the number of output cache units of the preset bus are all configured in advance using preset hardware parameters based on the resource information of the computing platform.

[0236] Here, the computing platform can include any platform that supports the AXI bus, such as a CPU, MCU, FPGA, or ASIC. The platform resources of the computing platform can include computing resources, on-chip memory, memory access bandwidth, and other resources.

[0237] For example, for any computing platform that requires a tensor computation coprocessor, the target data transfer bandwidth supported by the computing platform, the target number of input cache units (such as the number of cache units for matrix A, B, and C data and the number of cache units for computation parameters), the target number of output cache units, and the mnk of different computation arrays can be determined based on the platform's resource information. Then, the Chisel parameters are used for software configurability definition. After configuration, a tensor computation coprocessor matching the configured parameters can be generated for the computing platform to use.

[0238] In this way, by parameterizing the internal design of the tensor computing coprocessor, the characteristics of the tensor computing coprocessor can be generated according to different needs based on on-chip cache resources, computing resources, bus bandwidth, etc. using Chisel. The parameters can be flexibly set, and the remapping processing of the input matrix data can be completed in hardware while ensuring the accuracy of tensor computing, thereby improving the adaptability and computing efficiency of the tensor computing coprocessor.

[0239] Based on the above embodiments, compared with the prior art, this application provides a multi-precision, multi-size tensor computation coprocessor based on the AXI bus, which addresses the shortcomings of existing technologies such as low data rearrangement efficiency in tensor computation, single support for general data interfaces, and fixed bandwidth and computing arrays. It has the following significant advantages and technical effects: 1. It supports hardware data rearrangement, eliminating the need for additional zero-padding in software. Zero-padding of matrix data with non-compliant matrix sizes is performed directly within the hardware, along with data rearrangement and remapping for matrix computation. 2. It supports matrix data transmission and task parameter configuration via the AXI bus. It supports the input of data to be computed and configured task parameters to the tensor computation coprocessor through the AXI bus interface, as well as the writing of matrix computation results to external storage units. It is not oriented towards a single computing platform; leveraging the versatility of the AXI bus, the tensor computation coprocessor can be integrated into heterogeneous platforms, resulting in a more universal interface and compatibility with a wider range of computing platforms. 3. Supports tensor calculations with various numerical precisions and matrix shapes, such as FP32, FP16, INT8, FP8, FP6, FP4, and INT4, with input matrix sizes of M16N16K16, M32N8K16, and M8N32K16 for integer and floating-point precision. It also supports K-dimensional extended calculations with low-bit numerical precision, such as INT8, FP8, FP6, FP4, and INT4. 4. Supports various tensor calculation coprocessor parameter configurations. The AXI bus bandwidth supports 32 / 64 / 128 / 256 / 512 / 1024-bit input of A, B, and C matrix data, offering greater flexibility in supporting different storage types and interface bit widths. It supports selecting whether to use double-buffered data based on platform resource limitations and allows selection of the internal computing array size based on the computing platform. In summary, the tensor computing coprocessor of this application supports hardware data preprocessing remapping, AXI universal interface, flexible parameter configuration of the tensor computing coprocessor, and support for multiple numerical precisions and matrix calculation sizes. It not only improves the overall computing efficiency of the tensor computing coprocessor, but also achieves the comprehensive advantages of high flexibility and high adaptability, and has broad engineering practical value and industrialization prospects.

[0240] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0241] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0242] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0243] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A tensor computing coprocessor, comprising: The tensor computation coprocessor includes a preprocessing remapping module, a scheduling computation module, and an output mapping module; The scheduling calculation module is connected to the preprocessing remapping module and the output mapping module, respectively; wherein: The preprocessing remapping module is used to receive the task parameters of the tensor calculation task transmitted based on a preset bus and the matrix data of each matrix to be processed corresponding to the tensor calculation task, and to map the task parameters and matrix data to different input buffer units. The scheduling and calculation module is used to schedule the task parameters from different input cache units to the target computing array that matches the task parameters for the same tensor computing task, and to schedule the matrix data to the target computing array in blocks for tensor computing, so as to obtain the calculation results and block indexes corresponding to each block scheduling. The output mapping module is used to map the calculation results corresponding to each block scheduling to the output cache unit according to the block index; the number of input cache units, the size of the computing array and the number of output cache units are defined according to preset hardware parameters.

2. The tensor computing coprocessor of claim 1, wherein, The task parameters include at least the matrix size and data type of each of the matrices to be processed, the rounding method of the calculation results of the computing array, and the storage address of the external memory.

3. The tensor computing coprocessor of claim 2, wherein, The tensor computation coprocessor is further configured to transmit the computation results in the output buffer unit to the external memory based on the preset bus and the storage address of the external memory, according to the preset data format and the storage address.

4. The tensor computing coprocessor of claim 1, wherein, The preprocessing remapping module includes a task parameter mapping module and a matrix data mapping module; the input cache unit includes a task parameter cache unit and a matrix data cache unit; the number of task parameter cache units and the number of matrix data cache units are defined using the preset hardware parameters based on the resource information of the computing platform on which the coprocessor is deployed. The task parameter mapping module is used to receive the task parameters transmitted via a preset bus when there is an idle task parameter cache unit, and to map the task parameters to the idle task parameter cache unit. The matrix data mapping module is used to receive matrix data of each matrix to be processed transmitted based on a preset bus when there is an idle matrix data cache unit, and to map the matrix data to different matrix data cache units. The scheduling and calculation module is used to schedule the task parameters from the task parameter cache unit to the target computing array for the same computation tensor task, and to divide the matrix data in the matrix data cache unit into blocks and schedule them to the target computing array for tensor computation, so as to obtain the calculation results and block indexes corresponding to each block scheduling.

5. The tensor computing coprocessor of claim 4, wherein, If both the number of task parameter cache units and the number of matrix data cache units include multiple units, then after the scheduling calculation module outputs the calculation results and block index corresponding to each block scheduling, the task parameter cache unit and the new matrix data cache unit corresponding to the new tensor calculation task are switched. The scheduling and calculation module is used to schedule new task parameters from the switched task parameter cache unit to the new target computing array, and to schedule new matrix data from the switched matrix data cache unit in blocks to the new target computing array for tensor calculation.

6. The tensor computing coprocessor of claim 4, wherein, The matrices to be processed include a first matrix, a second matrix, and a third matrix, and the matrix data mapping module includes a temporary storage module and a mapping module corresponding to each matrix to be processed; The matrix data mapping module is used to receive the matrix to be processed and matrix category indication information transmitted in row priority order based on a preset bus, and to store the received matrix data in the corresponding temporary storage module of the matrix to be processed according to the matrix category indication information. The mapping module is used to map each row of matrix data of the matrix to be processed in the temporary storage module to a matrix data cache unit corresponding to the matrix category indication information of the matrix to be processed, according to the data type indicated by the task parameters.

7. The tensor computing coprocessor of claim 6, wherein, The matrix data mapping module is further configured to, after storing the received matrix data of each row into the corresponding temporary storage module of the matrix to be processed according to the matrix category indication information, for any matrix to be processed, if the data type of the matrix to be processed is a first data type, then pad each row of matrix data of the matrix to be processed with zeros to a first target bit width for each data element; the bit width corresponding to the first data type is smaller than the first target bit width.

8. The tensor computing coprocessor of claim 6, wherein, The temporary storage module includes a first temporary storage module corresponding to the first matrix, the mapping module includes a first mapping module corresponding to the first matrix, and the matrix data cache unit includes a first matrix data cache unit corresponding to the first matrix. The first mapping module is configured to, when the task parameters indicate that the column size of the first matrix is ​​a first preset size and the bit width corresponding to the data type of the first matrix is ​​less than the first preset bit width, pad each row of matrix data stored in the first temporary storage module with zeros to obtain matrix data with a column size of a second preset size and map it to the corresponding row in the first matrix data cache unit; or, When the task parameters indicate that the column size in the matrix size of the first matrix belongs to the first preset size and the bit width corresponding to the data type of the first matrix is ​​not less than the first preset bit width, or when the task parameters indicate that the column size in the matrix size of the first matrix does not belong to the first preset size, each row of matrix data stored in the first temporary storage module is mapped to the corresponding row in the first matrix data cache unit.

9. The tensor computing coprocessor of claim 8, wherein, The first temporary storage module is provided with a first buffer area of ​​a second preset bit width; The matrix data mapping module is used to store the received matrix data of each row of the first matrix into the first buffer area when the data transmission bandwidth of the preset bus is less than the preset bandwidth, if the column size of the first matrix is ​​a first preset size and the bit width corresponding to the data type of the first matrix is ​​less than the first preset bit width. The first mapping module is used to retrieve each row of matrix data from the first cache after the first cache is full, pad each row with zeros to obtain matrix data with a column size of a second preset size, and map it to the corresponding row in the first matrix data cache unit.

10. The tensor computation coprocessor according to claim 6, characterized in that, The temporary storage module includes a second temporary storage module corresponding to the second matrix, the mapping module includes a second mapping module corresponding to the second matrix, and the matrix data cache unit includes a second matrix data cache unit corresponding to the second matrix. The second mapping module is used to calculate the mapping address of each data element in each row of matrix data stored in the second temporary storage module in the second matrix data cache unit according to the matrix size and data type of the second matrix indicated by the task parameters, and to update the preset value at the mapping address using each data element.

11. The tensor computation coprocessor according to claim 10, characterized in that, The second temporary storage module is equipped with a second buffer area of ​​a third preset bit width; The matrix data mapping module is used to store the received matrix data of each row of the second matrix into the second buffer if the task parameters indicate that the row size of the second matrix is ​​a first preset size and the bit width corresponding to the data type of the second matrix is ​​less than the first preset bit width when the data transmission bandwidth of the preset bus is less than the preset bandwidth. The second mapping module is used to retrieve each row of matrix data from the second cache after the second cache is full, calculate the mapping address of each data element in the row of matrix data in the second matrix data cache unit, and update the preset value at the mapping address using each data element.

12. The tensor computation coprocessor according to claim 6, characterized in that, The temporary storage module includes a third temporary storage module corresponding to the third matrix, the mapping module includes a third mapping module corresponding to the third matrix, and the matrix data cache unit includes a third matrix data cache unit corresponding to the third matrix. The third mapping module is used to, when the task parameters indicate that the bit width corresponding to the data type of the third matrix is ​​a first preset bit width, pad each row of matrix data stored in the third temporary storage module with zeros for each data element to obtain matrix data with a bit width of the first target bit width and map it to the corresponding row in the third matrix data cache unit; or, When the task parameters indicate that the bit width corresponding to the data type of the third matrix is ​​the first target bit width, each row of matrix data stored in the third temporary storage module is mapped to the corresponding row in the third matrix data cache unit.

13. The tensor computation coprocessor according to claim 1, characterized in that, The scheduling and computing module includes schedulers and computing arrays corresponding to different data types; The scheduling and computing module is used to extract the task parameters from the input cache unit for the same tensor computing task, and select the target scheduler and target computing array from the schedulers and computing arrays corresponding to different data types according to the data type of the matrix to be processed indicated by the task parameters, and determine the hardwired between the input cache unit and the target computing array according to the matrix size of the matrix to be processed indicated by the task parameters. The target scheduler is used to schedule the task parameters and matrix data in the input buffer unit to the target computing array for tensor computation according to the hardwired configuration, and obtain the computation result and block index corresponding to each block scheduling.

14. The tensor computation coprocessor according to claim 13, characterized in that, The scheduler also maintains a state machine; the state machine has multiple states for triggering different matrix data scheduling; the number of states is related to the matrix size and the size of the target computing array. The target scheduler is used to schedule the task parameters to the target computing array, and determine the block data size and block scheduling order of each matrix to be processed in each state of the state machine according to the matrix size of the matrix to be processed indicated by the task parameters and the scale of the target computing array. And when the state machine enters a state, according to the hardwired and the block scheduling order, the block data with the block data size in the input buffer unit is scheduled to the target computing array for tensor calculation, so as to obtain the calculation result and block index corresponding to each block scheduling.

15. The tensor computation coprocessor according to claim 14, characterized in that, The computing array includes multiple dot product units that match the size of the computing array; the size of the input block data supported by the dot product unit is dynamically adjusted according to the data type of each matrix to be processed. The target scheduler is used to schedule the block data with the block data size in the input buffer unit to different dot product units in the target computing array for tensor calculation according to the hardwired wiring and the block scheduling order when the state machine enters a state, so as to obtain the calculation result and block index corresponding to each block scheduling.

16. The tensor computation coprocessor according to claim 1, characterized in that, The output mapping module includes a fourth temporary storage module and a fourth mapping module; The output mapping module is used to store the calculation results corresponding to each block scheduling to the fourth temporary storage module; The fourth mapping module is used to determine the write address of the calculation result corresponding to each block scheduling in the fourth temporary storage module according to the block index of the block data, and map the calculation result in the fourth temporary storage module to the write address in the output cache unit.

17. The tensor computation coprocessor according to claim 16, characterized in that, The output mapping module is further configured to, after storing the calculation result corresponding to each block scheduling to the fourth temporary storage module, if the bit width corresponding to the data type of the third matrix in the matrix to be processed is the first preset bit width, restore the bit width of the calculation result corresponding to each block scheduling to the first preset bit width.

18. The tensor computation coprocessor according to claim 1, characterized in that, The preprocessing remapping module is used to receive initial matrix data of each matrix to be processed transmitted based on a preset bus, and when the bit width corresponding to the data type of the matrix to be processed is less than a first preset bit width, it selects valid data from the initial matrix data as the matrix data of the matrix to be processed based on the valid data bit width signal transmitted based on the preset bus.

19. The tensor computation coprocessor according to claim 1, characterized in that, The coprocessor is deployed on any computing platform that supports a preset bus, and the data transmission bandwidth of the preset bus, the number of input cache units, the size of the computing array, and the number of output cache units are all configured in advance using preset hardware parameters based on the resource information of the computing platform.

Images