Data processing methods, pulsating arrays, devices, electronic equipment, and media for matrix computation.

Abstract

This disclosure relates to a data processing method, pulsating matrix, apparatus, device, and medium for matrix computation. The data processing method includes: reading data from a first input matrix and a second input matrix, wherein the first input matrix has M rows and K columns, and the second input matrix has K rows and N columns; inputting the read data from the first input matrix into the pulsating matrix and causing the data from the first input matrix to circulate laterally within the pulsating array; inputting the read data from the second input matrix into the pulsating matrix and causing the data from the second input matrix to circulate diagonally within the pulsating array; performing a multiply-accumulate operation on the data flowing through the first input matrix and the data from the second input matrix, and residing the intermediate results of the calculation in the processing unit; and, in response to determining that the multiply-accumulate operation is complete, unidirectionally propagating the final calculation results residing in each processing unit along the diagonal direction to remove the final calculation results from the pulsating array.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, particularly to the fields of deep learning and digital signal processing, and especially to a data processing method, pulsating array, apparatus, electronic device, and computer-readable storage medium for matrix computation. Background Technology

[0002] With the explosive growth in demand for artificial intelligence, digital signal processing, and high-performance computing, matrix multiplication, as a core operation in these fields, directly impacts the overall system performance. Traditional parallel computing architectures, such as multi-core CPUs and GPUs, while capable of handling parallel tasks, have gradually revealed bottlenecks such as high data transfer overhead, high power consumption, and low energy efficiency when performing large-scale matrix operations.

[0003] As a dataflow-driven dedicated computing architecture, systolic arrays provide an efficient hardware implementation for regularized computations such as matrix operations by arranging processing elements (PEs) in a regularized manner and transferring data between them in a pipelined manner. Output stationary (OS) is a common dataflow pattern in systolic arrays. Its core idea is that during matrix multiplication, the partial summation (psum) of the final output matrix C is fixed (i.e., "stationed") within the PE for accumulation, while the data of input matrices A and B flow through the entire array in a systolic manner. This design greatly enhances data reusability and reduces the need for data movement, thereby effectively improving computational efficiency and throughput. Ensuring that the correct matrix A data meets the correct matrix B data in the correct PE at the correct clock cycle, while reducing data flow latency, is crucial to the accuracy of the computational results and computational efficiency. Summary of the Invention

[0004] This disclosure provides a data processing method, a pulsating array, an apparatus, an electronic device, and a computer-readable storage medium for matrix computation.

[0005] According to one aspect of this disclosure, a data processing method for matrix computation is provided, the method comprising: reading data from a first input matrix and a second input matrix, wherein the first input matrix has M rows and K columns, the second input matrix has K rows and N columns, and wherein M, K, and N are all positive integers; inputting the read data of the first input matrix into a systolic array and causing the data of the first input matrix to circulate laterally within the systolic array, wherein the systolic array is composed of processing units with M rows and N columns; inputting the read data of the second input matrix into the systolic array and causing the data of the second input matrix to circulate diagonally within the systolic array; performing a multiply-accumulate operation on the data of the first input matrix and the data of the second input matrix flowing through each processing unit of the systolic array, and residing the intermediate result of the calculation in the processing unit; and, in response to determining that the multiply-accumulate operation is completed, unidirectionally propagating the final calculation result residing in each processing unit along the diagonal direction to remove the final calculation result from the systolic array.

[0006] According to another aspect of this disclosure, a pulsating array for matrix computation is provided, the pulsating array comprising M rows and N columns of processing units, where M and N are both positive integers, wherein each processing unit includes: a first data input interface and a first data output interface, respectively used to receive and transmit data of a first input matrix; a second data input interface and a second data output interface, respectively used to receive and transmit data of a second input matrix; a multiply-accumulate unit, used to perform multiply-accumulate operations on the received data of the first and second input matrices; a third data input interface and a third data output interface, respectively used to receive and transmit the results of the multiply-accumulate operations; wherein the ports of the M rows and N columns of processing units are configured to form a horizontal circulating data path within the pulsating array, wherein... A horizontal cyclic data path is formed by connecting the first data input interface and the first data output interface of the processing units in the same row of the pulsating array, and is used to realize the horizontal cyclic propagation of the data of the first input matrix in the pulsating array; a diagonal cyclic data path is formed by connecting the second data input interface and the second data output interface of the processing units located on the diagonal, and is used to realize the diagonal cyclic propagation of the data of the second input matrix; and a diagonal unidirectional data path is formed by connecting the third data input interface and the third data output interface of the processing units located on the diagonal, and is used to remove the final calculation result of the multiply-accumulate operation from the pulsating array.

[0007] According to another aspect of this disclosure, a data processing apparatus for matrix calculation is provided, comprising: a reading unit configured to read data from a first input matrix and a second input matrix, wherein the first input matrix has M rows and K columns, the second input matrix has K rows and N columns, and wherein M, K, and N are all positive integers; a first propagation unit configured to input the read data of the first input matrix into a pulsating array and to propagate the data of the first input matrix laterally and cyclically within the pulsating array, wherein the pulsating array is composed of M rows and N columns of processing units; and a second propagation unit configured to... To input the data of the second input matrix into the pulsating array and to circulate the data of the second input matrix in the pulsating array along the diagonal direction; a calculation unit is configured to perform a multiply-accumulate operation on the data of the first input matrix and the data of the second input matrix flowing through it in each processing unit of the pulsating array, and to retain the intermediate result of the calculation in the processing unit; and a third propagation unit is configured to, in response to determining that the multiply-accumulate operation is completed, propagate the final calculation result retained in each processing unit unidirectionally along the diagonal direction to remove the final calculation result from the pulsating array.

[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor, wherein each of the at least one processor includes: a processor cache; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the methods described above.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the above-described method.

[0010] According to another aspect of this disclosure, a computer program product is provided. This computer program product includes a computer program that, when executed by a processor, implements the above-described method.

[0011] According to embodiments of this disclosure, a data processing method for matrix computation is provided, which constructs an output-resident systolic array with a non-triangulated input structure through a novel matrix data propagation path. This scheme completely eliminates the large number of register groups used for data triangulation, thereby significantly reducing chip area and power consumption, simplifying backend layout, and reducing computational latency.

[0012] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0013] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0014] Figures 1a-1c A schematic diagram of the implementation of a pulsating array in related technologies is shown; Figure 2 A flowchart of a data processing method for matrix calculation according to an embodiment of the present disclosure; Figure 3 A schematic diagram illustrating the reading of a first input matrix and a second input matrix according to an embodiment of the present disclosure is shown; Figure 4 A schematic diagram illustrating the lateral cyclic propagation of data in a pulsating array according to an embodiment of the present disclosure is shown. Figure 5 A schematic diagram is shown illustrating the cyclic propagation of data in a diagonal direction in a pulsating array according to an embodiment of the present disclosure; Figure 6 A schematic diagram is shown illustrating the unidirectional propagation of data in a diagonal direction in a pulsating array according to an embodiment of the present disclosure; Figure 7 A schematic diagram is shown illustrating matrix calculation using a data processing method for matrix calculation according to an embodiment of the present disclosure; Figure 8 A schematic diagram is shown illustrating multiple matrix operations performed using a data processing method for matrix computation according to an embodiment of the present disclosure; Figure 9 An exemplary block diagram of a pulsating array for matrix computation according to an embodiment of the present disclosure is shown; Figure 10 A schematic diagram of the structure of a processing unit according to an embodiment of the present disclosure is shown; Figure 11 An exemplary block diagram of a data processing apparatus for matrix calculation according to embodiments of the present disclosure is shown; and Figure 12 A block diagram illustrating an example of an electronic device according to an exemplary embodiment of the present disclosure is shown. Detailed Implementation

[0015] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0016] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0017] The core objective of this disclosure is to efficiently handle the multiplication operation of two matrices, that is, to calculate C. M,N =A M,K ×B K,N This type of computation constitutes the most core and frequent computational task in modern artificial intelligence, especially in deep neural network models, particularly in advanced neural network architectures such as Large Language Models (LLMs) and Transformers. In typical applications of neural networks, the first input matrix (matrix A) usually refers to the input feature map (ifmap), which represents the data input to a certain layer of the neural network or the features calculated by the previous layer. The second input matrix (matrix B) usually refers to the weights, which are the parameters learned by the neural network model after training on a large amount of data, reflecting the model's knowledge. The result matrix (matrix C) obtained by multiplying these two matrices is the output feature map (ofmap), which will serve as the input to the next layer or the final output of the model. Therefore, the systolic array and its data processing method provided in this disclosure can efficiently accelerate the computation of key layers such as convolutional layers and fully connected layers in neural networks.

[0018] To facilitate understanding of this disclosure, some key terms used in the specification are defined herein: Systolic Array: A hardware architecture that efficiently computes matrix multiplication using a regularized array of Processing Elements (PEs) and a pipelined data flow.

[0019] Processing Element (PE): The basic computing and storage unit that makes up the systolic array.

[0020] Output Stationary (OS). This is a data flow pattern in which the intermediate and final results (ofmap) of matrix multiplication and accumulation remain stationary (stationary) in the PE for accumulation.

[0021] Input feature map (ifmap): Data input to a neural network or a specific layer thereof, which in this disclosure corresponds to the first input matrix.

[0022] Output feature map (ofmap): The result of the neural network calculation, which corresponds to the final calculation result in this disclosure.

[0023] Weight matrix: The weight parameters obtained by training the neural network, which in this disclosure correspond to the second input matrix.

[0024] Partial summation (psum): The intermediate sum generated during the computation of a neural network.

[0025] Column-major order: a data storage or retrieval method that means that data in the same column of a matrix is ​​stored contiguously in memory.

[0026] Row-major order: a data storage or retrieval method that means that data within the same row of a matrix is ​​stored contiguously in memory.

[0027] Register file: A high-speed storage circuit, commonly used in this field to cache data or perform timing adjustments on data streams.

[0028] In related technologies, to ensure that the correct first matrix data meets the correct second matrix data in the correct PE at the correct clock cycle, it is typically required that the input and output data streams exhibit a "triangular" shape. For example... Figure 1a As shown, to ensure computational continuity, the inputs of matrices A and B are triangulated. Specifically, both the input and output data streams resemble an oblique wavefront, intentionally offset in time to ensure the orderly execution of the entire computation process. Matrix A is input in column-major order and propagates from left to right within the PE array; matrix B is input in row-major order and propagates from top to bottom within the PE array. Taking a 3x3 PE array as an example, PE[i][j] represents the PE in the i-th row and j-th column, where i,j=0,1,2,3, as... Figure 1bAs shown, data a and 1 are multiplied in the top-left PE[0][0] during the first clock cycle (cycle 0); however, a must be multiplied by 4, and 1 must be multiplied by d, and these two events must be delayed until the second clock cycle (cycle 1), occurring in different PEs. Figure 1b As shown, the data flow sequence for each cycle is: In the first clock cycle (cycle 0), a and 1 are multiplied in PE[0][0].

[0029] In the second clock cycle (cycle 1), b and 2 are multiplied in PE[0][0] and added to the result of the previous multiplication; a and 4 are multiplied in PE[1][0]; 1 and d are multiplied in PE[0][1].

[0030] And so on, in the fourth clock cycle (cycle 3), the first complete multiplication-accumulation result... It is obtained in PE[0][0].

[0031] like Figure 1c As shown, matrix C, as the calculation result, is also output from the pulsating array in a triangular array.

[0032] In the fifth clock cycle (cycle 4), PE[1][0] obtains the multiply-accumulate result. PE[0][1] yields the multiplication and accumulation result. . Move down to PE[1][0].

[0033] In the sixth clock cycle (cycle 5), PE[2][0] obtains the multiply-accumulate result. PE[0][2] yields the multiplication and accumulation result. PE[1][1] yields the multiplication and accumulation result . Move down to PE[2][0], Move down to PE[1][1].

[0034] And so on, in the last calculation result of the eleventh clock cycle (cycle 10) Leaving PE[3][3], the result of multiplying matrix A and matrix B is output from the PE array.

[0035] Understandably, to achieve the triangular data configuration shown above, it is necessary to configure large-scale, triangularly arranged register sets at both the input and output ends of the systolic array. While these additional hardware units solve the data synchronization problem, they also introduce three levels of serious side effects, greatly limiting their application in modern high-performance chip design.

[0036] First, at the physical level, these triangular register groups are huge resource consumers. They not only occupy extremely valuable chip area but also bring significant static and dynamic power consumption. When the scale of the systolic array expands to meet higher computing power demands, this hardware overhead can become unbearable. Second, from a chip design perspective, these irregular triangular hardware structures are a nightmare for backend physical layout and routing. They disrupt the regularity of the chip layout, leading to low space utilization and making routing exceptionally complex. Finally, at the performance level, this data pattern imposes a heavy "latency constraint" on the entire computation process. Because data needs to "fill" the entire array like waves before slowly "draining," it results in significant pipeline start-up and end-of-pipeline delays, leaving many processing units idle at the beginning and end of the computation. This latency is costly, with a total duration approximately twice the array's side length, severely impacting the real-time performance of computation and failing to meet the demands of increasingly low-latency applications.

[0037] To address one or more of the problems existing in related technologies, this disclosure provides a data processing method for matrix computation. This method constructs an output-resident systolic array with a non-triangulated input structure through a novel matrix data propagation path. This scheme completely eliminates the large number of register sets used for data triangulation, thereby significantly reducing chip area and power consumption, simplifying backend layout, and reducing computational latency.

[0038] The embodiments of this disclosure are described in detail below with reference to the accompanying drawings.

[0039] Figure 2 A flowchart illustrating an exemplary process of a data processing method 200 for matrix calculation according to an embodiment of the present disclosure is shown.

[0040] like Figure 2 As shown, the data processing method 200 for matrix calculation includes: Step S201: Read the data from the first input matrix and the second input matrix, wherein the first input matrix has M rows and K columns, the second input matrix has K rows and N columns, and M, K and N are all positive integers; Step S202: Input the data of the first input matrix into the pulsation array, and make the data of the first input matrix propagate horizontally in the pulsation array, wherein the pulsation array is composed of M rows and N columns of processing units; Step S203: Input the data of the second input matrix into the pulsation array, and make the data of the second input matrix propagate cyclically along the diagonal direction in the pulsation array; Step S204: Within each processing unit of the pulsating array, perform a multiply-accumulate operation on the data flowing through the first input matrix and the data of the second input matrix, and store the intermediate results of the calculation in that processing unit; and Step S205: In response to determining that the multiplication-accumulation operation is completed, the final calculation result residing in each processing unit is propagated unidirectionally along the diagonal direction to remove the final calculation result from the pulsating array.

[0041] For example, the first input matrix can be the aforementioned 3x3 matrix A, and the second input matrix can be the aforementioned 3x3 matrix B. The data processing method 200 for matrix calculation is used for the multiplication calculation of matrix A and matrix B. For ease of description, the following description will use matrix A as the first input matrix, matrix B as the second input matrix, and matrix C as the calculation result of the multiplication of matrix A and matrix B to expand the technical solution of this disclosure. It can be understood that method 200 can process matrix M. K, K In matrix multiplication calculations of N, M, K, and N can be positive integers set according to needs and application scenarios. They do not need to be equal or limited to the 3x3 dimensions shown in matrices A and B.

[0042] Step S201 reads the two input matrices. For example, the data of the two input matrices can be read from memory respectively, so that in steps S202 and S203 the read data of the two input matrices can be input into the pulsation array respectively for matrix multiplication calculation.

[0043] Step S202 causes the data of the first input matrix to propagate laterally within the systolic array, and step S203 causes the data of the second input matrix to propagate diagonally within the systolic array. This ensures that the data of the first data matrix is ​​multiplied by the corresponding data of the second input matrix at the correct processing unit within the systolic array during the correct clock cycle, thus performing the multiplication of the two matrices in an orderly manner within the processing unit. After the multiplication and accumulation operation of the data from the two input matrices is completed, step S205 causes the calculation result to propagate unidirectionally along the diagonal direction to remove the final calculation result from the systolic array to obtain the output matrix.

[0044] Therefore, by employing a novel, parallel system of horizontal loops, diagonal loops, and diagonal unidirectional propagation paths, the design of traditional OS systolic arrays, which rely on triangular data structures, is overturned. This fundamentally eliminates the hardware dependency on large-scale triangular register sets, thus providing a foundation for reducing hardware costs, power consumption, and computational latency. In neural network applications, this means that AI accelerator chips can integrate larger-scale computing arrays or achieve lower power consumption within the same area.

[0045] According to some embodiments, the first input matrix is ​​read in column-major order, and the second input matrix is ​​read diagonally. The first input matrix is ​​read within a first plurality of clock cycles, and the second input matrix is ​​read within a second plurality of clock cycles. The first plurality of clock cycles and the second plurality of clock cycles start simultaneously. Step S201 includes: in each of the first plurality of clock cycles, reading the data column of the first input matrix corresponding to that clock cycle; and in each of the second plurality of clock cycles, reading data from one or more diagonals of the second input matrix. The one or more diagonals of the second input matrix are determined based on the number of columns K of the first input matrix, the number of columns N of the second input matrix, and the index of that clock cycle.

[0046] To accommodate the novel data propagation path within the systolic array, the data reading method in step S201 is further defined. The first input matrix is ​​read in column-major order, meaning that an entire column of data is read within each clock cycle. Since the first input matrix has K columns, its reading process is completed within K clock cycles (i.e., the "first multiple clock cycles"). The second input matrix is ​​read along a unique diagonal direction, requiring a second multiple clock cycle to complete. The reading processes of the two matrices begin synchronously.

[0047] This achieves a perfect match between the front-end data reading and the back-end data propagation path. In neural network applications, this means that ifmap data (activation values) and weight data (weight parameters) can be directly streamed from memory into the computation array for matrix calculations without additional timing adjustments. This directly eliminates the need for the triangular register set at the input end for data alignment, significantly reducing the hardware cost and power consumption for loading weights and activation values.

[0048] According to some embodiments, the one or more diagonals of the second input matrix are determined as follows: the i-th and i+max(N,K) diagonals are read in clock cycle i, and the j-th diagonal is read in clock cycle j, where i is a non-negative integer not greater than min(N, K)-2, j is an integer not less than min(N, K)-1 and not greater than max(N, K)-1, and the direction of the diagonals is from the upper right to the lower left of the second input matrix, and the diagonals of the second input matrix are numbered from the upper left to the lower right.

[0049] First, the direction of the diagonal is defined as from the top right to the bottom left of the second input matrix, and the diagonals are numbered starting from the top left corner of the matrix. This reading method can be determined by a unified mathematical formula: In clock cycle i, read the i-th and i+max(N,K)-th diagonal lines; At clock cycle j, read the j-th diagonal line; Where i is a non-negative integer not greater than min(N, K)-2, and j is an integer not less than min(N, K)-1 and not greater than max(N, K)-1.

[0050] for The calculation, Read column by column, requiring a total of K reads; Reading diagonally requires a total of max(N, K) reads.

[0051] Figure 3 A schematic diagram of reading a first input matrix and a second input matrix according to an embodiment of the present disclosure is shown.

[0052] like Figure 3 As shown, in the first clock cycle (clock cycle 0), i.e., when i is 0, the 0th column of the first input matrix A and the 0th and 3rd diagonals of the second input matrix B are read; in the second clock cycle (clock cycle 1), i.e., when i is 1, the 1st column of the first input matrix A and the 1st and 4th diagonals of the second input matrix B are read; in the third clock cycle (clock cycle 2), j.e., when j is 2, the 2nd column of the first input matrix A and the 2nd diagonal of the second input matrix B are read.

[0053] This provides a precise and unambiguous scheme to guide data reading in accordance with the propagation path of matrix data in the systolic array. This allows for deterministic data preprocessing and loading of input matrices of arbitrary dimensions based on the above formula, ensuring that the matrix data is fed into the systolic array in the correct sequence for multiplication calculations. In neural network computation, this ensures that complex weight matrices are fed in the correct sequence, guaranteeing that each weight value meets its corresponding activation value at the correct time. This is fundamental to ensuring the mathematical correctness of the results of convolution or matrix multiplication operations.

[0054] According to some embodiments, step S202 includes: in each of the first plurality of clock cycles, inputting M data from the data column of the first input matrix read into the M processing units of the 0th column of the pulsating array respectively.

[0055] Therefore, after each M data point in a column of the first input matrix is ​​read, it is input into the M processing units in the 0th column of the systolic array so that it propagates horizontally from the 0th column to the right in the systolic array.

[0056] According to some embodiments, step S203 includes: in each of the second plurality of clock cycles, inputting multiple data from one or more diagonals of the second input matrix into multiple processing units in the 0th row of the pulsating array respectively.

[0057] Therefore, after reading multiple data from one or more diagonals of the second input matrix, the data is input into multiple processing units in row 0 of the systolic array, so that it propagates cyclically in the systolic array starting from row 0 in the diagonal direction.

[0058] According to some embodiments, a third plurality of clock cycles are required from the start of reading the first input matrix and the second input matrix to the removal of the final calculation result from each processing unit from the systolic array, and wherein step S202 further includes: in each of the third plurality of clock cycles, in response to determining that the processing unit located at (x, y) has data containing the first input matrix, propagating the data to the right to (x, y+1), where x is a non-negative integer not greater than M-1 and y is a non-negative integer not greater than N-2; and in response to determining that the processing unit located at (o, N-1) has data containing the first input matrix, propagating the data to (o, 0), where o is a positive integer not greater than M-1.

[0059] Specifically, within one clock cycle, the PE at (x, y) responds to determining that it has data with the first input matrix by propagating its data to the right to the PE at (x, y+1). When the data reaches the last column (N-1 columns) of a row, it is propagated back to the PE in column 0 of that row in the next clock cycle, forming a loop. An exception is that the PE in row 0, after propagating the data to the last column, does not propagate it back.

[0060] Figure 4 A schematic diagram of data propagating laterally in a pulsating array according to an embodiment of the present disclosure is shown.

[0061] like Figure 4 As shown, within one clock cycle, the PE at (x, y) responds to determining that it has data with the first input matrix by propagating its data to the right to the PE at (x, y+1). For example, the data of the first input matrix at (1,0) will be propagated to (1,1), and the data of the first input matrix at (1,1) will be propagated to (1,2). When the data reaches the last column of a row, i.e., the second column, it will be propagated back to the PE in the 0th column of that row in the next cycle, forming a loop. An exception is that the PE in the 0th row, after propagating the data to the last column, does not propagate back; that is, the data of the first input matrix at (0,2) is not propagated back to (0,0).

[0062] Thus, by using inline lateral loop propagation, the maximum reuse of the first input matrix data (e.g., ifmap) is achieved. In convolution operations of neural networks, an input activation value typically needs to be multiplied by multiple weight values ​​within a convolution kernel. The lateral loop propagation of this disclosure allows an activation value, after being read into a row, to sequentially pass through all N PEs in that row, thereby performing operations with N different weight columns, without repeatedly reading from memory. This significantly reduces the demand for memory bandwidth and improves energy efficiency.

[0063] According to some embodiments, a third plurality of clock cycles are required from the start of reading the first input matrix and the second input matrix to the removal of the final calculation result from the systolic array in each processing unit, and wherein step S202 further includes: in each of the third plurality of clock cycles, except for the processing unit located at (M-1, N-1), in response to determining that the processing unit located at (p, q) has data containing the second input matrix, the data is propagated diagonally to ((p+1)%M, (q+1)%N), where p is a non-negative integer not greater than M-1 and q is a non-negative integer not greater than N-1.

[0064] Figure 5 A schematic diagram illustrating the diagonal propagation of data in a pulsating array according to an embodiment of the present disclosure is shown. Specifically, within one clock cycle, except for the PE (M-1, N-1) located at the lower right corner, other PEs located at (p, q) propagate their data diagonally to the PE at ((p+1)%M, (q+1)%N). The modulo (%) operation ensures that the data automatically "loops back" when it reaches the array boundary, forming a cyclic data stream covering the entire array.

[0065] This ingenious diagonal loop path allows the second input matrix data (e.g., weight data) to systematically and periodically sweep across the entire computational array. In neural network applications, this means that a weight parameter, once loaded, can be operated on with multiple ifmap activation values ​​from different spatial locations. This perfectly matches the core idea of ​​"weight sharing" in convolution operations and also greatly improves data reuse.

[0066] According to some embodiments, step S205 includes: in response to determining that the multiply-accumulate operation in the processing unit located at (M-1, n) is completed, removing the final calculation result residing in the processing unit at (M-1, n) from the pulsating array, where n is a non-negative integer not greater than N-1; and in response to determining that the multiply-accumulate operation in the processing unit located at (m, n) is completed, propagating the final calculation result residing in the processing unit at (m, n) along the diagonal direction to ((m+1)%M,(n+1)%N), where m is a non-negative integer not greater than M-2.

[0067] Figure 6 A schematic diagram illustrating the unidirectional propagation of data in a diagonal direction within a pulsating array according to an embodiment of the present disclosure is shown. Figure 6 As shown, once the calculation is complete, the PE located in row M-1, the last row, will directly output its final calculation result as the array output. The PEs located in other rows will propagate their results to the PEs located diagonally below ((m+1)%M,(n+1)%N), passing them down level by level, and finally outputting them from the last row as well.

[0068] As can be seen, the above embodiments define an efficient and well-organized data outgoing path that requires no additional timing adjustments, thereby completely avoiding the use of a triangular register set at the output. For neural networks, this means that the computed output feature map (ofmap) can be ejected from the array with minimal delay, allowing the next neural network layer in the pipeline to begin processing more quickly, thus improving the inference speed of the entire network.

[0069] According to some embodiments, the data processing method 100 for matrix calculation further includes: after removing the final calculation result residing in each processing unit from the pulsating array, performing inter-column data rotation on the output final calculation result to restore the column order of the output matrix data.

[0070] Due to the unique diagonal unidirectional shift path described above, the column order of the output results is disordered (e.g., the column order of the output row of a 3x3 array is columns 1, 2, and 0). Therefore, after the results are completely shifted out of the systolic array, it is necessary to perform inter-column data rotation on the final calculated output results to restore their original, correct column order.

[0071] Thus, the problem of column order disorder caused by efficient output paths is solved through a simple hardware post-processing step. This allows Method 200 to enjoy the advantages of low latency and low cost brought by diagonal output paths, while ensuring that the data finally output to system memory or the next computing unit is completely correct, forming a complete and practical technical closed loop of this disclosure.

[0072] Figure 7A schematic diagram illustrating matrix calculation using a data processing method for matrix calculation according to an embodiment of the present disclosure is shown. (Refer to...) Figure 7 Taking the calculation of the multiplication of two 3x3 matrices A and B using a 3x3 PE array with M=3, N=3, K=3 as an example, the process of data flow, calculation and final result output within the array is shown in stages and clock cycles.

[0073] like Figure 7 As shown, the first stage (clock cycles 0-5) is used for data reading, input, and computation. In this stage, the data of the first input matrix A (ifmap) and the second input matrix B (weight) are read cycle by cycle and input into the systolic array, while the PEs inside the array perform multiply-accumulate operations in parallel.

[0074] At clock cycle 0, read the 0th column data of A and the 0th and 3rd diagonal data of B from memory.

[0075] At clock cycle 1, the data in column 0 of matrix A is input to column 0 of the systolic array (PE), and the diagonal data of matrix B is input to row 0 of the systolic array (PE). Thus, the data from matrix A and matrix B located at PE(0,0) meet and perform the first multiply-accumulate operation. The remaining PEs in the systolic array remain in a waiting state.

[0076] At clock cycle 2, the first column of matrix A and the first and fourth diagonals of matrix B are input into the 0th column and 0th row of the systolic array. Simultaneously, the data input at clock cycle 1 begins to systolic: the data in the 0th column of matrix A moves horizontally from the 0th column of the systolic array to the 1st column; the data in the 0th and third diagonals of matrix B moves diagonally from the 0th row of the systolic array. Thus, more PEs in the systolic array are activated. For example, at PE(1,1), the data "4" from matrix A (from PE(1,0)) meets the data "A" from matrix B (from PE(0,0)) and is calculated.

[0077] During clock cycles 3-4, the process of "reading new data, inputting into the systolic array, and systolicing old data" continues. Data in matrix A continues to propagate to the right (data in rows 1 and 2 will cycle back to column 0 after reaching the rightmost position), and data in matrix B continues to propagate along the diagonal (it will cycle after reaching the boundary). As data continues to propagate and fill, all PEs in the array are activated and continuously perform multiply-accumulate operations. The psum value inside each PE is constantly updated.

[0078] At clock cycle 5, all input data has entered the systolic array and completed its respective propagation path. At the end of the 5th clock cycle, all PEs in the systolic array have completed all multiplication and accumulation operations of their respective partial sums (psum). At this moment, each PE contains an element value of the final calculation result matrix C.

[0079] After clock cycle 5 ends, the second stage (clock cycles 6-8) begins, where the calculation results are output. In this stage, the final calculation result (ofmap) residing inside the PE begins to move out of the pulsating array along a diagonal unidirectional path.

[0080] At clock cycle 6, the systolic array switches from computation mode to output mode. The PE located in row 2 (i.e., row M-1) outputs its result. The PEs in other rows pass their results to the PEs on the diagonal below. Finally, row 2 of matrix C is output.

[0081] At clock cycle 7, the data as the result of the calculation continues to move along the diagonal path, and the result of the first row moves to the second row. Thus, the first row of matrix C is output from the PE of the second row.

[0082] At clock cycle 8, the data representing the calculation results moves again along the diagonal path, with the result from row 0 moving to row 2. Thus, row 0 of matrix C is output. At this point, all calculation results have been removed from the array. It is worth noting that the output data column order is scrambled (the output order is columns 1, 2, 0), requiring subsequent column rotation to restore the correct column order.

[0083] Figure 7 This paper fully demonstrates the correctness and efficiency of the non-triangulated data flow of this disclosure. It shows that by combining a specific data reading scheme with a novel internal propagation path, matrix multiplication can be performed correctly without any additional triangulation register sets. Its most significant technical effect is a substantial reduction in computational latency. For a 3x3 computation, the traditional triangulation scheme requires approximately 11 cycles to output the complete result, while this invention completes the entire computation in only 5 cycles and outputs the result in the next 3 cycles, significantly reducing the total time and greatly improving processing efficiency. In neural network applications, this means that the computational tasks for each layer are processed in less time, thereby improving the inference or training speed of the entire network.

[0084] Figure 8 A schematic diagram is shown illustrating multiple matrix operations performed using a data processing method for matrix computation according to an embodiment of the present disclosure.

[0085] Reference Figure 8This disclosure also supports bubble-free continuous computation for multiple matrix multiplication tasks. Figure 8 Take three consecutive matrix multiplications (A0×B0=C0, A1×B1=C1, A2×B2=C2) as an example. In the diagram, light-colored blocks represent ongoing calculations, and dark-colored blocks represent completed and temporarily stored results.

[0086] During clock cycles 1-3, the data for the first matrix multiplication (A0×B0) is continuously read and input into the systolic array. PE is gradually activated and begins calculation for C0. By the end of cycle 3, all data for A0 and B0 has been input into the array.

[0087] After the last data of A0B0 is input in clock cycle 3, the array does not pause or idle in any cycle. It immediately starts reading and inputting the first data of the second matrix multiplication (A1×B1) in clock cycle 4, thus achieving a seamless connection between the two calculation tasks.

[0088] During clock cycles 4-6, data from A1B1 is continuously input, and the PE begins calculating C1 (as shown by the light-colored blocks in the diagram). Simultaneously, the result of the first calculation, C0, is fully calculated in clock cycle 5 (as shown by the dark-colored blocks in the diagram). Exemplarily, these completed C0 results are temporarily stored in a special register within each PE and, starting in clock cycle 6, are shifted out of the array along a diagonal unidirectional path without interfering with the ongoing calculation of C1. Thus, the systolic array achieves parallel operation by simultaneously calculating new tasks and shifting out old results.

[0089] In subsequent clock cycles, the above pattern continues. Data A2B2 is input immediately after A1B1, the systolic array begins calculating C2, and the result of C1 is shifted out in the background. It is evident that the systolic array's computational resources are constantly busy throughout this process.

[0090] Figure 8 The bubble-free computation demonstrated is a significant performance advantage of the method provided in this disclosure. Its technical effect lies in achieving near 100% hardware utilization. Traditional architectures typically require waiting for the results of one task to be completely removed (emptying the pipeline) before starting the next task, generating numerous idle cycles (i.e., "bubbles") and wasting valuable computational resources. This disclosure completely eliminates these bubbles by setting a result register within the PE (Process Execution Unit).

[0091] In neural network applications, a complex network model consists of dozens or even hundreds of layers, each representing an independent matrix operation. This invention can significantly improve the throughput of processing consecutive network layers because the switching between layers incurs almost no time overhead. This offers substantial performance gains for applications requiring high throughput, such as data center inference and video stream analysis.

[0092] In particular, matrix multiplication is the core and most computationally expensive operation in advanced neural network architectures such as large language models and Transformers. The self-attention mechanism and feed-forward networks in Transformer models heavily rely on large-scale matrix multiplication. These models typically contain hundreds or even trillions of parameters, resulting in astronomical matrix operations required for a single inference or training task. The method disclosed in this paper, with its bubble-free continuous computation and extremely low computational latency, can directly and efficiently accelerate the computation process of these complex models, significantly shortening the model training cycle and improving inference throughput. This has irreplaceable significance and performance improvement value for promoting the application of large language models in natural language processing, code generation, multimodal AI, and meeting future AI computing power demands.

[0093] In summary, Method 200 successfully solves the core problem of existing technologies that rely on large-scale triangular register sets for data synchronization by employing a co-designed data reading scheme, data input method, and three internal data propagation paths. This fundamental change brings about a series of significant technical benefits: First, at the physical level, it directly saves a significant amount of chip area and reduces power consumption, enabling the integration of larger-scale computing arrays or achieving better energy efficiency at the same cost. Second, at the performance level, by eliminating pipeline startup and drain latency, it reduces the total computation time to N+K-1 clock cycles, significantly reducing the latency of a single matrix operation. Furthermore, this method supports seamless continuous computation, eliminating "bubbles" between task switching, thereby greatly improving the effective utilization of hardware and overall data throughput. Ultimately, in applications such as neural networks, these effects collectively translate into higher inference speeds and better energy efficiency, providing a solid hardware foundation for deploying more complex and powerful AI models.

[0094] According to another aspect of this disclosure, a pulsating array for matrix computation is also provided. For example... Figure 9 As shown, the pulsating array 900 consists of M rows and N columns of processing units 901, where M and N are both positive integers. The internal structure of the processing unit 901 is as follows: Figure 10As shown. Each processing unit 901 includes: a first data input interface and a first data output interface, respectively used to receive and send data of a first input matrix; a second data input interface and a second data output interface, respectively used to receive and send data of a second input matrix; a multiply-accumulate unit, used to perform multiply-accumulate operations on the received data of the first and second input matrices; a third data input interface and a third data output interface, respectively used to receive and send the results of the multiply-accumulate operations; wherein, the ports of the M rows and N columns of the processing unit are configured to form a horizontal circular data path within the pulsating array, the horizontal circular data path connecting the processing units in the same row of the pulsating array. The system comprises: a first data input interface and a first data output interface connected together, used to realize the horizontal cyclic propagation of the data of the first input matrix in the pulsating array; a diagonal cyclic data path, formed by diagonally connecting the second data input interface and the second data output interface of the processing unit located on the diagonal, used to realize the diagonal cyclic propagation of the data of the second input matrix; and a diagonal unidirectional data path, formed by diagonally connecting the third data input interface and the third data output interface of the processing unit located on the diagonal, used to remove the final calculation result of the multiply-accumulate operation from the pulsating array.

[0095] For example, the first input matrix can be the aforementioned 3x3 matrix A, and the second input matrix can be the aforementioned 3x3 matrix B. The systolic array 900 is then used for the multiplication of matrices A and B. A completely new, parallel set of horizontal, diagonal, and diagonal unidirectional propagation paths is achieved through the horizontal cyclic data path, diagonal cyclic data path, and diagonal unidirectional data path in the systolic array 900. This overturns the traditional OS systolic array design that relies on triangulated data structures, fundamentally eliminating the hardware dependence on large-scale triangular register sets, thus providing a foundation for reducing hardware costs, power consumption, and computational latency. In neural network applications, this means that AI accelerator chips can integrate larger-scale computing arrays or achieve lower power consumption within the same area.

[0096] According to some embodiments, the systolic array is further configured to complete the reception of the first input matrix within a first plurality of clock cycles and to complete the reception of the second input matrix within a second plurality of clock cycles. The systolic array is also configured to simultaneously begin receiving data from both the first and second input matrices. Furthermore, the systolic array is further configured to: receive a data column of the first input matrix corresponding to that clock cycle within each of the first plurality of clock cycles; and receive data from one or more diagonals of the second input matrix within each of the second plurality of clock cycles, wherein the one or more diagonals of the second input matrix are determined based on the number of columns K of the first input matrix, the number of columns N of the second input matrix, and the index of that clock cycle.

[0097] Understandably, to accommodate the novel data propagation path within the systolic array, the data from the two matrices can be read and received by the systolic array in a specific manner. For example, the first input matrix is ​​read in column-major order, meaning that an entire column of data is read within each clock cycle. Since the first input matrix has K columns, its reading process is completed within K clock cycles (i.e., the "first multiple clock cycles"). The second input matrix is ​​read along a unique diagonal direction, requiring a second multiple clock cycle to complete. The reading processes of the two matrices begin synchronously.

[0098] This achieves a perfect match between the front-end data reading and the back-end data propagation path. In neural network applications, this means that ifmap data (activation values) and weight data (weight parameters) can be directly streamed from memory into the computation array for matrix calculations without additional timing adjustments. This directly eliminates the need for the triangular register set at the input end for data alignment, significantly reducing the hardware cost and power consumption for loading weights and activation values.

[0099] According to some embodiments, the one or more diagonals of the second input matrix are determined as follows: the i-th and i+max(N,K) diagonals are read in clock cycle i, and the j-th diagonal is read in clock cycle j, where i is a non-negative integer not greater than min(N, K)-2, j is an integer not less than min(N, K)-1 and not greater than max(N, K)-1, and the direction of the diagonals is from the upper right to the lower left of the second input matrix, and the diagonals of the second input matrix are numbered from the upper left to the lower right.

[0100] First, the direction of the diagonal is defined as from the top right to the bottom left of the second input matrix, and the diagonals are numbered starting from the top left corner of the matrix. This reading method can be determined by a unified mathematical formula: In clock cycle i, read the i-th and i+max(N,K)-th diagonal lines; At clock cycle j, read the j-th diagonal line; Where i is a non-negative integer not greater than min(N, K)-2, and j is an integer not less than min(N, K)-1 and not greater than max(N, K)-1.

[0101] for The calculation, Read column by column, requiring a total of K reads; Reading diagonally requires a total of max(N, K) reads.

[0102] According to some embodiments, the first input matrix has M rows and K columns, where K is a positive integer, and wherein the first data input interface of the M processing units in the 0th column of the systolic array is configured to receive M data from the data column of the first input matrix read from memory in each of the first plurality of clock cycles.

[0103] Therefore, after each M data points in a column of the first input matrix are read, the first data input interface of the M processing units in the 0th column of the pulsating array is configured to receive the aforementioned M data points so that they propagate horizontally to the right in the pulsating array from the 0th column.

[0104] According to some embodiments, the second input matrix has K rows and N columns, where K is a positive integer, and wherein the second data input interface of the N processing units in the 0th row of the systolic array is configured to receive multiple data from one or more diagonals of the second input matrix read from memory in each of the second plurality of clock cycles.

[0105] Therefore, after reading multiple data from one or more diagonals of the second input matrix, the data is input into multiple processing units in the 0th row of the systolic array. The second data input interface of the N processing units in the 0th row of the systolic array is configured to receive the aforementioned multiple data, so that it is cyclically propagated in the systolic array starting from the 0th row in the diagonal direction.

[0106] According to some embodiments, the pulsating array is configured to receive the first input matrix and the second input matrix and shift out the final calculation result within a third plurality of clock cycles, wherein the processing unit located at (x, y) is further configured to: in each of the third plurality of clock cycles, in response to determining data having the first input matrix, propagate the data along the horizontal cyclic data path to propagate the data to the right to (x, y+1), where x is a non-negative integer not greater than M-1 and y is a non-negative integer not greater than N-2; and the processing unit located at (o, N-1) is further configured to: in each of the third plurality of clock cycles, in response to determining data having the first input matrix, propagate the data along the horizontal cyclic data path to propagate the data to (o, 0), where o is a positive integer not greater than M-1.

[0107] Specifically, within one clock cycle, the PE at (x, y) responds to determining that it has data with the first input matrix by propagating its data right along the horizontal circular data path to the PE at (x, y+1). When the data reaches the last column (N-1 columns) of a row, it is propagated back to the PE in column 0 of that row in the next clock cycle, forming a loop. An exception is that the PE in row 0 does not propagate back after passing the data to the last column.

[0108] Thus, a lateral recurrent data path enables data to circulate laterally within rows, thereby maximizing the reuse of the first input matrix data (e.g., ifmap). In convolution operations of neural networks, an input activation value typically needs to be multiplied by multiple weight values ​​within a convolution kernel. The lateral recurrent data path disclosed herein allows an activation value, after being read into a row, to sequentially pass through all N PEs in that row, thereby performing operations with N different weight columns without repeatedly reading from memory. This significantly reduces the demand for memory bandwidth and improves energy efficiency.

[0109] According to some embodiments, the pulsating array is configured to complete the reception of the first input matrix and the second input matrix and the shifting out of the final calculation result within a third plurality of clock cycles, wherein, in addition to the processing unit located at (M-1, N-1), the processing unit located at (p, q) is further configured to: in each of the third plurality of clock cycles, in response to determining data having the second input matrix, propagate the data along the diagonal cyclic data path to propagate the data diagonally to ((p+1)%M, (q+1)%N), where p is a non-negative integer not greater than M-1 and q is a non-negative integer not greater than N-1.

[0110] Within one clock cycle, except for the PE (M-1, N-1) located in the lower right corner, other PEs located at (p, q) will propagate their data along the diagonal circular data path to the PE at ((p+1)%M, (q+1)%N). The modulo (%) operation ensures that the data automatically "loops back" when it reaches the array boundary, forming a circular data stream covering the entire array.

[0111] The diagonal cyclic data path in the systolic array 900 allows the second input matrix data (e.g., weight data) to systematically and periodically sweep across the entire computational array. In neural network applications, this means that a weight parameter, once loaded, can be operated on with multiple ifmap activation values ​​from different spatial locations. This perfectly matches the core idea of ​​"weight sharing" in convolution operations and also greatly improves data reuse.

[0112] According to some embodiments, the processing unit located at (M-1, n) is further configured to: in response to determining that the multiply-accumulate operation is completed, remove the result of the multiply-accumulate operation from the pulsation array, where n is a non-negative integer not greater than N-1; and the processing unit located at (m, n) is further configured to: in response to determining that the multiply-accumulate operation is completed, propagate the result of the multiply-accumulate operation along the diagonal unidirectional data path to propagate the result of the multiply-accumulate operation along the diagonal direction to ((m+1)%M,(n+1)%N), where m is a non-negative integer not greater than M-2.

[0113] Once the calculation is complete, the PE located in row M-1, the last row, will directly output its final calculation result as the array output. The PEs located in other rows will propagate their results to the PEs located below the diagonal at ((m+1)%M, (n+1)%N), passing them down level by level, and finally outputting them from the last row as well.

[0114] As can be seen, the diagonal unidirectional data path of the systolic array 900 eliminates the need for additional timing adjustments during output data processing, thus completely avoiding the use of a triangular register set at the output. For neural networks, this means that the computed output feature map (ofmap) can be ejected from the array with minimal latency, allowing the next neural network layer in the pipeline to begin processing more quickly, thereby improving the inference speed of the entire network.

[0115] Figure 10 A schematic diagram of the processing unit according to an embodiment of this disclosure is shown. The PE, as the basic building block of the pulsating array, has an ingenious internal structural design that is the core for realizing the functions of non-triangulated data flow, output dwell calculation, and bubble-free continuous processing described in this invention. Its operation is described in detail below: (Refer to...) Figure 10The PE receives external control signals through its configuration interface cfg and can dynamically switch between at least three main operating modes to perform different tasks.

[0116] 1. Calculation and Accumulation Mode This is the primary operating mode of the PE when performing matrix multiplication. In this mode, the first input matrix data (ifmap) and the second input matrix data (weight) are input from the act_in and weight_in interfaces, respectively. Simultaneously, the PE also passes these two data sets to downstream PEs via the act_out and weight_out interfaces to maintain the data pulsation of the entire array.

[0117] Furthermore, the ifmap and weight data input to the PE are fed into the multiplier. Under the control of the cfg signal, the demux multiplexer routes the multiplier's output to the adder's input, accumulating it with the partial sum (psum) currently stored internally in the PE (e.g., in a psum register not shown). The updated psum is then written back, thus achieving "residency" of the result. This process repeats every clock cycle.

[0118] 2. Result Output Mode Once all multiplication and accumulation operations on a matrix are completed, the PE (Predicted Calculation) needs to output the final result. Understandably, the PE will not perform any further multiplication and accumulation.

[0119] As the cfg signal changes, PE switches to output mode. At this point, demux retrieves the final result (ofmap) from the accumulation loop and latches it into the pipe register. Subsequently, the mux multiplexer is configured to select the path from the pipe register and send this final result out through the mac_out interface.

[0120] 3. Data Straight-Through Mode During the array's output phase, a PE may have already sent its own computation results, but it is still a necessary path for the output results of other "upstream" PEs. The ofmap data from the previous PE is input through the mac_in interface. The cfg signal configures the mux multiplexer to select the path from mac_in, allowing the input ofmap data to directly "pass through" that PE and be output from the mac_out interface, thus being passed to the next PE. This forms the physical basis for the final result's diagonal unidirectional propagation within the array.

[0121] Therefore, this PE internal structure design, which includes a pipe register, mux, and demux, achieves the following key technical effects through its high flexibility and configurability: First, precise control of the mux and demux via the cfg signal allows the PE to clearly distinguish between computation, output, and pass-through states. In particular, the mux's pass-through function is crucial for constructing a diagonal unidirectional output data path, thus avoiding the use of the output triangular register group, directly saving hardware area and reducing output latency. Second, it achieves bubble-free continuous computation, maximizing throughput. The pipe register design is a major highlight of this solution. Its core technical effect lies in decoupling the actions of "computation completion" and "result output" in both time and physical links. When a matrix computation (e.g., A0xB0) is completed, its result can be immediately latched into the pipe register for "temporary storage." This allows the core multiply-accumulate unit (MAC) to be released instantly and immediately put into the next matrix computation (A1xB1), without waiting for the result of A0xB0 to be completely removed from the matrix. In this way, the computing pipeline will not have any idle cycles ("bubbles"), achieving near 100% hardware utilization.

[0122] In neural network applications, a model contains hundreds or thousands of layers that need to be computed sequentially. This bubble-free computing capability means that there is almost no time overhead in handling the switching between layers, which can greatly improve the end-to-end inference speed and data throughput of the entire network. It has a decisive performance advantage for scenarios that require high throughput and low latency, such as data centers and autonomous driving.

[0123] In particular, matrix multiplication is the core and most computationally expensive operation in advanced neural network architectures such as large language models and Transformers. The self-attention mechanism and feed-forward networks in Transformer models heavily rely on large-scale matrix multiplication. These models typically contain hundreds or even trillions of parameters, resulting in astronomical matrix operations for a single inference or training task. The systolic array disclosed herein, with its bubble-free continuous computation and extremely low computational latency, can directly and efficiently accelerate the computation process of these complex models, significantly shortening the model training cycle and improving inference throughput. This has irreplaceable significance and performance improvement value for promoting the application of large language models in natural language processing, code generation, multimodal AI, and meeting future AI computing power demands.

[0124] According to another aspect of this disclosure, a data processing apparatus for matrix calculation is also provided.

[0125] like Figure 11As shown, the data processing apparatus 1100 for matrix calculation includes: a reading unit 1101 configured to read data from a first input matrix and a second input matrix, wherein the first input matrix has M rows and K columns, and the second input matrix has K rows and N columns, and wherein M, K, and N are all positive integers; a first propagation unit 1102 configured to input the read data of the first input matrix into a pulsating array and to propagate the data of the first input matrix laterally in the pulsating array, wherein the pulsating array is composed of M rows and N columns of processing units; and a second propagation unit 1103 configured to input the read data from the first input matrix into a pulsating array. The data of the second input matrix is ​​input into the pulsating array, and the data of the second input matrix is ​​cyclically propagated in the pulsating array along the diagonal direction; the calculation unit 1104 is configured to perform a multiply-accumulate operation on the data of the first input matrix and the data of the second input matrix flowing through it in each processing unit of the pulsating array, and to retain the intermediate result of the calculation in the processing unit; and the third propagation unit 1105 is configured to, in response to determining that the multiply-accumulate operation is completed, propagate the final calculation result retained in each processing unit unidirectionally along the diagonal direction to remove the final calculation result from the pulsating array.

[0126] For example, the first input matrix can be the aforementioned 3x3 matrix A, and the second input matrix can be the aforementioned 3x3 matrix B. The data processing device 1100 for matrix calculation is used for the multiplication calculation of matrix A and matrix B. For ease of description, the technical solution of this disclosure will be described below with matrix A as the first input matrix, matrix B as the second input matrix, and matrix C as the calculation result of the multiplication of matrix A and matrix B. It is understood that the data processing device 1100 for matrix calculation can process matrix A with dimension M. K, K In matrix multiplication calculations of N, M, K, and N can be positive integers set according to needs and application scenarios. They do not need to be equal or limited to the 3x3 dimensions shown in matrices A and B.

[0127] The reading unit 1101 reads the two input matrices. For example, the data of the two input matrices can be read from memory respectively, so that the first propagation unit 1102 and the second propagation unit 1103 respectively input the read data of the two input matrices into the pulsation array to perform matrix multiplication calculation.

[0128] The first propagation unit 1102 causes the data of the first input matrix to propagate horizontally within the systolic array, and the second propagation unit 1103 causes the data of the second input matrix to propagate diagonally within the systolic array. This ensures that the data of the first data matrix is ​​multiplied by the corresponding data of the second input matrix at the correct processing unit within the systolic array during the correct clock cycle, thus performing the multiplication calculation of the two matrices in an orderly manner. After the multiplication and accumulation operation of the data from the two input matrices is completed, the third propagation unit 1105 causes the calculation result to propagate unidirectionally along the diagonal direction to remove the final calculation result from the systolic array to obtain the output matrix.

[0129] Therefore, by employing a novel, parallel system of horizontal loops, diagonal loops, and diagonal unidirectional propagation paths, the design of traditional OS systolic arrays, which rely on triangular data structures, is overturned. This fundamentally eliminates the hardware dependency on large-scale triangular register sets, thus providing a foundation for reducing hardware costs, power consumption, and computational latency. In neural network applications, this means that AI accelerator chips can integrate larger-scale computing arrays or achieve lower power consumption within the same area.

[0130] It is understood that the components 1101-1105 of the data processing device 1100 for matrix calculation are used to execute steps S201-S205 of method 200, as well as other method steps defined by method 200, to realize the multiplication operation of two matrices, which will not be elaborated here.

[0131] According to another aspect of this disclosure, an electronic device is also provided, including at least one processor, wherein each of the at least one processor includes: a processor cache; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the methods described above.

[0132] According to another aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to cause the computer to perform the methods described above.

[0133] According to another aspect of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements the above-described method.

[0134] See Figure 12Electronic device 1200 will now be described as an example of a hardware device (electronic device) that can be applied to various aspects of this disclosure. Electronic device 1200 can be any machine configured to perform processing and / or calculations, and can be, but is not limited to, a workstation, server, desktop computer, laptop computer, tablet computer, personal digital assistant, robot, smartphone, in-vehicle computer, or any combination thereof. The methods described above can be implemented wholly or at least partially by electronic device 1200 or similar devices or systems.

[0135] Electronic device 1200 may include elements that are connected to or communicate with bus 1202 (possibly via one or more interfaces). For example, electronic device 1200 may include bus 1202, one or more processors 1204, one or more input devices 1206, and one or more output devices 1208. The one or more processors 1204 may be any type of processor and may include, but are not limited to, one or more general-purpose processors and / or one or more dedicated processors (e.g., special-purpose chips). Input devices 1206 may be any type of device capable of inputting information to electronic device 1200 and may include, but are not limited to, a mouse, keyboard, touchscreen, microphone, and / or remote control. Output devices 1208 may be any type of device capable of presenting information and may include, but are not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Electronic device 1200 may also include a non-transitory storage device 1210. The non-transitory storage device can be any storage device that is non-transitory and capable of storing data, including but not limited to disk drives, optical storage devices, solid-state storage, floppy disks, flexible disks, hard disks, magnetic tapes or any other magnetic media, optical discs or any other optical media, ROM (read-only memory), RAM (random access memory), cache memory and / or any other memory chip or cartridge, and / or any other medium from which a computer can read data, instructions, and / or code. The non-transitory storage device 1210 can be detached from an interface. The non-transitory storage device 1210 may have data / programs (including instructions) / code for implementing the methods and steps described above. Electronic device 1200 may also include a communication device 1212. The communication device 1212 can be any type of device or system that enables communication with external devices and / or with a network, and may include, but is not limited to, modems, network interface cards, infrared communication devices, wireless communication devices and / or chipsets, such as Bluetooth™ devices, 802.11 devices, Wi-Fi devices, Wi-Max devices, cellular communication devices, and / or the like.

[0136] Electronic device 1200 may also include working memory 1214, which may be any type of working memory that can store programs (including instructions) and / or data useful to the operation of processor 1204, and may include, but is not limited to, random access memory and / or read-only memory devices.

[0137] Software elements (programs) may reside in working memory 1214, including but not limited to operating system 1216, one or more application programs 1218, drivers, and / or other data and code. Instructions for performing the methods and steps described above may be included in one or more application programs 1218, and the method 200 described above can be implemented by processor 1204 reading and executing the instructions of one or more application programs 1218. More specifically, each step in the method 200 described above can be implemented, for example, by processor 1204 executing an application program 1218 having the instructions for the steps described above. Furthermore, other steps in the method described above can be implemented, for example, by processor 1204 executing an application program 1218 having the instructions for performing the corresponding steps. Executable code or source code of the instructions of the software elements (programs) may be stored in a non-transitory computer-readable storage medium (e.g., storage device 1210 described above) and may be stored in working memory 1214 during execution (possibly compiled and / or installed). Executable code or source code of the instructions of the software elements (programs) may also be downloaded from a remote location.

[0138] It should also be understood that various modifications can be made depending on specific requirements. For example, custom hardware can also be used, and / or specific elements can be implemented using hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. For example, some or all of the disclosed methods and apparatus can be implemented by programming hardware (e.g., programmable logic circuits including field-programmable gate arrays (FPGAs) and / or programmable logic arrays (PLAs)) using logic and algorithms according to this disclosure in assembly language or hardware programming languages ​​(such as Verilog, VHDL, C++).

[0139] It should also be understood that the aforementioned methods can be implemented using a server-client model. For example, the client can receive user input data and send it to the server. Alternatively, the client can receive user input data, perform a portion of the processing described in the aforementioned methods, and send the resulting data to the server. The server can receive data from the client, execute the aforementioned methods or a portion thereof, and return the execution result to the client. The client can receive the execution result from the server and, for example, present it to the user via an output device.

[0140] It should also be understood that the components of electronic device 1200 can be distributed across a network. For example, some processing can be performed by one processor, while other processing can be performed by another processor located far away from that processor. Other components of computing system 1200 can also be distributed similarly. In this way, electronic device 1200 can be interpreted as a distributed computing system that performs processing in multiple locations.

[0141] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A data processing method for matrix calculations, the method comprising: Read data from the first input matrix and the second input matrix, wherein the first input matrix has M rows and K columns, the second input matrix has K rows and N columns, and M, K and N are all positive integers; The data of the first input matrix is ​​read and input into the pulsation array, and the data of the first input matrix is ​​made to circulate horizontally within the pulsation array, wherein the pulsation array is composed of M rows and N columns of processing units; The data of the second input matrix is ​​read and input into the pulsation array, and the data of the second input matrix is ​​made to propagate cyclically along the diagonal direction within the pulsation array; Within each processing unit of the pulsating array, a multiply-accumulate operation is performed on the data flowing through the first input matrix and the data of the second input matrix, and the intermediate results of the calculation are stored in that processing unit; and In response to determining that the multiply-accumulate operation is complete, the final calculation result residing in each processing unit is propagated unidirectionally along the diagonal direction to remove the final calculation result from the pulsating array. The process from the start of reading the first and second input matrices to shifting the final calculation result from each processing unit out of the systolic array requires a third and more clock cycles, and the step of laterally circulating the data of the first input matrix within the systolic array includes: In each of the third plurality of clock cycles, in response to determining that the processing unit located at (x, y) contains data with the first input matrix, the data is propagated right to (x, y+1), where x is a non-negative integer not greater than M-1 and y is a non-negative integer not greater than N-2; and In response to determining that the processing unit located at (o, N-1) contains data with the first input matrix, the data is propagated to (o, 0), where o is a positive integer not greater than M-1.

2. The method of claim 1, wherein, The first input matrix is ​​read in column-major order, and the second input matrix is ​​read diagonally. The first input matrix is ​​read within a first plurality of clock cycles, and the second input matrix is ​​read within a second plurality of clock cycles. The first plurality of clock cycles and the second plurality of clock cycles start simultaneously. Reading data from the first and second input matrices includes: In each of the first plurality of clock cycles, read the data column of the first input matrix corresponding to that clock cycle; and In each of the second plurality of clock cycles, data is read from one or more diagonals of the second input matrix, wherein the one or more diagonals of the second input matrix are determined based on the number of columns K of the first input matrix, the number of columns N of the second input matrix, and the index of the clock cycle.

3. The method of claim 2, wherein the one or more diagonals of the second input matrix are determined in the following manner: In clock cycle i, read the i-th and i+max(N,K)-th diagonal lines; in clock cycle j, read the j-th diagonal line, where... i is a non-negative integer not greater than min(N, K)-2, j is an integer not less than min(N, K)-1 and not greater than max(N, K)-1, and the diagonal direction is from the upper right to the lower left of the second input matrix, and the diagonal of the second input matrix is ​​numbered from the upper left to the lower right.

4. The method of claim 2, wherein, The step of inputting the data of the first input matrix read into the pulsation array includes: In each of the first plurality of clock cycles, M data points from the data column of the first input matrix are respectively input into the M processing units of the 0th column of the pulsating array.

5. The method of claim 2, wherein, The step of inputting the data from the second input matrix into the pulsating array includes: In each of the second plurality of clock cycles, multiple data points on one or more diagonals of the second input matrix are respectively input into multiple processing units in the 0th row of the pulsating array.

6. The method according to any one of claims 1-5, wherein, From the start of reading the first input matrix and the second input matrix to shifting the final calculation result from each processing unit out of the systolic array, a third and more clock cycles are required, wherein the step of causing the data of the second input matrix to circulate diagonally within the systolic array includes: In each of the third plurality of clock cycles, except for the processing unit located at (M-1, N-1), in response to determining that the processing unit located at (p, q) has data with the second input matrix, the data is propagated diagonally to ((p+1)%M,(q+1)%N), where p is a non-negative integer not greater than M-1 and q is a non-negative integer not greater than N-1.

7. The method according to any one of claims 1-5, wherein, The step of responding to determining that the multiply-accumulate operation is complete by propagating the final calculation result residing in each processing unit unidirectionally along the diagonal direction to remove the final calculation result from the pulsating array includes: In response to determining that the multiply-accumulate operation in the processing unit located at (M-1, n) is complete, the final calculation result residing in the processing unit at (M-1, n) is removed from the systolic array, where n is a non-negative integer not greater than N-1; and In response to the completion of the multiply-accumulate operation in the processing unit located at (m, n), the final calculation result residing in the processing unit at (m, n) is propagated diagonally to ((m+1)%M,(n+1)%N), where m is a non-negative integer not greater than M-2.

8. The method according to any one of claims 1-5, further comprising: After removing the final calculation results residing in each processing unit from the pulsating array, column-to-column data rotation is performed on the output final calculation results to restore the column order of the output matrix data.

9. A systolic array for matrix computation, the systolic array comprising M rows and N columns of processing units, where M and N are both positive integers, wherein... Each processing unit includes: The first data input interface and the first data output interface are used to receive and send data from the first input matrix, respectively. The second data input interface and the second data output interface are used to receive and send data from the second input matrix, respectively. The multiply-accumulate unit is used to perform multiply-accumulate operations on the data of the first input matrix and the data of the second input matrix. The third data input interface and the third data output interface are used to receive and send the results of the multiplication and accumulation operation, respectively. The ports of the M rows and N columns of processing units are configured to form within the pulsating array: A horizontal circulating data path is formed by connecting the first data input interface and the first data output interface of the processing units in the same row of the pulsating array, and is used to realize the horizontal circulating propagation of the data of the first input matrix in the pulsating array; A diagonal cyclic data path, formed by diagonally connecting the second data input interface and the second data output interface of the processing unit located on the diagonal, is used to realize the diagonal cyclic propagation of data from the second input matrix; and A diagonal unidirectional data path, formed by diagonally connecting the third data input interface and the third data output interface of the processing unit located on the diagonal, is used to remove the final calculation result of the multiply-accumulate operation from the pulsation array. The pulsating array is configured to receive the first input matrix and the second input matrix and shift out the final calculation result within a third and a half clock cycles, and wherein... The processing unit located at (x, y) is further configured to: in each of the third plurality of clock cycles, in response to determining data having the first input matrix, propagate the data along the horizontal cyclic data path to propagate the data to the right to (x, y+1), where x is a non-negative integer not greater than M-1 and y is a non-negative integer not greater than N-2; and The processing unit located at (o, N-1) is further configured to: in each of the third plurality of clock cycles, in response to determining data having the first input matrix, propagate the data along the transverse cyclic data path to (o, 0), where o is a positive integer not greater than M-1.

10. The pulsating array as claimed in claim 9, wherein, The systolic array is further configured to complete the reception of the first input matrix within a first plurality of clock cycles, and to complete the reception of the second input matrix within a second plurality of clock cycles. The systolic array is also configured to simultaneously begin receiving data from both the first and second input matrices. Furthermore, the systolic array is further configured to: In each of the first plurality of clock cycles, a data column corresponding to that clock cycle of the first input matrix is ​​received; and In each of the second plurality of clock cycles, data from one or more diagonals of the second input matrix is ​​received, wherein the one or more diagonals of the second input matrix are determined based on the number of columns K of the first input matrix, the number of columns N of the second input matrix, and the index of the clock cycle.

11. The pulsating array as claimed in claim 10, wherein, The first input matrix has M rows and K columns, where K is a positive integer, and wherein the first data input interface of the M processing units in the 0th column of the systolic array is configured to receive M data from the data column of the first input matrix read from memory in each of the first plurality of clock cycles.

12. The pulsating array as claimed in claim 10, wherein, The second input matrix has K rows and N columns, where K is a positive integer, and wherein the second data input interface of the N processing units in the 0th row of the systolic array is configured to receive multiple data from one or more diagonals of the second input matrix read from memory in each of the second plurality of clock cycles.

13. The pulsating array as described in any one of claims 9-12, wherein, The pulsating array is configured to receive the first input matrix and the second input matrix and shift out the final calculation result within a third and a half clock cycles, wherein, In addition to the processing unit located at (M-1, N-1), the processing unit located at (p, q) is further configured to: in each of the third plurality of clock cycles, in response to determining data having the second input matrix, propagate the data along the diagonal cyclic data path to propagate the data diagonally to ((p+1)%M,(q+1)%N), where p is a non-negative integer not greater than M-1 and q is a non-negative integer not greater than N-1.

14. The pulsating array as claimed in any one of claims 9-12, wherein, The processing unit located at (M-1, n) is further configured to: in response to determining that the multiply-accumulate operation is complete, shift the result of the multiply-accumulate operation out of the systolic array, where n is a non-negative integer not greater than N-1; and The processing unit located at (m, n) is further configured to: in response to determining that the multiply-accumulate operation is completed, propagate the result of the multiply-accumulate operation along the diagonal unidirectional data path to propagate the result of the multiply-accumulate operation along the diagonal direction to ((m+1)%M, (n+1)%N), where m is a non-negative integer not greater than M-2.

15. A data processing apparatus for matrix calculations, the apparatus comprising: The reading unit is configured to read data from a first input matrix and a second input matrix, wherein the first input matrix has M rows and K columns, the second input matrix has K rows and N columns, and M, K, and N are all positive integers; The first propagation unit is configured to input the data of the first input matrix read into the pulsation array and to propagate the data of the first input matrix laterally in the pulsation array, wherein the pulsation array is composed of M rows and N columns of processing units; The second propagation unit is configured to input the data of the second input matrix read into the pulsation array, and to cause the data of the second input matrix to propagate cyclically in the pulsation array along the diagonal direction. A computational unit is configured to perform a multiply-accumulate operation on the data flowing through the first input matrix and the data of the second input matrix within each processing unit of the pulsating array, and to store the intermediate results of the computation in that processing unit; and The third propagation unit is configured to, in response to determining that the multiply-accumulate operation is complete, propagate the final calculation result residing in each processing unit unidirectionally along the diagonal direction to remove the final calculation result from the pulsating array. The process from the start of reading the first and second input matrices to shifting the final calculation result from each processing unit out of the systolic array requires a third and more clock cycles, and the first propagation unit is further configured as follows: In each of the third plurality of clock cycles, in response to determining that the processing unit located at (x, y) contains data with the first input matrix, the data is propagated right to (x, y+1), where x is a non-negative integer not greater than M-1 and y is a non-negative integer not greater than N-2; and In response to determining that the processing unit located at (o, N-1) contains data with the first input matrix, the data is propagated to (o, 0), where o is a positive integer not greater than M-1.

16. An electronic device comprising: At least one processor, wherein each of the at least one processor comprises: Processor cache; and A memory communicatively connected to the at least one processor, wherein The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

17. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

18. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the method of any one of claims 1-8.

Images