Prefix and computation apparatus, method, chip, board and electronic device

Abstract

The application discloses a prefix sum and calculation device, method, chip, board card and electronic equipment, and relates to the technical field of prefix sum and calculation, in particular to a prefix sum and calculation device, method, chip, board card and electronic equipment.The device comprises a data segmentation unit, which is used for grouping vector data to be processed according to a preset segmentation granularity, splicing each vector obtained by grouping in sequence to form a matrix to be processed; a matrix operation unit, which is used for batch calculating prefix sums of each row data in the matrix to be processed to obtain an output matrix; a vector operation unit, which is used for performing addition operation on each row data in the output matrix and a corresponding scalar value to obtain a result vector corresponding to each row data, wherein the corresponding scalar value is equal to the accumulated value of the last data of each row before the row; and a data splicing unit, which is used for splicing each result vector in sequence to obtain a prefix sum vector corresponding to the vector data to be processed.The application can fully utilize the matrix operation resources and vector operation resources inside a chip, improve the overall processing efficiency of prefix sum calculation while ensuring the calculation accuracy.

Description

Technical Field

[0001] This invention relates to the field of chips, and more specifically, to a prefix sum computing device, method, chip, board, and electronic device. Background Technology

[0002] Prefix sums are an efficient technique in computer science for calculating the cumulative sum of sequences. The core idea is to preprocess an array so that the cumulative sum of any interval can be retrieved in O(1) time, making it suitable for large-scale vocabulary searches. In large models (such as Transformers), prefix sums are widely used in probability distribution calculations, sampling strategies (such as top-p), and efficiency optimization, playing a crucial role, especially in the decoding stage of text generation. In tasks such as natural language generation, dialogue systems, and code generation, prefix sums have become a core technology for improving the efficiency and controllability of model generation by accelerating probability filtering, optimizing path evaluation, and supporting parallel computing.

[0003] For arrays:

[0004]

[0005] Its prefix sum array S is defined as:

[0006]

[0007] like Figure 2 The example shown is for calculating the prefix sum:

[0008] S[1] = A[1] = 1;

[0009] S[2] = A[1] + A[2] = 3;

[0010] S[3]= A[1] + A[2] + A[3] = 6;

[0011] …

[0012] S[8]= A[1] + A[2] + A[3] + … +A[8] = 36.

[0013] Existing technologies typically use vector computation (such as SIMD, CUDA cores) to perform prefix sum calculations (Single-pass Parallel Prefix Scan with Decoupled Look-back). In artificial intelligence (AI) chip architectures, matrix operations and vector / scalar operations are usually implemented using different hardware units. Matrix operations typically use dedicated hardware (such as Tensor Cores) to efficiently perform matrix multiplication and accumulation operations, resulting in high data parallelism efficiency. Vector operations are usually performed using SIMD or CUDA cores, which have low data parallelism efficiency. Furthermore, since matrix operations account for a large proportion of neural network models, while vector / scalar operations account for a relatively small proportion, the computational power ratio between matrix operations and vector / scalar operations differs significantly in AI chips, typically by tens to thousands of times. Therefore, using only vector operations to perform prefix sum calculations results in low computational efficiency.

[0014] Therefore, how to make full use of the hardware resources on the chip (matrix operation resources and vector operation resources) to effectively accelerate the calculation process of prefix sums and improve the efficiency of prefix sum calculation is a technical problem that needs to be solved urgently by existing technologies. Summary of the Invention

[0015] In order to solve at least one of the technical problems in the background art described above, the present invention proposes a prefix sum computing device, method, chip, board, and electronic device.

[0016] In one aspect, a prefix sum calculation device is provided, the device comprising:

[0017] The data segmentation unit is used to group the vector data to be processed according to a preset segmentation granularity, and then concatenate the grouped vectors in sequence to form the matrix to be processed.

[0018] A matrix operation unit is used to batch calculate the prefix sum of each row of data in the matrix to be processed to obtain an output matrix, wherein each row of data in the output matrix is ​​the prefix sum of the corresponding row of data in the matrix to be processed;

[0019] The vector operation unit is used to perform addition operations on each row of data in the output matrix with the corresponding scalar value to obtain the result vector corresponding to each row of data, wherein the corresponding scalar value is equal to the sum of the last data of each row before that row;

[0020] The data splicing unit is used to splice the result vectors in sequence to obtain the prefix sum vector corresponding to the vector data to be processed.

[0021] Optionally, the matrix operation unit is specifically used to perform matrix multiplication on the matrix to be processed and the upper triangular matrix to obtain the output matrix, wherein the diagonal and upper elements of the upper triangular matrix are all 1.

[0022] Optionally, the prefix sum calculation device further includes:

[0023] A memory for storing the output matrix;

[0024] A vector register is used to store the data of the rows to be calculated in the output matrix during vector computation;

[0025] A scalar register is used to store the scalar value corresponding to the row to be calculated during vector computation;

[0026] The controller is used to control the vector operation unit to add the data in the vector register to the data in the scalar register during vector computation, so as to generate the corresponding result vector.

[0027] Optionally, the controller is specifically configured to read the output matrix from the memory, and sequentially load the current row data into the vector register for each row of data in the output matrix except for the first row; control the vector operation unit to add each data in the vector register to the last data of the result vector corresponding to the previous row data stored in the scalar register to obtain the result vector corresponding to the current row data; write the result vector corresponding to the current row data into the memory; if the current row data is the last row in the output matrix, then all vector calculations are completed; otherwise, the last data in the result vector corresponding to the current row data is updated to the scalar register for use in the next row operation.

[0028] Optionally, the controller is further configured to, for the first row of data in the output matrix, take that row of data as a result vector and update the last data of that row to the scalar register.

[0029] Optionally, the controller is specifically configured to write the result vector corresponding to the current row of data into the memory, replacing the original data corresponding to that row in the output matrix.

[0030] Optionally, the data splicing unit is specifically used to obtain the result vectors corresponding to each row of data in the output matrix from the memory, and splice the result vectors in row order to obtain the prefix sum vector corresponding to the vector data to be processed.

[0031] Optionally, the matrix operation unit adopts a pulsating array, and the weight matrix loaded in the pulsating array is the upper triangular matrix. The pulsating array is used to perform matrix multiplication on the matrix to be processed and the weight matrix to generate the output matrix.

[0032] Optionally, the granularity of the segmentation is set according to the parallel processing scale of the matrix operation unit.

[0033] In another aspect, the present invention also provides a method for calculating prefix sums, the method comprising:

[0034] The data segmentation unit groups the vector data to be processed according to a preset segmentation granularity, and the grouped vectors are then concatenated in sequence to form the matrix to be processed.

[0035] The prefix sum of each row of data in the matrix to be processed is calculated in batches by the matrix operation unit to obtain the output matrix, wherein each row of data in the output matrix is ​​the prefix sum of the corresponding row of data in the matrix to be processed;

[0036] The vector operation unit performs addition operations on each row of data in the output matrix with the corresponding scalar value to obtain the result vector corresponding to each row of data, wherein the corresponding scalar value is equal to the cumulative value of the last data of each row before that row;

[0037] The data splicing unit sequentially splices the result vectors to obtain the prefix sum vector corresponding to the vector data to be processed.

[0038] In another aspect, the present invention provides a chip including the aforementioned prefix and computing device.

[0039] In another aspect, the present invention provides a board card comprising the aforementioned chip.

[0040] In another aspect, the present invention provides an electronic device comprising the aforementioned chip.

[0041] The beneficial effects of this invention are as follows:

[0042] This invention utilizes a data segmentation unit to group the vector data to be processed according to a preset segmentation granularity, and then sequentially concatenates the grouped vectors to form a matrix to be processed. This allows the vector data to be input into the hardware computing unit in matrix form for parallel processing. A matrix operation unit performs batch calculations of the prefix sums of each row of data in the matrix to be processed, obtaining an output matrix. This leverages the high parallel computing capability of the matrix operation unit to achieve parallel prefix sum calculations for multiple rows of data. A vector operation unit performs addition operations on each row of data in the output matrix with its corresponding scalar value, obtaining the result vector for each row, thus achieving continuous concatenation of prefix sum results between different rows. Finally, a data concatenation unit sequentially concatenates the result vectors to obtain the prefix sum vector corresponding to the vector data to be processed. Therefore, this invention can allocate prefix sum calculation tasks between the matrix operation unit and the vector operation unit, fully utilizing the internal matrix and vector operation resources of the chip, and improving the overall processing efficiency of prefix sum calculation while ensuring calculation accuracy. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0044] Figure 1 This is a schematic diagram of the prefix sum calculation device according to an embodiment of the present invention;

[0045] Figure 2 This is a diagram illustrating prefix sum calculation;

[0046] Figure 3 This is a schematic diagram of the overall process for calculating the prefix sum in an embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram of the matrix operation unit calculation in an embodiment of the present invention;

[0048] Figure 5 This is a schematic diagram of the vector operation unit calculation in an embodiment of the present invention;

[0049] Figure 6 This is a schematic diagram illustrating the vector operation processing implemented in an embodiment of the present invention;

[0050] Figure 7 This is a flowchart of vector operation processing according to an embodiment of the present invention;

[0051] Figure 8 This is a schematic diagram of the hardware data flow process for prefix sum calculation based on a vector operation unit according to an embodiment of the present invention;

[0052] Figure 9 This is a first schematic diagram illustrating a calculation example of the present invention;

[0053] Figure 10 This is a second schematic diagram illustrating a calculation example of the present invention;

[0054] Figure 11 This is the third schematic diagram of the calculation example of the present invention. Detailed Implementation

[0055] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0056] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0057] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products or devices.

[0058] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0059] This invention transforms prefix sum calculation into a combination of matrix operations (intra-group prefix sum calculation) and vector operations (inter-group addition). This transformation fully utilizes the hardware resources on the chip (matrix calculation resources and vector calculation resources), effectively accelerates the prefix sum calculation process, and improves the execution efficiency of the prefix sum operator.

[0060] Figure 1This is a schematic diagram of the prefix sum calculation device according to an embodiment of the present invention, as shown below. Figure 1 As shown, in one embodiment of the present invention, the prefix sum calculation device of the present invention includes:

[0061] The data segmentation unit is used to group the vector data to be processed according to a preset segmentation granularity, and then concatenate the grouped vectors in sequence to form the matrix to be processed.

[0062] In one embodiment of the present invention, the data segmentation unit may include a data scheduling logic circuit or an input buffer control module, used to receive vector data to be processed during the data loading stage, and to group the input data according to a preset segmentation granularity, reorganizing the grouped data into a matrix form. This matrix can be stored in an on-chip cache or local memory for subsequent matrix operation units to read, realizing parallel loading of data at the matrix level.

[0063] The matrix operation unit is used to batch calculate the prefix sum of each row of data in the matrix to be processed to obtain an output matrix, wherein each row of data in the output matrix is ​​the prefix sum of the corresponding row of data in the matrix to be processed.

[0064] In one embodiment of the present invention, the matrix operation unit can be implemented using a systolic array or other parallel matrix multiplication and addition structure, and is used to perform prefix sum calculations on each row of data in the matrix to be processed in matrix-level parallel mode. This operation unit can load an upper triangular weight matrix into the array, and obtain the local prefix sum result of each row at once through matrix multiplication operations with fixed weights, significantly improving the parallel computation capability of prefix sums.

[0065] The vector operation unit is used to perform addition operations on each row of data in the output matrix with the corresponding scalar value to obtain the result vector corresponding to each row of data, wherein the corresponding scalar value is equal to the cumulative value of the last data of each row preceding that row.

[0066] In one embodiment of the present invention, the vector operation unit may employ SIMD-structured vector addition logic to perform cross-row accumulation operations on the results output by the matrix operation unit. Specifically, the vector operation unit reads the scalar value corresponding to the current row (i.e., the accumulated value of the last data of the previous row result) from the scalar register according to the control signal, and performs an addition operation with each element of the current row to generate the result vector corresponding to the current row.

[0067] The data splicing unit is used to splice the result vectors in sequence to obtain the prefix sum vector corresponding to the vector data to be processed.

[0068] The data concatenation unit, which can be implemented by an output buffer or a result reassembly module, is used to concatenate the result vectors generated by the vector operation unit in row order to form a complete prefix sum vector. The concatenated result data can be directly written to external memory or passed to the next-level calculation module for use.

[0069] Through the above-described structured module division and data flow control, the prefix sum computing device of this invention can achieve parallel and pipelined processing of the prefix sum computing process at the chip hardware level, effectively improving the execution efficiency and hardware resource utilization of prefix sum computing.

[0070] In one embodiment of the present invention, the matrix operation unit is specifically used to perform matrix multiplication on the matrix to be processed and the upper triangular matrix to obtain the output matrix, wherein the diagonal and above elements of the upper triangular matrix are all 1.

[0071] In one embodiment of the present invention, each row of the matrix to be processed corresponds to a group of grouped vector data. The dimension of the upper triangular matrix is ​​the same as the length of each group of vectors, and the element values ​​at and above the diagonal are set to 1, while the element values ​​below the diagonal are set to 0. When the matrix operation unit performs a matrix multiplication operation between the matrix to be processed and the upper triangular matrix, each row of vector data is multiplied and added element-wise with the corresponding weight position in the upper triangular matrix, so that the calculation result of the j-th element of the row is equal to the cumulative value of the first j elements of the original row vector, thereby realizing the prefix sum calculation of each row of data at the matrix level.

[0072] In hardware implementation, the matrix operation unit can be implemented using a systolic array structure or other matrix computation circuits capable of parallel multiplication and addition operations. By loading an upper triangular weight matrix inside the array and keeping the weight matrix unchanged during computation, the high parallelism of matrix multiplication and addition can be utilized to process multiple row vectors simultaneously, thereby significantly improving the overall computational efficiency of prefix sum calculation, reducing computational latency, and increasing the utilization rate of the matrix operation unit. Through this method, the matrix operation unit can achieve efficient prefix sum calculation for vectors of different dimensions without changing the hardware structure, providing basic data support for cross-row accumulation in subsequent vector operation units.

[0073] In one embodiment of the present invention, the matrix operation unit adopts a pulsating array, the weight matrix loaded in the pulsating array is the upper triangular matrix, and the pulsating array is used to perform matrix multiplication operation on the matrix to be processed and the weight matrix to generate the output matrix.

[0074] In one embodiment of the present invention, the pulsating array may include multiple multiply-accumulate processing units arranged in a two-dimensional array. Each processing unit is used to perform multiply-accumulate operations between input data and corresponding weights. Intermediate calculation results are passed sequentially between processing units in the row and column directions, realizing the pipelined flow and accumulation of data in the array. The upper triangular weight matrix is ​​loaded into the pulsating array before the operation begins, wherein the weight values ​​at and above the diagonal are all set to 1, and the weight values ​​at and below the diagonal are set to 0.

[0075] When the data of each row of the matrix to be processed is sequentially input into the systolic array, each processing unit in the array performs element-wise multiplication and addition operations according to the configuration of the upper triangular weight matrix and the matrix multiplication rules, thereby obtaining the prefix sum of each row of data in a single calculation. This result is then processed in parallel within the array, forming an output matrix at the array output. The matrix operation unit can fully utilize the parallel multiplication and addition characteristics of the systolic array at the hardware level to achieve batch calculation of prefix sums for multiple sets of vector data, improving the throughput and computational efficiency of prefix sum calculation.

[0076] In one embodiment of the present invention, the segmentation granularity is set according to the parallel processing scale of the matrix operation unit.

[0077] In one embodiment of the present invention, the matrix operation unit can be implemented using a systolic array structure with a fixed array size. Its parallel processing scale can be determined by the number of rows and columns of the multiply-accumulate processing unit (MAC unit) in the array. For example, when the matrix operation unit is a 4×4 systolic array, it can process the multiply-accumulate operations of 4 rows of input data in parallel within the same clock cycle. Each row contains 4 data units, thereby realizing the synchronous execution of 16 element-level multiply-accumulate operations within the array.

[0078] Therefore, during the data loading and task allocation phase, the granularity of the segmentation can be set to match the parallel processing scale of the matrix operation unit. That is, when the matrix operation unit is an N×N array structure, the granularity of the segmentation can be set to N, so that the number of elements of each group of vector data to be processed corresponds to the parallel processing capability of the systolic array, thereby completing the prefix sum calculation of the entire group of data in one matrix operation, avoiding the idle or repeated use of array resources, and improving computational efficiency and hardware resource utilization.

[0079] In practical applications, the parallel processing scale of the matrix operation unit can vary depending on the chip architecture and application scenario. For example, it can be a 4×4, 8×8, 16×16, or higher-dimensional systolic array structure. By dynamically setting the partitioning granularity according to the parallel processing scale of the array, an adaptive match between the computational load and the hardware parallelism can be achieved, thereby further improving the overall execution performance of prefix sum calculation.

[0080] Figure 3 This is a schematic diagram of the overall process of prefix sum calculation in an embodiment of the present invention, as shown below. Figure 3 As shown, the input data of the prefix sum calculation device of the present invention is a vector to be processed, which is a vector data containing m elements (such as an array [a1, a2, ..., am]).

[0081] The present invention first determines the segmentation granularity k, which is determined based on the parallel processing capability of the matrix operation unit (such as a systolic array).

[0082] The matrix operation unit in the device of the present invention divides the vector to be processed into groups of k elements, for a total of n = m / k groups (rounded up). Assuming that m is an integer multiple of k, if m is not an integer multiple of k, then 0 is padded to the end of the input vector to make it an integer multiple.

[0083] Then, the n sets of data (each set of k elements) are concatenated row by row to form the matrix A to be processed (n×k, each row corresponds to a set of data).

[0084] Furthermore, this invention pre-establishes an upper triangular matrix U (k×k), where all elements on the diagonal and above of the triangular matrix U are 1. For example, when k=3, the triangular matrix U is:

[0085]

[0086] The upper triangular matrix U is used to simulate the accumulation logic of prefix sums (the prefix sum within the group is calculated in each row).

[0087] like Figure 4 As shown, the matrix operation unit in the device of the present invention specifically performs matrix multiplication operation A×U on the matrix to be processed A (n×k) and the triangular matrix U (k×k) to obtain the output matrix (n×k).

[0088] Each row of the output matrix (n×k) is the prefix sum of each row of the matrix A to be processed (n×k). For example, the first row of the matrix A to be processed is:

[0089]

[0090] The first row of data in the output matrix is:

[0091]

[0092] This invention utilizes the parallelism of matrix operations, allowing n rows of data to be calculated simultaneously, thus improving computational efficiency.

[0093] The output matrix is ​​the prefix sum of each of the n data groups, but to obtain the prefix sum vector corresponding to the vector to be processed, it is also necessary to generate the global prefix sum.

[0094] like Figure 5 As shown, this invention generates a global prefix sum using a vector operation unit.

[0095] The first row (P1 row) of the output matrix is ​​directly used as the local prefix sum (the prefix sum of the first k elements).

[0096] Subsequent lines (P) i For each element in the row (2≤i≤n-1), add the sum of all the preceding rows (i.e., the prefix of the first (i-1)k+j elements), for example:

[0097] The k elements in the second line (line P2) become:

[0098]

[0099] The k elements in the third row (row P3) become:

[0100]

[0101] Until the nth row (P) n The k elements in row ) become:

[0102]

[0103] After the vector operation is completed, the data concatenation unit concatenates the result vectors obtained from the vector operation in row order to obtain the prefix sum vector (m data) corresponding to the vector data to be processed.

[0104] Therefore, the core logic of the prefix sum calculation device of the present invention is as follows:

[0105] Intra-group parallelism: Prefix sums within each group are calculated in batches through matrix operations, accelerating the process using hardware parallelism capabilities (such as Tensor Cores and systolic arrays).

[0106] Inter-group addition: The results of each group are accumulated through vector operation units to ensure the continuity of the global prefix sum (i.e., the prefix sum of the first i×k data is equal to the sum of the first (i-1)k data plus the prefix sum within the i-th group, where 1≤i≤n).

[0107] In one embodiment of the present invention, the prefix sum calculation device further includes:

[0108] A memory for storing the output matrix;

[0109] A vector register is used to store the data of the rows to be calculated in the output matrix during vector computation;

[0110] A scalar register is used to store the scalar value corresponding to the row to be calculated during vector computation;

[0111] The controller is used to control the vector operation unit to add the data in the vector register to the data in the scalar register during vector computation, so as to generate the corresponding result vector.

[0112] In one embodiment of the present invention, the memory is used to store the output matrix output by the matrix operation unit and to provide data to be calculated to the vector register during the vector calculation stage. The vector register is used to store data of the rows to be calculated in the output matrix during vector calculation, so that the vector operation unit can directly read data from the register for parallel addition operations. The scalar register is used to store the scalar value corresponding to the row to be calculated during vector calculation. The scalar value can be obtained by updating the last data of the previous row's result vector, and is used to implement cross-row accumulation operations in the vector calculation of the current row.

[0113] The controller coordinates the operation of the aforementioned modules during the vector computation phase, controlling the vector operation unit to add the data in the vector register to the data in the scalar register to generate the corresponding result vector. After each computation, the controller can also write the result vector back to the memory and update the last data of the result vector to the scalar register so that it can be used again in the next line of data computation.

[0114] With the above settings, the prefix sum calculation device can complete the data connection between matrix operations and vector operations at the hardware level, use registers for high-speed caching and data exchange, realize automated control of cross-row accumulation, and further improve the parallel processing performance and overall operating efficiency of prefix sum calculation.

[0115] In one embodiment of the present invention, the controller is specifically configured to read the output matrix from the memory, and sequentially load the current row data into the vector register for each row of data in the output matrix excluding the first row; control the vector operation unit to add each data in the vector register to the last data of the result vector corresponding to the previous row data stored in the scalar register to obtain the result vector corresponding to the current row data; write the result vector corresponding to the current row data into the memory; if the current row data is the last row in the output matrix, then all vector calculations are completed; otherwise, the last data in the result vector corresponding to the current row data is updated to the scalar register for use in the next row operation.

[0116] In one embodiment of the present invention, the controller is further configured to, for the first row of data in the output matrix, take the row of data as a result vector and update the last data of the row of data to the scalar register.

[0117] In one embodiment of the present invention, the controller is configured to execute cross-row accumulation control logic for the prefix sum result during the vector computation stage. Specifically, the controller is configured to read the output matrix from the memory, and sequentially load the current row data into the vector register for each row of data in the output matrix excluding the first row; control the vector operation unit to perform addition operations on each data stored in the vector register and the last data of the result vector corresponding to the previous row data stored in the scalar register to obtain the result vector corresponding to the current row data; and write the result vector into the memory to update the data content of the corresponding row in the output matrix.

[0118] When the controller determines that the current row of data is the last row in the output matrix, it confirms that all vector calculations are complete. If the current row of data is not the last row, it controls the scalar register to update the last data in the result vector corresponding to the current row of data, for use in the calculation of the next row of vectors. For the first row of data in the output matrix, the data in that row is directly used as the corresponding result vector, and the last data in that row is updated to the scalar register to provide an initial scalar value for the calculation of subsequent rows.

[0119] Through the above process, the controller can realize row-level cyclic control of prefix sum calculation, enabling automatic loading, calculation, updating and write-back operations of each row vector calculation at the hardware level, thereby ensuring the continuity of the cross-row accumulation process and the correctness of the calculation results.

[0120] In one embodiment of the present invention, the controller is specifically used to write the result vector corresponding to the current row of data into the memory, replacing the original data corresponding to that row in the output matrix.

[0121] In this embodiment, after the vector operation unit completes the addition operation on the current row of data, the controller receives the result vector output by the vector operation unit and controls the memory interface to write the result vector back to the memory to update the storage content of the corresponding row in the output matrix. By replacing the original data with the result vector, the prefix sum calculation result can be updated in real time without occupying additional storage space, thereby reducing on-chip storage requirements and simplifying data management logic. This process also ensures that the output matrix maintains data consistency during the calculation process, that is, the storage content of each row reflects the latest calculation result, which facilitates subsequent concatenation operations to directly read the final result data from the memory, achieving efficient data stream transmission and storage access.

[0122] In one embodiment of the present invention, the data splicing unit is specifically used to obtain the result vectors corresponding to each row of data in the output matrix from the memory, and splice each result vector in row order to obtain the prefix sum vector corresponding to the vector data to be processed.

[0123] In this embodiment, the data concatenation unit may include data recombination logic or an output buffer control module, used to sequentially read the result vectors of each row from memory after the prefix sum calculation is completed, and concatenate them on-chip according to row order to generate a complete prefix sum vector. The concatenation operation can be implemented through address control logic, that is, according to the starting address and row number information of the result vector in memory, the data of each row vector is sequentially written into the contiguous storage space of the output buffer. In this way, the data concatenation unit can complete the orderly integration of result data at the hardware level without additional software recombination operations, achieving one-time output of the prefix sum calculation result. This setup not only ensures the consistency of data order but also reduces the overhead of data movement and intermediate caching, thereby improving the overall efficiency of calculation and output.

[0124] Figure 6 This is a schematic diagram illustrating the vector operation processing implemented in an embodiment of the present invention. Figure 6 A schematic diagram illustrating vector operation processing is shown, demonstrating the complete process from instruction fetching, decoding, scheduling to achieving efficient parallel vector computation using vector operation units.

[0125] SRAM: Static Random Access Memory, used to store program instructions and data to meet the data access speed requirements of the system architecture.

[0126] Instruction Cache: A high-speed cache space that caches instructions read from SRAM, reducing the time the processor waits for instructions.

[0127] Instruction fetch cache: Further stores instructions fetched from the instruction cache. During the instruction fetch phase, RISC-V architecture processors use the instruction fetch cache to store instructions in an orderly manner, preparing them for subsequent decoding and execution.

[0128] Instruction decoding: The instruction decoding unit can quickly parse these instructions, rapidly identify the type of operation to be performed and the source of operands, and prepare for subsequent computation and processing.

[0129] Scheduler: The scheduler arranges the execution order of instructions reasonably based on the resource situation (such as register resources, execution unit status, etc.) and instruction characteristics in different application scenarios.

[0130] Vector register set: A register file used to store vector operands. The data bit width of each vector register is equal to the data processing bit width of the vector operation unit.

[0131] Vector operation unit: Used to perform some operations in parallel (such as SIMD operations, which process multiple elements at once, and can perform vector addition, vector multiplication and other operations), improving computational density and efficiency.

[0132] Figure 7 This is a flowchart of vector operation processing according to an embodiment of the present invention. Figure 7 A method for calculating the prefix sum of input data based on vector operation units is described. The calculation result is accumulated step by step through vector and scalar operations. The specific steps and principles are as follows:

[0133] 1. Data loading:

[0134] Read the first row vector P1 and the second row vector P2 from the output matrix in SRAM. Take the last data (element) of vector P1 and load it into scalar register rs1. Load vector P2 into vector register vs1.

[0135] 2. Vector-Scalar Addition:

[0136] The vector-scalar addition instruction is used to control the vector arithmetic unit to add each data (element) in the vector register vs1 to the value in the scalar register rs1, resulting in the vector vs2.

[0137] 3. Result rewrite:

[0138] Write the resulting vector vs2 back into SRAM and save the partial calculation results of the prefix sum.

[0139] 4. Looping and data updating:

[0140] Determine if the vector in vector register vs1 is the last row vector Pn of the output matrix. If it is not the last row vector, read the next row vector from SRAM and load it into vector register vs1.

[0141] Use the assignment instruction to assign the last data in the result vector vs2 to the scalar register rs1. Then repeat the steps of vector-scalar addition, result write-back, and loop check.

[0142] If the last row vector is Pn, the loop ends. At this point, all prefix sums of the input data have been calculated, and the results have been stored in SRAM.

[0143] Figure 8 This is a schematic diagram of the hardware data flow process for prefix sum calculation based on a vector operation unit according to an embodiment of the present invention. Figure 8 The hardware data flow process for prefix sum calculation based on the vector operation unit is demonstrated. The data flow direction is explained as follows:

[0144] The blue arrows represent the flow of data being read from SRAM and loaded into registers. The scalar register rs1 has two main data sources: one is loading the last data (element) of vector P1 directly into rs1 during the initial phase, and the other is assigning the last data (element) of the result vector vs2 to rs1 during subsequent loop phases; data is then read from P2 to Pn and loaded into vs1 sequentially.

[0145] The red arrow indicates the process where, after the vector operation unit performs the operation, the result vector flows from the vector operation unit to vs2, and then the result vector in vs2 is written back to SRAM.

[0146] Figure 9 , Figure 10 and Figure 11 A specific calculation example of the present invention is shown.

[0147] In this computational example, taking the calculation of the prefix sum of three sets of input vectors (i.e., the vectors to be processed) as an example, each set of input vectors contains 16 data points; the matrix operation unit is a 4×4 systolic array, and the systolic array adopts a fixed weight data working mode; each set of vectors is split according to the size of the systolic array (4), and each set of vectors can be split into four 1×4 sub-vectors, which are then concatenated into a 4×4 matrix, such as Figure 9 As shown, the three sets of vectors to be processed can be converted into three sets of 4×4 matrices to be processed.

[0148] Arrange the three sets of 4×4 matrices to be processed sequentially, as follows: Figure 10 As shown, the input activations are sequentially input into the systolic array. At this time, the weight matrix loaded in the systolic array is an upper triangular matrix, and all elements above the diagonal are 1. The matrix multiplication operation between the input activation matrix and the weight matrix is ​​completed through the systolic array, resulting in 12 1×4 result vectors. According to the order of the output data stream, the first 4 vectors are the block prefix sum matrix of input vector 1 (i.e., the output matrix mentioned above), the middle 4 vectors are the block prefix sum matrix of vector 2, and the last 4 vectors correspond to the block prefix sum matrix of vector 3.

[0149] The block prefix sum matrices of the three sets of vectors are fed into the vector operation unit, and the complete prefix sums of the three sets of input vectors are obtained according to the calculation method of the vector operation unit described above. Subsequently, the complete prefix sums of each set of input vectors are concatenated to obtain the prefix sum vectors corresponding to the three sets of input vectors (i.e., the vectors to be processed).

[0150] Another aspect of the present invention provides a prefix sum calculation method. In one embodiment of the present invention, the prefix sum calculation method includes:

[0151] The data segmentation unit groups the vector data to be processed according to a preset segmentation granularity, and the grouped vectors are then concatenated in sequence to form the matrix to be processed.

[0152] The prefix sum of each row of data in the matrix to be processed is calculated in batches by the matrix operation unit to obtain the output matrix, wherein each row of data in the output matrix is ​​the prefix sum of the corresponding row of data in the matrix to be processed;

[0153] The vector operation unit performs addition operations on each row of data in the output matrix with the corresponding scalar value to obtain the result vector corresponding to each row of data, wherein the corresponding scalar value is equal to the cumulative value of the last data of each row before that row;

[0154] The data splicing unit sequentially splices the result vectors to obtain the prefix sum vector corresponding to the vector data to be processed.

[0155] In one embodiment of the present invention, the step of batch calculating the prefix sum of each row of data in the matrix to be processed by the matrix operation unit to obtain the output matrix specifically includes:

[0156] The matrix operation unit performs matrix multiplication on the matrix to be processed and the upper triangular matrix to obtain the output matrix, wherein all elements on the diagonal and above of the upper triangular matrix are 1.

[0157] In one embodiment of the present invention, the prefix sum calculation method of the present invention further includes:

[0158] The output matrix is ​​stored in a memory.

[0159] The data of the rows to be calculated in the output matrix are stored in the vector register during vector computation.

[0160] The scalar value corresponding to the row to be calculated is stored in a scalar register during vector computation.

[0161] When the controller performs vector computation, it controls the vector operation unit to add the data in the vector register to the data in the scalar register to generate the corresponding result vector.

[0162] In one embodiment of the present invention, the step of controlling the vector operation unit to add the data in the vector register to the data in the scalar register to generate a corresponding result vector during vector computation by the controller specifically includes:

[0163] The controller reads the output matrix from the memory, and sequentially loads the data of each row of the output matrix (excluding the first row) into the vector register. The vector operation unit adds each data point in the vector register to the last data point of the result vector corresponding to the previous row stored in the scalar register, obtaining the result vector corresponding to the current row. The result vector corresponding to the current row is written into the memory. If the current row is the last row of the output matrix, all vector calculations are completed; otherwise, the last data point of the result vector corresponding to the current row is updated in the scalar register for use in the next row's operation. For the first row of the output matrix, this row is used as the result vector, and the last data point of this row is updated in the scalar register.

[0164] In one embodiment of the present invention, writing the result vector corresponding to the current row of data into the memory specifically includes:

[0165] Write the result vector corresponding to the current row of data into the memory, replacing the original data corresponding to that row in the output matrix.

[0166] In one embodiment of the present invention, the step of sequentially concatenating the result vectors by the data concatenation unit to obtain the prefix sum vector corresponding to the vector data to be processed specifically includes:

[0167] The data splicing unit retrieves the result vectors corresponding to each row of data in the output matrix from the memory, and splices the result vectors in row order to obtain the prefix sum vector corresponding to the vector data to be processed.

[0168] For specific embodiments of the prefix sum calculation method of the present invention, please refer to the embodiments of the prefix sum calculation device described above, and will not be repeated here.

[0169] In another aspect, the present invention provides a chip including the aforementioned prefix and computing device.

[0170] In one embodiment of the present invention, the chip may be an artificial intelligence chip (AI chip).

[0171] In another aspect, the present invention provides a circuit board including the aforementioned chip.

[0172] In another aspect, the present invention provides an electronic device comprising the aforementioned chip.

[0173] The present invention is applicable to AI chips, artificial intelligence servers, and artificial intelligence server clusters. Using the present invention can effectively improve the execution efficiency of hardware in processing prefixes and tasks. As can be seen from the above embodiments, the present invention has at least the following beneficial effects compared to the prior art:

[0174] 1. Improved computational efficiency: This invention groups the vector data to be processed into a preset granularity and inputs it into the matrix operation unit in matrix form. It utilizes the high parallel computing capability of the matrix operation unit to perform batch calculation of the prefix sum of each row of data, which significantly improves the throughput and execution efficiency of prefix sum calculation.

[0175] 2. Make full use of heterogeneous hardware resources: The present invention adopts a collaborative computing method between matrix operation units and vector operation units, which respectively undertake the tasks of local prefix sum calculation and inter-row accumulation calculation, realizing the division of labor and collaboration between matrix operation units and vector operation units, and giving full play to the hardware performance of different types of computing units within the chip.

[0176] 3. Reduce the usage of low-parallelism operation units: This invention distributes the main computational load to matrix operation units and only calls vector operation units for lightweight operations during the cross-row accumulation stage, thereby reducing the frequency of use of low-parallelism vector operation units and lowering the overall computational latency.

[0177] 4. Improve chip resource utilization: This invention enables the full utilization of idle matrix computing resources in the chip by grouping, matrixing and inter-row concatenation of vector data, achieving load balancing of prefix sum calculation on different types of computing units, and improving the overall computing power utilization of the chip.

[0178] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A prefix sum calculation device, characterized in that, include: The data segmentation unit is used to group the vector data to be processed according to a preset segmentation granularity, and then concatenate the grouped vectors in sequence to form the matrix to be processed. The matrix operation unit is used to perform matrix multiplication on the matrix to be processed and the upper triangular matrix to obtain an output matrix, wherein the diagonal and above elements of the upper triangular matrix are all 1, and each row of data in the output matrix is ​​the prefix sum of the corresponding row of data in the matrix to be processed; The vector operation unit is used to perform addition operations on each row of data in the output matrix with the corresponding scalar value to obtain the result vector corresponding to each row of data, wherein the corresponding scalar value is equal to the cumulative value of the last data of each row before that row; The data splicing unit is used to splice the result vectors in sequence to obtain the prefix sum vector corresponding to the vector data to be processed.

2. The prefix sum calculation device according to claim 1, characterized in that, Also includes: A memory for storing the output matrix; A vector register is used to store the data of the rows to be calculated in the output matrix during vector computation; A scalar register is used to store the scalar value corresponding to the row to be calculated during vector computation; The controller is used to control the vector operation unit to add the data in the vector register to the data in the scalar register during vector computation, so as to generate the corresponding result vector.

3. The prefix sum calculation device according to claim 2, characterized in that, The controller is specifically configured to read the output matrix from the memory, sequentially load the current row data into the vector register for each row of data in the output matrix except for the first row; control the vector operation unit to add each data in the vector register to the last data of the result vector corresponding to the previous row data stored in the scalar register to obtain the result vector corresponding to the current row data; and write the result vector corresponding to the current row data into the memory. If the current row of data is the last row in the output matrix, then all vector calculations are completed; otherwise, the last data in the result vector corresponding to the current row of data is updated to the scalar register for use in the next row of calculations.

4. The prefix sum calculation device according to claim 3, characterized in that, The controller is further configured to, for the first row of data in the output matrix, take the row of data as a result vector and update the last data of the row of data to the scalar register.

5. The prefix sum calculation device according to claim 3, characterized in that, The controller is specifically used to write the result vector corresponding to the current row of data into the memory, replacing the original data corresponding to that row in the output matrix.

6. The prefix sum calculation device according to any one of claims 3 to 5, characterized in that, The data splicing unit is specifically used to obtain the result vectors corresponding to each row of data in the output matrix from the memory, and splice the result vectors in row order to obtain the prefix sum vector corresponding to the vector data to be processed.

7. The prefix sum calculation device according to claim 1, characterized in that, The matrix operation unit uses a pulsating array, and the weight matrix loaded in the pulsating array is the upper triangular matrix. The pulsating array is used to perform matrix multiplication on the matrix to be processed and the weight matrix to generate the output matrix.

8. The prefix sum calculation device according to claim 1, characterized in that, The granularity of the segmentation is set according to the parallel processing scale of the matrix operation unit.

9. A method for calculating prefix sums, characterized in that, include: The data segmentation unit groups the vector data to be processed according to a preset segmentation granularity, and the grouped vectors are then concatenated in sequence to form the matrix to be processed. The matrix operation unit performs matrix multiplication on the matrix to be processed and the upper triangular matrix to obtain an output matrix, wherein all elements on the diagonal and above of the upper triangular matrix are 1, and each row of data in the output matrix is ​​the prefix sum of the corresponding row of data in the matrix to be processed. The vector operation unit performs addition operations on each row of data in the output matrix with the corresponding scalar value to obtain the result vector corresponding to each row of data, wherein the corresponding scalar value is equal to the cumulative value of the last data of each row before that row; The data splicing unit sequentially splices the result vectors to obtain the prefix sum vector corresponding to the vector data to be processed.

10. A chip, characterized in that, The prefix and calculation device includes any one of claims 1 to 8.

11. A circuit board, characterized in that, Includes the chip described in claim 10.

12. An electronic device, characterized in that, Includes the chip described in claim 10.

Images