Prefix and calculation device and method, chip, board card and electronic equipment

Abstract

The invention discloses a prefix and calculation device and method, a chip, a board card and electronic equipment, and the device comprises a data segmentation unit which is used for grouping to-be-processed vector data according to a preset segmentation granularity, and splicing all vectors obtained through grouping in sequence to form a to-be-processed matrix; the matrix operation unit is used for calculating the prefix sum of each row of data in the to-be-processed matrix in batches to obtain an output matrix; the vector operation unit is used for performing additive operation on each row of data in the output matrix and a corresponding scalar value to obtain a result vector corresponding to each row of data, and the corresponding scalar value is equal to an accumulated value of the last data of each row before the row; and the data splicing unit is used for splicing the result vectors in sequence to obtain prefixes and vectors corresponding to the vector data to be processed. According to the method, matrix operation resources and vector operation resources in the chip can be fully utilized, and the overall processing efficiency of prefix and calculation is improved while the calculation accuracy is ensured.

Description

Technical Field

[0001] This invention relates to the field of chips, and more specifically, to a prefix sum computing device, method, chip, board, and electronic device. Background Technology

[0002] Prefix sums are an efficient technique in computer science for calculating the cumulative sum of sequences. The core idea is to preprocess an array so that the cumulative sum of any interval can be retrieved in O(1) time, making it suitable for large-scale vocabulary searches. In large models (such as Transformers), prefix sums are widely used in probability distribution calculations, sampling strategies (such as top-p), and efficiency optimization, playing a crucial role, especially in the decoding stage of text generation. In tasks such as natural language generation, dialogue systems, and code generation, prefix sums have become a core technology for improving the efficiency and controllability of model generation by accelerating probability filtering, optimizing path evaluation, and supporting parallel computing.

[0003] For arrays:

[0004] Its prefix sum array S is defined as:

[0005] like Figure 2 The example shown is for calculating the prefix sum: S[1] = A[1] = 1; S[2] = A[1] + A[2] = 3; S[3]= A[1] + A[2] + A[3] = 6; … S[8]= A[1] + A[2] + A[3] + … +A[8] = 36.

[0006] Existing technologies typically use vector computation (such as SIMD, CUDA cores) to perform prefix sum calculations (Single-pass Parallel Prefix Scan with Decoupled Look-back). In artificial intelligence (AI) chip architectures, matrix operations and vector / scalar operations are usually implemented using different hardware units. Matrix operations typically use dedicated hardware (such as Tensor Cores) to efficiently perform matrix multiplication and accumulation operations, resulting in high data parallelism efficiency. Vector operations are usually performed using SIMD or CUDA cores, which have low data parallelism efficiency. Furthermore, since matrix operations account for a large proportion of neural network models, while vector / scalar operations account for a relatively small proportion, the computational power ratio between matrix operations and vector / scalar operations differs significantly in AI chips, typically by tens to thousands of times. Therefore, using only vector operations to perform prefix sum calculations results in low computational efficiency.

[0007] Therefore, how to make full use of the hardware resources on the chip (matrix operation resources and vector operation resources) to effectively accelerate the calculation process of prefix sums and improve the efficiency of prefix sum calculation is a technical problem that needs to be solved urgently by existing technologies. Summary of the Invention

[0008] In order to solve at least one of the technical problems in the background art described above, the present invention proposes a prefix sum computing device, method, chip, board, and electronic device.

[0009] In one aspect, a prefix sum calculation device is provided, the device comprising: The data segmentation unit is used to group the vector data to be processed according to a preset segmentation granularity, and then concatenate the grouped vectors in sequence to form the matrix to be processed. A matrix operation unit is used to batch calculate the prefix sum of each row of data in the matrix to be processed to obtain an output matrix, wherein each row of data in the output matrix is ​​the prefix sum of the corresponding row of data in the matrix to be processed; The vector operation unit is used to perform addition operations on each row of data in the output matrix with the corresponding scalar value to obtain the result vector corresponding to each row of data, wherein the corresponding scalar value is equal to the cumulative value of the last data of each row before that row; The data splicing unit is used to splice the result vectors in sequence to obtain the prefix sum vector corresponding to the vector data to be processed.

[0010] Optionally, the matrix operation unit is specifically used to perform matrix multiplication on the matrix to be processed and the upper triangular matrix to obtain the output matrix, wherein the diagonal and upper elements of the upper triangular matrix are all 1.

[0011] Optionally, the prefix sum calculation device further comprises: a memory configured to store the output matrix; a vector register configured to store data of a to-be-calculated row in the output matrix during vector calculation; a scalar register configured to store the scalar value corresponding to the to-be-calculated row during vector calculation; a controller configured to control the vector operation unit to add data in the vector register and data in the scalar register to generate a corresponding result vector during vector calculation.

[0012] Optionally, the controller is specifically configured to read the output matrix from the memory, load data of each row in the output matrix except the first row to the vector register in sequence, control the vector operation unit to add each data in the vector register and the last data of a result vector corresponding to data of the last row stored in the scalar register to obtain a result vector corresponding to the current row data, write the result vector corresponding to the current row data into the memory, and if the current row data is the last row in the output matrix, complete the entire vector calculation, or update the last data in the result vector corresponding to the current row data to the scalar register for use in the next row operation.

[0013] Optionally, the controller is further configured to, for data of the first row in the output matrix, take the row data as a result vector and update the last data of the row data to the scalar register.

[0014] Optionally, the controller is specifically configured to write the result vector corresponding to the current row data into the memory to replace original data corresponding to the row in the output matrix.

[0015] Optionally, the data splicing unit is specifically configured to obtain the result vector corresponding to each row data in the output matrix from the memory, splice the result vectors in sequence to obtain the prefix sum vector corresponding to the to-be-processed vector data.

[0016] Optionally, the matrix operation unit adopts a systolic array, a weight matrix loaded in the systolic array is the upper triangular matrix, and the systolic array is configured to perform matrix multiplication on the to-be-processed matrix and the weight matrix to generate the output matrix.

[0017] Optionally, the segmentation granularity is set according to a parallel processing scale of the matrix operation unit.

[0018] Another aspect of the present application further provides a prefix sum calculation method, which comprises: group the to-be-processed vector data according to a preset grouping granularity by a data grouping unit, and sequentially splice each vector obtained by the grouping to form a to-be-processed matrix; calculate prefix sums of each row of data in the to-be-processed matrix in batches by a matrix operation unit to obtain an output matrix, wherein each row of data in the output matrix is a prefix sum of a corresponding row of data in the to-be-processed matrix; perform an addition operation on each row of data in the output matrix and a corresponding scalar value by a vector operation unit to obtain a result vector corresponding to the row of data, wherein the corresponding scalar value is equal to an accumulated value of the last data of each row before the row; splice each result vector in sequence by a data splicing unit to obtain a prefix sum vector corresponding to the to-be-processed vector data.

[0019] Another aspect of the present application also provides a chip comprising the prefix sum calculation device.

[0020] Another aspect of the present application also provides a board card comprising the chip.

[0021] Another aspect of the present application also provides an electronic device comprising the chip.

[0022] The present application has the following advantages: The embodiment of the present application groups the to-be-processed vector data according to a preset grouping granularity by a data grouping unit, and sequentially splices each vector obtained by the grouping to form a to-be-processed matrix, so that the vector data can be input into a hardware calculation unit in a matrix form for parallel processing; the prefix sums of each row of data in the to-be-processed matrix are calculated in batches by a matrix operation unit to obtain an output matrix, so that the high parallel calculation capability of the matrix operation unit is utilized to realize parallel prefix sum calculation of multiple rows of data; an addition operation is performed on each row of data in the output matrix and a corresponding scalar value by a vector operation unit to obtain a result vector corresponding to the row of data, so that the continuous connection of prefix sum results between different rows is realized; each result vector is spliced in sequence by a data splicing unit to obtain a prefix sum vector corresponding to the to-be-processed vector data. Thus, the present application can allocate prefix sum calculation tasks between a matrix operation unit and a vector operation unit, fully utilize the matrix operation resources and the vector operation resources inside the chip, and improve the overall processing efficiency of the prefix sum calculation while ensuring the calculation accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor. In the drawings: Figure 1 is a schematic diagram of the prefix and calculation device of the embodiment of the present application; Figure 2 is a schematic diagram of the prefix and calculation; Figure 3 is a schematic diagram of the overall flow of the prefix and calculation of the embodiment of the present application; Figure 4 is a schematic diagram of the matrix operation unit calculation of the embodiment of the present application; Figure 5 is a schematic diagram of the vector operation unit calculation of the embodiment of the present application; Figure 6 is a schematic diagram of the vector operation processing of the embodiment of the present application; Figure 7 is a flowchart of the vector operation processing of the embodiment of the present application; Figure 8 is a schematic diagram of the hardware data flow process of the prefix and calculation based on the vector operation unit of the embodiment of the present application; Figure 9 is a first schematic diagram of the calculation example of the present application; Figure 10 is a second schematic diagram of the calculation example of the present application; Figure 11 is a third schematic diagram of the calculation example of the present application. DETAILED DESCRIPTION

[0024] In order to make the person skilled in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0025] Those skilled in the art will appreciate that embodiments of the application can be supplied as a method, a system, or a computer program product. Accordingly, the application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code thereon.

[0026] It should be noted that the terms "comprising", "having", "including", and "containing" and any variations thereof in the specification and in the claims are intended to cover a non-exclusive inclusion, such that a process, method, system, product, or apparatus that comprises, has, includes, or contains an item or list of items who have not been explicitly stated or processes, methods, systems, products, or apparatuses that essentially or potentially include, or contain, other steps or units can include, without limitation, such an explicitly listed or inherent or potential steps or units.

[0027] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0028] The present application converts the prefix sum calculation into a form of combination of matrix operation (prefix sum calculation within a group) and vector operation (addition between groups), which can not only make full use of the hardware resources (matrix calculation resources and vector calculation resources) on the chip, but also effectively accelerate the calculation process of the prefix sum and improve the execution efficiency of the prefix sum operator.

[0029] Figure 1 is a schematic diagram of a prefix sum calculation device according to an embodiment of the present application, as shown in the figure, in an embodiment of the present application, the prefix sum calculation device of the present application comprises: Figure 1 a data splitting unit, configured to group the to-be-processed vector data according to a preset splitting granularity, and splice the vectors obtained by the grouping in sequence to form a to-be-processed matrix.

[0030] In an embodiment of the present application, the data splitting unit can comprise a data scheduling logic circuit or an input buffer control module, configured to receive the to-be-processed vector data in a data loading stage, and group the input data according to the preset splitting granularity, and reorganize the grouped data into a matrix form. The matrix can be stored in an on-chip cache or a local memory for reading by a subsequent matrix operation unit, so as to realize parallel loading of data at the matrix level.

[0031] ​The matrix operation unit is configured to calculate prefix sums of each row of the to-be-processed matrix in batches to obtain an output matrix, wherein each row of the output matrix is a prefix sum of a corresponding row of the to-be-processed matrix.

[0032] In an embodiment of the present application, the matrix operation unit can be implemented by using a systolic array or other parallel matrix multiplication structure to perform prefix sum calculation on each row of the to-be-processed matrix in a matrix-level parallel mode. The operation unit can load a triangular weight matrix in the array, and obtain the local prefix sum result of each row through a fixed weight matrix multiplication operation at one time, thereby significantly improving the parallel calculation capability of the prefix sum.

[0033] The vector operation unit is configured to perform addition operation on each row of the output matrix and a corresponding scalar value to obtain a result vector corresponding to the row, wherein the corresponding scalar value is equal to an accumulated value of the last data of each row before the row.

[0034] In an embodiment of the present application, the vector operation unit can be implemented by using a vector addition logic of the SIMD structure to perform cross-row accumulation operation on each row of the result output by the matrix operation unit. Specifically, the vector operation unit reads the scalar value corresponding to the current row (i.e., the accumulated value of the last data of the previous row result) from the scalar register according to a control signal, and performs addition operation on each element of the current row to generate the result vector corresponding to the current row.

[0035] The data splicing unit is configured to splice each result vector in sequence to obtain a prefix sum vector corresponding to the to-be-processed vector data.

[0036] The data splicing unit can be implemented by an output buffer or a result recombination module to splice each row of the result vector generated by the vector operation unit in sequence according to the row order to form a complete prefix sum vector. The result data after splicing can be directly written into an external memory or transmitted to a next-level calculation module for use.

[0037] Through the above structured module division and data flow control, the prefix sum calculation device of the embodiment of the present application can realize parallelization and pipelining of the prefix sum calculation process at the chip hardware level, thereby effectively improving the execution efficiency of the prefix sum calculation and the utilization rate of hardware resources.

[0038] In an embodiment of the present application, the matrix operation unit is specifically configured to perform matrix multiplication on the to-be-processed matrix and an upper triangular matrix to obtain the output matrix, wherein the elements on and above the diagonal line of the upper triangular matrix are all 1.

[0039] In one embodiment of the present application, each row of the to-be-processed matrix corresponds to a group of grouped vector data, the upper triangular matrix has the same dimension as the length of each group of vectors, and the element values in the diagonal and above are set to 1 and the element values below the diagonal are set to 0. When the matrix operation unit performs the matrix multiplication operation of the to-be-processed matrix and the upper triangular matrix, each row of vector data is element-wise multiplied and added with the corresponding weight position in the upper triangular matrix, so that the calculation result of the jth element of the row is equal to the accumulated value of the first j elements of the original row vector, thereby realizing the prefix sum calculation of each row of data at the matrix level.

[0040] In hardware implementation, the matrix operation unit can be implemented by using a systolic array structure or other matrix calculation circuit capable of performing parallel multiplication and addition operations. By loading the upper triangular weight matrix inside the array and keeping the weight matrix unchanged during the calculation process, the high parallelism of matrix multiplication and addition can be utilized to process multiple rows of vector data at the same time, thereby significantly improving the overall operation efficiency of the prefix sum calculation, reducing the operation delay and improving the utilization rate of the matrix operation unit. In the above manner, the matrix operation unit can realize efficient prefix sum calculation of vectors of different dimensions without changing the hardware structure, thereby providing basic data support for the cross-row accumulation of the subsequent vector operation unit.

[0041] In one embodiment of the present application, the matrix operation unit uses a systolic array, the weight matrix loaded in the systolic array is the upper triangular matrix, and the systolic array is used to perform matrix multiplication operation on the to-be-processed matrix and the weight matrix to generate the output matrix.

[0042] In one embodiment of the present application, the systolic array can include a plurality of multiplication and addition processing units arranged in a two-dimensional array, and each processing unit is used to perform multiplication and addition operation between input data and corresponding weights. The intermediate calculation results are sequentially transmitted between the processing units in the row and column directions, thereby realizing the pipeline flow and accumulation of data in the array. The upper triangular weight matrix is loaded into the systolic array before the operation starts, wherein the weight values in the diagonal and above are set to 1 and the weight values below the diagonal are set to 0.

[0043] When each row of data of the to-be-processed matrix is sequentially input into the systolic array, each processing unit in the array performs element-wise multiplication and addition operation according to the configuration of the upper triangular weight matrix according to the matrix multiplication rule, thereby obtaining the prefix sum result of each row of data in a single calculation process. The result is processed by parallel accumulation inside the array, and the output matrix is formed at the output end of the array. The matrix operation unit can fully utilize the parallel multiplication and addition characteristics of the systolic array at the hardware level to realize batch calculation of prefix sums of multiple groups of vector data, thereby improving the throughput and calculation efficiency of the prefix sum calculation.

[0044] In an embodiment of the present application, the split granularity is set according to the parallel processing scale of the matrix operation unit.

[0045] In an embodiment of the present application, the matrix operation unit can be implemented in a systolic array structure with a fixed array scale, and the parallel processing scale thereof can be determined by the number of rows and columns of multiply-accumulate processing units (MAC units) in the array. For example, when the matrix operation unit is a 4x4 systolic array, multiply-accumulate operations on 4 rows of input data, each containing 4 data units, can be processed in parallel in the same clock cycle, thereby realizing the synchronous execution of 16 element-level multiply-accumulate operations within the array.

[0046] Therefore, in the data loading and task allocation stage, the split granularity can be set to match the parallel processing scale of the matrix operation unit, i.e., when the matrix operation unit is an NxN array structure, the split granularity can be set to N, so that the number of elements of each group of vector data to be processed corresponds to the parallel processing capability of the systolic array, thereby completing the prefix sum calculation of the entire group of data in one matrix operation, avoiding idle or repeated invocation of array resources, and improving the calculation efficiency and hardware resource utilization.

[0047] In practical applications, the parallel processing scale of the matrix operation unit can vary depending on the chip architecture and application scenarios, such as a 4x4, 8x8, 16x16, or higher-dimensional systolic array structure. By dynamically setting the split granularity according to the parallel processing scale of the array, adaptive matching of the calculation load and hardware parallelism can be achieved, thereby further improving the overall execution performance of the prefix sum calculation.

[0048] Figure 3 is the overall flowchart of the prefix sum calculation of an embodiment of the present application, as shown in Figure 3 The input data of the prefix sum calculation device of the present application is a vector to be processed, which is a vector data (such as an array [a1, a2, …, am]) containing m elements.

[0049] The present application first determines the split granularity k, which is determined according to the parallel processing capability of the matrix operation unit (such as a systolic array).

[0050] The matrix operation unit in the device of the present application splits the vector to be processed into groups of k elements, a total of n=m / k groups (rounded up). Assuming that m is an integer multiple of k, if m is not an integer multiple of k, 0s are added at the tail of the input vector to make up the integer multiple.

[0051] Then, the n groups of data (each group containing k elements) are spliced by rows to form the matrix A to be processed (n x k, each row corresponding to a group of data).

[0052] In addition, the present application also pre-establishes an upper triangular matrix U (k x k), the diagonal and the elements above the diagonal of the triangular matrix U are all 1, for example, when k = 3, the triangular matrix U is as follows:

[0053] The upper triangular matrix U is used to simulate the accumulation logic of the prefix sum (calculating the prefix sum in the group for each row).

[0054] As shown in the figure, the matrix operation unit in the device of the present application specifically performs matrix multiplication A x U on the to-be-processed matrix A (n x k) and the triangular matrix U (k x k), to obtain an output matrix (n x k). Figure 4

[0055] The data in each row of the output matrix (n x k) is the prefix sum of the data in each row of the to-be-processed matrix A (n x k), for example, the first row of data of the to-be-processed matrix A is as follows:

[0056] The first row of data in the output matrix is as follows:

[0057] The present application utilizes the parallelism of matrix operation, and n rows of data can be calculated at the same time, thereby improving the calculation efficiency.

[0058] The output matrix is the prefix sum in each group of n groups of data, but to obtain the prefix sum vector corresponding to the to-be-processed vector, the generation of the global prefix sum is also needed.

[0059] As shown in the figure, the present application generates the global prefix sum through the vector operation unit. Figure 5

[0060] The first row (P1 row) in the output matrix is directly used as the local prefix sum (the prefix sum of the first k elements).

[0061] The elements in the subsequent rows (P i , 2≤i≤n-1 rows) are added with the sum of all the previous rows (i.e., the prefix sum of the first (i-1)k+j elements), for example: The k elements in the second row (P2 row) become as follows:

[0062] The k elements in the third row (P3 row) become as follows:

[0063] The k elements in the n-th row (P n row) become as follows:

[0064] ​​After the vector operation is completed, the data splicing unit splices the result vectors obtained by the vector operation in row order to obtain a prefix sum vector (m data) corresponding to the to-be-processed vector data.

[0065] It can be seen that the core logic of the prefix sum calculation device is as follows: Intra-group parallelism: prefix sums in each group are calculated in batches through matrix operation, and hardware parallelism (such as Tensor Core and systolic array) is used for acceleration.

[0066] Inter-group addition: the result vectors of each group are accumulated through the vector operation unit to ensure the continuity of the global prefix sum (that is, the prefix sum of the first i×k data is equal to the accumulated sum of the first (i-1)k data plus the prefix sum of the i-th group, where 1≤i≤n).

[0067] In an embodiment of the present application, the prefix sum calculation device further comprises: a memory for storing the output matrix; a vector register for storing data of a to-be-calculated row in the output matrix during vector calculation; a scalar register for storing the scalar value corresponding to the to-be-calculated row during vector calculation; a controller for controlling the vector operation unit to add the data in the vector register and the data in the scalar register to generate a corresponding result vector during vector calculation.

[0068] In an embodiment of the present application, the memory is used to store the output matrix output by the matrix operation unit and provide to-be-calculated data for the vector register during vector calculation. The vector register is used to store data of a to-be-calculated row in the output matrix during vector calculation, so that the vector operation unit directly reads data from the register for parallel addition operation. The scalar register is used to store the scalar value corresponding to the to-be-calculated row during vector calculation, which can be obtained by updating the last data of the previous row result vector and used for cross-row accumulation operation in the vector calculation of the current row.

[0069] The controller is used to coordinate the operation of the above modules during vector calculation, control the vector operation unit to add the data in the vector register and the data in the scalar register to generate a corresponding result vector. The controller can also write the result vector back to the memory after each calculation is completed, and update the last data of the result vector to the scalar register, so as to continue to be used in the calculation of the next row of data.

[0070] Through the above arrangement, the prefix sum calculation device can complete the data connection between the matrix operation and the vector operation at the hardware level, utilize the register to perform the cache and the data exchange, realize the automatic control of the cross-row accumulation, and further improve the parallel processing performance and the overall operation efficiency of the prefix sum calculation.

[0071] In an embodiment of the present application, the controller is specifically configured to read the output matrix from the memory, sequentially load the current row data to the vector register for each row data in the output matrix except the first row, control the vector operation unit to add each data in the vector register and the last data of the result vector corresponding to the previous row data stored in the scalar register to obtain the result vector corresponding to the current row data, write the result vector corresponding to the current row data into the memory, and if the current row data is the last row in the output matrix, complete the entire vector calculation, otherwise update the last data in the result vector corresponding to the current row data to the scalar register for use in the next row operation.

[0072] In an embodiment of the present application, the controller is further configured to take the first row data in the output matrix as the result vector and update the last data of the first row data to the scalar register.

[0073] In an embodiment of the present application, the controller is configured to execute the cross-row accumulation control logic of the prefix sum result in the vector calculation stage. Specifically, the controller is configured to read the output matrix from the memory, sequentially load the current row data to the vector register for each row data in the output matrix except the first row, control the vector operation unit to perform the addition operation on each data in the vector register and the last data of the result vector corresponding to the previous row data stored in the scalar register to obtain the result vector corresponding to the current row data, and write the result vector into the memory to update the data content of the corresponding row in the output matrix.

[0074] When it is judged that the current row data is the last row in the output matrix, the controller determines that the entire vector calculation is completed, and if the current row data is not the last row, the controller controls the scalar register to update the last data in the result vector corresponding to the current row data for use in the next row vector calculation. For the first row data in the output matrix, the first row data is directly taken as the corresponding result vector, and the last data of the first row data is updated to the scalar register to provide an initial scalar value for the calculation of the subsequent row.

[0075] Through the above process, the controller can realize the prefix sum and the row-level loop control, so that the row vector calculation realizes automatic loading, calculation, updating and write-back operation at the hardware level, thereby ensuring the continuity of the cross-row accumulation process and the correctness of the calculation result.

[0076] In an embodiment of the present application, the controller is specifically configured to write the result vector corresponding to the current row data into the memory, and replace the original data corresponding to the row in the output matrix.

[0077] In this embodiment, when the vector operation unit completes the addition operation of the current row data, the controller receives the result vector output by the vector operation unit, and controls the memory interface to write the result vector back to the memory for updating the storage content of the corresponding row in the output matrix. By replacing the original data with the result vector, the prefix sum calculation result can be updated in real time without occupying additional storage space, thereby reducing the on-chip storage requirement and simplifying the data management logic. This process also ensures the data consistency of the output matrix during the calculation, i.e., the storage content of each row reflects the latest calculation result, which facilitates the subsequent splicing operation to directly read the final result data from the memory, thereby realizing efficient data streaming and storage access.

[0078] In an embodiment of the present application, the data splicing unit is specifically configured to obtain the result vector corresponding to each row data in the output matrix from the memory, and splice the result vectors in sequence to obtain the prefix sum vector corresponding to the to-be-processed vector data.

[0079] In this embodiment, the data splicing unit can include a data reorganization logic or an output buffer control module, which is configured to sequentially read the result vectors of each row from the memory after the prefix sum calculation is completed, and splice the result vectors in sequence on the chip to generate a complete prefix sum vector. The splicing operation can be realized through address control logic, i.e., according to the starting address of the result vector in the memory and the row number information, the data of each row vector is sequentially written into the continuous storage space of the output buffer. In this way, the data splicing unit can complete the ordered integration of the result data at the hardware level without additional software reorganization operation, thereby realizing one-time output of the prefix sum calculation result. This setting not only ensures the consistency of the data order, but also reduces the data movement and the overhead of intermediate caching, thereby improving the efficiency of the overall calculation and output.

[0080] Figure 6 is a schematic diagram of an embodiment of the present application for realizing vector operation processing, Figure 6 which shows a specific schematic diagram for realizing vector operation processing, and shows the complete process from instruction acquisition, decoding, scheduling to efficient parallel vector calculation using a vector operation unit.

[0081] SRAM: Static Random Access Memory, used to store program instructions and data to meet the data access speed requirements of the system architecture.

[0082] Instruction Cache: A high-speed cache space that caches instructions read from SRAM, reducing the time the processor waits for instructions.

[0083] Instruction fetch cache: Further stores instructions fetched from the instruction cache. During the instruction fetch phase, RISC-V architecture processors use the instruction fetch cache to store instructions in an orderly manner, preparing them for subsequent decoding and execution.

[0084] Instruction decoding: The instruction decoding unit can quickly parse these instructions, rapidly identify the type of operation to be performed and the source of operands, and prepare for subsequent computation and processing.

[0085] Scheduler: The scheduler arranges the execution order of instructions reasonably based on the resource situation (such as register resources, execution unit status, etc.) and instruction characteristics in different application scenarios.

[0086] Vector register set: A register file used to store vector operands. The data bit width of each vector register is equal to the data processing bit width of the vector operation unit.

[0087] Vector operation unit: Used to perform some operations in parallel (such as SIMD operations, which process multiple elements at once, and can perform vector addition, vector multiplication and other operations), improving computational density and efficiency.

[0088] Figure 7 This is a flowchart of vector operation processing according to an embodiment of the present invention. Figure 7 A method for calculating the prefix sum of input data based on vector operation units is described. The calculation result is accumulated step by step through vector and scalar operations. The specific steps and principles are as follows: 1. Data loading: Read the first row vector P1 and the second row vector P2 from the output matrix in SRAM. Take the last data (element) of vector P1 and load it into scalar register rs1. Load vector P2 into vector register vs1.

[0089] 2. Vector-Scalar Addition: The vector-scalar addition instruction is used to control the vector arithmetic unit to add each data (element) in the vector register vs1 to the value in the scalar register rs1, resulting in the vector vs2.

[0090] 3. Result rewrite: Write the resulting vector vs2 back into SRAM and save the partial calculation results of the prefix sum.

[0091] 4. Looping and data updating: Determine if the vector in vector register vs1 is the last row vector Pn of the output matrix. If it is not the last row vector, read the next row vector from SRAM and load it into vector register vs1.

[0092] Use the assignment instruction to assign the last data in the result vector vs2 to the scalar register rs1. Then repeat the steps of vector-scalar addition, result write-back, and loop check.

[0093] If the last row vector is Pn, the loop ends. At this point, all prefix sums of the input data have been calculated, and the results have been stored in SRAM.

[0094] Figure 8 This is a schematic diagram of the hardware data flow process for prefix sum calculation based on a vector operation unit according to an embodiment of the present invention. Figure 8 The hardware data flow process for prefix sum calculation based on the vector operation unit is demonstrated. The data flow direction is explained as follows: The blue arrows represent the flow of data being read from SRAM and loaded into registers. The scalar register rs1 has two main data sources: one is loading the last data (element) of vector P1 directly into rs1 during the initial phase, and the other is assigning the last data (element) of the result vector vs2 to rs1 during subsequent loop phases; data is then read from P2 to Pn and loaded into vs1 sequentially. The red arrow indicates the process where, after the vector operation unit performs the operation, the result vector flows from the vector operation unit to vs2, and then the result vector in vs2 is written back to SRAM.

[0095] Figure 9 , Figure 10 and Figure 11 A specific calculation example of the present invention is shown.

[0096] In this computational example, taking the calculation of the prefix sum of three sets of input vectors (i.e., the vectors to be processed) as an example, each set of input vectors contains 16 data points; the matrix operation unit is a 4×4 systolic array, and the systolic array adopts a fixed weight data working mode; each set of vectors is split according to the size of the systolic array (4), and each set of vectors can be split into four 1×4 sub-vectors, which are then concatenated into a 4×4 matrix, such as Figure 9 As shown, the three sets of vectors to be processed can be converted into three sets of 4×4 matrices to be processed.

[0097] Arrange the three sets of 4×4 matrices to be processed sequentially, as follows: Figure 10As shown, the input activations are sequentially input into the systolic array. At this time, the weight matrix loaded in the systolic array is an upper triangular matrix, and all elements above the diagonal are 1. The matrix multiplication operation between the input activation matrix and the weight matrix is ​​completed through the systolic array, resulting in 12 1×4 result vectors. According to the order of the output data stream, the first 4 vectors are the block prefix sum matrix of input vector 1 (i.e., the output matrix mentioned above), the middle 4 vectors are the block prefix sum matrix of vector 2, and the last 4 vectors correspond to the block prefix sum matrix of vector 3.

[0098] The block prefix sum matrices of the three sets of vectors are fed into the vector operation unit, and the complete prefix sums of the three sets of input vectors are obtained according to the calculation method of the vector operation unit described above. Subsequently, the complete prefix sums of each set of input vectors are concatenated to obtain the prefix sum vectors corresponding to the three sets of input vectors (i.e., the vectors to be processed).

[0099] Another aspect of the present invention provides a prefix sum calculation method. In one embodiment of the present invention, the prefix sum calculation method includes: The data segmentation unit groups the vector data to be processed according to a preset segmentation granularity, and the grouped vectors are then concatenated in sequence to form the matrix to be processed. The prefix sum of each row of data in the matrix to be processed is calculated in batches by the matrix operation unit to obtain the output matrix, wherein each row of data in the output matrix is ​​the prefix sum of the corresponding row of data in the matrix to be processed; The vector operation unit performs addition operations on each row of data in the output matrix with the corresponding scalar value to obtain the result vector corresponding to each row of data, wherein the corresponding scalar value is equal to the cumulative value of the last data of each row before that row; The data splicing unit sequentially splices the result vectors to obtain the prefix sum vector corresponding to the vector data to be processed.

[0100] In one embodiment of the present invention, the step of batch calculating the prefix sum of each row of data in the matrix to be processed by the matrix operation unit to obtain the output matrix specifically includes: The matrix operation unit performs matrix multiplication on the matrix to be processed and the upper triangular matrix to obtain the output matrix, wherein all elements on the diagonal and above of the upper triangular matrix are 1.

[0101] In one embodiment of the present invention, the prefix sum calculation method of the present invention further includes: The output matrix is ​​stored in a memory. The data of the rows to be calculated in the output matrix are stored in the vector register during vector computation. The scalar value corresponding to the row to be calculated is stored in a scalar register during vector computation. When the controller performs vector computation, it controls the vector operation unit to add the data in the vector register to the data in the scalar register to generate the corresponding result vector.

[0102] In one embodiment of the present invention, the step of controlling the vector operation unit to add the data in the vector register to the data in the scalar register to generate a corresponding result vector during vector computation by the controller specifically includes: The controller reads the output matrix from the memory, and sequentially loads the data of each row of the output matrix (excluding the first row) into the vector register. The vector operation unit adds each data point in the vector register to the last data point of the result vector corresponding to the previous row stored in the scalar register, obtaining the result vector corresponding to the current row. The result vector corresponding to the current row is written into the memory. If the current row is the last row of the output matrix, all vector calculations are completed; otherwise, the last data point of the result vector corresponding to the current row is updated in the scalar register for use in the next row's operation. For the first row of the output matrix, this row is used as the result vector, and the last data point of this row is updated in the scalar register.

[0103] In one embodiment of the present invention, writing the result vector corresponding to the current row of data into the memory specifically includes: Write the result vector corresponding to the current row of data into the memory, replacing the original data corresponding to that row in the output matrix.

[0104] In one embodiment of the present invention, the step of sequentially concatenating the result vectors by the data concatenation unit to obtain the prefix sum vector corresponding to the vector data to be processed specifically includes: The data splicing unit retrieves the result vectors corresponding to each row of data in the output matrix from the memory, and splices the result vectors in row order to obtain the prefix sum vector corresponding to the vector data to be processed.

[0105] For specific embodiments of the prefix sum calculation method of the present invention, please refer to the embodiments of the prefix sum calculation device described above, and will not be repeated here.

[0106] In another aspect, the present invention provides a chip including the aforementioned prefix and computing device.

[0107] In one embodiment of the present invention, the chip may be an artificial intelligence chip (AI chip).

[0108] In another aspect, the present invention provides a circuit board including the aforementioned chip.

[0109] In another aspect, the present invention provides an electronic device comprising the aforementioned chip.

[0110] The present invention is applicable to AI chips, artificial intelligence servers, and artificial intelligence server clusters. Using the present invention can effectively improve the execution efficiency of hardware in processing prefixes and tasks. As can be seen from the above embodiments, the present invention has at least the following beneficial effects compared to the prior art: 1. Improved computational efficiency: This invention groups the vector data to be processed into a preset granularity and inputs it into the matrix operation unit in matrix form. It utilizes the high parallel computing capability of the matrix operation unit to perform batch calculation of the prefix sum of each row of data, which significantly improves the throughput and execution efficiency of prefix sum calculation.

[0111] 2. Make full use of heterogeneous hardware resources: The present invention adopts a collaborative computing method between matrix operation units and vector operation units, which respectively undertake the tasks of local prefix sum calculation and inter-row accumulation calculation, thereby realizing the division of labor and collaboration between matrix operation units and vector operation units and giving full play to the hardware performance of different types of computing units within the chip.

[0112] 3. Reduce the usage of low-parallelism operation units: This invention distributes the main computational load to matrix operation units and only calls vector operation units for lightweight operations during the cross-row accumulation stage, thereby reducing the frequency of use of low-parallelism vector operation units and lowering the overall computational latency.

[0113] 4. Improve chip resource utilization: This invention enables the full utilization of idle matrix computing resources in the chip by grouping, matrixing and inter-row concatenation of vector data, achieving load balancing of prefix sum calculation on different types of computing units, and improving the overall computing power utilization of the chip.

[0114] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A prefix sum calculation device, characterized in that, include: The data segmentation unit is used to group the vector data to be processed according to a preset segmentation granularity, and then concatenate the grouped vectors in sequence to form the matrix to be processed. A matrix operation unit is used to batch calculate the prefix sum of each row of data in the matrix to be processed to obtain an output matrix, wherein each row of data in the output matrix is ​​the prefix sum of the corresponding row of data in the matrix to be processed; The vector operation unit is used to perform addition operations on each row of data in the output matrix with the corresponding scalar value to obtain the result vector corresponding to each row of data, wherein the corresponding scalar value is equal to the sum of the last data of each row before that row; The data splicing unit is used to splice the result vectors in sequence to obtain the prefix sum vector corresponding to the vector data to be processed.

2. The prefix sum calculation device according to claim 1, characterized in that, The matrix operation unit is specifically used to perform matrix multiplication on the matrix to be processed and the upper triangular matrix to obtain the output matrix, wherein the diagonal and above elements of the upper triangular matrix are all 1.

3. The prefix sum calculation device according to claim 1, characterized in that, Also includes: A memory for storing the output matrix; A vector register is used to store the data of the rows to be calculated in the output matrix during vector computation; A scalar register is used to store the scalar value corresponding to the row to be calculated during vector computation; The controller is used to control the vector operation unit to add the data in the vector register to the data in the scalar register during vector computation, so as to generate the corresponding result vector.

4. The prefix sum calculation device according to claim 3, characterized in that, The controller is specifically configured to read the output matrix from the memory, sequentially load the current row data into the vector register for each row of data in the output matrix except for the first row; control the vector operation unit to add each data in the vector register to the last data of the result vector corresponding to the previous row data stored in the scalar register to obtain the result vector corresponding to the current row data; and write the result vector corresponding to the current row data into the memory. If the current row of data is the last row in the output matrix, then all vector calculations are completed; otherwise, the last data in the result vector corresponding to the current row of data is updated to the scalar register for use in the next row of calculations.

5. The prefix sum calculation device according to claim 4, characterized in that, The controller is further configured to, for the first row of data in the output matrix, take the row of data as a result vector and update the last data of the row of data to the scalar register.

6. The prefix sum calculation device according to claim 4, characterized in that, The controller is specifically used to write the result vector corresponding to the current row of data into the memory, replacing the original data corresponding to that row in the output matrix.

7. The prefix sum calculation device according to any one of claims 4 to 6, characterized in that, The data splicing unit is specifically used to obtain the result vectors corresponding to each row of data in the output matrix from the memory, and splice the result vectors in row order to obtain the prefix sum vector corresponding to the vector data to be processed.

8. The prefix sum calculation device according to claim 2, characterized in that, The matrix operation unit adopts a pulsating array, and the weight matrix loaded in the pulsating array is the upper triangular matrix. The pulsating array is used to perform matrix multiplication operations on the matrix to be processed and the weight matrix to generate the output matrix.

9. The prefix sum calculation device according to claim 1, characterized in that, The granularity of the segmentation is set according to the parallel processing scale of the matrix operation unit.

10. A method for calculating prefix sums, characterized in that, include: The data segmentation unit groups the vector data to be processed according to a preset segmentation granularity, and the grouped vectors are then concatenated in sequence to form the matrix to be processed. The prefix sum of each row of data in the matrix to be processed is calculated in batches by the matrix operation unit to obtain the output matrix, wherein each row of data in the output matrix is ​​the prefix sum of the corresponding row of data in the matrix to be processed; The vector operation unit performs addition operations on each row of data in the output matrix with the corresponding scalar value to obtain the result vector corresponding to each row of data, wherein the corresponding scalar value is equal to the cumulative value of the last data of each row before that row; The data splicing unit sequentially splices the result vectors to obtain the prefix sum vector corresponding to the vector data to be processed.

11. A chip, characterized in that, The prefix and computing device includes any one of claims 1 to 9.

12. A circuit board, characterized in that, Includes the chip described in claim 11.

13. An electronic device, characterized in that, Includes the chip described in claim 11.

Images