Vector processing method, device and system for non-linear operation

Abstract

The application provides a vector processing method, device and system of nonlinear operation, which divides the nonlinear operation into a reduction stage and a normalization stage, adopts element serial scheduling execution, and comprises the following steps: in the reduction stage, a processing unit array is controlled to perform a first matrix vector multiplication in an inner product mode, and elements in a first result vector are output in element serial; a reduction unit is synchronously controlled to receive each element in the first result vector, and to update a global statistic quantity on line; after the global statistic quantity is ready, the normalization stage is entered, the processing unit array is switched to an outer product mode; a normalization unit is controlled to normalize each element of the first result vector temporarily stored based on the global statistic quantity, to generate an element in a second result vector, and to broadcast the element to the processing unit array in element serial; and the processing unit array is synchronously controlled to receive the element in the second result vector in each cycle, to perform a second matrix vector multiplication in the outer product mode, and to output a third result vector, so as to improve the vector processing speed of the nonlinear operation.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence-specific accelerators and computer architecture technology, and in particular to a vector processing method, apparatus and system for nonlinear operations. Background Technology

[0002] In recent years, deep learning models based on attention mechanisms have been widely used in natural language processing, computer vision, and multimodal learning. Attention mechanisms effectively enhance the expressive power of models by weighted aggregation of input features. A typical attention computation process usually involves multiple matrix multiplications and nonlinear operators such as Softmax and LayerNorm.

[0003] In existing hardware acceleration solutions, matrix multiplication is typically performed by highly parallel processing unit arrays, while nonlinear operations such as Softmax and LayerNorm are executed by dedicated function units. Since these nonlinear operations require all data in the complete vector or feature dimensions to be ready before computation can begin, the processing unit array remains idle during the execution of the dedicated function units, significantly reducing overall hardware resource utilization.

[0004] To alleviate the above problems, some existing technologies reduce nonlinear operation latency by increasing the number of dedicated function units or using approximate calculation methods. However, these solutions often introduce additional hardware resource overhead or sacrifice computational accuracy. Summary of the Invention

[0005] The purpose of this application is to provide a vector processing method, apparatus and system for nonlinear operations, which solves the problem of idle processing unit arrays during nonlinear operations in the prior art.

[0006] Firstly, this application provides a vector processing method for nonlinear operations, dividing the nonlinear operation into a reduction stage and a normalization stage, and employing element-wise serial scheduling for execution. The method includes: in the reduction stage, controlling the processing unit array to perform a first matrix-vector multiplication in inner product mode, and sequentially outputting the elements of the first result vector element by element; synchronously controlling the reduction unit to receive each element of the first result vector and updating the global statistics online; after the global statistics are ready, entering the normalization stage, switching the processing unit array to outer product mode; and controlling the normalization unit to normalize the temporarily stored first result vector element by element based on the global statistics, generating... The elements of the second result vector are serially broadcast to the processing unit array element by element; the synchronous control processing unit array receives the elements of the second result vector periodically, performs the second matrix-vector multiplication in outer product mode, and outputs the third result vector; wherein, the processing unit array is a reconfigurable processing unit that can switch between outer product mode and inner product mode; the nonlinear operation includes: Softmax or LayerNorm operation; the input operands of the first matrix-vector multiplication include: the first input vector and the first input matrix; the input operands of the second matrix-vector multiplication include: the second result vector and the second input matrix.

[0007] Furthermore, the step of the above-mentioned control processing unit array performing the first matrix-vector multiplication in inner product mode and sequentially outputting the elements in the first result vector element by element includes: configuring the processing unit array to inner product mode through a mode control signal; performing multiplication operations on the first input vector and the row vectors in the first input matrix in parallel on a periodic basis through each processing unit of the processing unit array; reducing the multiple products output in parallel by each processing unit through a hierarchical addition tree structure; and outputting the elements in the first result vector element by element in a serial manner on a periodic basis.

[0008] Furthermore, the aforementioned hierarchical addition tree structure includes at least a first-level addition tree and a second-level addition tree; the addition nodes of the first-level and second-level addition trees are reconstructed from the adders and partial sum forwarding paths within the processing unit in inner product mode; the first-level addition tree is used to perform parallel merging of the product results generated by adjacent multipliers within each processing unit row or local processing unit group to form multiple local partial sums; the second-level addition tree is used to further merge the local partial sums output by the first-level addition tree to generate a complete inner product result in the current cycle.

[0009] Furthermore, the step of the aforementioned synchronous control reduction unit receiving each element in the first result vector and updating the global statistics online includes: when the nonlinear operation is a Softmax operation, the global statistics include the maximum value and exponential sum of each element in the first result vector; during the process of receiving each element cycle by cycle, the reduction unit updates the current maximum value through a comparator, and after detecting the maximum value update, performs online scaling correction on the stored exponential sum, and then adds the exponential value of the current element to the exponential sum; when the nonlinear operation is a LayerNorm operation, the global statistics include the element-wise sum and element-wise sum of squares of each element in the first result vector; during the process of receiving each element cycle by cycle, the reduction unit updates the element-wise sum through an accumulator and updates the element-wise sum of squares through a multiply-accumulate unit; after all elements have been processed, the reduction unit latches the final global statistics for use in the normalization stage.

[0010] Furthermore, the step of latching the final global statistics after all elements have been processed by the reduction unit includes: counting the elements of the received first result vector using an element counter; when the count value reaches the sequence length value in the preset configuration register, determining that the global statistics are ready, and latching the final global statistics.

[0011] Furthermore, the above-mentioned switching of the processing unit array to the external product mode includes: in response to the global statistics ready signal, generating an external product mode selection signal, controlling the multiplexer and mode control logic in the processing unit array, cutting off part of the transmission path, and enabling each processing unit to perform local accumulation independently.

[0012] Furthermore, the step of the control normalization unit normalizing the temporarily stored first result vector element by element based on the global statistics to generate the elements in the second result vector includes: activating the normalization unit through a normalization unit start signal, causing the normalization unit to read the current element in the first result vector element by element from the buffer storage unit; the normalization unit performing a subtraction operation on the current element and the global statistics, and performing a division operation on the subtraction result to obtain the corresponding element in the second result vector; wherein, when the nonlinear operation is a Softmax operation, the normalization unit performs an exponential operation on the subtraction result after the subtraction operation and before the division operation; when the nonlinear operation is a LayerNorm operation, the method further includes performing an optional affine transformation on the result of the division operation.

[0013] Furthermore, the steps of receiving elements from the second result vector in the aforementioned synchronous control processing unit array cycle by cycle, performing the second matrix-vector multiplication in outer product mode, and outputting the third result vector include: in each cycle, broadcasting one element of the second result vector to each processing unit of the processing unit array, and reading the corresponding row vector of the second input matrix; each processing unit multiplies the received element with the weight component in the corresponding row vector, and accumulates the multiplication into its own local accumulator register; after accumulating for multiple cycles, the accumulation is ended by the processing unit turn-off signal, and the third result vector is jointly output by the local accumulator registers in each processing unit.

[0014] Furthermore, the aforementioned processing unit array includes a first type of processing unit and a second type of processing unit; the adder of the first type of processing unit has at least one input that receives the local multiplication result, and the other input that selects between the local accumulated value and the partial sum passed from the adjacent processing unit; the two inputs of the adder of the second type of processing unit can both come from other processing units, and are used to form higher-level addition tree nodes in inner product mode; the inner product mode and the outer product mode are implemented by jointly configuring the mode control logic of the first type of processing unit and the second type of processing unit, so that the same processing unit array can be reconfigured at runtime.

[0015] Furthermore, the first input vector is the query vector; the first input matrix is ​​the key matrix; the second input matrix is ​​the value matrix; the key matrix and the value matrix are uniformly stored row-wise in a two-dimensional matrix format. The multiplication operation between the query vector and the transpose of the key matrix is ​​completed by the first matrix-vector multiplication in the inner product mode to obtain the attention score vector; the multiplication operation between the attention score vector and the value matrix is ​​completed by the second matrix-vector multiplication in the outer product mode to obtain the attention output vector.

[0016] Secondly, this application also provides a vector processing device for nonlinear operations, which divides the nonlinear operation into a reduction stage and a normalization stage, and executes it using element-wise serial scheduling. The device includes: a reduction stage control module, used to control the processing unit array to perform a first matrix-vector multiplication in inner product mode during the reduction stage, and to output the elements of the first result vector element-wise serially; synchronously controlling the reduction unit to receive each element in the first result vector and update the global statistics online; and a normalization stage control module, used to enter the normalization stage after the global statistics are ready, and to switch the processing unit array to outer product mode; and to control the normalization unit to perform a temporary first matrix-vector multiplication based on the global statistics. The result vector is element-wise normalized to generate the elements in the second result vector, and then broadcast element-wise serially to the processing unit array. The synchronous control processing unit array receives the elements in the second result vector periodically, performs the second matrix-vector multiplication in outer product mode, and outputs the third result vector. The processing unit array is a reconfigurable processing unit that can switch between outer product mode and inner product mode. The nonlinear operations include Softmax or LayerNorm operations. The input operands of the first matrix-vector multiplication include the first input vector and the first input matrix. The input operands of the second matrix-vector multiplication include the second result vector and the second input matrix.

[0017] Thirdly, this application also provides a vector processing system for nonlinear operations, the system comprising: a scheduling control module, a processing unit array, a hierarchical addition tree structure, and dedicated function units; the dedicated function units comprising: a reduction unit, a normalization unit, and a storage unit; the scheduling control module being connected to the processing unit array, the hierarchical addition tree structure, and the dedicated function units respectively; the processing unit array, the hierarchical addition tree structure, and the dedicated function units being connected in sequence; the scheduling control module being used to execute the method as described in any of the first aspects.

[0018] Furthermore, the aforementioned scheduling control module includes a finite state machine, an element counter, a vector length configuration register, and a handshake control circuit, used to generate control signals according to the current calculation stage and coordinate the data flow order between the processing unit array, the hierarchical addition tree structure, and the dedicated function units.

[0019] The vector processing method, apparatus, and system for nonlinear operations provided in this application first configure the processing unit array in inner product mode, perform matrix-vector multiplication on the first input vector and the first input matrix, reduce the product step by step through a hierarchical addition tree structure, and output the elements in the first result vector periodically in an element-by-element serial manner; synchronously control the reduction unit to receive each element in the first result vector and update the global statistics online; after the global statistics are ready, switch the processing unit array to outer product mode; the above-mentioned processing unit array is a reconfigurable processing unit that can switch between outer product mode and inner product mode; control the normalization unit to normalize the temporarily stored first result vector element by element based on the global statistics, generate the elements in the second result vector, and broadcast them to the processing unit array in outer product mode in an element-by-element serial manner; control the processing unit array to receive the elements in the second result vector periodically, perform multiplication and accumulation with the row vectors of the second input matrix, and output the third result vector. The processing unit array is a reconfigurable processing unit that can switch between outer product mode and inner product mode. Nonlinear operations include Softmax or LayerNorm operations. The input operands for the first matrix-vector multiplication include a first input vector and a first input matrix. The input operands for the second matrix-vector multiplication include a second result vector and a second input matrix. In this method, the process of the reduction unit receiving the elements of the first result vector and updating the global statistics overlaps in time with the element-wise output process of the matrix-vector multiplication in the inner product mode. The process of the normalization unit generating the elements of the second result vector overlaps in time with the element-wise input process of the matrix-vector multiplication in the outer product mode. This solves the problem of idle processing unit arrays during nonlinear operations in existing technologies, greatly improving the vector processing speed of nonlinear operations. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 A schematic diagram of the structure of a vector processing system for nonlinear operations provided in this application embodiment; Figure 2 A flowchart illustrating a vector processing method for nonlinear operations provided in this application embodiment; Figure 3 This is a schematic diagram of a reconfigurable processing unit structure provided in an embodiment of this application; Figure 4This application provides a schematic diagram of a processing unit array structure in an external product calculation mode. Figure 5 This application provides a schematic diagram of a processing unit array structure in an inner product calculation mode. Figure 6 This is a schematic diagram of a hierarchical addition tree structure provided in an embodiment of this application.

[0022] Figure 7 A schematic diagram comparing the delay of a traditional scheduling method and an element serial scheduling method provided for embodiments of this application; Figure 8 A unified abstract diagram of Softmax and LayerNorm operations provided for embodiments of this application; Figure 9 This is a schematic diagram of a hardware configuration based on element serial scheduling provided in an embodiment of this application.

[0023] Figure 10 This is a structural block diagram of a vector processing device for nonlinear operations provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions of this application will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] Attention computation requires significantly different computational patterns at different stages. For example, in the decoding stage, the query-key computes q×K. T =(1,d)×(d,l)=(1,l) is more suitable for using an inner product data stream, which maps the sequence length l to the time dimension and generates attention scores element by element; the Score-Value calculation s'×V=(1,l)×(l,d)=(1,d) is more suitable for using an outer product data stream, which multiplies each normalized attention score by the corresponding Value row vector and accumulates them locally within the processing unit. Traditional fixed-structure processing unit arrays cannot simultaneously meet the efficiency requirements of the above different computational stages.

[0026] Based on this, embodiments of this application provide a vector processing method, apparatus, and system for nonlinear operations, which solves the problem of idle processing unit arrays during nonlinear operations in the prior art, thereby improving the utilization rate of processing unit arrays during nonlinear operations and adapting to the computational characteristics of different stages of attention computation.

[0027] To facilitate understanding of this embodiment, a vector processing system for nonlinear operations disclosed in this application will first be described in detail.

[0028] This application provides a vector processing system for nonlinear operations, see [link to relevant documentation]. Figure 1 As shown, the system includes: a scheduling control module 10, a processing unit array 20, a hierarchical addition tree structure 30, and a dedicated function unit 40; the dedicated function unit 40 includes: a reduction unit 401, a normalization unit 402, and a storage unit 403; the scheduling control module 10 is connected to the processing unit array 20, the hierarchical addition tree structure 30, and the dedicated function unit 40 respectively; the processing unit array 20, the hierarchical addition tree structure 30, and the dedicated function unit 40 are connected in sequence; the scheduling control module 10 is used to execute the steps of the vector processing method for nonlinear operations described later.

[0029] The multiplication results and partial sum propagation paths of the processing unit array 20 are connected to the hierarchical addition tree structure 30. The hierarchical addition tree structure 30 performs step-by-step reduction on multiple product results corresponding to the same inner product in inner product mode, and outputs the matrix-vector multiplication results in a serial element-wise manner. This serial output is written to the first-in-first-out storage unit 403 of the dedicated function unit 40 on one hand, and sent to the reduction unit 401 on the other. The normalization unit 402 of the dedicated function unit 40 reads the data to be normalized element by element from the storage unit and broadcasts the normalization result as a scalar input to the processing unit array 20 in outer product mode. The scheduling control module 10 is connected to the control terminals of the processing unit array 20, the hierarchical addition tree structure 30, and the dedicated function unit 40, respectively, and is used to generate mode configuration, read / write enable, statistical update, and operation start signals.

[0030] In a preferred embodiment, the scheduling control module 10 includes a finite state machine, an element counter, a vector length configuration register, and a handshake control circuit. It generates control signals based on the current computation stage to coordinate the data flow order among the processing unit array 20, the hierarchical addition tree structure 30, and the dedicated function unit 40, thereby achieving a serial element scheduling mode. Specifically, it receives the current operator type, computation stage, element number, vector length l, feature dimension d, valid output signal of the processing unit array, full / empty state of the first-in-first-out memory, and reduction statistics completion signal, and determines the order of data reading, computation, and write-back accordingly.

[0031] The control signals generated by the scheduling control module 10 include at least: the mode control signal of the processing unit, the inner product / outer product mode selection signal, the local accumulator and partial sum transfer selection signal, the accumulator register clear and processing unit shutdown signal, the first-level and second-level addition tree path enable signal, the first-in-first-out memory read / write enable signal, the reduction unit and normalization unit start signal, and the maximum value, exponential accumulator, element sum and element square sum reset and update signals.

[0032] Taking the Softmax operation in the attention mechanism as an example, the scheduling control module 10 first configures the processing unit array to inner product mode and clears the relevant registers; whenever the array outputs a valid value, it simultaneously activates memory writing and reduction statistics updates; after the element count reaches the sequence length, it latches the final maximum value and the exponential sum, and switches the array to outer product mode; then it reads the temporarily stored values ​​element by element, so that the normalization unit generates and broadcasts them to the processing unit array, and performs multiplication and accumulation with the corresponding values; after all elements are processed, the final attention vector is output.

[0033] The aforementioned processing unit array 20 is used to perform matrix multiplication and addition operations in the attention calculation process, including inner product mode and outer product mode, and is the core computing module in the system that undertakes the main parallel computing tasks.

[0034] The hierarchical addition tree structure 30 is formed by the adders and part of the transfer path inside the processing unit in the inner product mode. It is used to reduce multiple product results corresponding to the same inner product step by step, thereby generating a matrix-vector multiplication output element. The cross-element global statistics required by Softmax or LayerNorm are obtained by the reduction unit of the dedicated function unit by further accumulating the above serial output.

[0035] The dedicated function unit 40 includes a reduction unit, a normalization unit, and a first-in-first-out (FIFO) memory for storing elements or data blocks to be normalized. The reduction unit receives the serial output generated by the processing unit array via a hierarchical addition tree and is used to update the maximum value, exponential sum, element-wise sum, or element-wise sum of squares. The normalization unit performs subtraction, exponentiation, division, and optional affine transformations on the elements read from the FIFO memory based on the final statistics, and sends the results element-wise to the next outer product mode processing unit array.

[0036] Compared with existing technologies, the key difference in this embodiment lies in the collaborative operation of the scheduling control module with the reconfigurable processing unit array, the hierarchical addition tree structure, and the dedicated function unit. This enables the processing unit array to switch between inner product mode and outer product mode during operation, and overlaps the reduction phase of nonlinear operations with the preceding matrix-vector multiplication and the normalization phase with the subsequent matrix-vector multiplication. Its technical contribution lies in the collaborative design of data flow, hardware structure, and scheduling mechanism.

[0037] Based on the above system embodiments, this application provides a vector processing method for nonlinear operations, which is implemented by the scheduling control module of the system. The nonlinear operation is divided into a reduction stage and a normalization stage, and is executed using element-wise serial scheduling. The nonlinear operation includes Softmax or LayerNorm operations; see [link to relevant documentation]. Figure 2 The flowchart shown illustrates the method, which specifically includes the following steps: In step S202, during the reduction phase, the control processing unit array performs the first matrix-vector multiplication in inner product mode and outputs the elements in the first result vector sequentially element by element; the synchronous control reduction unit receives each element in the first result vector and updates the global statistics online. The input operands for the first matrix-vector multiplication include: the first input vector and the first input matrix; The aforementioned "synchronous control" refers to the control process of simultaneously updating the global statistics online by the reduction unit during the execution of the first matrix-vector multiplication. In other words, the reception of the reduction unit and the execution of the first matrix-vector multiplication overlap in time.

[0038] Step S204: After the global statistics are ready, the normalization stage is entered. The processing unit array is switched to the outer product mode. The normalization unit is controlled to normalize the temporarily stored first result vector element by element based on the global statistics to generate the elements in the second result vector, and broadcast them to the processing unit array element by element. The processing unit array is synchronously controlled to receive the elements in the second result vector periodically, perform the second matrix-vector multiplication in the outer product mode, and output the third result vector.

[0039] In this embodiment, the input operands of the second matrix-vector multiplication include: a second result vector and a second input matrix; the above-mentioned processing unit array is a reconfigurable processing unit that can switch between outer product mode and inner product mode during operation.

[0040] The aforementioned "synchronization control" refers to the simultaneous execution of the second matrix-vector multiplication operation during the broadcasting of elements in the second result vector generated by the normalization unit. In other words, the output of the normalization unit and the second matrix-vector multiplication in the outer product mode of the processing unit array overlap in time.

[0041] The vector processing method described above is suitable for nonlinear operations in attention mechanisms. Taking the Softmax and LayerNorm operations in attention mechanisms as examples, the attention calculation process is accelerated based on the processing unit array, including the following steps: Step 1: Perform a unified modeling of the Softmax and LayerNorm operations in the attention mechanism, dividing the operations into a reduction phase and a normalization phase; Step 2: In the reduction phase, receive the serial output generated element-wise from the previous inner product pattern matrix-vector multiplication; for Softmax, update the maximum value and exponential summation based on the serial output; for LayerNorm, update the element-wise sum and element-wise sum of squares based on the serial output to obtain global statistics for subsequent normalization processing. Step 3: In the normalization stage, based on the global statistics, perform subtraction of the maximum value, exponentiation and division operations on each of the temporarily stored input elements, or perform subtraction of the mean, division of the standard deviation and optional affine transformation, and use the obtained normalized elements directly as the serial input of the next outer product pattern matrix vector multiplication. Step 4: Employ an element-sequential scheduling method to overlap the reduction phase with the previous matrix-vector multiplication in time, and to overlap the normalization phase with the subsequent matrix-vector multiplication in time. For the same vector, the normalization phase begins after the required global statistics are ready, thus hiding the delay of nonlinear operations while satisfying data dependencies.

[0042] Taking the Softmax operation as an example, the first input vector is the query vector q; the first input matrix is ​​the key matrix K; and the second input matrix is ​​the value matrix V. The key matrix K and the value matrix V are stored row-wise in a two-dimensional matrix format (l, d). Through the first matrix-vector multiplication in the inner product mode, the transpose matrix K of the query vector q and the key matrix K is obtained. T The multiplication operation between the two matrices yields the attention score vector s; the multiplication operation between the attention score vector s and the value matrix V is completed through the second matrix vector multiplication in the outer product mode, yielding the attention output vector O.

[0043] In the Softmax operation, the aforementioned first result vector is q×K. T The obtained attention score vector s, and the second result vector is the attention score vector after Softmax normalization. The third result vector is The attention output vector O is obtained by multiplying and adding the V matrix.

[0044] In practice, the data flow is as follows: the query vector and the key vector stored row by row enter the inner product mode processing unit array; the hierarchical addition tree structure is reduced to an attention score in each cycle; the maximum value and exponential sum required to update Softmax are temporarily stored and synchronously sent to the reduction unit; after the global statistics are ready, the normalization unit generates them element by element; the scheduling control module switches the processing unit array to outer product mode, multiplies it with the corresponding Value row vector and accumulates it in the local accumulation register of each processing unit, and finally obtains the attention output vector.

[0045] This embodiment provides a vector acceleration method for attention-based computation. By uniformly dividing the Softmax and LayerNorm operations into a reduction phase and a normalization phase, and employing an element-wise serial scheduling method: the reduction phase receives the element-wise serial output of the previous inner product pattern matrix-vector multiplication and executes it concurrently with the previous matrix-vector multiplication; after the global statistics calculation is completed, the normalization result is element-wise fed into the subsequent outer product pattern matrix-vector multiplication and executed concurrently with the subsequent matrix-vector multiplication. This hides most of the latency of nonlinear operations within the preceding and following matrix-vector multiplication processes, and continuously improves the utilization rate of the processing unit array and dedicated function units.

[0046] The processing unit array in this embodiment will be described in detail below: The aforementioned processing unit array includes a first type of processing unit and a second type of processing unit. The adder of the first type of processing unit has at least one input that receives the local multiplication result, and the other input that selects between the local accumulated value and the partial sum passed from the adjacent processing unit. The two inputs of the adder of the second type of processing unit can both come from other processing units and are used to form higher-level addition tree nodes in inner product mode. The inner product mode and outer product mode are implemented by jointly configuring the mode control logic of the first type of processing unit and the second type of processing unit, so that the same processing unit array can be reconfigured at runtime.

[0047] like Figures 3 to 5 As shown, each processing unit (PE) includes at least an input register, a weight register, a multiplier, an adder, an accumulator register, a multiplexer, and mode control logic. The input register and weight register provide operands to the multiplier, and the adder selects the local product, the local accumulated value, or a partial sum from an adjacent processing unit as input based on the current mode.

[0048] The adder of the type A processing unit (same as the first type processing unit) has at least one input that can receive the local multiplication result, and another input that can be selected between the local accumulated value and the partial sum passed from the adjacent processing unit. Therefore, it can perform local accumulation in the outer product mode and participate in the first-level reduction in the inner product mode.

[0049] The Type B processing unit (same as the Type II processing unit) further sets up partial selection and forwarding paths on the basis of the above, so that both inputs of the adder can come from other processing units, which are used to form higher-level adder tree nodes in inner product mode and complete further reduction between the outputs of the first-level adder tree.

[0050] Each processing unit includes mode control logic to switch between outer product and inner product calculation modes to adapt to the computational needs of different stages of attention computation. The mode control logic can be implemented using 2 or more bits of control signals to select states such as local partial sum accumulation, partial sum transfer, clearing the accumulation register, or shutting down the processing unit. By jointly configuring the modes of type A and type B processing units, the same array can be reconfigured to run between outer product and inner product modes.

[0051] Furthermore, the step 202 above, "controlling the processing unit array to perform the first matrix-vector multiplication in inner product mode and sequentially outputting the elements of the first result vector element by element," includes: The processing unit array is configured to inner product mode by the mode control signal; each processing unit of the processing unit array performs multiplication operations on the first input vector and the row vectors in the first input matrix in parallel on a periodic basis; the multiple products output in parallel by each processing unit are reduced step by step through the hierarchical addition tree structure; and the elements in the first result vector are output periodically in a serial manner.

[0052] See Figure 6 As shown, the hierarchical addition tree structure includes at least a first-level addition tree and a second-level addition tree. The addition nodes of the first-level and second-level addition trees are reconstructed from the adders and partial sum forwarding paths within the processing unit in inner product mode. Among them, the partial sum represents the intermediate result in the addition operation of the addition tree. The first-level addition tree is used to merge the product results generated by adjacent multipliers in parallel within each processing unit row or local processing unit group to form multiple local partial sums. The second-level addition tree is used to further merge the local partial sums output by the first-level addition tree to generate a complete inner product result in the current cycle.

[0053] like Figure 5 As shown, in inner product calculation mode, the processing unit array is used to perform multiplication between the query vector and the key matrix. Each cycle reads a row-by-row key vector. Each processing unit performs parallel computation It then performs spatial reduction using first- and second-level addition trees to generate an attention score for that cycle. That is, an element in the aforementioned first result vector.

[0054] For example, let the query vector be... Sequence length Key matrix and Value Matrix All are arranged in 3 rows and 4 columns Formatted storage. In the first cycle of the inner product mode, the four processing units calculate respectively. A hierarchical addition tree adds four products together to obtain... Read in the second and third cycles respectively and ,get and This process, through three cycles, forms a complete attention score vector. .

[0055] Generation of hierarchical addition tree structure A single output element belonging to matrix-vector multiplication, not a cross-element global statistic from Softmax or LayerNorm. Continuously generated. The reduction unit, which sends elements sequentially to the dedicated function unit, further updates the maximum value and exponential summation required by Softmax, or the element-wise sum and element-wise sum of squares required by LayerNorm. The next step is to "synchronously control the reduction unit to receive each element in the first result vector and update the global statistics online."

[0056] The step S202 above, "the synchronous control reduction unit receives each element in the first result vector and updates the global statistics online," includes: When the nonlinear operation is a Softmax operation, the global statistics include the maximum value and the exponential sum of each element in the first result vector. During the process of receiving each element in cycle by cycle, the reduction unit updates the current maximum value through the comparator, and after detecting the maximum value update, performs online scaling correction on the stored exponential sum, and then adds the exponential value of the current element to the exponential sum. When the nonlinear operation is LayerNorm, the global statistics include the sum of elements and the sum of squares of each element in the first result vector; the reduction unit updates the sum of elements through the accumulator and updates the sum of squares of elements through the multiply-accumulate unit during the process of receiving each element in cycle by cycle. After processing all elements, the reduction unit latches the final global statistics for use in the normalization stage. Specifically, it counts the elements of the received first result vector using an element counter. When the count reaches the sequence length value in the preset configuration register, it determines that the global statistics are ready and latches the final global statistics.

[0057] The step of "switching the processing unit array to outer product mode" in step S204 above includes: In response to the global statistics ready signal, an outer product mode selection signal is generated to control the multiplexers and mode control logic in the processing unit array, cutting off some transmission paths so that each processing unit can perform local accumulation independently.

[0058] Furthermore, the step S204 above, which involves "controlling the normalization unit to normalize the temporarily stored first result vector element by element based on global statistics to generate elements in the second result vector," includes: The normalization unit is activated by a normalization unit start signal, causing it to read the current element from the first result vector element by element from the buffer storage unit. The normalization unit subtracts the current element from the global statistics, performs a division operation on the subtraction result, and obtains the corresponding element in the second result vector. When the nonlinear operation is a Softmax operation, the normalization unit performs an exponential operation on the subtraction result after the subtraction operation and before the division operation. When the nonlinear operation is a LayerNorm operation, the method also includes performing an optional affine transformation on the result of the division operation.

[0059] Furthermore, the step S204 above, which involves "the synchronous control processing unit array receiving elements of the second result vector cycle by cycle, performing the second matrix-vector multiplication in outer product mode, and outputting the third result vector," includes: In each cycle, an element of the second result vector is broadcast to each processing unit in the processing unit array, and the corresponding row vector of the second input matrix is ​​read. Each processing unit multiplies the received element with the weight component in the corresponding row vector and accumulates the multiplication into its local accumulator register. After accumulating for multiple cycles, the accumulation ends by the processing unit turn-off signal, and the third result vector is jointly output by the local accumulator registers in each processing unit.

[0060] like Figure 4 As shown, in the outer product calculation mode, the processing unit array is used to perform the multiplication between the normalized attention score and the value matrix. Each cycle will assign a normalized attention score. Broadcast to each processing unit and read the corresponding Value row vector. Each processing unit is responsible for one output dimension, and will... The corresponding components are accumulated to the local accumulator register.

[0061] Specifically, Softmax obtains Then, the processing unit array switches to outer product mode. The first cycle broadcasts. (i.e., an element in the aforementioned second result vector) and read , No. Each processing unit will Write to the local accumulator register; broadcast in the second cycle. And accumulate The third cycle continues to accumulate. Ultimately, each processing unit receives... Together they form the output vector That is, the third result vector.

[0062] The inner product mode is characterized by the need for spatial partial summation and hierarchical addition tree reduction to achieve the product of products generated by different processing units; the outer product mode is characterized by each processing unit corresponding to a different output dimension and performing independent local accumulation. Since both modes read data row-by-row... or , Matrix and Matrix can be uniformly adopted Formatted storage, no need to modify at runtime Perform physical transpose on the matrix.

[0063] By jointly configuring the processing unit type and computation mode at runtime, the same processing unit array can reuse multiple stages in the attention computation, thereby reducing the overall hardware resource overhead.

[0064] like Figure 7 As shown, the Softmax and LayerNorm operations in attention computation are uniformly abstracted. This unified abstraction method divides both the Softmax and LayerNorm operations into a reduction stage and a normalization stage. The reduction stage performs statistical operations on the input vector, including maximum value calculation, cumulative summation, or mean and variance calculation, thereby generating global statistics. The normalization stage performs element-wise nonlinear operations on each element of the input vector based on the global statistics, generating a normalized output result.

[0065] For Softmax, let the input vector be... N represents the length of the input vector. The reduction phase calculates the maximum value. and exponential summation The normalization stage calculates the output result element by element. To ensure numerical stability and adapt to serial input, the reduction unit can be updated online: upon receiving... Later orders and order Finally obtained and .

[0066] For LayerNorm, the reduction phase can synchronously compute the element sum as elements arrive serially. Sum of squares of elements And from this, the mean is obtained. and standard deviation ; This represents a preset small positive number to prevent the value inside the square root from being negative; the normalization stage calculates the result element by element and outputs the result. In implementations that include affine parameters, another output result can be further calculated. ;in, , These are the parameters that are included in both operations.

[0067] The aforementioned "element-by-element scheduling" does not mean that the final normalization of the previous element is performed on the same vector before the global statistics are completed. For the same Softmax or LayerNorm vector, the normalization phase still begins after the required global statistics are ready. This means that the reduction unit receives the serial output of the previous inner product pattern matrix-vector multiplication one element per cycle, causing the reduction phase to overlap with the previous matrix-vector multiplication; the normalization unit then provides serial input to the next outer product pattern matrix-vector multiplication one element per cycle, causing the normalization phase to overlap with the next matrix-vector multiplication.

[0068] like Figure 8 As shown, the hardware configuration for Softmax or LayerNorm operations is based on the element serial scheduling method, which realizes the computational processing of the reduction and normalization stages, so that the scheduling overhead of nonlinear operations does not increase linearly with the vector length, thereby improving the overall computational efficiency.

[0069] With a Softmax input of length 4 For example Figure 7 The hardware execution process involves the inner product mode processing unit array outputting the four attention scores for four consecutive cycles. The reduction unit synchronously receives each element and writes the original elements or corresponding data blocks into the first-in, first-out memory in their original order. After the first element arrives, the current maximum value is updated to 1, and the exponential sum is 1; after the second element arrives, the maximum value is updated to 2, and the exponential sum is scaled and updated according to the online formula. After the third element arrives, the maximum value remains 2, and the exponent sum is updated to... After the fourth element arrives, the maximum value is updated to 3, and the final sum of exponents is... It is approximately 1.5530.

[0070] After all elements have been processed, the normalization unit reads 1, 2, 0, and 3 from the first-in-first-out memory in sequence and calculates them respectively. This yields normalized attention scores of approximately [0.0871, 0.2369, 0.0321, 0.6439]. The process requires only a constant number of comparators, exponentiation units, accumulators, and dividers, without the need for dedicated function units for each element in the vector.

[0071] Subsequently, the processing unit array is configured in outer product mode. In the first cycle, 0.0871 is broadcast to all processing units and read... Each processing unit calculates The corresponding component is recorded and written to the local accumulator register; in subsequent cycles, 0.2369, 0.0321, and 0.6439 are broadcast sequentially, and the corresponding values ​​are accumulated respectively. , and Ultimately obtained Therefore, each normalized unit generates one The outer product mode processing unit array can immediately consume the element without waiting for the complete normalized vector to be ready in parallel all at once.

[0072] For LayerNorm Figure 8 The reduction path can be reused as a serial accumulation path for element-wise sums and element-wise sums of squares, while the normalization path reuses subtractors and dividers for element-wise calculation. The result is then immediately fed into the subsequent outer product pattern matrix-vector multiplication. Thus, Softmax and LayerNorm can share the same element serial interface, storage structure, and scheduling mechanism.

[0073] Through the unified abstraction process described above, different nonlinear operators can adopt a consistent execution flow. Softmax and LayerNorm operations can share the same scheduling mechanism and hardware resources, solving the problem that global data dependencies exist between nonlinear operators such as Softmax and LayerNorm during attention computation, causing the processing unit array to be idle during the computation of dedicated function units, thus providing a foundation for subsequent serial scheduling of elements.

[0074] like Figure 9 As shown, this implementation uses an element-based serial scheduling method to uniformly schedule the reduction and normalization phases. Compared with the traditional serial scheduling method, the element-based serial scheduling method can significantly reduce the overall execution latency.

[0075] In the element-serial scheduling mode, the processing unit array and the dedicated function unit adopt an element-granular serial interface. For each valid output element generated by the previous inner product mode matrix-vector multiplication, the reduction unit synchronously receives and updates the statistics; after the statistics are completed, for each normalized element generated by the normalization unit, the next outer product mode matrix-vector multiplication synchronously receives and performs local multiplication-accumulation.

[0076] For the same input vector, the normalization operation depends on the vector’s final maximum value, exponential sum, mean, or standard deviation. Therefore, the normalization phase begins after the corresponding global statistics are ready. This implementation does not require the final normalization to be performed before the reduction of the same vector is completed, thus ensuring that the data dependencies of Softmax and LayerNorm are satisfied.

[0077] Through the above scheduling method, the reduction phase is overridden during the element-wise output of the preceding matrix-vector multiplication, and the normalization phase is overridden during the element-wise input of the subsequent matrix-vector multiplication, ensuring that the processing unit array and the dedicated function units are in operation simultaneously for most of the time. Since the subsequent outer product mode only requires obtaining one normalized scalar per cycle, configuring a set or a constant number of high-cost exponentiation and division units can match the data consumption rate of the processing unit array, reducing the overhead of the dedicated function units from increasing with parallel width to a constant level.

[0078] The vector processing method for nonlinear operations provided in this application has the following beneficial effects: (1) By using the serial scheduling method of elements, the utilization rate of the processing unit array during nonlinear operations is effectively improved; (2) By unifying the abstraction of Softmax and LayerNorm operations, the number of dedicated function units is reduced and the hardware resource reuse rate is improved; (3) Adapt to the computational requirements of different stages of attention computation through a runtime reconfigurable processing unit array; (4) Reduce overall hardware overhead and execution latency while ensuring computational accuracy.

[0079] Based on the above method embodiments, this application also provides a vector processing device for nonlinear operations, which divides the nonlinear operation into a reduction stage and a normalization stage, and uses element-sequential scheduling for execution. See [link to relevant documentation]. Figure 10As shown, the device includes: a reduction stage control module 102, used to control the processing unit array to perform a first matrix-vector multiplication in inner product mode during the reduction stage, and to output the elements of the first result vector element by element serially; synchronously control the reduction unit to receive each element of the first result vector and update the global statistics online; a normalization stage control module 104, used to enter the normalization stage after the global statistics are ready, and switch the processing unit array to outer product mode; control the normalization unit to normalize the temporarily stored first result vector element by element based on the global statistics, generate the elements of the second result vector, and broadcast them element by element serially to the processing unit array; synchronously control the processing unit array to receive the elements of the second result vector periodically, perform a second matrix-vector multiplication in outer product mode, and output a third result vector; wherein, the processing unit array is a reconfigurable processing unit that can switch between outer product mode and inner product mode during operation; the nonlinear operation includes: Softmax or LayerNorm operation; the input operands of the first matrix-vector multiplication include: a first input vector and a first input matrix; the input operands of the second matrix-vector multiplication include: a second result vector and a second input matrix.

[0080] Furthermore, the reduction stage control module 102 is used to configure the processing unit array into inner product mode through mode control signals; through each processing unit of the processing unit array, the first input vector and the row vector in the first input matrix are multiplied in parallel on a periodic basis, and the multiple products output in parallel by each processing unit are reduced step by step through a hierarchical addition tree structure, and the elements in the first result vector are output periodically in a serial manner.

[0081] Furthermore, the aforementioned hierarchical addition tree structure includes at least a first-level addition tree and a second-level addition tree; the addition nodes of the first-level and second-level addition trees are reconstructed from the adders and partial sum forwarding paths within the processing unit in inner product mode; wherein, the partial sum represents the intermediate result in the addition operation of the addition tree; the first-level addition tree is used to merge the product results generated by adjacent multipliers in parallel within each processing unit row or local processing unit group to form multiple local partial sums; the second-level addition tree is used to further merge the local partial sums output by the first-level addition tree to generate a complete inner product result in the current cycle.

[0082] Furthermore, the reduction stage control module 102 is configured such that, when the nonlinear operation is a Softmax operation, the global statistics include the maximum value and exponential sum of each element in the first result vector; during the process of receiving each element cycle by cycle, the reduction unit updates the current maximum value through a comparator, and after detecting the maximum value update, performs online scaling correction on the stored exponential sum, and then adds the exponential value of the current element to the exponential sum; when the nonlinear operation is a LayerNorm operation, the global statistics include the element sum and element square sum of each element in the first result vector; during the process of receiving each element cycle by cycle, the reduction unit updates the element sum through an accumulator and updates the element square sum through a multiply-accumulate unit; after all elements have been processed, the reduction unit latches the final global statistics for use in the normalization stage.

[0083] Furthermore, the reduction stage control module 102 is used to count the elements of the received first result vector using an element counter. When the count value reaches the sequence length value in the preset configuration register, it determines that the global statistics are ready and latches the final global statistics.

[0084] Furthermore, the normalization stage control module 104 is used to generate an outer product mode selection signal in response to the global statistics ready signal, control the multiplexer and mode control logic in the processing unit array, cut off part and transmission path, and enable each processing unit to perform local accumulation independently.

[0085] Furthermore, the normalization stage control module 104 is used to activate the normalization unit through a normalization unit start signal, so that the normalization unit reads the current element in the first result vector element by element from the buffer storage unit; the normalization unit performs a subtraction operation on the current element and a division operation on the subtraction result to obtain the corresponding element in the second result vector; wherein, when the nonlinear operation is a Softmax operation, the normalization unit performs an exponential operation on the subtraction result after the subtraction operation and before the division operation; when the nonlinear operation is a LayerNorm operation, the method also includes performing an optional affine transformation on the result of the division operation.

[0086] Furthermore, the normalization stage control module 104 described above is used to broadcast an element of the second result vector to each processing unit of the processing unit array in each cycle, and read the corresponding row vector of the second input matrix; each processing unit multiplies the received element with the weight component in the corresponding row vector, and accumulates the multiplication into its own local accumulator register; after accumulating for multiple cycles, the accumulation is ended by the processing unit turn-off signal, and the third result vector is jointly output by the local accumulator registers in each processing unit.

[0087] Furthermore, the aforementioned processing unit array includes a first type of processing unit and a second type of processing unit; the adder of the first type of processing unit has at least one input that receives the local multiplication result, and the other input that selects between the local accumulated value and the partial sum passed from the adjacent processing unit; the two inputs of the adder of the second type of processing unit can both come from other processing units, and are used to form higher-level addition tree nodes in inner product mode; the inner product mode and the outer product mode are implemented by jointly configuring the mode control logic of the first type of processing unit and the second type of processing unit, so that the same processing unit array can be reconfigured at runtime.

[0088] Furthermore, the first input vector is the query vector; the first input matrix is ​​the key matrix; the second input matrix is ​​the value matrix; the key matrix and the value matrix are uniformly stored row-wise in a two-dimensional matrix format. The multiplication operation between the query vector and the transpose of the key matrix is ​​completed by the first matrix-vector multiplication in the inner product mode to obtain the attention score vector; the multiplication operation between the attention score vector and the value matrix is ​​completed by the second matrix-vector multiplication in the outer product mode to obtain the attention output vector.

[0089] The device provided in this application embodiment has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts of the device embodiment not mentioned can be referred to the corresponding content in the aforementioned method embodiment.

[0090] This application also provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions cause the processor to implement the above-described method. For specific implementation details, please refer to the foregoing method embodiments, which will not be repeated here.

[0091] The computer program products of the methods, apparatus, and electronic devices provided in the embodiments of this application include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementations, please refer to the method embodiments, which will not be repeated here.

[0092] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application.

[0093] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0094] In the description of this application, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0095] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the technical scope disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A vector processing method for nonlinear operations, characterized in that, The nonlinear operation is divided into a reduction phase and a normalization phase, and is executed using element-wise serial scheduling, including: In the reduction phase, the control processing unit array performs the first matrix-vector multiplication in inner product mode and outputs the elements of the first result vector element by element in sequence; the synchronous control reduction unit receives each element of the first result vector and updates the global statistics online. After the global statistics are ready, the normalization stage is entered, and the processing unit array is switched to outer product mode. The normalization unit is controlled to normalize the temporarily stored first result vector element by element based on the global statistics to generate the elements in the second result vector, and broadcast them to the processing unit array element by element. The processing unit array is simultaneously controlled to receive the elements in the second result vector periodically, perform the second matrix-vector multiplication in outer product mode, and output the third result vector. The processing unit array is a reconfigurable processing unit that can switch between the outer product mode and the inner product mode during operation; the nonlinear operation includes Softmax or LayerNorm operation; the input operands of the first matrix-vector multiplication include a first input vector and a first input matrix; the input operands of the second matrix-vector multiplication include a second result vector and a second input matrix.

2. The method according to claim 1, characterized in that, The steps of controlling the processing unit array to perform the first matrix-vector multiplication in inner product mode and sequentially outputting the elements of the first result vector element by element include: The processing unit array is configured in inner product mode by a mode control signal; Each processing unit in the processing unit array performs a multiplication operation on the first input vector and the row vectors in the first input matrix in parallel on a periodic basis. The multiple products output in parallel by each processing unit are reduced step by step through a hierarchical addition tree structure, and the elements in the first result vector are output periodically in a serial manner.

3. The method according to claim 2, characterized in that, The hierarchical addition tree structure includes at least a first-level addition tree and a second-level addition tree; the addition nodes of the first-level addition tree and the second-level addition tree are reconstructed by the adders inside the processing unit and the partial sum forwarding path in inner product mode; wherein, the partial sum represents the intermediate result in the addition operation of the addition tree; The first-level addition tree is used to merge the product results of adjacent multipliers in parallel within each processing unit row or local processing unit group to form multiple local partial sums; The second-level adder tree is used to further merge the partial sums of the output of the first-level adder tree to generate a complete inner product result in the current cycle.

4. The method according to claim 1, characterized in that, The synchronous control reduction unit receives each element from the first result vector and updates the global statistics online, including: When the nonlinear operation is a Softmax operation, the global statistics include the maximum value and exponential sum of each element in the first result vector; during the process of receiving each element periodically, the reduction unit updates the current maximum value through a comparator, and after detecting the maximum value update, performs online scaling correction on the stored exponential sum, and then adds the exponential value of the current element to the exponential sum. When the nonlinear operation is LayerNorm operation, the global statistics include the sum of elements and the sum of squares of each element in the first result vector; the reduction unit updates the sum of elements through an accumulator and updates the sum of squares of elements through a multiply-accumulate unit during the process of receiving each element periodically. After all elements have been processed, the reduction unit latches the final global statistics for use in the normalization stage.

5. The method according to claim 4, characterized in that, The step of latching the final global statistics after all elements have been processed by the reduction unit includes: The elements of the received first result vector are counted by an element counter. When the count value reaches the sequence length value in the preset configuration register, the global statistics are determined to be ready, and the final global statistics are latched.

6. The method according to claim 1, characterized in that, Switching the processing unit array to outer product mode includes: In response to the global statistics ready signal, an outer product mode selection signal is generated to control the multiplexers and mode control logic in the processing unit array, cutting off some transmission paths so that each processing unit can perform local accumulation independently.

7. The method according to claim 1, characterized in that, The step of controlling the normalization unit to normalize the temporarily stored first result vector element by element based on the global statistics to generate the elements in the second result vector includes: The normalization unit is activated by a normalization unit start signal, causing the normalization unit to read the current element in the first result vector element by element from the buffer storage unit; the normalization unit subtracts the current element from the global statistics, performs a division operation on the subtraction result, and obtains the corresponding element in the second result vector; Wherein, when the nonlinear operation is a Softmax operation, the normalization unit performs an exponential operation on the subtraction result after the subtraction operation and before the division operation; when the nonlinear operation is a LayerNorm operation, the method further includes performing an optional affine transformation on the result of the division operation.

8. The method according to claim 1, characterized in that, The steps of synchronously controlling the processing unit array to receive elements of the second result vector cycle by cycle, performing a second matrix-vector multiplication in outer product mode, and outputting a third result vector include: In each cycle, an element of the second result vector is broadcast to each processing unit of the processing unit array, and the corresponding row vector of the second input matrix is ​​read. Each processing unit multiplies the received element with the weight component in the corresponding row vector and accumulates the multiplication into its local accumulator register. After accumulating for multiple cycles, the accumulation ends by the processing unit turn-off signal, and the third result vector is jointly output by the local accumulator registers in each processing unit.

9. The method according to claim 1, characterized in that, The processing unit array includes a first type of processing unit and a second type of processing unit; The adder of the first type of processing unit has at least one input that receives the local multiplication result, and another input that selects between the local accumulated value and the partial sum passed from the adjacent processing unit; The two inputs of the adder in the second type of processing unit can both come from other processing units, and are used to form higher-level addition tree nodes in inner product mode; The inner product mode and outer product mode are implemented by jointly configuring the mode control logic of the first type of processing unit and the second type of processing unit, so that the same processing unit array can be reconfigured at runtime.

10. The method according to claim 1, characterized in that, The first input vector is a query vector; the first input matrix is ​​a key matrix; the second input matrix is ​​a value matrix; the key matrix and the value matrix are uniformly stored row-wise in a two-dimensional matrix format. The multiplication operation between the query vector and the transpose of the key matrix is ​​completed by the first matrix-vector multiplication in the inner product mode to obtain the attention score vector; the multiplication operation between the attention score vector and the value matrix is ​​completed by the second matrix-vector multiplication in the outer product mode to obtain the attention output vector.

11. A vector processing device for nonlinear operations, characterized in that, The nonlinear operation is divided into a reduction stage and a normalization stage, and is executed using element-sequential scheduling. The device includes: The reduction stage control module is used to control the processing unit array to perform a first matrix-vector multiplication in inner product mode during the reduction stage, and to output the elements of the first result vector element by element in sequence; and to synchronously control the reduction unit to receive each element of the first result vector and update the global statistics online. The normalization stage control module is used to enter the normalization stage after the global statistics are ready, switch the processing unit array to the outer product mode, control the normalization unit to normalize the temporarily stored first result vector element by element based on the global statistics, generate the elements in the second result vector, and broadcast them to the processing unit array element by element serially; synchronously control the processing unit array to receive the elements in the second result vector periodically, perform the second matrix-vector multiplication in the outer product mode, and output the third result vector. The processing unit array is a reconfigurable processing unit that can switch between the outer product mode and the inner product mode during operation; the nonlinear operation includes Softmax or LayerNorm operation; the input operands of the first matrix-vector multiplication include a first input vector and a first input matrix; the input operands of the second matrix-vector multiplication include a second result vector and a second input matrix.

12. A vector processing system for nonlinear operations, characterized in that, The system includes: a scheduling control module, a processing unit array, a hierarchical addition tree structure, and dedicated function units; the dedicated function units include: a reduction unit, a normalization unit, and a storage unit; the scheduling control module is connected to the processing unit array, the hierarchical addition tree structure, and the dedicated function units respectively; the processing unit array, the hierarchical addition tree structure, and the dedicated function units are connected sequentially; the scheduling control module is used to execute the method as described in any one of claims 1-10.

13. The system according to claim 12, characterized in that, The scheduling control module includes a finite state machine, an element counter, a vector length configuration register, and a handshake control circuit, which are used to generate control signals according to the current calculation stage and coordinate the data flow order between the processing unit array, the hierarchical addition tree structure, and the dedicated function units.

Images