A transformer neural network system and an operation method thereof

Abstract

The application discloses a kind of Transformer neural network system and its operation method, model includes: sequence feature extraction algorithm module, joint multi-head attention algorithm module, efficient calculation algorithm module, Transformer hardware acceleration algorithm module and judgment, control and scheduling system method algorithm module.The application can not only design the more excellent AI processing core of terminal application, SOC integration, or the technology brought by the development of newer algorithm module on similar platform, but also directly use the method of the application, use current hardware resources, and play the advantage of Transformer.The application can overcome the complexity of quadratic complexity of storage space and calculation time, and speed up hardware operation speed, and can be widely applied in artificial intelligence field.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a Transformer neural network system and its computation method. Background Technology

[0002] The Transformer network architecture, proposed in recent years, has demonstrated powerful expressive capabilities and processing performance in natural language processing tasks. However, this application has encountered bottlenecks. The two main modules of the model method are the multi-head attention residual module and the forward prediction residual module. While the computational complexity of attention can be overcome, this alone is not enough. The system's performance is also related to the performance of hardware such as processors and GPUs.

[0003] The Transformer model has achieved great success in the field of natural language processing, and its application in fields such as autonomous driving and robotics has shown great potential. However, the current technology has the following technical problems: there is a lack of algorithms that reduce storage and computational complexity from network feature extraction to attention calculation, specifically tailored to the characteristics of visual images; and improvements in front-end algorithms, when deployed on hardware platforms, do not achieve overall performance improvements. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a Transformer neural network system and its operation method, which overcomes the complexity of the square of storage space and computation time, and accelerates the hardware operation speed.

[0005] One aspect of this invention provides a Transformer neural network system, comprising:

[0006] The sequence feature extraction algorithm module is used to extract features from a sequence of signals by means of a backbone neural network, and output sparse N:M neural network weights.

[0007] A multi-head attention algorithm module is used to output the input signal sequence as a dynamic attention matrix obtained after passing through a sparse neural network. The dynamic attention matrix is ​​obtained by calculating the correlation between the Q, K, and V attention matrices.

[0008] The high-efficiency computing algorithm module is used to schedule and organize operations based on computational characteristics, judge, schedule and control input signals, and perform feature extraction operations with reduced complexity based on dense computing engines or sparse computing engines; it is also used to process multi-head attention, and perform sparse operations of attention N:M with reduced complexity on the input dynamic attention matrix Q, K, V based on judgment, scheduling and control signals and dense computing engines or sparse computing engines.

[0009] The Transformer hardware acceleration algorithm module is used to handle instruction and job scheduling between hardware computing and data transfer.

[0010] The judgment, control, and scheduling system algorithm module is used to schedule and coordinate efficiency according to preset rules based on the system status of each algorithm module, and output the control and scheduling of each algorithm module.

[0011] Another aspect of this invention provides a Transformer neural network system, comprising:

[0012] The data storage module includes an input matrix storage unit, a weight matrix storage unit, and a temporary matrix storage unit. The input end of the input matrix storage unit serves as the input interface of the data storage module.

[0013] Tensor operation kernel, used to perform operations on the input tensors;

[0014] The input signal backbone neural network processing module outputs the input image signal to be processed as a feature extraction signal, which serves as the tensor input of the sequence signal to the Transformer.

[0015] The multi-head attention processing module outputs the sequence tensor of the input Transformer as an attention matrix;

[0016] The feedforward neural network processing module outputs the normalized evaluation of the input multi-head attention as a vector encoded from the sequence.

[0017] The high-efficiency computing module is used to convert the input image series and neural network output into N:M sparse network sparse features; it is also used to convert the input dynamic attention matrix, which is obtained by the correlation between Q, K, and V attention matrices, into a sparse N:M attention matrix that meets the hardware acceleration conditions.

[0018] The Transformer hardware acceleration processing module is used to output the input irregular sparse attention matrix as a structured sparse matrix conforming to N:M; it is also used to output the input Q and K matrices as matrix pairs that satisfy preset data flow characteristics; it is also used to compress the Q and K features using the Autoencoder and output the compressed result to the external storage unit; it is also used to load the Q and K features from the external storage, perform reverse encoding, obtain the result in the on-chip storage, and complete the specified operation under the control module.

[0019] The residual addition and normalization operation module includes a residual addition operation module and a layer normalization function operation module. The residual addition module includes multiple adders, the number of which is consistent with the number of rows of any input matrix in the multi-head attention layer. The input of the residual addition module is connected to the output of the input matrix storage unit, and the output is connected to the input of the layer normalization function operation module. The output of the layer normalization function operation module serves as the output interface.

[0020] The computing engine array comprises multiple configurable computing engines and an on-chip cache module. Each computing engine includes configurable levels of multi-precision adders, accumulators, nonlinear operation units, and multiple multi-precision parallel multipliers. The computing engine array is used to execute linear layer operations, nonlinear layer operations, single-head attention layers, multi-head attention layers, fully connected layer operations, and residual connection operations in parallel. Operations include dense matrix / tensor computation, sparse matrix / tensor computation, and their mixed operations. The on-chip cache module includes a weight cache area, a parameter cache area, and multiple feature cache areas. The computing engine array is connected to off-chip storage via the on-chip cache module, which is used to implement the connection between the computing engine array and off-chip storage. Data interaction between storage units; the nonlinear layer includes a Softmax operation module and a normalization operation module. The input terminals of both the Softmax and normalization operation modules are connected to the output terminals of the computing engine array, and both are connected to the on-chip cache module; the Softmax and normalization operation modules are used to execute the nonlinear layer operation control module; the computing engine array, Softmax operation module, and normalization operation module are all connected to the control module; the control module is used to perform operation configuration and memory access logic scheduling control for the computing engine array, Softmax operation module, and LayerNorm operation module.

[0021] The judgment, scheduling, and control system method module serves as the control module. This control module connects to and controls the data storage module, the input backbone neural network processing module, the joint multi-head attention processing module, the feedforward neural network processing module, the residual and addition module, the layer normalization function operation module, the data access module, the tensor operation kernel, the high-efficiency calculation module, and the Tramsformer acceleration processing module.

[0022] Another aspect of this invention provides a computation method for a Transformer neural network system, comprising:

[0023] Set a sparse objective, determine that in each set of M weights, the maximum number of non-zero weights is N, and set the objective function for learning and training; calculate the gradient of the continuous parameter set spatial subnet, and perform backpropagation gradient calculation of the network.

[0024] Determine the approximate objective function; select a random matrix; train an approximate matrix that satisfies the dynamic N:M sparsity requirement; predict the self-attention matrix of dynamic N:M using the approximate matrix; and dynamically select the weights of the attention matrix of N:M based on the approximate matrix.

[0025] Another aspect of this invention provides a computation method for a Transformer neural network system, comprising:

[0026] Each individual attention evaluation tensor is concatenated into a single overall tensor; this overall tensor is then subjected to a one-dimensional convolution to obtain a new tensor; the joint multi-head attention coefficients are determined based on this new tensor.

[0027] Calculate the cross-attention coefficient matrix, split the cross-attention coefficient matrix into multiple tensors, and calculate the cross-attention matrix between multiple layers on the same side according to the multi-head attention calculation process;

[0028] Calculate the joint cross-attention between the encoder and decoder sides based on the dynamic attention matrices Q, K, and V.

[0029] Another aspect of this invention provides a computation method for a Transformer neural network system, comprising:

[0030] The objective function is determined; the encoder input sequence is embedded vectors Q and K, which are used as the input to the neural network; the decoder input is the output of the encoder; the encoded output data of the encoder and decoder is stored in the external memory; the decoded data is read from the external memory, and the decoded vectors are stored in the on-chip memory.

[0031] By leveraging the sparsity of the Q and K matrices of the image, data is stored using the K-Stationary Dataflow approach; and by utilizing the sparsity of the Q and K matrices of the image and the operational order of the Transformer, hardware kernels are scheduled to perform matrix operations.

[0032] Encoding is performed using a preset method; the order of matrix operations is selected based on the data stream.

[0033] Another aspect of this invention provides a computing device for a Transformer neural network system, comprising:

[0034] The first operational unit is used to set the sparsity objective, determine that in each set of weights of number M, the maximum number of non-zero weights is N, set the objective function for learning and training, calculate the gradient of the continuous parameter set spatial subnet, and perform backpropagation gradient calculation of the network.

[0035] The second operational unit is used to determine the approximate objective function; select a random matrix; train an approximate matrix that satisfies the dynamic N:M sparsity requirement; predict the self-attention matrix of dynamic N:M using the approximate matrix; and dynamically select the weights of the attention matrix of N:M based on the approximate matrix.

[0036] Another aspect of this invention provides a computing device for a Transformer neural network system, comprising:

[0037] The third computational unit is used to concatenate the individual attention evaluation tensors of each head into a single overall tensor; to perform a one-dimensional convolution on the overall tensor to obtain a new tensor; and to determine the joint multi-head attention coefficients based on the new tensor.

[0038] The fourth operation unit is used to calculate the cross-attention coefficient matrix, split the cross-attention coefficient matrix into multiple tensors, and calculate the cross-attention matrix between multiple layers on the same side according to the multi-head attention calculation process.

[0039] The fifth computational unit is used to calculate the joint cross-attention between the encoding and decoding sides based on the dynamic attention matrices Q, K, and V.

[0040] Another aspect of this invention provides a computing device for a Transformer neural network system, comprising:

[0041] The sixth arithmetic unit is used to determine the objective function; the encoder input sequence is embedded vectors Q and K, which are used as the input to the neural network; the decoder input is the output of the encoder; the encoded output data of the encoder and decoder is stored in the external memory; the decoded data is read from the external memory, and the decoded vector is stored in the on-chip memory.

[0042] The seventh operation unit is used to store data using the sparsity of the Q and K matrices of the image in the K-StationaryDataflow manner; and to schedule the hardware kernel to perform matrix operations by utilizing the sparsity of the Q and K matrices of the image and the operation order of the Transformer.

[0043] The eighth operation unit is used to encode using a preset method and to select the order of matrix operations based on the data stream.

[0044] Another aspect of the present invention provides an electronic device, including a processor and a memory;

[0045] The memory is used to store programs;

[0046] The processor executes the program to implement the above-described method.

[0047] Another aspect of this invention provides a computer-readable storage medium storing a program that is executed by a processor to implement the above-described method.

[0048] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the method described above.

[0049] This invention addresses the characteristics of visual images by proposing an algorithmic approach to reduce the complexity of the feature extraction network and subsequent attention operations. It also proposes a control and scheduling method for coordinated improvement of the algorithm and hardware, and a novel Transformer model architecture based on these solutions. This invention solves the problem of matching hardware resources, leveraging overall performance advantages. The method of this invention provides technical support for designing superior AI processing cores and SOC integrations for terminal applications, or developing newer algorithm modules on similar platforms. It also allows for direct application of the method to current hardware resources, maximizing the advantages of the Transformer. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This invention provides an algorithm module structure diagram of a Transformer neural network system according to an embodiment of the present invention.

[0052] Figure 2 A structural module diagram of a Transformer neural network system provided in an embodiment of the present invention;

[0053] Figure 3 This is an example diagram illustrating an application of self-attention processing provided by an embodiment of the present invention;

[0054] Figure 4 This is an example diagram illustrating an application of cross-attention processing provided by an embodiment of the present invention;

[0055] Figure 5 A schematic diagram of an encoding process provided in an embodiment of the present invention;

[0056] Figure 6A schematic diagram of an attention score input provided in an embodiment of the present invention;

[0057] Figure 7 and Figure 8 This is an example scenario diagram provided for an embodiment of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0059] Reference Figure 1 This invention provides a Transformer neural network system, specifically including:

[0060] 1.1 Sequence Feature Extraction Algorithm Module: The sequence feature extraction takes a sequence of signals as input, performs feature extraction through a pre-defined neural network, and outputs sparse N:M neural network weights. Here, sparse structured computation can refer to sparse N:M operations.

[0061] 1.2 Joint Multi-Head Attention Algorithm Module: The joint multi-head attention algorithm module takes an input signal sequence as input and outputs a dynamic attention matrix Q, K, V obtained after passing through a sparse neural network. The dynamic attention matrix can be obtained by finding and calculating the correlations between the attentions of the Q, K, and V heads. The dynamic attention matrix can be calculated using the following expression:

[0062]

[0063] 1.3 High-efficiency computation module: If the module is handling feature extraction, its input is signals—judgment, scheduling, and control signals. Under the action of either a dense or sparse computation engine, it performs feature extraction operations with reduced complexity while ensuring model accuracy. If it is handling multi-head attention, its input is the attention matrix Q, K, and V signals. Under the action of either a dense or sparse computation engine, it performs attention sparse operations with reduced complexity while ensuring model accuracy.

[0064] 1.4 The Transformer hardware acceleration algorithm module takes as its input the sparsity of the attention matrix and the characteristics of efficient dataflow. Its role is to determine how to leverage hardware performance in hardware computation and data transfer through instructions, job scheduling, etc.

[0065] 1.5 Judgment, Control & Scheduling System Method Algorithm Module: Its input is based on the system state of the relevant component module or algorithm module, and its output is the control, scheduling, etc. in the aforementioned component module or algorithm module, which coordinates and schedules time according to relevant methods.

[0066] Reference Figure 2 This invention provides another Transformer neural network system, specifically including:

[0067] 2.1 Data storage module, including input matrix storage unit, weight matrix storage unit, and temporary matrix storage unit, wherein the input terminal of the input matrix storage unit serves as the input interface of the relevant modules.

[0068] 2.2 Tensor Operation Kernel, including GEMM, SDDMM, and SpMM tensor processing operations, receives input tensors from the backbone neural network processing module, the joint multi-head attention processing module, the feedforward neural network processing module, the residual addition and layer normalization operation module, the high-efficiency computation processing module, and the Transformer hardware acceleration processing module, respectively. The operation structure can be either a statement issued by this module and returned to this module, or it can be related to modules in the system under control and scheduling signal processing.

[0069] 2.3 Input signal backbone neural network processing module, its input is the image signal to be processed, its output is the feature extraction signal, which is the tensor input of the sequence signal as the input of the Transformer.

[0070] 2.4. The multi-head attention processing module takes a Transformer sequence tensor as input and outputs an attention matrix, which, after normalization, becomes the normalized evaluation score of the attention.

[0071] 2.5. The feedforward neural network processing module takes the normalized evaluation of multi-head attention as input and outputs the encoded vector of the sequence.

[0072] 2.6 High-efficiency computing module: If it is to process sparse networks, the input is an image series and a neural network, and the output is the sparse features of the N:M sparse network. If it is to process sparse attention, the input is Q, K, V, and the output is the sparse N:M attention matrix that meets the hardware acceleration conditions.

[0073] 2.7 The Transformer hardware acceleration module, when processing sparse computation tasks, takes an irregular sparse attention matrix as input and outputs a structured sparse matrix conforming to the N:M ratio. When processing parallel multiplexing of units, its input is the Q and K matrices, and its output is a pair of corresponding matrices that meet the timing alignment requirements of the K-Stationary for attention computation. When balancing the computational load between structured and unstructured sparse computations, it processes the compression of Q and K features using an Autoencoder and outputs the compressed result to external storage. To meet the attention computation requirements, the Q and K features are loaded from external storage, reverse encoded, and the result is obtained in on-chip storage before being processed by the control module. The hardware's ability to perform the aforementioned computational collaboration requires that the hardware acceleration conform to dataflow characteristics. This embodiment of the invention can employ various implementation methods that meet timing requirements; optionally, this embodiment can adopt the timing requirements of the K-Stationary.

[0074] 2.8 Residual Addition and Normalization Module: The residual addition and normalization module includes a residual addition module and a layer normalization function operation module. The residual addition module includes multiple adders, the number of which is consistent with the number of rows in any input matrix of the multi-head attention layer. The input terminal of the residual addition module is connected to the output terminal of the input matrix storage unit, and the output terminal is connected to the input terminal of the layer normalization function operation module. The output terminal of the layer normalization function operation module serves as the output interface of the module.

[0075] 2.9 A computing engine array, comprising multiple configurable parallel computing engines and on-chip caches for high-speed inference; the computing engines include configurable multi-precision adders, accumulators, ReLU units, and multiple multi-precision parallel multipliers; the computing engine array is used to perform linear layer operations, nonlinear layer operations, single-head attention layers, multi-head attention operations, fully connected layer operations, and residual connection operations in parallel; the operations include dense matrix / tensor computation (GEMM), sparse matrix / tensor computation, and their hybrid operations (SDMM / SpMM); the on-chip cache module includes a weight cache, a parameter cache, and multiple feature caches; the computing engine array is connected to off-chip storage through the on-chip cache module, which is used to realize data interaction between the computing engine array and the off-chip storage; the nonlinear layer includes Soft The Softmax and Normalization (Norm) operation modules are connected, with their input terminals both connected to the output terminals of the computing engine array. Both modules are also connected to the on-chip cache module. These modules are used to execute nonlinear layer operation control modules. The computing engine array, the Softmax module, and the Normalization (Norm) module are all connected to the control module, which performs operation configuration and memory access logic scheduling control on the computing engine array, the Softmax module, and the LayerNorm module. These connections can be either hardware connections or software method message transmissions.

[0076] 2.10 Judgment, scheduling and control system method module, as control module, the control module is connected to and controls the data storage module, input backbone neural network processing module, joint multi-head attention processing module, feedforward neural network processing module, residual and addition module and layer normalization function operation module, data storage and retrieval module, tensor operation kernel, high-efficiency calculation module and Tramsformer acceleration processing module respectively.

[0077] The efficient computation process in the aforementioned Transformer neural network system is shown below:

[0078] 3.1 If the backbone network for feature extraction is subjected to dynamic sparse compression N:M, then it includes:

[0079] 3.1.1. Based on the hardware-friendly characteristics, a sparsity objective is set, determining that in each group of weights with a value of M, the maximum number of non-zero values ​​is N. The objective function for learning and training is set as follows:

[0080]

[0081] in To observe the data, For loss function, For the network parameters of layer L, For N:M sparse network parameters.

[0082] 3.1.2, If we assume: It is a continuous parameter set. This represents the parameter set required for the corresponding N:M. Let ξ be the threshold for the maximum ξ-th in this embodiment of the invention, and w = {|w1|, |w2|, ..., |w...} M |}, Algorithm search is arrive The mapping space satisfies the threshold ξ of N:M such that:

[0083]

[0084] 3.1.3 Calculate based on the Straight Through Estimator (STE). The gradient of the spatial subnet is calculated through backpropagation of the network. The calculation formula is as follows:

[0085]

[0086]

[0087] Where γt is the learning rate, λW is a user-defined hyperparameter representing the relative weights of the refined expression for the sparse iteration of the tightly connected units, and represents the dense weights for the t-th iteration. The dense weights in the t-th iteration are represented by [the value]. The mask represents the pruning weights. ⊙ represents the Hadmard product.

[0088] 3.1.4. Following the iterative method described above, complete the network training to achieve structured N:M network sparsity.

[0089] 3.2. For multi-head attention matrices, the dynamic N:M sparsity requirement includes:

[0090] 3.2.1 Determine the approximate objective function:

[0091]

[0092] 3.2.2 Choosing a random matrix:

[0093]

[0094] 3.2.3 Training an approximate matrix that satisfies the dynamic N:M sparsity requirement:

[0095]

[0096] 3.2.4 Predicting the dynamic N:M self-attention matrix using an approximation matrix:

[0097]

[0098] 3.2.5. Based on the above approximate matrix, dynamically select the weights of the N:M attention matrix.

[0099] The joint multi-head attention calculation process in the aforementioned Transformer neural network system is shown below:

[0100] 4.1 Single-layer joint multi-head self-attention algorithm, including:

[0101] 4.1.1. Concatenate the individual attention evaluation tensors into a single unified tensor, as shown in the following expression:

[0102]

[0103] in The shape of the tensor (3-D Tensor Shape) is nXmXh.

[0104] 4.1.2. By performing a one-dimensional convolution on the above tensor, a new tensor is obtained. So:

[0105]

[0106] in: The kernel is learnable.

[0107] 4.1.3. Therefore, the joint tensor of so many heads can be represented as:

[0108]

[0109] Among them, σ, It is a nonlinear function.

[0110] 4.1.4, the above... Further splitting into h separate attention coefficients yields new... Take each Inputting each into Softmax yields the normalized result. The final joint bullish attention coefficient is as follows:

[0111]

[0112]

[0113]

[0114] 4.2 Joint multi-head, multi-layer cross-attention algorithm, including:

[0115] 4.2.1 Considering the multi-head, multi-layer correlation between the k-th layer and the l-th layer, the cross-attention coefficient matrix can be calculated according to the following relationship:

[0116]

[0117] in: Cl stands for CrossLayer, while Hk is a non-linear transformation function.

[0118] 4.2.2, Put Split into h or more heads Right now:

[0119]

[0120] New Following the multi-head attention calculation process, the subsequent calculation operations are completed to obtain the cross-attention matrix between multiple layers on the same side.

[0121] 4.3 If it is a joint cross-attention between encoding and decoding, then:

[0122] 4.3.1 Its K and V come from the encoding side, while Q comes from the decoding side.

[0123] 4.3.2 The rules of its operation matrix are the same as those of the self-attention algorithm.

[0124] 4.4 If it involves multiple layers of cross-interaction on the encoding side, and then cross-interaction with a certain layer on the decoding side, then it includes:

[0125] 4.4.1 The crossover on its encoding side follows the calculation steps in 4.2.

[0126] 4.4.2 The final output is obtained in accordance with the method in 4.3.

[0127] 4.5 If the code involves intersecting with the history of a previous layer on the decoding side, and then intersecting with the current layer on the decoding side, then it includes:

[0128] 4.5.1 The history of its decoding side involves multiple layers of crossover, and its K and V originate from the decoding side.

[0129] 4.5.2 When performing cross-attention calculations on the current layer on the decoding side, its Q comes from this layer.

[0130] The hardware-accelerated computation process of Transformer in the aforementioned Transformer neural network system is shown below:

[0131] 5.1 To reduce data movement between off-chip memory and the computing engine, methods include:

[0132] 5.1.1 Determine the objective function:

[0133]

[0134] 5.1.2 The encoder input sequence required is an embedded vector Q and K, which is used as the input to the neural network.

[0135] 5.1.3 The input of the decoder that meets the requirements is the output of the encoder.

[0136] 5.1.4 The output data encoded by the codec is placed in the external memory of the device.

[0137] 5.1.5 The decoded data is read from the external memory, and the decoded vector is placed in the on-chip memory.

[0138] 5.2. If it is unstructured sparse computation, solving the reuse of data streams includes:

[0139] 5.2.1. Utilizing the sparsity of the Q and K matrices of the image, the data is stored using the K-Stationary Dataflow method.

[0140] 5.2.2 Utilizing the sparsity of the Q and K matrices of the image and the operational order of the Transformer, the SDMM (Sampled-based Dense-Dense Matrix Multiplication) hardware kernel and the SpMM (Sparse-Dense Matrix Multiplication) hardware kernel are scheduled to perform matrix operations in accordance with the K-Stationary Dataflow method.

[0141] 5.3. If it involves unstructured sparse operations, solving the encoding of sparse matrices includes:

[0142] 5.3.1 Use Compressed Sparse Column Format encoding.

[0143] 5.3.2. Based on the characteristics of dataflow, determine the order of matrix operations.

[0144] To describe the invention in more detail, specific examples will be used to illustrate its practical application.

[0145] Example 1: Design of a smartly accelerated microarchitecture kernel based on the new Transformer architecture.

[0146] Figure 3 This is an application of the present invention to handle self-attention, and Figure 4 This is an application of the present invention to handle cross attention. A direct application of the present invention, in embodiment one, is the design of a microarchitecture embedded within a visual processing intelligent accelerator.

[0147] The microarchitecture for designing intelligent accelerators to process vision tasks consists of off-chip memory on the left, with the computing engine in the middle. Considering that image data can be sparsely distributed in an N:M ratio, and that image sparse compression ratios can be very high, typically 90%–95%, the computing engine can be designed in two types: a dense computing engine and a sparse computing engine. The right side of the engine module shows the arrangement of the processing element (PE) array. The deployment of multiple PEs is for parallel computing, improving computational performance. These multiple PE arrays, according to the requirements of the algorithm task module, form an additive-multiplicative task line (MAC Line) to complete a specific task.

[0148] The accelerator's computing engine, while performing the tasks of the algorithm modules, also includes an integrated encoder / decoder neural network.

[0149] The intensive computing engine uses SDDMM (Sampled-based Dense-Dense Matrix Multiplication) to handle Q / K operations, while SpMM (Sparse-Dense Matrix Multiplication) is used to handle S / V operations.

[0150] The sparse engine uses SpMM to calculate the Q, K, V, and S matrices, as well as Softmax and other nonlinear calculations.

[0151] The intensive computing engine has dedicated caches, including an output cache (OBuf), a weight cache (WBuf), a K / S vector cache (K / SBuf), and a Q / V vector cache (Q / VBuf). Each of these has parallel read / write ports.

[0152] The sparse computing engine also requires dedicated caches, including an output cache (OBuf), a weight cache (WBuf), a K / S vector cache (K / SBuf), an index (IdxBuf), and a Q / V vector cache (Q / VBuf). Each of these has parallel read / write ports. In addition, there is control logic.

[0153] For self-attention, the encoder uses a hierarchical approach to its computational steps, which are as follows:

[0154] 1. First-order linear transformation: This involves multiplying the input feature vector / matrix by the corresponding weight matrix (Wq(kV)) to perform a linear transformation, generating Q, K, and V matrices.

[0155] 2. Divide the data into blocks based on each individual head in the multi-head attention, and use the multi-head processing line (MAC line) in parallel.

[0156] 3. Based on the parallel processing of PE and the calculation of Dataflow, a sparse computation N:M algorithm is adopted when processing individual attention operations.

[0157] 4. Adopt the K-parameter stationary method (K-Stationary Dataflow) to improve data reuse and reduce data migration overhead.

[0158] 5. Building on step 4, Autoenoder is used to further improve the load balance between intensive and sparse computing.

[0159] 6. After the above processing is completed, merge the results according to the multi-head attention requirements.

[0160] 7. Based on step 5, complete the subsequent linear operations of FFN to finish the coding of this module.

[0161] For cross-attention, the Q matrix needs to come from the encoder, and the remaining encoding steps are the same as above.

[0162] Example 2: Using microarchitecture, design key technology core solutions for encoding and decoding applications with different input and output scenarios.

[0163] The advantage of the Transformer architecture is its ability to express mutual attention between adjacent and global components. For many computer vision perception tasks in complex environments, intensive and sparse computations often need to be coordinated according to different task characteristics. In such cases, the algorithm structure and the hardware supporting it need to be usable within the scope required to complete the task. According to the microarchitecture technology provided by this invention, tasks requiring only the encoder, or tasks requiring both encoding and decoding, can be completed based on task needs. It can also establish the relationship between historical attention and current attention, the relationship between certain features of different encoders and the features needed for current decoding, or even the solution of features for specific problems, such as the relationship between encoded or decoded historical features and the features of the current query. (See...) Figure 5 Due to the innovative structure of this invention, its encoding and decoding system is composed of multiple multi-head attention layers stacked together. These different multi-head attention layers can be interconnected to achieve joint prediction, providing a feasible solution for accomplishing such tasks.

[0164] The steps to solve this type of problem are as follows:

[0165] 1. If we are considering the joint association between multiple branches at the same level, then:

[0166] 1.1. Allocate the attention scores for each sub-subject according to... Figure 6 The fraction input is shown.

[0167] 1.2 Finally, each new branch output is obtained. At this point, the branch output has new features that take into account the useful information of the other branches.

[0168] 2. If we are considering the combination of different multi-head attention layers from the same encoder or decoder, then the combination relationships between different layers are as follows: Figure 5 .

[0169] 3. If we consider that K and V come from the encoder, while Q comes from the decoder, then the implementation of multi-head joint is as follows, see Figure 5 .

[0170] 4. If considering fully combining the multi-head attention at the encoder end, and then considering the cross-combination of multi-graph attention at the decoder end, see [link to relevant documentation]. Figure 5 .

[0171] Example 3: By utilizing microarchitecture and embedding it into processors used in real-world applications such as autonomous driving, the terminal system can achieve stronger processing capabilities.

[0172] In the visual perception of autonomous driving and robotics, BEV (Bird's Eye View) is an important perception tool used to describe the perspective or coordinate system of the perceived world. BEV also refers to an end-to-end technology in the field of computer vision that uses neural networks to transform visual information from image space to BEV space.

[0173] While theoretically BEV (Battery Electric Vehicle) technology can be applied to front-end, mid-end, and rear-end fusion processes, it's rarely used in front-end fusion due to its high implementation difficulty. Occasionally, it's used in rear-end fusion, but more often in feature-level fusion—a process between data-level and target-level fusion—i.e., mid-level fusion. Mid-level fusion involves first extracting intermediate-layer features (effective features) from each sensor using a neural network model, then fusing the main effective features from multiple sensors to increase the likelihood of optimal inference. Fusing effective features in the BEV space results in less data loss and lower computational cost (compared to front-end fusion), hence its prevalence in mid-level fusion. Typical processing methods for neural network backbones and Transformers currently used in the automotive industry can be found in [link to relevant documentation]. Figure 7 As shown.

[0174] exist Figure 7 In this invention, the structure of the Transformer can be adopted as follows:

[0175] 1. Extract features from the backbone network and sparsify N:M to leverage its hardware advantages in improving computational performance.

[0176] 2. The cross-attention processing method achieves better image feature processing and information fusion, while simultaneously realizing high computational efficiency.

[0177] The specific method is as follows:

[0178] 1. Input the features of multiple shots into the backbone of the selected pre-selected neural network model to obtain the features fused.

[0179] 2. According to the method of the invention, efficient operation of feature extraction is achieved.

[0180] 3. Use the features of the shot-around image sequence before time t as the input for self-attention. Figure 8 Bird's EyeView Bt-1, and Bird's Eye View Bt.

[0181] 4. The cross-attention processing access method between different layers according to Embodiment 2.

[0182] 5. Simultaneously considering the multiple signals required in the unified stereo view of Bird's Eye View, attention is applied to the characteristics of these multiple spatial signals through cross-attention. This requires Q from the decoding end to query K and V from the cameras at different spatial positions. Finally, the attention matrix is ​​derived.

[0183] The application of Transformers in the field of computer vision is relatively new. Existing technologies in this field generally fail to address the overall performance bottleneck of Transformers. It is precisely the collaboration between the sparse implementation of the front-end and the back-end processing engine that solves the overall performance bottleneck. Therefore, the performance is not significant on resource-constrained terminals. The advantage of this invention lies in its proposal of an overall architecture design and related technologies compatible with the overall architecture design, building upon previous research. For example:

[0184] Existing network sparse design technologies generally employ structured or unstructured sparse algorithms. While unstructured sparse can achieve a significantly higher compression ratio than structured sparse, its subsequent hardware processing performance improvement is often less than that of structured sparse. In terms of visual characteristics, numerous existing studies have demonstrated that sparsity rates can be very high, typically reaching around 90%. However, if the algorithm does not consider the characteristics of subsequent hardware, such processing generally does not lead to an overall performance improvement. This invention designs an algorithm in an N:M sparse pattern that meets the requirements of subsequent GPU hardware processing, and also designs a structural design that satisfies the algorithm's operational requirements.

[0185] Of course, going further, some existing technologies, if they only consider the above N:M structure, but are still static, or use a priori fixed sparse mode, then they cannot adapt to the dynamic and variable characteristics of visual signals. The approach of this invention is to use a dynamic and learnable sparse algorithm that conforms to the signal processing characteristics of the human brain for each multi-head output. Moreover, this algorithm also considers overcoming the original Transformer (Vanilla) approach of simply concatenating the multi-heads to express image features. For all multi-heads, the joint statistical properties (collaborate) are also considered to express the overall features, so that the model's expressive ability can be better.

[0186] The fundamental goal of solving the bottleneck problem of Transformer is to design a usable accelerator circuit, the core of which is the design of the acceleration engine. Existing technologies generally do not consider the coordination between algorithm and hardware implementation, and typically fail to address the computation of processing elements (PEs) and the communication between the processor and memory. To ensure the strong application value of this invention, it combines the characteristics of Transformer and Auto Encoder, proposing a new network form for Transformer. This new form can adapt to algorithms with the aforementioned image characteristics, achieving the requirement of simplified processor design, and can also balance and efficiently address the communication bottleneck between the computational unit and memory unit based on the size of the input image features and the network size.

[0187] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the method described above.

[0188] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0189] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0190] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0191] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0192] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0193] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0194] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0195] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0196] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A Transformer neural network system, characterized by, Comprise: Data storage module, including input matrix storage unit, weight matrix storage unit and temporary matrix storage unit, the input end of the input matrix storage unit as the input interface of the data storage module; Tensor operation core, for operating the input tensor; Input signal main trunk neural network processing module, output the input image signal to be processed as a feature extraction signal, as a sequence signal tensor input of the Transformer; Joint multi-head attention processing module, output the input sequence tensor of the Transformer as an attention matrix; Forward neural network processing module, output the normalization evaluation of the input multi-head attention as a sequence of encoded vectors; Efficient calculation module, for outputting the input image series and neural network as N:M sparse network sparse features; also for outputting the input dynamic attention matrix, the dynamic attention matrix is obtained by the correlation operation between Q, K, V attention matrices, output as a sparse N:M attention matrix that meets the hardware acceleration condition; Transformer hardware acceleration processing module, for outputting the input irregular sparse attention matrix as a structured sparse matrix that meets N:M; also for outputting the input Q, K matrix as a matrix pair that meets the preset data stream characteristics; also for compressing the Q, K features by using Autoencoder, outputting the compression result to the external storage unit; also for loading the Q, K features from the external storage, performing reverse encoding, obtaining the result in the on-chip storage, and completing the specified operation under the control of the control module; Residual addition and normalization operation module, including residual addition operation module and layer normalization function operation module, the residual addition module includes a plurality of adders, the number of adders is consistent with the number of rows of any input matrix in the multi-head attention layer, the input end of the residual addition module is connected to the output end of the input matrix storage unit, and the output end is connected to the input end of the layer normalization function operation module; the output end of the layer normalization function operation module as the output interface; The computing engine array includes a plurality of configurable computing engines and an on-chip cache module; the computing engine includes a configurable number of multi-precision adders, accumulators, nonlinear operation units, and a plurality of multi-precision parallel multipliers; the computing engine array is used to perform linear layer operation, nonlinear layer operation, single-head attention layer, multi-head attention operation, fully connected layer operation, and residual connection operation in parallel; the operation includes dense matrix / tensor calculation, sparse matrix / tensor calculation, and mixed operation thereof; the on-chip cache module includes a weight cache area, a parameter cache area, and a plurality of feature cache areas; the computing engine array is connected to off-chip storage through the on-chip cache module, and the on-chip cache module is used to realize data interaction between the computing engine array and the off-chip storage; the nonlinear layer includes a Softmax operation module and a normalization operation module, the input end of the Softmax operation module and the input end of the normalization operation module are connected with the output end of the computing engine array, and the Softmax operation module and the normalization operation module are connected with the on-chip cache module; the Softmax operation module and the normalization operation module are used to perform nonlinear layer operation control module; the computing engine array, the Softmax operation module, and the normalization operation module are connected with the control module; the control module is used to configure operation and memory logic scheduling control of the computing engine array, the Softmax operation module, and the LayerNorm operation module; The judgment, scheduling and control system method module is a control module, which is connected with and controls the data storage module, the input backbone neural network processing module, the joint multi-head attention processing module, the forward neural network processing module, the residual and addition module, the layer normalization function operation module, the data access module, the tensor operation core, the high-efficiency computing module and the Tramsformer acceleration processing module.

2. An operation method of a Transformer neural network system, characterized by, The dynamic sparse compression calculation process applied to the Transformer neural network system in claim 1 comprises: Setting a sparse target, determining that the maximum number of non-zero numbers in each group of M weights is N, and setting a target function for learning and training; calculating the gradient of the continuous parameter group space subnetwork to perform backward propagation gradient calculation of the network; Determining an approximate target function; selecting a random matrix; training an approximate matrix that meets the dynamic N:M sparsity requirement; predicting the dynamic N:M self-attention matrix through the approximate matrix; and dynamically screening the weights of the attention matrix of N:M according to the approximate matrix.

3. An operation method of a Transformer neural network system, characterized by, The joint multi-head attention calculation process applied to the Transformer neural network system in claim 1 comprises: Splicing each sub-head attention evaluation into a whole tensor; obtaining a new tensor through one-dimensional convolution of the whole tensor; and determining joint multi-head attention coefficients according to the new tensor; Calculating a cross-attention coefficient matrix, splitting the cross-attention coefficient matrix into a plurality of tensors, and obtaining the cross-attention matrix between multiple layers on the same side according to the multi-head attention calculation process; The joint cross-attention of the encoding and decoding mutual sides is calculated according to the dynamic attention matrices Q, K and V.

4. An operation method of a Transformer neural network system, the method comprising: The application is applied to the Transformer hardware acceleration calculation process in the Transformer neural network system in claim 1, and comprises: A target function is determined; the encoder input sequence is an embedded vector Q and K, which is input to the neural network; the input of the decoder is the output of the encoder; the encoding output data of the encoder-decoder is stored in the external memory; the decoded data is read from the external memory, and the decoded vector is stored in the on-chip memory; The sparse characteristics of the Q and K matrices of the image are utilized, and the K-Stationary Dataflow is adopted to store the data; the sparse characteristics of the Q and K matrices of the image and the operation order of the Transformer are utilized to schedule the hardware kernel to perform the matrix operation; The preset mode is adopted to perform the encoding; and the operation order of the data stream selection matrix is determined.

5. An arithmetic device of a Transformer neural network system, characterized by comprising: The application is applied to the joint multi-head attention calculation process in the Transformer neural network system in claim 1, and comprises: A third operation unit is configured to splice each sub-head attention evaluation tensor into an integral tensor, pass the integral tensor through a one-dimensional convolution to obtain a new tensor, and determine a joint multi-head attention coefficient according to the new tensor; A fourth operation unit is configured to calculate a cross-attention coefficient matrix, split the cross-attention coefficient matrix into multiple tensors, and obtain the cross-attention matrix between multiple layers on the same side according to the multi-head attention calculation process; The joint cross-attention of the encoding and decoding mutual sides is calculated according to the dynamic attention matrices Q, K and V.

6. An arithmetic device of a Transformer neural network system, characterized by comprising: The application is applied to the Transformer hardware acceleration calculation process in the Transformer neural network system in claim 1, and comprises: A sixth operation unit is configured to determine a target function; the encoder input sequence is an embedded vector Q and K, which is input to the neural network; the input of the decoder is the output of the encoder; the encoding output data of the encoder-decoder is stored in the external memory; the decoded data is read from the external memory, and the decoded vector is stored in the on-chip memory; A seventh operation unit is configured to utilize the sparse characteristics of the Q and K matrices of the image, adopt the K-Stationary Dataflow to store the data, and utilize the sparse characteristics of the Q and K matrices of the image and the operation order of the Transformer to schedule the hardware kernel to perform the matrix operation; An eighth operation unit is configured to adopt a preset mode to perform the encoding, and determine the operation order of the data stream selection matrix.

7. An electronic device, comprising: The application comprises a processor and a memory. The memory is configured to store a program. The processor executes the program to implement the method in any one of claims 2, 3 or 4.

8. A computer-readable storage medium, characterized in that, The storage medium stores a program, and the program is executed by the processor to implement the method in any one of claims 2, 3 or 4.

Images