Data processing method, system and terminal of multi-head potential attention model based on TTD compression

Abstract

The invention discloses a data processing method and system for a multi-head potential attention model based on TTD compression and a terminal, and the method comprises the steps: constructing a large language model, and processing a plurality of linear layers in the large language model in a TTD compression and decomposition mode, thereby reducing the weight number in the model, and improving the data processing efficiency. And on a hardware level, targeted optimization is carried out on a data stream subjected to linear layer reasoning after TTD compression, so that a final model reasoning result is output. According to the method, a multi-head potential attention structure in a large language model is converted, so that the requirement for KV cache during model reasoning is reduced, the weight number is reduced, the long text output capacity of the model on edge equipment is improved, matrix calculation in the reasoning process is carried out subsequently by adopting a calculation structure of a group vector systolic array, and the reasoning efficiency is improved. And limited hardware resources are efficiently utilized.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data processing method, system, terminal and computer-readable storage medium for a multi-head latent attention model based on TTD compression. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs) have made breakthrough progress in the fields of natural language processing, machine translation, text generation, etc. Quantization can convert the weights of the network from floating-point numbers to fixed-point numbers, which not only reduces the memory space occupied by the weights, but also replaces the floating-point calculations in the network with fixed-point calculations, making the inference of the network more friendly to hardware.

[0003] However, for existing Grouped-Query Attention (GQA) and Multi-Query Attention (MQA), while reducing the KV cache (key-value cache, a key technology for optimizing model inference efficiency), they will cause a loss of model performance.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the Invention

[0005] The main purpose of the present invention is to provide a data processing method, system, terminal and computer-readable storage medium for a multi-head latent attention model based on TTD compression, aiming to solve the problem in the prior art that large language models have a high demand for KV Cache, resulting in too low model inference efficiency and inaccurate data processing results.

[0006] To achieve the above object, the present invention provides a data processing method for a multi-head latent attention model based on TTD compression, and the data processing method for the multi-head latent attention model based on TTD compression includes the following steps:

[0007] Construct a large language model, perform tensorization processing on multiple linear layers in the large language model to obtain multiple high-dimensional tensors, perform TTD compression on all the high-dimensional tensors to obtain tensor data of all the linear layers, and construct corresponding tensor forms according to all the tensor data;

[0008] Perform SVD decomposition on the key matrix and value matrix in the large language model to obtain multiple sampling matrices, perform TTD compression on all the sampling matrices, and then combine all the tensor forms to construct a TTD-MLA network structure;

[0009] Input the training data into the TTD-MLA network structure, rearrange the training data according to the tensor kernels of each linear layer to obtain a vector sequence, input the vector sequence into the group vector systolic array architecture in the TTD-MLA network structure, and output multiple intermediate data;

[0010] Input all the intermediate data into a digital signal processor. The digital signal processor determines the displacement according to all the intermediate data, and processes all the intermediate data according to the displacement to output the final result.

[0011] Optionally, for the data processing method of the multi-head latent attention model based on TTD compression, in which the large language model is constructed, the multiple linear layers in the large language model are tensorized to obtain multiple high-dimensional tensors, and all the high-dimensional tensors are compressed by TTD to obtain the tensor data of all the linear layers, and the corresponding tensor forms are constructed according to all the tensor data, specifically including:

[0012] Construct a large language model, obtain the weight matrices corresponding to the multiple linear layers in the large language model, and tensorize all the weight matrices to obtain multiple high-dimensional tensors:

[0013]

[0014] Among them, T j represents the j-th weight matrix, M1 and M2 represent two dimensions of T, represents the set of real numbers, d represents the number of dimensions after tensorization, v i represents the i-th dimension of the high-dimensional tensor, T ′ j represents the j-th high-dimensional tensor;

[0015] Perform TTD compression on each high-dimensional tensor to obtain the tensor data and corresponding tensor kernels corresponding to each linear layer:

[0016] T ′ j (l 1j ,...,l dj )=G 1j [l 1j ...G dj [l dj ;

[0017]

[0018] Among them, T ′ j (l 1j ,...,l dj ) represents T′ j The tensor data, l 1j represents the first tensor core G after TTD compression 1j of the l 1j th slice, l dj represents the dth tensor core G after TTD compression dj of the l dj th slice, l kj represents the slice of the kth tensor core in the jth linear layer, r k-1 is the tensor core G k-1j of the rank, r k is the tensor core G kj of the rank;

[0019] Construct the tensor form corresponding to each said linear layer according to each said tensor data:

[0020]

[0021]

[0022] where Y(j 1j ,...,j dj ) represents the tensor form of the linear layer corresponding to T j , G 1j [i 1j ,j 1j represents the first tensor form of the original weight of the linear layer corresponding to T j , G dj [i dj ,j dj represents the dth tensor form of the original weight of the linear layer corresponding to T j , G kj represents the kth tensor form of the original weight of the linear layer corresponding to T j , X represents the tensor form of the original input of the linear layer, i dj and j dj both represent the indices of G dj [i dj ,j dj , m d represents the last dimension of Y(j 1j ,...,j dj ), G kj [i k ,j k represents the kth tensor form of the original weight of the linear layer corresponding to T j , m k and n k both represent G kj [i k ,jk dimensions.

[0023] Optionally, in the data processing method of the multi-head latent attention model based on TTD compression, when performing TTD compression on each of the high-dimensional tensors to obtain the tensor data and corresponding tensor kernels corresponding to each linear layer, the method further includes:

[0024] When each linear layer performs inference, use the corresponding tensor kernel as the weight data of each linear layer, and use each weight data to replace the linear layer weight of the corresponding linear layer.

[0025] Optionally, in the data processing method of the multi-head latent attention model based on TTD compression, when performing SVD decomposition on the key matrix and value matrix in the large language model to obtain multiple sampling matrices, and after performing TTD compression on all the sampling matrices, combining all the tensor forms to construct a TTD-MLA network structure, it specifically includes:

[0026] Obtain the key matrix and value matrix in the large language model, and use the SVD decomposition method to decompose the key matrix and the value matrix into multiple upsampling matrices and a single downsampling matrix;

[0027] After performing TTD compression on all the upsampling matrices and the downsampling matrix, obtain an MLA structure network;

[0028] According to all the tensor forms, combine the MLA structure network with all the linear layers to obtain a TTD-MLA network structure.

[0029] Optionally, in the data processing method of the multi-head latent attention model based on TTD compression, when inputting training data into the TTD-MLA network structure, rearranging the training data according to the tensor kernel of each linear layer to obtain a vector sequence, and inputting the vector sequence into the group vector systolic array architecture in the TTD-MLA network structure to output multiple intermediate data, it specifically includes:

[0030] Obtain the training data input by the user, and input the training data into the TTD-MLA network structure;

[0031] The TTD-MLA network structure rearranges the training data according to all the tensor kernels to obtain a vector sequence:

[0032]

[0033] where represents the vector sequence, t k-1 represents the dimension of the k-th stage of the TTD decomposition process, jk The tensor form representing the k-th stage, n k The number of training data representing the k-th stage, P k-1 [i k ,t k-1 represents the vector sequence of the (k - 1)-th stage;

[0034] Divide the vector sequence into multiple group vectors, and input each of the group vectors into multiple vector operation units in the group vector systolic array architecture in the TTD-MLA network structure for parallel processing, and output corresponding intermediate data respectively.

[0035] Optionally, in the data processing method of the multi-head latent attention model based on TTD compression, where the dividing the vector sequence into multiple group vectors, and inputting each of the group vectors into multiple vector operation units in the group vector systolic array architecture in the TTD-MLA network structure for parallel processing, and outputting corresponding intermediate data respectively, further includes:

[0036] Cache each of the intermediate data into the ping-pong buffer in the TTD-MLA network structure, and accumulate it into the accumulator in the TTD-MLA network structure to obtain accumulated data;

[0037] Write the accumulated data into the double data rate memory through the direct memory access unit in the TTD-MLA network structure.

[0038] Optionally, in the data processing method of the multi-head latent attention model based on TTD compression, where the inputting all the intermediate data into a digital signal processor, and the digital signal processor determines a displacement amount according to all the intermediate data, and processes all the intermediate data according to the displacement amount to output a final result, specifically includes:

[0039] Input all the intermediate data and all the weight data into the digital signal processor;

[0040] The digital signal processor determines the displacement amount of the maximum exponent bit in all the weight data, performs a shift processing on all the intermediate data according to the displacement amount, and then inputs them into an adder tree;

[0041] Input the accumulated data into the adder tree, calculate the product between each of the intermediate data and the accumulated data, and output the final result.

[0042] In addition, to achieve the above object, the present invention further provides a data processing system of a multi-head latent attention model based on TTD compression, where the data processing system of the multi-head latent attention model based on TTD compression includes:

[0043] A tensor compression module, used to construct a large language model, perform tensorization processing on multiple linear layers in the large language model to obtain multiple high-dimensional tensors, perform TTD compression on all the high-dimensional tensors to obtain the tensor data of all the linear layers, and construct corresponding tensor forms according to all the tensor data;

[0044] A model construction module, used to perform SVD decomposition on the key matrix and value matrix in the large language model to obtain multiple sampling matrices, perform TTD compression on all the sampling matrices, and combine all the tensor forms to construct a TTD-MLA network structure;

[0045] A group vector systolic array module, used to input training data into the TTD-MLA network structure, rearrange the training data according to the tensor core of each linear layer to obtain a vector sequence, input the vector sequence into the group vector systolic array architecture in the TTD-MLA network structure, and output multiple intermediate data;

[0046] A floating-point calculation module, used to input all the intermediate data into a digital signal processor, the digital signal processor determines a displacement amount according to all the intermediate data, and processes all the intermediate data according to the displacement amount to output a final result.

[0047] In addition, to achieve the above object, the present invention also provides a terminal, wherein the terminal includes: a VCU 128, a processor, and a data processing program of a multi-head latent attention model based on TTD compression stored on the VCU 128 and executable on the processor. When the data processing program of the multi-head latent attention model based on TTD compression is executed by the processor, the steps of the data processing method of the multi-head latent attention model based on TTD compression as described above are implemented.

[0048] In addition, to achieve the above object, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a data processing program of a multi-head latent attention model based on TTD compression. When the data processing program of the multi-head latent attention model based on TTD compression is executed by a processor, the steps of the data processing method of the multi-head latent attention model based on TTD compression as described above are implemented.

[0049] In the present invention, a large language model is constructed, and the multiple linear layers in the large language model are tensorized to obtain corresponding high-dimensional tensors. Then, all the high-dimensional tensors are compressed by TTD to obtain the tensor data of all the linear layers, and corresponding tensor forms are constructed according to all the tensor data. The key matrix and value matrix in the large language model are subjected to SVD decomposition to obtain multiple sampling matrices. After all the sampling matrices are compressed by TTD, combined with all the tensor forms, a TTD-MLA network structure is constructed. Training data is input into the TTD-MLA network structure, and the training data is rearranged into a vector sequence according to the tensor kernel of each linear layer. The vector sequence is input into the group vector systolic array architecture in the TTD-MLA network structure, and multiple intermediate data are output. All the intermediate data are input into a digital signal processor, and the digital signal processor determines the displacement amount according to all the intermediate data and processes all the intermediate data according to the displacement amount to output the final result. By transforming into the multi-head latent attention structure in the large language model, the present invention reduces the demand for KV cache during model inference, and improves the long text output ability of the model on edge devices by reducing the number of weights. Subsequently, the group vector systolic array computing structure is adopted to perform matrix calculations during the inference process, efficiently utilizing limited hardware resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flowchart of a preferred embodiment of the data processing method of the multi-head latent attention model based on TTD compression of the present invention;

[0051] Figure 2 is a schematic diagram of a linear layer of a preferred embodiment of the data processing method of the multi-head latent attention model based on TTD compression of the present invention;

[0052] Figure 3 is a schematic diagram of the TTD principle of a preferred embodiment of the data processing method of the multi-head latent attention model based on TTD compression of the present invention;

[0053] Figure 4 is a schematic diagram of TTD-MLA of a preferred embodiment of the data processing method of the multi-head latent attention model based on TTD compression of the present invention;

[0054] Figure 5 is a general architecture diagram of a preferred embodiment of the data processing method of the multi-head latent attention model based on TTD compression of the present invention;

[0055] Figure 6 is a schematic diagram of the group vector systolic array architecture of a preferred embodiment of the data processing method of the multi-head latent attention model based on TTD compression of the present invention;

[0056] Figure 7 It is a floating-point calculation schematic diagram of a preferred embodiment of the data processing method of the multi-head latent attention model based on TTD compression of the present invention;

[0057] Figure 8 It is a structural diagram of a preferred embodiment of the data processing system of the multi-head latent attention model based on TTD compression of the present invention;

[0058] Figure 9 It is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed implementation manners

[0059] To make the objectives, technical solutions and advantages of the present invention clearer and more definite, the following further describes the present invention in detail with reference to the accompanying drawings and by way of examples. It should be understood that the specific examples described herein are only used to explain the present invention and are not used to limit the present invention.

[0060] The data processing method of the multi-head latent attention model based on TTD compression according to a preferred embodiment of the present invention, as Figure 1 shown, the data processing method of the multi-head latent attention model based on TTD compression includes the following steps:

[0061] Step S10: Construct a large language model, perform tensorization processing on multiple linear layers in the large language model to obtain multiple high-dimensional tensors, and perform TTD compression (Tensor-train Decomposition) on all the high-dimensional tensors to obtain tensor data of all the linear layers, and construct corresponding tensor forms according to all the tensor data.

[0062] Among them, to construct a large language model, its structure is as Figure 2 shown, including word embedding, multiple cascaded transformer blocks and an output layer. Among them, the Transformer block is composed of MHA (Multi-Head Attention) and MLP (Multilayer Perceptron). MHA is mainly composed of a linear layer, rotary position embedding (RoPE), matrix multiplication and softmax. The MLP is composed of a linear layer, matrix multiplication and an activation function.

[0063] Specifically, to construct a large language model, obtain weight matrices corresponding to multiple linear layers in the large language model, and perform tensorization processing on all the weight matrices to obtain multiple high-dimensional tensors:

[0064]

[0065] Among them, T j represents the j-th weight matrix, and M1 and M2 represent the two dimensions of the weight matrix. represents the set of real numbers, d represents the number of dimensions after tensorization, and v i represents the i-th dimension of the high-dimensional tensor, and T ′ j represents the j-th high-dimensional tensor; perform TTD compression on each of the high-dimensional tensors to obtain the tensor data and corresponding tensor kernels corresponding to each of the linear layers:

[0066] T ′ j (l 1j ,...,l dj ) = G 1j [l 1j ...G dj [l dj ;

[0067]

[0068] Among them, T ′ j (l 1j ,...,l dj ) represents the tensor data of T ′ j , l 1j represents the l 11j -th slice of the first tensor kernel G 1j , l dj represents the l dj -th slice of the d-th tensor kernel G dj , l kj represents the slice of the k-th tensor kernel in the j-th linear layer, and r k-1 is the rank of the tensor kernel G k-1j , and r k is the rank of the tensor kernel G kj ; construct the tensor form corresponding to each of the linear layers according to each of the tensor data:

[0069]

[0070] Among them, Y(j 1j ,...,j dj ) represents the tensor form of the linear layer corresponding to T j , and G 1j [i 1j ,j 1j represents the first tensor form of the original weight of the linear layer corresponding to T j , and Gdj [i dj ,j dj represents the d-th tensor form of the original weight of the corresponding linear layer, G j represents the d-th tensor form of the original weight of the corresponding linear layer, G kj represents T j represents the k-th tensor form of the original weight of the corresponding linear layer, X represents the tensor form of the original input of the linear layer, i dj and j dj both represent G dj [i dj ,j dj represents the index of m d represents Y(j 1j ,...,j dj ), the last dimension of G kj [i k ,j k represents T j represents the k-th tensor form of the original weight of the corresponding linear layer, m k and n k both represent G kj [i k ,j k represents the dimension of

[0071] Among them, as Figure 3 shown, after the TTD of the linear layer in the model, tensor compression is performed. Since this process is mainly for high-dimensional tensor data, the weight matrix of the linear layer needs to be tensorized, and then the high-dimensional tensor is represented as the product of multiple low-dimensional tensors.

[0072] Among them, the compression ratio (CR) of tensor compression is:

[0073]

[0074] Among them, v k represents the k-th dimension of the high-dimensional tensor. Since the rank of the tensor core is small, the compression effect of TTD is obvious.

[0075] Further, when each of the linear layers performs inference, the corresponding tensor core is used as the weight data of each of the linear layers, and each of the weight data is used to replace the linear layer weight of the corresponding linear layer.

[0076] Among them, through TTD compression processing of three linear layers, the tensor kernels obtained by TTD compression are used as the weight data in the model inference process, that is, the multi-head attention (MHA) part in the LLM is decomposed and transformed into a structure similar to multi-head latent attention (MLA), thereby reducing the demand for KV cache during model inference. And TTD is used to compress the linear layers in the MLA part and multi-layer perceptron (MLP) part of the LLM, thereby further reducing the number of weights and improving the long text output ability of the model on edge devices.

[0077] Step S20: Perform SVD decomposition on the key matrix and value matrix in the large language model to obtain multiple sampling matrices. After performing TTD compression on all the sampling matrices, combine all the tensor forms to construct a TTD-MLA network structure.

[0078] Specifically, obtain the key matrix and value matrix in the large language model, and use the SVD decomposition method to decompose the key matrix and the value matrix into multiple upsampling matrices and a single downsampling matrix; after performing TTD compression on all the upsampling matrices and the downsampling matrix, obtain an MLA structure network; according to all the tensor forms, combine the MLA structure network with all the linear layers to obtain the Figure 4 shown TTD-MLA network structure.

[0079] Among them, Figure 4 W in K represents the weight of linear layer K, W V represents the weight of linear layer V, W UK represents the downsampling matrix, W UV and W DKV represent the upsampling matrices, and respectively represent the first, second, and third tensor kernels obtained after TTD of W DKV , represents the result of the output of the linear layer with W UK as the weight, and respectively represent the first, second, and third tensor kernels obtained after TTD of W UK , and respectively represent the first, second, and third tensor kernels obtained after TTD of W UV .

[0080] Among them, in the structure of MHA, first, Singular Value Decomposition (SDV) is used to decompose the key matrix (K) and the value matrix (V) to obtain two upsampling matrices and a shared downsampling matrix, thus forming the MLA structure, where the latent vector of KV is the product of the downsampling matrix and the input. To further compress the network, the two upsampling matrices and the shared downsampling matrix will be decomposed by TTD. This can convert the K and V that need to be cached during LLM inference into cached latent vectors and V, and the size of the latent vectors is smaller, thus reducing the need for KV cache during LLM inference.

[0081] Step S30: Input the training data into the TTD-MLA network structure, rearrange the training data according to the tensor kernel of each linear layer to obtain a vector sequence, and input the vector sequence into the group vector systolic array architecture in the TTD-MLA network structure to output multiple intermediate data.

[0082] Among them, as Figure 5 shown, when accelerating the inference process of the TTD-MLA network structure, it also includes a linear layer, TTD linear layer operators, and non-linear operation operators. The linear layer and TTD linear layer operators include a direct memory access (DMA), a group vector systolic array (GVSA), an on-chip input and weight buffer (Buffer), and a ping-pong buffer (Ping-Pong Buffer).

[0083] Specifically, obtain the training data input by the user and input the training data into the TTD-MLA network structure; the TTD-MLA network structure rearranges the training data according to all the tensor kernels to obtain a vector sequence:

[0084]

[0085] Among them, represents the vector sequence, t k-1 represents the dimension of the k-th stage of the TTD decomposition process, j k represents the tensor form of the k-th stage, n k represents the number of training data of the k-th stage, P k-1 [i k ,t k-1represents the vector sequence of the (k - 1)-th stage; the vector sequence is divided into multiple group vectors, and each group vector is respectively input into multiple vector operation units in the group vector systolic array architecture in the TTD-MLA network structure for parallel processing, and the corresponding intermediate data is output respectively.

[0086] Among them, GVSA contains 2 groups of vector operation units (PEs), and each group of operation units supports operations of 32x128 groups of INT4xFP16 (two data formats: 4-bit integer (INT4) and half-precision floating-point number (FP16)). Parallel processing is adopted, and the output of each vector operation unit is accumulated by an accumulator.

[0087] As Figure 6 shown, assume the specification of GVSA is Tin×Tout, where Tin is the parallelism degree in the accumulation direction and Tout is the parallelism degree in the output direction. The array is composed of two groups of Group Vector Systolic PEs (group vector systolic processing units) with the specification of Tin×Tout / 2 (where Tout / 2 represents Tn), and data is transmitted between them in a systolic form. GVSA inputs feature and weight data in parallel in the form of group vectors (vector data with a length of Tin and a quantity of Tout / 2). In each clock cycle, Tout vectors participate in the calculation simultaneously, significantly improving the hardware utilization rate, and being able to process irregular matrix operations, such as matrix transpose, matrix multiplication, etc. in the multi-head self-attention mechanism. Through the optimization of data flow and matrix packing, the overhead of reconstructing the matrix is reduced. Compared with traditional systolic arrays and VSA (vector systolic array), while maintaining high throughput, GVSA reduces register occupancy and power consumption, making it suitable for the application scenarios of edge devices.

[0088] Furthermore, each intermediate data is cached into the ping-pong buffer in the TTD-MLA network structure and accumulated into the accumulator in the TTD-MLA network structure to obtain accumulated data; the accumulated data is written into the double data rate memory through the direct memory access unit in the TTD-MLA network structure.

[0089] Among them, the intermediate data between stages during the TTD inference process needs to be cached in the on-chip ping-pong buffer, and the output after completing all stages of TTD inference is written back to the DDR (Double Data Rate SDRAM). During the process of performing linear layer inference calculations, the output data of the accumulator is directly written back to the DDR through DMA. The TTD-MLA network structure also includes other non-linear operators, and all calculations for LLM inference can be completed through these operators. The weights used for calculation are stored in the HBM (High-Bandwidth-Memory), achieving higher bandwidth and lower latency.

[0090] Step S40: Input all the intermediate data into a digital signal processor. The digital signal processor determines a displacement amount based on all the intermediate data, and processes all the intermediate data according to the displacement amount, and outputs a final result.

[0091] Specifically, input all the intermediate data and all the weight data into a digital signal processor; the digital signal processor determines the displacement amount of the maximum exponent bit in all the weight data, shifts all the intermediate data according to the displacement amount, and then inputs them into an adder tree; input the accumulated data into the adder tree, calculate the product between each intermediate data and the accumulated data, and output a final result.

[0092] Among them, as Figure 7 shown, the vector calculation unit in GVSA adopts a scheme of sharing DSP (Digital Signal Processor) to execute (Tin×Tout / 2) groups of INT4xFP16 operations in parallel. The FP16 format consists of 1 sign bit, 5 exponent bits, and 10 mantissa bits. To implement the multiplication of FP16 and INT4, it is necessary to multiply the INT4 signed fixed-point number by the two's complement format of {sign bit, hidden bit, mantissa bit} with 12 bits.

[0093] As Figure 7 shown in (b) of [], since the input bit widths of the multipliers in the DSP on the platform are 27 and 18, it is allowed to simultaneously execute the multiplication of the two's complement of the mantissas of two 12-bit FP16s and the multiplication of two INT4 weights. The specific data input format of the DSP is as Figure 7 shown in (c) of []. Through this DSP sharing strategy, not only the data flow requirements of GVSA are met, but also DSP resources are saved. By using multiple such PEs in parallel externally, different levels of parallelism of GVSA can be achieved.

[0094] Further, for the vector floating-point calculation unit shared by the DSP, the calculation is performed in the form of the pipeline shown in (a) of Figure 7 A feature vector in FP16 format and two weight vectors in INT4 are input into the PE. First, each part of the FP16 format is separated, and complement operations are performed on the data of {sign bit, hidden bit, mantissa bit}; next, the 12-bit complement data and the fixed-point weight data are input into the DSP, and at the same time, the floating-point exponent bits are compared to determine the displacement amount for the maximum exponent bit; then, the DSP output is shifted according to the determined displacement amount and input into the adder tree; in the final stage, the input quantization scale is multiplied by the accumulated result to obtain the final result.

[0095] The present invention reduces the demand for KV cache during model inference by transforming it into a multi-head latent attention structure in the large language model, improves the long text output ability of the model on edge devices by reducing the number of weights, and subsequently adopts a computational structure of a group vector systolic array to perform matrix calculations during the inference process, efficiently utilizing limited hardware resources.

[0096] Further, as shown in Figure 8 Based on the above data processing method of the multi-head latent attention model based on TTD compression, the present invention also correspondingly provides a data processing system for the multi-head latent attention model based on TTD compression, wherein the data processing system for the multi-head latent attention model based on TTD compression includes:

[0097] A tensor compression module 51, configured to construct a large language model, perform tensorization processing on multiple linear layers in the large language model to obtain corresponding high-dimensional tensors, perform TTD compression on all the high-dimensional tensors to obtain tensor data of all the linear layers, and construct corresponding tensor forms according to all the tensor data;

[0098] A model construction module 52, configured to perform SVD decomposition on the key matrix and value matrix in the large language model to obtain multiple sampling matrices, perform TTD compression on all the sampling matrices, and combine all the tensor forms to construct a TTD-MLA network structure;

[0099] A group vector systolic array module 53, configured to input training data into the TTD-MLA network structure, rearrange the training data according to the tensor kernel of each linear layer to obtain a vector sequence, input the vector sequence into the group vector systolic array architecture in the TTD-MLA network structure, and output multiple intermediate data;

[0100] The floating-point calculation module 54 is used to input all the intermediate data into a digital signal processor. The digital signal processor determines a displacement amount based on all the intermediate data, and processes all the intermediate data according to the displacement amount to output a final result.

[0101] Further, as Figure 9 shown, based on the above data processing method and system of the multi-head latent attention model based on TTD compression, the present invention also correspondingly provides a terminal, which includes a processor ( Figure 9 10 in Figure 9 ), a VCU 128 ( Figure 9 20 in Figure 9 ), and a display (

[0102] 30 in

[0103] ). Only some components of the terminal are shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented.

[0104] The VCU 128 is a model of an FPGA (Field-Programmable Gate Array), a programmable semiconductor device. It and the CPU perform data writing and control of the FPGA through PCIE (Peripheral Component Interconnect Express, the mainstream high-speed serial bus standard in the current computer system). HBM, DDR, and the accelerator are all on the VCU 128; the DSP is a resource on the FPGA, and the vector floating-point calculation unit based on the shared DSP is designed to make more full use of the resources on the FPGA.

[0105] In an embodiment, when the processor 10 executes the data processing program 40 of the multi-head latent attention model based on TTD compression in the VCU 128, the following steps are implemented:

[0106] Build a large language model, perform tensorization on multiple linear layers in the large language model to obtain multiple high-dimensional tensors, perform TTD compression on all the high-dimensional tensors to obtain the tensor data of all the linear layers, and construct corresponding tensor forms according to all the tensor data;

[0107] Perform SVD decomposition on the key matrix and value matrix in the large language model to obtain multiple sampling matrices. After performing TTD compression on all the sampling matrices, combine all the tensor forms to construct a TTD-MLA network structure;

[0108] Input training data into the TTD-MLA network structure, rearrange the training data according to the tensor core of each linear layer to obtain a vector sequence, input the vector sequence into the group vector systolic array architecture in the TTD-MLA network structure, and output multiple intermediate data;

[0109] Input all the intermediate data into a digital signal processor. The digital signal processor determines the displacement amount according to all the intermediate data, and processes all the intermediate data according to the displacement amount to output the final result.

[0110] Among them, the building of the large language model, performing tensorization on multiple linear layers in the large language model to obtain multiple high-dimensional tensors, performing TTD compression on all the high-dimensional tensors to obtain the tensor data of all the linear layers, and constructing corresponding tensor forms according to all the tensor data specifically includes:

[0111] Build a large language model, obtain the weight matrices corresponding to multiple linear layers in the large language model, and perform tensorization on all the weight matrices to obtain multiple high-dimensional tensors:

[0112]

[0113]

[0114] Among them, T j represents the j-th weight matrix, M1 and M2 represent the two dimensions of the weight matrix, represents the set of real numbers, d represents the dimension number after tensorization, v i represents the i-th dimension of the high-dimensional tensor, T ′ j represents the j-th high-dimensional tensor;

[0115] Perform TTD compression on each high-dimensional tensor to obtain the tensor data and corresponding tensor core corresponding to each linear layer:

[0116] T ′j (l 1j ,...,l dj ) = G 1j [l 1j ...G dj [l dj ;

[0117]

[0118] Among them, T ′ j (l 1j ,...,l dj ) represents the tensor data of T ′ j . l 1j represents the l 1j -th slice of the first tensor core G 1j after TTD compression. l dj represents the l dj -th slice of the d-th tensor core G dj after TTD compression. l kj represents the slice of the k-th tensor core in the j-th linear layer. r k-1 is the rank of the tensor core G k-1j . r k is the rank of the tensor core G kj ;

[0119] Construct the tensor form corresponding to each said linear layer according to each said tensor data:

[0120]

[0121] Among them, Y(j 1j ,...,j dj ) represents the tensor form of the linear layer corresponding to T j . G 1j [i 1j ,j 1j represents the first tensor form of the original weight of the linear layer corresponding to T j . G dj [i dj ,j dj represents the d-th tensor form of the original weight of the linear layer corresponding to T j . G kj represents the k-th tensor form of the original weight of the linear layer corresponding to T j . X represents the tensor form of the original input of the linear layer. i dj and j dj both represent G dj [i dj ,j djIndex of ], m d Denote Y(j 1j ,..., j dj )'s last dimension, G kj [i k , j k denote T j The k-th tensor form of the original weight of the corresponding linear layer, m k and n k both denote G kj [i k , j k 's dimension.

[0122] Among them, each of the high-dimensional tensors is compressed by TTD to obtain the tensor data and the corresponding tensor cores corresponding to each linear layer. After that, it further includes:

[0123] When each linear layer performs inference, the corresponding tensor core is used as the weight data of each linear layer, and each weight data is used to replace the linear layer weight of the corresponding linear layer.

[0124] Among them, the key matrix and the value matrix in the large language model are decomposed by SVD to obtain multiple sampling matrices. After TTD compression of all the sampling matrices, combined with all the tensor forms, a TTD-MLA network structure is constructed, specifically including:

[0125] Obtain the key matrix and the value matrix in the large language model, and decompose the key matrix and the value matrix into multiple upsampling matrices and a single downsampling matrix by using the SVD decomposition method;

[0126] After TTD compression of all the upsampling matrices and the downsampling matrix, an MLA structure network is obtained;

[0127] According to all the tensor forms, the MLA structure network is combined with all the linear layers to obtain a TTD-MLA network structure.

[0128] Among them, the training data is input into the TTD-MLA network structure, and the training data is rearranged according to the tensor cores of each linear layer to obtain a vector sequence. The vector sequence is input into the group vector systolic array architecture in the TTD-MLA network structure, and multiple intermediate data are output, specifically including:

[0129] Obtain the training data input by the user, and input the training data into the TTD-MLA network structure;

[0130] The TTD-MLA network structure rearranges the training data according to all the tensor cores to obtain a vector sequence:

[0131]

[0132] Among them, represents a vector sequence, and t k-1 represents the dimension of the k-th stage of the TTD decomposition process, and j k represents the tensor form of the k-th stage, and n k represents the number of training data of the k-th stage, and P k-1 [i k , t k-1 represents the vector sequence of the (k - 1)-th stage;

[0133] Divide the said vector sequence into multiple group vectors, and input each said group vector into multiple vector operation units in the group vector systolic array architecture in the TTD-MLA network structure for parallel processing, and output corresponding intermediate data respectively.

[0134] Among them, after dividing the said vector sequence into multiple group vectors, and inputting each said group vector into multiple vector operation units in the group vector systolic array architecture in the TTD-MLA network structure for parallel processing, and outputting corresponding intermediate data respectively, it further includes:

[0135] Cache each said intermediate data into the ping-pong buffer in the TTD-MLA network structure, and accumulate it into the accumulator in the TTD-MLA network structure to obtain accumulated data;

[0136] Write the accumulated data into the double data rate memory through the direct memory access in the TTD-MLA network structure.

[0137] Among them, inputting all the said intermediate data into a digital signal processor, the digital signal processor determines a displacement amount according to all the said intermediate data, and processes all the said intermediate data according to the displacement amount, and outputs a final result, specifically including:

[0138] Input all the said intermediate data and all the said weight data into the digital signal processor;

[0139] The digital signal processor determines the displacement amount of the maximum exponent bit in all the said weight data, shifts all the said intermediate data according to the displacement amount, and then inputs them into an adder tree;

[0140] Input the accumulated data into the adder tree, calculate the product between each said intermediate data and the accumulated data, and output the final result.

[0141] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a data processing program of a multi-head latent attention model based on TTD compression, and when the data processing program of the multi-head latent attention model based on TTD compression is executed by a processor, the steps of the data processing method of the multi-head latent attention model based on TTD compression as described above are implemented.

[0142] In summary, the present invention provides a data processing method and related devices for a multi-head latent attention model based on TTD compression. The method includes: constructing a large language model, performing tensorization processing on multiple linear layers in the large language model to obtain multiple high-dimensional tensors, performing TTD compression on all the high-dimensional tensors to obtain tensor data of all the linear layers, and constructing corresponding tensor forms according to all the tensor data; performing SVD decomposition on the key matrix and value matrix in the large language model to obtain multiple sampling matrices, performing TTD compression on all the sampling matrices, and combining all the tensor forms to construct a TTD-MLA network structure; inputting training data into the TTD-MLA network structure, rearranging the training data according to the tensor kernel of each linear layer to obtain a vector sequence, inputting the vector sequence into the group vector systolic array architecture in the TTD-MLA network structure, and outputting multiple intermediate data; inputting all the intermediate data into a digital signal processor, and the digital signal processor determines a displacement amount according to all the intermediate data and processes all the intermediate data according to the displacement amount to output a final result. By transforming into the multi-head latent attention structure in the large language model, the present invention reduces the demand for KV cache during model inference, improves the long text output ability of the model on edge devices by reducing the number of weights, and subsequently uses the computing structure of the group vector systolic array to perform matrix calculations during the inference process, efficiently utilizing limited hardware resources.

[0143] It should be noted that in this article, the terms "including", "comprising" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or terminal. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or terminal including the element.

[0144] Of course, those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware (such as a processor, a controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium readable by a computer. When the program is executed, it can include the processes of the above method embodiments. The computer-readable storage medium can be a VCU 128, a magnetic disk, an optical disk, etc.

[0145] It should be understood that the application of the present invention is not limited to the above examples. For those of ordinary skill in the art, improvements or transformations can be made according to the above description. All such improvements and transformations should fall within the protection scope of the appended claims of the present invention.

Claims

1. A data processing method for a multi-head latent attention model based on TTD compression, characterized in that The data processing method of the multi-head latent attention model based on TTD compression includes: Construct a large language model, perform tensorization processing on multiple linear layers in the large language model to obtain multiple high-dimensional tensors, perform TTD compression on all the high-dimensional tensors to obtain the tensor data of all the linear layers, and construct corresponding tensor forms according to all the tensor data; Perform SVD decomposition on the key matrix and value matrix in the large language model to obtain multiple sampling matrices. After performing TTD compression on all the sampling matrices, combine all the tensor forms to construct a TTD-MLA network structure; Input training data into the TTD-MLA network structure, rearrange the training data according to the tensor cores of each linear layer to obtain a vector sequence, and input the vector sequence into the group vector systolic array architecture in the TTD-MLA network structure to output multiple intermediate data; Input all the intermediate data into a digital signal processor. The digital signal processor determines a displacement amount according to all the intermediate data, and processes all the intermediate data according to the displacement amount to output a final result.

2. The data processing method of the multi-head latent attention model based on TTD compression according to claim 1, wherein The construction of the large language model, the tensorization processing of multiple linear layers in the large language model to obtain multiple high-dimensional tensors, and the TTD compression of all the high-dimensional tensors to obtain the tensor data of all the linear layers and the construction of corresponding tensor forms according to all the tensor data specifically include: Construct a large language model, obtain the weight matrices corresponding to multiple linear layers in the large language model, and perform tensorization processing on all the weight matrices to obtain multiple high-dimensional tensors: Among them, T j represents the j-th weight matrix, and M1 and M2 represent the two dimensions of the weight matrix. represents the set of real numbers, d represents the number of dimensions after tensorization, and v i represents the i-th dimension of the high-dimensional tensor, and T ′ j represents the j-th high-dimensional tensor; Perform TTD compression on each high-dimensional tensor to obtain the tensor data and corresponding tensor cores corresponding to each linear layer: T ′ j (l 1j ,...,l dj )=G 1j [l 1j ...G dj [l dj ; Among them, T ′ j (l 1j ,...,l dj ) represents the tensor data of T ′ j . l 1j represents the l 1j -th slice of the first tensor core G 1j after TTD compression. l dj represents the l dj -th slice of the d-th tensor core G dj after TTD compression. l kj represents the slice of the k-th tensor core in the j-th linear layer. r k-1 is the rank of the tensor core G k-1j . r k is the rank of the tensor core G kj ; Construct the tensor form corresponding to each linear layer according to each tensor data: Among them, Y(j 1j ,...,j dj ) represents the tensor form of the corresponding linear layer of T j , G 1j [i 1j ,j 1j represents the first tensor form of the original weight of the corresponding linear layer of T j , G dj [i dj ,j dj represents the d-th tensor form of the original weight of the corresponding linear layer of T j , G kj represents the k-th tensor form of the original weight of the corresponding linear layer of T j , X represents the tensor form of the original input of the linear layer, i dj and j dj both represent the indices of G dj [i dj ,j dj , m d represents the last dimension of Y(j 1j ,...,j dj ), G kj [i k ,j k represents the k-th tensor form of the original weight of the corresponding linear layer of T j , m k and n k both represent the dimensions of G kj [i k ,j k .

3. The data processing method of the multi-head latent attention model based on TTD compression according to claim 1, wherein After performing TTD compression on each high-dimensional tensor to obtain the tensor data and corresponding tensor cores corresponding to each linear layer, it further includes: When each linear layer performs inference, use the corresponding tensor core as the weight data of each linear layer, and use each weight data to replace the linear layer weight of the corresponding linear layer.

4. The data processing method of the multi-head latent attention model based on TTD compression according to claim 1, wherein The SVD decomposition of the key matrix and value matrix in the large language model to obtain multiple sampling matrices, the TTD compression of all the sampling matrices, and the combination of all the tensor forms to construct a TTD-MLA network structure specifically include: Obtain the key matrix and value matrix in the large language model, and decompose the key matrix and the value matrix into multiple upsampling matrices and a single downsampling matrix by using the SVD decomposition method; After performing TTD compression on all the upsampling matrices and the downsampling matrix, obtain an MLA structure network; According to all the tensor forms, combine the MLA structure network with all the linear layers to obtain a TTD-MLA network structure.

5. The data processing method of the multi-head latent attention model based on TTD compression according to claim 2, wherein Input the training data into the TTD-MLA network structure, rearrange the training data according to the tensor of each linear layer to obtain a vector sequence, and input the vector sequence into the group vector systolic array architecture in the TTD-MLA network structure to output multiple intermediate data, specifically including: Obtain the training data input by the user and input the training data into the TTD-MLA network structure; The TTD-MLA network structure rearranges the training data according to all the tensors to obtain a vector sequence: Among them, represents the vector sequence, and t k-1 represents the dimension of the k-th stage of the TTD decomposition process, and j k represents the tensor form of the k-th stage, and n k represents the number of training data of the k-th stage, and P k-1 [i k ,t k-1 represents the vector sequence of the (k - 1)-th stage; Divide the vector sequence into multiple group vectors, and input each group vector into multiple vector operation units in the group vector systolic array architecture in the TTD-MLA network structure for parallel processing, and output the corresponding intermediate data respectively.

6. The data processing method of the multi-head latent attention model based on TTD compression according to claim 1, wherein After dividing the vector sequence into multiple group vectors and inputting each group vector into multiple vector operation units in the group vector systolic array architecture in the TTD-MLA network structure for parallel processing and outputting the corresponding intermediate data respectively, it further includes: Cache each intermediate data into the ping-pong buffer in the TTD-MLA network structure and accumulate it into the accumulator in the TTD-MLA network structure to obtain accumulated data; Write the accumulated data into the double data rate memory through the direct memory access in the TTD-MLA network structure.

7. The data processing method of the multi-head latent attention model based on TTD compression according to claim 3, characterized in that Input all the intermediate data into a digital signal processor. The digital signal processor determines the displacement amount according to all the intermediate data, and processes all the intermediate data according to the displacement amount to output the final result, specifically including: Input all the intermediate data and all the weight data into the digital signal processor; The digital signal processor determines the displacement amount of the maximum exponent bit in all the weight data, performs a shift operation on all the intermediate data according to the displacement amount, and then inputs it into the adder tree; Input the accumulated data into the adder tree, calculate the product between each intermediate data and the accumulated data, and output the final result.

8. A data processing system for a multi-head latent attention model based on TTD compression, characterized in that, The data processing of the multi-head latent attention model based on TTD compression is applied to the data processing method of the multi-head latent attention model based on TTD compression according to any one of claims 1-7. The data processing system of the multi-head latent attention model based on TTD compression includes: A tensor compression module for constructing a large language model, performing tensorization processing on multiple linear layers in the large language model to obtain corresponding high-dimensional tensors, performing TTD compression on all the high-dimensional tensors to obtain tensor data of all the linear layers, and constructing corresponding tensor forms according to all the tensor data; A model construction module for performing SVD decomposition on the key matrix and value matrix in the large language model to obtain multiple sampling matrices, performing TTD compression on all the sampling matrices, and then constructing a TTD-MLA network structure in combination with all the tensor forms; A group of vector systolic array modules is used to input training data into the TTD-MLA network structure, rearrange the training data according to the tensor cores of each linear layer to obtain a vector sequence, input the vector sequence into the group vector systolic array architecture in the TTD-MLA network structure, and output multiple intermediate data; A floating-point calculation module is used to input all the intermediate data into a digital signal processor. The digital signal processor determines a displacement amount according to all the intermediate data, and processes all the intermediate data according to the displacement amount to output a final result.

9. A terminal, characterized in that, The terminal includes: a VCU 128, a processor, and a data processing program of a multi-head latent attention model based on TTD compression stored on the VCU 128 and executable on the processor. When the data processing program of the multi-head latent attention model based on TTD compression is executed by the processor, the steps of the data processing method of the multi-head latent attention model based on TTD compression as described in any one of claims 1-7 are implemented.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a data processing program of a multi-head latent attention model based on TTD compression. When the data processing program of the multi-head latent attention model based on TTD compression is executed by a processor, the steps of the data processing method of the multi-head latent attention model based on TTD compression as described in any one of claims 1-7 are implemented.

Images