Systems and methods for compressing large language models using tensor networks

Abstract

A computer-implemented method for compressing pre-training of a large language model (LLM) is provided. The method comprises identifying (S101) a layer of the LLM (47) with a weight matrix (48), decomposing (S104a) the weight matrix (48) of the LLM (47) into a tensor network (49), compressing (S104b) the tensor network (49), and storing (S104c) the compressed tensor network (49) in a data storage unit (40).

Description

[0001] Cross-reference to related applications This application claims priority to European Patent Application No. EP 23383011.6, filed October 2, 2023; European Patent Application No. EP 24172279.2, filed April 24, 2024; and U.S. Patent Application No. EP18 / 654,223, filed May 3, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present invention relates to methods and systems for computer implementation of pre-trained layers for compressing large language models (LLM). Background Technology

[0003] A large language model (LLM) is a type of language model renowned for its ability to achieve general language understanding and generation. LLMs acquire these capabilities by learning billions of parameters using massive amounts of input data during their training. LLMs consume significant computational resources during both training and operation. They are based on artificial neural networks (primarily transformers) and can be (pre-)trained using both self-supervised and semi-supervised learning. As a so-called "autoregressive language model," an LLM works by taking input text and repeatedly predicting the next token or word.

[0004] The transformer concept is based on a deep learning architecture developed by Google and a multi-head attention mechanism. Input text is converted into a numerical representation—called a "tag"—and this tag is transformed into a vector via a lookup from a table. This tag is then contextualized with other tags in a leaching layer via the multi-head attention mechanism. This contextualization allows signals for key tags to be amplified, while signals for less important tags are attenuated.

[0005] Neural network architectures based solely on attention mechanisms are known in the field. For example, Ashish Vaswani et al., in their paper "Attention is All You Need," published on December 4, 2017 (arXiv:1706.03762v7), used this approach for machine translation. This attention mechanism is a neural network layer added to a deep learning model, focusing the model's attention on specific parts of the data by assigning different weights to different parts of the data. For instance, in machine translation, the attention mechanism is used to align and selectively focus on relevant parts of the source sentence during the translation process by amplifying signals corresponding to key parts of the source sentence. Therefore, a learning model using an attention mechanism is able to assign different weights to signals of more important words or phrases in the source sentence.

[0006] The fundamental element of Vaswani's LLM model is the so-called "layer." Vaswani's paper describes a multi-head attention module (MNB) used to compute attention scores between different parts of an input sequence (such as the source sentence mentioned above). The input sequence is presented to the MNB from the input of the LLM model. The MNB consists of so-called "multiple attention heads." Each attention head computes a separate set of attention scores based on the (same) input sequence. These attention scores are then combined to produce a final set of attention scores used to weight the input sequence. For example, by allowing the LLM to focus on different parts of the input sequence (i.e., the source sentence or other text) at different times, the MNB can be used for natural language processing tasks. Focusing on different parts of the input sequence allows the LLM model to capture long-range dependencies between different words in the sentence of the input sequence and to handle the case of variable-length input sequences (i.e., source sentences with different numbers of words).

[0007] Since the publication of Vaswani et al.'s paper, attention mechanisms have played a role in several tasks that rely on deep networks. Attention mechanisms in attention-based architectures have been used in the known domain of large language models (LLMs). For example, attention mechanisms have been integrated into large language models such as BERT, GPT, BART, or LLAMA. These large language models are used for tasks in deep networks, such as translation, text summarization, question answering, and chatbot functions.

[0008] The architecture of LLM consists of multiple layers of different types of neural networks, such as recurrent layers, feedforward layers, embedding layers, and the aforementioned attention layers. These different types of neural network layers work together to process the input text and generate output predictions. LLM is trained on massive text datasets using deep learning techniques to learn syntax, semantics, and context, and typically employs a so-called "transformer architecture".

[0009] The transformer architecture is the neural network described in the Vaswani paper mentioned above. The transformer architecture is able to understand the relationships in the input sequence and allows prediction of the next word in a sentence within the input sequence. The transformer architecture is a type of neural network that processes the entire data sequence at once, rather than processing one element at a time like a "traditional" recurrent neural network. As described in the Vaswani paper, the transformer architecture consists of two main components: an encoder and a decoder. The encoder receives the input sequence and generates a hidden representation of that input sequence. The decoder then takes that hidden representation and generates the output sequence.

[0010] Attention-based architectures have been used to accomplish tasks such as translation, text summarization, question answering, and chatbot functionality. However, attention-based architectures require significant computational power (and energy) for training and use in deep networks. For example, pre-training a 3 LLaMA-2-Chat language model requires a cumulative 3.3 million GPU hours of computation on an A100-80GB hardware with a maximum thermal design power (TDP) of 350-400W. The estimated carbon dioxide (CO2) emissions during the pre-training of the 3 LLaMA-2-Chat language model are 539 tons of CO2 equivalent.

[0011] Furthermore, even after pre-training a large language model, it is impossible to run it on a classic computing device with GPU capacity, even with the minimum number of parameters (e.g., the Falcon-7B requires 7 billion parameters). The Falcon 7B language model alone requires approximately 27GB of RAM to run with 7 billion parameters on a classic computing device. Therefore, it is challenging for users to run large language models even with the minimum number of parameters on a classic computing device.

[0012] One of the challenges of using attention-based architectures is the sheer amount of resources required to store the language model being trained and to run the pre-trained language model on classical computing devices. The large amount of resources required to train a language model stems from the massive datasets needed and the very high dimensionality of the parameter space. In machine learning, a neural network is a model composed of directed graphs with weights (i.e., real numbers) on the edges of that graph. The parameter space is called the weight space, and the learning process involves updating the parameters, for example, through gradient descent.

[0013] Therefore, it is necessary to reduce the amount of resources required to train and store large language models.

[0014] There are two known challenges in existing technologies: finding compression methods to compress multi-head attention modules in converter architectures while preserving sufficiently high information quality. The first challenge relates to the nonlinearity of converter architectures, making compression difficult. The second challenge is that compressed multi-head attention models cannot be directly integrated into the encoder and decoder frameworks of converter architectures.

[0015] The purpose of this document is to provide a method and system that reduces the amount of resources required to train and store large language models. Summary of the Invention

[0016] This document describes a method for training and running language models only on data processing machines with extremely high computing power. It is well known that not all parameters of a language model are independent. Therefore, the parameter space can be reduced if some parameters are represented as a compilation of the other parameters of the language model. Representing parameters in a compiled manner can be achieved using tensor networks that are known to represent the states of a physical system.

[0017] This document describes integrating pre-trained layers of an LLM architecture into a tensor network layer, which stores the parameters of a language model in a more efficient manner.

[0018] The approach proposed in this document involves "tensification" of the self-attention layers and multi-perceptron layers of a large language model using tensor networks. Tensification effectively truncates the correlations present in the large language model (LLM). The degree of truncation can be controlled via the bond dimension of the tensor network. This method can significantly reduce the size of the LLM (and the required storage space) while maintaining accuracy. In practice, the compressed LLM requires less energy and memory, and operations such as training, retraining, and inference become more efficient and require fewer resources.

[0019] In the first aspect, the method is a computer implementation for compressing pre-trained layers of a Large Language Model (LLM). A Large Language Model has multiple layers and weight matrices. The method involves identifying the layers of the LLM with weight matrices, then decomposing the LLM's weight matrices into a tensor network, and then compressing this tensor network. The compressed tensor network can be stored in a data storage unit. Decomposing the weight matrix into a tensor network reduces the size of the tensor network, thus eliminating storage requirements.

[0020] The layers in an LLM are, for example, at least one of self-attention layers or multi-perceptron layers. These layers are used in an LLM.

[0021] On one hand, the weight matrix is ​​decomposed into a tensor network by creating a tensor star composed of multiple tensors. The dimension of the tensors is smaller than the dimension of the weight matrix.

[0022] On one hand, multiple tensors (in a tensor network) include at least one pre-programmed core tensor. In one implementation, the LLM has a predefined core tensor.

[0023] Compression methods include, for example, performing edge permutations on the nodes of a tensor network using a random search algorithm. This allows for experimentation with various compression methods to identify the optimal compression that reduces data size while preserving information.

[0024] On the other hand, the method further includes splitting the edges of the nodes of the tensor network into n groups and merging the edges of the nodes of the tensor network into a single index vector.

[0025] On the other hand, the method further includes determining the optimal virtual edge dimension of at least one of the matrix multiplication operator (MPO) form or the matrix multiplication state (MPS) form.

[0026] Finally, the computer-implemented method includes reconstructing the initial weight matrix to enable the LLM to be used. On the other hand, the method further includes calculating the differences between the elements of the initial weight matrix and the reconstructed weight matrix. The method described below can be repeated several times to determine the optimal decomposition of the tensor network, which reduces the required storage while providing sufficient information.

[0027] On the other hand, the document describes a computer system for compressing parameters of a large language model (LLM). This computer system includes a compression module that implements algorithms for compressing the parameters of a large language model.

[0028] This method can be used in large language models (LLMs) within deep neural networks, which are used to implement at least one of the following tasks: translation, text summarization, question answering, or chatbot functionality. Attached Figure Description

[0029] Figure 1 An example of a computer system for implementing the methods described in this document is shown.

[0030] Figure 2 The tensor representation of the weight matrix is ​​shown.

[0031] Figure 3 A flowchart of the method described in this document is shown.

[0032] Figure 4 The Tucker decomposition of the attention tensor is shown.

[0033] Figure 5 A flowchart of the Tucker decomposition is shown.

[0034] Figure 6 A flowchart of the random search algorithm is shown.

[0035] Figure 7 The flowchart for tensor layer initialization is shown.

[0036] Figure 8 A comparison of the time required for convergence of the Heston model for tensor neural networks and classical dense neural networks is shown.

[0037] Figure 9The benchmarks are shown as follows: (a) Retraining time (in hours) for each epoch on the Gigaword and XSum datasets for the compressed (left) and uncompressed (right) LlaMa-2 models using distributed retraining. (b) Forward and backward propagation time (in hours) for the compressed (left) and uncompressed (right) LlaMa-2 models on the Gigaword and XSum datasets using distributed retraining.

[0038] Figure 10 The table shows the Rouge scores of the compressed (left) and uncompressed (right) LlaMa-2 models for the (a) Gigaword and (b) XSum datasets. Rouge1 refers to the overlap of unary phrases, Rouge2 refers to the overlap of bigrams, RougeL refers to the statistics of the longest common subsequence in the text, and RougeLsum is similar to RougeL but takes into account sentence-level information. Detailed Implementation

[0039] The invention will now be described with reference to the accompanying drawings. It should be understood that the embodiments and aspects of the invention described herein are merely examples and do not limit the scope of the claims in any way. The invention is defined by the claims and their equivalents. It should be understood that features of one aspect or embodiment of the invention may be combined with features of different aspects and / or embodiments of the invention.

[0040] Figure 1 An example of a computing system for implementing the computer-based methods of this document is shown. Figure 1 A computing system 100 is shown with a (classical) central processing unit (CPU) 10, which has at least one input device 20 and at least one output device 30. The system 100 is shown using a single CPU 10, but it should be understood that the system 100 can be expanded by adding a quantum processor 50 or a quantum emulator (running classically) connected to the CPU 10.

[0041] In one example, the computing system 100 includes a (classical) central processing unit 10 connected to a data storage unit 40 (i.e., one or more memory devices), and one or more input devices 20 and one or more output devices 30. The one or more input devices 20 are capable of inputting training data 45 stored in the data storage unit 40 and later used as input to the input sequence 60.

[0042] One or more graphics processing units (GPUs) 35 for handling vector and tensor computations and a field-programmable gate array (FGPA) 41 for control logic, which may also be connected to the CPU 10. If more than one GPU 35 is provided, vector and tensor computations for training a large language model (LLM) can be distributed across different GPUs 35. This enables distributed training of the LLM if needed.

[0043] The computing system 100 is connected to a computer network such as the Internet. It should be understood that... Figure 1 The arrangement of the computing system 100 is merely exemplary, and other units or elements may exist in the computing system 100. It should also be understood that there may be more than one input device 20 and more than one output device 30 located in multiple locations, and multiple data storage units 40 also located in multiple locations. Many input devices 20 and output devices 30, as well as data storage units 40, are connected via a computer network.

[0044] The large language model 47 is stored in data storage unit 40 and fed data via a variety of input devices 20. The sources of data are diverse. For example, it is well known that some large language model developers have crawled websites and scraped the internet to obtain data. Other large language models are built from internal data. The methods described in this document are not limited to any particular data source.

[0045] Now we will combine Figure 3 A method for compressing the amount of data stored in data storage unit 40 is described. Figure 3 A flowchart of the method is shown. In the first step S101, the initial layer of the LLM 47 is identified. The selection of the identified initial layer is arbitrary. When the initial layer is identified, the next step S102 is to integrate the identified initial layer of the LLM 47 into the tensor network layer. This step S102 is referred to as the recycling or upgrading of the LLM 47 layer.

[0046] In an unrestricted example, the language model Llama-2-7b-hf This is used to implement the method, but it does not limit the invention, and other language models can be used. Llama-2-7b-hf The model is a fine-tuned text generation model with (currently) 7 billion parameters. Llama-2-7b-hf The model has been optimized for dialogue use cases and converted to the Hugging Face Transformer format. The Hugging Face Transformer is provided by the Hugging Face community at huggingface.co. Llama-2-7b-hfThe module is part of the Llama-2 family of large language models (LLMs), which includes pre-trained and fine-tuned text-generating models ranging from 7 billion to 70 billion parameters. Llama-2-7b-hf The model has a self-attention layer and a multi-perceptron layer, which have a weight matrix 48, which can be tensorized as described below.

[0047] Once the identified layers have been recovered in step S102, the next step is to begin the tensor quantization process in S103. The tensor is generated by T... αβγ The multidimensional array represented by T αβγ This describes a multilinear relationship between sets of algebraic objects associated with a vector space. The subscripts α, β, and γ denote the tensor dimension (3 in this case, due to the three subscripts), and this value is called the tensor's "rank." Tensors can map between different objects, such as vectors, scalars, and other tensors. There are many types of tensors, including scalars and vectors, dual vectors, multilinear mappings between vector spaces, and dot products. The definition of a tensor is independent of any basis.

[0048] Tensor quantization is the process of transforming or mapping low-order data to high-order data. For example, the low-order data can be a vector, and the tensor quantization result can be a matrix, a third-order tensor (i.e., rank-3), or a higher-order tensor. The low-order data can also be a matrix or a third-order tensor. As mentioned above, tensor quantization is commonly used to compress data and reduce its complexity while preserving its original characteristics. In a non-limiting example of this application, the data is at least one of the text data required for translation, text summarization, question answering, and chatbot functions.

[0049] In step S103, to initiate the tensor quantization process, a weight matrix 48 representing the layer is required. The weight matrix 48 is also stored in the data storage unit 40. In machine learning, the weight matrix 48 is a numerical matrix representing the connection strength between nodes in a layer of an LLM 47 neural network. Numerical values ​​are used to transform input data from the input sequence within the hidden layers of the neural network. For example, when input data enters a node in the neural network, it is multiplied by a weight value. The resulting output from the node is either observed or passed to the next layer in the neural network. The weight matrix 48 is typically contained within the hidden layers of the neural network. The weight matrix 48 is adjusted during neural network training to improve the accuracy of language model predictions.

[0050] Tensor Neural Networks (TNNs) and Tensor Convolutional Neural Networks (TCNNs) are examples of deep neural networks (NNs) in which the weight matrix 48 of the hidden layers of a large language model 47 is replaced by a tensor network 49 created using, for example, singular value decomposition (SVD), as described later. Tensor Neural Networks offer better performance and accuracy than standard deep neural networks by reducing the parameter space and thus the amount of storage required to store the LLM 47. In tensor Neural Networks, tensorization occurs only at the level of the hidden layers (e.g., trainable weights). However, training of the language model is typically performed by a CPU 10 and one or more GPUs by optimizing the shrinking trainable weight matrix 48 of each layer based on standard optimization techniques (e.g., gradient descent and automatic differentiation).

[0051] Standard optimization techniques are efficient and accurate. However, these known standard optimization techniques only target the global minimum of the loss function. Correlating and entangled with the parameters of the weight matrix 48 in LLM 47 is challenging for standard optimization techniques. Standard optimization techniques are also difficult to extend. Monitoring the behavior of the loss function to assess training convergence in these methods, and in principle, distinguishing between local minima and the actual global minimum, is very difficult.

[0052] An efficient representation of the weight matrix 48 of the large language model 47 can be obtained by replacing it with a matrix product operator (MPO) having the key dimension χ. The MPO is created by performing sequential singular value decomposition (SVD) on the weight matrix 48 and preserving the largest χ singular value at each SVD. The replacement weight matrix 48 with a new tensor quantized layer containing several trainable weights is then represented by the MPO. The resulting TNN is scalable and can have any desired number of TN layers to form a deep neural network.

[0053] It is necessary to know the final shape of the weight matrix 48 (i.e., the input dimension and the output dimension). The most common way to handle the weight matrix representation problem is to split the initial input and output dimensions into smaller dimensions such that the tensor rank of each new dimension is a prime number. However, the use of tensor ranks as prime numbers does not limit this invention. The product of the tensor ranks of all new dimensions is equal to the product of the input and output dimensions of the module layer.

[0054] Figure 2 The tensor representation of the initial weight matrix W, with dimensions of 216x216, is shown to illustrate this. It should be understood that the value 216x216 is merely illustrative and does not limit the invention. The initial two dimensions of the weight matrix are split into smaller dimensions, such as... Figure 2 The central "tensor star" is shown, where the product of the smaller dimensions equals the initial dimension. Then, the tensor star is transformed through two consecutive SVD transformations to form 2×36χ+36χ. 2The resulting tensor network of parameters is equal to the sum of the parameters of each tensor, where χ is the MPO bond dimension, used as the truncation parameter. Figure 2 In the illustrated representation of MPO, circles represent individual tensors within a tensor, lines represent tensor indices, and lines connecting circles represent shrink-shared indices between tensors.

[0055] It should be understood that in step S104a, there are infinitely many possibilities to decompose the weight matrix W into Figure 2 The "tensor star" shown in the middle left is then decomposed into Figure 2 The tensor network on the right. However, not all possibilities lead to efficient decomposition. To reduce the number of possibilities used for tensor decomposition, several core tensors are adapted for each training model task. For example, the user training the model can pre-program, set, or determine several core tensors.

[0056] Following the decomposition in step S104a, the tensors are compressed in step S104b via the indices of the tensors in the truncated tensor network 49, and stored in the data storage unit 40 in step S104c. The compressed tensors can be used to recalculate new weight matrices for the correlation layers. It should be noted that the size of the (recalculated) new weight matrix will be slightly different from the size of the initial (before decomposition) weight matrix because less correlated information is lost during the compression process.

[0057] The reconstruction of the weight matrix 48 in step S105 is achieved by shrinking the MPO in step S106 and then reshaping the MPO into the form of a tensor star in step S107. Finally, the tensor star is reshaped back into matrix form in step S108.

[0058] In step S109, the differences between the elements of the initial weight matrix 48 and the reconstructed weight matrix are calculated to determine whether the reconstructed weight matrix is ​​smaller than the initial weight matrix 48. In step S110, this process is repeated several times using different permutations, keeping the smallest value in the reconstructed weight matrix. This value will provide an acceptable decomposition.

[0059] like Figure 4 As shown, the core tensor is a compressed tensor used in the Tucker decomposition of a larger tensor. Figure 4 The diagram shows a block representing the query matrix Q, key matrix K, and value matrix V of LLM 47. The query matrix Q and key matrix K are derived from the input sequence of LLM 47. The key matrix K represents the relation strength of words in the input sequence. The value matrix V is computed from the query matrix Q and key matrix K to be used as the output. Figure 4In this example, we assume the matrix consists of three factor matrices formed by three sets of orthogonal basis vectors. R is the rank of the tensor, and N is the length of the input sequence. The dimension of the matrix is ​​given by d. The new single-block attention is the core tensor G.

[0060] In step S104, Tucker decomposition decomposes the tensor network 49 into a set of matrices and a small core tensor. Tucker decomposition reduces the size of the index of the large tensor with minimal information loss. The resulting compressed tensor is the "core" tensor. The core tensor ranges from the number 2 to the rank of the tensor representation. The number of core tensors is a hyperparameter in the neural network.

[0061] Figure 5 The method of Tucker decomposition S104 is illustrated. A single-block attention module is constructed based on Tucker decomposition. Tucker decomposition is a low-rank decomposition method. In Vaswani et al.'s paper, the multi-head attention module is the main part of the transformer architecture. The multi-head attention module consists of a large number of parameters. As described in the paper, Vaswani et al. computed the multi-head attention module based on the query set derived simultaneously from the input sequence. The query set is packaged together into the query matrix Q, and the key matrix and value matrix are packaged together into the key matrix K and value matrix V, respectively. The multi-head attention module employs a nonlinear function on the two matrices Q and K. softmax .

[0062] The nonlinear function of the converter in the converter architecture is difficult to compress. To overcome this challenge, in step S401, the output of the multi-head attention function of the self-attention model is linearly represented by a set of orthogonal basis vectors. After the linear representation, the low-rank core tensor is initialized using Tucker decomposition in step S402. In step S403, the new multi-head attention representation is reconstructed using factor matrices Q, K, and V through Tucker decomposition.

[0063] Block-term tensor decomposition (BTD) is used to construct multi-head attention using factor matrices Q, K, and V, thereby compressing the language model. Compression of the large language model using BTD occurs in step S404. The multilinear attention module uses the idea of ​​parameter sharing, for example, sharing factor matrices across multiple blocks. Block-term tensor decomposition (BTD) is a combination of CP (CANDECOMP / PARAFAC) decomposition and Tucker decomposition. Unlike prior art, in step S405, the three factor matrices Q, K, and V are shared when constructing each 3rd-order block tensor.

[0064] In step S406, the 3rd-order block tensor is used to reconstruct the scaled dot-product self-attention module in the transformer by summing along a specific dimension. This document discloses an LLM compression method that combines the low-rank approximation principle and the parameter sharing principle. Therefore, higher compression ratios are achieved through LLM compression using this method. The self-attention modules (e.g., scaled dot-product attention) in the transformer network are split into 3rd-order block tensors (outputs of multilinear attention), which allows for improved accuracy.

[0065] A known challenge in the prior art is that multi-head attention models cannot be directly integrated into the encoder and decoder framework of a transformer network after LLM compression. Three steps are required to address this challenge. In the first step, the average value of each block tensor is calculated. In the second step, multiple matrices are formed through tensor splitting. In the third step, the concatenation of these multiple matrices is used as input to the next layer in the transformer network. After performing these three steps, the multi-head attention model can be integrated into the encoder and decoder framework of the transformer network and trained end-to-end.

[0066] The principle will now be explained in detail. The self-attention function can be represented by a linear function, for example, by a linear combination of basis vectors.

[0067] The output of the self-attention function is represented by a linear combination of the basis vector sets: in These are basis vectors from the vector space S. Are they linearly independent and Q , K , V It is a factor matrix, which is composed of basis vectors. The set is represented linearly (as described in step S401). It is a coefficient matrix, and (as described above) d It is a factor matrix Q , K ,and V Dimensions.

[0068] A new attention function can be constructed using a single attention module. In step S402, the 3rd-order diagonal tensor is initialized. g Third-order diagonal tensor g It is a trainable tensor. R It is the rank of the tensor. N This is the length of the input sequence. The function of the single-block attention module is computed based on Tucker decomposition: in G It is the core tensor.i , j and m It is the core tensor G The index. "It is the outer product, the operator" "represented as order and Tensor-tensor product on ). , and It comes from the matrix Q , K and V , where and In a non-restrictive example, I = J = M = R Core tensor G The definition is as follows: .

[0069] in rand(0,1) It is a random function, and the core tensor G The diagonal entries form a vector g Each entry .vector g It is a trainable weight matrix. g It can be done softmax Function calculation. softmax The function converts a real vector into a probability distribution.

[0070] The output of the single-block attention function is a 3rd-order tensor obtained through linear computation. Existing techniques for compressing multi-head modules involve multiple sets of linear mappings. This document addresses the matrix... Q , K and V Three linear mappings are used. At the output of the three linear mappings, in step S403, the three factor matrices are... Q , K and V These are considered to be the three factor matrices for reconstructing multilinear attention in step S403.

[0071] Figure 4 The reconstruction of multilinear attention is shown. Let's assume... h It refers to the number of multi-head modules. d It is a factor matrix Q , K and V The dimension. The compression ratio is calculated by the following formula: (3 xhxd ) / (3 x d+h In a non-restricted example, the number of multi-head modules...h Set to 8, size d It was set to 512. In this unrestricted example, the compression ratio reached 8. In other words, the parameters in the attention layer were reduced by nearly eight times.

[0072] Hyperparameters in machine learning are adjustable parameters defined by the user to control the machine learning process. Hyperparameters are used to improve the learning of large language models. The values ​​of the hyperparameters are set before the learning process of a large language model begins. In contrast, the values ​​of other parameters (typically node weights) are derived through training the language model.

[0073] For each core tensor, the location of physical edges is chosen, along with the method for connecting the core tensor to other nodes in the tensor network. A physical edge is an edge that connects to only one node in the tensor network. In a non-restricted example, nodes are connected in an MPS (matrix product state) structure.

[0074] Another hyperparameter required by the user is the selection of the maximum virtual edge dimension. A virtual edge is the edge between two nodes in a tensor network. A virtual edge dimension that is too large will destroy the parameter reduction effect provided by the tensor quantization process. A virtual edge dimension that is too small may cause the output matrix to differ significantly from the input matrix when the MPS structure shrinks.

[0075] The optimal virtual edge dimension for the MPS form should be selected to achieve the best algorithmic performance. For example, the genetic permutation optimization algorithm can be used to determine the optimal virtual edge dimension for the MPS form. In a non-restricted example, a random search of permutations is used to determine the optimal virtual edge dimension for the MPS form. In machine learning, random search is a strategy that uses random combinations of hyperparameters to identify the best answer for an established model.

[0076] Figure 6 The random search algorithm is illustrated. In the first step S201 of the random search algorithm, random permutations are performed on the edges of the nodes of the tensor network 49.

[0077] In step S202, the edges of the node are split into n Group, among which n This is the number of tensors in TensorNet49.

[0078] In step S203, all physical edges of the tensor network 49 are merged into an index. This index is a multi-index tensor transformed into a single index vector.

[0079] In step S204, MPS decomposition is performed.

[0080] In step S205, the initial weight matrix is ​​reconstructed through the following steps: first, the MPS form is shrunk, then the MPS form is reshaped into a permuted Cartesian form, followed by a step of permuting the edges of the tensor network with random permutations, and finally the Cartesian form is reshaped into a matrix form.

[0081] In step S206, the differences between the elements of the initial (i.e., before decomposition) weight matrix and the elements of the reconstructed weight matrix are calculated. If the number of parameters is less than the number of elements in the initial weight matrix, and the maximum difference between the two factors is less than the previous best value, then MPS is saved as the optimal shrinkage strategy.

[0082] In step S207, steps S201-S207 are repeated. m Next, among them m These are user-defined extrinsic parameters. After repeating steps S201-S207, the optimal MPS decomposition is defined.

[0083] After defining the optimal MPS decomposition in step S104, the tensor quantization layer is constructed in step S105. The optimal MPS decomposition is the decomposition that achieves the highest possible accuracy while losing as little relevant information as possible during weight layer compression. In one non-limiting example, the optimal accuracy is 87% of the original model, compressed to 70% of the original model. In another non-limiting example, the accuracy is 85% of the original model, compressed to 60% of the original model. In yet another non-limiting example, the accuracy is 81% of the original model, compressed to 50% of the original model.

[0084] Figure 8 This diagram compares the convergence time of a Tensor Neural Network (TNN) and a Classical Dense Neural Network (DNN) to the Heston model. The Heston model is a mathematical model used in finance to describe the evolution of the volatility of an underlying asset. According to... Figure 8 TNNs converge 12 times faster than equivalent DNNs with the same number of parameters. For example... Figure 8 As shown, even the smallest TNN architecture (e.g., with 161 parameters) outperforms the equivalent DNN in terms of convergence speed.

[0085] The construction of a tensor layer involves four steps, such as... Figure 7 As shown. In the first step S301, to initialize the tensor layer, the weight matrix is ​​replaced with a tensor network 49 to collect information about the tensor quantization technique. This information allows for the reconstruction of the initial weight matrix. Using the (compressed) tensor network 49, the initial weight matrix can be reconstructed with high accuracy.

[0086] In the second step S302, the parameters of the tensor layer are extracted from the tensor network 49.

[0087] In the third step S303, values ​​are removed from the tensor network 49 to free up memory and avoid retaining unnecessary data in the data storage unit 40 of the system 100.

[0088] In the fourth step S304, the selected technology and bias are initialized based on the input from the user.

[0089] The second change to the tensor layer occurs during the feedforward step. Before the classic forward step, the weight matrix needs to be reconstructed. This reconstruction provides a more efficient strategy for MPS shrinkage. For MPS shrinkage, the language model parameters need to be reintroduced into the tensor network 49. The tensor layer is then shrunk to obtain the weight matrix. After shrinking the tensor layer, the memory of the tensor network 49 structure is freed again to avoid retaining unnecessary data in the data storage unit 40.

[0090] When TensorNet49 is initialized, the tensor quantization layer is replaced in the initial language model. It is crucial to accurately allocate the tensor quantization layer where it is recycled. In a non-restrictive example, the tensor layer is precisely allocated to the same location by preserving the name of the initial module.

[0091] Benchmarking To evaluate the method proposed in this document, it was used to compress the LlaMA-2 7B model. This model represents the "smallest" LLM in the "large" LLM category of the open-source LlaMA family developed by META. As mentioned above, the model contains 7 billion parameters and has been pre-trained on over 2 trillion labels. The model is provided with a context length of 4096 and has been fine-tuned with over 1 million human annotations. In float32, the model consumes 24GB of memory, and after lightweight scalarization, it consumes 12GB of memory in float16.

[0092] This method incorporates an MPO with a key dimension of χ≈100 in the SA and MLP layers of the float16 version of LlaMA-2 7B. Consequently, the model is reduced to 2 billion parameters and 3.7 Gb of memory, which is 30% of its original non-tensor size in float16 and 15% of the original LlaMA-27B in float32 (if lightweight scaling is also considered). In other words, the tensor network compression method has already reduced the number of parameters in the model and its memory size to 30% of the original size, while the lightweight scaling from float32 to float16 further reduces the size by a factor of 2.

[0093] To evaluate the model's performance, a text summarization task was used. For this, two open-source datasets were selected: XSum and Gigaword. Both the original and compressed models underwent a limited number of additional training iterations using these datasets. Notably, the training time of the compressed model was approximately twice that of the uncompressed version, see [link to relevant documentation]. Figure 9 Following this training, the Rouge score was calculated. The Rouge score is a metric used to evaluate automatic text summarization and machine translation. Figure 10 The illustration compares the Rouge scores of the two models after retraining on two datasets. This comparison shows that the compressed model retains approximately 90% of the accuracy of the original model in float16, despite being only 30% of its original size.

[0094] The computation for this benchmark was performed on a single AWS machine with eight NVIDIA A100 Tensor Core GPUs using distributed retraining, which in turn demonstrates the method's GPU compatibility.

[0095] Figure Labels 10 Central Processing Units (CPU) 20 input devices 30 output devices 35 graphics processing units (GPUs) 37 Field Programmable Gate Array 40 data storage units 41 Field Programmable Gate Array (FGPA) 45 training data 47 major language models 48 weight matrix 49 Tensor Networks 50 quantum processors 60 Input Data 100 system.

Claims

1. A computer-implemented method for compressing pre-trained layers of a large language model (LLM) (47) having multiple layers and weight matrices (48), the method comprising: Identify (S101) the layer of the LLM (47) having the weight matrix (48); The weight matrix (48) of the LLM (47) is decomposed (S104a) into a tensor network (49). Compress (S104b) the tensor network (49); and The tensor (49) is stored (S104c) in the data storage unit (40).

2. The computer-implemented method of claim 1, wherein the layer of the LLM (47) is at least one of a self-attention layer or a multi-sensor layer.

3. The computer-implemented method of claim 1 or 2, wherein decomposing the weight matrix (S104a) into the tensor network (49) includes creating a tensor star formed by a plurality of tensors, the dimensions of which are smaller than the dimensions of the weight matrix (48).

4. The computer-implemented method of claim 3, wherein the plurality of tensors includes at least one pre-programmed core tensor.

5. The computer-implemented method as described in any of the preceding claims, wherein the compression (S104b) comprises performing a permutation (S201) on the edges of the nodes of the tensor network using a random search algorithm.

6. The computer-implemented method of claim 5, further comprising partitioning (S202) the edges of the nodes of the tensor network (49) into n Group.

7. The computer-implemented method of any one of claims 5 to 6, further comprising merging the edges of the nodes of the tensor network (S203) into a single index vector.

8. The computer-implemented method as described in any of the preceding claims further includes determining (S204) the optimal virtual edge dimension of at least one of a matrix multiplication operator (MPO) form or a matrix multiplication state (MPS) form.

9. The computer-implemented method as described in any of the preceding claims further includes reconstructing (S105, S205) the weight matrix (48).

10. The computer-implemented method as described in any of the preceding claims further includes (S206) calculating (the difference between the elements of the weight matrix (48) and the reconstructed weight matrix.

11. The computer-implemented method as described in any of the preceding claims, further comprising repeating steps S201-S206 (S207). m Second-rate.

12. A computer system for compressing parameters of a large language model (LLM), the computer system comprising a compression module (110) implementing an algorithm for compressing the parameters of the large language model, wherein the algorithm uses the method of at least one of claims 1 to 11.

13. Use of the method as described in any one of claims 1 to 12, wherein the large language model (LLM) is a deep neural network used to perform at least one of translation, text summarization, question answering, or chatbot functional tasks.

Images