Large language model training method and device
By performing matrix and vector processing in the target attention head of the large language model, combined with linear attention mechanism and mask layer pruning technology, the high computing overhead problem of the large language model under safe multi-party computing is solved, and the acceleration is achieved Improved reasoning and accuracy.
Patent Information
- Application Number
- CN202410842855.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-27
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-06-27
AI Technical Summary
The overhead of model encryption inference calculation of large language models under secure multi-party computing is huge, hindering its practicality in real-life services and affecting user experience.
By using query transformation matrix, key transformation matrix and value transformation matrix in the target attention head of the large language model, and compressing it into a global query vector and global attention vector, replacing the dot product attention with a linear attention mechanism. The mechanism reduces the computational complexity and further optimizes through mask layer and model pruning technology.
It effectively reduces the complexity of model calculation, accelerates the inference process, improves the efficiency of security inference, and maintains the accuracy of the model and improves the user experience.
Smart Images

Figure CN118445379B_ABST
Abstract
Description
Technical Field
[0001] One or more embodiments of this specification relate to the field of machine learning technology, and more particularly, to a method and device for training a large language model. Background Art
[0002] Large Language Model (LLM) has been widely used to process various natural language tasks, such as text summary generation, question answering, text translation, etc., due to its excellent semantic understanding ability.
[0003] In real scenarios, LLM-based services need to transmit the query information entered by users to the server. The query information often involves sensitive information such as personal interests, medical data, financial information, and source code, which causes users to worry about privacy issues.
[0004] One way to mitigate privacy issues is to use secure multi-party computing (MPC) technology. MPC ensures privacy by encrypting sensitive data such as user data and model weights and sharing them securely during LLM model reasoning.
[0005] However, the computational overhead of dense model reasoning under MPC greatly hinders its practicality in real-world services. Therefore, a solution is needed that can reduce the computational overhead of dense model reasoning under MPC while ensuring the accuracy of the LLM model, improve the reasoning speed, thereby reducing the waiting time for user queries and improving user experience. Summary of the invention
[0006] The embodiments of this specification describe a method and device for training a large language model, which can better meet the needs of practical applications.
[0007] According to a first aspect, a training method for a large language model is provided. The training method comprises: in a target attention head of a large language model, respectively processing the output of the previous layer for input text using a query transformation matrix, a key transformation matrix, and a value transformation matrix to obtain a query matrix, a key matrix, and a value matrix; compressing the query matrix into a global query vector, and fusing the global query vector with multiple key vectors in the key matrix to obtain an attention matrix; compressing the attention matrix into a global attention vector, and fusing the global attention vector with multiple value vectors in the value matrix to obtain the output of the target attention head to determine the target output of the large language model; and training the large language model based on the target output.
[0008] In a possible implementation, the rows and columns of the query matrix correspond to the sequence length and feature dimension of the input text, respectively; wherein, compressing the query matrix into a global query vector includes: normalizing each column of the query matrix to obtain a first weight matrix; performing a bitwise multiplication process on the query matrix and the first weight matrix to obtain a first weight matrix; and superimposing each row of the first weight matrix to obtain the global query vector.
[0009] Furthermore, in some possible embodiments, the trained large language model is used for reasoning under secure multi-party computing (MPC); wherein, each column of the query matrix is normalized separately, including: for each column of the query matrix, determining the function value of the linear rectification function ReLU corresponding to each column element; for each column element, calculating the proportion of the function value of the corresponding ReLU function in the accumulated value of the function value as the corresponding normalization result.
[0010] In some possible embodiments, the global query vector and multiple key vectors in the key matrix are fused separately to obtain an attention matrix, including: for each key vector, after performing a bitwise multiplication process on it and the global query vector, superimposing the query vectors at the same position corresponding to the key vector in the query matrix to obtain an attention vector for forming the attention matrix.
[0011] In some possible embodiments, the rows and columns of the attention matrix correspond to the sequence length and feature dimension of the input text, respectively; wherein, compressing the attention matrix into a global attention vector includes: normalizing each column of the attention matrix to obtain a second weight matrix; performing bitwise multiplication on the attention matrix and the second weight matrix to obtain a second weight matrix; and superimposing each row of the second weight matrix to obtain the global attention vector.
[0012] In some possible embodiments, the output of the target attention head is obtained by respectively fusing the global attention vector and multiple value vectors in the value matrix, including: respectively fusing the global attention vector and the multiple value vectors to obtain an encoding matrix for the input text; processing the encoding matrix using a linear transformation matrix to obtain the output of the target attention head.
[0013] Furthermore, in some possible examples, the global attention vector and multiple value vectors in the value matrix are fused separately to obtain an encoding matrix for the input text, including: for each value vector, after performing a bitwise multiplication process on it and the global attention vector, superimposing the attention vectors at the same position corresponding to the value vector in the attention matrix to obtain an encoding vector and form the encoding matrix.
[0014] In a possible implementation, before obtaining the query matrix, the key matrix, and the value matrix, the method further includes: adding a mask layer based on a pre-trained Transformer model to obtain a mask model; training the mask model according to the distillation loss between the Transformer model and the mask model and the sparsity constraint related to the mask layer parameters; and pruning the mask model according to the parameters of the trained mask layer to obtain a pruned model as the large language model.
[0015] Furthermore, in some possible embodiments, the trained large language model is used for reasoning under secure multi-party computing (MPC) calculations; wherein, adding a mask layer based on a pre-trained Transformer model comprises: for the pre-trained Transformer model, replacing the nonlinear activation function contained therein with an approximate function, wherein each operator in the approximate function has a corresponding MPC protocol; and adding the mask layer to the replaced Transformer model.
[0016] In some possible embodiments, the attention layer in the Transformer model includes h attention heads and a linear transformation sublayer; wherein, adding a mask layer includes: adding a first mask layer between the h attention heads and the linear transformation sublayer, and the first mask vector in the first mask layer is used to mask the outputs of the h attention heads.
[0017] Furthermore, in some possible examples, the mask model is pruned according to the parameters of the trained mask layer, including: sorting the vector elements in the first mask vector from small to large; and according to the sorting result, removing the attention heads corresponding to the vector elements ranked in a predetermined front range from the h attention heads of the mask model.
[0018] In some specific examples, model pruning is performed according to the parameters of the trained mask layer, including: sorting the vector elements in the first mask vector from small to large; and according to the sorting result, replacing the attention heads whose corresponding vector elements are within a predetermined front range among the h attention heads of the mask model with the target attention heads.
[0019] In some possible embodiments, the feedforward layer in the Transformer model includes a dimensionality increase sublayer, a nonlinear activation function layer and a dimensionality reduction sublayer; wherein, the adding of the mask layer includes: adding a second mask layer between the nonlinear activation function layer and the dimensionality reduction sublayer, the second mask matrix in the second mask layer is used to mask the input of the dimensionality reduction sublayer, and the second mask matrix is obtained by diagonalizing the second mask vector.
[0020] Furthermore, in some possible examples, the dimensionality reduction sublayer includes a dimensionality reduction weight matrix; wherein, the mask model is pruned according to the parameters of the trained mask layer, including: sorting the vector elements in the second mask vector from small to large; and according to the sorting result, removing the matrix rows corresponding to the vector elements ranked in a predetermined front range from the dimensionality reduction weight matrix of the mask model.
[0021] In some possible embodiments, the Transformer model includes a linear layer; wherein, adding a mask layer includes: adding a third mask layer before the linear layer, a third mask matrix in the third mask layer is used to mask the input of the linear layer, and the third mask matrix is obtained by diagonalizing a third mask vector.
[0022] Furthermore, in some possible examples, the linear layer includes a linear transformation matrix; wherein, the mask model is pruned according to the parameters of the trained mask layer, including: sorting the vector elements in the third mask vector from small to large; and according to the sorting result, removing the matrix rows corresponding to the vector elements within a predetermined front range from the linear transformation matrix of the mask model.
[0023] In some possible embodiments, the initial parameters of the mask layer are determined in the following manner: performing the i-th sampling on a predetermined uniform distribution to obtain the i-th sampling element; processing the i-th variable using a sigmoid function to obtain the i-th result as the i-th initial parameter; wherein the i-th variable is a combination result of the i-th sampling element and the parameter to be trained.
[0024] In some possible embodiments, the sparsity constraint includes minimizing the gap between a current sparsity and a preset target sparsity, wherein the current sparsity is calculated according to current parameters of the mask layer.
[0025] In some possible embodiments, the large language model is trained based on the target output, including: inputting the input text into the Transformer model for model processing to obtain a supervisory signal; and performing distillation learning based on the target output and the supervisory signal to obtain a trained large language model.
[0026] In some possible embodiments, the training of the large language model is an i-th round iteration of the large language model, and the i-th round iteration includes replacing part of the attention heads in the i-th last attention layer of the large language model obtained after the previous round iteration with the target attention head, and updating the last i attention layers in the large language model.
[0027] According to a second aspect, a training device for a large language model is provided, comprising: a mapping unit, configured to process the output of the previous layer for input text in a target attention head of the large language model using a query transformation matrix, a key transformation matrix and a value transformation matrix, respectively, to obtain a query matrix, a key matrix and a value matrix; a first interaction unit, configured to compress the query matrix into a global query vector, and fuse the global query vector with multiple key vectors in the key matrix, respectively, to obtain an attention matrix; a second interaction unit, configured to compress the attention matrix into a global attention vector, and fuse the global attention vector with multiple value vectors in the value matrix, respectively, to obtain the output of the target attention head, so as to determine the target output of the large language model; a training unit, configured to train the large language model based on the target output.
[0028] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method provided in the first aspect.
[0029] According to a fourth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method provided in the first aspect is implemented.
[0030] In summary, the large language model training method and device disclosed in the embodiments of this specification can effectively reduce the complexity of model calculation and accelerate reasoning, and is particularly suitable for accelerating secret security reasoning under MPC, while enabling the trained large language model to maintain the accuracy of reasoning. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0032] Figure 1 A schematic diagram showing the structure of the basic modules in the Transformer model;
[0033] Figure 2 A schematic diagram of the process steps of the method for pruning a Transformer model disclosed in an embodiment of this specification;
[0034] Figure 3 This is a schematic diagram of an implementation of adding a first mask layer to an attention head according to an embodiment of this specification;
[0035] Figure 4 This is a schematic diagram of implementing adding a second mask layer to a dimension reduction sublayer in a feedforward layer disclosed in an embodiment of this specification;
[0036] Figure 5 A schematic diagram of the steps of the training method for a large language model disclosed in the embodiments of this specification;
[0037] Figure 6 A data flow diagram for implementing a linear attention mechanism in an attention head according to an embodiment of this specification;
[0038] Figure 7 This is a schematic diagram of the structure of the functional modules of the large language model training device disclosed in the embodiments of this specification. DETAILED DESCRIPTION
[0039] The solution provided in this specification is described below in conjunction with the accompanying drawings.
[0040] As mentioned above, the computational overhead of dense inference of large language models under MPC is huge, which hinders its practicality in real-world services.
[0041] At present, most large language models are implemented based on the Transformer mechanism. Transformer is a deep learning framework based on the self-attention mechanism. Its core advantage is that it can imitate the way humans allocate attention when processing information, selectively focusing on more important information, thereby effectively improving the efficiency and accuracy of natural language tasks.
[0042] In order to improve the efficiency of dense-state reasoning, previous work mainly explored simplifying the bottleneck operation of the Transformer model (hereinafter referred to as the LLM model) in dense-state calculations, namely the nonlinear activation function.
[0043] Different from previous work, the applicant proposed an improvement scheme to improve the efficiency of secure reasoning by reducing the number of multiplication calculations in the Transformer model.
[0044] Specifically, the improvement plan involves two stages. In the first stage, the Transformer model is pruned to eliminate redundant parameters in the model, thereby reducing the number of multiplication operations in the Transformer. In the second stage, the original dot product attention mechanism is replaced by a linear attention mechanism, thereby reducing the multiplication complexity of the attention mechanism from the square of the length of the input text sequence to linear, that is, from a quadratic relationship to a linear relationship.
[0045] It should be noted that stage one and stage two can be performed independently, such as executing only stage one or only stage two, or they can be performed jointly. For example, stage two is performed based on the model obtained in stage one, and for example, the standard for performing attention head replacement in stage two adopts the attention head mask variable derived in stage one.
[0046] Next, we will first introduce stage one of the above improvement scheme, or the pruning stage, and then introduce stage two, or the attention mechanism replacement stage.
[0047] For the pruning stage, the main purpose is to remove redundant components in certain specific structures in the Transformer model, such as redundant attention heads in the attention layer, redundant dimensions in the intermediate dimensions of the feedforward layer, redundant dimensions in the output dimensions of the attention layer and the feedforward layer, etc. To help understand, the overall structure of the Transformer model is briefly introduced below.
[0048] The Transformer model consists of a series of L (≥1) blocks, where the structure of any j-th block can be found in Figure 1 .like Figure 1 As shown in Figure 3, the jth module includes an attention layer, a feed-forward layer, a residual connection, and a layer normalization operation.
[0049] 1) Attention Layer
[0050] The attention layer uses the Multi-Head Attention (MHA) mechanism, which means that the attention layer contains multiple attention heads. Figure 1 Indicated in indivual.
[0051] In the attention layer, The input of the attention heads is the same, denoted as ,in Indicates the length of the text sequence, for example, 1024 characters; Indicates the feature dimension corresponding to the sequence element, such as 32 dimensions. Any of the attention heads Attention heads process the input The output produced is recorded as , where the processing involves computing:
[0052] (1)
[0053] in, Represents the self-attention mechanism; , , , represent the query transformation matrix, key transformation matrix and value transformation matrix respectively.
[0054] Afterwards, the weight matrix in the linear transformation sublayer Acts on The output of an attention head { H i } [h] , thus obtaining the output of the attention layer.
[0055] From the above, we can consider the attention layer to be The calculation process is simply recorded as:
[0056] (2)
[0057] in, Express The outputs of the attention heads are concatenated. This concatenation is not in the sequence length dimension, but in the feature dimension. The output before concatenation is located at dimensional space, the result after splicing is located in dimensional space, where ,This concatenation operation enables the model to fuse information from different representation spaces.
[0058] The above introduces the composition of the attention layer in the module and the computational processing involved.
[0059] 2) Feedforward layer
[0060] The feedforward layer, or feedforward neural network, includes a dimensionality increase sublayer, a nonlinear transformation sublayer (or a nonlinear activation function layer), and a dimensionality reduction sublayer.
[0061] The dimension-raising sublayer contains the dimension-raising weight matrix , used to In indivual Dimension vector is upgraded to dimensional space. It is important to understand that the input of the feedforward layer, that is, the input of the dimension-raising sublayer, has the same dimension as the input of the attention layer. To express this intuitively, reuse Represents the input of the feed-forward layer.
[0062] The nonlinear activation function is used to perform nonlinear activation processing on the output of the dimension-increased sub-layer.
[0063] The dimension reduction sublayer contains the dimension reduction weight matrix , which is used to convert the nonlinear activation indivual dimensional vector is mapped back dimensional space.
[0064] From the above, we can conclude that The calculation process is recorded as:
[0065] (3)
[0066] in, represents the activation function. For example, a Gaussian Error Linear Unit (Gaussian Error Linear Unit, referred to as ), or Tanh, Swish, etc.
[0067] 3) Residual connection and layer normalization operations
[0068] After the attention layer and the feed-forward layer, there are residual connections and layer normalization operations, respectively.
[0069] The form of the residual connection is:
[0070] (4)
[0071] in, is the input of the current layer (such as the attention layer), is the output calculated by this layer, Represents the result of residual connection. Residual connection can alleviate the gradient vanishing problem and accelerate convergence.
[0072] The form of the layer normalization operation is:
[0073] (5)
[0074] in, is the input of the operation, and They are The mean and variance of and are learnable parameters.
[0075] Combination of the above Figure 1, briefly introduces the structure of the basic module in the Transformer model. The pruning stage of the improved scheme involves pruning redundant components in the attention layer, feedforward layer, etc. in the basic module.
[0076] Next, the specific implementation steps of the pruning stage are introduced with reference to more drawings and embodiments. Figure 2 The following is a schematic diagram of the steps of the method for pruning the Transformer model disclosed in the embodiments of this specification. It can be understood that the execution subject of the method can be any device, platform, server or device cluster with computing and processing capabilities. Figure 2 As shown, the method comprises the following steps:
[0077] Step S210, adding a mask layer based on the pre-trained Transformer model to obtain a mask model; step S220, training the mask model according to the distillation loss between the Transformer model and the mask model, and the sparsity constraint related to the mask layer parameters; step S230, performing model pruning processing according to the parameters of the trained mask layer to obtain a pruned model.
[0078] The following is an expansion of the above steps:
[0079] First, in step S210, a mask layer is added based on a pre-trained Transformer model to obtain a mask model.
[0080] It should be understood that pre-training means that the model has been trained in advance using massive text data. There are many ways to obtain a pre-trained Transformer model. For example, you can directly download a pre-trained open source large model from the Internet, or you can train it yourself.
[0081] In addition, when the pruned model is used for MPC dense inference, the applicant proposes that before pruning, or before executing this step, the nonlinear activation function in the Transformer model, for example, Function, for example, used to instantiate the activation function in equation (3) above Functions are replaced with approximate functions that support MPC calculations, thereby obtaining a more MPC-friendly pruning model.
[0082] In one embodiment, the Softmax function can be approximated as:
[0083] (6)
[0084] In another embodiment, the GeLU function can be approximated as:
[0085] (7)
[0086] The above introduces the approximate replacement of activation functions in the Transformer model. It should be noted that the pruned model is particularly suitable for MPC computing scenarios, but not only for scenarios that require reducing computing overhead, such as scenarios with limited hardware computing power. Figure 2 The method shown performs model pruning.
[0087] Back to the implementation of this step, the addition of the mask layer is targeted at the specific structure that needs to be cut. The following introduces the addition of the mask layer in combination with several examples of structural components to be cut.
[0088] 1) Attention heads in the attention layer
[0089] like Figure 3 As shown, a first mask layer can be added between the h attention heads and the linear transformation sublayer of the attention layer to perform mask processing on the outputs of the h attention heads.
[0090] It should be noted that the "first" in the first mask layer, as well as the "second", "third" and other similar terms in other places in the text, are all used to distinguish similar things and do not have other limiting functions such as sorting. In addition, the first mask layer can be selectively added to any number of modules (blocks). For example, the first mask layer can be set in L modules. When more than one module in the L modules is added with the first mask layer, the parameters in different first mask layers can be independent of each other or shared.
[0091] The parameters in the first mask layer include h first mask variables defined for h attention heads { z head i } [h] , where any The first mask variable is denoted by , accordingly, the calculation process in the attention layer changes from the above formula (2) to:
[0092] (8)
[0093] It should be noted that during the training phase, Can be a continuous value, the value range is generally [0,1], and will be used during clipping Perform binarization and remove or retain the corresponding attention heads according to the 0 or 1 obtained by binarization.
[0094] The above introduction adds the first mask layer to the attention head in the attention layer.
[0095] 2) Input dimension of the dimension reduction sublayer in the feedforward layer
[0096] like Figure 4 As shown, a second mask layer can be added between the nonlinear activation function of the feedforward layer and the dimensionality reduction sublayer to filter the input of the dimensionality reduction sublayer. The second mask layer can be selectively added to any number of blocks. For example, the second mask layer can be added to all L modules. When more than one of the L modules is added with the second mask layer, the parameters in different second mask layers can be independent of each other or shared.
[0097] The parameters in the second mask layer include Dimensional feature definition Second mask variable { z FFN i } [ d f ] , forming the second mask vector , accordingly, the calculation process in the feedforward layer changes from the above formula (3) to:
[0098] (9)
[0099] in, Express The second mask matrix is obtained by diagonalizing.
[0100] It should be noted that during the training phase, The second mask variable in can be a continuous value, and the value range is generally [0,1]. After that, each second mask variable will be binarized during pruning, and the dimensionality reduction weight matrix will be processed according to the 0 or 1 obtained by binarization. In row vectors to remove or keep.
[0101] The above describes adding a second mask layer to the input dimension of the dimensionality reduction sublayer in the feed-forward layer.
[0102] 3) Hidden state (or hidden representation) Dimensional Features
[0103] A third mask layer can be added before or after the linear layer in the module to Dimensional input or The dimensional output is masked.
[0104] Specifically, the linear layer refers to a linear transformation processing layer including a weight matrix, where the weight matrix can be the query transformation matrix mentioned above. , key transformation matrix , value transformation matrix , the weight matrix in the linear transformation sublayer , the dimension-increasing weight matrix in the feedforward layer and the dimension reduction weight matrix . Observe that the input dimension or output dimension of each weight matrix is Therefore, a third mask layer can be added before or after the linear layer to achieve Masking of dimensional features.
[0105] It should be noted that a third mask layer can be added to some or all of the linear layers in the Transformer model. For the third mask layer added to the same type of linear layers (such as the dimensionality-increasing sublayers in different modules), the parameters therein can be configured independently or shared.
[0106] The parameters in the third mask layer include Dimensional feature definition A third mask variable { z hidn i } [ d ] , forming the third mask vector For example, assume that a third mask layer is added before the query transform layer. In this case, the query transform layer Processing by , becomes ,in Express The third mask matrix obtained by diagonalizing.
[0107] It should be noted that during the training phase, The third mask variable in can be a continuous value, and the value range is generally [0,1]. After that, each third mask variable will be binarized during clipping, and the 0 or 1 obtained by binarization will be used to calculate the weight matrix. vectors to remove or keep.
[0108] The above introduction is for the hidden state dimensional features to add a third masking layer.
[0109] The above introduces the addition of mask layers in combination with three specific structural component examples. It should be understood that the methods of adding mask layers for different structural components can be implemented independently or in combination, for example, adding the first mask layer, the second mask layer, and the third mask layer at the same time in the Transformer model.
[0110] For the mask variables mentioned above, in one implementation, each mask variable is a parameter that needs to be trained, which means that there are as many mask parameters that need to be learned as there are mask variables.
[0111] In another embodiment, it is proposed to use L0 regularization optimization to reduce the amount of mask parameters that need to be trained. The distribution involved in L0 regularization optimization is as follows:
[0112]
[0113] (10)
[0114]
[0115] in, represents a uniform distribution in the interval [0,1], is a hyperparameter used to control the steepness of the sigmoid function. is a learnable parameter.
[0116] During the training process, according to formula (10), each mask variable is sampled separately , and maps it to a real number in the range [0,1] In this way, no matter how many mask variables need to be determined, only one parameter needs to be trained. In this way, the number of parameters that need to be trained is greatly reduced.
[0117] From the above, we can add a mask layer to the pre-trained Transformer model to obtain a mask model.
[0118] Then, in step S220, the mask model is trained according to the distillation loss between the Transformer model and the mask model, and the sparsity constraint related to the mask layer parameters.
[0119] The following first introduces the determination of distillation loss, and then introduces the necessity and specific method of introducing sparsity constraints.
[0120] 1) Distillation loss
[0121] Distillation loss is the loss function used in distillation learning. The distillation learning in this step aims to transfer the knowledge of the pre-trained Transformer model (as the teacher model) to the mask model (as the student model). This is achieved by setting the distillation loss to measure the difference between the outputs of each layer between the teacher model and the student model, thereby reducing the difference.
[0122] Exemplarily, the composition of distillation losses may include:
[0123] ① Alignment of hidden states after each module:
[0124] (11)
[0125] in, represents the number of modules in the teacher model and the student model, and Represent the teacher model and the student model respectively. The output of each module for the same input text is It is the abbreviation of Mean Square Error, which is translated into mean square error in Chinese.
[0126] ②Alignment of the output of the final prediction layer:
[0127] (12)
[0128] It should be understood that the prediction layer follows After the modules, its structure varies depending on the specific natural language task. For example, for text classification tasks, the prediction layer is usually a fully connected layer with an output dimension equal to the number of categories, and finally an activation function is applied to obtain the probability distribution of each category.
[0129] In formula (12), and Respectively represent the probability distribution output by the prediction layer of the teacher model and the student model for the same input text, It stands for KL divergence, which is used to measure the difference between two probability distributions, where K and L refer to the names Kullback and Leibler respectively.
[0130] By weighting the loss terms in ① and ②, we can get the distillation loss:
[0131] (13)
[0132] in, It is used for balance and The specific value of the hyperparameter is manually selected from the interval [0,1]. For example, .
[0133] The above introduces distillation losses.
[0134] 2) Sparsity Constraint
[0135] The applicant found that if only the distillation loss is used to train the mask model, the mask model is likely to be lazy and directly learn all mask variables to be 1. In this case, the distillation loss is 0. Therefore, it is proposed to further introduce a sparsity constraint to minimize the current sparsity of the mask model. and the preset target sparsity , thereby avoiding invalid learning of the mask model.
[0136] The current sparsity above It can be calculated based on the addition of mask layers in the mask model. For example, when the first mask layer, the second mask layer, and the third mask layer are added to the mask model, the current sparsity can be calculated using the following formula: :
[0137] (14)
[0138] in, represents the total number of parameters in the mask model, represents the dimensionality of the query, key, and value spaces, represents the number of modules in the mask model, represents the number of attention heads in the attention layer, represents the dimension of the hidden state, Indicates that for Module The mask variable of the attention head, Represents the hidden state The mask vector mask variables, represents the dimension of the up-dimensional space in the feed-forward layer, Indicates In the module The first dimension of the mask vector mask variable.
[0139] The sparsity constraint is based on the current sparsity and the preset target sparsity The corresponding calculation formula can be obtained as follows:
[0140] (15)
[0141] in, and These are all hyperparameters.
[0142] Based on the above distillation loss and sparsity constraint, the comprehensive loss can be obtained:
[0143] (16)
[0144] So in this step, we can minimize the comprehensive loss As a training goal, update the parameters of the mask model. In this way, by taking the sparsity of the mask model close to the target sparsity as one of the training goals, the complexity of model calculation can be effectively reduced and reasoning can be accelerated. At the same time, taking reducing the distillation loss as another training goal can make the mask model maintain the accuracy of reasoning.
[0145] In this step, the mask model is iteratively trained for multiple rounds until the model converges, thereby obtaining a trained mask model.
[0146] Afterwards, in step S230, model pruning is performed according to the mask layer parameters in the trained mask model to obtain a pruned model.
[0147] Specifically, for each mask layer, the corresponding structural components in the trained mask model can be removed or retained according to the values of each mask variable therein, thereby obtaining a pruned model. It should be understood that the value of the mask variable can be used to measure the importance of the component. The smaller the value, the lower the importance, and the tendency to be pruned or removed; in addition, in the pruning process, the mask layer is removed together, and the pruned model does not include the mask layer.
[0148] In one embodiment, it is assumed that the mask model includes a first mask layer. As described above, the first mask layer includes h first mask variables defined for h attention heads: { z head i } [h] , at this time, the pruning process in this step may include: { z head i } [h] The corresponding h variable values are sorted from small to large, and then according to the sorting result, the attention head corresponding to the first mask variable in the predetermined front range is removed from the corresponding h attention heads. For example, 3 attention heads are removed from 10 attention heads.
[0149] In another embodiment, it is assumed that the mask model includes a second mask layer. As described above, the parameters in the second mask layer include Dimensional feature definition Second mask variable { z FFN i } [ d f ] , at this time, the pruning process in this step may include: { z FFN i } [ d f ] Corresponding The variable values are sorted from small to large, and then according to the sorting results, the corresponding dimension reduction weight matrix In From the matrix rows, the row vectors corresponding to the second mask variables that are in the predetermined front range are removed. For example, 16 matrix rows are removed from the 64 matrix rows.
[0150] In yet another embodiment, it is assumed that the mask model includes a third mask layer. As described above, the parameters in the third mask layer include Dimensional feature definition A third mask variable { z hidn i } [ d ] , at this time, the pruning process in this step may include: { z hidn i } [ d ] Corresponding The variable values are sorted from small to large, and then according to the sorting result, the matrix row or column corresponding to the third mask variable that is in a predetermined front range is removed from the corresponding weight matrix.
[0151] From the above, we can prune the pre-trained Transformer model and obtain a pruned model.
[0152] In summary, the method for pruning the Transformer model disclosed in the embodiments of this specification can effectively reduce the complexity of model calculation, accelerate security reasoning, and at the same time enable the pruned model to maintain the accuracy of reasoning.
[0153] The above introduces the pruning phase of the improvement scheme. Next, we will introduce the specific implementation steps of the attention mechanism replacement phase. It should be noted that the relationship between the two phases will be further explained below.
[0154] Figure 5 The following is a schematic diagram of the steps of the training method for a large language model disclosed in the embodiments of this specification. It can be understood that the execution subject of the method can be any device, platform, server or device cluster with computing and processing capabilities. Figure 5 As shown, the method comprises the following steps:
[0155] Step S510, in the target attention head of the large language model, respectively use the query transformation matrix, the key transformation matrix and the value transformation matrix to process the output of the previous layer for the input text to obtain the query matrix, the key matrix and the value matrix. Step S520, compress the query matrix into a global query vector, and fuse the global query vector with multiple key vectors in the key matrix to obtain an attention matrix; Step S530, compress the attention matrix into a global attention vector, and fuse the global attention vector with multiple value vectors in the value matrix to obtain the output of the target attention head to determine the target output of the large language model; Step S540, train the large language model based on the target output.
[0156] The following is an expansion of the above steps:
[0157] First, in step S510, in the target attention head of the large language model, the query transformation matrix, the key transformation matrix and the value transformation matrix are used to process the output of the previous layer for the input text, respectively, to obtain the query matrix, the key matrix and the value matrix.
[0158] It should be noted that the large language model in this step contains one or more target attention heads. The attention mechanism used in the target attention head is not the traditional dot product attention mechanism, but the linear attention mechanism. For the dot product attention mechanism, please refer to the above formula (1), and for the linear attention mechanism, please refer to the introduction in the following text.
[0159] The large language model in this step can be obtained by replacing the traditional attention head with the target attention head. At this time, the initial large language model that has not been replaced can be the pruned model obtained in the above-mentioned pruning stage, or it can also be a pre-trained model without pruning.
[0160] Regarding the replacement of attention heads, in one implementation, all traditional attention heads in the initial large language model can be replaced with target attention heads. However, although the target attention heads can reduce computational complexity, they will also lead to a decrease in model accuracy.
[0161] In order to better balance the computational complexity and model accuracy, another implementation method is proposed to replace only some traditional attention heads. Compared with setting a replacement ratio threshold for random replacement, it is further proposed to replace the attention head according to the attention head mask vector derived in the pruning stage.
[0162] Specifically, the initial large language model that has not been replaced is the pruned model produced in the pruning stage. The mask model in the pruning stage includes a first mask layer set for the attention head in the attention layer, which includes h first mask variables that can measure the importance of the attention head. { z head i } [h] , thus, we can use the information retained in the attention layer after pruning The attention heads correspond to the variable values of the first mask variable, and decide which attention heads with relatively low importance are replaced.
[0163] Further, in one embodiment, the pruning model is first The attention head corresponds to The first mask variable values are sorted from small to large, and then the attention heads in the predetermined front range are replaced with the target attention heads. Experiments have shown that only retaining the last (or most important) traditional attention head can achieve higher model accuracy, and the rest of the attention heads can be replaced with the target attention heads.
[0164] On the other hand, assuming that the attention heads of multiple attention layers in the initial large language model need to be replaced, if the attention heads in all attention layers are replaced at once, the knowledge in the original attention heads will be insufficiently utilized. Therefore, it is proposed that in each round of iterative training, only the attention heads in one attention layer are replaced. In addition, for the replacement order, a back-to-front strategy can be adopted to reduce the error accumulation caused by the replacement of attention heads.
[0165] For example, suppose we need to prune the model Each attention layer replaces the attention head. In the iterative training rounds, the penultimate Replace the attention heads except the most important attention head in the attention layer, and only replace the replaced ones. Attention layer or The model parameters in each module are updated until the model converges, and then enter the next training round or end the training.
[0166] The large language model in this step is described above as being obtained by replacing the traditional attention heads with the target attention heads, and the replacement strategy is specifically introduced. In fact, the large language model in this step can also be obtained by self-design, in which one or more attention heads are initially the target attention heads.
[0167] For the implementation of this step, in the target attention head, the query transformation matrix is used respectively , key transformation matrix Sum transformation matrix Process the output of the previous layer for user query content , get the query matrix , key matrix Sum Matrix This can be written as:
[0168]
[0169] (17)
[0170]
[0171] in, , , , , , , .
[0172] It should be noted that formula (17) is the same as the calculation process in the traditional attention head. The improvement of the attention mechanism in the target attention head is reflected in the subsequent steps S520 and S530.
[0173] In addition, the previous layer of the target attention head needs to be discussed separately. When the target attention head is located in the first module, the previous layer can be the embedding layer used to embed the input text, and when the target attention head is not located in the first module, the previous layer can be the last processing layer in the previous module (for example, Figure 1 The layer normalization shown in 2).
[0174] From the above, in the target attention head, we can get the query matrix for the user's query content , key matrix Sum Matrix .
[0175] Next, in step S520, the query matrix Compressed into a global query vector , and the global query vector With key matrix The multiple key vectors in are fused separately to obtain the attention matrix .
[0176] For intuitive description, the query matrix Recorded as Q= [ q 1 , q 2 ,…, q N ] T ,in Indicates the first The corresponding elements dimensional query vector.
[0177] In this step, the query matrix Compressed into a global query vector There are many implementations, for example, indivual The vectors are directly added or averaged to obtain the global query vector ,at this time, The query matrix is integrated into The global context information in .
[0178] In addition, the applicant also proposed a more effective compression method, which can be combined with Figure 6 to understand. Specifically:
[0179] ①First, normalize each column of the query matrix to obtain the first weight matrix.
[0180] The above normalization process can use the Softmax function, etc. In addition, if the trained large language model is subsequently used in the MPC dense inference scenario, the normalization process can use an approximate function of the Softmax function, for example, the above formula (6), thereby converting the normalization of each column into The normalization process is recorded as:
[0181] (18)
[0182] Then the first weight matrix can be written as B= [ b 1 , b 2 ,…, b N ] T .
[0183] ② For the query matrix and the first weight matrix Perform bitwise multiplication processing to obtain a first weighted matrix.
[0184] ③Superimpose the rows of the first weighted matrix to obtain the global query vector .
[0185] The calculations in ② and ③ can also be written as:
[0186] (19)
[0187] in, Represents the bitwise multiplication operation, also known as element-wise multiplication operation or Hadamard product.
[0188] From the above, we can get the global query vector .
[0189] Furthermore, the global query vector With key matrix The multiple key vectors in are fused separately to obtain the attention matrix For intuitive description, the bond matrix Recorded as K= [ k 1 , k 2 ,…, k N ] T ,in Indicates the first The corresponding elements dimensional key vector.
[0190] In one implementation, the global query vector With key vector Add or concatenate.
[0191] In another embodiment, in order to better capture the global query vector With key vector The nonlinear relationship between them proposes a global query vector With key vector Perform a bitwise multiplication operation, which can be written as:
[0192] (20)
[0193] Furthermore, the applicant proposes that the idea of residual connection can also be introduced. The query vector is superimposed on , which can be written as:
[0194] (twenty one)
[0195] Based on the attention vector , we can get the attention matrix P= [ p 1 , p 2 ,…, p N ] T .
[0196] From the above, we can get the attention matrix .
[0197] Then, in step S530, the attention matrix Compressed into a global attention vector , and by the global attention vector With value matrix The multiple value vectors in are fused separately to obtain the output of the target attention head to determine the target output of the large language model.
[0198] In this step, the attention matrix Compressed into a global attention vector There are many implementations, for example, indivual Directly add or average to get the global attention vector ,at this time, The attention matrix is integrated into The global context information in .
[0199] In addition, the applicant also proposed a compression method with better effect, which can also be seen in Figure 6 . Specifically:
[0200] ①First, normalize each column of the attention matrix to obtain the second weight matrix.
[0201] The above normalization process can use the Softmax function, etc. In addition, if the trained large language model is subsequently used in the MPC dense inference scenario, the normalization process can use an approximate function of the Softmax function, for example, the above formula (6), thereby converting the normalization of each column into The normalization process is recorded as:
[0202] (twenty two)
[0203] The second weight matrix can then be written as C= [ c 1 , c 2 ,…, c N ] T .
[0204] ② Attention Matrix and the second weight matrix Perform bitwise multiplication processing to obtain a second weighting matrix.
[0205] ③Superimpose the rows of the second weighted matrix to obtain the global attention vector .
[0206] The calculations in ② and ③ can also be written as:
[0207] (twenty three)
[0208] From the above, we can get the global attention vector .
[0209] Furthermore, the global attention vector With value matrix The multiple value vectors in are fused separately to obtain the encoding matrix For intuitive description, the value matrix Recorded as V= [ v 1 , v 2 ,…, v N ] T ,in Indicates the first The corresponding elements dimensional vector of values.
[0210] In one implementation, the global attention vector With value vector Add or concatenate.
[0211] In another embodiment, in order to better capture the global attention vector With value vector The nonlinear relationship between them proposes a global attention vector With value vector Perform a bitwise multiplication operation, which can be written as:
[0212] (twenty four)
[0213] Furthermore, the applicant proposes that the idea of residual connection can also be introduced. The attention vector is superimposed on , which can be written as:
[0214] (25)
[0215] Based on the encoding vector , we can get the encoding matrix U= [ u 1 , u 2 ,…, u N ] T .
[0216] From the above, we can get the encoding matrix . Furthermore, the encoding matrix can be directly As the output of the target attention head, or, the encoding matrix can be processed by the linear transformation sublayer in the attention layer where the target attention head is located , and get the output of the target attention head. Therefore, based on the output of the target attention head, the target output of the large language model for the input text can be obtained. The specific process depends on the specific structure of the large language model, which is not the focus of this article and will not be elaborated.
[0217] Thereafter, in step S540, the large language model is trained based on the target output.
[0218] In one implementation, the large language model is obtained by replacing the traditional attention head with the target attention head. In this case, the large language model can be trained by distillation learning. Specifically, distillation learning can use the pre-trained traditional large language model as the teacher model, for example, the pre-trained Transformer model before pruning, and the current large language model with the attention head replaced as the student model.
[0219] In distillation learning, the difference between the outputs of each layer between the teacher model and the student model can be measured by setting the distillation loss, so as to reduce the difference and realize the transfer of knowledge from the teacher model to the student model. For details, please refer to the description of the distillation learning of the mask model in the above embodiment, which will not be repeated here.
[0220] In another implementation, the large language model is designed by itself, and one or more of the attention heads are initially the target attention heads, which have not been trained. In this case, the parameters in the large language model can be updated based on the target output and the corresponding label.
[0221] From the above, introducing the linear attention mechanism to replace the traditional dot product attention mechanism in the large language model can effectively reduce the computational complexity. Specifically, the computational complexity is proportional to the length of the text sequence. The square relationship decreases to a linear relationship. For ease of understanding, the following is a comparative analysis of the main differences in the two attention mechanisms.
[0222] 1) Computational complexity of the dot product attention mechanism
[0223] Observe the calculation in the traditional attention head, that is, formula (1):
[0224] (1)
[0225] It can be analyzed that in the query matrix and key matrix When performing computational interactions, the attention matrix obtained The number of elements in is , each element must be obtained correspondingly dimensional query vector and The dot product operation between the dimensional key vectors, so the computational complexity is Similarly, in the attention matrix With value matrix When performing computational interaction, it can be analyzed that the computational complexity is also .
[0226] Therefore, the computational complexity in the traditional attention head is .
[0227] 2) Computational complexity of linear attention mechanism
[0228] Observe the calculation in the target attention head and query the matrix in step S520 and key matrix When interacting, assuming that the compression process uses formula (19) and the fusion process uses formula (21), it can be analyzed that the computational complexity of these two formulas is Similarly, in step S530, the attention matrix Sum Matrix When interacting, we can analyze that the computational complexity is also .
[0229] Therefore, the computational complexity of the linear attention mechanism is .
[0230] Generally, The value is much larger than , so the computational complexity mainly depends on From the above analysis, we can see that the computational complexity of the newly designed linear attention mechanism is significantly reduced compared to the traditional dot product attention mechanism.
[0231] In summary, the large language model training method disclosed in the embodiments of this specification can effectively reduce the complexity of model calculation and accelerate security reasoning. At the same time, the trained large language model can maintain the accuracy of reasoning.
[0232] Corresponding to the above training method, the embodiment of this specification also discloses a training device. Figure 7 This is a schematic diagram of the structure of the functional modules of the large language model training device disclosed in the embodiments of this specification. Figure 7 As shown, the training device 700 includes the following components:
[0233] The mapping unit 710 is configured to process the output of the previous layer for the input text in the target attention head of the large language model using the query transformation matrix, the key transformation matrix and the value transformation matrix respectively to obtain the query matrix, the key matrix and the value matrix; the first interaction unit 720 is configured to compress the query matrix into a global query vector, and fuse the global query vector with multiple key vectors in the key matrix respectively to obtain the attention matrix; the second interaction unit 730 is configured to compress the attention matrix into a global attention vector, and fuse the global attention vector with multiple value vectors in the value matrix respectively to obtain the output of the target attention head to determine the target output of the large language model; the training unit 740 is configured to train the large language model based on the target output.
[0234] In one possible design, the rows and columns of the query matrix correspond to the sequence length and feature dimension of the input text respectively; wherein the first compression subunit 721 in the first interaction unit 720 is specifically configured as follows: normalizing each column of the query matrix to obtain a first weight matrix; performing bitwise multiplication processing on the query matrix and the first weight matrix to obtain a first weight matrix; and superimposing each row of the first weight matrix to obtain the global query vector.
[0235] Furthermore, in some possible design examples, the trained large language model is used for reasoning under secure multi-party computing MPC; wherein, the first compression subunit 721 is specifically configured to perform normalization processing on each column of the query matrix respectively, specifically including: for each column of the query matrix, determining the function value of the linear rectification function ReLU corresponding to each column element; for each column element, calculating the proportion of the function value of the corresponding ReLU function in the accumulated value of the function value as the corresponding normalization result.
[0236] In one possible design, the first fusion subunit 722 in the first interaction unit 720 is specifically configured as follows: for each key vector, after performing a position-wise multiplication process on it and the global query vector, the query vector at the same position corresponding to the key vector in the query matrix is superimposed to obtain an attention vector for forming the attention matrix.
[0237] In one possible design, the rows and columns of the attention matrix correspond to the sequence length and feature dimension of the input text respectively; wherein the second compression subunit 731 in the second interaction unit 730 is specifically configured as follows: normalizing each column of the attention matrix to obtain a second weight matrix; performing bitwise multiplication on the attention matrix and the second weight matrix to obtain a second weight matrix; and superimposing each row of the second weight matrix to obtain the global attention vector.
[0238] In one possible design, the second fusion subunit 732 in the second interaction unit 730 is specifically configured as follows: fusing the global attention vector and the multiple value vectors respectively to obtain an encoding matrix for the input text; and processing the encoding matrix using a linear transformation matrix to obtain the output of the target attention head.
[0239] Furthermore, in some possible design instances, the second fusion sub-unit 732 is further configured as follows: for each value vector, after performing a bitwise multiplication process on it and the global attention vector, the attention vectors at the same position corresponding to the value vector in the attention matrix are superimposed to obtain a coding vector to form the coding matrix.
[0240] In one possible design, the training device 700 also includes a pruning unit 750, which specifically includes: a mask layer adding subunit 751, configured to add a mask layer based on a pre-trained Transformer model to obtain a mask model; a training subunit 752, configured to train the mask model according to the distillation loss between the Transformer model and the mask model and the sparsity constraint related to the mask layer parameters; a pruning subunit 753, configured to prune the mask model according to the parameters of the trained mask layer to obtain a pruned model as the large language model.
[0241] Furthermore, in some possible design examples, the trained large language model is used for reasoning under secure multi-party computing (MPC) calculations; wherein the mask layer adding subunit 751 is specifically configured as follows: for the pre-trained Transformer model, the nonlinear activation function contained therein is replaced with an approximate function, and each operator in the approximate function has a corresponding MPC protocol; and the mask layer is added to the replaced Transformer model.
[0242] In some possible design instances, the attention layer in the Transformer model includes h attention heads and a linear transformation sublayer; wherein the mask layer adding subunit 751 is specifically configured as: adding a first mask layer between the h attention heads and the linear transformation sublayer, and the first mask vector in the first mask layer is used to mask the outputs of the h attention heads.
[0243] Further, in some possible examples, the pruning subunit 753 is specifically configured to: sort the vector elements in the first mask vector from small to large; and according to the sorting result, remove the attention heads corresponding to the vector elements ranked in a predetermined front range from the h attention heads of the mask model.
[0244] In some other possible examples, the pruning subunit 753 is specifically configured to: sort the vector elements in the first mask vector from small to large; and according to the sorting result, replace the attention heads whose corresponding vector elements are within a predetermined front range among the h attention heads of the mask model with the target attention heads.
[0245] In some possible design instances, the feedforward layer in the Transformer model includes a dimensionality increase sublayer, a nonlinear activation function layer, and a dimensionality reduction sublayer; wherein the mask layer adding subunit 751 is specifically configured as follows: a second mask layer is added between the nonlinear activation function layer and the dimensionality reduction sublayer, a second mask matrix in the second mask layer is used to mask the input of the dimensionality reduction sublayer, and the second mask matrix is obtained by diagonalizing the second mask vector.
[0246] Further, in some possible examples, the dimensionality reduction sublayer includes a dimensionality reduction weight matrix; wherein the pruning subunit 753 is specifically configured to: sort the vector elements in the second mask vector from small to large; and according to the sorting result, remove the matrix rows corresponding to the vector elements ranked in a predetermined front range from the dimensionality reduction weight matrix of the mask model.
[0247] In some possible design instances, the Transformer model includes a linear layer; wherein the mask layer adding subunit 751 is specifically configured as: adding a third mask layer before the linear layer, the third mask matrix in the third mask layer is used to mask the input of the linear layer, and the third mask matrix is obtained by diagonalizing the third mask vector.
[0248] Further, in some possible examples, the pruning subunit 753 is specifically configured to: sort the vector elements in the third mask vector from small to large; and according to the sorting result, remove the matrix rows corresponding to the vector elements ranked in a predetermined front range from the linear transformation matrix of the mask model.
[0249] In some possible design examples, the initial parameters of the mask layer are determined in the following manner: performing the i-th sampling on a predetermined uniform distribution to obtain the i-th sampling element; processing the i-th variable using a sigmoid function to obtain the i-th result as the i-th initial parameter; wherein the i-th variable is the combination result of the i-th sampling element and the parameter to be trained.
[0250] In some possible design examples, the sparsity constraint includes minimizing the gap between a current sparsity and a preset target sparsity, where the current sparsity is calculated based on current parameters of the mask layer.
[0251] In some possible design examples, the training unit 740 is specifically configured as follows: inputting the input text into the Transformer model for model processing to obtain a supervision signal; and performing distillation learning based on the target output and the supervision signal to obtain a trained large language model.
[0252] In some possible design instances, the training of the large language model is an i-th round iteration of the large language model, and the i-th round iteration includes replacing part of the attention heads in the i-th last attention layer of the large language model obtained after the previous round iteration with the target attention heads, and updating the last i attention layers in the large language model.
[0253] According to another embodiment, there is also provided a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute Figure 2 or Figure 5 The method described.
[0254] According to another embodiment, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, Figure 2 or Figure 5 The method described.
[0255] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.
[0256] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for training a large language model, comprising: In the target attention head of the large language model, the query transformation matrix, key transformation matrix, and value transformation matrix are used to process the output of the previous layer for the input text to obtain the query matrix, key matrix, and value matrix. Compressing the query matrix into a global query vector, and fusing the global query vector with a plurality of key vectors in the key matrix to obtain an attention matrix; Compressing the attention matrix into a global attention vector, and fusing the global attention vector with a plurality of value vectors in the value matrix to obtain the output of the target attention head, so as to determine the target output of the large language model; Based on the target output, training the large language model; Among them, the training of the large language model is an i-th round iteration of the large language model, and the i-th round iteration includes replacing some attention heads in the i-th last attention layer of the large language model obtained after the previous round iteration with the target attention head, and updating the last i attention layers in the large language model.
2. The method according to claim 1, wherein: The rows and columns of the query matrix correspond to the sequence length and feature dimension of the input text respectively; wherein the query matrix is compressed into a global query vector, including: Normalizing each column of the query matrix to obtain a first weight matrix; Performing a bitwise multiplication process on the query matrix and the first weight matrix to obtain a first weight matrix; The rows of the first weighting matrix are superimposed to obtain the global query vector.
3. The method according to claim 2, wherein: The trained large language model is used for reasoning under secure multi-party computing (MPC); wherein each column of the query matrix is normalized respectively, including: For each column of the query matrix, determine the function value of the linear rectification function ReLU corresponding to each column element; For each column element, calculate the proportion of the function value of the corresponding ReLU function in the accumulated value of the function value as the corresponding normalized result.
4. The method according to claim 1, wherein: The global query vector and multiple key vectors in the key matrix are fused respectively to obtain an attention matrix, including: For each key vector, after performing a position-wise multiplication process on it and the global query vector, the query vectors corresponding to the same position as the key vector in the query matrix are superimposed to obtain an attention vector for forming the attention matrix.
5. The method according to claim 1, wherein: The rows and columns of the attention matrix correspond to the sequence length and feature dimension of the input text respectively; wherein the attention matrix is compressed into a global attention vector, including: Normalizing each column of the attention matrix to obtain a second weight matrix; Performing a bitwise multiplication process on the attention matrix and the second weight matrix to obtain a second weight matrix; The rows of the second weighted matrix are superimposed to obtain the global attention vector.
6. The method according to claim 1, wherein: The output of the target attention head is obtained by fusing the global attention vector with multiple value vectors in the value matrix, including: The global attention vector and the multiple value vectors are respectively fused to obtain a coding matrix for the input text; The encoding matrix is processed using a linear transformation matrix to obtain the output of the target attention head.
7. The method according to claim 6, wherein: The global attention vector and a plurality of value vectors in the value matrix are respectively fused to obtain an encoding matrix for the input text, including: For each value vector, after performing a position-wise multiplication process on it and the global attention vector, the attention vectors at the same position corresponding to the value vector in the attention matrix are superimposed to obtain a coding vector to form the coding matrix.
8. The method according to claim 1, wherein: Before obtaining the query matrix, the key matrix and the value matrix, the method further includes: Add a mask layer based on the pre-trained Transformer model to obtain a mask model; training the mask model according to a distillation loss between the Transformer model and the mask model and a sparsity constraint associated with mask layer parameters; The mask model is pruned according to the parameters of the trained mask layer to obtain a pruned model as the large language model.
9. The method according to claim 8, wherein: The trained large language model is used for reasoning under secure multi-party computing (MPC). The mask layer is added based on the pre-trained Transformer model, including: For the pre-trained Transformer model, the nonlinear activation function contained therein is replaced with an approximate function, and each operator in the approximate function has a corresponding MPC protocol; The mask layer is added to the replaced Transformer model.
10. The method according to claim 8, wherein: The attention layer in the Transformer model includes h attention heads and a linear transformation sublayer; wherein the added mask layer includes: A first mask layer is added between the h attention heads and the linear transformation sublayer, and a first mask vector in the first mask layer is used to mask the outputs of the h attention heads.
11. The method according to claim 10, wherein: The mask model is pruned according to the parameters of the trained mask layer, including: Sort the vector elements in the first mask vector from small to large; According to the sorting result, the attention heads corresponding to the vector elements ranked in a predetermined front range are removed from the h attention heads of the mask model.
12. The method according to claim 10 or 11, wherein: Model pruning is performed based on the parameters of the trained mask layer, including: Sort the vector elements in the first mask vector from small to large; According to the sorting result, the attention heads whose corresponding vector elements are ranked in a predetermined front range among the h attention heads of the mask model are replaced by the target attention heads respectively.
13. The method according to claim 8, wherein: The feedforward layer in the Transformer model includes a dimension-increasing sublayer, a nonlinear activation function layer, and a dimension-reducing sublayer; wherein the adding mask layer includes: A second mask layer is added between the nonlinear activation function layer and the dimensionality reduction sublayer, and a second mask matrix in the second mask layer is used to mask the input of the dimensionality reduction sublayer, and the second mask matrix is obtained by diagonalizing the second mask vector.
14. The method according to claim 13, wherein: The dimension reduction sublayer includes a dimension reduction weight matrix; wherein the mask model is pruned according to the parameters of the trained mask layer, including: Sort each vector element in the second mask vector from small to large; According to the sorting result, the matrix rows corresponding to the vector elements ranked in a predetermined front range are removed from the dimensionality reduction weight matrix of the mask model.
15. The method according to claim 8, wherein: The Transformer model includes a linear layer; wherein the adding mask layer includes: A third mask layer is added before the linear layer. A third mask matrix in the third mask layer is used to perform mask processing on the input of the linear layer. The third mask matrix is obtained by diagonalizing the third mask vector.
16. The method according to claim 15, wherein: The linear layer includes a linear transformation matrix; wherein the mask model is pruned according to the parameters of the trained mask layer, including: Sort the vector elements in the third mask vector from small to large; According to the sorting result, the matrix rows corresponding to the vector elements arranged in a predetermined front range are removed from the linear transformation matrix of the mask model.
17. The method according to claim 8, wherein: The initial parameters of the mask layer are determined in the following way: Perform the i-th sampling on the predetermined uniform distribution to obtain the i-th sampling element; Use the sigmoid function to process the i-th variable and get the i-th result as the i-th initial parameter; The i-th variable is the combination result of the i-th sampling element and the parameter to be trained.
18. The method according to claim 8, wherein: The sparsity constraint includes minimizing the gap between the current sparsity and a preset target sparsity, and the current sparsity is calculated according to the current parameters of the mask layer.
19. The method according to claim 8, wherein: Based on the target output, training the large language model includes: Inputting the input text into the Transformer model for model processing to obtain a supervision signal; Distillation learning is performed based on the target output and the supervision signal to obtain a trained large language model.
20. A large language model training device, comprising: A mapping unit is configured to process the output of the previous layer for the input text using the query transformation matrix, the key transformation matrix, and the value transformation matrix in the target attention head of the large language model to obtain a query matrix, a key matrix, and a value matrix; A first interaction unit is configured to compress the query matrix into a global query vector, and fuse the global query vector with a plurality of key vectors in the key matrix to obtain an attention matrix; A second interaction unit is configured to compress the attention matrix into a global attention vector, and obtain the output of the target attention head by fusing the global attention vector with a plurality of value vectors in the value matrix, so as to determine the target output of the large language model; A training unit configured to train the large language model based on the target output; Among them, the training of the large language model is an i-th round iteration of the large language model, and the i-th round iteration includes replacing some attention heads in the i-th last attention layer of the large language model obtained after the previous round iteration with the target attention head, and updating the last i attention layers in the large language model.
21. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 19.
22. A computing device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 19 is implemented.
