Continuous prefix fine tuning method and device of large language model, equipment and medium
The continuous prefix fine-tuning of large language models is solved through the variational automatic encoder framework, which solves the problem of insufficient prompting capabilities when switching between multitasking, improves training efficiency and generalization capabilities, and achieves the high level of large language models in multitasking scenarios Precision switching.
Patent Information
- Application Number
- CN202410992146.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-26
AI Technical Summary
The existing large language models have limited prompt capabilities when switching between multitasks, making it difficult to efficiently adapt to different task scenarios, and the training cost is high and the generalization ability is insufficient.
Using the variational automatic encoder framework, by obtaining word sequences and fine-tuning data, the encoder and decoder links are constructed, and the activation layer vector is generated by combining pre-trained large language models, and the coordinate optimization of the loop is run two-stage coordinates to optimize the multi-layer perceptron parameters. Mix parameters with weights to achieve continuous prefix fine-tuning of large language models.
It improves the training efficiency and generalization capabilities of large language models, reduces training costs, realizes high-precision switching between multitasks, provides a more complete computing framework, and is suitable for multitask scenarios.
Smart Images

Figure CN119026688A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and medium for fine-tuning continuous prefixes of a large language model, and a computer program product. Background Art
[0002] Large Language Model (LLM) refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. Large language model can handle a variety of natural language tasks, such as text classification, question answering, dialogue, etc., and is an important path to artificial intelligence.
[0003] Currently, LLM mainly uses fine-tuning methods to train downstream tasks. The most commonly used method is prefix fine-tuning, which facilitates switching LLM between multiple tasks. The idea of prefix fine-tuning is to use the powerful capabilities of pre-trained language models (such as GPT-2 or BART) to optimize the performance of the model on specific natural language generation (NLG) tasks by adding a continuous sequence of task-specific vectors (called prefixes) before the input data. However, the prompting ability of this method is limited. Summary of the invention
[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a method, device, equipment, medium and product for fine-tuning continuous prefixes of a large language model, which can be trained to obtain an LLM suitable for multi-task scenarios and capable of switching between multiple tasks with high precision.
[0005] In order to achieve the above objectives, the technical solutions provided by the embodiments of the present disclosure are as follows:
[0006] In a first aspect, the present disclosure provides a method for fine-tuning continuous prefixes of a large language model, comprising:
[0007] Obtain a word sequence and a fine-tuning data set, and sample data from the fine-tuning data set;
[0008] Constructing an encoder chain of a variational autoencoder, the encoder corresponding to a first multivariate Gaussian distribution associated with the multilayer perceptron parameters;
[0009] Constructing a decoder chain of a variational autoencoder, the decoder corresponding to a second multivariate Gaussian distribution associated with a weight mixture parameter, the decoder chain including generating an activation layer vector by pre-training a large language model;
[0010] Construct the objective function of variational autoencoder training, and repeatedly run the two-stage coordinate optimization until the multilayer perceptron parameters and weight mixture parameters converge to obtain a fine-tuned large language model; wherein the objective function includes the sampled data.
[0011] In a second aspect, the present disclosure provides a continuous prefix fine-tuning device for a large language model, comprising:
[0012] An acquisition module is used to acquire a word sequence and a fine-tuning data set, and to sample sample data from the fine-tuning data set;
[0013] An encoder chain building module for building an encoder chain of a variational autoencoder, the encoder corresponding to a first multivariate Gaussian distribution associated with multilayer perceptron parameters;
[0014] A decoder chain building module, used to build a decoder chain of a variational autoencoder, the decoder corresponds to a second multivariate Gaussian distribution associated with a weight mixture parameter, and the decoder chain includes generating an activation layer vector by pre-training a large language model;
[0015] A training module is used to construct an objective function for variational autoencoder training and cyclically run two-stage coordinate optimization until the multilayer perceptron parameters and weight mixture parameters converge to obtain a fine-tuned large language model; wherein the objective function includes sampled data.
[0016] In a third aspect, the present disclosure provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, a method for fine-tuning continuous prefixes of a large language model as in the first aspect or any one of its optional embodiments is implemented.
[0017] In a fourth aspect, the present disclosure provides a computer-readable storage medium, comprising: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the continuous prefix fine-tuning method of a large language model as in the first aspect or any one of its optional embodiments.
[0018] In a fifth aspect, the present disclosure provides a computer program product, including: the computer program product includes a computer program, and when the computer program runs on a computer, the computer implements the continuous prefix fine-tuning method for a large language model as in the first aspect or any one of its optional embodiments.
[0019] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:
[0020] The disclosed embodiment provides a method, device, equipment, medium and product for fine-tuning continuous prefixes of a large language model, wherein the method acquires a word sequence and a fine-tuning data set, samples the fine-tuning data set to obtain sampled data, constructs an encoder link and a decoder link of a variational autoencoder, and then constructs an objective function for training the variational autoencoder, and cyclically runs the two-stage coordinate optimization until the multi-layer perceptron parameters and the weight mixing parameters converge to obtain a fine-tuned large language model. In this way, the present disclosure is based on the variational autoencoder framework to learn continuous prefix vectors under conditional coding, and optimizes and iterates the variational autoencoder and the large language model together to perform alignment training on the large language model, improve the richness of the prompting ability, reduce the training cost of the large language model, improve the training efficiency and generalization ability of the large language model, and the trained large language model is suitable for multi-task scenarios, and can switch between multiple tasks with high precision, providing a more complete computing framework for taking into account both the switching efficiency between multiple tasks and the accuracy of question and answer. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0023] Figure 1 This is a schematic diagram of the principle of prefix fine-tuning;
[0024] Figure 2 A flow chart of a method for fine-tuning continuous prefixes of a large language model provided in an embodiment of the present disclosure Figure 1 ;
[0025] Figure 3 A schematic diagram of the structure of a pre-trained large language model LLM provided in an embodiment of the present disclosure;
[0026] Figure 4A The embodiment of the present disclosure provides a schematic diagram of the input of the inner product attention module;
[0027] Figure 4B is an input schematic diagram of a multi-head attention module provided in an embodiment of the present disclosure;
[0028] Figure 5 A schematic diagram of the structure of an interface model for prefix fine-tuning provided in an embodiment of the present disclosure;
[0029] Figure 6A schematic diagram of a beam search provided by an embodiment of the present disclosure;
[0030] Figure 7 A flow chart of a method for fine-tuning continuous prefixes of a large language provided by an embodiment of the present disclosure Figure 2 ;
[0031] Figure 8 A schematic diagram of the structure of a continuous prefix fine-tuning device for a large language model provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0032] The prefix is a small, continuous, task-specific sequence of vectors that are learned and trained to represent the key information of the task and can be "seen" and "understood" by the model during the generation process.
[0033] The idea of prefix tuning is to use the powerful capabilities of pre-trained language models (such as GPT-2 or BART) to optimize the performance of the model on a specific natural language generation (NLG) task by adding a continuous sequence of task-specific vectors (called prefixes) before the input data. The core idea of this method is to enable the model to "pay attention" to the prefix, as if it is composed of actual tokens, thereby guiding the model to generate output that better meets the task requirements. Figure 1 As shown, Figure 1 The figure is a schematic diagram of the principle of prefix fine-tuning. Tasks A, B, and C are first processed in a mixed-task batch, and then the same pre-trained model is reused to make the output more efficient.
[0034] Variational autoencoder (VAE) is an information reconstruction method. The autoencoder, by encoding latent variables and generating samples from latent variables, achieves the purpose of learning information compression rules from data and reconstructing samples from compressed information. In fact, variational autoencoder is another architectural paradigm for multimodal and large models, such as Dall-E and other multimodal large models. We introduce VAE into prefix fine-tuning, focusing on the mechanism improvement of the existing prefix fine-tuning method, and also provides a prefix fine-tuning solution for the fusion of large models of two architectures.
[0035] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.
[0036] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.
[0037] The application scenarios of the embodiments of the present disclosure include but are not limited to the following:
[0038] (1) Vehicle sales market consultation scenario
[0039] Vehicle sales market consulting scenarios include intelligence consulting on the used car market, market trend analysis, and news and public opinion analysis of major brands, etc., providing users with a vehicle market intelligence assistant based on interactive inquiries. Its main service objects are automobile brands and marketing departments, vehicle dealers, and used car dealers.
[0040] By capturing and utilizing historical public opinion news in real time, fine-tuning the large model and building a knowledge graph, we can form a clear thinking chain for price analysis, thereby forming a three-dimensional intelligence analysis report that combines multiple factors such as brand, vehicle condition, region, and policy.
[0041] (2) Vehicle maintenance and vehicle condition analysis scenarios
[0042] In the vehicle maintenance and condition analysis scenarios, it is necessary to query the maintenance manual of the specific vehicle and form a systematic maintenance procedure based on the big model analysis, so as to provide a task list for vehicle maintenance; based on the historical records of the user's big data, a vehicle maintenance record table for the model is formed and stored in the vector retrieval engine to realize the vehicle condition analysis function; combined with the user's vehicle history records and maintenance manual, the vehicle data can be analyzed in real time and the report can be pushed to the user, thereby speeding up the vehicle maintenance efficiency and shortening the troubleshooting cycle.
[0043] (3) Vehicle-to-machine human-machine interaction scenarios
[0044] In the vehicle-computer human-computer interaction scenario, it is necessary to identify the user's semantic interaction intention, so as to form interaction habits with the user in terms of cabin control, power control, vehicle-computer entertainment control, etc., generate shortcuts for vehicle-computer control, and complex vehicle-computer data extraction and push functions.
[0045] (4) Insurance Broker Assistant and Insurance Risk Control Event Scenarios
[0046] In the insurance broker assistant and insurance risk control event scenarios, the big model analyzes a large number of insurance claims events to form a semantic query analysis model for insurance events, thereby providing risk warnings for the formulation of insurance contracts. The insurance broker assistant can provide insurance products based on the user's travel needs, conveniently serving the user's diverse travel scenarios.
[0047] In order to solve some or all of the technical problems existing in the related art, the embodiments of the present disclosure provide a method, device, electronic device, medium and program product for continuous prefix fine-tuning of a large language model, wherein the method combines VAE and a large language model for optimization and iteration to perform alignment training on the large language model, thereby reducing the training cost of the large language model and improving the training efficiency and generalization ability of the large language model. The trained LLM is suitable for multi-task scenarios and can switch between multiple tasks with high precision.
[0048] A method for fine-tuning a continuous prefix of a large language model provided in an embodiment of the present disclosure can be implemented by a continuous prefix fine-tuning device of a large language model or an electronic device, and the electronic device includes but is not limited to a vehicle terminal, a server, a personal computer, a laptop computer, a tablet computer, a smart phone, etc. The operating system of the electronic device may include Android, a mobile operating system (iOS) developed by Apple, an operating system (Windows) developed by Microsoft Corporation of the United States, etc., and the embodiment of the present disclosure does not limit this. The electronic device can be operated alone to implement the present disclosure, or it can be connected to a network and implement the present disclosure through interactive operations with other computer devices in the network. Among them, the network in which the electronic device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN) network, etc.
[0049] It should be noted that the protection scope of the continuous prefix fine-tuning method for a large language model in an embodiment of the present disclosure is not limited to the execution order of the steps listed in this embodiment, and all solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present disclosure are included in the protection scope of the present disclosure.
[0050] like Figure 2 As shown, Figure 2 A flow chart of a method for fine-tuning continuous prefixes of a large language model according to an embodiment of the present disclosure Figure 1 The method can be performed by a continuous prefix fine-tuning device for a large language model, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 2 As shown, the method mainly includes the following steps S200 to S204:
[0051] It should be noted that before steps S201 to S204, the method includes a pre-training loading step S200, which performs the operation of pre-loading the pre-trained large language model LLM. Step S200 will be described in the following three parts: 1) Transformer architecture paradigm of large language model; 2) Parameter Efficient Fine-tuning (PEFT) algorithm paradigm; 3) Interface model of prefix fine-tuning.
[0052] 1) Transformer architecture paradigm for large language models
[0053] ①Auto-regressive Language Model
[0054] First, let’s consider how to generate output y from input x via a large language model. Let’s write the token sequence as: Z = [x, y], where x is the previous context and y is the next context. id Indicates the token number containing x, Y id Indicates the token number containing y.
[0055] At the i-th tag position, the activation layer of the autoregressive language model has n layers, which can be expressed as They are stacked and represented as vectors The activation vector of the jth layer can be expressed as
[0056] The autoregressive language model is based on the input variable Z of the i-th token position i and the previous activation layer context h <i , output the activation layer h at this position i , satisfying: h i =LM φ (z i ,h <i ). The last layer activation vector Used to predict the next (i+1th) word token, which can be expressed as: In the formula, φ last_layer is the activation vector of the last layer The fully connected layer parameters are used to output the probability distribution of the i+1th word token through softmax.
[0057] Case 1: Z = [x, y], x is the previous text, y is the subsequent text to be generated. i ,h <i And loop to generate the sequence:
[0058] z i+1,h i ←LLM(z i ,h <i ,φ)
[0059] Case 2: Z = [x, y], x is the preceding context, y is the following context to be fitted. id There is a fitting objective function:
[0060]
[0061] ②Autoregressive language model based on Transformer architecture
[0062] The large model we loaded is an autoregressive language model based on the Transformer architecture, which includes an encoder component and a decoder component, such as Figure 3 As shown in the figure, the encoder includes inputs, input embedding layers, positional encoding layers, multiple multi-head attention layers, multiple residual connections and normalization layers, and multiple feed-forward layers; the decoder includes outputs, output embedding layers, multiple masked multi-head attention layers, multiple Add&Norm layers, multiple Multi-Head Attention layers, multiple feed-forward layers, as well as linear layers, fully connected layers, and output probabilities.
[0063] The encoder receives the input part X id x, the decoder loop outputs part Y id y, GPT-3 and later large language model architectures implement a mask offset-based decoder to avoid "peek at tokens with larger idx". Expressed as:
[0064] h←LLM_Encoder(x,φ)
[0065] y i ←LLM_Decoder(y <i ,h,φ)
[0066] 2) Parameter effective fine-tuning algorithm paradigm
[0067] The LLM-based multi-head attention mechanism module can increase the prefix encoding parameters and freeze the parameters φ of the rest of the LLM to achieve the purpose of fine-tuning for specific tasks.
[0068] ①Multi-Head Attention Module
[0069] Figure 3 The Multi-Head Attention layer shown in is composed of multiple Scaled Dot-Product Attention modules stacked together, such as Figure 4A As shown, Figure 4A The embodiment of the present disclosure provides an input schematic diagram of the inner product attention module. First, the query vector Q and the key value vector K are input into another matrix multiplication layer MatMul together with the numerical vector V through the matrix multiplication layer (MatMul), the scaling translation layer (Scale), the mask layer (Mask) and the fully connected layer (SoftMax), and the output value is used as the input of the inner product attention module. Figure 4B As shown, Figure 4B This is an input diagram of the multi-head attention module provided in an embodiment of the present disclosure. Multiple groups of query vectors Q, key-value vectors K, and numerical vectors V are input into multiple linear layers Linear and then correspondingly input into multiple inner product attention modules. The obtained multiple outputs are then input into a concatenation layer (Concat) and another linear layer Linear before being input into the next multi-head attention layer.
[0070] The output matrix of the inner product attention module is as follows (1):
[0071]
[0072] In formula (1), Represents the word token sequence vector matrix with input length N, d model is the feature dimension of the input; It is the projection matrix of the vector; for the input text Z, it is projected to the corresponding dimension d k , then there is but is a Type matrix:
[0073]
[0074] Softmax operation pair For each row of , there is the (i, j)th element:
[0075]
[0076] in, Used to control the calculation variance;
[0077] Combined with the above formula (1), the inner product is performed with the numerical vector V to obtain the output corresponding to the query vector:
[0078]
[0079] In the formula, “i,*” indicates Any row of .
[0080] That is, the Attention function receives the embedding matrix of the token As input, and output matrix.
[0081] The h attention heads of the multi-head attention module can be expressed as: MultiHead(Q,K,V)=Concat(head1,...,head h ) T W O ,in, is the linear projection output layer. Each attention head i (1≤i≤h) corresponds to a set of projection matrices The attention output is: That is, the Multi-head Attention function receives the embedding matrix of the token As input, and output matrix.
[0082] ②Prefix-tuning
[0083] The prefix fine-tuning method directly adds the prefix of the K and V matrices, that is, The matrix is prefixed with the following (assuming length is L) and has the same hidden dimension as K and V:
[0084]
[0085] at this time This setting does not change the input and output structure of each attention head. The parameter matrix, parameterized as prefix, can be encoded by a multilayer perceptron (MLP), denoted as: prefix = MLP W (Z task ), where Z taskIt is a matrix smaller than prefix. It corresponds to a prefix fine-tuning task, which can be the label information of a task. When fine-tuning is completed, only the parameter prefix needs to be maintained for the corresponding task. The above MLP can be simplified as: prefix = W up φ(W down T Z task ),if
[0086] 3) Interface model for prefix fine-tuning
[0087] The structure of the interface model for prefix fine-tuning is as follows Figure 5 shown.
[0088] ① Calculation link graph and back propagation
[0089] Consider the standard fine-tuning paradigm, x is the prefix of the input, y is the predicted suffix, and we have:
[0090] h←LLM_Encoder φ (x)
[0091] y i ←LLM_Decoder φ (y <i ,h)
[0092] Where LLM_Encoder φ with LLM_Decoder φ It should be noted that this is a simplified notation. The parameters φ of the large model are completely frozen, and x and y are input variables. φ Contains the fine-tuning parameter prefix.
[0093] Computational graph of variables (simplified variable dependency flow): LLM_Encoder φ (x)→h→LLM_Decoder φ (y <i ,h)→u i ,h→LLM_Decoder φ (y <i+1 ,h)→...
[0094] LLM_Encoder φ One of the attention modules head i :x→Z→head i (Z), head in prefix fine-tuning i (Z) has the following unlocking parameters: That is, the simplified calculation graph link is:
[0095]
[0096] Based on the deep learning automatic differentiation framework (Tensorflow and other deep learning frameworks come with compilers), based on the above calculation graph (the framework can infer the dependency structure of variables), the deep learning compiler can automatically infer the back propagation calculation formula required for gradient descent. All of the above parameters can be updated through back propagation, that is:
[0097] prefix=MLP W (Z task )
[0098]
[0099] ② Fine-tune the model interface
[0100] We load the large language model LLM and build the above The computational graph of the interface, usually in practice, is downloaded and loaded into the GPU through the transformers library provided by Huggingface, which converts the model into a form that can be operated by deep learning libraries such as Tensorflow or Pytorch.
[0101] Next, according to the above calculation diagram, find the multi-head attention component and implant the prefix encoder prefix=MLP for each attention head W (Z task ) and returns LLM. In this example, we will use the variational autoencoder VAE to implement prefix encoding instead.
[0102] Through the above content, we can build an interface link for prefix fine-tuning model, which we call preloading. The following will introduce the method of implementing continuous prefix fine-tuning of large language model based on variational autoencoder VAE:
[0103] S201, obtain a word unit token sequence Z = [x, y], x is the previous context, y is the following context; and obtain a fine-tuning data set From the fine-tuning data set Sampling E (x,y)~D .
[0104] In some embodiments, in step S201, the input text is converted into a corresponding token sequence and a corresponding word embedding vector, and the subsequent pre-trained large language model receives the word embedding vector of the corresponding token sequence as input. In S202, S203, and S204, the token sequence input form is the word vector after module conversion.
[0105] S202, Encoder of Variational Autoencoder VAE Link construction.
[0106] According to the following formula (2), the multivariate Gaussian distribution of prefix is set as:
[0107]
[0108] The prefix prefix is expressed by the generating function and noise as:
[0109]
[0110] Among them, φ VAE The model parameters of the multilayer perceptron as the generating function are simplified as the multilayer perceptron parameters in the following text; is the generating function of the mean, is the generating function of the covariance, and the noise ε is a multivariate Gaussian distribution with the identity matrix I as the covariance and mean 0;
[0111] in, Satisfies Cholesky decomposition. Cholesky decomposition is the decomposition of a symmetric positive definite matrix expressed as the product of a lower triangular matrix and its transpose: as follows:
[0112] In some embodiments, it is assumed that the multivariate Gaussian distribution covariance matrix is an n×n matrix, then:
[0113] is a multilayer perceptron with input y and output (n×(n-1)) / 2 variables, used to construct the lower triangular matrix;
[0114] is a multilayer perceptron with input y and output n variables, used to construct a diagonal matrix; optionally, the multilayer perceptron parameter φ VAE include: and It is understandable that and Abbreviated as φ VAE .
[0115] The 1 / 2 indicates that the exponent is 1 / 2, that is, the square root of each variable element in the matrix is taken.
[0116] Understandably, It is an information encoding function based on a multi-layer perceptron, that is, the input y usually has a higher dimension, while the output prefix prefix usually has a lower dimension. Therefore, the prefix generation function has the ability to summarize the question and answer type and prompts.
[0117] In the training phase, y is used as The coding goal can effectively summarize the type of problem and find the prompt pattern from the following y. It can be understood that y can also contain the above related information. In some embodiments, it can be selected , or just use x above.
[0118] In the prediction phase, the prefix encoding input is the template engine or key information prompt about y. template(y) represents the reference format, document, topic, and prompt information of the answer that constrains the answer content. In some embodiments, you can select That is, the information input into the encoder of the prediction stage is the overall description of the task dataset.
[0119] S203, Decoder of Variational Autoencoder VAE Link construction.
[0120] Generate predicted token sequence based on pre-trained large language model LLM Generate according to the following forward calculation graph
[0121] y or template(y),ε~N(0,I)→prefix
[0122] prefix,x→LLM_Encoder,head1(Z),...,head N (Z)→h
[0123] Case 1: Using the Soft_Topk_Selection operator (SoftTopK) when calculating back propagation
[0124]
[0125] Case 2: Beam Search sampling from SoftTopK when calculating forward propagation:
[0126]
[0127] And set the probability of the following y to satisfy the Gaussian weighted mixed distribution of the K cluster search results, as shown in formula (3):
[0128]
[0129] in, So is the mean and I is the multivariate Gaussian distribution of covariance; is a weighted mixing parameter that satisfies and
[0130] Since the "differentiability" of the reverse derivative of the objective function needs to be considered, we use a "Differentiable Top-k Operator with Optimal Transport" as the activation function of the outermost layer of the decoder. Beam_Search(·) represents beam search, LLM_Encoder is the encoder of LLM, LLM_Decoder is the decoder of LLM, k is the sorting index of beam search, 1≤k≤K;
[0131] Set to use beam search to generate K candidate y. Beam search is a heuristic graph search algorithm, which is mainly used to reduce the space and time occupied by the search. The workflow of beam search is as follows: Use the breadth-first strategy to establish a search tree; at each level of the tree, sort the paths according to the heuristic cost; only leave a predetermined number of paths (beam width) to continue to expand at the next level, and other paths sorted after K are cut off; the condition for the end of the algorithm is to meet the maximum token sequence length of the following expansion. Specifically, by accumulating the probability values on the search path, retain the K paths with the largest probability, and obtain K predicted subsequent contexts.
[0132] like Figure 6 As shown, Figure 6 A schematic diagram of a beam search provided by an embodiment of the present disclosure. The figure shows a beam search result when K=2.
[0133] S204, objective function of variational autoencoder training Construct and iterate a two-stage coordinate optimization until the parameters converge.
[0134] Specifically, for each token sequence, the LLM is trained according to the following objective function:
[0135]
[0136] in, Sampling from empirical data; and for each sample Variational autoencoder, solves the following function form:
[0137]
[0138] in, We can take the cross entropy, is the Shannon entropy, Corresponding decoder, Corresponding to the encoder. This formula is equivalent to maximizing the objective function
[0139] The specific optimization steps are as follows:
[0140] Because of the
[0141]
[0142] and
[0143]
[0144] The objective function is equivalent to maximizing the following formula (4):
[0145]
[0146] It includes the following two stages:
[0147] 1) Known multilayer perceptron parameters φ VAE , and the weight mixing parameter unknown;
[0148] Given the multilayer perceptron parameters φ VAE , weight mixing parameters In the unknown case, the objective function Only weight mixing parameters If optimization is required, then A-part needs to be maximized based on the above formula (4). Only forward calculation is required, that is, the "case 2" link in S203 is generated
[0149] When optimizing A-part, you can use any of the following implementation methods: Alternatively, you can use a simplex maximum iteration limit, which sets the maximum number of simplex iterations to be performed before the algorithm terminates without reaching optimality; or, use a second-order Newton method; or, use a convex optimization package such as CVOPT, where convex optimization problems typically involve minimizing a convex function, such as the log-linear function being convex under the simplex constraint. CVOPT is a Python module for parameter search and feature selection for convex optimization in machine learning.
[0150] 2) Known weight mixing parameters Multilayer Perceptron Parameters φ VAA unknown;
[0151] In optimizing φ VAE When , based on the above formula (4), it is necessary to maximize the two parts of A-part and B-part, and it is necessary to consider the "differentiability" of the reverse derivative of the objective function. The "case 1" link in S203 is used to generate
[0152] The multilayer perceptron parameters φ can be back-propagated based on the forward computation graph VAE Perform stochastic gradient optimization. The multilayer perceptron parameter φ can be optimized based on machine learning inference frameworks such as tensorflow and pytorch. VAE Perform automatic differentiation and use the random optimization algorithm to optimize until the multilayer perceptron parameter φ VAE convergence.
[0153] So far, the loop runs for φ VaE and The two-stage optimization of the block coordinates is performed until the multilayer perceptron parameter φ VAE and weight mixing parameters Convergence, continuous prefix fine-tuning training of the pre-trained large language model is completed.
[0154] The fine-tuned large language model obtained through training has model parameters including prefix generation parameters and context weight prediction parameters. The prefix generation parameters are the converged multi-layer perceptron parameters φ obtained during the training process. VAE ; The weight prediction parameter in the following text is the converged weight mixing parameter obtained during the training process
[0155] The above steps generate prefixes based on VAE, then perform continuous prefix fine-tuning, and optimize the objective function to perform alignment training on the large language model. This is to combine VAE and LLM for optimization iteration, which reduces the training cost of the large language model and improves the training efficiency and generalization ability of the large language model. The trained LLM is suitable for multi-task scenarios and can switch between multiple tasks with high precision.
[0156] After step S204, a model publishing step is also included, in which the fine-tuned large language model is published as an application service, so that devices and the like can call the fine-tuned large language model and apply it to various task scenarios.
[0157] Figure 7 A flow chart of a method for fine-tuning continuous prefixes of a large language provided by an embodiment of the present disclosure Figure 2 ,like Figure 7As shown, the method includes the following steps S701 to S704:
[0158] S701 , obtaining a word sequence and a fine-tuning data set, and sampling from the fine-tuning data set to obtain sampled data.
[0159] The word sequence includes the preceding and succeeding contexts. As mentioned above, let Z be the word sequence, Z = [x, y], where x is the preceding context and y is the succeeding context. Let the fine-tuning data set be From the fine-tuning dataset The sampling data E is obtained by sampling (x,y)~D .
[0160] In some embodiments, before step S701, the method further includes a preloading step, preloading an interface model for determining prefix fine-tuning and setting fine-tuning parameters. The detailed implementation process of the preloading step can refer to the aforementioned step S200, which will not be described in detail in this disclosure.
[0161] S702: Construct an encoder chain of a variational autoencoder.
[0162] Encoder of a Variational Autoencoder Satisfies the multilayer perceptron parameter φ VAE The associated first multivariate Gaussian distribution:
[0163]
[0164] The first multivariate Gaussian distribution is based on the mean generating function is the mean, and the covariance generating function is the multivariate Gaussian distribution with the prefix of the variance .
[0165] The prefix prefix is related to the multilayer perceptron parameter φ VAE Related functions, and satisfy the mean generating function and the covariance generating function The multivariate Gaussian distribution of .
[0166] Prefix and mean generating functions Covariance Generating Function Related to the noise ε, the prefix prefix is generated by the mean function and the covariance generating function It is expressed as:
[0167]
[0168] Where y is the context; ε is the noise, which satisfies the multivariate Gaussian distribution with the unit matrix I as the covariance and the mean being 0.
[0169] The following relationship (5) is satisfied:
[0170]
[0171] In formula (5), It is a multilayer perceptron with input y and output (n×(n-1)) / 2 variables; It is a multi-layer perceptron with input y and output n variables.
[0172] About the mean generating function and the covariance generating function The multivariate Gaussian distribution of is expressed as:
[0173] The specific implementation process of the above step S702 can refer to the above step S202, and the present disclosure will not elaborate on it here.
[0174] S703: Construct a decoder link of the variational autoencoder.
[0175] Among them, the decoder Corresponding to a second multivariate Gaussian distribution associated with the weight mixture parameters; the decoder chain includes generating an activation layer vector through a pre-trained large language model.
[0176] Specifically, the activation layer vector h is first generated by pre-training the large language model LLM according to the following forward calculation graph:
[0177] y or template(y),ε~N(0,I)→prefix
[0178] prefix,x→LLM_Encoder,head1(Z),...,head N (Z)→h
[0179] Among them, template(y) is the overall description information of y, and LLM_Encoder is the encoder of LLM.
[0180] Then it is divided into the following two situations
[0181] Case 1: When calculating the back propagation, the activation layer vector h generates the predicted context using the preset operator according to the first calculation link. The first calculation link is as follows:
[0182]
[0183] The preset operator may be the Soft_Topk_Selection operator. Considering the "differentiability" of the reverse derivative of the target function of subsequent training, the present disclosure uses the optimal transmission soft topK operator as the activation function of the outermost layer of the decoder. LLM_Decoder is the decoder of LLM.
[0184] Case 2: When calculating the forward propagation, the activation layer vector h performs a beam search according to the second calculation link to obtain a predicted context, and the predicted context satisfies the second multivariate Gaussian distribution; the second multivariate Gaussian distribution is a Gaussian weighted mixed distribution of multiple beam search results. The second calculation link is as follows:
[0185]
[0186] Wherein, Beam_Search(·) represents beam search, k is the sorting index of beam search, 1≤k≤K.
[0187] The second multivariate Gaussian distribution is shown in formula (3):
[0188]
[0189] in, So is the mean and I is the multivariate Gaussian distribution of covariance; is a weighted mixing parameter that satisfies and
[0190] The specific implementation process of the above step S703 can refer to the above step S203, and the present disclosure will not elaborate on it here.
[0191] S704, constructing the objective function of variational autoencoder training, and cyclically running the two-stage coordinate optimization until the multi-layer perceptron parameters and the weight mixing parameters converge, to obtain a fine-tuned large language model.
[0192] Among them, the objective function includes the sampling data
[0193] Specifically, for the sampled data Each sample The variational autoencoder solves the following function form:
[0194]
[0195] in, We can take the cross entropy, is the Shannon entropy, Corresponding decoder, Corresponding to the encoder. This formula is equivalent to maximizing the objective function
[0196] Objective Function As shown below:
[0197]
[0198] The specific optimization steps are as follows:
[0199] Because of the
[0200]
[0201] and
[0202]
[0203] The objective function is equivalent to maximizing the following formula (4):
[0204]
[0205] The objective function includes the first loss function and the second loss function. The first loss function is as shown in the A-part of formula (4): The second loss function is the B-part in equation (4):
[0206] The two-stage coordinate optimization is repeated until the multilayer perceptron parameters and weight mixing parameters converge. It consists of the following two stages:
[0207] 1) Known multilayer perceptron parameters φ VAE , and the weight mixing parameter unknown;
[0208] Given the multilayer perceptron parameters φ VAE , weight mixing parameters In the unknown case, the objective function Only weight mixing parameters If optimization is required, then A-part needs to be maximized based on the above formula (4). Only forward calculation is required, that is, the "case 2" link in S203 is generated
[0209] When optimizing A-part, you can use any of the following implementation methods: Alternatively, you can use a simplex maximum iteration limit, which sets the maximum number of simplex iterations to be performed before the algorithm terminates without reaching optimality; or, use a second-order Newton method; or, use a convex optimization package such as CVOPT, where convex optimization problems typically involve minimizing a convex function, such as the log-linear function being convex under the simplex constraint. CVOPT is a Python module for parameter search and feature selection for convex optimization in machine learning.
[0210] 2) Known weight mixing parameters Multilayer Perceptron Parameters φ VAE unknown;
[0211] In optimizing φ VAE When , based on the above formula (4), it is necessary to maximize the two parts of A-part and B-part, and it is necessary to consider the "differentiability" of the reverse derivative of the objective function. The "case 1" link in S203 is used to generate
[0212] The multilayer perceptron parameters φ can be back-propagated based on the forward computation graph VAE Perform stochastic gradient optimization. The multilayer perceptron parameter φ can be optimized based on machine learning inference frameworks such as tensorflow and pytorch. VAE Perform automatic differentiation and use the random optimization algorithm to optimize until the multilayer perceptron parameter φ VAE convergence.
[0213] So far, the loop runs for φ VAe and The two-stage optimization of the block coordinates is performed until the multilayer perceptron parameter φ VAE and weight mixing parameters Convergence, continuous prefix fine-tuning training of the pre-trained large language model is completed.
[0214] The fine-tuned large language model obtained through training has model parameters including prefix generation parameters and context weight prediction parameters. The prefix generation parameters are the converged multi-layer perceptron parameters φ obtained during the training process. VAE ; The weight prediction parameter in the following text is the converged weight mixing parameter obtained during the training process
[0215] After step S704, a model publishing step is also included.
[0216] In summary, the disclosed embodiments provide a continuous prefix fine-tuning method for a large language model, which generates prefixes through a variational autoencoder, and then performs continuous prefix fine-tuning to optimize the objective function until the multi-layer perceptron parameters and the weight mixing parameters converge, so as to perform alignment training on the large language model, thereby realizing the combined optimization iteration of the variational autoencoder and the large language model, reducing the training cost of the large language model, improving the training efficiency and generalization ability of the large language model, and the trained large language model is suitable for multi-task scenarios and can switch between multiple tasks with high precision.
[0217] like Figure 8 As shown, Figure 8 A schematic diagram of the structure of a continuous prefix fine-tuning device for a large language model provided by an embodiment of the present disclosure, the device comprising:
[0218] An acquisition module 801 is used to acquire a word-unit sequence and a fine-tuning data set, and to obtain sampled data by sampling from the fine-tuning data set;
[0219] An encoder chain construction module 802 for constructing an encoder chain of a variational autoencoder, the encoder corresponding to a first multivariate Gaussian distribution associated with multilayer perceptron parameters;
[0220] A decoder chain construction module 803 is used to construct a decoder chain of the variational autoencoder, the decoder corresponds to a second multivariate Gaussian distribution related to the weight mixing parameter, and the decoder chain includes generating an activation layer vector by pre-training a large language model;
[0221] The training module 804 is used to construct an objective function for variational autoencoder training and to cyclically run the two-stage coordinate optimization until the multilayer perceptron parameters and the weight mixing parameters converge to obtain a fine-tuned large language model; wherein the objective function includes the sampled data.
[0222] As an optional implementation of the embodiment of the present disclosure, the first multivariate Gaussian distribution is a multivariate Gaussian distribution with a mean generating function as a mean and a covariance generating function as a prefix of a variance;
[0223] Here, the prefix is related to the mean generating function, the covariance generating function, and the noise.
[0224] As an optional implementation of the embodiment of the present disclosure, the decoder link construction module 803 is specifically used to: when calculating back propagation, the activation layer vector uses a preset operator according to the first calculation link to generate a predicted context; when calculating forward propagation, the activation layer vector performs a beam search according to the second calculation link to obtain a predicted context, and the predicted context satisfies a second multivariate Gaussian distribution, which is a Gaussian weighted mixed distribution of multiple beam search results.
[0225] As an optional implementation of an embodiment of the present disclosure, the objective function includes a first loss function and a second loss function; the training module 804 is specifically used to: when the multi-layer perceptron parameters are known but the weight mixing parameters are unknown, optimize the first loss function until the weight mixing parameters converge; when the weight mixing parameters are known but the multi-layer perceptron parameters are unknown, optimize the first loss function and the second loss function until the multi-layer perceptron parameters converge.
[0226] As an optional implementation of the embodiment of the present disclosure, the training module 804 is specifically used to: optimize the first loss function by any of the following methods: increasing the maximum iteration limit of the simplex method of the weight mixing parameter; using the second-order Newton method; using a convex optimization software package.
[0227] As an optional implementation of the embodiment of the present disclosure, the training module 804 is specifically used to: optimize the first loss function and the second loss function by any of the following methods: based on the back propagation of the first computing link, stochastic gradient optimization of the multilayer perceptron parameters; based on the machine learning reasoning framework, automatically differentiate the multilayer perceptron parameters and optimize them using a random optimization algorithm.
[0228] The disclosed embodiment provides an electronic device, which includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, and when the computer program is executed by the processor, each process of the continuous prefix fine-tuning method for a large language model in the above method embodiment is implemented. And the same technical effect can be achieved, and to avoid repetition, it will not be repeated here.
[0229] Those skilled in the art will understand that the structure of the above-mentioned electronic device is only a partial structure related to the scheme of the present disclosure, and does not constitute a limitation on the electronic device to which the scheme of the present disclosure is applied. The specific electronic device may include more or fewer components than those mentioned above, or combine certain components, or have a different arrangement of components.
[0230] An embodiment of the present disclosure provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the continuous prefix fine-tuning method for a large language model in the above method embodiment is implemented, and the same technical effect can be achieved.
[0231] The computer readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0232] The disclosed embodiment provides a computer program product, which stores a computer program. When the computer program is executed by a processor, each process of the method for intelligent monitoring of structural safety during a multi-stage construction process of a building in the above-mentioned method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0233] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.
[0234] In several embodiments provided by the present disclosure, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a module, a program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0235] In the present disclosure, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0236] In the present disclosure, memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.
[0237] In the present disclosure, computer-readable media include permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.
[0238] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0239] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for fine-tuning continuous prefixes for large language models, It is characterized in that include: Acquire a word unit sequence and a fine-tuning data set, and sample data from the fine-tuning data set; constructing an encoder chain of a variational autoencoder corresponding to a first multivariate Gaussian distribution associated with parameters of the multilayer perceptron; Constructing a decoder chain of the variational autoencoder, the decoder corresponding to a second multivariate Gaussian distribution associated with a weight mixing parameter, the decoder chain comprising generating an activation layer vector by pre-training a large language model; Constructing an objective function for variational autoencoder training, and cyclically running two-stage coordinate optimization until the multilayer perceptron parameters and the weight mixing parameters converge, thereby obtaining a fine-tuned large language model; wherein the objective function includes the sampled data.
2. The method according to claim 1, It is characterized in that The first multivariate Gaussian distribution is a multivariate Gaussian distribution with a mean generating function as a mean and a covariance generating function as a prefix of a variance; The prefix is related to the mean generating function, the covariance generating function and the noise.
3. The method according to claim 1, It is characterized in that The constructing a decoder link of the variational autoencoder comprises: When calculating the back propagation, the activation layer vector generates a predicted context using the soft topK operator according to the first calculation link; When calculating forward propagation, the activation layer vector performs a beam search according to the second calculation link to obtain a predicted context, and the predicted context satisfies the second multivariate Gaussian distribution, which is a Gaussian weighted mixed distribution of multiple beam search results.
4. The method according to claim 1, It is characterized in that The objective function includes a first loss function and a second loss function; The objective function of the variational autoencoder training is constructed, and the two-stage coordinate optimization is cyclically executed until the multilayer perceptron parameters and the weight mixing parameters converge, including cyclically executing the following steps: When the multilayer perceptron parameters are known but the weight mixing parameters are unknown, optimizing the first loss function until the weight mixing parameters converge; When the weight mixing parameters are known but the multilayer perceptron parameters are unknown, the first loss function and the second loss function are optimized until the multilayer perceptron parameters converge.
5. The method according to claim 4, It is characterized in that When the multilayer perceptron parameters are known but the weight mixing parameters are unknown, optimizing the first loss function includes optimizing the first loss function in any of the following ways: Increased the maximum iteration limit of the simplex method for weighted mixing parameters; Use the second-order Newton method; Use the convex optimization package.
6. The method according to claim 4, It is characterized in that When the weight mixing parameters are known but the multilayer perceptron parameters are unknown, optimizing the first loss function and the second loss function comprises optimizing the first loss function and the second loss function by any of the following methods: Based on the back propagation of the first computation link, stochastic gradient optimization is performed on the multilayer perceptron parameters; Based on the machine learning reasoning framework, the multi-layer perceptron parameters are automatically differentiated and optimized using a stochastic optimization algorithm.
7. A continuous prefix fine-tuning device for a large language model, It is characterized in that include: An acquisition module, used for acquiring a word unit sequence and a fine-tuning data set, and sampling from the fine-tuning data set to obtain sampled data; An encoder chain building module for building an encoder chain of a variational autoencoder corresponding to a first multivariate Gaussian distribution associated with multilayer perceptron parameters; A decoder chain construction module, for constructing a decoder chain of the variational autoencoder, the decoder corresponding to a second multivariate Gaussian distribution associated with a weight mixing parameter, the decoder chain comprising generating an activation layer vector by a pre-trained large language model; A training module is used to construct an objective function for variational autoencoder training, and to cyclically run two-stage coordinate optimization until the multilayer perceptron parameters and the weight mixing parameters converge to obtain a fine-tuned large language model; wherein the objective function includes the sampled data.
8. An electronic device, It is characterized in that include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the continuous prefix fine-tuning method for a large language model according to any one of claims 1 to 6 is implemented.
9. A computer-readable storage medium, It is characterized in that include: A computer program is stored on a computer-readable storage medium, and when the computer program is executed by a processor, the continuous prefix fine-tuning method for a large language model as claimed in any one of claims 1 to 6 is implemented.
10. A computer program product, It is characterized in that include: The computer program product stores a computer program, and when the computer program is executed by a processor, the continuous prefix fine-tuning method for a large language model according to any one of claims 1 to 6 is implemented.
