Method and apparatus for fine-tuning a pre-trained model
By fusing the encoding layer with the prompt text template to generate a fused feature vector, the problem of privacy information leakage during the fine-tuning of pre-trained LLM models is solved, thereby improving the accuracy of logical reasoning results while protecting privacy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ANT BLOCKCHAIN TECHNOLOGY (SHANGHAI) CO LTD
- Filing Date
- 2024-02-27
- Publication Date
- 2026-07-03
AI Technical Summary
Existing pre-trained LLM models are prone to leaking privacy information during fine-tuning, and traditional methods cannot accurately identify and replace all privacy information, resulting in a high risk of information leakage.
A fine-tuning method using a pre-trained model is employed. By encoding user text data at the data holder's encoding layer and fusing it with a prompt text template, a fused feature vector is generated and sent to the model for logical reasoning. The parameters in the prompt text template are adjusted with the goal of minimizing the loss, thus completing the fine-tuning training and preventing the leakage of privacy information.
While protecting privacy, it improves the accuracy of logical reasoning results output by the LLM service model, reducing the risk of sensitive information leakage due to identification errors.
Smart Images
Figure CN118095487B_ABST
Abstract
Description
Technical Field
[0001] This specification relates to the field of computer technology, and in particular to a method and apparatus for fine-tuning a pre-trained model. Background Technology
[0002] With the development of computer technology, deep learning is being used more and more widely in various service scenarios. Pre-trained LLM models only require fine-tuning with a small amount of data before they can be applied to various service scenarios.
[0003] Currently, pre-trained LLM models are typically not open-sourced. Furthermore, even open-source pre-trained LLM models require extremely high computing power for fine-tuning. This necessitates sending one's service data to companies that own the LLM model or possess the computing power, potentially leading to the leakage of private information contained within that data.
[0004] To protect privacy, services typically identify and replace sensitive information within their data to prevent leaks. However, this method cannot accurately identify all private information and may still result in leaks.
[0005] Therefore, how to avoid the leakage of privacy information during the fine-tuning of pre-trained LLM models is an urgent problem to be solved. Summary of the Invention
[0006] This specification provides a method, apparatus, storage medium, and electronic device for fine-tuning a pre-trained model, so as to avoid the leakage of privacy information during the fine-tuning of a pre-trained LLM model.
[0007] The following technical solution is adopted in this specification:
[0008] This specification provides a method for fine-tuning a pre-trained model. The pre-trained LLM base model includes an encoding layer deployed on the data holder and a network layer deployed on the model. The method is applied to the data holder and includes:
[0009] Obtain user text data corresponding to the target application scenario;
[0010] The user text data is input into the encoding layer, whereby the encoding layer encodes the user text data to obtain a text feature vector. The text feature vector is then fused with a prompt text template corresponding to the target application scenario to obtain a fused feature vector.
[0011] The fused feature vector is sent to the model, which then inputs the fused feature vector into the network layer for logical reasoning and returns the logical reasoning result to the data holder.
[0012] The system receives the logical reasoning result returned by the model, calculates the loss between the logical reasoning result and the expected reasoning result labeled for the text feature vector, and adjusts the text character feature vectors contained in the prompt text template as parameters to be fine-tuned with the goal of minimizing the loss, so as to complete the fine-tuning training for the LLM base model and obtain the LLM service model corresponding to the target application scenario.
[0013] Optionally, the model provider is a service platform that has deployed the LLM basic model.
[0014] Optionally, the prompt text template includes several prompt text character feature vectors;
[0015] With minimizing the loss as the optimization objective, the text characters contained in the prompt text template, which serve as parameters to be fine-tuned, are adjusted to complete the fine-tuning training of the LLM base model, resulting in an LLM service model corresponding to the application scenario, including:
[0016] With minimizing the loss as the optimization objective, the feature vectors of several prompt text characters, which are parameters to be fine-tuned, contained in the prompt text template, and the positions of the several prompt text character feature vectors in the prompt text template are adjusted to complete the fine-tuning training of the LLM base model and obtain the LLM service model corresponding to the target application scenario.
[0017] Optionally, the network layer includes: a plurality of network sub-layers; the parameters to be fine-tuned further include parameter matrices corresponding one-to-one with the plurality of network sub-layers; the parameter matrix is a matrix composed of the parameters to be fine-tuned contained in the network sub-layers;
[0018] With minimizing the loss as the optimization objective, the text character feature vectors contained in the prompt text template, which serve as parameters to be fine-tuned, are adjusted to complete the fine-tuning training of the LLM base model, resulting in an LLM service model corresponding to the target application scenario, including:
[0019] With minimizing the loss as the optimization objective, the text character feature vectors contained in the prompt text template, which serve as parameters to be fine-tuned, are adjusted, and the loss is sent to the model. The model then adjusts the parameters contained in the parameter matrix corresponding to each of the network sub-layers, with minimizing the loss as the optimization objective, to complete the fine-tuning training of the LLM base model and obtain an LLM service model corresponding to the target application scenario.
[0020] Optionally, the training method used to fine-tune the LLM base model is a LoRA low-rank adaptive fine-tuning method; the parameter matrix includes: the LoRA weight matrix.
[0021] This specification provides a method for fine-tuning a pre-trained model. The pre-trained LLM base model includes an encoding layer deployed on the data holder and a network layer deployed on the model. The method is applied to the data holder and includes:
[0022] Obtain user text data corresponding to the target application scenario;
[0023] The user text data is input into the encoding layer, whereby the encoding layer encodes the user text data to obtain a text feature vector. The text feature vector is then fused with a prompt text template corresponding to the target application scenario to obtain a fused feature vector.
[0024] The fused feature vector is sent to the model, which then inputs the fused feature vector into the network layer for logical reasoning and returns the logical reasoning result to the data holder.
[0025] The system receives the logical inference result returned by the model, calculates the loss between the logical inference result and the expected inference result labeled for the text feature vector, and sends the loss to the model. The model then adjusts the parameters contained in the parameter matrix corresponding to the network layer with the optimization objective of minimizing the loss, thereby completing the fine-tuning training of the LLM base model and obtaining the LLM service model corresponding to the target application scenario.
[0026] Optionally, the network layer includes: a plurality of network sub-layers; the parameters to be fine-tuned further include parameter matrices corresponding one-to-one with the plurality of network sub-layers; the parameter matrix is a matrix composed of the parameters to be fine-tuned contained in the network sub-layers;
[0027] The loss is sent to the model, which then adjusts the parameters in the parameter matrix corresponding to the network layer with the optimization objective of minimizing the loss. This completes the fine-tuning training of the LLM base model, resulting in an LLM service model corresponding to the target application scenario, including:
[0028] The loss is sent to the model, which then adjusts the parameters in the parameter matrix corresponding to each of the network sub-layers with the optimization objective of minimizing the loss. This completes the fine-tuning training of the LLM base model, resulting in an LLM service model corresponding to the target application scenario.
[0029] Optionally, the prompt text template is an automatically generated prompt text template based on the fine-tuned LLM base model after fine-tuning training using the p-tuning method.
[0030] This specification provides a fine-tuning device for a pre-trained model. The pre-trained LLM base model includes an encoding layer deployed on the data holder and a network layer deployed on the model. The method is applied to the data holder, and the device includes:
[0031] The acquisition module is used to acquire user text data corresponding to the target application scenario.
[0032] The fusion module is used to input the user text data into the encoding layer, so that the encoding layer encodes the user text data to obtain a text feature vector, and fuses the text feature vector with the prompt text template corresponding to the target application scenario to obtain a fused feature vector;
[0033] The inference module is used to send the fused feature vector to the model, so that the model can input the fused feature vector into the network layer for logical inference and return the logical inference result to the data holder.
[0034] The adjustment module is used to receive the logical reasoning result returned by the model, calculate the loss between the logical reasoning result and the expected reasoning result labeled for the text feature vector, and adjust the text character feature vectors contained in the prompt text template as parameters to be fine-tuned with the goal of minimizing the loss, so as to complete the fine-tuning training for the LLM base model and obtain the LLM service model corresponding to the target application scenario.
[0035] This specification provides a fine-tuning device for a pre-trained model. The pre-trained LLM base model includes an encoding layer deployed on the data holder and a network layer deployed on the model. The method is applied to the data holder, and the device includes:
[0036] The acquisition module is used to acquire user text data corresponding to the target application scenario.
[0037] The fusion module is used to input the user text data into the encoding layer, so that the encoding layer encodes the user text data to obtain a text feature vector, and fuses the text feature vector with the prompt text template corresponding to the target application scenario to obtain a fused feature vector;
[0038] The inference module is used to send the fused feature vector to the model, so that the model can input the fused feature vector into the network layer for logical inference and return the logical inference result to the data holder.
[0039] The adjustment module is used to receive the logical inference result returned by the model, calculate the loss between the logical inference result and the expected inference result labeled for the text feature vector, and send the loss to the model so that the model can adjust the parameters contained in the parameter matrix corresponding to the network layer with the optimization objective of minimizing the loss, so as to complete the fine-tuning training of the LLM base model and obtain the LLM service model corresponding to the target application scenario.
[0040] This specification provides an electronic device, including a communication interface, a processor, a memory, and a bus, wherein the communication interface, the processor, and the memory are interconnected via the bus;
[0041] The memory stores machine-readable instructions, and the processor executes the fine-tuning method of the pre-trained model by calling the machine-readable instructions.
[0042] This specification provides a machine-readable storage medium storing machine-readable instructions that, when called and executed by a processor, implement the fine-tuning method of the pre-trained model described above.
[0043] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:
[0044] In this specification, the pre-trained LLM base model includes an encoding layer deployed on the data holder and a network layer deployed on the model. This method encodes user text data using the encoding layer to obtain text feature vectors, and then fuses these text feature vectors with a prompt text template corresponding to the target application scenario to obtain a fused feature vector. This prevents the model from distinguishing which part of the obtained fused feature vector is the text feature vector and which part is the prompt text template, thus avoiding the leakage of privacy information.
[0045] Then, the fused feature vector is sent to the model, which then inputs it into the network layer for logical inference. Finally, the logical inference result returned by the model is received, the loss between the logical inference result and the expected inference result labeled with the text feature vector is calculated, and the text character feature vectors contained in the prompt text template as parameters to be fine-tuned are adjusted with the goal of minimizing the loss. This completes the fine-tuning training of the LLM base model, resulting in an LLM service model corresponding to the target application scenario. Thus, the accuracy of the logical inference results output by the LLM service model is improved while protecting privacy. Attached Figure Description
[0046] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:
[0047] Figure 1 This is a flowchart illustrating a fine-tuning method for a pre-trained model, as shown in an exemplary embodiment.
[0048] Figure 2 This is a schematic diagram illustrating a prompt text template as an exemplary embodiment;
[0049] Figure 3 This is an exemplary embodiment illustrating another prompt text template;
[0050] Figure 4 This is a schematic diagram illustrating the model structure of an exemplary embodiment of the LLM basic model;
[0051] Figure 5 This is a flowchart illustrating a fine-tuning method for a pre-trained model, as shown in an exemplary embodiment.
[0052] Figure 6 This is a structural diagram of an electronic device containing a fine-tuning device for a pre-trained model, as shown in an exemplary embodiment.
[0053] Figure 7This is a structural diagram illustrating a fine-tuning device for a pre-trained model, as shown in an exemplary embodiment.
[0054] Figure 8 This is a structural diagram of a fine-tuning device for a pre-trained model, as shown in an exemplary embodiment. Detailed Implementation
[0055] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.
[0056] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.
[0057] To enable those skilled in the art to better understand the technical solutions in the embodiments of this specification, the relevant technologies involved in the embodiments of this specification will be briefly described below.
[0058] Vertical federated learning: a distributed machine learning approach for training models across multiple stakeholders while protecting data privacy.
[0059] LLM (Large Language Model) is an artificial intelligence model designed to understand and generate human language. Trained on massive amounts of text data, it can perform a wide range of tasks, including text summarization, translation, sentiment analysis, and more. LLM models are characterized by their massive scale, containing billions of parameters, and their ability to learn complex patterns in language data.
[0060] In practical applications, pre-trained LLM models are typically not open-sourced. Furthermore, even open-source pre-trained LLM models require extremely high computing power for fine-tuning. This necessitates sending one's service data to companies that own the LLM model or possess the computing power, potentially leading to the leakage of private information contained within that data.
[0061] To protect privacy, services typically identify and replace sensitive information within their data to prevent leaks. However, this method cannot accurately identify all private information and may still result in leaks.
[0062] Based on this, this specification proposes a technical solution to adjust the text character feature vectors contained in the prompt text template as parameters to be fine-tuned, thereby completing the fine-tuning training of the LLM base model and obtaining the LLM service model. This improves the accuracy of the logical reasoning results output by the LLM service model while protecting privacy information.
[0063] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.
[0064] Figure 1 This is a flowchart illustrating an exemplary embodiment of a fine-tuning method for a pre-trained model, the method specifically including the following steps:
[0065] S100: Obtain user text data corresponding to the target application scenario.
[0066] S102: Input the user text data into the encoding layer so that the encoding layer encodes the user text data to obtain a text feature vector, and fuse the text feature vector with the prompt text template corresponding to the target application scenario to obtain a fused feature vector.
[0067] S104: The fused feature vector is sent to the model, so that the model can input the fused feature vector into the network layer for logical reasoning and return the logical reasoning result to the data holder.
[0068] S106: Receive the logical reasoning result returned by the model, calculate the loss between the logical reasoning result and the expected reasoning result labeled for the text feature vector, and adjust the text character feature vectors contained in the prompt text template as parameters to be fine-tuned with minimizing the loss as the optimization objective, so as to complete the fine-tuning training for the LLM base model and obtain the LLM service model corresponding to the target application scenario.
[0069] In practical applications, the encoding layer of the pre-trained LLM base model maps user text data into text feature vectors based on the mapping relationship between user text data and text feature vectors. In fact, it is a table lookup operation and does not involve the knowledge information learned by the pre-trained LLM base model through a large amount of data.
[0070] Therefore, the model can also infer user text data based on the mapping relationship between user text data and text feature vectors, thus failing to protect privacy information.
[0071] Based on this, the data holder can fuse the text feature vector with the prompt text template corresponding to the target application scenario, making it impossible for the model to distinguish which part of the obtained fused feature vector is the text feature vector and which part is the prompt text template, thereby avoiding the leakage of privacy information.
[0072] In the embodiments described in this specification, the pre-trained LLM base model includes an encoding layer deployed on the data holder and a network layer deployed on the model. The data holder acquires user text data corresponding to the target application scenario.
[0073] Then, the user text data is input into the encoding layer, which encodes the user text data to obtain a text feature vector. The text feature vector is then fused with the prompt text template corresponding to the target application scenario to obtain a fused feature vector.
[0074] Next, the fused feature vector is sent to the model, which then inputs the fused feature vector into the network layer for logical reasoning and returns the logical reasoning result to the data holder.
[0075] Finally, the logical reasoning result returned by the receiving model is used to calculate the loss between the logical reasoning result and the expected reasoning result labeled for the text feature vector. With minimizing the loss as the optimization objective, the text character feature vectors contained in the prompt text template as parameters to be fine-tuned are adjusted to complete the fine-tuning training of the LLM base model and obtain the LLM service model corresponding to the target application scenario.
[0076] The model provider refers to the service platform that deploys the LLM basic model. This service platform can refer to a cloud service platform.
[0077] It can be seen that during the fine-tuning training of the LLM base model, the original user text data was not replaced or altered, preserving its integrity and knowledge content, thus resulting in better training performance for the LLM base model. Furthermore, it eliminates the need to identify sensitive information in the user text data, reducing the risk of sensitive information leakage due to identification errors.
[0078] In practical applications, technicians can manually create prompt text templates based on the application scenario. However, manually created prompt text templates are often influenced by the technician's subjective awareness and experience, resulting in templates that may lack objectivity and comprehensiveness, failing to cover all possible situations and needs. Furthermore, manually creating prompt text templates is inefficient.
[0079] Based on this, data holders can automatically construct prompt text templates according to application scenarios and adjust the feature vectors of each prompt text character in the prompt text template, as well as the position of the prompt text character feature vectors in the prompt text template.
[0080] In the embodiments of this specification, the prompt text template includes several prompt text character feature vectors.
[0081] The data holder can optimize by minimizing the loss by adjusting several prompt text character feature vectors (which are parameters to be fine-tuned) and their positions within the prompt text template. This will allow them to complete the fine-tuning training of the LLM base model and obtain an LLM service model that corresponds to the target application scenario.
[0082] The prompt text template can have multiple variations. For example, in a prompt text template, several prompt text character feature vectors are placed before the text feature vectors. Specifically, as shown... Figure 2 As shown.
[0083] Figure 2 This is an exemplary embodiment illustrating a prompt text template.
[0084] exist Figure 2 In the diagram, u1, u2, u3, u4, u5, and u6 are the feature vectors of the prompt text characters. M stands for mask symbol, and the LLM base model needs to predict the masked text characters. u1, u2, u3, u4, u5, and u6 are located before the text feature vectors.
[0085] For example, in a prompt text template, some prompt text character feature vectors are located before the text feature vectors, and some prompt text character feature vectors are located after the text feature vectors. Specifically, as follows... Figure 3 As shown.
[0086] Figure 3 This is an exemplary embodiment illustrating another prompt text template.
[0087] exist Figure 3 In the diagram, u1, u2, u3, u4, u5, and u6 are the feature vectors of the prompt text characters. M stands for mask symbol, and the LLM base model needs to predict the masked text characters. u1, u2, and u3 are located before the text feature vectors, while u4, u5, and u6 are located after the text feature vectors.
[0088] It should be noted that the data holder can also adjust the number of prompt text character feature vectors in the prompt text template. Furthermore, the prompt text character feature vectors in the prompt text template may not have actual corresponding text characters.
[0089] In practical applications, LLM models typically consist of billions to tens of billions of parameters. Training an LLM model requires significant time and computational resources. Furthermore, because LLM models are general-purpose, they cannot meet the personalized needs and preferences of users. In new application scenarios, traditional model training methods require retraining the LLM model to adapt to the new application, which consumes substantial time and computational resources.
[0090] Based on this, data holders can add parameter matrices to the network layers of the LLM base model. By adjusting the parameters contained in the parameter matrices, they can fine-tune the training of the LLM base model. This reduces the number of parameters required for training and improves the efficiency of model training.
[0091] In the embodiments described in this specification, the network layer includes several network sub-layers. The parameters to be fine-tuned also include parameter matrices that correspond one-to-one with each of the several network sub-layers. The parameter matrices are matrices composed of the parameters to be fine-tuned contained in the network sub-layers.
[0092] With minimizing the loss as the optimization objective, the text character feature vectors contained in the prompt text template, which serve as parameters to be fine-tuned, are adjusted. The loss is then sent to the model, which, with minimizing the loss as the optimization objective, adjusts the parameters contained in the parameter matrix corresponding to several network sub-layers to complete the fine-tuning training of the LLM base model and obtains an LLM service model corresponding to the target application scenario.
[0093] The training method used for fine-tuning the LLM base model is a LoRA-based low-rank adaptive fine-tuning method. The parameter matrix includes the LoRA weight matrix. LoRA is a parameter-efficient fine-tuning method. LoRA freezes the weight parameters of the pre-trained LMM base model and adds additional LoRA weight matrices to it. Since these newly added LoRA weight matrices have a small number of parameters, training these additional parameters not only reduces the cost of model training but also achieves similar results to full model fine-tuning.
[0094] In the embodiments of this specification, the model structure of the LLM basic model is varied. The network layer may include several network sub-layers. A network sub-layer may be composed of a single neural network structure. Alternatively, a network sub-layer may be composed of multiple neural network structures. For example, a network sub-layer may consist of a self-attention mechanism sub-layer and a multi-layer perceptron layer. Specifically... Figure 4 As shown.
[0095] Figure 4 This is a schematic diagram of the model structure of the LLM basic model as shown in an exemplary embodiment.
[0096] exist Figure 4 In this process, user text data is input into the encoding layer, which encodes the user text data to obtain a text feature vector. The text feature vector is then fused with a prompt text template corresponding to the target application scenario to obtain a fused feature vector.
[0097] The fused feature vector is sent to the model, which then inputs the fused feature vector into the normalization layer of the first sub-layer of the network for normalization. The normalized fused feature vector is then input into the first self-attention mechanism sub-layer of the first sub-layer and the LoRA weight matrix corresponding to the first self-attention mechanism sub-layer for logical reasoning to obtain the first adaptive feature vector. The fused feature vector and the first adaptive feature vector are then fused to obtain the first adaptive fused feature vector.
[0098] The first adaptive fused feature vector is input into the normalization layer in the first network sub-layer for normalization. Then, the normalized first adaptive fused feature vector is input into the first multilayer perceptron layer in the first network sub-layer, along with the corresponding LoRA weight matrix, for logical reasoning to obtain the second adaptive feature vector. The first and second adaptive feature vectors are then fused to obtain the second adaptive fused feature vector. This second adaptive fused feature vector is then input into the second network sub-layer for the same process, and so on, until the last network sub-layer completes the logical reasoning for the user text data, yielding the logical reasoning result.
[0099] In the fine-tuning training of the LLM base model, there are several methods for fusing feature vectors. For example, the fused feature vector and the first adaptive feature vector can be concatenated to obtain the first adaptive fused feature vector. Another example is the weighted summation of the fused feature vector and the first adaptive feature vector to obtain the first adaptive fused feature vector. The weights in this weighted summation are adjusted by the LLM base model.
[0100] It should be noted that this specification does not limit the structure of the network layers in the LLM basic model.
[0101] Figure 5 This is a flowchart illustrating an exemplary embodiment of a fine-tuning method for a pre-trained model, the method specifically including the following steps:
[0102] S500: Obtain user text data corresponding to the target application scenario.
[0103] S502: Input the user text data into the encoding layer so that the encoding layer encodes the user text data to obtain a text feature vector, and fuse the text feature vector with the prompt text template corresponding to the target application scenario to obtain a fused feature vector.
[0104] In practical applications, the model can deduce user text data based on the mapping relationship between user text data and text feature vectors, thus failing to protect privacy information.
[0105] Based on this, the data holder can fuse the text feature vector with the prompt text template corresponding to the target application scenario, making it impossible for the model to distinguish which part of the obtained fused feature vector is the text feature vector and which part is the prompt text template, thereby avoiding the leakage of privacy information.
[0106] In the embodiments described in this specification, the pre-trained LLM base model includes an encoding layer deployed on the data holder and a network layer deployed on the model. The data holder can obtain user text data corresponding to the target application scenario.
[0107] Then, the user text data is input into the encoding layer, which encodes the user text data to obtain a text feature vector. The text feature vector is then fused with the prompt text template corresponding to the target application scenario to obtain a fused feature vector.
[0108] It should be noted that the prompt text template mentioned here can be manually constructed in advance by technical personnel. Alternatively, it can be automatically generated based on the fine-tuned LLM base model after fine-tuning using the p-tuning method. The p-tuning training method mentioned here refers to keeping the model parameters of the pre-trained LLM base model unchanged, inputting the initialized prompt text template and user text data into the LLM base model to obtain the logical reasoning result. Then, using a distance metric (e.g., cosine similarity), the distance or similarity between the logical reasoning result and the expected reasoning result is calculated. Based on the calculated distance or similarity, an optimization algorithm (e.g., gradient descent) is used to optimize the text character feature vectors in the prompt text template, making the logical reasoning result closer to the expected reasoning result. The prompt text template is automatically updated through multiple iterations until a certain convergence condition or metric is reached to complete the training of the prompt text template.
[0109] S504: The fused feature vector is sent to the model, so that the model can input the fused feature vector into the network layer for logical reasoning and return the logical reasoning result to the data holder.
[0110] S506: Receive the logical inference result returned by the model, calculate the loss between the logical inference result and the expected inference result labeled for the text feature vector, and send the loss to the model so that the model can adjust the parameters contained in the parameter matrix corresponding to the network layer with the optimization objective of minimizing the loss, so as to complete the fine-tuning training of the LLM base model and obtain the LLM service model corresponding to the target application scenario.
[0111] In practical applications, the LLM model is a general-purpose model and cannot meet the personalized needs and preferences of users. In new application scenarios, traditional model training methods require retraining the LLM model to adapt to the new application scenarios, which consumes a lot of time and computing resources.
[0112] Based on this, data holders can add parameter matrices to the network layers of the LLM base model. By adjusting the parameters contained in the parameter matrices, they can fine-tune the training of the LLM base model. This reduces the number of parameters required for training and improves the efficiency of model training.
[0113] In the embodiments described in this specification, the data holder can send the fused feature vector to the model, so that the model can input the fused feature vector into the network layer for logical reasoning and return the logical reasoning result to the data holder.
[0114] Next, the logical reasoning result returned by the model is received, the loss between the logical reasoning result and the expected reasoning result labeled for the text feature vector is calculated, and the loss is sent to the model so that the model can adjust the parameters contained in the parameter matrix corresponding to the network layer with the optimization objective of minimizing the loss, so as to complete the fine-tuning training of the LLM base model and obtain the LLM service model corresponding to the target application scenario.
[0115] The network layer comprises several network sublayers. The parameters to be fine-tuned also include parameter matrices that correspond one-to-one with each of the network sublayers. The parameter matrices are matrices composed of the parameters to be fine-tuned contained in the network sublayers.
[0116] Furthermore, the loss is sent to the model, which then adjusts the parameters in the parameter matrix corresponding to each of the network sub-layers with the optimization objective of minimizing the loss. This completes the fine-tuning training of the LLM base model, resulting in an LLM service model that corresponds to the target application scenario.
[0117] In the embodiments of this specification, after obtaining the LLM service model, the data holder can input user text data into the encoding layer so that the encoding layer can encode the user text data to obtain a text feature vector, and then fuse the text feature vector with the prompt text template corresponding to the target application scenario to obtain a fused feature vector.
[0118] Then, the fused feature vector is sent to the model, which then inputs the fused feature vector into the network layer for logical reasoning and returns the logical reasoning result to the data holder to execute the logical reasoning result.
[0119] As can be seen from the above method, the pre-trained LLM base model includes an encoding layer deployed on the data holder and a network layer deployed on the model. This method encodes the user's text data using the encoding layer on the data holder to obtain a text feature vector. This text feature vector is then fused with a prompt text template corresponding to the target application scenario to obtain a fused feature vector. This prevents the model from distinguishing which part of the obtained fused feature vector is the text feature vector and which part is the prompt text template, thus avoiding the leakage of privacy information.
[0120] Then, the fused feature vector is sent to the model, which then inputs it into the network layer for logical inference. Finally, the logical inference result returned by the model is received, the loss between the logical inference result and the expected inference result labeled with the text feature vector is calculated, and the text character feature vectors contained in the prompt text template as parameters to be fine-tuned are adjusted with the goal of minimizing the loss. This completes the fine-tuning training of the LLM base model, resulting in an LLM service model corresponding to the target application scenario. Thus, the accuracy of the logical inference results output by the LLM service model is improved while protecting privacy.
[0121] Corresponding to the embodiments of the fine-tuning method for the pre-trained model described above, this specification also provides an embodiment of a fine-tuning device for the pre-trained model.
[0122] Please see Figure 6 , Figure 6 This is a structural diagram of an electronic device containing a fine-tuning device for a pre-trained model, as illustrated in an exemplary embodiment. At the hardware level, the device includes a processor 602, an internal bus 604, a network interface 606, memory 608, and non-volatile memory 610, and may also include other necessary hardware. One or more embodiments of this specification can be implemented in software, for example, the processor 602 reads the corresponding computer program from the non-volatile memory 610 into memory 608 and then runs it. Of course, besides software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.
[0123] Please see Figure 7 , Figure 7 This is a structural diagram illustrating a fine-tuning device for a pre-trained model, as shown in an exemplary embodiment. This fine-tuning device for the pre-trained model can be applied to, for example... Figure 6 The electronic device shown is used to implement the technical solution described in this specification.
[0124] The device may include:
[0125] The acquisition module 700 is used to acquire user text data corresponding to the target application scenario;
[0126] The fusion module 702 is used to input the user text data into the encoding layer, so that the encoding layer encodes the user text data to obtain a text feature vector, and fuses the text feature vector with the prompt text template corresponding to the target application scenario to obtain a fused feature vector;
[0127] The inference module 704 is used to send the fused feature vector to the model, so that the model can input the fused feature vector into the network layer for logical inference and return the logical inference result to the data holder.
[0128] The adjustment module 706 is used to receive the logical reasoning result returned by the model, calculate the loss between the logical reasoning result and the expected reasoning result labeled for the text feature vector, and adjust the text character feature vector contained in the prompt text template as a parameter to be fine-tuned with the goal of minimizing the loss, so as to complete the fine-tuning training for the LLM base model and obtain the LLM service model corresponding to the target application scenario.
[0129] Optionally, the model provider is a service platform that has deployed the LLM basic model.
[0130] Optionally, the prompt text template includes several prompt text character feature vectors; the adjustment module 706 is specifically used to adjust the several prompt text character feature vectors included in the prompt text template as parameters to be fine-tuned, as well as the positions of the several prompt text character feature vectors in the prompt text template, with the optimization objective of minimizing the loss, so as to complete the fine-tuning training for the LLM base model and obtain the LLM service model corresponding to the target application scenario.
[0131] Optionally, the network layer includes: several network sub-layers; the parameters to be fine-tuned also include parameter matrices corresponding one-to-one with the several network sub-layers; the parameter matrix is a matrix composed of the parameters to be fine-tuned contained in the network sub-layers; the adjustment module 706 is specifically used to adjust the text character feature vectors contained in the prompt text template as parameters to be fine-tuned with minimizing the loss as the optimization objective, and send the loss to the model, so that the model can adjust the parameters contained in the parameter matrix corresponding one-to-one with the several network sub-layers with minimizing the loss as the optimization objective, so as to complete the fine-tuning training of the LLM base model and obtain the LLM service model corresponding to the target application scenario.
[0132] Optionally, the training method used for fine-tuning the LLM base model is a LoRA low-rank adaptive fine-tuning method; the parameter matrix includes: the LoRA weight matrix.
[0133] Please see Figure 8 , Figure 8 This is a structural diagram illustrating a fine-tuning device for a pre-trained model, as shown in an exemplary embodiment. This fine-tuning device for the pre-trained model can be applied to, for example... Figure 6 The electronic device shown is used to implement the technical solution described in this specification.
[0134] The device may include:
[0135] The acquisition module 800 is used to acquire user text data corresponding to the target application scenario;
[0136] The fusion module 802 is used to input the user text data into the encoding layer, so that the encoding layer encodes the user text data to obtain a text feature vector, and fuses the text feature vector with the prompt text template corresponding to the target application scenario to obtain a fused feature vector;
[0137] The inference module 804 is used to send the fused feature vector to the model, so that the model can input the fused feature vector into the network layer for logical inference and return the logical inference result to the data holder.
[0138] The adjustment module 806 is used to receive the logical inference result returned by the model, calculate the loss between the logical inference result and the expected inference result labeled for the text feature vector, and send the loss to the model so that the model can adjust the parameters contained in the parameter matrix corresponding to the network layer with the optimization objective of minimizing the loss, so as to complete the fine-tuning training of the LLM base model and obtain the LLM service model corresponding to the target application scenario.
[0139] Optionally, the network layer includes: several network sub-layers; the parameters to be fine-tuned also include parameter matrices corresponding one-to-one with the several network sub-layers; the parameter matrix is a matrix composed of the parameters to be fine-tuned contained in the network sub-layers; the adjustment module 806 is specifically used to send the loss to the model, so that the model can adjust the parameters contained in the parameter matrix corresponding one-to-one with the several network sub-layers with minimizing the loss as the optimization objective, so as to complete the fine-tuning training for the LLM base model and obtain an LLM service model corresponding to the target application scenario.
[0140] Optionally, the prompt text template is an automatically generated prompt text template based on the fine-tuned LLM base model after fine-tuning training using the p-tuning method.
[0141] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.
[0142] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0143] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.
[0144] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0145] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0146] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0147] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.
[0148] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0149] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.
[0150] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.
[0151] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."
[0152] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.
Claims
1. A method for fine-tuning a pre-trained model, wherein a pre-trained LLM base model comprises an encoding layer deployed at a data holder and a network layer deployed at a model holder, wherein the data holder and the model holder are different parties. The method is applied to the data holder, and the method includes: Obtain user text data corresponding to the target application scenario; The user text data is input into the encoding layer, whereby the encoding layer encodes the user text data to obtain a text feature vector. The text feature vector is then fused with a prompt text template corresponding to the target application scenario to obtain a fused feature vector. The fused feature vector is sent to the model, which then inputs the fused feature vector into the network layer for logical reasoning and returns the logical reasoning result to the data holder. The system receives the logical reasoning result returned by the model, calculates the loss between the logical reasoning result and the expected reasoning result labeled for the text feature vector, and adjusts the text character feature vectors contained in the prompt text template as parameters to be fine-tuned with the goal of minimizing the loss, so as to complete the fine-tuning training for the LLM base model and obtain the LLM service model corresponding to the target application scenario.
2. The method as described in claim 1, wherein the model is a service platform on which the LLM basic model is deployed.
3. The method as described in claim 1, wherein the prompt text template includes a plurality of prompt text character feature vectors; With minimizing the loss as the optimization objective, the text characters contained in the prompt text template, which serve as parameters to be fine-tuned, are adjusted to complete the fine-tuning training of the LLM base model, resulting in an LLM service model corresponding to the application scenario, including: With minimizing the loss as the optimization objective, the feature vectors of several prompt text characters, which are parameters to be fine-tuned, contained in the prompt text template, and the positions of the several prompt text character feature vectors in the prompt text template are adjusted to complete the fine-tuning training of the LLM base model and obtain the LLM service model corresponding to the target application scenario.
4. The method of claim 1, the network layer comprising: Several network sub-layers; the parameters to be fine-tuned also include parameter matrices that correspond one-to-one with the several network sub-layers; The parameter matrix is a matrix composed of the parameters to be fine-tuned contained in the network sub-layer; With minimizing the loss as the optimization objective, the text character feature vectors contained in the prompt text template, which serve as parameters to be fine-tuned, are adjusted to complete the fine-tuning training of the LLM base model, resulting in an LLM service model corresponding to the target application scenario, including: With minimizing the loss as the optimization objective, the text character feature vectors contained in the prompt text template, which serve as parameters to be fine-tuned, are adjusted, and the loss is sent to the model. The model then adjusts the parameters contained in the parameter matrix corresponding to each of the network sub-layers, with minimizing the loss as the optimization objective, to complete the fine-tuning training of the LLM base model and obtain an LLM service model corresponding to the target application scenario.
5. The method as described in claim 4, wherein the training method used for fine-tuning the LLM base model is a LoRA-based low-rank adaptive fine-tuning method; the parameter matrix includes: LoRA weight matrix.
6. A method for fine-tuning a pre-trained model, wherein the pre-trained LLM base model includes an encoding layer deployed on the data holder and a network layer deployed on the model, wherein the data holder and the model are different entities; The method is applied to the data holder, and the method includes: Obtain user text data corresponding to the target application scenario; The user text data is input into the encoding layer, whereby the encoding layer encodes the user text data to obtain a text feature vector. The text feature vector is then fused with a prompt text template corresponding to the target application scenario to obtain a fused feature vector. The fused feature vector is sent to the model, which then inputs the fused feature vector into the network layer for logical reasoning and returns the logical reasoning result to the data holder. The system receives the logical inference result returned by the model, calculates the loss between the logical inference result and the expected inference result labeled for the text feature vector, and sends the loss to the model. The model then adjusts the parameters contained in the parameter matrix corresponding to the network layer with the optimization objective of minimizing the loss, thereby completing the fine-tuning training of the LLM base model and obtaining the LLM service model corresponding to the target application scenario.
7. The method of claim 6, wherein the network layer comprises: Several network sub-layers; the parameters to be fine-tuned also include parameter matrices that correspond one-to-one with the several network sub-layers; The parameter matrix is a matrix composed of the parameters to be fine-tuned contained in the network sub-layer; The loss is sent to the model, which then adjusts the parameters in the parameter matrix corresponding to the network layer with the optimization objective of minimizing the loss. This completes the fine-tuning training of the LLM base model, resulting in an LLM service model corresponding to the target application scenario, including: The loss is sent to the model, which then adjusts the parameters in the parameter matrix corresponding to each of the network sub-layers with the optimization objective of minimizing the loss. This completes the fine-tuning training of the LLM base model, resulting in an LLM service model corresponding to the target application scenario.
8. The method as described in claim 6, wherein the prompt text template is an automatically generated prompt text template based on the fine-tuned LLM base model after fine-tuning training using the p-tuning method.
9. A fine-tuning device for a pre-trained model, wherein the pre-trained LLM base model includes an encoding layer deployed on a data holder and a network layer deployed on a model, wherein the data holder and the model are different entities; The device is used by the data holder, and the device includes: The acquisition module is used to acquire user text data corresponding to the target application scenario. The fusion module is used to input the user text data into the encoding layer, so that the encoding layer encodes the user text data to obtain a text feature vector, and fuses the text feature vector with the prompt text template corresponding to the target application scenario to obtain a fused feature vector; The inference module is used to send the fused feature vector to the model, so that the model can input the fused feature vector into the network layer for logical inference and return the logical inference result to the data holder. The adjustment module is used to receive the logical reasoning result returned by the model, calculate the loss between the logical reasoning result and the expected reasoning result labeled for the text feature vector, and adjust the text character feature vectors contained in the prompt text template as parameters to be fine-tuned with the goal of minimizing the loss, so as to complete the fine-tuning training for the LLM base model and obtain the LLM service model corresponding to the target application scenario.
10. A fine-tuning device for a pre-trained model, wherein the pre-trained LLM base model includes an encoding layer deployed on a data holder and a network layer deployed on a model, wherein the data holder and the model are different entities; The device is used by the data holder, and the device includes: The acquisition module is used to acquire user text data corresponding to the target application scenario. The fusion module is used to input the user text data into the encoding layer, so that the encoding layer encodes the user text data to obtain a text feature vector, and fuses the text feature vector with the prompt text template corresponding to the target application scenario to obtain a fused feature vector; The inference module is used to send the fused feature vector to the model, so that the model can input the fused feature vector into the network layer for logical inference and return the logical inference result to the data holder. The adjustment module is used to receive the logical inference result returned by the model, calculate the loss between the logical inference result and the expected inference result labeled for the text feature vector, and send the loss to the model so that the model can adjust the parameters contained in the parameter matrix corresponding to the network layer with the optimization objective of minimizing the loss, so as to complete the fine-tuning training of the LLM base model and obtain the LLM service model corresponding to the target application scenario.
11. An electronic device, comprising a communication interface, a processor, a memory, and a bus, wherein the communication interface, the processor, and the memory are interconnected via the bus; The memory stores machine-readable instructions, and the processor executes the method according to any one of claims 1 to 8 by invoking the machine-readable instructions.
12. A machine-readable storage medium storing machine-readable instructions that, when invoked and executed by a processor, implement the method of any one of claims 1 to 8.