Method and apparatus for fine-tuning a large language model

Abstract

Embodiments of the present application disclose a fine-tuning method and device of a large language model, which are used to balance the effectiveness and security of the large language model. The method of the embodiments of the present application comprises: obtaining a reward model set, the reward model set comprising an effectiveness reward model and a security reward model, the effectiveness reward model being used to determine an effectiveness loss of an output of the large language model, and the security reward model being used to determine a security loss of the output of the large language model. Gradient information is determined based on the reward model set, the gradient information being used to indicate a parameter update direction of fine-tuning of the large language model, the gradient information comprising gradient information corresponding to the effectiveness loss and gradient information corresponding to the security loss, the gradient information corresponding to the effectiveness loss indicating an effectiveness preference of the fine-tuning of the large language model, and the gradient information corresponding to the security loss indicating a security preference of the fine-tuning of the large language model. Model parameters of the large language model are updated based on the gradient information.

Description

Technical Field

[0001] This application relates to the field of computers, and more particularly to a method and apparatus for fine-tuning a large language model. Background Technology

[0002] With the development of artificial intelligence technology, large language models (LLMs) have outlined a blueprint for the development of general artificial intelligence. Large language models are natural language processing models based on deep learning. Through training with massive amounts of text data, large language models can generate, understand and reason about text.

[0003] In the training of large language models, to ensure that the model's output aligns with human preferences, reinforcement learning from human feedback (RLHF) is often used for fine-tuning. This allows the model to extract subjective human preferences from the feedback, thereby optimizing the model and generating higher-quality text. However, human preference data sometimes focuses solely on the quality of the generated text, neglecting potential security risks. This can lead to unethical, biased, or misleading content in the generated text.

[0004] Therefore, to prevent large language models from outputting unsafe content, current large language model training schemes often enhance the output security of the model by adding security constraints. For example, explicit security rules and restrictions are introduced during the training process, including setting specific vocabularies to limit the model's use of words that may incite bias, hatred, or violate ethics. However, adding security constraints to large language models may impair their normal capabilities due to excessive constraints. Furthermore, security-constrained schemes cannot cover all potential security risks, resulting in an imbalance between the effectiveness and security of large language models. Summary of the Invention

[0005] This application provides a method for fine-tuning a large language model, balancing the effectiveness and security of the large language model. This application also provides a fine-tuning device, computing device, computing device cluster, computer-readable storage medium, and computer program product corresponding to this method.

[0006] In a first aspect, embodiments of this application provide a method for fine-tuning a large language model. This method can be executed by a computing device, or by a component of the computing device, such as a processor, chip, or chip system, or by a logic module or software capable of implementing all or part of the functions of the computing device. The method provided in the first aspect includes: the computing device acquiring a set of reward models, which includes an effectiveness reward model and a security reward model. The effectiveness reward model is used to determine the effectiveness loss of the large language model output, and the security reward model is used to determine the security loss of the large language model output. The computing device determines fused gradient information based on the set of reward models. The fused gradient information is used to indicate the parameter update direction of the large language model fine-tuning. The fused gradient information includes gradient information corresponding to the effectiveness loss and gradient information corresponding to the security loss. The gradient information corresponding to the effectiveness loss indicates the effectiveness preference of the large language model fine-tuning, and the gradient information corresponding to the security loss indicates the security preference of the large language model fine-tuning. The computing device updates the model parameters of the large language model based on the fused gradient information.

[0007] In this embodiment, the computing device can determine the fusion gradient information during the fine-tuning process of a large language model based on the effectiveness reward model and the security reward model, and update the model parameters of the large language model based on the fusion gradient information. Since the effectiveness reward model and the security reward model can fit human effectiveness and security preferences, the fusion gradient information determined based on the effectiveness reward model and the security reward model can indicate the update preference of the large language model fine-tuning. Compared with the prior art solution that adds security constraints to the large language model, in this embodiment, the computing device can adjust the effectiveness and security preferences of the large language model fine-tuning through the fusion gradient information, thereby balancing the effectiveness and security of the large language model.

[0008] In one possible implementation, during the process of determining the fusion gradient information based on the reward model set, the computing device calculates the effectiveness gradient and the security gradient based on the effectiveness reward model and the security reward model, respectively. The effectiveness gradient indicates the descent direction of the effectiveness loss, and the security gradient indicates the descent direction of the security loss. The computing device projects the effectiveness gradient onto the orthogonal complement space of the security gradient to obtain the effectiveness gradient projection. The effectiveness gradient projection is used to eliminate conflicting components between the effectiveness gradient and the security gradient. The computing device determines the fusion gradient information based on the effectiveness gradient projection and the security gradient.

[0009] In this embodiment, the computing device can project the validity gradient, thereby reducing the conflicting components between the validity gradient and the safety gradient, and avoiding the difference between validity and safety during the fine-tuning of large language models.

[0010] In one possible implementation, during the process of calculating the validity gradient and security gradient based on the validity reward model and the security reward model, respectively, the computing device also calculates the validity loss and security loss based on the validity reward model and the security loss, respectively. The validity loss is used to evaluate the validity of the large language model output, and the security loss is used to evaluate the security of the large language model output. The computing device then calculates the validity gradient and security gradient based on the validity loss and security loss, respectively.

[0011] In this application, the computing device can calculate the effectiveness loss and security loss based on the effectiveness reward model and the security reward model, respectively, and update the model parameters of the large language model based on the effectiveness loss and security loss, thereby improving the effectiveness and security of the large language model.

[0012] In one possible implementation, during the process of calculating the effectiveness loss and security loss based on the effectiveness reward model and the security reward model, respectively, the computing device calculates the effectiveness reward value and the security reward value based on the effectiveness reward model and the security reward value, respectively. The effectiveness reward value is used to indicate the immediate effectiveness feedback of the large language model output, and the security reward value is used to indicate the immediate security feedback of the large language model output. The computing device calculates an effectiveness advantage estimate based on the generalized advantage estimation (GAE) method, using the effectiveness reward value and the effectiveness value estimate. The effectiveness advantage estimate is used to quantify the net benefit of the current morpheme of the large language model for effectiveness, and the effectiveness value estimate is used to predict the expected effectiveness benefit of the large language model output based on the generated morphemes. Similarly, the computing device calculates a security advantage estimate based on the security reward value and the security value estimate, using the GAE method. The security advantage estimate is used to quantify the net benefit of the current morpheme of the large language model for security, and the security value estimate is used to predict the expected security benefit of the large language model output based on the generated morphemes. Finally, the computing device calculates the effectiveness loss and the security loss based on the proximal policy optimization (PPO) algorithm, respectively, using the effectiveness advantage estimate and the security advantage estimate.

[0013] In this embodiment, the computing device calculates the effectiveness advantage estimate and the security advantage estimate based on the reward value and the value estimate, and further calculates the effectiveness loss and the security loss based on the effectiveness advantage estimate and the security advantage estimate, thereby improving the feasibility of calculating the effectiveness loss and the security loss.

[0014] In one possible implementation, during the process of the computing device projecting the validity gradient to the orthogonal complement space of the safety gradient, the computing device projects the validity gradient to the orthogonal complement space of the safety gradient based on the projection intensity, where the projection intensity is used to control the elimination ratio of conflict components. The projection intensity is related to the safety margin decay coefficient. When the safety loss is less than the safety loss threshold, the larger the safety margin decay coefficient, the faster the projection intensity decays, which allows large language models to perform more aggressive validity optimization. The smaller the safety margin decay coefficient, the slower the projection intensity decays, preserving more safety constraints and performing conservative validity optimization.

[0015] In the embodiments of this application, during the process of valid gradient projection, the computing device can control the elimination ratio of conflict components between valid gradient and security gradient based on the projection intensity. This allows the direction of the valid gradient to be more strictly limited based on the projection intensity when the security loss is large, thereby improving the security of the large language model.

[0016] In one possible implementation, during the process of determining the fused gradient information based on the validity gradient projection and the security gradient, the computing device performs gradient scaling on the validity gradient projection and the security gradient respectively based on a gradient scaling algorithm. The gradient scaling algorithm is used to balance the difference in gradient magnitude between the validity gradient projection and the security gradient. The scaled validity gradient projection and the scaled security gradient are then combined to obtain the fused gradient information. The gradient scaling algorithm includes a gradient scaling mechanism based on the L(1 / 2) norm. The L(1 / 2) norm is a sparsity measure between the L1 norm and the L2 norm. Gradient scaling based on the L(1 / 2) norm can balance the difference in magnitude between the validity gradient projection and the security gradient.

[0017] In this embodiment, the computing device can perform gradient scaling on the effective gradient projection and the safety gradient respectively based on the gradient scaling algorithm, thereby balancing the difference in gradient magnitude between the effective gradient projection and the safety gradient, avoiding the optimization direction being dominated by a single gradient in the fine-tuning of large language models, and thus improving the stability of the fine-tuning of large language models.

[0018] In one possible implementation, the computing device determines weight parameters based on a user demand feature vector. These weight parameters include validity weight parameters and security weight parameters. The validity weight parameters indicate the weights corresponding to the scaled validity gradient projection, and the security weight parameters indicate the weights corresponding to the scaled security gradient. The computing device then synthesizes the scaled validity gradient projection and the scaled security gradient using the validity weight parameters and security weight parameters to obtain fused gradient information.

[0019] In this embodiment, users can customize the weight parameters of the effectiveness gradient and safety gradient for large model fine-tuning based on user-demand feature vectors, thereby improving the flexibility of large language model fine-tuning.

[0020] In one possible implementation, the computing device acquires a validity training dataset and a security training dataset. A validity reward model is trained based on the validity training dataset, which includes training data on validity human preferences. A security reward model is trained based on the security training dataset, which also includes training data on security human preferences. Specifically, the fine-tuning system for the large language model uses the Bradley-Terry model to model the validity reward model and the security reward model respectively, and constructs pairwise comparison losses for the validity reward model and the security reward model. The validity reward model and the security reward model are then updated based on their respective pairwise comparison losses, resulting in the trained validity reward model and the trained security reward model.

[0021] In this embodiment, the computing device can train an effectiveness reward model and a security reward model based on an effectiveness training dataset and a security training dataset, respectively, so that the effectiveness reward model and the security reward model can fit human effectiveness preferences and human security preferences, thereby improving the accuracy of the effectiveness reward model and the security reward model.

[0022] In one possible implementation, during the acquisition of a secure training dataset, the fine-tuning system for the large language model can collect cases of attacks on the security of the large language model through red team attacks, thereby identifying and adding red team attack prompts that are not present in the original training dataset, thus expanding the security of the training dataset.

[0023] In this embodiment, the fine-tuning system for the large language model can simulate the behavior of a malicious attacker through a red team attack, inputting designed prompt words into the large language model, thereby collecting a security-enhanced training dataset and improving the feasibility of expanding the training dataset.

[0024] Secondly, embodiments of this application provide a fine-tuning apparatus for a large language model. This apparatus includes an acquisition unit and a processing unit. The acquisition unit acquires a reward model set, which includes an effectiveness reward model and a security reward model. The effectiveness reward model is used to determine the effectiveness loss of the large language model's output, and the security reward model is used to determine the security loss of the large language model's output. The processing unit determines fused gradient information based on the reward model set. The fused gradient information indicates the parameter update direction for fine-tuning the large language model. The fused gradient information includes gradient information corresponding to the effectiveness loss and gradient information corresponding to the security loss. The gradient information corresponding to the effectiveness loss indicates the effectiveness preference of the large language model's fine-tuning, and the gradient information corresponding to the security loss indicates the security preference of the large language model's fine-tuning. The processing unit also updates the model parameters of the large language model based on the fused gradient information.

[0025] In one possible implementation, the processing unit is specifically used to calculate the effectiveness gradient and the security gradient based on the effectiveness reward model and the security reward model, respectively. The effectiveness gradient indicates the descent direction of the effectiveness loss, and the security gradient indicates the descent direction of the security loss. The effectiveness gradient is projected onto the orthogonal complement space of the security gradient to obtain the effectiveness gradient projection, which is used to eliminate conflicting components between the effectiveness gradient and the security gradient. The fused gradient information is determined based on the effectiveness gradient projection and the security gradient.

[0026] In one possible implementation, the processing unit is specifically used to calculate the validity loss and security loss based on the validity reward model and the security reward model, respectively. The validity loss is used to evaluate the validity of the large language model output, and the security loss is used to evaluate the security of the large language model output. The validity gradient and security gradient are then calculated based on the validity loss and security loss, respectively.

[0027] In one possible implementation, the processing unit is specifically used to calculate effectiveness reward values ​​and security reward values ​​based on an effectiveness reward model and a security reward model, respectively. The effectiveness reward value indicates the immediate effectiveness feedback output by the large language model, and the security reward value indicates the immediate security feedback output by the large language model. Based on the Generalized Advantage Estimation (GAE) method, an effectiveness advantage estimate is calculated from the effectiveness reward value and effectiveness value estimate. The effectiveness advantage estimate quantifies the net benefit of the current morpheme of the large language model to effectiveness, and the effectiveness value estimate is used to predict the expected effectiveness benefit of the large language model output based on the generated morphemes. Based on the GAE method, a security advantage estimate is calculated from the security reward value and security value estimate. The security advantage estimate quantifies the net benefit of the current morpheme of the large language model to security, and the security value estimate is used to predict the expected security benefit of the large language model output based on the generated morphemes. Based on the Proximal Policy Optimization (PPO) algorithm, effectiveness loss and security loss are calculated from the effectiveness advantage estimate and security advantage estimate, respectively.

[0028] In one possible implementation, the processing unit is specifically used to project the validity gradient onto the orthogonal complement space of the security gradient based on the projection intensity, whereby the projection intensity is used to control the elimination ratio of conflict components.

[0029] In one possible implementation, the processing unit is specifically used to perform gradient scaling on the validity gradient projection and the safety gradient respectively based on a gradient scaling algorithm, which is used to balance the difference in gradient magnitude between the validity gradient projection and the safety gradient. The scaled validity gradient projection and the scaled safety gradient are then combined to obtain fused gradient information.

[0030] In one possible implementation, the processing unit is further configured to determine weight parameters based on the user demand feature vector. These weight parameters include validity weight parameters and security weight parameters. The validity weight parameters indicate the weights corresponding to the scaled validity gradient projection, and the security weight parameters indicate the weights corresponding to the scaled security gradient. Specifically, the processing unit is configured to synthesize the scaled validity gradient projection and the scaled security gradient based on the validity weight parameters and the security weight parameters to obtain fused gradient information.

[0031] In one possible implementation, the processing unit is further configured to acquire an effectiveness training dataset and a security training dataset, train an effectiveness reward model based on the effectiveness training dataset (which includes training data on effectiveness human preferences), and train a security reward model based on the security training dataset (which includes training data on security human preferences).

[0032] Thirdly, embodiments of this application provide a computing device including a processor coupled to a memory. The processor stores instructions, which, when executed by the processor, cause the computing device to perform the method described in the first aspect or any possible implementation thereof.

[0033] Fourthly, embodiments of this application provide a computing device cluster, which includes one or more computing devices. Each computing device includes a processor coupled to a memory. The processor is used to store instructions, which, when executed by the processor, cause the computing device cluster to perform the method described in the first aspect or any possible implementation thereof.

[0034] Fifthly, embodiments of this application provide a computer-readable storage medium having instructions stored thereon, which, when executed, cause a computer to perform the method described in the first aspect or any possible implementation thereof.

[0035] Sixthly, embodiments of this application provide a computer program product including instructions that, when executed, cause a computer to implement the method described in the first aspect or any possible implementation thereof.

[0036] It is understood that the beneficial effects achievable by any of the large language model fine-tuning devices, computing devices, computing device clusters, computer-readable media, or computer program products provided above can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0037] Figure 1 A schematic diagram of the system architecture of a fine-tuning system for a large language model provided in this application embodiment;

[0038] Figure 2 A flowchart illustrating a method for fine-tuning a large language model provided in an embodiment of this application;

[0039] Figure 3 A flowchart illustrating another method for fine-tuning a large language model provided in this application embodiment;

[0040] Figure 4 A flowchart illustrating a training reward model provided in an embodiment of this application;

[0041] Figure 5 A schematic diagram illustrating the calculation of effectiveness gradient projection provided in an embodiment of this application;

[0042] Figure 6 A schematic diagram illustrating the determination of gradient weight parameters based on user feature demand vectors, provided in an embodiment of this application;

[0043] Figure 7 A flowchart illustrating another method for fine-tuning a large language model provided in this application embodiment;

[0044] Figure 8 A schematic diagram of a fine-tuning device for a large language model provided in an embodiment of this application;

[0045] Figure 9 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0046] Figure 10 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0047] Figure 11 This is a schematic diagram of another computing device cluster provided in an embodiment of this application. Detailed Implementation

[0048] This application provides a method and apparatus for fine-tuning a large language model, which improves the balance between the effectiveness and security of the large language model.

[0049] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0050] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0051] First, some of the terms used in the embodiments of this application are introduced to facilitate understanding of the technical solutions by those skilled in the art.

[0052] Reinforcement learning from human feedback (RLHF) is a technique that matches large language models with human preferences. It trains a reward model based on human preference data to establish human biases towards model responses, and then optimizes the language model using reinforcement learning algorithms to maximize cumulative rewards, thereby aligning the language model output with human preferences.

[0053] The policy model is the large language model to be optimized in RLHF. It takes user queries as input and outputs model responses. The ultimate goal of RLHF is to optimize the policy model to produce high-quality text that aligns with human preferences. The fine-tuning method for the large language model in this embodiment can also be called the fine-tuning method for the policy model.

[0054] In RLHF, the reference model refers to the original large language model used to initialize the strategy model. Its role is to provide a reference answer to a given user question, compare it with the model response generated by the strategy model, and limit the range of changes in the strategy model.

[0055] The reward model (RM) is a reward-guided policy optimization process in RLHF. The reward model takes {user question, model output} as input and outputs a scalar score to evaluate the quality of the responses generated by the policy model.

[0056] Proximal policy optimization (PPO) is an Actor-Critic-based reinforcement learning algorithm used to train policy models to complete complex tasks. PPO ensures that the difference between the new and old policies is controllable by constraining the magnitude of policy updates, and its core is to maintain stability during policy updates.

[0057] Direct preference optimization (DPO) is a reinforcement learning algorithm based on preference learning. It improves the alignment performance and generation quality of the model by directly optimizing the model output to be consistent with human preferences.

[0058] Group relative policy optimization (GRPO) is a reinforcement learning algorithm for aligning large language models. It optimizes the policy model by introducing group relative advantage values, thereby improving the overall performance and alignment effect of the model under various tasks and preferences.

[0059] Generalized advantage estimation (GAE) is a method for estimating advantage in reinforcement learning, designed to more efficiently compute policy gradients. By combining the advantages of Monte Carlo estimation and TD estimation, GAE provides a low-variance, low-bias advantage estimation approach, thereby accelerating the policy optimization process.

[0060] The Bradley-Terry model is a classic probabilistic model used to model preference relationships between pairs of data. It represents the relative strength of preference by assigning a positive real score to each option and calculating the relative probability of preference between the two options. For example, for options i and j, the probability that preference i exceeds preference j is:

[0061]

[0062] Where, p i and p j These are the scores for options i and j, respectively. The Bradley-Terry model is widely used in sports competitions, market research, and preference learning.

[0063] To make the technical solution of this application clearer and easier to understand, the system architecture of this application will be described below with reference to the accompanying drawings.

[0064] Please see Figure 1 , Figure 1 This application provides a schematic diagram of the system architecture for a fine-tuning system of a large language model. Figure 1 In the example shown, the large language model fine-tuning system 10 includes a large language model module 101, a model set module 102, and a multi-objective gradient fusion module 103. The multi-objective gradient fusion module 103 includes an adaptive gradient projection submodule 1031, a gradient scaling submodule 1032, and a user-defined requirement customization submodule 1033. The specific functions of each part of the large language model fine-tuning system 10 are described below.

[0065] The large language model module 101 is used to perform reinforcement learning on the large language model based on human feedback, so that the output of the large language model can conform to human preferences. Specifically, the large language model module 101 can update the model parameters of the large language model based on the fused gradient information provided by the multi-objective gradient fusion module 103, so that the large language model can be fine-tuned according to the preference direction indicated by the fused gradient information. The fused gradient information includes validity gradient and safety gradient, which correspond to validity preference and safety preference, respectively. In this embodiment, the large language model can be called a policy model, and the large language model module 101 can also be called a policy model module.

[0066] The model set module 102 manages various models required for reinforcement learning of large language models, including reference models, effectiveness reward models, safety reward models, effectiveness evaluation models, and safety evaluation models. The reference model serves as the initial model as a benchmark during the reinforcement learning process of large language models, preventing the optimized large language model from deviating excessively from the original distribution and thus generating low-quality output.

[0067] The effectiveness reward model evaluates the effectiveness of content generated by large language models, including performance in task completion, information accuracy, and logical coherence. The effectiveness reward model provides a reward value related to effectiveness; a higher reward value indicates better effectiveness of the large language model's output. The safety reward model evaluates the safety of content generated by large language models, identifying and penalizing harmful, biased, misleading, or unethical content. The safety reward model provides a reward value related to safety; a higher reward value indicates better safety of the large language model's output.

[0068] The effectiveness assessment model is used to predict the long-term benefits of content generated by the large language model on the effectiveness objective. In other words, the effectiveness estimation model can predict the future effectiveness score, i.e., the effectiveness value estimate, starting from the current state. The effectiveness value estimate generated by the effectiveness assessment model is used to calculate the effectiveness advantage estimate, which quantifies the net benefit of the current morpheme of the large language model on effectiveness. The security assessment model is used to predict the long-term benefits of content generated by the large language model on the security objective. In other words, the security estimation model can predict the future security score, i.e., the security value estimate, starting from the current state. The security value estimate generated by the security assessment model is used to calculate the security advantage estimate, which quantifies the net benefit of the current morpheme of the large language model on security.

[0069] The multi-objective gradient fusion module 103 generates fused gradient information by fusing the effectiveness gradient and safety gradient in the reinforcement learning process of the large language model. This fused gradient information includes the total gradient after fusing the effectiveness gradient and the safety gradient. The fused total gradient is used to fine-tune the large language model. The effectiveness gradient indicates the direction of fine-tuning of the large language model as the effectiveness preference direction, and the safety gradient indicates the direction of fine-tuning of the large language model as the safety preference direction.

[0070] The multi-objective gradient fusion module 103 includes an adaptive gradient projection submodule 1031, a gradient scaling submodule 1032, and a user-customized submodule 1033. The adaptive gradient projection submodule 1031 projects the validity gradient onto the orthogonal complement space of the safety gradient, reducing the conflict between the validity gradient direction and the safety gradient direction. This ensures that the parameter updates of the large language model do not violate safety constraints while improving validity.

[0071] The adaptive gradient projection submodule 1031 can also control the removal ratio of components in the validity gradient that conflict with the safety gradient based on the projection intensity. Specifically, the adaptive gradient projection submodule 1031 dynamically adjusts the projection intensity according to the safety loss and the safety loss threshold, so that when the safety constraint is violated, that is, when the safety loss is greater than or equal to the safety loss threshold, the components in the validity gradient that conflict with the safety gradient are forcibly eliminated, ensuring that the update of large language model parameters strictly guarantees safety priority.

[0072] The gradient scaling submodule 1032 is used to scale the effective gradient projection and the safety gradient separately according to the gradient scaling algorithm, thereby mitigating the difference in magnitude between the effective gradient and the safety gradient and preventing optimization skew caused by one gradient dominating the update direction of the large language model. The gradient scaling algorithm can be an L1 / 2 norm-based algorithm. The L1 / 2 norm can be defined as the square of the sum of the square roots of the absolute values ​​of each element. When optimizing the effectiveness and safety of a large language model, the effective gradient and the safety gradient may have a large difference in magnitude. Directly adding them together would lead to one gradient direction dominating the update. Gradient scaling based on the L1 / 2 norm can adjust this difference, making their contributions to the fine-tuning of the large language model more balanced.

[0073] The user-defined requirements submodule 1033 provides the weight parameters for the effectiveness gradient and the safety gradient during the reinforcement learning process of a large language model, thereby meeting the user's customized requirements for the effectiveness and safety optimization objectives. The user-defined requirements submodule 1033 can determine the weight parameters for the effectiveness gradient and the safety gradient in the fused gradient information based on the user's requirement feature vector, and control these weight parameters to ensure that the update direction of the large language model meets the user's customized requirements.

[0074] It should be noted that the large language model module 101, model set module 102 and multi-objective gradient fusion module 103 in the large language model fine-tuning system 10 can all be deployed on computing devices or computing device clusters. Therefore, in this embodiment, computing devices or computing device clusters can also be used to refer to the various modules in the large language model fine-tuning system 10.

[0075] based onFigure 1 The present application also provides a method for fine-tuning a large language model, in addition to the large language model fine-tuning system 10 shown. The following describes the method for fine-tuning a large language model provided in the embodiments of this application.

[0076] Please see Figure 2 , Figure 2 This is a flowchart illustrating a method for fine-tuning a large language model, as provided in an embodiment of this application. Figure 2 In the example shown, the method includes the following steps:

[0077] 201. Obtain the reward model set, which includes the validity reward model and the security reward model. The validity reward model is used to determine the validity loss of the output of the large language model, and the security reward model is used to determine the security loss of the output of the large language model.

[0078] Before performing reinforcement learning based on human feedback, the large language model fine-tuning system 10 first needs to acquire a set of reward models, which includes an effectiveness reward model and a safety reward model. The effectiveness reward model is used to determine the effectiveness loss of the large language model output, and the safety reward model is used to determine the safety loss of the large language model output. The large language model fine-tuning system 10 can fine-tune the large language model based on these effectiveness losses and safety losses, thereby optimizing the effectiveness and safety of the large language model.

[0079] Specifically, the fine-tuning system 10 of the large language model acquires training datasets for the effectiveness reward model and the safety reward model. These training datasets are human preference datasets based on human feedback. The training datasets include effectiveness datasets and safety datasets. The fine-tuning system 10 of the large language model trains the effectiveness reward model and the safety reward model based on the effectiveness training dataset and the safety training dataset, respectively. The effectiveness reward model can provide reward signals for optimizing the effectiveness of the content generated by the large language model, and the safety reward model can provide judgments and reward signals for the safety of the content generated by the large language model.

[0080] The training process for the effectiveness reward model and the security reward model is described below:

[0081] The fine-tuning system 10 for the large language model first acquires a valid training dataset and a safe training dataset. The valid training dataset includes training data on valid human preferences, and the safe training dataset includes training data on safe human preferences. The process of acquiring the valid training dataset and the safe training dataset by the fine-tuning system 10 for the large language model includes steps such as generating candidate outputs, collecting human preferences, and data labeling and organization. Among these, human preferences include valid human preferences and safe human preferences.

[0082] In this application embodiment, the effectiveness training dataset can also be referred to as the effectiveness human data preference dataset. The sample data format in the effective training dataset is as follows Where, x i Input the i-th prompt word. and Given two candidate outputs for the input of the i-th cue word, and Compared to It answered the prompt word input x more accurately. i .

[0083] In this application embodiment, the security training dataset can also be referred to as the security human preference dataset. The sample data format in the security training dataset is as follows: Where, x j Input the j-th prompt word. and Given two candidate outputs for the input of the j-th cue word, and Compared to For safer candidate outputs, in addition, for and Two-category safety labels, that is, when When it is safe, when When it is unsafe,

[0084] In one possible implementation, during the acquisition of the training dataset, the large language model fine-tuning system 10 can collect cases of attacks on the security of the large language model through red team attacks. This allows it to identify and add red team attack prompts that are not present in the original training dataset, thereby expanding the training dataset. Red team attacks are a proactive security testing method that simulates the behavior of malicious attackers by inputting carefully designed prompts into the large language model to detect the risk of the model generating harmful, biased, misleading, or dangerous content.

[0085] After acquiring the validity training dataset and the security training dataset, the large language model fine-tuning system 10 trains a validity reward model based on the validity training dataset and a security reward model based on the security training dataset. Specifically, the large language model fine-tuning system 10 uses the Bradley-Terry model to train the validity reward model R... h and security reward model R s Modeling is performed, and a pairwise comparison loss for the effectiveness reward model is constructed. Pairwise comparison loss with security reward model Pairwise comparison loss based on an effectiveness reward model Pairwise comparison loss with security reward model The effectiveness reward model and the security reward model are updated separately. Specifically, the pairwise comparison loss of the effectiveness reward model is used. Satisfy the following formula:

[0086]

[0087] Pairwise comparison loss of the security reward model Satisfy the following formula:

[0088]

[0089] Please see Figure 3 , Figure 3 This is a schematic diagram of another major language model fine-tuning method provided in an embodiment of this application. Figure 3 In step a of the example shown, the computing device constructs a dataset of effective human data preferences. and safety human preference dataset Among them, the effective human data preference dataset The sample data includes prompt word input from a large language model and human validity preference output, as well as a safety human preference dataset. The sample data includes prompt words from the large language model, human safety preference outputs, and safety classification labels, which include two categories: safe outputs and unsafe outputs.

[0090] exist Figure 3 In step b of the example shown, the fine-tuning system 10 of the large language model is based on a valid human data preference dataset. Training effectiveness reward model R h Based on a dataset of human preferences for security Training the safety reward model R s During the training phase of the reward model, the effectiveness of the reward model R... h The training uses pairwise comparison loss. Security Reward Model R s The training uses pairwise comparison loss that includes safety labels. Pairwise comparison loss can be used as an optimization objective for the reward model. By adjusting the model parameters of the reward model through backpropagation, the predicted reward value can be made to conform to the preferences of human annotation.

[0091] Please see Figure 4 , Figure 4 This is a flowchart illustrating a training reward model provided in an embodiment of this application. Figure 4In steps a through b of the example shown, the fine-tuning system 10 of the large language model constructs a human preference dataset, including a valid human data preference dataset, before training the reward model. and safety human preference dataset

[0092] For example, users build datasets of human preferences for security. During the process, users collect sensitive language data of various types, such as hate speech, offensive language, discrimination, and stereotypes, from various channels and platforms as prompt words. Simultaneously, a large language model is used to generate two candidate outputs for each prompt word. The safety of the candidate outputs is then binary-classified and compared. For example, candidate output A is labeled "1" to indicate it is safe, while candidate output B is labeled "-1" to indicate it is unsafe. The safety comparison label is, for example, that candidate output A is safer than candidate output B. The fine-tuning system 10 of the large language model uses the prompt words and the labeled candidate outputs as sample data to construct a human preference dataset.

[0093] exist Figure 4 In steps c to d of the example shown, during the training phase of the reward model, the fine-tuning system 10 of the large language model models models the reward model using the Bradley-Terry model. Subsequently, the fine-tuning system 10 of the large language model models models the reward model based on the effectiveness human preference dataset. and safety human preference dataset Training effectiveness reward model R h and security reward model R s In training the effectiveness reward model R h At that time, the fine-tuning system of the large language model 10 is based on the effectiveness pairwise comparison loss. Optimize reward model R h The model parameters, in training the security reward model R s At that time, the fine-tuning system 10 of the large language model uses a security pairwise comparison loss that includes binary classification security labels. Optimize reward model R s Model parameters.

[0094] 202. Determine the fusion gradient information based on the reward model set. The fusion gradient information is used to indicate the parameter update direction for fine-tuning of the large language model.

[0095] After acquiring the reward model, the large language model fine-tuning system 10 determines the fused gradient information based on the reward model set. The fused gradient information is used to indicate the parameter update direction of the large language model fine-tuning. The fused gradient information includes gradient information corresponding to the effectiveness loss and gradient information corresponding to the safety loss. The gradient information corresponding to the effectiveness loss indicates the effectiveness preference of the large language model fine-tuning, and the gradient information corresponding to the safety loss indicates the safety preference of the large language model fine-tuning.

[0096] In the process of determining the fusion gradient information based on the reward model set in the fine-tuning system of the large language model 10, the effective reward model R is used. h and security reward model R s Calculate the effectiveness gradient g respectively h and safety gradient g s Then, based on the effectiveness gradient g h and safety gradient g s Determine the fusion gradient information, where the effectiveness gradient g h The direction of descent of the effectiveness loss, and the safety gradient g. s Used to indicate the direction of descent of security loss. The effectiveness gradient g is introduced below. h and safety gradient g s The calculation process:

[0097] In one possible implementation, the fine-tuning system 10 of the large language model is based on the effectiveness reward model R. h and security reward model R s Calculate the effectiveness gradient g respectively h and safety gradient g s During the process, the fine-tuning system 10 of the large language model first calculates the effectiveness loss based on the proximal policy optimization algorithm (PPO). and security loss Among them, effectiveness loss Used to evaluate the effectiveness of large language model output, security loss Used to evaluate the security of large language model output.

[0098] The effectiveness loss was calculated using the fine-tuning system 10 of the large language model. and security loss Subsequently, based on effectiveness loss and security loss Calculate the effectiveness gradient g respectively h and safety gradient g s Among them, the effectiveness gradient g h and safety gradient g s They respectively satisfy the following calculation formulas:

[0099]

[0100] The following details the effectiveness loss in the embodiments of this application. and security loss The calculation process:

[0101] In one possible implementation, the fine-tuning system 10 of the large language model is based on the effectiveness reward model R. h and security reward model R s In calculating the effectiveness loss and security loss separately, based on the effectiveness reward model R... h and security reward model R s Calculate the validity reward value and the security reward value separately. The validity reward value is used to indicate the immediate validity feedback of the large language model output, and the security reward value is used to indicate the immediate security feedback of the large language model output.

[0102] Subsequently, the fine-tuning system 10 of the large language model calculates the efficiency advantage estimate based on the generalized advantage estimation (GAE) method for the efficiency reward value and efficiency value estimate. The effectiveness advantage estimate, also known as the effectiveness advantage value, is used to quantify the net benefit of the current morpheme of a large language model to effectiveness. The effectiveness value estimate is used to predict the expected effectiveness benefit of the output of the large language model based on the generated morphemes. The effectiveness value estimate can also be called the effectiveness estimate.

[0103] Similarly, the fine-tuning system 10 of the large language model calculates the security advantage estimate based on the generalized advantage estimation (GAE) method for the security reward value and security value estimate. The effectiveness advantage estimate, also known as the effectiveness advantage value, and the security advantage estimate are used to quantify the net security gain of the current morphemes in the large language model. The security value estimate is used to predict the expected security gain of the output of the large language model based on the generated morphemes. The security value estimate can also be called the security estimate.

[0104] The effectiveness advantage estimate was obtained by fine-tuning the large language model system 10. and security advantage estimates Subsequently, the PPO algorithm was optimized based on the proximal strategy to improve its effectiveness advantage value. and security advantages The effectiveness loss was calculated separately. and security loss Effectiveness loss and security loss They respectively satisfy the following calculation formulas:

[0105]

[0106] Where x is the prompt word of the large language model, y is the response of the large language model to prompt word x, and R h (x,y) represents the effective reward model R. h Generate an estimate of the effectiveness reward, R s (x,y) represents the security reward model R. s Generate a security reward estimate. and The advantage value is calculated from the reward value and the value estimate based on the Generalized Advantage Estimation (GAE) method.

[0107] Please continue reading. Figure 3 ,exist Figure 3 In step c of the example shown, the fine-tuning system 10 of the large language model calculates the effectiveness reward value and the security reward value based on the input prompt words and output content of the large language model, respectively, using the effectiveness reward model and the security reward model. It then calculates the effectiveness estimate value and the security estimate value using the effectiveness evaluation model and the security evaluation model. Next, it calculates the effectiveness advantage value based on the effectiveness reward value and the effectiveness estimate value, and the security advantage value based on the security reward value and the security estimate value. The fine-tuning system 10 of the large language model calculates the security loss and the effectiveness loss based on the effectiveness advantage value and the security advantage value, respectively. Based on the security loss and the effectiveness loss, the effectiveness gradient and the security gradient can be further calculated.

[0108] In one possible implementation, to avoid discrepancies between the validity gradient and the security gradient direction of the large language model during fine-tuning, the large language model fine-tuning system 10 will use the validity gradient g as a reference. h Projected onto safety gradient g s The orthogonal complement space is used to obtain the effective gradient projection g′. h , validity gradient projection g′ h This is used to eliminate conflicting components between the validity gradient and the safety gradient. After projecting the validity gradient onto the orthogonal complement space of the safety gradient, the fine-tuning system 10 of the large language model projects the validity gradient onto the validity gradient projection g′. h and safety gradient g s Determine the fusion gradient information.

[0109] Specifically, the fine-tuning system of the large language model has 10 pairs of effective gradients g. h During the projection process, the effectiveness gradient is projected onto the orthogonal complement space of the safety gradient based on the projection intensity. The projection intensity controls the elimination ratio of conflict components. In this embodiment, the effectiveness gradient projection g′... h The following calculation formula is satisfied:

[0110]

[0111] in, The intensity is the projected intensity.

[0112] In this embodiment, the large language model fine-tuning system 10 can adjust based on security losses. and safety loss threshold Calculate projection intensity Security loss threshold This can be a preset threshold. Projection intensity The following calculation formula is satisfied:

[0113]

[0114] Wherein, β is the safety margin decay coefficient, used to control safety loss. Less than the safety loss threshold The release speed of optimization for the effectiveness of large language models. The larger β is, the stronger the projection. exist The faster the time decay, the more aggressively large language models can be optimized for effectiveness once safety standards are met. The smaller the β, the stronger the projection intensity. The slower the decay, the more safety constraints can be preserved and the effectiveness optimization can be carried out conservatively.

[0115] It should be noted that when When, then the projected intensity That is, the adaptive gradient projection adopts a strict safety mode, and the effective gradient projection g′ h With safety gradient g s Orthogonality ensures that security dominates the update direction, guaranteeing that maintaining effectiveness during fine-tuning of the large language model does not compromise security improvements. When At that time, That is, adaptive gradient projection adopts a partial projection balance mode, and the projection intensity... Decreasing exponentially, gradually allowing the effectiveness gradient projection g′. h There is a controllable residual component in the direction of safety refueling, which accelerates the effectiveness optimization while ensuring the safety of the large language model.

[0116] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating the calculation of an effectiveness gradient projection, provided as an embodiment of this application. Figure 5 In step a of the example shown, during the fine-tuning of the large language model, for the response y of the large language model under the prompt word input x, the fine-tuning system 10 of the large language model is based on the effectiveness reward model R. h and security reward model R sGenerate effectiveness reward value R h (x,y) and security bonus value R s (x,y), based on the effectiveness evaluation model V h and security assessment model V s Generate effectiveness value estimates and security value estimates, and use the Generalized Advantage Estimation (GAE) method to calculate the effectiveness advantage estimate from the reward and value estimates. Compared with the security advantage estimate The fine-tuning system 10 for the large language model then uses the Proximal Policy Optimization (PPO) algorithm to calculate the effectiveness loss. With security loss

[0117] exist Figure 5 In steps b to c of the example shown, the fine-tuning system 10 of the large language model calculates the effectiveness loss. With security loss Calculate the effectiveness gradient g h and safety gradient g s Define the security loss threshold based on the maximum security risk that the system can tolerate. Calculate projection intensity And based on the effective gradient projection of the projection intensity, the effective gradient g is projected. h Projected onto safety gradient g s The orthogonal complement space is used to obtain the effective gradient projection g′. h .

[0118] In one possible implementation, during the process of determining the fused gradient information based on the validity gradient projection and the security gradient, the large language model fine-tuning system 10 determines the validity gradient projection g′. h and safety gradient g s Subsequently, gradient scaling is applied to both the validity gradient projection and the security gradient using a gradient scaling algorithm to obtain the scaled validity gradient projection. and scaled security gradient Gradient scaling algorithms are used to balance the difference in gradient magnitude between validity gradient projections and safety gradients. The fine-tuning system for large language models uses 10 pairs of scaled validity gradient projections. and scaled security gradient Perform total gradient synthesis to obtain fused gradient information.

[0119] The gradient scaling algorithm in this application includes a gradient scaling mechanism based on the L(1 / 2) norm. The L(1 / 2) norm is a sparsity measure between the L1 norm and the L2 norm. Gradient scaling based on the L(1 / 2) norm can balance the difference between the magnitude of the effective gradient projection and the magnitude of the safety gradient, avoid a single gradient dominating the optimization direction, and thus improve stability.

[0120] Specifically, the fine-tuning system 10 of the large language model uses an adaptive projection method to obtain the effectiveness gradient g′ after projection. h and safety gradient g s Then, a gradient scaling mechanism based on the L(1 / 2) norm is used to determine the scaled effective gradient projection. and scaled security gradient This achieves a balance between efficiency and safety in gradient updates. The L(1 / 2) norm is calculated using the following formula:

[0121]

[0122] For the effective gradient projection g′ h and safety gradient g s The fine-tuning system 10 for large language models uses L(1 / 2) norm for gradient scaling to obtain the scaled effective gradient projection. and scaled security gradient Scaled validity gradient projection and scaled security gradient The following calculation formula is satisfied:

[0123]

[0124] Where ∈ is a local minimum to prevent the L(1 / 2) norm of the gradient from being zero.

[0125] In one possible implementation, the fine-tuning system 10 of the large language model determines weight parameters based on user demand feature vectors, wherein the weight parameters include a validity weight parameter β. h and safety weight parameter β s , validity weight parameter β h The safety weight parameter β is used to indicate the weights corresponding to the scaled validity gradient projection. s The weights used to indicate the scaled safety gradient are used in the fine-tuning system 10 of the large language model during the synthesis of the total gradient from the scaled validity gradient projection and the scaled safety gradient. This is based on the validity weight parameter β. h and safety weight parameter β h Scaled validity gradient projection and scaled security gradient The total gradient is synthesized to obtain fused gradient information, which includes the total gradient g, and the total gradient g satisfies the following calculation formula:

[0126]

[0127] In the process of determining weight parameters based on user demand feature vectors in the large language model fine-tuning system 10, users can use a user demand feature vector p of length 1 to quantify their customized weight requirements for effectiveness and security during the fine-tuning of the large language model. The user demand feature vector p can be represented as follows:

[0128]

[0129] To train a large language model that meets user needs, the fine-tuning system 10 needs to control both efficiency and security losses. Within the neighborhood of the user demand feature vector p, in this embodiment of the application, the neighborhood p of the user demand feature vector p + and p - This can be represented as follows:

[0130]

[0131] in, It is a local minimum.

[0132] The fine-tuning system of the large language model 10 determines the domain p of the user demand feature vector p. + and p - Then, through constraints The angle between p and p is less than p + and p - The angle between them, i.e., control Within the neighborhood defined by the user demand feature vector p, the following multi-objective constrained optimization problem is obtained:

[0133]

[0134] The above multi-objective constrained optimization problem is transformed into solving the following dual problem:

[0135]

[0136] By solving the dual problem described above, the effectiveness weight parameter β can be determined. h and safety weight parameter β s Based on the effectiveness weight parameter β h and safety weight parameter β h Scaled validity gradient projection and scaled security gradient Perform fusion and determine the total gradient g after fusion.

[0137] Please see Figure 6 , Figure 6 This is a schematic diagram illustrating how to determine gradient weight parameters based on a user feature demand vector, as provided in an embodiment of this application. Figure 6 In steps a to b of the example shown, the fine-tuning system 10 of the large language model receives the user feature demand vector and, based on the user feature demand vector p = [p h ,p s Determine the domain p + and p - The fine-tuning system for large language models controls effectiveness loss and security loss during the process of determining weight parameters based on user feature vectors. Within the neighborhood of the user demand feature vector p, a multi-objective constrained optimization problem is obtained.

[0138] exist Figure 6 In steps b to c of the embodiment, the fine-tuning system 10 of the large language model can transform the multi-objective constrained optimization problem into a dual problem, and solve the dual problem to determine the weight parameter β of the effectiveness gradient. h and safety weight parameter β s The fine-tuning system for large language models determines the total gradient based on effectiveness gradient weight parameters and safety gradient weight parameters.

[0139] Please continue reading. Figure 3 , Figure 3 In step d of the example shown, after the large language model fine-tuning system 10 determines the validity gradient and the safety gradient, the large language model fine-tuning system 10 determines the fused gradient information based on steps such as adaptive gradient projection, gradient scaling based on the L(1 / 2) norm, and determining gradient weights based on user demand feature vectors.

[0140] 203. Updating the model parameters of a large language model based on fused gradient information.

[0141] After determining the fused gradient information, the fine-tuning system 10 of the large language model updates the model parameters based on the fused gradient information. Specifically, this is done by using the total gradient... As the update direction for fine-tuning the large language model, the model parameters of the large language model are updated, where the updated model parameters θ′ satisfy the following formula:

[0142]

[0143] Where η is the learning rate, θ′ is the updated model parameters, and θ is the original model parameters.

[0144] Please continue reading. Figure 3 , Figure 3 In step e of the example shown, after the large language model fine-tuning system 10 determines the fusion gradient information, the large language model fine-tuning system 10 updates the model parameters of the large language model based on the fusion gradient information, so that the large language model is fine-tuned according to the effectiveness preference and safety preference indicated by the fusion gradient information.

[0145] Please see Figure 7 , Figure 7 This is a flowchart illustrating another method for fine-tuning a large language model provided in an embodiment of this application. Figure 7 In the example shown, the fine-tuning system 10 of the large language model determines the effectiveness loss, security loss, effectiveness gradient, and security gradient based on the prompts and outputs of the large language model, through models such as the effectiveness reward model, security reward model, effectiveness evaluation model, and security evaluation model in the model set. The fine-tuning system 10 then determines the fused gradient information by performing adaptive gradient projection based on the effectiveness gradient and security gradient, gradient scaling based on the L(1 / 2) norm, and determining gradient weights based on user demand feature vectors. Finally, it updates the large language model based on the fused gradient information.

[0146] As can be seen from the above embodiments, in the embodiments of this application, the computing device can determine the fusion gradient information in the fine-tuning process of the large language model based on the effectiveness reward model and the security reward model, and update the model parameters of the large language model based on the fusion gradient information. Since the effectiveness reward model and the security reward model can fit human effectiveness preferences and security preferences, the fusion gradient information determined based on the effectiveness reward model and the security reward model can indicate the effectiveness preferences and security preferences of the large language model fine-tuning, thereby balancing the effectiveness and security of the output of the large language model.

[0147] It should be noted that the method provided in this application embodiment is not only applicable to the fine-tuning of large language models based on human feedback under security constraints, but also applicable to the fine-tuning of general multi-objective large language models under various scenarios, thereby achieving the balance and fusion of the fine-tuning gradient of large language models under multiple optimization objectives, such as in scenarios balancing the effectiveness and redundancy of large language models.

[0148] In addition, the fine-tuning method for large language models provided in this application, besides combining the human feedback reinforcement learning (RLHF) method, is also applicable to methods such as direct preference optimization (DPO) and group relative policy optimization (GRPO), thus providing a general framework for multi-objective alignment of large models. It can be combined with various existing single-objective alignment methods for large language models, and is not specifically limited.

[0149] Based on the above method embodiments, this application also provides a fine-tuning device for a large language model. The fine-tuning device for a large language model provided in this application embodiment is described in detail below.

[0150] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a fine-tuning device for a large language model provided in an embodiment of this application. Figure 8 In the example shown, the large language model fine-tuning device 800 is used to implement the various steps performed by the large language model fine-tuning system in the above embodiments. The large language model fine-tuning device 800 includes an acquisition unit 801 and a processing unit 802.

[0151] The acquisition unit 801 acquires a set of reward models, including an effectiveness reward model and a safety reward model. The effectiveness reward model determines the effectiveness loss of the large language model output, and the safety reward model determines the safety loss of the large language model output. The processing unit 802 determines fused gradient information based on the set of reward models. This fused gradient information indicates the parameter update direction for fine-tuning the large language model. The fused gradient information includes gradient information corresponding to the effectiveness loss and gradient information corresponding to the safety loss. The gradient information corresponding to the effectiveness loss indicates the effectiveness preference of the large language model's fine-tuning, and the gradient information corresponding to the safety loss indicates the safety preference of the large language model's fine-tuning. The processing unit 802 also updates the model parameters of the large language model based on the fused gradient information.

[0152] In one possible implementation, processing unit 802 is specifically used to calculate the effectiveness gradient and the security gradient based on the effectiveness reward model and the security reward model, respectively. The effectiveness gradient indicates the descent direction of the effectiveness loss, and the security gradient indicates the descent direction of the security loss. The effectiveness gradient is projected onto the orthogonal complement space of the security gradient to obtain the effectiveness gradient projection, which is used to eliminate conflicting components between the effectiveness gradient and the security gradient. Based on the effectiveness gradient projection and the security gradient, fused gradient information is determined.

[0153] In one possible implementation, processing unit 802 is specifically used to calculate validity loss and security loss based on a validity reward model and a security reward model, respectively. The validity loss is used to evaluate the validity of the large language model output, and the security loss is used to evaluate the security of the large language model output. Validity gradient and security gradient are calculated based on the validity loss and security loss, respectively.

[0154] In one possible implementation, processing unit 802 is specifically used to calculate effectiveness reward values ​​and security reward values ​​based on an effectiveness reward model and a security reward model, respectively. The effectiveness reward value indicates the immediate effectiveness feedback of the large language model output, and the security reward value indicates the immediate security feedback of the large language model output. Based on the Generalized Advantage Estimation (GAE) method, an effectiveness advantage estimate is calculated from the effectiveness reward value and effectiveness value estimate. The effectiveness advantage estimate quantifies the net benefit of the current morpheme of the large language model to effectiveness, and the effectiveness value estimate is used to predict the expected effectiveness benefit of the large language model output based on the generated morphemes. Based on the GAE method, a security advantage estimate is calculated from the security reward value and security value estimate. The security advantage estimate quantifies the net benefit of the current morpheme of the large language model to security, and the security value estimate is used to predict the expected security benefit of the large language model output based on the generated morphemes. Based on the Proximal Policy Optimization (PPO) algorithm, effectiveness loss and security loss are calculated from the effectiveness advantage estimate and security advantage estimate, respectively.

[0155] In one possible implementation, the processing unit 802 is specifically used to project the validity gradient to the orthogonal complement space of the security gradient based on the projection intensity, wherein the projection intensity is used to control the elimination ratio of conflict components.

[0156] In one possible implementation, the processing unit 802 is specifically used to perform gradient scaling on the validity gradient projection and the security gradient respectively based on a gradient scaling algorithm, wherein the gradient scaling algorithm is used to balance the difference in gradient magnitude between the validity gradient projection and the security gradient. The scaled validity gradient projection and the scaled security gradient are then combined to obtain fused gradient information.

[0157] In one possible implementation, the processing unit 802 is further configured to determine weight parameters based on the user demand feature vector. These weight parameters include validity weight parameters and security weight parameters. The validity weight parameters indicate the weights corresponding to the scaled validity gradient projection, and the security weight parameters indicate the weights corresponding to the scaled security gradient. Specifically, the processing unit 802 is configured to synthesize the scaled validity gradient projection and the scaled security gradient based on the validity weight parameters and the security weight parameters to obtain fused gradient information.

[0158] In one possible implementation, the processing unit 802 is further configured to acquire an effectiveness training dataset and a security training dataset, train an effectiveness reward model based on the effectiveness training dataset (which includes training data on effectiveness human preferences), and train a security reward model based on the security training dataset (which includes training data on security human preferences).

[0159] It is understandable that the acquisition unit 801 and processing unit 802 in the fine-tuning device 800 of the large language model can serve as functional modules. Figure 1 The various modules in the fine-tuning system 10 of the large language model are mapped to each other, thereby realizing the functions of each module in the fine-tuning system 10 of the large language model.

[0160] It should be understood that the division of units in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, all units in the device can be implemented entirely through software calls from processing elements; all units can be implemented entirely in hardware; or some units can be implemented through software calls from processing elements, and others in hardware. For example, each unit can be a separate processing element, or it can be integrated into a chip within the device. Alternatively, it can be stored as a program in memory, called and executed by a processing element of the device. Moreover, these units can be fully or partially integrated together, or implemented independently. The processing element mentioned here can also be called a processor, which can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above units can be implemented through integrated logic circuits in the processor element or through software calls from processing elements.

[0161] It is worth noting that, for the sake of simplicity, the above method embodiments are described as a series of actions. However, those skilled in the art should know that this application is not limited to the order of the described actions. Furthermore, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by this application.

[0162] Other reasonable combinations of steps that can be conceived by those skilled in the art based on the above description also fall within the scope of protection of this application. Furthermore, those skilled in the art should also be aware that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0163] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application.Figure 9 As shown, the computing device 900 includes a processor 901, a memory 902, a communication interface 903, and a bus 904. The processor 901, memory 902, and communication interface 903 are coupled via the bus (not shown in the figure). The memory 902 stores instructions. When the instructions in the memory 902 are executed, the computing device 900 executes the method performed by the fine-tuning system of the large language model in the above method embodiment.

[0164] The computing device 900 may be one or more integrated circuits configured to implement the methods described above, such as: one or more application-specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs), or a combination of at least two of these forms of integrated circuits. Furthermore, when the units in the device can be implemented in the form of a processing element scheduler, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling programs. Alternatively, these units may be integrated together and implemented as a system-on-a-chip (SOC).

[0165] Processor 901 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0166] The memory 902 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0167] The memory 902 stores executable program code, and the processor 901 executes the executable program code to implement the functions of the aforementioned units or modules, thereby realizing the fine-tuning method of the large language model described above. That is, the memory 902 stores instructions for executing the fine-tuning method of the large language model described above.

[0168] The communication interface 903 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 900 and other devices or communication networks.

[0169] In addition to the data bus, the 904 bus can also include a power bus, a control bus, and a status signal bus. The bus can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. The bus can be categorized into address bus, data bus, and control bus.

[0170] Please see Figure 10 , Figure 10 This is a schematic diagram of a computing device cluster provided in an embodiment of this application. Figure 10 As shown, the computing device cluster 1000 includes at least one computing device 900.

[0171] like Figure 10 As shown, the computing device cluster 1000 includes at least one computing device 900. The memory 902 of one or more computing devices 900 in the computing device cluster 1000 may store the same instructions for executing the fine-tuning method of the large language model described above.

[0172] In some possible implementations, the memory 902 of one or more computing devices 900 in the computing device cluster 1000 may also store partial instructions for executing the fine-tuning method of the large language model described above. In other words, a combination of one or more computing devices 900 can jointly execute the instructions for executing the fine-tuning method of the large language model described above.

[0173] It should be noted that the memories 902 in the different computing devices 900 within the computing device cluster 1000 can store different instructions, each used to execute a portion of the functions of the fine-tuning device for the aforementioned large language model. That is, the instructions stored in the memories 902 of the different computing devices 900 can implement the functions of one or more modules in the processing unit and the acquisition unit.

[0174] In some possible implementations, one or more computing devices 900 in the computing device cluster 1000 can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc.

[0175] Please see Figure 11 , Figure 11This is a schematic diagram illustrating the network connection of computer devices in a computer cluster, as provided in an embodiment of this application. Figure 11 As shown, the two computing devices 900A and 900B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device.

[0176] In one possible implementation, the memory in computing device 900A stores instructions for performing the fetch unit function. Meanwhile, the memory in computing device 900B stores instructions for performing the processing unit function.

[0177] It should be understood that Figure 11 The functions of computing device 900A shown can also be performed by multiple computing devices. Similarly, the functions of computing device 900B can also be performed by multiple computing devices.

[0178] In another embodiment of this application, a computer-readable storage medium is also provided, which stores computer-executable instructions. When the processor of the device executes the computer-executable instructions, the device executes the method executed by the fine-tuning system of the large language model in the above method embodiment.

[0179] In another embodiment of this application, a computer program product is also provided, which includes computer-executable instructions stored in a computer-readable storage medium. When the processor of the device executes the computer-executable instructions, the device performs the method executed by the fine-tuning system of the large language model in the above method embodiments.

[0180] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0181] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0182] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0183] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0184] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for fine-tuning a large language model, characterized in that, include: Obtain a set of reward models, which includes an effectiveness reward model and a security reward model. The effectiveness reward model is used to determine the effectiveness loss of the output of the large language model, and the security reward model is used to determine the security loss of the output of the large language model. Based on the reward model set, fusion gradient information is determined. The fusion gradient information is used to indicate the parameter update direction of the large language model fine-tuning. The fusion gradient information includes the gradient information corresponding to the effectiveness loss and the gradient information corresponding to the safety loss. The gradient information corresponding to the effectiveness loss indicates the effectiveness preference of the large language model fine-tuning, and the gradient information corresponding to the safety loss indicates the safety preference of the large language model fine-tuning. The model parameters of the large language model are updated based on the fused gradient information.

2. The method according to claim 1, characterized in that, The determination of the fusion gradient information based on the reward model set includes: The effectiveness gradient and the safety gradient are calculated based on the effectiveness reward model and the safety reward model, respectively. The effectiveness gradient indicates the descent direction of the effectiveness loss, and the safety gradient indicates the descent direction of the safety loss. The validity gradient is projected onto the orthogonal complement space of the security gradient to obtain the validity gradient projection, which is used to eliminate conflicting components between the validity gradient and the security gradient. The fusion gradient information is determined based on the effectiveness gradient projection and the security gradient.

3. The method according to claim 2, characterized in that, The calculation of the effectiveness gradient and the security gradient based on the effectiveness reward model and the security reward model respectively includes: The effectiveness loss and the security loss are calculated based on the effectiveness reward model and the security reward model, respectively. The effectiveness loss is used to evaluate the effectiveness of the output of the large language model, and the security loss is used to evaluate the security of the output of the large language model. The effectiveness gradient and the security gradient are calculated based on the effectiveness loss and the security loss, respectively.

4. The method according to claim 3, characterized in that, The calculation of the effectiveness loss and the security loss based on the effectiveness reward model and the security reward model respectively includes: The effectiveness reward value and the security reward value are calculated based on the effectiveness reward model and the security reward model, respectively. The effectiveness reward value is used to indicate the immediate effectiveness feedback output by the large language model, and the security reward value is used to indicate the immediate security feedback output by the large language model. The effectiveness advantage estimate is calculated based on the generalized advantage estimation (GAE) method to evaluate the effectiveness reward value and the effectiveness value estimate. The effectiveness advantage estimate is used to quantify the net benefit of the current morpheme of the large language model to effectiveness. The effectiveness value estimate is used to predict the expected effectiveness benefit of the output of the large language model based on the generated morphemes. Based on the GAE method, a security advantage estimate is calculated from the security reward value and the security value estimate. The security advantage estimate is used to quantify the net security gain of the current morpheme of the large language model. The security value estimate is used to predict the expected security gain of the output of the large language model based on the generated morphemes. The effectiveness loss and the security loss are calculated based on the Proximity Policy Optimization (PPO) algorithm, which optimizes the effectiveness advantage estimate and the security advantage estimate, respectively.

5. The method according to any one of claims 2 to 4, characterized in that, The step of projecting the validity gradient onto the orthogonal complement space of the security gradient includes: The effectiveness gradient is projected onto the orthogonal complement space of the security gradient based on the projection intensity, and the projection intensity is used to control the elimination ratio of the conflict components.

6. The method according to any one of claims 2 to 5, characterized in that, The step of determining the fusion gradient information based on the validity gradient projection and the security gradient includes: The effective gradient projection and the safety gradient are scaled using a gradient scaling algorithm, which is used to balance the difference in gradient magnitude between the effective gradient projection and the safety gradient. The scaled validity gradient projection and the scaled security gradient are combined to obtain the fused gradient information.

7. The method according to claim 6, characterized in that, The method further includes: The weight parameters are determined based on the user demand feature vector. The weight parameters include an effectiveness weight parameter and a security weight parameter. The effectiveness weight parameter is used to indicate the weight corresponding to the scaled effectiveness gradient projection, and the security weight parameter is used to indicate the weight corresponding to the scaled security gradient. The process of synthesizing the total gradient from the scaled validity gradient projection and the scaled security gradient includes: Based on the validity weight parameter and the security weight parameter, the scaled validity gradient projection and the scaled security gradient are synthesized into a total gradient to obtain the fused gradient information.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: Obtain the validity training dataset and the security training dataset; The effectiveness reward model is trained based on the effectiveness training dataset, which includes training data on effective human preferences. The security reward model is trained based on the security training dataset, which includes training data on human security preferences.

9. A fine-tuning device for a large language model, characterized in that, include: An acquisition unit is used to acquire a set of reward models, which includes an effectiveness reward model and a security reward model. The effectiveness reward model is used to determine the effectiveness loss of the output of the large language model, and the security reward model is used to determine the security loss of the output of the large language model. The processing unit is configured to determine fusion gradient information based on the reward model set. The fusion gradient information is used to indicate the parameter update direction of the large language model fine-tuning. The fusion gradient information includes gradient information corresponding to the effectiveness loss and gradient information corresponding to the safety loss. The gradient information corresponding to the effectiveness loss indicates the effectiveness preference of the large language model fine-tuning, and the gradient information corresponding to the safety loss indicates the safety preference of the large language model fine-tuning. The processing unit is also used to update the model parameters of the large language model based on the fused gradient information.

10. The apparatus according to claim 9, characterized in that, The processing unit is specifically used for: The effectiveness gradient and the safety gradient are calculated based on the effectiveness reward model and the safety reward model, respectively. The effectiveness gradient indicates the descent direction of the effectiveness loss, and the safety gradient indicates the descent direction of the safety loss. The validity gradient is projected onto the orthogonal complement space of the security gradient to obtain the validity gradient projection, which is used to eliminate conflicting components between the validity gradient and the security gradient. The fusion gradient information is determined based on the effectiveness gradient projection and the security gradient.

11. The apparatus according to claim 10, characterized in that, The processing unit is specifically used for: The effectiveness loss and the security loss are calculated based on the effectiveness reward model and the security reward model, respectively. The effectiveness loss is used to evaluate the effectiveness of the output of the large language model, and the security loss is used to evaluate the security of the output of the large language model. The effectiveness gradient and the security gradient are calculated based on the effectiveness loss and the security loss, respectively.

12. The apparatus according to claim 11, characterized in that, The processing unit is specifically used for: The effectiveness reward value and the security reward value are calculated based on the effectiveness reward model and the security reward model, respectively. The effectiveness reward value is used to indicate the immediate effectiveness feedback output by the large language model, and the security reward value is used to indicate the immediate security feedback output by the large language model. The effectiveness advantage estimate is calculated based on the generalized advantage estimation (GAE) method to evaluate the effectiveness reward value and the effectiveness value estimate. The effectiveness advantage estimate is used to quantify the net benefit of the current morpheme of the large language model to effectiveness. The effectiveness value estimate is used to predict the expected effectiveness benefit of the output of the large language model based on the generated morphemes. Based on the GAE method, a security advantage estimate is calculated from the security reward value and the security value estimate. The security advantage estimate is used to quantify the net security gain of the current morpheme of the large language model. The security value estimate is used to predict the expected security gain of the output of the large language model based on the generated morphemes. The effectiveness loss and the security loss are calculated based on the Proximity Policy Optimization (PPO) algorithm, which optimizes the effectiveness advantage estimate and the security advantage estimate, respectively.

13. The apparatus according to any one of claims 10 to 12, characterized in that, The processing unit is specifically used for: The effectiveness gradient is projected onto the orthogonal complement space of the security gradient based on the projection intensity, and the projection intensity is used to control the elimination ratio of the conflict components.

14. The apparatus according to any one of claims 10 to 13, characterized in that, The processing unit is specifically used for: The effective gradient projection and the safety gradient are scaled using a gradient scaling algorithm, which is used to balance the difference in gradient magnitude between the effective gradient projection and the safety gradient. The scaled validity gradient projection and the scaled security gradient are combined to obtain the fused gradient information.

15. The apparatus according to claim 14, characterized in that, The processing unit is also used for: The weight parameters are determined based on the user demand feature vector. The weight parameters include an effectiveness weight parameter and a security weight parameter. The effectiveness weight parameter is used to indicate the weight corresponding to the scaled effectiveness gradient projection, and the security weight parameter is used to indicate the weight corresponding to the scaled security gradient. The processing unit is specifically used for: Based on the validity weight parameter and the security weight parameter, the scaled validity gradient projection and the scaled security gradient are synthesized into a total gradient to obtain the fused gradient information.

16. The apparatus according to any one of claims 9 to 15, characterized in that, The processing unit is also used for: Obtain the validity training dataset and the security training dataset; The effectiveness reward model is trained based on the effectiveness training dataset, which includes training data on effective human preferences. The security reward model is trained based on the security training dataset, which includes training data on human security preferences.

17. A computing device, characterized in that, The device includes a processor coupled to a memory, the processor storing instructions which, when executed by the processor, cause the computing device to perform the method of any one of claims 1 to 8.

18. A computing device cluster, characterized in that, The system includes at least one computing device, the computing device including a processor coupled to a memory, the processor being used to store instructions that, when executed by the processor, cause the cluster of computing devices to perform the method of any one of claims 1 to 8.

19. A computer-readable storage medium having instructions stored thereon, characterized in that, When the instructions are executed, they cause the computer to perform the method of any one of claims 1 to 8.

20. A computer program product, the computer program product comprising instructions, characterized in that, When the instructions are executed, they cause the computer to implement the method of any one of claims 1 to 8.

Images