RLHF-based large language model generation content privacy protection method

Abstract

The invention discloses an RLHF-based large language model generation content privacy protection method. The method comprises the following steps: training a base model as an instruction fine tuning model based on supervised instruction fine tuning; disassembling the original reward model into a benefit model and a cost model; expanding preference understanding of the cost model; training a scoring model based on the data set containing the multi-dimensional score and the multi-dimensional weight; a traditional preference probability calculation formula is improved based on a weighted score and a Bradley-Terry preference model; constructing a human privacy preference probability calculation method through a multi-dimensional score and regret theory; training a benefit model and a cost model; and optimizing the training instruction fine tuning model based on the benefit model and the cost model in combination with a human privacy preference probability calculation method. According to the method, potential sensitive information can be adaptively and dynamically identified and avoided, so that efficient and accurate privacy protection is realized, and meanwhile, the common problem of performance reduction in a traditional privacy protection method is effectively avoided.

Description

Technical Field

[0001] The present invention relates to the field of computer artificial intelligence and large model security technology, and specifically to a method for protecting content privacy generated by a large language model based on RLHF. Background Art

[0002] In recent years, large language models based on deep learning (such as the GPT series, BERT, LLaMA, etc.) have made significant progress in the field of natural language processing. These models are trained with massive amounts of data to generate fluent and coherent texts, and are widely used in tasks such as intelligent question answering, content generation, translation, and text summarization. However, as the application scope of large language models continues to expand, their potential privacy protection issues have gradually attracted widespread attention.

[0003] During the training process, large language models usually rely on a large amount of public data that has not been strictly screened, which may contain sensitive information (such as personal identity information, medical records, etc.). During the inference phase, the model may inadvertently generate content containing this sensitive information, resulting in the risk of privacy leakage. In addition, attackers may also infer the private content in the input or training data based on the model output through reverse reasoning, which exacerbates the threat of privacy leakage.

[0004] At present, the privacy protection methods of large language models mainly include data screening and desensitization, differential privacy, federated learning and output filtering. Data screening and desensitization reduce the risk of privacy leakage by removing or processing sensitive information in training data, but due to the diversity of sensitive information and high processing costs, privacy issues cannot be completely avoided. Differential privacy provides certain privacy protection by adding noise to model parameters or gradients during the training phase, but it may lead to performance degradation when generating high-quality text. Federated learning, as a distributed training method, avoids direct data leakage by only sharing parameter updates, but it faces problems such as high communication costs, parameter leakage risks and uneven data distribution. Output filtering quickly reduces the risk of privacy leakage by post-processing the generated content, but this method usually relies on predefined rules, lacks flexibility, and may affect the coherence and integrity of the text. Therefore, existing methods still find it difficult to find an ideal balance between privacy protection and model performance. In this context, the application of reinforcement learning from human feedback (RLHF) technology shows great potential. By combining the reinforcement learning framework, RLHF can dynamically adjust the model behavior so that its generated content is more in line with specific preferences. However, currently RLHF is mainly used to improve generation quality, and its application to privacy protection issues is still in the exploratory stage. Summary of the invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and propose a method for privacy protection of content generation by a large language model based on RLHF. By introducing the framework of reinforcement learning, the privacy protection goal is integrated into the model training process to achieve dynamic and adaptive protection of the privacy of generated content and achieve a better balance between privacy protection and generation quality.

[0006] In order to achieve the above object, the technical solution specifically adopted by the present invention is as follows:

[0007] A method for protecting content privacy based on a large language model generated by RLHF, comprising the following steps:

[0008] S1. Train the base model based on supervised instruction fine-tuning: Based on the instruction fine-tuning dataset annotated for specific downstream tasks of generative type, train the open source base model as the instruction fine-tuning model;

[0009] S2. Decoupling usefulness and harmlessness: Decompose the original reward model into benefit model and cost model;

[0010] S3. Expand the preference understanding of the cost model: Expand the linear mapping layer of the cost model so that the cost model has a multi-dimensional refined scoring head structure, and add a gating network to dynamically analyze the weight of each dimension for privacy protection based on the input content;

[0011] S4. Training a multidimensional scoring model: training a scoring model based on a labeled dataset containing multidimensional scores and multidimensional weights;

[0012] S5. Optimize the traditional preference probability calculation formula: Improve the traditional preference probability calculation formula based on weighted scores and Bradley-Terry preference model;

[0013] S6. Construct a new probability calculation method for human privacy preference: Construct a new probability calculation method for human privacy preference by combining multidimensional scores with regret theory;

[0014] S7. Train benefit model and cost model: Use the preference to train benefit model and cost model on the dataset;

[0015] S8. Reinforcement learning training instruction fine-tuning model: Based on the benefit model and cost model described above combined with a new method for calculating the probability of human privacy preferences, the training instruction fine-tuning model is optimized through proximal strategies.

[0016] Furthermore, in the step S2, the last non-embedded layer of the Transformer pre-trained language model is removed, and an additional linear layer is added to the last Transformer layer to construct a benefit model and a cost model; given any text, the benefit model and the cost model will assign a scalar reward value to the last token, which are used as indicators for evaluating the usefulness and privacy of the answer, respectively.

[0017] Furthermore, in the step S3, an additional linear layer of the cost model obtained in step S2 is expanded into five separate linear layers, and compliance, sensitivity, correctness, objectivity, and credibility are scored in a fine-grained manner respectively; and a gating network designed based on the idea of ​​a hybrid expert model is added to an additional linear layer of the cost model obtained in step S2, and the impact of each dimension on privacy protection is dynamically analyzed for the input content to obtain different weights.

[0018] Furthermore, in step S4, the mean square error between each dimension score output by the model and the annotation score, as well as the mean square error between each dimension weight generated by the model and the annotation weight are first calculated, and this is used as the main optimization target to obtain the following loss function with the introduction of the regularization term:

[0019]

[0020] In the formula, represents the score error loss of the i-th dimension, represents the weighted error loss of the i-th dimension; d is the number of dimensions of the score, is the regularization coefficient, which controls the influence of the regularization term on the total loss; The regularization term for the score, Regularization term for weights.

[0021] Then, the loss function is used to penalize the model for situations where the variance of the score prediction and weight prediction on a single sample is too large, avoiding the generation of unreasonable extreme values ​​in a single sample, thereby outputting a smoother and more stable score and weight distribution.

[0022] Furthermore, in step S5, firstly, based on the cost model obtained in step S3, a weighted score after dynamic analysis weighting of the input content can be obtained. Then use the weighted score Instead of a single evaluation score, the model is more likely to select answers that meet human privacy preferences. The new preference probability calculation formula is:

[0023]

[0024] In the formula, Represents the preference model judgment for a given input x Better than probability; For input x and output The weighted score of For input x and output The numerical value of the weighted score; It is used to convert the scores into probabilities and form a softmax function to ensure that the sum of the probabilities is 1.

[0025] Furthermore, in step S6, regret is defined as the deviation between each trajectory segment and the optimal decision, which is used to measure its loss in expected reward; regret can be decomposed into partial reward and state value change, the former is the difference between the reward of each state-action pair and the optimal reward, and the latter is the value difference between the start and end states of the trajectory segment. In the preference model based on Boltzmann rational distribution, the discounted partial reward of the trajectory segment is calculated by the reward model score, the discount factor and the total time reward of the trajectory.

[0026] Furthermore, in the step S6, in the cost model obtained in step S3, the state represents the score of the model in each dimension, and the action is represented by the weight output by the gating network, which is used to adjust the score of each dimension; first, the score of a certain action after weight adjustment relative to the benchmark (unadjusted score), i.e., the negative regret (advantage), is calculated by the following formula: ,in, The benchmark return is the score of the better answer in the i-th dimension is the average score of all samples on the i-th dimension;

[0027] Then use the gating weight to weight the advantages of each dimension ,in is the weight of a better answer on the i-th dimension. Combined with the discount factor , calculate the future weighted total advantage: , where t is the time step and T is the sequence length (time step);

[0028] Then, for each pair of samples in the batch, the Boltzmann distribution is used to calculate the preference between the "better" and "worse" answers, i.e. the probability of regret, exponentially summed up to the weighted advantage of each dimension:

[0029]

[0030] In the formula, It represents the probability that the model judges that the better answer is better than the worse answer given a pair of samples; represents the cumulative weighted advantage value of the better answer, represents the weighted advantage value accumulated by the poorer responses; It is used to convert the scores into probabilities and form a softmax function to ensure that the sum of the probabilities is 1.

[0031] Finally, the preference probability between the two weighted answers is calculated.

[0032] Furthermore, in step S7, the benefit model is used to evaluate the usefulness of the generated content by maximizing Better than The probability of adjusting the parameters of the benefit model , construct the loss function for benefit model training: , where represents the expected value, represents the training dataset of the benefit model, is a triplet sampled from the dataset, where x is the input data, and are better and worse responses to the input data, respectively; The benefit model is better for the input x and output The probability function of The benefit model has a poor answer for input x and output The probability function of is the logarithmic form of the logistic sigmoid function and is used to convert probabilities into logits.

[0033] Furthermore, in step S7, the cost model is used to evaluate the privacy protection performance of the generated content, and by introducing the design of regret loss, the gap between the generated content and the ideal privacy protection target is quantified. The regret loss is expressed as: ; In order to better utilize this information to train the cost model, the original pairwise comparison loss is corrected by introducing classification terms, and the loss function for cost model training is constructed:

[0034]

[0035]

[0036] In the formula, represents the expected value, Represents the training dataset for the cost model and are triples and quintuples sampled from the dataset, where x is the input data, and The responses are more privacy-preserving and less privacy-preserving for input data, respectively. and are the scaling factors for the classification losses of the two responses respectively; The cost model is more privacy-preserving for input x and output answer The probability function of The cost model lacks privacy protection for input x and output The probability function of is the logarithmic form of the logistic sigmoid function, used to convert probabilities into logits; is the regret loss introduced above.

[0037] Furthermore, in step S8, the goal of reinforcement learning training is to maximize the usefulness reward of the generated content while ensuring that the generated response meets the privacy protection standard, that is, for each given input x and corresponding output y, all outputs are guaranteed to be safe. Then the goal of reinforcement learning training is as follows:

[0038]

[0039] In order to deal with this constrained optimization problem, the Lagrange multiplier method is used to introduce a non-negative Lagrange multiplier λ, and the constraints are incorporated into the optimization objective, thus transforming it into the following unconstrained problem:

[0040]

[0041] In the formula, represents the expected benefit objective function, Represents the expected cost objective function; λ represents the degree of penalty for violating security constraints. If the response generated by the model is safer, then λ will decrease, otherwise it will increase, thereby dynamically adjusting the penalty for privacy protection.

[0042] The above-mentioned method of generating content privacy protection based on a large language model of RLHF in the present invention can adaptively and dynamically identify and avoid potential sensitive information, thereby achieving efficient and accurate privacy protection, while effectively avoiding the common performance degradation problem in traditional privacy protection methods, and has the following beneficial effects:

[0043] 1) By decoupling the utility model and the cost model, the privacy protection goal and the quality goal of the generated content are modeled separately, and the two requirements are deeply integrated in model training, which significantly reduces the conflict between privacy protection and model performance.

[0044] 2) Design a multi-dimensional privacy evaluation mechanism and a gating network based on a hybrid expert model, which can accurately evaluate the generated content from multiple dimensions such as compliance, sensitivity, and correctness. The model can dynamically activate the appropriate expert network according to the characteristics of the input content and strike a balance between privacy protection and the quality of generated content.

[0045] 3) Modeling human privacy preferences based on regret theory dynamically optimizes privacy protection strategies to achieve a balance between privacy protection and generation quality. Through the regret value feedback mechanism, the model can adaptively adjust generation behavior to improve privacy protection efficiency while reducing long-term privacy risks and training costs. In addition, this mechanism enhances the multi-dimensional optimization capabilities of the model, supports the coordinated improvement of goals such as harmlessness, helpfulness, and privacy, and improves user trust and actual application effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A training flow chart of a method for generating content privacy protection using a large language model based on human feedback reinforcement learning according to an embodiment of the present invention;

[0047] Figure 2 4 is a diagram of the cost model structure in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

[0049] like Figure 1 As shown, the present invention provides a method for privacy protection of content generation based on a large language model of human feedback reinforcement learning. By decoupling the benefit model and the cost model, the understanding of privacy protection is improved; a multi-dimensional scoring head and a gating network are designed to dynamically analyze the weight of the input content in each dimension of privacy protection; combined with regret theory, the traditional preference probability calculation is optimized, the benefit model and the cost model are better trained, and the accuracy of privacy protection calculation is improved; finally, the dual guarantee of privacy protection and generated content quality is successfully achieved through the fine-tuning model of reinforcement learning training instructions; specifically, the following steps are included:

[0050] S1. Train the base model based on supervised instruction fine-tuning: Based on the instruction fine-tuning dataset annotated for specific downstream tasks of generative models, train the open source base model as the instruction fine-tuning model. The open source base models include the gpt2 model, LLama model, Baichuan model, InternLM model, etc.; instruction fine-tuning uses the Lora instruction fine-tuning method provided by the LLAMA-Factory framework.

[0051] S2. Decoupling usefulness and harmlessness: The original reward model is decomposed into a benefit model and a cost model. Specifically, the last embedding layer of the pre-trained language model based on the Transformer structure (such as GPT-2, LLama, etc.) is removed, and its original Transformer architecture and weights are retained to construct a decoupled model. On this basis, two models (the instruction fine-tuning training base model obtained in step S1, one model to build the benefit model, and the other to build the cost model) are used to add an independent linear mapping layer at the output of the last Transformer layer as a benefit model and a cost model. The linear mapping layer is used to receive the hidden state output by the last Transformer layer as input and project it into a scalar value to evaluate the usefulness and privacy of the input text respectively. During the training process, the benefit model uses usefulness annotated data (such as scores for dimensions such as correctness and coherence) for supervised training, while the cost model uses privacy-related annotated data (such as whether it contains sensitive information) for separate optimization. Through this design, the benefit model and the cost model can independently evaluate the performance of the input text in different dimensions, thereby achieving effective decoupling of usefulness and harmlessness. This decoupled approach not only ensures a balance between usefulness and privacy of the generated content, but also provides greater flexibility for subsequent customized optimization.

[0052] S3. Expand the preference understanding of the cost model: Expand the linear mapping layer of the cost model so that the cost model has a multi-dimensional refined scoring head structure, and add a gating network to dynamically analyze the weight of each dimension for privacy protection based on the input content. Specifically, the single linear mapping layer in the cost model obtained in step S2 is expanded into five independent linear mapping layers to score compliance, sensitivity, correctness, objectivity and credibility respectively. Each linear mapping layer extracts features from the hidden state of the last layer of Transformer and generates a scoring value for the corresponding dimension, thereby realizing a multi-dimensional refined evaluation of privacy protection. In addition, in the scoring head structure of the cost model obtained in step S2, a gating network based on the design idea of ​​the hybrid expert model is introduced. The gating network generates weight coefficients for the above five dimensions by dynamically analyzing the input content. These weight coefficients are used to measure the importance of different dimensions to privacy protection under specific input content, and then dynamically adjust the overall scoring method of the cost model. By combining the multidimensional scoring head with the gating network, the cost model can flexibly assign dimension weights according to the characteristics of the input content while evaluating the privacy risks of the generated content in a fine-grained manner, thereby achieving a more comprehensive and accurate modeling and understanding of privacy protection preferences.

[0053] S4. Training multidimensional scoring model: The scoring model is trained based on a labeled dataset containing multidimensional scoring and multidimensional weights. During the training process, the mean square error between the score of each dimension output by the model and the labeled score, as well as the mean square error between the weight of each dimension generated by the model and the labeled weight are calculated first, and this is used as the main optimization target to obtain the following loss function with the introduction of regularization terms:

[0054]

[0055] In the formula, represents the score error loss of the i-th dimension, represents the weighted error loss of the i-th dimension; d is the number of dimensions of the score, is the regularization coefficient, which controls the influence of the regularization term on the total loss; The regularization term for the score, Regularization term for weights.

[0056] Then, the loss function of the regularization term is introduced to penalize the model for excessive variance in the score prediction and weight prediction on a single sample. Through the constraints of the regularization term, the model can avoid generating unreasonable extreme values ​​in a single sample, thereby outputting a smoother and more stable score and weight distribution. This training method aims to enable the model to generate a distribution that better meets actual needs while maintaining high prediction accuracy, thereby improving the robustness and applicability of the multidimensional scoring model.

[0057] S5. Optimize the traditional preference probability calculation formula: Improve the traditional preference probability calculation formula by combining weighted scores with the Bradley-Terry preference model. Specifically, dynamically analyze the input content through the cost model obtained in step S3, generate the weights of each dimension, and calculate the weighted scores. The weighted score The comprehensive score and weight of multiple dimensions can more comprehensively reflect the comprehensive performance of the input content in terms of privacy protection. Traditional preference prediction is based on the Bradley-Terry model, that is, given an input In this case, the model assumes Output is better than The probability of the output is used to describe how the model determines which output is more in line with human preferences during the training process. Replace the single score in the traditional model , so that the preference calculation can more accurately express the comprehensive performance of the generated content in terms of privacy protection preferences. The new preference probability calculation formula is:

[0058]

[0059] By introducing weighted scores generated by dynamic analysis, the model can more effectively judge and prioritize answers that meet human needs in terms of privacy protection and other preference goals, thereby improving the performance and practicality of the model in privacy protection tasks.

[0060] S6. Construct a new probability calculation method for human privacy preference: Construct a probability method for human privacy preference by combining multidimensional scores with regret theory. In this embodiment, regret is defined as the deviation between each segment and the optimal decision, which measures the loss of each segment in expected return. This decision is more in line with human privacy protection awareness. Specifically, regret can be decomposed into partial returns and state value changes, where partial returns represent the direct loss of each segment in expected returns, that is, the difference between the reward brought by each state-action pair and the reward brought by the optimal strategy. The state value change represents the change between the start and end state values ​​of each segment, which reflects the degree of movement of each segment in the state space. In the regret preference model, a model based on Boltzmann rational distribution is usually used. The core idea of ​​this model is expert preference. Compared to The probability is based on the discounted partial return of each trajectory segment: ,in is the score of the reward model on each state-action pair, which is used to calculate the quality of each trajectory, is the time step The sum of the discounted rewards represents the total reward value of the trajectory segment over time.

[0061] In reinforcement learning, the state (State, s) usually represents the characteristics of the environment or input content, and the action (Action, a) represents the decision or behavior taken in the current state. In the cost model obtained in step 3, the state s can be expressed as the score of each dimension, that is, , specifically, the model’s prediction score. For each input , the model predicted score in the i-th dimension is . is the score of the better answer in the i-th dimension, is the score of the worse answer in the i-th dimension, and these scores constitute the state. The action is represented by the weight output by the gating network, and the weight of the gating network in each dimension is , then the gating weights will adjust the score of each dimension. represents the gating weight of the i-th dimension, and , so it can be mapped by the Sigmoid function .

[0062] Advantage indicates how good an action (i.e., the weight-adjusted score) is relative to a benchmark (the unadjusted score). Advantage can be calculated using the following formula: Advantages is the difference between the return after selecting an action a in a certain state s and the benchmark return (i.e., the average score). is the average score of all samples in the i-th dimension. The advantage of each dimension is weighted by the gating weight In this way, the advantage reflects the influence of each dimension by adjusting the weights.

[0063] For each sample and , calculate its weighted total advantage (i.e. the sum of weighted advantages), and use the discount factor to calculate the future weighted advantage: , where T is the sequence length (time steps).

[0064] For each pair of samples in the batch, the Boltzmann distribution is used to calculate the preference between the “better” and “worse” answers, i.e. the probability of regret:

[0065]

[0066] The implementation exponentiates the sum of the weighted advantages of each dimension and then calculates the probability of preference between the two weighted answers.

[0067] S7. Training benefit model and cost model: Use the preference pair dataset to train benefit model and cost model. The benefit model aims to evaluate the usefulness of generated content, and the model parameters are adjusted by maximizing the preference probability during training. Specifically, the model determines which of the two candidate outputs is better based on the preference data, and uses this to construct the training loss function:

[0068]

[0069] By minimizing the loss function, the benefit model can more accurately predict the quality of the generated content, thereby improving its performance in terms of task completion.

[0070] The goal of the cost model is to evaluate the privacy protection performance of the generated content and quantify the gap between the generated content and the ideal privacy protection goal by introducing the design of regret loss. The regret loss can be expressed as:

[0071]

[0072] In order to better utilize this information to train the cost model, the original pairwise comparison loss is modified by introducing a classification term:

[0073]

[0074] In the formula, represents the expected value, Represents the training dataset for the cost model and are triples and quintuples sampled from the dataset, where x is the input data, and The responses are more privacy-preserving and less privacy-preserving for input data, respectively. and are the scaling factors for the classification losses of the two responses respectively; The cost model is more privacy-preserving for input x and output answer The probability function of The cost model lacks privacy protection for input x and output The probability function of is the logarithmic form of the logistic sigmoid function, used to convert probabilities into logits; is the regret loss introduced above.

[0075] The classification loss term is used to explicitly classify the privacy risks of the generated content, and improves the model's ability in fine-grained privacy assessment through joint optimization with the regret loss.

[0076] Finally, the comprehensive loss function effectively balances the two optimization objectives, allowing the cost model to perform a more accurate dynamic evaluation of privacy protection. The loss function for cost model training is constructed:

[0077]

[0078]

[0079] S8. Reinforcement learning training instruction fine-tuning model: Based on the benefit model and cost model described above combined with the new human privacy preference probability calculation method, the training instruction fine-tuning model is optimized through the proximal strategy. The training goal is to maximize the usefulness reward of the generated content while ensuring that the generated response meets the privacy protection standard, that is, for each given input x and corresponding output y, all outputs do not contain privacy issues. Therefore, the reinforcement learning training goal can be expressed as:

[0080]

[0081] In the formula, Represents sampling an input 𝑥 from the data distribution 𝐷 (i.e., the prompt dataset used for RL training); Indicates that from the current LLM strategy Sample an answer from the generated answer distribution ; Express expectations, represents the reward value given by the helpfulness reward model, Represents the privacy-preserving cost evaluated by the cost model.

[0082] Since optimization problems that strictly satisfy the harmlessness constraint usually significantly increase the difficulty of reinforcement learning training, a new hyperparameter d is introduced to rewrite the original hard constraint into an expected form, making the objective function more optimizable. Specifically, the objective function consists of the following two parts:

[0083] , represents the expected usefulness reward;

[0084] , represents the desired harmlessness cost target, and the probability of generating an unsafe response is controlled by the hyperparameter d.

[0085] The rewritten objective function becomes as follows:

[0086] To further deal with this constrained optimization problem, the Lagrange multiplier method is used to transform the harmlessness constraint into a form of constraint penalty. By introducing non-negative Lagrange multipliers, the usefulness objective and the harmlessness constraint are unified into one optimization objective. The transformed objective function is:

[0087]

[0088] In the formula, λ represents the degree of penalty for violating security constraints. When the model generates safer responses, λ will gradually decrease. On the contrary, if the generated responses have more privacy issues, λ will increase, thereby dynamically adjusting the penalty for privacy protection.

[0089] 1. Experimental setup

[0090] The dataset used in the simulation experiment of the present invention is PKU-SafeRLHF in PKU-Alignment, which contains more than 30,000 expert comparison data, each of which contains two answers to a question, as well as safety meta tags and preferences based on helpfulness and harmlessness. The dataset provides detailed rankings for answers through independent helpfulness and harmlessness evaluations, aiming to improve the safety of language model output through this two-dimensional ranking system. The dataset is used for research purposes, with a particular emphasis on research on reducing the harmfulness of the model, and contains potentially offensive or harmful content.

[0091] 2. Results Analysis

[0092] The simulation experiment of the present invention adopts the present invention and an existing related technology Safe RLHF training framework. The related technology Safe RLHF training framework refers to the training framework based on human feedback reinforcement learning proposed by Dai J et al. in "Safe RLHF: Safe Reinforcement Learningfrom Human Feedback, 2023", which decouples human preferences for usefulness and harmlessness, allows training of separate reward and cost models, formalizes the safety problem of LLMs as an optimization task of maximizing the reward function while satisfying specified cost constraints, and dynamically adjusts the balance between the two objectives during fine-tuning.

[0093] In order to verify the effect of the simulation experiment of the present invention, GPT-2 and GPT-Neo models were used for verification. First, the accuracy of correct human privacy preferences was obtained by using the validation set evaluation model in the benefit model and the cost model; secondly, the target model after completing the entire training framework and the model that was only fine-tuned with supervised instructions were used to answer prompts that induced privacy leakage, and the GPT-4 score was used for comparison. Since it involves the comparison of two models, the Elo score calculation is used, which is expressed as the target model after completing the entire training framework has a higher score than the model that was only fine-tuned with supervised instructions. All results are plotted in the following table:

[0094]

[0095] Combined with the above table, it can be seen that the cost model and GPT-4 score in the present invention have shown significant improvements. Specifically, for the GPT-2 model, the effect of the present invention is better than the traditional Safe-RLHF method, in which the accuracy of the cost model is improved by 1.57%, and the GPT-4 score is improved by 14.78 points; for the GPT-Neo model, the accuracy of the benefit model and the cost model are improved by 0.74% and 0.97% respectively, and the GPT-4 score is improved by 26.53 points. As the number of model parameters increases, the effect of the present invention is more significant, especially in the improvement of GPT-4 scores. The present invention combines a multidimensional scoring system with dynamic weight analysis of a gated network, so that the model can more comprehensively understand and respond to human preferences, especially in the field of privacy protection, showing stronger adaptability and accuracy. By improving the modeling of the probability of human privacy preferences, the model's ability to protect privacy data can be more effectively improved in reinforcement learning.

[0096] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for protecting content privacy based on a large language model generated by RLHF, characterized in that: The following steps are involved: S1. Train the base model based on supervised instruction fine-tuning: Based on the instruction fine-tuning dataset annotated for specific downstream tasks of generative type, train the open source base model as the instruction fine-tuning model; S2. Decoupling usefulness and privacy: Decomposing the original reward model into a benefit model and a cost model; S3. Expand the preference understanding of the cost model: Expand the linear layer of the cost model so that the cost model has a multi-dimensional refined scoring head structure, and add a gating network to dynamically analyze the weight of each dimension for privacy protection based on the input content; S4. Training a multidimensional scoring model: training a scoring model based on a labeled dataset containing multidimensional scores and multidimensional weights; S5. Optimize the traditional preference probability calculation formula: Improve the traditional preference probability calculation formula based on weighted scores and Bradley-Terry preference model; S6. Construct a new probability calculation method for human privacy preference: Construct a new probability calculation method for human privacy preference by combining multidimensional scores with regret theory; S7. Train benefit model and cost model: Use the preference to train benefit model and cost model on the dataset; S8. Reinforcement learning training instruction fine-tuning model: Based on the benefit model and cost model described above combined with a new method for calculating the probability of human privacy preferences, the training instruction fine-tuning model is optimized through proximal strategies.

2. The method for generating content privacy protection based on a large language model of RLHF as claimed in claim 1, characterized in that: In the step S2, the last non-embedded layer of the Transformer pre-trained language model is removed, and an additional linear layer is added to the last Transformer layer to construct a benefit model and a cost model; given any text, the benefit model and the cost model will assign a scalar reward value to the last token, which are used as indicators for evaluating the usefulness and privacy of the answer, respectively.

3. The method for protecting content privacy by generating a large language model based on RLHF as claimed in claim 1, characterized in that: In the step S3, an additional linear layer of the cost model obtained in step S2 is expanded into five separate linear layers, and fine-grained scoring is performed on compliance, sensitivity, correctness, objectivity, and credibility respectively; A gating network designed based on the idea of ​​a hybrid expert model is added to an additional linear layer of the cost model obtained in step S2 to dynamically analyze the impact of each dimension on privacy protection based on the input content and obtain different weights.

4. The method for protecting content privacy based on a large language model generated by RLHF as claimed in claim 1, characterized in that: In step S4, the mean square error between the score of each dimension output by the model and the annotation score, as well as the mean square error between the weight of each dimension generated by the model and the annotation weight are first calculated, and this is used as the main optimization target to obtain the following loss function with the introduction of the regularization term: : ； In the formula, represents the score error loss of the i-th dimension, Represents the weight error loss of the i-th dimension; d is the number of dimensions of the rating, is the regularization coefficient, which controls the influence of the regularization term on the total loss; The regularization term for the score, Regularization term for weights; Then, the loss function is used to penalize the model for situations where the variance of the score prediction and weight prediction on a single sample is too large, thereby avoiding the generation of unreasonable extreme values ​​in a single sample.

5. The method for generating content privacy protection based on a large language model of RLHF as claimed in claim 1, characterized in that: In step S5, firstly, based on the cost model obtained in step S3, a weighted score after dynamic analysis weighting of the input content can be obtained. Then use the weighted score Instead of a single evaluation score, the new preference probability calculation formula is: ； In the formula, Represents the preference model judgment for a given input x Better than The probability of For input x and output The weighted score of For input x and output The numerical value of the weighted score; It is used to convert the scores into probabilities and form a softmax function to ensure that the sum of the probabilities is 1.

6. The method for protecting content privacy by generating a large language model based on RLHF as claimed in claim 1, characterized in that: In step S6, regret is defined as the deviation between each trajectory segment and the optimal decision, which is used to measure its loss in expected reward; regret can be decomposed into partial reward and state value change, the former is the difference between the reward of each state-action pair and the optimal reward, and the latter is the value difference between the start and end states of the trajectory segment.

7. The method for protecting content privacy by generating a large language model based on RLHF as claimed in claim 1, characterized in that: In the step S6, in the cost model obtained in step S3, the state represents the score of the model in each dimension, and the action is represented by the weight output by the gating network, which is used to adjust the score of each dimension; first, the score of a certain action after weight adjustment relative to the benchmark is calculated by the following formula: ,in, is the score of a better answer on the i-th dimension, the benchmark return is the average score of all samples on the i-th dimension; Then use the gating weight to weight the advantages of each dimension ,in is the weight of the better answer on the i-th dimension; combined with the discount factor , calculate the future weighted total advantage: , where t is the time step and T is the sequence length; Then, for each pair of samples in the batch, the Boltzmann distribution is used to calculate the preference between the "better" and "worse" answers, i.e. the probability of regret, exponentially summed up to the weighted advantage of each dimension: ； In the formula, It represents the probability that the model judges that the better answer is better than the worse answer given a pair of samples; represents the cumulative weighted advantage value of the better answer, represents the weighted advantage value accumulated by the poorer responses; Used to convert scores into probabilities, forming a softmax function to ensure that the sum of probabilities is 1; Finally, the preference probability between the two weighted answers is calculated.

8. The method for protecting content privacy by generating a large language model based on RLHF as claimed in claim 1, characterized in that: In step S7, the benefit model is used to evaluate the usefulness of the generated content by maximizing Better than The probability of adjusting the parameters of the benefit model , construct the loss function for benefit model training: , where represents the expected value, represents the training dataset of the benefit model, is a triplet sampled from the dataset, where x is the input data, and are better and worse responses to the input data, respectively; The benefit model is better for the input x and output The probability function of The benefit model has a poor answer for input x and output The probability function of is the logarithmic form of the logistic sigmoid function and is used to convert probabilities into logits.

9. The method for protecting content privacy by generating a large language model based on RLHF as claimed in claim 1, characterized in that: In step S7, the cost model is used to evaluate the privacy protection performance of the generated content, and by introducing the design of regret loss, the gap between the generated content and the ideal privacy protection target is quantified. The regret loss is expressed as: , where B represents the total number of samples; by introducing classification terms to correct the original pairwise comparison loss, the loss function of the cost model training is constructed: ； ； In the formula, represents the expected value, Represents the training dataset for the cost model and are triples and quintuples sampled from the dataset, where x is the input data, and They are respectively the responses that provide more privacy protection and lack of privacy protection for the input data. and are the scaling factors for the classification losses of the two responses respectively; The cost model is more privacy-preserving for input x and output answer The probability function of The cost model lacks privacy protection for input x and output The probability function of is the logarithmic form of the logistic sigmoid function, used to convert probabilities into logits; is the regret loss introduced above.

10. The method for protecting content privacy by generating a large language model based on RLHF as claimed in claim 1, characterized in that: In step S8, the goal of reinforcement learning training is to maximize the usefulness reward of generated content while ensuring that the generated response meets the privacy protection standard; the Lagrange multiplier method is adopted to introduce a non-negative Lagrange multiplier λ, and the constraint condition is incorporated into the reinforcement learning training, thereby converting it into the following unconstrained problem: ； In the formula, represents the expected benefit objective function, represents the expected cost objective function; λ represents the degree of penalty for violating the safety constraint. If the response generated by the model is safer, then λ will decrease, otherwise it will increase.

Images