Large model security alignment method based on dynamic constraint reinforcement learning

Abstract

The invention provides a large model security alignment method based on dynamic constraint reinforcement learning. The method comprises the following steps: constructing a dialogue preference data set; respectively training an initial reward model and an initial cost model based on the dialogue preference data set to obtain a trained reward model and a trained cost model; generating a first fine tuning strategy and a second fine tuning strategy based on the trained reward model and the trained cost model; and updating the initial large language model according to the first fine tuning strategy and / or the second fine tuning strategy to obtain an updated large language model. According to the method, the reward model and the cost model are respectively trained through the constructed dialogue preference data set, and the first fine tuning strategy and the second fine tuning strategy are generated through the trained reward model and the cost model, so that the large language model is updated, and the balance between the safety and the usefulness of the large language model is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a large model security alignment method based on dynamic constraint reinforcement learning. Background Art

[0002] Existing security alignment methods for large language models mainly rely on reinforcement learning from human feedback (RLHF), which captures human preferences by modeling reward models and uses reinforcement learning algorithms such as proximal policy optimization to fine-tune large models, thereby guiding the model to generate content that is more in line with human values. RLHF can effectively reduce the deviation or harmful content generated by the model and improve the reliability and compliance of the output.

[0003] In conversation scenarios with large language models (LLMs), traditional RLHF methods have significant deficiencies in the balance between usefulness and security. Although these models perform well in providing useful answers, they often generate inappropriate or even harmful content when it comes to sensitive topics or complex interactions. This imbalance not only reduces the security of the model, but also affects its acceptability in actual conversation scenarios. That is, in existing technical solutions, large language models have significant deficiencies in the balance between security and usefulness. Summary of the invention

[0004] The embodiment of the present application provides a large model security alignment method based on dynamic constrained reinforcement learning to solve the problem that the large language model in the prior art has a significant inadequacy in balancing security and usefulness.

[0005] To solve the above problems, this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a large model security alignment method based on dynamic constraint reinforcement learning, the method comprising:

[0007] Constructing a dialogue preference data set, wherein the dialogue preference data set includes a plurality of dialogue data, each of the dialogue data includes a prompt question and an answer;

[0008] Based on the dialogue preference dataset, the initial reward model and the initial cost model are trained respectively to obtain a trained reward model and a trained cost model;

[0009] Generate a first fine-tuning strategy based on the trained reward model and the trained cost model, and generate a second fine-tuning strategy based on the trained reward model and the trained cost model, wherein the first fine-tuning strategy is used to improve the response quality and safety of the large model, and the second fine-tuning strategy is used to dynamically adjust the safety constraints of the large model;

[0010] The initial large language model is updated according to the first fine-tuning strategy and / or the second fine-tuning strategy to obtain an updated large language model.

[0011] Optionally, the training of the initial reward model and the initial cost model based on the dialogue preference dataset to obtain a trained reward model and a trained cost model includes:

[0012] Dividing the conversation preference data set according to content to obtain a first preference data set and a second preference data set, wherein the first preference data set includes a plurality of conversation data that focuses on the usefulness of answers, and the second preference data set includes a plurality of conversation data that focuses on the safety of answers;

[0013] Training the initial reward model based on the first preference data set to obtain a trained reward model;

[0014] The initial cost model is trained based on the second preference data set to obtain a trained cost model.

[0015] Optionally, updating the initial large language model according to the first fine-tuning strategy and / or the second fine-tuning strategy to obtain an updated large language model includes:

[0016] updating the initial large language model according to the first fine-tuning strategy to obtain an updated first large language model;

[0017] The initial large language model is updated according to the second fine-tuning strategy to obtain an updated second large language model.

[0018] Optionally, updating the initial large language model according to the first fine-tuning strategy to obtain an updated first large language model includes:

[0019] Initializing parameters of the first fine-tuning strategy, and generating a first constraint strategy according to the trained reward model and the trained cost model;

[0020] Use the gradient ascent method to perform non-parametric space optimization and determine the optimal update strategy for the non-parametric space;

[0021] Projecting the optimal update strategy into a parameterized space, updating the optimal update strategy, and obtaining a target fine-tuning strategy;

[0022] The initial large language model is iteratively updated based on a target fine-tuning strategy to obtain the updated large language model.

[0023] Optionally, updating the initial large language model according to the second fine-tuning strategy to obtain an updated second large language model includes:

[0024] Initializing parameters of the second fine-tuning strategy;

[0025] Calculating a first loss corresponding to the trained reward model, and calculating a second loss corresponding to the trained cost model;

[0026] Setting a dynamic adjustment factor, where the dynamic adjustment factor is used to ensure that the large model always maintains the ability to suppress harmful output when generating content;

[0027] Updating the parameters of the initial large language model by a gradient descent method to obtain target parameters;

[0028] The initial large language model is iteratively updated based on the target parameter to obtain the updated large language model.

[0029] Optionally, after the initial large language model is updated according to the first fine-tuning strategy and the second fine-tuning strategy to obtain an updated large language model, the method further includes:

[0030] Setting the trained reward model to obey a first normal distribution, and setting the trained cost model to obey a second normal distribution;

[0031] Scoring the updated large language model based on the first normal distribution and the second normal distribution to obtain a target score value;

[0032] When the target score value is lower than a preset threshold, the updated large language model is iteratively updated again.

[0033] In a second aspect, an embodiment of the present application further provides a large model security alignment device based on dynamic constraint reinforcement learning, the device comprising:

[0034] A construction module, used to construct a dialogue preference data set, wherein the dialogue preference data set includes a plurality of dialogue data, each of which includes a prompt question and an answer;

[0035] A training module, used to train the initial reward model and the initial cost model based on the dialogue preference dataset to obtain a trained reward model and a trained cost model;

[0036] A generation module, configured to generate a first fine-tuning strategy based on the trained reward model and the trained cost model, and to generate a second fine-tuning strategy based on the trained reward model and the trained cost model, wherein the first fine-tuning strategy is used to improve the response quality and safety of the large model, and the second fine-tuning strategy is used to dynamically adjust the safety constraints of the large model;

[0037] An updating module is used to update the initial large language model according to the first fine-tuning strategy and / or the second fine-tuning strategy to obtain an updated large language model.

[0038] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a transceiver, a memory, a processor, and a program stored in the memory and executable on the processor; the processor is used to read the program in the memory to implement the steps of the method described in the first aspect above.

[0039] In a fourth aspect, an embodiment of the present application further provides a readable storage medium for storing a program, which, when executed by a processor, implements the steps in the method described in the first aspect above.

[0040] In a fifth aspect, an embodiment of the present application further provides a computer program product, wherein the computer program product is stored in a storage medium, and the computer program product is executed by at least one processor to implement the steps in the method described in the first aspect above.

[0041] The present application provides a large model security alignment method based on dynamic constraint reinforcement learning, the method comprising: constructing a dialogue preference dataset, the dialogue preference dataset comprising multiple dialogue data, each of the dialogue data comprising prompt questions and answers; training the initial reward model and the initial cost model respectively based on the dialogue preference dataset to obtain a trained reward model and a trained cost model; generating a first fine-tuning strategy based on the trained reward model and the trained cost model, and generating a second fine-tuning strategy based on the trained reward model and the trained cost model, the first fine-tuning strategy being used to improve the response quality and security of the large model, and the second fine-tuning strategy being used to dynamically adjust the security constraints of the large model; updating the initial large language model according to the first fine-tuning strategy and / or the second fine-tuning strategy to obtain an updated large language model. The present application trains the reward model and the cost model respectively by constructing a dialogue preference dataset, generates the first fine-tuning strategy and the second fine-tuning strategy by the trained reward model and the cost model, thereby updating the large language model and improving the balance between security and usefulness of the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 It is a flowchart of a large model security alignment method based on dynamic constraint reinforcement learning provided in an embodiment of the present application;

[0044] Figure 2 It is a structural schematic diagram of a large model security alignment device based on dynamic constraint reinforcement learning provided in an embodiment of the present application;

[0045] Figure 3 It is a schematic diagram of the structure of an electronic device provided by the implementation of this application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] The terms "first", "second" etc. in the embodiments of the present application are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. In addition, the terms "include" and "have" and any variation thereof are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment. In addition, "and / or" is used in the present application to represent at least one of connected objects, such as A and / or B and / or C, representing the inclusion of separate A, separate B, separate C, and A and B all exist, B and C all exist, A and C all exist, and 7 situations that A, B and C all exist.

[0048] See also Figure 1 , Figure 1 It is a flow chart of a large model security alignment method based on dynamic constraint reinforcement learning provided in an embodiment of the present application. Figure 1 The large model security alignment method based on dynamic constraint reinforcement learning shown can be executed by a computer or other devices.

[0049] Step 101: construct a dialogue preference dataset, wherein the dialogue preference dataset includes a plurality of dialogue data, each of which includes a prompt question and an answer.

[0050] In this embodiment, a dialogue preference dataset for training is first constructed, wherein the dialogue preference dataset includes a plurality of dialogue data, wherein each dialogue data packet includes a prompt question and an answer. r With D c , corresponding to human’s usefulness preference and security preference respectively, expressing the two types of human preferences through fine labeling and feature extraction.

[0051] Step 102: Train the initial reward model and the initial cost model based on the dialogue preference dataset to obtain a trained reward model and a trained cost model.

[0052] In this embodiment, the initial reward model refers to a reward model based on the BT probability model, and the initial cost model refers to a cost model based on the BT probability model. φ With cost model C ψ Training is performed so that the model can accurately predict and distinguish between usefulness and safety preference ratings.

[0053] Step 103: Generate a first fine-tuning strategy based on the trained reward model and the trained cost model, and generate a second fine-tuning strategy based on the trained reward model and the trained cost model, wherein the first fine-tuning strategy is used to improve the response quality and safety of the large model, and the second fine-tuning strategy is used to dynamically adjust the safety constraints of the large model.

[0054] In this embodiment, the first fine-tuning strategy is fine-tuning based on the first-order optimization method of the policy space, using the feedback of the reward model and the cost model, and adopting the FOCOPS algorithm to fine-tune the strategy of the large language model, so as to achieve the enhancement of security constraints while meeting the usefulness requirements.

[0055] The second fine-tuning strategy is proximal strategy optimization fine-tuning based on the penalty mechanism. It uses the signals provided by the reward model and the cost model, and uses the P3O algorithm to optimize the strategy of the large language model. It injects a risk control mechanism into the strategy fine-tuning to enhance the robustness of the model.

[0056] Step 104: Update the initial large language model according to the first fine-tuning strategy and / or the second fine-tuning strategy to obtain an updated large language model.

[0057] In this embodiment, the initial large language model is updated through the first fine-tuning strategy and / or the second fine-tuning strategy to obtain an updated large language model. Compared with the traditional RLHF method which usually relies on a single reward model and is difficult to meet the requirements of usefulness and harmlessness at the same time, and often results biased towards one side in application scenarios, in this proposal, through the dual model feedback mechanism, that is, the combination of the reward model and the cost model, the output probability of harmful content in the large model is effectively reduced, while the usefulness is improved.

[0058] With the popularity of large language models in various applications, especially in high-risk fields such as medicine, finance and law, the demand for model security is increasing. The improved security alignment strategy can effectively reduce the risk of models in practical applications and meet the industry's strict requirements for security. The dynamic constrained reinforcement learning optimization strategy in this proposal emphasizes robustness and flexibility, and can adapt to high-dimensional complex tasks. This feature makes the technology not only suitable for current alignment scenarios, but also can be expanded to emerging application scenarios in the future, enhancing its market competitiveness. By solving the problems of existing algorithms in training stability and parameter sensitivity, this technical solution is expected to significantly improve the efficiency and performance of the algorithm. This will attract enterprises and institutions that want to improve the security of models and further expand its market application scope. The improved security alignment framework can be applied in a variety of tasks, including natural language processing, machine translation, automatic content generation, etc., and has a wide range of applicability. This makes the application prospects of this technology bright in multiple industries, especially in areas that require high security.

[0059] This application trains the reward model and the cost model respectively by constructing a dialogue preference dataset, generates a first fine-tuning strategy and a second fine-tuning strategy through the trained reward model and cost model, and thus updates the large language model, thereby improving the balance between security and usefulness of the large language model.

[0060] In some feasible implementations, optionally, the training of the initial reward model and the initial cost model based on the dialogue preference dataset to obtain a trained reward model and a trained cost model includes:

[0061] Dividing the conversation preference data set according to content to obtain a first preference data set and a second preference data set, wherein the first preference data set includes a plurality of conversation data that focuses on the usefulness of answers, and the second preference data set includes a plurality of conversation data that focuses on the safety of answers;

[0062] Training the initial reward model based on the first preference data set to obtain a trained reward model;

[0063] The initial cost model is trained based on the second preference data set to obtain a trained cost model.

[0064] In this embodiment, optionally, the conversation preference data set can be divided manually or in other ways into a first preference data set and a second preference data set. Specifically, the multiple conversation data included in the first preference data set focus on the usefulness of the answer, and the multiple conversation data included in the second preference data set focus on the safety of the answer.

[0065] Among them, the first preference data set is the conversation preference data set Reflecting human preference for the usefulness of model output, the entire dataset D r Contains N dialogue prompts {x1, x2…, x N} and the corresponding response pair for each prompt {(y w,1 ,y l,1 ),(y w,2 ,y l,2 )…,(y w,N ,y l,N )}. Each response to (y w,i ,y l,i ) reflects the human expert's opinion based on the prompt x i Reply w,i Compared to y l,i More useful. For example, if the expert thinks that the response y w,i y l,iMore helpful or more in line with user needs, then during training, the model will learn to generate similar w,i The answer.

[0066] The second preference dataset is the conversation preference dataset reflects the human preference for the harmlessness of the model output response, and each response pair reflects the human expert's opinion on the response y w,i Compared to y l,i For greater security, each reply pair also introduces a safety signal identification pair (s w,i ,s l,i ), which is used to indicate whether the marked reply is safe or not. Such marking helps the model to better judge and avoid generating harmful replies through the cost model during the training process, thereby improving the safety of the conversation. An example of this dataset is shown in Table 1.

[0067]

[0068] Table 1 Example of cost model dialogue scenario training dataset

[0069] Optionally, updating the initial large language model according to the first fine-tuning strategy and / or the second fine-tuning strategy to obtain an updated large language model includes:

[0070] updating the initial large language model according to the first fine-tuning strategy to obtain an updated first large language model;

[0071] The initial large language model is updated according to the second fine-tuning strategy to obtain an updated second large language model.

[0072] In this embodiment, the initial large language model is updated by the first fine-tuning strategy and the second fine-tuning strategy respectively. It should be noted that the two do not interfere with each other and can be performed simultaneously or sequentially, which is not specifically limited in this embodiment. That is, after the large language model is adjusted by the first fine-tuning strategy and the second fine-tuning strategy at the same time, a large language model with better adjustment effect can be obtained.

[0073] In this embodiment, the training process of the reward model and the cost model is described. Specifically, in this application, the probabilistic Bradley-Terry (BT) model is used to model human usefulness preferences. By maximizing the observed preference probability (i.e., D r The log-likelihood of the probability difference between good answers and bad answers generated in the BT probability model is used to train a reward model R based on the BT probability model. φ , the loss function of this process is defined as follows:

[0074]

[0075] For harmless preference, the method of the present invention also trains a cost model C based on the BT probability model ψ , based on the harmlessness preference dataset D constructed in the above steps c , the loss function of this process includes a preference discrimination term and a security loss term L S , where the preference discrimination term is consistent with the goal of the reward model, aiming to maximize the log-likelihood of the observed preference probability. The goal of the safety loss term is to focus on pushing up the response scores that meet the safety signal and ignore those that do not meet the safety standards, further enhancing the cost model's ability to distinguish harmful and harmless responses. The loss function of this process is defined as follows:

[0076]

[0077]

[0078] The scores of the trained reward model and cost model follow a normal distribution and where μ r , μ c are the average scores of the reward model and the cost model on the human preference dataset, are the variances of the reward model and the cost model on the human preference dataset, respectively.

[0079] Optionally, updating the initial large language model according to the first fine-tuning strategy to obtain an updated first large language model includes:

[0080] Initializing parameters of the first fine-tuning strategy, and generating a first constraint strategy according to the trained reward model and the trained cost model;

[0081] Use the gradient ascent method to perform non-parametric space optimization and determine the optimal update strategy for the non-parametric space;

[0082] Projecting the optimal update strategy into a parameterized space, updating the optimal update strategy, and obtaining a target fine-tuning strategy;

[0083] The initial large language model is iteratively updated based on a target fine-tuning strategy to obtain the updated large language model.

[0084] In this example, the First Order Constrained Optimization in Policy Space (FOCOPS) is introduced to fine-tune the large language model. The core purpose of this step is to enhance the usefulness of the model while meeting security requirements, so that the model can not only generate high-quality responses but also ensure security. In this fine-tuning process, the reward model R trained in the above steps is used. φ and cost model C ψ Provides critical feedback for the policy optimization process. Reward Model R φ Evaluate the usefulness score of the output to ensure that the generated content is efficient and informative. ψ A harmlessness assessment is provided, and by scoring the output for safety, it is ensured that the model’s response is consistent with human safety preferences. The FOCOPS method ensures robustness and stability in high-dimensional continuous control tasks by finding the optimal update in the non-parametric policy space and then projecting it into the parameterized space. It uses the first-order information of the gradient to achieve efficient policy updates at a lower computational cost. This method establishes a constrained optimization framework in the policy space, so that the policy adjustment process not only focuses on the reward signal, but also incorporates safety requirements into the optimization constraints. Specifically, FOCOPS will gradually exclude unsafe outputs from the optimal policy based on the feedback from the cost model. The goal of FOCOPS optimization is to incorporate the harmlessness constraints provided by the cost model into the policy space while maximizing the reward model score, that is, to ensure that the policy π of LLMs θ (·|x) implements the following formula:

[0085]

[0086] The optimization objective after the introduction of the advantage function can be expressed as follows:

[0087]

[0088] This formula indicates that in reinforcement learning, by maximizing the policy advantage over reward to improve the overall return of the strategy while constraining the strategy to cost advantage The expectation does not exceed the limit C limit Among them, π θ (a t |s t )and Represent the current policy and the old policy respectively, and the ratio of the two is used for importance sampling adjustment to take into account the difference when the policy is updated. This optimization problem aims to balance the return and cost so that the policy can achieve high returns while maintaining reasonable limits on cost or risk. This ensures that the policy update strictly follows safety standards while improving the quality of the model output.

[0089] The specific process of using the FOCOPS method to update the strategy of a large language model is as follows: Initialize strategy parameters: First, initialize the strategy parameters to ensure that the strategy meets basic task requirements.

[0090] Non-parametric space optimization: In the non-parametric strategy space, the gradient ascent method is used to find the direction of the optimal strategy. The specific non-parametric optimal strategy form is as follows:

[0091]

[0092] The use of non-parametric space makes the optimization process more flexible and not limited to fixed parameter forms, which helps to explore a wide area of ​​the policy space and find the optimal potential policy direction.

[0093] Projection to parameterized space: After finding the optimal update strategy in the non-parametric space, FOCOPS projects it back to the parameterized strategy space, that is, minimizing the difference between the parameterized strategy and the non-parametric strategy. In the specific implementation, KL divergence is used to measure the difference between the parameterized strategy and the non-parametric strategy. By minimizing the parameterized strategy π θ The KL divergence between the non-parametric strategy π* and the parameterized strategy π that is closest to π* θ , the loss function is defined as follows:

[0094]

[0095] Replace π in the above formula * Substituting the above formula, the gradient update term of the loss function is as follows:

[0096]

[0097] Among them, the indicator function The role is to reject in the gradient update Too large a sample size ensures that the updated strategy is close to the original strategy.

[0098] Iterative update: Repeat the non-parametric spatial optimization and projection steps, and continuously adjust the policy parameters until the optimal policy is reached, meeting the dual goals of stability and high performance.

[0099] The existing Lagrangian-based iterative primal-dual method is prone to instability during the training process. The non-parametric policy space optimization mechanism of FOCOPS is introduced. Through the two-step method of "optimization first and then projection", the instability caused by direct optimization in the parameter space is avoided, especially when dealing with high-dimensional complex tasks.

[0100] Optionally, updating the initial large language model according to the second fine-tuning strategy to obtain an updated second large language model includes:

[0101] Updating the initial large language model according to the second fine-tuning strategy;

[0102] The updating of the initial large language model according to the second fine-tuning strategy includes:

[0103] Initializing parameters of the second fine-tuning strategy;

[0104] Calculating a first loss corresponding to the trained reward model, and calculating a second loss corresponding to the trained cost model;

[0105] Setting a dynamic adjustment factor, where the dynamic adjustment factor is used to ensure that the large model always maintains the ability to suppress harmful output when generating content;

[0106] Updating the parameters of the initial large language model by a gradient descent method to obtain target parameters;

[0107] The initial large language model is iteratively updated based on the target parameter to obtain the updated large language model.

[0108] In this embodiment, a proximal optimization strategy (Penalized Proximal Policy Optimization, P3O) based on the ReLU exact penalty function is introduced to dynamically adjust the security constraints during the optimization process to further ensure that the model does not ignore the security constraints in the process of optimizing usefulness and achieve the optimal solution under the constraints. The training goal of this stage is consistent with the training goal in the above method. The specific process of using the P3O method to update the LLMs policy is as follows:

[0109] Initialize model parameters: Similar to FOCOPS, first initialize the model parameters to ensure the feasibility of the initial state.

[0110] Calculate reward and cost loss: Based on the current strategy, calculate the reward loss separately and cost loss The reward loss is used to quantify the usefulness of the model output, and the cost loss is used to assess the potential risk of the content generated by the model.

[0111]

[0112] Dynamic adjustment of factors: According to the current cost loss, the penalty factor κ is dynamically adjusted to ensure that the model always maintains the ability to suppress harmful output when generating content. When the cost loss exceeds the preset threshold, the value of the penalty factor is appropriately increased, and the adaptive adjustment mechanism is used to strengthen the suppression of harmful content generation, thereby improving the security of the model.

[0113] Gradient update: By introducing the composite loss function L P3O , which can more accurately capture the deviation of model behavior. The model parameters are updated by gradient descent to minimize the overall loss function, which is specifically expressed as:

[0114]

[0115] Iterative training: After completing the parameter update, repeat the above steps until the model reaches a dynamic balance between rewards and costs. This process uses adaptive learning rate and momentum strategy to accelerate convergence and improve the robustness of the model, thereby achieving the goal of being both practical and not producing harmful outputs. Through this mechanism, the model will be gradually optimized to ensure stability and effectiveness under multi-level constraints.

[0116] Traditional algorithms are sensitive to the selection of initial hyperparameters, which affects the optimization efficiency. P3O introduces an adaptive dynamic penalty mechanism, which dynamically adjusts the penalty factor κ to achieve faster strategy optimization, further improving the adaptability of the model in different tasks and environments.

[0117] Optionally, after the initial large language model is updated according to the first fine-tuning strategy and the second fine-tuning strategy to obtain an updated large language model, the method further includes:

[0118] Setting the trained reward model to obey a first normal distribution, and setting the trained cost model to obey a second normal distribution;

[0119] Scoring the updated large language model based on the first normal distribution and the second normal distribution to obtain a target score value;

[0120] When the target score value is lower than a preset threshold, the updated large language model is iteratively updated again.

[0121] In this embodiment, the present application designs a performance evaluation experiment for the usefulness and harmlessness of a large language model, and conducts multi-task tests on multiple test benchmarks (such as Blimp, CoQA, and TruthfulQA). The results show that the method of the present invention performs well, especially in reducing the generation of harmful content, achieving higher safety standards, and greatly improving the usefulness compared to the benchmark algorithm. Specifically, the combination of P3O and FOCOPS enables the model to effectively reduce harmful outputs while maintaining the high quality of the output content. Compared with the existing PPO-Lag method, the optimization strategy of the present invention exhibits higher stability and efficiency, especially in high-dimensional complex tasks, achieving significant performance improvements, as shown in Table 2.

[0122]

[0123] In addition, the output content of the large language model before and after alignment is quantitatively evaluated based on the reward model and cost model trained in the above steps. The specific evaluation scheme is as follows: the output scores of the reward model and cost model after training on the human preference dataset are set to follow the first normal distribution and the second normal distribution, that is, and where μ r , μ c are the average scores of the reward model and the cost model on the human preference dataset, are the variances of the reward model and the cost model on the human preference dataset. The scoring results are normalized using the Z-score, i.e. and This normalization process maps the original scores to the standard normal distribution N(0,1), making the scoring results of different models comparable.

[0124] The standardized scoring interval division uses σ (standard deviation) as the benchmark, and the specific division is as follows:

[0125] Responses above 2σ (Z>2.0) were considered excellent performance, above the 97.7% percentile, indicating extremely good performance in terms of usefulness or safety;

[0126] 1σ to 2σ (1.0≤Z<2.0) indicates excellent performance, which is located at the 84.1%-97.7% quantile, reflecting significant positive characteristics;

[0127] Between 0 and 1σ (0≤Z<1.0) indicates good, located at the 50%-84.1% quantile, with a moderate effect;

[0128] -1σ to 0 (-1.0≤Z<0) indicates average, located at the 15.9%-50% percentile, and the effect needs to be improved;

[0129] A value lower than -1σ (Z<-1.0) indicates poor performance, which is below the 15.9% percentile and requires significant improvement.

[0130] This quantitative method based on statistical distribution not only provides a standardized measurement benchmark for the evaluation of model output, but also can calculate the exact percentile corresponding to any score through the cumulative distribution function, that is, This evaluation scheme is suitable for quantitatively evaluating the performance of the model in both usefulness and security dimensions, providing reliable quantitative indicators for model optimization.

[0131] Through the above process and design, the large model security alignment method based on dynamic constrained reinforcement learning of this application can ensure that large language models can significantly reduce potential security risks while improving their usefulness, thereby providing an efficient solution for building a more reliable and secure large language model.

[0132] This application trains the reward model and the cost model respectively by constructing a dialogue preference dataset, generates a first fine-tuning strategy and a second fine-tuning strategy through the trained reward model and cost model, and thus updates the large language model, thereby improving the balance between security and usefulness of the large language model.

[0133] See also Figure 2 , Figure 2 is a structural diagram of a large model security alignment device based on dynamic constraint reinforcement learning provided in an embodiment of the present application. Figure 2 As shown, the large model security alignment device 200 based on dynamic constraint reinforcement learning includes:

[0134] A construction module 210 is used to construct a dialogue preference data set, wherein the dialogue preference data set includes a plurality of dialogue data, each of which includes a prompt question and an answer;

[0135] A training module 220, configured to train the initial reward model and the initial cost model based on the conversation preference dataset to obtain a trained reward model and a trained cost model;

[0136] A generation module 230, configured to generate a first fine-tuning strategy based on the trained reward model and the trained cost model, and to generate a second fine-tuning strategy based on the trained reward model and the trained cost model, wherein the first fine-tuning strategy is used to improve the response quality and safety of the large model, and the second fine-tuning strategy is used to dynamically adjust the safety constraints of the large model;

[0137] The updating module 240 is configured to update the initial large language model according to the first fine-tuning strategy and / or the second fine-tuning strategy to obtain an updated large language model.

[0138] Optionally, the training module 220 includes:

[0139] a division submodule, configured to divide the conversation preference data set according to content to obtain a first preference data set and a second preference data set, wherein the first preference data set includes a plurality of conversation data that focuses on the usefulness of answers, and the second preference data set includes a plurality of conversation data that focuses on the safety of answers;

[0140] A first training submodule, configured to train the initial reward model based on the first preference data set to obtain a trained reward model;

[0141] The second training submodule is used to train the initial cost model based on the second preference data set to obtain a trained cost model.

[0142] Optionally, the updating module 240 includes:

[0143] A first updating submodule, configured to update the initial large language model according to the first fine-tuning strategy to obtain an updated first large language model;

[0144] The second updating submodule is used to update the initial large language model according to the second fine-tuning strategy to obtain an updated second large language model.

[0145] Optionally, the first updating submodule includes:

[0146] A generating unit, configured to initialize parameters of the first fine-tuning strategy and to generate a first constraint strategy according to the trained reward model and the trained cost model;

[0147] A determination unit, used for performing non-parametric space optimization using a gradient ascent method to determine an optimal update strategy for the non-parametric space;

[0148] A projection unit, used to project the optimal update strategy into a parameterized space, update the optimal update strategy, and obtain a target fine-tuning strategy;

[0149] The first iterative unit is used to iteratively update the initial large language model based on a target fine-tuning strategy to obtain the updated large language model.

[0150] Optionally, the second updating submodule includes:

[0151] An initialization unit, used to initialize the parameters of the second fine-tuning strategy;

[0152] A calculation unit, used to calculate a first loss corresponding to the trained reward model, and to calculate a second loss corresponding to the trained cost model;

[0153] A setting unit, used to set a dynamic adjustment factor, wherein the dynamic adjustment factor is used to ensure that the large model always maintains the ability to suppress harmful output when generating content;

[0154] An updating unit, used for updating the parameters of the initial large language model by a gradient descent method to obtain target parameters;

[0155] The second iterative unit is used to iteratively update the initial large language model based on the target parameter to obtain the updated large language model.

[0156] Optionally, also include:

[0157] A setting module, used to set the trained reward model to obey a first normal distribution, and to set the trained cost model to obey a second normal distribution;

[0158] a scoring module, configured to score the updated large language model based on the first normal distribution and the second normal distribution to obtain a target scoring value;

[0159] The iteration module is used to iteratively update the updated large language model again when the target score value is lower than a preset threshold.

[0160] This application trains the reward model and the cost model respectively by constructing a dialogue preference dataset, generates a first fine-tuning strategy and a second fine-tuning strategy through the trained reward model and cost model, and thus updates the large language model, thereby improving the balance between security and usefulness of the large language model.

[0161] The present application also provides an electronic device. Figure 3 , the electronic device may include a processor 301, a memory 302, and a program 3021 stored in the memory 302 and executable on the processor 301.

[0162] When the program 3021 is executed by the processor 301, it can achieve Figure 1 Any step in the corresponding method embodiment:

[0163] Constructing a dialogue preference data set, wherein the dialogue preference data set includes a plurality of dialogue data, each of the dialogue data includes a prompt question and an answer;

[0164] Based on the dialogue preference dataset, the initial reward model and the initial cost model are trained respectively to obtain a trained reward model and a trained cost model;

[0165] Generate a first fine-tuning strategy based on the trained reward model and the trained cost model, and generate a second fine-tuning strategy based on the trained reward model and the trained cost model, wherein the first fine-tuning strategy is used to improve the response quality and safety of the large model, and the second fine-tuning strategy is used to dynamically adjust the safety constraints of the large model;

[0166] The initial large language model is updated according to the first fine-tuning strategy and / or the second fine-tuning strategy to obtain an updated large language model.

[0167] Optionally, the training of the initial reward model and the initial cost model based on the dialogue preference dataset to obtain a trained reward model and a trained cost model includes:

[0168] Dividing the conversation preference data set according to content to obtain a first preference data set and a second preference data set, wherein the first preference data set includes a plurality of conversation data that focuses on the usefulness of answers, and the second preference data set includes a plurality of conversation data that focuses on the safety of answers;

[0169] Training the initial reward model based on the first preference data set to obtain a trained reward model;

[0170] The initial cost model is trained based on the second preference data set to obtain a trained cost model.

[0171] Optionally, updating the initial large language model according to the first fine-tuning strategy and / or the second fine-tuning strategy to obtain an updated large language model includes:

[0172] updating the initial large language model according to the first fine-tuning strategy to obtain an updated first large language model;

[0173] The initial large language model is updated according to the second fine-tuning strategy to obtain an updated second large language model.

[0174] Optionally, updating the initial large language model according to the first fine-tuning strategy to obtain an updated first large language model includes:

[0175] Initializing parameters of the first fine-tuning strategy, and generating a first constraint strategy according to the trained reward model and the trained cost model;

[0176] Use the gradient ascent method to perform non-parametric space optimization and determine the optimal update strategy for the non-parametric space;

[0177] Projecting the optimal update strategy into a parameterized space, updating the optimal update strategy, and obtaining a target fine-tuning strategy;

[0178] The initial large language model is iteratively updated based on a target fine-tuning strategy to obtain the updated large language model.

[0179] Optionally, updating the initial large language model according to the second fine-tuning strategy to obtain an updated second large language model includes:

[0180] Initializing parameters of the second fine-tuning strategy;

[0181] Calculating a first loss corresponding to the trained reward model, and calculating a second loss corresponding to the trained cost model;

[0182] Setting a dynamic adjustment factor, where the dynamic adjustment factor is used to ensure that the large model always maintains the ability to suppress harmful output when generating content;

[0183] Updating the parameters of the initial large language model by a gradient descent method to obtain target parameters;

[0184] The initial large language model is iteratively updated based on the target parameter to obtain the updated large language model.

[0185] Optionally, after the initial large language model is updated according to the first fine-tuning strategy and the second fine-tuning strategy to obtain an updated large language model, the method further includes:

[0186] Setting the trained reward model to obey a first normal distribution, and setting the trained cost model to obey a second normal distribution;

[0187] Scoring the updated large language model based on the first normal distribution and the second normal distribution to obtain a target score value;

[0188] When the target score value is lower than a preset threshold, the updated large language model is iteratively updated again.

[0189] This application trains the reward model and the cost model respectively by constructing a dialogue preference dataset, generates a first fine-tuning strategy and a second fine-tuning strategy through the trained reward model and cost model, and thus updates the large language model, thereby improving the balance between security and usefulness of the large language model.

[0190] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned large model security alignment method embodiment based on dynamic constraint reinforcement learning is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. Among them, the computer-readable storage medium is such as a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0191] An embodiment of the present application further provides a computer program product, which is stored in a storage medium. The computer program product is executed by at least one processor to implement the various processes of the above-mentioned large model security alignment method embodiment based on dynamic constraint reinforcement learning, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0192] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0193] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0194] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A large model security alignment method based on dynamic constraint reinforcement learning, characterized in that: The method comprises: Constructing a dialogue preference data set, wherein the dialogue preference data set includes a plurality of dialogue data, each of the dialogue data includes a prompt question and an answer; Based on the dialogue preference dataset, the initial reward model and the initial cost model are trained respectively to obtain a trained reward model and a trained cost model; Generate a first fine-tuning strategy based on the trained reward model and the trained cost model, and generate a second fine-tuning strategy based on the trained reward model and the trained cost model, wherein the first fine-tuning strategy is used to improve the response quality and safety of the large model, and the second fine-tuning strategy is used to dynamically adjust the safety constraints of the large model; The initial large language model is updated according to the first fine-tuning strategy and / or the second fine-tuning strategy to obtain an updated large language model.

2. The method according to claim 1, characterized in that The step of training the initial reward model and the initial cost model based on the dialogue preference dataset to obtain a trained reward model and a trained cost model includes: Dividing the conversation preference data set according to content to obtain a first preference data set and a second preference data set, wherein the first preference data set includes a plurality of conversation data that focuses on the usefulness of answers, and the second preference data set includes a plurality of conversation data that focuses on the safety of answers; Training the initial reward model based on the first preference data set to obtain a trained reward model; The initial cost model is trained based on the second preference data set to obtain a trained cost model.

3. The method according to claim 1, characterized in that The updating of the initial large language model according to the first fine-tuning strategy and / or the second fine-tuning strategy to obtain an updated large language model includes: updating the initial large language model according to the first fine-tuning strategy to obtain an updated first large language model; The initial large language model is updated according to the second fine-tuning strategy to obtain an updated second large language model.

4. The method according to claim 3, characterized in that: The updating of the initial large language model according to the first fine-tuning strategy to obtain an updated first large language model includes: Initializing parameters of the first fine-tuning strategy, and generating a first constraint strategy according to the trained reward model and the trained cost model; Use the gradient ascent method to perform non-parametric space optimization and determine the optimal update strategy for the non-parametric space; Projecting the optimal update strategy into a parameterized space, updating the optimal update strategy, and obtaining a target fine-tuning strategy; The initial large language model is iteratively updated based on a target fine-tuning strategy to obtain the updated large language model.

5. The method according to claim 3, characterized in that: The updating of the initial large language model according to the second fine-tuning strategy to obtain an updated second large language model includes: Initializing parameters of the second fine-tuning strategy; Calculating a first loss corresponding to the trained reward model, and calculating a second loss corresponding to the trained cost model; Setting a dynamic adjustment factor, where the dynamic adjustment factor is used to ensure that the large model always maintains the ability to suppress harmful output when generating content; Updating the parameters of the initial large language model by a gradient descent method to obtain target parameters; The initial large language model is iteratively updated based on the target parameter to obtain the updated large language model.

6. The method according to claim 1, characterized in that After the initial large language model is updated according to the first fine-tuning strategy and the second fine-tuning strategy to obtain an updated large language model, the method further includes: Setting the trained reward model to obey a first normal distribution, and setting the trained cost model to obey a second normal distribution; Scoring the updated large language model based on the first normal distribution and the second normal distribution to obtain a target score value; When the target score value is lower than a preset threshold, the updated large language model is iteratively updated again.

7. A large model security alignment device based on dynamic constraint reinforcement learning, characterized in that: The device comprises: A construction module, used to construct a dialogue preference data set, wherein the dialogue preference data set includes a plurality of dialogue data, each of which includes a prompt question and an answer; A training module, used to train the initial reward model and the initial cost model based on the dialogue preference dataset to obtain a trained reward model and a trained cost model; A generation module, configured to generate a first fine-tuning strategy based on the trained reward model and the trained cost model, and to generate a second fine-tuning strategy based on the trained reward model and the trained cost model, wherein the first fine-tuning strategy is used to improve the response quality and safety of the large model, and the second fine-tuning strategy is used to dynamically adjust the safety constraints of the large model; An updating module is used to update the initial large language model according to the first fine-tuning strategy and / or the second fine-tuning strategy to obtain an updated large language model.

8. An electronic device, comprising: A memory, a processor, and a program stored in the memory and executable on the processor; characterized in that the processor is used to read the program in the memory to implement the steps in the large model security alignment method based on dynamic constraint reinforcement learning as described in any one of claims 1 to 6.

9. A readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps in the large model security alignment method based on dynamic constraint reinforcement learning as described in any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product is stored in a storage medium, and the computer program product is executed by at least one processor to implement the steps in the large model security alignment method based on dynamic constraint reinforcement learning as described in any one of claims 1 to 6.

Images