A large model security reinforcement method and system of category adaptive reinforcement learning

Abstract

The application discloses a large model security reinforcement method and system of category adaptive reinforcement learning, and the method comprises the following steps: dividing safety risk data obtained from a large model security evaluation system into multiple safety categories, generating corresponding safety responses through an auxiliary large language model, and constructing training data sets of the safety categories; constructing a loss function by comparing original large model answers with safety responses in the training data sets of the safety categories, training a corresponding reward model by minimizing the loss function; obtaining an advantage function based on the reward model corresponding to each safety category and a reward scaling factor thereof; constructing an overall objective function based on the advantage function, and alternately and iteratively optimizing the objective functions of the safety categories in the large model training process, so that the strategy model can simultaneously learn and align safety preferences of multiple safety categories, and finally obtain a large model optimized by multiple safety categories. The application enhances the safety and robustness of the model.

Description

Technical Field

[0001] This application relates to the field of reinforcement learning technology, and in particular to a method and system for security hardening of large models using category-adaptive reinforcement learning. Background Technology

[0002] With the rapid development and widespread application of large language models (LLMs), their output may contain false information, biases, and privacy breaches, posing serious security risks. Therefore, security hardening of large models has become a crucial aspect of research and application. Its core objective is to effectively resist potential attacks and abuse risks during model operation through a series of security enhancement measures, while ensuring data privacy, model stability, and prediction accuracy. Among existing technologies, reinforcement learning is widely used for security hardening of large models because it can establish a closed-loop optimization mechanism between model generation and external feedback. Compared with methods relying solely on supervised fine-tuning, reinforcement learning-based optimization methods show more significant advantages in constraining model behavior and improving multi-dimensional security performance.

[0003] Currently, large-scale model security optimization methods based on reinforcement learning mainly fall into two categories: reward-based optimization methods (such as PPO) and reward-free optimization methods (such as DPO). In contrast, reward-based methods score candidate responses using reward models and maximize expected rewards through policy optimization, typically providing stronger alignment capabilities and security constraint effects. However, existing methods still face two challenges in practical applications: First, the sources of reward signals are limited, usually relying on a single global reward model, making it difficult to comprehensively cover complex and diverse security scenarios, and easily leading to the model generating responses that are "superficially compliant" but still risky. Second, different security categories have significant differences in data scale and learning difficulty, while existing methods mostly adopt simple parallel training strategies, lacking adaptive processing of category differences. This causes the model to overfit on easily learned categories and underlearn on difficult categories or long-tailed samples, ultimately resulting in an unbalanced overall security alignment effect. Summary of the Invention

[0004] To address the issues of sparsity of security signals and optimization imbalance in large models during security hardening reinforcement learning, this application provides a method and system for security hardening of large models using category adaptive reinforcement learning. This method can fully utilize the reward signals of each security category and adaptively balance the contributions of each category in policy updates, thereby significantly improving the multidimensional security alignment effect of large models and enhancing the security and robustness of the models.

[0005] This application discloses a method for security hardening of large models using category-adaptive reinforcement learning, which includes:

[0006] Step 1: Divide the security risk data obtained from the large model security evaluation system into multiple security categories. For each security category of harmful instructions, generate the corresponding security response through an auxiliary large language model, thereby constructing the training dataset for each security category.

[0007] Step 2: For each security category, construct a loss function by comparing the original large model's answer with the security responses in the training dataset for each security category, and train the corresponding reward model by minimizing this loss function;

[0008] Step 3: Based on the reward model and reward scaling factor corresponding to each security category, obtain the advantage function;

[0009] Step 4: Construct an overall objective function based on the advantage function. During the training of the large model, iteratively optimize the objective functions of each security category, so that the policy model can learn and align the security preferences of multiple security categories at the same time, and finally obtain a large model jointly optimized by multiple security categories.

[0010] Further, step 1 includes:

[0011] Step 11: Obtain automatically detected and labeled security risk data from the large-scale model security assessment system, and classify it into... One security category;

[0012] Step 12: For the first Each security category has a set of security features. A set of harmful instructions, denoted as , For the first The security category corresponding to the first A set of harmful instructions, representing all security categories of harmful instructions, is uniformly represented as: Harmful instructions are text instructions that can induce large models to generate unsafe or illegal content;

[0013] Step 13: Targeting The auxiliary generator GPT-4o is used to generate a security response. ;in, In order to target the The security category corresponding to the first Security response to a malicious instruction This indicates a structured reasoning answer. For the first Step-by-step structured reasoning answer, The final answer; the final answer refers to the final output conclusion generated by the logical synthesis of the GPT-4o answer based on structured reasoning.

[0014] Step 14: Based on harmful instructions and its corresponding security response The training data for each security category was organized into , For the first Training data for each security category; summarizing the training data for all security categories into... , This is the training dataset for all security categories.

[0015] Further, step 13 includes:

[0016] Will As input prompts, a thought chain prompt template is constructed and input into the large model GPT-4o to generate structured reasoning answers and the final answer; whereby the structured reasoning answer refers to the GPT-4o thinking... Whether the reasoning process is safe, each structured reasoning answer uses... and The tags are used for annotation, and the final answer is encapsulated in... and Between tags; The starting label for the reasoning step. This is the end label for the reasoning step. The starting tag for the final answer, This is the closing tag for the final answer.

[0017] Further, step 2 includes:

[0018] For each security category, a contrastive learning-based ranking method is employed to maximize the security response by minimizing the loss function. Compared with the original response By gradually optimizing the reward model parameters based on the score difference between them. This is to enable the reward model to distinguish between safe and insecure responses under the same query, where, In order to target the The security category corresponding to the first The original response to a harmful instruction. For the first The reward model parameters corresponding to each security category;

[0019] Update reward model parameters using gradient descent method :

[0020]

[0021] in, For the first The reward update step size for each security category For the first The loss function corresponding to each security category For the loss function to the reward model parameters The gradient;

[0022] The process continues until the loss function converges or the set number of training epochs is reached, yielding the [number of epochs]. Final reward model for each security category ;

[0023] After the reward models for all safety categories have been trained, a reward model set is formed. , It can provide multi-security-category security score signals for the subsequent policy optimization stage, realize differentiated modeling and signal constraints for different security categories, and thus support the policy model in the adaptive optimization of security categories in reinforcement learning.

[0024] Furthermore, the process of obtaining the loss function includes:

[0025] For each security category, in each iteration, from the training dataset... Training data for each security category Harmful Sampling Commands and Their Reference Security Responses And utilize the generative strategy of the original large model. Generate the original response Soon Input the original large model to obtain its original answer;

[0026] According to the security response and the original response Construct the loss function.

[0027] Furthermore, the loss function is obtained through the following formula:

[0028]

[0029] in, For loss function, For the first The number of data points for each security category For the Sigmoid function, This is the reward score for a safe response under the current reward policy. Given the reward score of the original response under the current reward policy, the Sigmoid function guarantees that on the [number]th [day]... Under each safety category, the reward model tends to output higher scores for safe responses and lower scores for unsafe responses, in order to establish a ranking preference for safety categories.

[0030] Further, step 3 includes:

[0031] Using reward models and reward scaling factor Calculate the advantage function of the population relative policy optimization GRPO; in each iteration of GRPO, from Sample a batch of harmful instructions ,in, For from the first The sampled from the set of all harmful instructions in the security category is the first... A collection of harmful instructions in a batch; for each harmful instruction in the current batch Based on the current large model generation strategy generate A set of candidate responses ,in, For the current large model generation strategy, the first The security category corresponding to the first Candidate responses to malicious instructions, calculate malicious instructions. Corresponding reward scaling factor ;in accordance with The set of reward scores for each candidate response , To calculate the advantage function for the reward model parameters:

[0032]

[0033]

[0034]

[0035] in, For the first Advantage functions corresponding to each security category; Indicates harmful instructions Corresponding candidate response Reward points; Based on harmful instructions The mean reward of the current batch of samples is used to standardize the reward distribution; Based on harmful instructions The standard deviation of the reward for the current batch of samples is used to normalize reward volatility. The standard deviation of the reward includes a stable term. ; As an advantage estimate, it is used in the GRPO shearing target to achieve policy optimization based on safety category difficulty adaptation.

[0036] Furthermore, the process of obtaining the reward scaling factor includes:

[0037] For the Harmful instructions in each security category and its corresponding security response Q&A Generate the original response using the original large model strategy. Construct a paired response dataset:

[0038]

[0039] in, For the first The paired response dataset corresponding to each security category Harmful instructions Paired security response and the original response gather;

[0040] Using the first Reward model for each security category Calculate the reward gap :

[0041]

[0042] in, For the first The first security category The reward gap between the security response and the original response for each harmful instruction. Reward points for a safe response. The reward score for the original response;

[0043] The reward gap can be calculated using the following formula. After processing, the average reward gap estimate is obtained:

[0044]

[0045] in, Estimate the average reward gap;

[0046] The reward scaling factor is obtained using the following formula. :

[0047]

[0048] in, This is the initial default scaling factor, used to set the baseline level for reward scaling; Used to control the weight ratio between static and dynamic components; It is an adjustable hyperparameter, which can be adjusted. Sensitivity to the difficulty of security categories; limitations This is to avoid excessive coefficient updates.

[0049] Further, step 4 includes:

[0050] For the For each safety category, the large model generation strategy is iteratively updated by optimizing the following objective function:

[0051]

[0052]

[0053] in, For the first Objective function for each security category The learnable parameters for generating policy networks in current large reinforcement learning models. This indicates the generated answer under the current large model generation strategy. The probability, This indicates the answers generated under the large model generation strategy before each iteration. The probability, For the large model generation strategy prior to this iteration, targeting the first... The security category corresponding to the first Candidate responses for a malicious instruction, The output probability is the reference policy output of the original large model. for and KL divergence between the two large model generation strategies For harmful issues The corresponding reward scaling factor; This is a truncation function to prevent the reinforcement learning policy from updating too much and to avoid the model training from becoming too divergent.

[0054] During training, the objective function for each security category is iteratively optimized alternately, and the reward model for different security categories is applied in turn. and reward scaling factor GRPO is executed to progressively integrate the reward gradient information output by the reward models of each security category into a unified system. In this process, by iterating alternately across security categories, the policy model learns and aligns with multiple security categories simultaneously, outputting a large security model optimized by multiple security categories.

[0055] This application also discloses a large model security hardening system based on category adaptive reinforcement learning, which implements the above-described method and includes:

[0056] The training dataset construction module is used to divide the security risk data obtained from the large model security evaluation system into multiple security categories. For each security category of harmful instructions, the module generates corresponding security responses through an auxiliary large language model, thereby constructing the training dataset for each security category.

[0057] The reward model training module is used to construct a loss function for each security category by comparing the answers of the original large model with the security responses in the training dataset of each security category, and to train the corresponding reward model by minimizing the loss function.

[0058] The strategy optimization module is used to obtain the advantage function based on the reward model and reward scaling factor corresponding to each security category. The overall objective function is constructed based on the advantage function. During the training of the large model, the objective function of each security category is alternately and iteratively optimized, so that the strategy model can learn and align the security preferences of multiple security categories at the same time, and finally obtain the large model jointly optimized by multiple security categories.

[0059] This application addresses the reward-based reinforcement learning problem in large-scale model security hardening scenarios, proposing a method and system for large-scale model security hardening based on category-adaptive reinforcement learning. This method integrates thought chain technology, a fine-grained categorical reward modeling mechanism, and a dynamic reward scaling optimization strategy. While ensuring the consistency and robustness of multi-class security alignment, it effectively alleviates the instability problems caused by training signal conflicts between different classes and extreme values ​​in reward distribution. Through group relative policy optimization with alternating class iterations, this method can simultaneously absorb multi-class security constraint signals in a unified large model, thereby achieving a more refined, stable, and scalable security hardening effect. Compared with existing technologies, this invention has the following beneficial effects and advantages:

[0060] 1. This invention is based on publicly available secure data and a thought chain reasoning generation method to automatically construct high-quality categorized secure training samples, avoiding reliance on large-scale manual preference labeling, significantly reducing data construction costs, and ensuring the structure and consistency of the data.

[0061] 2. For different categories of potential hazard scenarios, this invention trains a categorized reward model to form a multi-category reward function set. This mechanism enables more refined modeling and constraints on complex and diverse security risks, improving the model's adaptability and robustness in multiple scenarios;

[0062] 3. Introducing a dynamic scaling factor driven by intra-class reward differences during reinforcement learning effectively alleviates the training instability caused by extreme values ​​in reward distribution and imbalance between class samples, achieving a more balanced and comprehensive safety alignment effect;

[0063] 4. This security hardening system combines the GRPO framework, uses the relative ranking signal of multiple candidate responses to stabilize policy updates, and estimates the advantage function through the relative reward within the group. It does not require storing additional value network parameters, can make more efficient use of hardware resources, and saves memory usage. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0065] Figure 1 This is a flowchart illustrating a method for security hardening of large models using category-adaptive reinforcement learning, as described in an embodiment of this application.

[0066] Figure 2 This is a block diagram of a large model security hardening system for adaptive reinforcement learning according to an embodiment of this application. Detailed Implementation

[0067] The present application will be further described in conjunction with the accompanying drawings and embodiments. The described embodiments are only some, not all, of the embodiments of the present application. All other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of the present application.

[0068] In the field of large-scale model security hardening, reward-based reinforcement learning is currently the mainstream approach. Its core mechanism involves transforming "security compliance" into an optimizable signal through a reward model, and then using policy optimization to drive iterative model improvement. This method uses a language model as the policy network, and the candidate responses generated are scored for security by the reward model. The policy gradient algorithm maximizes long-term rewards to update model parameters, allowing the model to gradually converge to a better security policy over multiple rounds of interaction. This mechanism essentially achieves "security-oriented self-reinforcement," enabling large models to form a more robust security alignment during iterative training.

[0069] Building upon this foundation, the proposed solution introduces a category-adaptive mechanism. Through multi-security category reward learning and a dynamic reward scaling strategy, it achieves more refined and balanced security optimization. This method adaptively adjusts the reward signal based on the learning difficulty and sample distribution of different security categories. While ensuring overall training stability, it enables the model to improve in a balanced manner across various security tasks, thereby significantly enhancing the overall security alignment performance and generalization ability of large models.

[0070] See Figure 1This application provides an embodiment of a class-adaptive reinforcement learning method for security hardening of large models, comprising:

[0071] Step 1: Divide the security risk data obtained from the large model security evaluation system into multiple security categories. For each security category of harmful instructions, generate the corresponding security response through an auxiliary large language model, thereby constructing the training dataset for each security category.

[0072] Step 2: For each security category, construct a loss function by comparing the original large model's answer with the security responses in the training dataset for each security category, and train the corresponding reward model by minimizing this loss function;

[0073] Step 3: Based on the reward model and reward scaling factor corresponding to each security category, obtain the advantage function;

[0074] Step 4: Construct an overall objective function based on the advantage function. During the training of the large model, iteratively optimize the objective functions of each security category, so that the policy model can learn and align the security preferences of multiple security categories at the same time, and finally obtain a large model jointly optimized by multiple security categories.

[0075] Optionally, step 1 includes:

[0076] Step 11: Obtain automatically detected and labeled security risk data from the large-scale model security assessment system, and classify it into... One security category;

[0077] Step 12: For the first Each security category has a set of security features. A set of harmful instructions, denoted as , For the first The security category corresponding to the first A set of harmful instructions, representing all security categories of harmful instructions, is uniformly represented as: Harmful instructions are text instructions that can induce large models to generate unsafe or illegal content;

[0078] Step 13: Targeting The auxiliary generator GPT-4o is used to generate a security response. ;in, In order to target the The security category corresponding to the first Security response to a malicious instruction This indicates a structured reasoning answer. For the first Step-by-step structured reasoning answer, The final answer; the final answer refers to the final output conclusion generated by the logical synthesis of the GPT-4o answer based on structured reasoning.

[0079] Step 14: Based on harmful instructions and its corresponding security response The training data for each security category was organized into , For the first Training data for each security category; summarizing the training data for all security categories into... , This is the training dataset for all security categories.

[0080] Optionally, step 13 includes:

[0081] Will As input prompts, a thought chain prompt template is constructed and input into the large model GPT-4o to generate structured reasoning answers and the final answer; whereby the structured reasoning answer refers to the GPT-4o thinking... Whether the reasoning process is safe, each structured reasoning answer uses... and The tags are used for annotation, and the final answer is encapsulated in... and Between tags; The starting label for the reasoning step. This is the end label for the reasoning step. The starting tag for the final answer, This is the closing tag for the final answer.

[0082] Optionally, step 2 includes:

[0083] For each security category, a contrastive learning-based ranking method is employed to maximize the security response by minimizing the loss function. Compared with the original response By gradually optimizing the reward model parameters based on the score difference between them. This is to enable the reward model to distinguish between safe and insecure responses under the same query, where, In order to target the The security category corresponding to the first The original response to a harmful instruction. For the first The reward model parameters corresponding to each security category;

[0084] Update reward model parameters using gradient descent method :

[0085]

[0086] in, For the first The reward update step size for each security category For the first The loss function corresponding to each security category For the loss function to the reward model parameters The gradient;

[0087] The process continues until the loss function converges or the set number of training epochs is reached, yielding the [number of epochs]. Final reward model for each security category ;

[0088] After the reward models for all safety categories have been trained, a reward model set is formed. , It can provide multi-security-category security score signals for the subsequent policy optimization stage, realize differentiated modeling and signal constraints for different security categories, and thus support the policy model in the adaptive optimization of security categories in reinforcement learning.

[0089] Optionally, the process of obtaining the loss function includes:

[0090] For each security category, in each iteration, from the training dataset... Training data for each security category Harmful Sampling Commands and Their Reference Security Responses And utilize the generative strategy of the original large model. Generate the original response Soon Input the original large model to obtain its original answer;

[0091] According to the security response and the original response Construct the loss function.

[0092] Alternatively, the loss function can be obtained using the following formula:

[0093]

[0094] in, For loss function, For the first The number of data points for each security category For the Sigmoid function, This is the reward score for a safe response under the current reward policy. Given the reward score of the original response under the current reward policy, the Sigmoid function guarantees that on the [number]th [day]... Under each safety category, the reward model tends to output higher scores for safe responses and lower scores for unsafe responses, in order to establish a ranking preference for safety categories.

[0095] Optionally, step 3 includes:

[0096] Using reward models and reward scaling factor Calculate the advantage function of the population relative policy optimization GRPO; in each iteration of GRPO, from Sample a batch of harmful instructions ,in, For from the first The sampled from the set of all harmful instructions in the security category is the first... A collection of harmful instructions in a batch; for each harmful instruction in the current batch Based on the current large model generation strategy generate A set of candidate responses ,in, For the current large model generation strategy, the first The security category corresponding to the first Candidate responses to malicious instructions, calculate malicious instructions. Corresponding reward scaling factor ;in accordance with The set of reward scores for each candidate response , To calculate the advantage function for the reward model parameters:

[0097]

[0098]

[0099]

[0100] in, For the first Advantage functions corresponding to each security category; Indicates harmful instructions Corresponding candidate response Reward points; Based on harmful instructions The mean reward of the current batch of samples is used to standardize the reward distribution; Based on harmful instructions The standard deviation of the reward for the current batch of samples is used to normalize reward volatility. The standard deviation of the reward includes a stable term. ; As an advantage estimate, it is used in the GRPO shearing target to achieve policy optimization based on safety category difficulty adaptation.

[0101] Optionally, the process of obtaining the reward scaling factor includes:

[0102] For the Harmful instructions in each security category and its corresponding security response Q&A Generate the original response using the original large model strategy. Construct a paired response dataset:

[0103]

[0104] in, For the first The paired response dataset corresponding to each security category Harmful instructions Paired security response and the original response gather;

[0105] Using the first Reward model for each security category Calculate the reward gap :

[0106]

[0107] in, For the first The first security category The reward gap between the security response and the original response for each harmful instruction. Reward points for a safe response. The reward score for the original response;

[0108] The reward gap can be calculated using the following formula. After processing, the average reward gap estimate is obtained:

[0109]

[0110] in, Estimate the average reward gap;

[0111] The reward scaling factor is obtained using the following formula. :

[0112]

[0113] in, This is the initial default scaling factor, used to set the baseline level for reward scaling; Used to control the weight ratio between static and dynamic components; It is an adjustable hyperparameter, which can be adjusted. Sensitivity to the difficulty of security categories; limitations This is to avoid excessive coefficient updates.

[0114] Optionally, step 4 includes:

[0115] For the For each safety category, the large model generation strategy is iteratively updated by optimizing the following objective function:

[0116]

[0117]

[0118] in, For the first Objective function for each security category The learnable parameters for generating policy networks in current large reinforcement learning models. This indicates the generated answer under the current large model generation strategy. The probability, This indicates the answers generated under the large model generation strategy before each iteration. The probability, For the large model generation strategy prior to this iteration, targeting the first... The security category corresponding to the first Candidate responses for a malicious instruction, The output probability is the reference policy output of the original large model. for and KL divergence between the two large model generation strategies For harmful issues The corresponding reward scaling factor; This is a truncation function to prevent the reinforcement learning policy from updating too much and to avoid the model training from becoming too divergent.

[0119] During training, the objective function for each security category is iteratively optimized alternately, and the reward model for different security categories is applied in turn. and reward scaling factor GRPO is executed to progressively integrate the reward gradient information output by the reward models of each security category into a unified system. In this process, by iterating alternately across security categories, the policy model learns and aligns with multiple security categories simultaneously, outputting a large security model optimized by multiple security categories.

[0120] See Figure 2This application also provides an embodiment of a large model security hardening system based on category adaptive reinforcement learning, wherein the method for implementing the above embodiment includes:

[0121] The training dataset construction module is used to divide the security risk data obtained from the large model security evaluation system into multiple security categories. For each security category of harmful instructions, the module generates corresponding security responses through an auxiliary large language model, thereby constructing the training dataset for each security category.

[0122] The reward model training module is used to construct a loss function for each security category by comparing the answers of the original large model with the security responses in the training dataset of each security category, and to train the corresponding reward model by minimizing the loss function.

[0123] The strategy optimization module is used to obtain the advantage function based on the reward model and reward scaling factor corresponding to each security category. The overall objective function is constructed based on the advantage function. During the training of the large model, the objective function of each security category is alternately and iteratively optimized, so that the strategy model can learn and align the security preferences of multiple security categories at the same time, and finally obtain the large model jointly optimized by multiple security categories.

[0124] Through the synergistic effect of the above modules, this application can achieve modular and scalable architecture support at the system level, thereby outputting a large security model.

[0125] This application first constructs multiple reward models for different security categories and performs multi-dimensional security scoring on candidate responses. Then, based on the reward difference between the reference security response and the model-generated response, a dynamic scaling factor is designed to apply difficulty-aware weighting to the advantage function within each category. Finally, the dynamically scaled advantage function is integrated into the reinforcement learning policy optimization process, guiding the model to be trained under the combined influence of multi-category security signals. In terms of system construction, a reinforcement learning fine-tuning architecture consisting of a security department dialogue data construction module, a category reward model training module, and a policy optimization module is constructed to achieve the engineering implementation of the above methods. By combining the above methods with the system, reliable signals from each security category can be fully utilized, and the contribution of each category in policy updates can be adaptively balanced, thereby significantly improving the multi-dimensional security alignment effect of large models and enhancing the model's security and robustness.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and not to limit them. Although this application has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of this application. Any modifications or equivalent substitutions that do not depart from the spirit and scope of this application should be covered within the protection scope of the claims of this application.

Claims

1. A method for security hardening of large models using category-adaptive reinforcement learning, characterized in that, include: Step 1: Divide the security risk data obtained from the large model security evaluation system into multiple security categories. For each security category of harmful instructions, generate the corresponding security response through an auxiliary large language model, thereby constructing the training dataset for each security category. Step 2: For each security category, construct a loss function by comparing the original large model's answer with the security responses in the training dataset for each security category, and train the corresponding reward model by minimizing this loss function; Step 3: Based on the reward model and reward scaling factor corresponding to each security category, obtain the advantage function; Step 4: Construct the overall objective function based on the advantage function, and iteratively optimize the objective functions of each security category during the training of the large model, so that the policy model can learn and align the security preferences of multiple security categories at the same time, and finally obtain the large model jointly optimized by multiple security categories. Step 3 includes: Using reward models and reward scaling factor Calculate the advantage function of the population relative policy optimization GRPO; in each iteration of GRPO, from Sample a batch of harmful instructions ,in, for A set of harmful instructions. For from the first The sampled from the set of all harmful instructions in the security category is the first... A collection of harmful instructions in a batch; for each harmful instruction in the current batch Based on the current large model generation strategy generate A set of candidate responses ,in, For the first The security category corresponding to the first A harmful instruction. For the current large model generation strategy, the first The security category corresponding to the first Candidate responses to malicious instructions, calculate malicious instructions. Corresponding reward scaling factor ;in accordance with The set of reward scores for each candidate response , To calculate the advantage function for the reward model parameters: in, For the first Advantage functions corresponding to each security category; Indicates harmful instructions Corresponding candidate response Reward points; Based on harmful instructions The mean reward of the current batch of samples is used to standardize the reward distribution; Based on harmful instructions The standard deviation of the reward for the current batch of samples is used to normalize reward volatility. The standard deviation of the reward includes a stable term. ; As an advantage estimate, it is used in the shearing target of GRPO to achieve policy optimization based on safety category difficulty adaptation; The process of obtaining the reward scaling factor includes: For the Harmful instructions in each security category and its corresponding security response Q&A Generate the original response using the original large model strategy. Construct a paired response dataset: in, For the first The paired response dataset corresponding to each security category Harmful instructions Paired security response and the original response gather, In order to target the The security category corresponding to the first Security response to a malicious instruction; Using the first Reward model for each security category Calculate the reward gap : in, For the first The first security category The reward gap between the security response and the original response for each harmful instruction. Reward points for a safe response, The reward score for the original response; The reward gap can be calculated using the following formula. After processing, the average reward gap estimate is obtained: in, Estimate the average reward gap; The reward scaling factor is obtained using the following formula. : in, This is the initial default scaling factor, used to set the baseline level for reward scaling; Used to control the weight ratio between static and dynamic components; It is an adjustable hyperparameter, which can be adjusted. Sensitivity to the difficulty of security categories; limitations This is to avoid excessive coefficient updates.

2. The method according to claim 1, characterized in that, Step 1 includes: Step 11: Obtain automatically detected and labeled security risk data from the large-scale model security assessment system, and classify it into... One security category; Step 12: For the first Each security category has a set of security features. A set of harmful instructions, denoted as The set of harmful instructions for all security categories is uniformly represented as Harmful instructions are text instructions that can induce large models to generate unsafe or illegal content; Step 13: Targeting The auxiliary generator GPT-4o is used to generate a security response. ;in, This indicates a structured reasoning answer. For the first Step-by-step structured reasoning answer, The final answer; the final answer refers to the final output conclusion generated by the logical synthesis of the GPT-4o answer based on structured reasoning. Step 14: Based on harmful instructions and its corresponding security response The training data for each security category was organized into , For the first Training data for each security category; summarizing the training data for all security categories into... , This is the training dataset for all security categories.

3. The method according to claim 2, characterized in that, Step 13 includes: Will As input prompts, a thought chain prompt template is constructed and input into the large model GPT-4o to generate structured reasoning answers and the final answer; whereby the structured reasoning answer refers to the GPT-4o thinking... Whether the reasoning process is safe, each structured reasoning answer uses... and The tags are used for annotation, and the final answer is encapsulated in... and Between tags; The starting label for the reasoning step. This is the end label for the reasoning step. The starting tag for the final answer, This is the closing tag for the final answer.

4. The method according to claim 2, characterized in that, Step 2 includes: For each security category, a contrastive learning-based ranking method is employed to maximize the security response by minimizing the loss function. Compared with the original response By gradually optimizing the reward model parameters based on the score difference between them. This is to enable the reward model to distinguish between safe and insecure responses under the same query, where, In order to target the The security category corresponding to the first The original response to a harmful instruction. For the first The reward model parameters corresponding to each security category; Update reward model parameters using gradient descent method : in, For the first The reward update step size for each security category For the first The loss function corresponding to each security category For the loss function to the reward model parameters The gradient; The process continues until the loss function converges or the set number of training epochs is reached, yielding the [number of epochs]. Final reward model for each security category ; After the reward models for all safety categories have been trained, a reward model set is formed. , It can provide multi-security-category security score signals for the subsequent policy optimization stage, realize differentiated modeling and signal constraints for different security categories, and thus support the policy model in the adaptive optimization of security categories in reinforcement learning.

5. The method according to claim 4, characterized in that, The process of obtaining the loss function includes: For each security category, in each iteration, from the training dataset... Training data for each security category Harmful Sampling Commands and Their Reference Security Responses And utilize the generative strategy of the original large model. Generate the original response Soon Input the original large model to obtain its original answer; According to the security response and the original response Construct the loss function.

6. The method according to claim 5, characterized in that, According to the security response and the original response Construct the loss function, including: The loss function is obtained using the following formula: in, For loss function, For the first The number of data points for each security category For the Sigmoid function, This is the reward score for a safe response under the current reward policy. Given the reward score of the original response under the current reward policy, the Sigmoid function guarantees that on the [number]th [day]... Under each safety category, the reward model tends to output higher scores for safe responses and lower scores for unsafe responses, in order to establish a ranking preference for safety categories.

7. The method according to claim 1, characterized in that, Step 4 includes: For the For each safety category, the large model generation strategy is iteratively updated by optimizing the following objective function: in, For the first Objective function for each security category The learnable parameters for generating policy networks in current large reinforcement learning models. This indicates the generated answer under the current large model generation strategy. The probability, This indicates the answers generated under the large model generation strategy before each iteration. The probability, For the large model generation strategy prior to this iteration, targeting the first... The security category corresponding to the first Candidate responses for a malicious instruction, The output probability is the reference policy output of the original large model. for and KL divergence between the two large model generation strategies For harmful issues The corresponding reward scaling factor; This is a truncation function to prevent the reinforcement learning policy from updating too much and to avoid the model training from becoming too divergent. During training, the objective function for each security category is iteratively optimized alternately, and the reward model for different security categories is applied in turn. and reward scaling factor GRPO is executed to progressively integrate the reward gradient information output by the reward models of each security category into a unified system. In this process, by iterating alternately across security categories, the policy model learns and aligns with multiple security categories simultaneously, outputting a large security model optimized by multiple security categories.

8. A large model security hardening system based on category adaptive reinforcement learning, implementing the method described in any one of claims 1-7, characterized in that, include: The training dataset construction module is used to divide the security risk data obtained from the large model security evaluation system into multiple security categories. For each security category of harmful instructions, the module generates corresponding security responses through an auxiliary large language model, thereby constructing the training dataset for each security category. The reward model training module is used to construct a loss function for each security category by comparing the answers of the original large model with the security responses in the training dataset of each security category, and to train the corresponding reward model by minimizing the loss function. The strategy optimization module is used to obtain the advantage function based on the reward model and reward scaling factor corresponding to each security category. The overall objective function is constructed based on the advantage function. During the training of the large model, the objective function of each security category is alternately and iteratively optimized, so that the strategy model can learn and align the security preferences of multiple security categories at the same time, and finally obtain the large model jointly optimized by multiple security categories.

Images