A large model safety optimization method based on segmented kl constraint and multi-dimensional reward

Abstract

The application provides a large model safety optimization method based on segmented KL constraint and multi-dimensional reward, first makes the model generate multiple answers for the same prompt word, then obtains five-dimensional scores of safety, utility, thinking analysis, detail and rationality through a DeepSeek-Reasoner scoring system, then calculates the advantage value through standardization, and then dynamically determines the position threshold k based on the risk level of the input instruction, applies strict KL constraint to the first k tokens of the generated sequence to ensure the initial safety, applies weak constraint to the subsequent tokens to release creativity, the total constraint is the weighted sum of two KL divergences, finally, based on the GRPO algorithm, the model parameters are updated by using the Lora method in combination with the segmented KL constraint and the multi-dimensional reward, and the optimization effect is verified through the diversity index and the calculation efficiency. The application scheme realizes the safety controllability and quality improvement of the large model generated content, balances the safety and creativity, and reduces the deployment cost.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence security technology, and in particular to a large model security optimization method based on segmented KL constraints and multidimensional rewards. Background Technology

[0002] Large language models are artificial intelligence models with massive parameters built based on deep learning technology. Large language models, represented by ChatGPT, can understand and generate natural and fluent text through large-scale pre-training and reinforcement learning based on human feedback (RLHF), providing users with high-quality answers and interactive experiences. Their applications cover multiple fields such as intelligent customer service, content creation, educational assistance, and code generation, greatly improving work efficiency and user satisfaction.

[0003] However, with the continuous expansion of applications of large language models, their security issues are becoming increasingly prominent. In terms of content generation, the models may generate harmful, false, or biased information; when maliciously manipulated, large language models may generate text containing harmful content such as violence, discrimination, and fraud, misleading and harming users. Regarding privacy protection, because model training relies on large amounts of data, it may unintentionally leak users' sensitive information. Furthermore, the models may be used for malicious purposes, such as generating fake news and automated phishing messages, posing a threat to social order and information security. Therefore, ensuring the security of the content generated by large language models while maintaining their powerful generative capabilities has become a critical issue that urgently needs to be addressed in the field of artificial intelligence.

[0004] In the safety optimization of large language models, existing reinforcement learning (RLHF) methods based on human feedback, such as proximal policy optimization (PPO) and its variant group relative policy optimization (GRPO), have played an important role in ensuring model safety and improving generation quality, but they also have some obvious drawbacks.

[0005] Most existing methods employ global Kullback-Leibler divergence (KL divergence) constraints to limit policy changes during model updates. In the GRPO algorithm, this global constraint subjects the model to the same level of constraint throughout the text generation process. While this ensures the stability and security of the generated content to some extent, it also leads to overly conservative generation. To meet strict KL divergence constraints, the model often sacrifices the information content and diversity of the generated content, resulting in text that is too bland and lacks originality, failing to fully utilize the creativity and potential of large language models and thus failing to meet users' demands for rich and personalized content.

[0006] Current security optimization methods typically employ a single security reward function, primarily focusing on whether the generated content conforms to security standards, while neglecting other important dimensions. This singular reward mechanism cannot comprehensively measure the quality of the generated content. As a result, while the model may ensure content security, it may perform poorly in areas such as utility, rationality, and detail. The generated text may lack practicality and fail to truly meet the user's actual needs; it may be logically flawed, containing inconsistencies and semantic ambiguity; or it may be too brief, failing to provide sufficient detail and information, thus impacting user experience and practical application effectiveness. Summary of the Invention

[0007] To address the aforementioned technical issues, this invention proposes a large model security optimization method based on piecewise KL constraints and multidimensional rewards. Through innovative constraint mechanisms and reward function design, it achieves fine-grained control over the large model generation process, effectively balancing security with the richness and innovation of the content.

[0008] A large-scale model safety optimization method based on piecewise KL constraints and multidimensional rewards includes the following steps:

[0009] Step S1: Set up a hardware and software environment that meets the requirements of large-scale language model training, and initialize the core training hyperparameters, including model parameters, training parameters, optimizer parameters and generation parameters;

[0010] Step S2: Load the pre-trained language model base and correctly attach the low-rank adaptation LoRA parameter module;

[0011] Step S3: Import the dataset, perform text cleaning and classification, and word segmentation to ensure that the input data meets the model training requirements;

[0012] Step S4: Set up a multidimensional reward function and standardize the reward value to obtain the advantage value;

[0013] Step S5: Flexibly set token-level KL divergence weights for different segments of the generated sequence to balance model security and generation innovation;

[0014] Step S6: Employ a policy optimization-based reinforcement learning method, combining advantage value calculation and piecewise KL constraints, to iteratively adjust the generation policy and optimize model behavior.

[0015] Step S7: Model saving and performance evaluation.

[0016] Furthermore, the model parameters include: model name, LoRA rank, LoRA target module, and dropout rate; the training parameters include: total training steps and first KL divergence constraint threshold. Location threshold k, second KL divergence constraint threshold PPO cutting parameters The optimizer parameters include: learning rate, Adam optimizer betas, and eps weight decay parameter; the generation parameters include: number of sequences generated for each input, generation temperature, and top_p sampling parameter.

[0017] Furthermore, the scoring dimensions of the multidimensional reward function include: security, utility, analytical thinking, innovativeness, and detail. The multidimensional reward function is expressed as follows:

[0018]

[0019] in, This represents the reward for the i-th reply. , , , , These represent the weights of each rating dimension. This represents the security score for the i-th response. This represents the utility score for the i-th response. This represents the analytical score for the i-th response. This represents the score for the innovativeness of the i-th response. This represents the detail rating for the i-th response, using the DeepSeek-Reasoner rating system.

[0020] Furthermore, the reward function for each response is updated to the dominance function, expressed as:

[0021]

[0022] in, Let represent the advantage value of the i-th response, G represent the number of responses under the same prompt, mean represent the average value, and std represent the standard deviation.

[0023] Furthermore, piecewise KL constraint refers to segmenting the KL divergence based on a location threshold k parameter and using two different hyperparameters, namely the first KL divergence constraint threshold. Second KL divergence constraint threshold Apply constraints.

[0024] Furthermore, the training loss function for reinforcement learning in step S6 is:

[0025]

[0026] in, This represents the i-th reply. This indicates the length of the i-th reply, which is also the number of tokens. Let t represent the t-th token of the i-th reply, and q represent the prompt word. This represents the first t-1 tokens of the i-th reply. This indicates that the current model is in a state where the prompt word is q and the first t-1 tokens are... At that time, the next token is The probability distribution, This indicates that the model trained in the previous step was in a state where the prompt word was q and the first t-1 tokens were... At that time, the next token is The probability distribution, This represents the advantage value of the current i-th response, and the clip function represents clipping. This means constraining the ratio of the probability distribution of the model trained in the previous step to that of the current model. between, This represents the summation of the KL divergences of the first k tokens. This represents the summation of the KL divergences of all tokens after the (k+1)th token.

[0027] Furthermore, the KL divergence is expressed as:

[0028]

[0029]

[0030] in, Denotes KL divergence, The parameter is The probability distribution of the model at that time is the current model. Represents the probability distribution of the reference model. Let represent the probability distribution of the current model at the position of the t-th token. This represents the probability distribution of the reference model at the position of the t-th token. This represents the sum of the logarithms of the quotients of the two probability distributions at each token position, where logits represents the log odds.

[0031] This invention, based on the theoretical foundation that large model security alignment mainly focuses on the tokens in the early stages, innovatively proposes a large model security optimization method based on segmented KL constraints and multidimensional rewards. It introduces a position threshold k, which is used to clearly divide the text sequence generated by the large language model into two key regions: a security guidance region (1→k) and a free creation region (k+1→end). By strictly restricting the first few tokens that are most important to the security of the large model output, and encouraging the updating of subsequent tokens that have an important impact on the utility and innovation of content generation, the model is allowed to explore freely within a certain range, generating more diverse and richer content, and meeting users' needs for personalized and innovative text.

[0032] This invention aims to comprehensively measure the quality of content generated by a large model. Through a designed multi-dimensional reward function and utilizing the cutting-edge DeepSeek-Reasoner model, it deeply evaluates the generated content from five key dimensions: safety score, utility score, analytical thinking score, innovativeness score, and detailedness score. Finally, training is performed based on these scores, thereby improving the overall quality of the generated content in a multi-dimensional and comprehensive manner. This method allows the model to ensure the generated content meets safety standards while also producing more diverse content, avoiding repetitive and monotonous expressions, and providing users with richer and more novel text content to meet their needs for personalized and innovative content.

[0033] By using segmented KL divergence constraint design, strict constraints are imposed on the front-end tokens that have a significant impact on security alignment, while variations are encouraged for the subsequent tokens that have a significant impact on content richness and creativity. In the low-risk part, the constraints on KL divergence can be reduced, the computational efficiency is optimized, security and creativity are balanced, and the contradiction between excessive constraints and insufficient constraints is resolved.

[0034] The segmented constraint + multi-dimensional reward paradigm proposed in this invention provides quantifiable and reproducible technical standards (such as KL threshold range and reward weight ratio) for the safe optimization of large models, and solves the problem of traditional safety mechanisms relying on empirical parameter tuning. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart illustrating a large model safety optimization method based on piecewise KL constraints and multidimensional rewards provided in an embodiment of the present invention.

[0037] Figure 2 This is a schematic diagram of the model optimization process provided in the embodiments of the present invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] This invention aims to address the critical issue of balancing the security and creativity of content generated by large models. It seeks an effective method to fully unleash the creativity of content generated by large models while ensuring its security, generating high-quality text that is both safe and reliable, as well as rich and diverse, to meet multi-dimensional evaluation needs. At the same time, it can significantly optimize computational efficiency and reduce the cost of large-scale deployment. Furthermore, it can provide quantifiable and reproducible technical standards for the security optimization of large models, breaking through the limitations of traditional experience-based parameter tuning.

[0040] Group Relative Policy Optimization (GRPO) optimizes the model by utilizing content generated during training. Its goal is to maximize the advantage function while limiting the model to the vicinity of the reference policy. GRPO includes the following steps:

[0041] Group generation: When given a hint, the model does not generate just one answer; instead, it generates multiple attempts for the same question (usually 4, 8, or 16 different attempts).

[0042] Preference learning: GRPO can use any function or model to evaluate the quality of a solution. For example, a length function can be used to reward shorter answers, or a mathematical solver can be used to reward accurate mathematical solutions.

[0043] Iterative optimization mainly includes two principles: encouraging the model to generate more solutions similar to successful cases, while avoiding methods that perform poorly; GRPO includes a safety mechanism (i.e., KL divergence penalty) to prevent the model from changing too much at once.

[0044] This invention, based on the GRPO algorithm, employs innovative constraint mechanisms and reward function design to develop a large-model safety optimization method based on piecewise KL constraints and multidimensional rewards. This method achieves fine-grained control over the large-model generation process, effectively balancing safety and creativity. Figure 1 As shown, the method specifically includes the following steps:

[0045] S1: Set up the hardware and software environment that meets the requirements of large-scale language model training and initialize the core training hyperparameters.

[0046] In one feasible implementation, the hardware environment consists of two NVIDIA L20 48GB GPU servers equipped with 20 vCPU Intel(R) Xeon(R) Platinum 8457C CPUs and 100GB of memory. The software environment consists of an Ubuntu 20.04 LTS operating system, PyTorch 2.7.0 deep learning framework, CUDA version 12.8, and Python version 3.10.16. Hugging Face's Transformers library (version 4.51.3) is used for model loading and processing, and the Datasets library (version 3.5.0) is used for data processing. The selection of these software versions has undergone compatibility testing to ensure stable collaboration among the components.

[0047] The core training hyperparameters include: model parameters, training parameters, optimizer parameters, and generation parameters. In one feasible implementation, the model parameters include: the model is qwen2.5-7b, the LoRA rank is 8, the target module is ["q_proj", "v_proj"], and the dropout rate is 0.1. The training parameters include: a total of 1000 training steps, and a first KL divergence constraint threshold. The value is 0.1, and the position threshold k (i.e., how many tokens are needed before it can be used) is... The parameter (for KL encouragement) is 20, and the second KL divergence constraint threshold is... The PPO trimming parameter is 0.03. The value is 0.2. The optimizer parameters include: a learning rate of 2x10⁻¹⁰. -4 The Adam optimizer has betas parameters of (0.9, 0.999) and eps of 1x10. -8 The weight decay is 0.001. The generation parameters include: 8 sequences generated per input, a generation temperature of 1.2, and a top_p sampling parameter of 0.9.

[0048] S2: Load the pre-trained language base model and correctly attach the low-rank adaptation (LoRA) parameter module.

[0049] A pre-trained language base model refers to a model that learns the general rules of language, semantic understanding, grammatical structure, and other basic knowledge by pre-training and aligning it on large-scale text data, and learns how to use this knowledge to generate content, thus obtaining a model with basic language capabilities.

[0050] LoRA (Low-Rank Adaptation) is a technique that adds a bypass to the original pre-trained model to perform dimensionality reduction and then dimensionality increase operations, as shown below:

[0051]

[0052]

[0053] in, This represents the parameter matrix of the training layer in the pre-trained model. , Let represent a real number space with n rows and m columns, where n represents the input dimension of the parameter matrix of the layer to be trained, and m represents the output dimension of the parameter matrix of the layer to be trained. This indicates an added bypass. , Represents a dimension reduction matrix. , Represents an upgraded matrix. d represents the low-rank dimension. This reduces the number of training parameters by using a low-rank dimension. During training, the pre-trained model parameters are fixed, and only the reduced-dimensional matrix A and the increased-dimensional matrix B are trained. During inference, the matrix product is used. Compared with the original weight matrix When combined as new weight parameters, they can reduce the computational cost and memory overhead of fine-tuning large models.

[0054] S3: Import the dataset and perform preprocessing operations such as text cleaning and word segmentation to ensure that the input data meets the requirements for model training.

[0055] The dataset refers to the prompt words used to train the model. The dataset used is a refined version of the ToxiGen dataset, comprising 10K high-risk texts, 20K medium-risk texts, and 30K low-risk texts. This dataset is used to evaluate the model's security, generated content quality, and diversity. It includes not only malicious and benign instructions but also standardizes the domain of each instruction, which facilitates text classification during preprocessing. High-risk instructions are characterized by directly involving personal safety, illegal activities, and significant ethical issues, including medical diagnosis, violent crime, privacy violations, and legal circumvention. Medium-risk instructions are characterized by involving professional fields but not high-risk scenarios, with minor potential risks requiring management, including financial operation advice, basic medical information, and controversial topics. Low-risk instructions are characterized by not involving sensitive topics, having no legal / ethical risks, and minimal impact from generated content deviations, including general knowledge questions and answers, creative writing, daily conversations, and mathematical calculations.

[0056] Preprocessing includes cleaning and classifying the text in the dataset, as well as word segmentation. Text cleaning refers to removing invalid characters, special symbols, and duplicate content to ensure data standardization; text classification refers to classifying prompt words into high-risk, medium-risk, and low-risk texts based on the original labels of the ToxiGen dataset and the features described above; word segmentation refers to setting the maximum sequence length of the labeled dataset to 2048 and constructing training subsets according to the risk level ratio.

[0057] S4: Set up a multidimensional reward function and standardize the reward value to obtain the advantage value.

[0058] The reward function refers to the core method of linking large models with actual needs; it defines the target direction of the large model's response. In one feasible implementation, the response of the large model is scored by calling the cutting-edge DeepSeek-Reasoner. DeepSeek-Reasoner, an AI model with deep reasoning capabilities launched by DeepSeek, possesses outstanding reasoning and discriminative abilities, making it a qualified and excellent evaluation model.

[0059] A multi-dimensional reward function refers to a scoring principle that evaluates content across multiple dimensions, rather than solely on one. In a feasible implementation, the scoring dimensions include: safety, utility, analytical thinking, innovativeness, and detail, with each dimension scored from 1 to 10. The safety score primarily assesses whether the generated content contains harmful, sensitive, or unethical / legal information, ensuring compliance with social norms and safety standards. The utility score focuses on whether the generated content meets the user's actual needs and has practical application value and assistance. The analytical thinking score focuses on whether the model considers the requirements, reflects on the issues, and whether the logic is sound and the reasoning process clear. The innovativeness score focuses on whether the generated content is innovative and original, including fresh perspectives, creative analogies, or unique insights. The detail score measures the level of detail in the generated content, ensuring sufficient information and detail are provided, avoiding overly simplistic and vague content. The multi-dimensional reward function is represented as follows:

[0060] ;

[0061] in, This represents the reward for the i-th reply. , , , , These represent the weights of each rating dimension. This represents the security score for the i-th response. This represents the utility score for the i-th response. This represents the analytical score for the i-th response. This represents the score for the innovativeness of the i-th response. This represents the detail score for the i-th response. In this embodiment, the security weight is set to... The weights for the other dimensions are all 0.15, which emphasizes the security of the response while maintaining the multidimensional capabilities of the large model.

[0062] To standardize the rewards and stabilize the training process, the reward function for each response is updated to the advantage function:

[0063] ;

[0064] in, Let represent the advantage value of the i-th response, G represent the number of responses under the same prompt, mean represent the average value, and std represent the standard deviation.

[0065] S5: Flexibly set token-level segmented KL divergence weights for different segments of the generated sequence to balance model security and generation innovation.

[0066] The generated sequence refers to the portion of the content generated by the large model after removing prompt words. Each position in this part is a vector the size of the vocabulary, called logits. A token is the smallest unit of processing language data. To ensure that the input paragraph can be understood by the large model, it needs to be broken down into smaller units. For example, "I loveChina." would be broken down into "I", "(space)", "love", "(space)", and "China". This method standardizes the input and helps the model understand the context more accurately.

[0067] KL divergence is an asymmetric measure of the difference between two probability distributions P and Q, based on information entropy theory. Its definition is as follows:

[0068]

[0069]

[0070] in, Denotes KL divergence, The parameter is The probability distribution of the model at that time is the current model. Represents the probability distribution of the reference model. This represents the probability distribution of the current model at the position of the t-th token. This represents the position of the t-th token, referencing the probability distribution of the model. This represents the sum of the logarithms of the quotients of the two probability distributions at each token position, where logits represents the log odds.

[0071] Piecewise KL constraints refer to dividing the KL divergence into a safe guidance zone and a free creation zone based on a location threshold parameter k, and using two different hyperparameters, namely the first KL divergence constraint threshold. Second KL divergence constraint threshold Perform constraint loss function calculation, first KL divergence constraint threshold. The value range is 0.01-0.1, which is the second KL divergence constraint threshold. The value range is 1.0-5.0, and The value of k is dynamically adjusted according to the security requirements of the generated task. In one feasible method, the dynamic mapping rule for the position threshold k is: k ≥ 15 when the risk level is high, 10 ≤ k < 15 when it is medium, and k < 10 when it is low. The risk level is determined by pre-defined labels in the dataset. In the security guidance zone, the first k tokens generated by the model play a crucial guiding role in the theme and direction of the entire text. To ensure the security of the content's initial state, a strict first KL divergence constraint threshold is applied to this region. By tightly constraining the model's policy changes, the initial content generated by the model conforms to security standards, preventing the generation of harmful information at its source. In the free creation area, subsequent token generation focuses more on unleashing the model's creativity. To release the model's creativity, a relatively weak second KL divergence constraint threshold is applied to this area. ,in < This setup allows the model to explore freely within certain limits, generating more diverse and richer content to meet users' needs for personalized and innovative text.

[0072] S6: Employ a policy optimization-based reinforcement learning approach, combining advantage value calculation and piecewise KL constraints, to iteratively adjust the generation policy and optimize model behavior.

[0073] Policy optimization is a core method in reinforcement learning. Its core idea is to directly optimize the agent's policy to maximize the cumulative reward through interactions with the environment. In this embodiment, the policy is represented by a probability distribution function. .

[0074] The loss function for optimizing training is specifically expressed as follows:

[0075]

[0076] in, This represents the i-th reply; This represents the length of the i-th reply, i.e., the number of tokens. represents the t-th token of the i-th reply, and q represents the prompt word. This represents all tokens in the first t-1 of the i-th reply; This indicates that the current model is in a state where the prompt word is q and all the first t-1 tokens are... At that time, the next token is The probability distribution; This indicates that the model trained in the previous step has the prompt word q and all the first t-1 tokens are... At that time, the next token is The probability distribution; This represents the advantage value of the i-th response at this point; the clip function represents clipping, specifically... This means constraining the ratio of the probability distribution of the model trained in the previous step to that of the current model. In this embodiment ; This represents the summation of the KL divergences of the first k tokens; This represents the summation of the KL divergences of all tokens after the (k+1)th token.

[0077] In one feasible implementation, the Group Relative Policy Optimization (GRPO) algorithm is used for model optimization, such as... Figure 2 As shown, the specific method is as follows:

[0078] Step S601: Group generation. Input a prompt word into the model, let the model generate multiple responses to the prompt word, and combine the multiple responses into a group;

[0079] Step S602: Reward calculation, using the multidimensional reward function designed in step S4 to score each response and calculate its respective advantage value;

[0080] Step S603: Determine the location threshold and assign different location thresholds k according to the risk level of the prompt words;

[0081] Step S604: Loss calculation, using the previously designed loss function to calculate the loss function of the model response;

[0082] Step S605: Iterative optimization. Repeat the above steps until the preset number of iterations is completed.

[0083] S7: Model Saving and Performance Evaluation

[0084] Model saving is divided into two stages: saving once every certain number of steps (in this embodiment, the certain number of steps is set to 100 steps) and saving the final model. This method can ensure the security of the training process and facilitate model performance analysis and selection.

[0085] Performance evaluation refers to assessing the model's performance, i.e., the score of the response, using the test portion of the dataset. The evaluation is divided into safety, utility, analytical thinking, innovativeness, and detail. Each dimension is scored with an average score and a comprehensive score. At the same time, the distribution of the comprehensive score is analyzed. This can not only comprehensively measure the model's performance, but also provide key basis for model optimization and application.

[0086] This invention provides a large-scale model safety optimization method based on segmented KL constraints and multi-dimensional rewards, aiming to solve the "safety and creativity imbalance" problem caused by global KL constraints in existing reinforcement learning optimization. Through the collaborative design of segmented constraints and multi-dimensional rewards, the method achieves both safety controllability and quality improvement of the content generated by the large model. First, the model generates multiple responses to the same prompt word. Then, a five-dimensional score (safety, utility, analytical thinking, detail, and rationality) is obtained using the DeepSeek-Reasoner scoring system. Next, the advantage value is calculated through standardization. Then, a position threshold k is dynamically determined based on the risk level of the input instruction. Strict KL constraints are applied to the first k tokens of the generated sequence to ensure initial safety, while weak constraints are applied to subsequent tokens to unleash creativity. The total constraint is a weighted sum of the KL divergences of the two segments. Finally, based on the GRPO algorithm, the LoRa method is used to update the model parameters, combining segmented KL constraints and multi-dimensional rewards. The optimization effect is verified through diversity indicators and computational efficiency. This invention achieves a risk interception rate of 95% on the test set and reduces KL computation overhead by 60%. It can be widely applied to large model optimization in high-security scenarios, balancing security and creativity while reducing deployment costs.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A large-scale model safety optimization method based on piecewise KL constraints and multidimensional rewards, characterized in that, The method includes: Step S1: Set up a hardware and software environment that meets the requirements of large-scale language model training, and initialize the core training hyperparameters, including model parameters, training parameters, optimizer parameters and generation parameters; Step S2: Load the pre-trained language model base and correctly attach the low-rank adaptation LoRA parameter module; Step S3: Import the dataset, perform text cleaning and classification, and word segmentation to ensure that the input data meets the model training requirements; Step S4: Set up a multidimensional reward function and standardize the reward value to obtain the advantage value; Step S5: Flexibly set token-level KL divergence weights for different segments of the generated sequence to balance model security and generation innovation; Step S6: Employ a policy optimization-based reinforcement learning method, combining advantage value calculation and piecewise KL constraints, to iteratively adjust the generation policy and optimize model behavior; Step S7: Model saving and performance evaluation.

2. The large model safety optimization method based on piecewise KL constraints and multidimensional rewards according to claim 1, characterized in that, The model parameters include: model name, LoRA rank, LoRA target module, and dropout rate; the training parameters include: total training steps and the first KL divergence constraint threshold. Location threshold k, second KL divergence constraint threshold PPO cutting parameters The optimizer parameters include: learning rate, Adam optimizer betas, and eps weight decay parameter; the generation parameters include: number of sequences generated for each input, generation temperature, and top_p sampling parameter.

3. The large model safety optimization method based on piecewise KL constraints and multidimensional rewards according to claim 1, characterized in that, The scoring dimensions of the multidimensional reward function include: security, utility, analytical thinking, innovativeness, and detail. The multidimensional reward function is expressed as follows: in, This represents the reward for the i-th reply. , , , , These represent the weights of each rating dimension. This represents the security score for the i-th response. This represents the utility score for the i-th response. This represents the analytical score for the i-th response. This represents the score for the innovativeness of the i-th response. This represents the detail rating for the i-th response, using the DeepSeek-Reasoner rating system.

4. The large model safety optimization method based on piecewise KL constraints and multidimensional rewards according to claim 3, characterized in that, Update the reward function for each response to the advantage function, expressed as: in, Let represent the advantage value of the i-th response, G represent the number of responses under the same prompt, mean represent the average value, and std represent the standard deviation.

5. The large model safety optimization method based on piecewise KL constraints and multidimensional rewards according to claim 1, characterized in that, Piecewise KL constraint refers to segmenting the KL divergence based on a location threshold k parameter and using two different hyperparameters: the first KL divergence constraint threshold and the second hyperparameter. Second KL divergence constraint threshold Apply constraints.

6. The large model safety optimization method based on piecewise KL constraints and multidimensional rewards according to claim 4, characterized in that, The training loss function for step S6 using reinforcement learning is: in, This represents the i-th reply. This indicates the length of the i-th reply, which is also the number of tokens. Let t represent the t-th token of the i-th reply, and q represent the prompt word. This represents the first t-1 tokens of the i-th reply. This indicates that the current model is in a state where the prompt word is q and the first t-1 tokens are... At that time, the next token is The probability distribution, This indicates that the model trained in the previous step was in a state where the prompt word was q and the first t-1 tokens were... At that time, the next token is The probability distribution, This represents the advantage value of the current i-th response, and the clip function represents clipping. This means constraining the ratio of the probability distribution of the model trained in the previous step to that of the current model. between, This represents the summation of the KL divergences of the first k tokens. This represents the summation of the KL divergences of all tokens after the (k+1)th token.

7. The large model safety optimization method based on piecewise KL constraints and multidimensional rewards according to claim 6, characterized in that, The KL divergence is expressed as: in, Denotes KL divergence, The parameter is The probability distribution of the model at that time is the current model. Represents the probability distribution of the reference model. Let represent the probability distribution of the current model at the position of the t-th token. This represents the probability distribution of the reference model at the position of the t-th token. This represents the sum of the logarithms of the quotients of the two probability distributions at each token position, where logits represents the log odds.

Images