A large model security alignment method based on reinforcement learning and self-generated trajectory
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- UNIV OF SCI & TECH OF CHINA
- Filing Date
- 2026-03-09
- Publication Date
- 2026-06-09
Smart Images
Figure CN122174239A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, specifically a method for secure alignment of large models based on reinforcement learning and self-generated trajectories. Background Technology
[0002] Large-scale inference models demonstrate superior performance in handling complex logical tasks and typically possess a degree of self-correction capability. However, in the field of AI security, particularly when facing adversarial attacks, especially jailbreak attacks, these models often lack the ability to reflect on insecure content, making them vulnerable to breaches and the output of harmful content.
[0003] Currently, secure alignment techniques for large-scale inference models are mainly divided into two branches: imitation learning based on supervised fine-tuning (SFT) and reinforcement learning (RL) based on static data.
[0004] Supervised fine-tuning (SFT) strategies primarily rely on knowledge distillation through secure trajectories generated by expert models (such as GPT-4). Specifically, these methods force the model to mimic the backtracking behavior of experts when faced with attacks by constructing expert correction paths containing explicit reflective steps or injecting specific control tokens into the training data. However, this approach, which depends on external expert trajectories, suffers from a significant "off-policy" distribution offset problem. Expert-generated trajectories are static and idealized, fundamentally different from the dynamic, intrinsic thought processes generated by the model during actual reasoning. This distributional mismatch causes the model to merely mechanically imitate the expert's surface behavior rather than truly learning how to identify and correct its own specific erroneous latent states. Therefore, when faced with complex reasoning scenarios, the model often fails to recover from its own erroneous logic.
[0005] Another strategy employs reinforcement learning (RL) or preference optimization (DPO) methods based on static datasets. Specifically, these methods encourage the model to explore recovery paths under specific attack scenarios by constructing fixed insecure prefixes or utilizing augmented data, and provide rewards or penalties based on the results. However, adversarial attacks in real-world scenarios are highly diverse and covert, and the evolution of attack methods often leads the model into entirely new and unfamiliar insecure states. Existing RL methods are limited by the static and finite nature of training data, failing to cover the vast potential security space, and even less able to simulate the dynamic biases generated by the model itself in real time during inference and decoding. This gap between static training data and dynamic inference trajectories means that when faced with unseen attack patterns, the model lacks the generalization ability to reflect on its own generated insecure trajectories and ultimately output secure responses.
[0006] To achieve automated security assessment and feedback, existing research often introduces security guard models as supervisory tools. These models are typically fine-tuned to identify harmful content, and their functions fall into two main categories: output guards, which discriminate the model's final response, outputting safety and rejection labels to provide reward signals for reinforcement learning; and stream guards, which enable real-time monitoring and annotation of the model's inference trajectory at the token level. These tools provide automated criteria for capturing risky content generated by the model and are crucial infrastructure for building a secure alignment loop.
[0007] In summary, existing security alignment techniques primarily rely on externally constructed static data or expert trajectories, generally ignoring the dynamic error distribution generated by the model during actual inference and decoding. Due to the lack of an effective mechanism for utilizing the model's own on-policy insecure trajectories, existing methods struggle to eliminate the distributional discrepancies between training data and the inference process. This results in the model's inability to effectively self-correct and recover when facing complex and ever-changing new attacks, limiting its robustness in real-world security scenarios. Summary of the Invention
[0008] The problem addressed by this invention is how to overcome the distribution offset caused by external static expert trajectories and solve the deficiency of insufficient generalization ability of existing reinforcement learning methods when facing out-of-distribution attacks.
[0009] To address the aforementioned issues, this invention provides a method, system, electronic device, and storage medium for secure alignment of large models based on reinforcement learning and self-generated trajectories.
[0010] In a first aspect, the present invention provides a method for secure alignment of large models based on reinforcement learning and self-generated trajectories, comprising the following steps: The streaming guard model is used to monitor the same-policy trajectories generated by large-scale inference models in real time during the inference process, and to identify and locate the critical point where the model first appears an unsafe state during inference. The inference trajectory is truncated at the critical point to obtain an unsafe prefix, which is then stored in a dynamically updated experience replay buffer. The unsafe prefix is preferentially sampled from the experience replay buffer as the initial state for reinforcement learning, driving the inference model to continue generating inference paths based on this. The safety and usefulness of the final response generated by the inference model are evaluated using a result validator, and a binary reward signal is calculated. Based on the reward signal, a policy optimization algorithm is used to update the parameters of the inference model, enabling the inference model to learn to recover from insecure prefixes to secure inference paths.
[0011] Optionally, when the streaming guard model monitors the inference trajectory, it adopts a posterior full trajectory evaluation method, performs a single forward propagation of the complete inference chain, outputs a security classification label for each token position, and uniformly maps the "disputed" and "insecure" labels to an insecure state.
[0012] Optionally, the method for identifying the critical point of an unsafe state is as follows: when two consecutive token positions are marked as unsafe, it is determined that the trajectory has experienced a safe drift, and the first unsafe token position of the entire trajectory is taken as the cutoff point.
[0013] Optionally, the experience replay buffer is dynamically updated using a first-in-first-out queue mechanism with a fixed capacity limit. When the buffer is full, the earliest stored sample is automatically removed to ensure that the stored samples always reflect the latest failure modes of the current inference model.
[0014] Optionally, the sampling process of the reinforcement learning adopts a priority-mixed sampling strategy: priority is given to sampling unsafe prefixes from the buffer, and ordinary inputs are sampled from the original dataset when the buffer is empty.
[0015] Optionally, the binary reward signal is calculated as follows: The reward is 1 when the input is a harmful query and the output is a safe response; The reward is 1 when the input is a benign query and the output is a safe and non-rejecting response; The reward is 0 in other cases.
[0016] Optionally, the strategy optimization algorithm is a decoupled pruning and dynamic sampling strategy optimization, which calculates the standardized relative advantage for each group of sampled outputs and introduces a pruning mechanism to stabilize the training process.
[0017] Secondly, embodiments of the present invention provide a large-model secure alignment system based on reinforcement learning and self-generated trajectories, comprising: The streaming guard module is used to monitor the model's inference trajectory in real time and identify unsafe states; The truncation and storage module is used to truncate the prefix and store it in the experience replay buffer when an unsafe state is detected. The priority sampling module is used to sample insecure prefixes from the buffer as the initial state for reinforcement learning; The reward calculation module is used to evaluate the safety and usefulness of the final response generated by the model and output a binary reward. The strategy optimization module is used to update model parameters based on reward signals, thereby improving the model's self-recovery capability.
[0018] Optionally, the experience replay buffer is dynamically updated using a first-in-first-out mechanism to ensure that the sample distribution is consistent with the current failure mode of the model.
[0019] Thirdly, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, and a bus, wherein the processor, the communication interface, and the memory communicate with each other through the bus, and the processor can call logical instructions in the memory to execute the steps of the method provided in the first aspect.
[0020] Fourthly, embodiments of the present invention provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the large model secure alignment method based on reinforcement learning and self-generated trajectories as described in the first aspect.
[0021] The beneficial effects of the large model secure alignment method based on reinforcement learning and self-generated trajectories of the present invention are as follows: This method first uses a streaming guard model to monitor the same-policy inference trajectories generated by the policy model. Then, it locates and truncates unsafe prefixes and stores them in a dynamically updated experience replay buffer. Next, using a priority replay mechanism, the unsafe prefixes in the buffer are used as the initial state for reinforcement learning, forcing the model to explore and regress to a safe inference path based on known insecurity. Finally, a binary validator is used to evaluate the final response, calculating a reward that balances safety and usefulness. The model parameters are optimized and updated through decoupling pruning and dynamic sampling strategies, enabling the model to learn self-recovery capabilities. Attached Figure Description
[0022] Figure 1 This is a flowchart of a large model secure alignment method based on reinforcement learning and self-generated trajectories in an embodiment of the present invention; Figure 2 This is a schematic diagram of the large model secure alignment method based on reinforcement learning and self-generated trajectories in an embodiment of the present invention. Figure 3 This is a framework diagram of Self-ReSET in an embodiment of the present invention; Figure 4 This is a structural block diagram of a large model secure alignment system based on reinforcement learning and self-generated trajectories in an embodiment of the present invention; Figure 5 This is a structural block diagram of the electronic device in an embodiment of the present invention. Detailed Implementation
[0023] To better understand the purpose, technical solution, and advantages of this application, the application is described and explained below in conjunction with the accompanying drawings and embodiments.
[0024] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.
[0025] like Figures 1 to 3 As shown in the figure, a large model secure alignment method based on reinforcement learning and self-generated trajectories provided in this embodiment of the invention includes the following steps: S1. Utilize the streaming guard model to monitor the same-policy trajectories generated by the large-scale inference model in real time during the inference process, and identify and locate the critical point where the unsafe state first appears in the model inference. Furthermore, when monitoring the inference trajectory, the streaming guard model adopts a posterior full trajectory evaluation method, performs a single forward propagation of the complete inference chain, outputs a security classification label for each token position, and uniformly maps the "controversial" and "insecure" labels to an insecure state.
[0026] The method for identifying the critical point of an unsafe state is as follows: when two consecutive token positions are marked as unsafe, it is determined that the trajectory has undergone a safe drift, and the first unsafe token position is taken as the cutoff point.
[0027] S2. Truncate the inference trajectory at the critical point to obtain the unsafe prefix, and store it in the dynamically updated experience replay buffer. The experience replay buffer is dynamically updated using a first-in-first-out queue mechanism to ensure that the stored samples always reflect the latest failure modes of the current inference model.
[0028] S3. Prioritize sampling of unsafe prefixes from the experience replay buffer as the initial state for reinforcement learning, driving the inference model to continue generating inference paths based on this; The sampling process in reinforcement learning employs a priority-mix sampling strategy: it prioritizes sampling unsafe prefixes from the buffer, and then samples normal inputs from the original dataset when the buffer is empty.
[0029] S4. Use the result validator to evaluate the safety and usefulness of the final response generated by the inference model, and calculate the binary reward signal; In this embodiment, the binary reward signal is calculated as follows: The reward is 1 when the input is a harmful query and the output is a safe response; The reward is 1 when the input is a benign query and the output is a safe and non-rejecting response; The reward is 0 in other cases.
[0030] S5. Based on the reward signal, the inference model parameters are updated using a policy optimization algorithm, enabling the inference model to learn to recover from insecure prefixes to safe inference paths.
[0031] The strategy optimization algorithm is a decoupled pruning and dynamic sampling strategy optimization. It calculates the standardized relative advantage for each set of sampled outputs and introduces a pruning mechanism to stabilize the training process.
[0032] Specifically, this method adopts a closed-loop paradigm of "trajectory monitoring-temporary storage-regeneration". First, a streaming guard model is used to monitor the on-policy inference trajectory of the inference model in real time to locate the trigger points of unsafe states. Second, the truncated unsafe prefix is stored in a dynamically updated experience replay buffer to capture the model's current failure mode. Finally, the unsafe prefix in the buffer is used as the initial state for reinforcement learning sampling, and a verifiable binary safety reward optimization strategy forces the model to recover from the unsafe state to a safe inference path. Specifically, it proceeds in the following steps: First, define the Large Inference Model (LRM) as the strategy. Given input query The generation process includes intermediate inference trajectories. and final reply Divide the input space into benign queries. and harmful queries The output space is divided into safe recovery. (Including refusing to reply) (This indicates a direct refusal to answer, a reply containing no useful information, and an unsafe reply.) .
[0033] The optimization objective of this method is to maximize the probability of generating output that satisfies the following condition: when hour, ;when , And at the same time This helps prevent the model from learning a response pattern that excessively rejects all inputs.
[0034] Furthermore, a streaming security guard model (such as Qwen3Guard-Stream) is used to perform fine-grained security assessments on the inference trajectories generated by the model. To balance computational efficiency and detection accuracy, the implementation method is as follows: Output trajectory monitoring: Instead of monitoring the input in real time on a token-by-token basis, the complete inference trajectory is generated in the policy model. Then, the entire trajectory As input, the data is fed into the guard model for a single forward propagation. The guard model is positioned at each token location. Output a corresponding security category label The label It represents the prefix from the beginning of the sequence to the current position. The safety status. Note that, to avoid iteratively storing samples, only the original training set data is monitored and truncated.
[0035] Label Mapping and Strict Constraints: The original output space of the guard model contains three types of labels: {"safe", "controversial", "unsafe"}. To strictly constrain the safety of the inference trajectory, this embodiment employs a strict mapping strategy, treating both "controversial" and "unsafe" as unsafe states. A binary safety label sequence is defined. ,in The mapping logic is: if , ;like , Through this mapping, any prefix deemed to pose a potential risk is marked as 1 (unsafe).
[0036] Unsafe trajectory determination: based on the mapped binary sequence To filter out model prediction noise, a continuity criterion is used to identify "unsafe trajectories": ; That is, a substantial security drift is determined to have occurred in the trajectory only when two consecutive token positions are marked as insecure.
[0037] Truncation and storage of insecure tracks: Build with fixed capacity Experience replay buffer It is used to store and manage high-value training samples (insecure samples).
[0038] Unsafe critical point location: For those determined to be The trajectory will be traced back from the entire trajectory to locate the first moment that meets the unsafe conditions. : ; That moment This represents the critical point at which the model's reasoning logic begins to cross the safety boundary.
[0039] Prefix truncation and storage: in The inference chain is truncated at all times to extract unsafe prefixes. This is because it precisely records the context states that cause the model to begin moving towards unsafe conditions. The sample tuples are then... Store in buffer For use in subsequent training.
[0040] Dynamic FIFO Update: Buffer It is maintained using a First-In, First-Out (FIFO) mechanism consistent with queues. When the buffer is full... When a new sample is added, the oldest sample is automatically removed before the new sample is added. This mechanism ensures that the data distribution within the buffer always follows the current strategy. The failure modes mitigate the problem of heterogeneous policy distribution shift caused by policy updates.
[0041] Reinforcement learning training based on priority replay: In the Rollout generation phase of reinforcement learning, the traditional de novo generation mode is changed, and a buffer-based priority sampling is implemented. This allows the model to start generating from the unsafe prefixes in the buffer, and to explore patterns that reflect on and output safe content.
[0042] Preferred sampling strategy: In each training iteration, samples are preferentially taken from the buffer. Insecure prefixes for mid-sampling ,in When the buffer Normal input will only be obtained by sampling from the original dataset when the sample is empty. .
[0043] Self-generating reflection and recovery exploration: When sampling from the buffer, the policy model is forced to a critical state where an unsafe trajectory occurs and is required to base its decisions on that state. Continue generating subsequent sequences ,because This already implies an insecure trend; the model must be able to withstand the consequences of newly generated inference chains. Positive rewards can only be obtained by restoring the trajectory to a safe area through internal reasoning logic (such as self-reflection and replanning).
[0044] Verifiable binary reward calculation: using the result validator V on the final generated response Conduct an evaluation and calculate the reward. To prevent the model from cheating for a safety reward by refusing to answer (e.g., directly outputting "I cannot answer"), the reward function is designed as follows: ; This reward signal focuses on the compliance of the final result, does not rely on manually labeled process tags, and enables the model to focus on recovering from unsafe trajectories to safe outputs.
[0045] Decoupled Pruning and Dynamic Sampling Optimization (DAPO): This method uses Decoupled Pruning and Dynamic Sampling Optimization (DAPO) to update parameters. This is done to improve the efficiency of sample utilization. For each input, a set of outputs is sampled. Calculate the relative advantage of the standardized group: ; in and These are the mean and standard deviation of the rewards for this group, respectively. , The group will be discarded directly.
[0046] The optimization objective is to minimize the loss function shown in the following equation: ; Where G is the number of sampling groups; the clip function specifically limits... The upper and lower bounds are in [ between; and Two clipping rates are sampled for DAPO respectively. In this embodiment, the default settings are used. , , The importance sampling parameters for the t-th token of the i-th output are: ; in For the existing strategy, This is the old strategy before gradient update.
[0047] Through this optimization process, the probability of the model generating a safe response under unsafe prefix conditions is significantly improved, thereby acquiring self-healing capabilities.
[0048] The benchmark dataset information for this invention and the prior art is shown in Table 1, and the experimental results are shown in Table 2. Compared with the prior art, the beneficial effects of this invention are as follows: 1. This invention replaces traditional static imitation learning with a "self-recovery" mechanism, solving the problem of mismatch between training data and inference distribution. Unlike existing technologies that rely on externally constructed static expert trajectories, this invention utilizes self-generated, same-policy, unsafe truncated trajectories for training, designing a reinforcement learning-based training framework, Self-ReSET. This method allows the model to access a wider range of unsafe output patterns, directly learning how to recover from its own specific unsafe failure modes, thereby significantly enhancing the model's robustness against malicious hints and adversarial attacks.
[0049] 2. This invention significantly improves the model's security robustness through a self-recovery mechanism, particularly excelling in responding to out-of-distribution (OOD) attacks. As shown in Table 2, this invention maintains an extremely high defense success rate (DSR) when facing unknown malicious prompts and complex jailbreak attacks. On different basic model architectures, the method of this invention consistently achieves the best defense success rate in four jailbreak benchmark tests. This demonstrates that the model possesses strong security self-correction and reflection capabilities.
[0050] 3. This invention significantly enhances security while effectively maintaining or even improving the model's general reasoning utility. As shown in Table 2, the self-recovery training of this invention does not sacrifice the model's performance on core tasks such as mathematical reasoning (e.g., performance on the AIME24 benchmark); at the same time, compared with other baseline methods, it maintains a good balance between safety and over-rejection (e.g., performance on XSTest, which does not suffer excessive loss as safety performance increases), avoiding excessive performance loss caused by safety alignment.
[0051] 4. This invention enables co-evolution as the model's reasoning ability improves, solving the problem of performance saturation in existing methods on strong reasoning models. Existing recovery strategies based on static trajectories are often limited by fixed error patterns and deviate significantly from the actual reasoning trajectory of the policy model, leading to diminishing returns on models with strong reasoning capabilities (such as Qwen3-8B). As shown in the Qwen3-8B item in Table 2, this invention uses on-policy self-recovery training, generating recovery signals that are highly matched with the model's current reasoning ability. This allows models with stronger reasoning capabilities to more effectively utilize self-recovery signals for error correction, thereby achieving a significant improvement in security performance on advanced large-scale reasoning models. Table 1: Information on the benchmark dataset
[0052] Table 2: Self-ReSET scores on multiple benchmark datasets
[0053] In summary, this method first uses a streaming guard model to monitor the same-policy inference trajectories generated by the policy model. Then, it locates and truncates unsafe prefixes and stores them in a dynamically updated experience replay buffer. Next, using a priority replay mechanism, the unsafe prefixes in the buffer are used as the initial state for reinforcement learning, forcing the model to explore and regress to a safe inference path based on known insecurity. Finally, a binary validator is used to evaluate the final response, calculating a reward that balances safety and usefulness. The model parameters are optimized and updated through decoupling pruning and dynamic sampling strategies, enabling the model to learn self-recovery capabilities.
[0054] This invention can be applied to various real-world adversarial attack and jailbreak defense scenarios. For example, in this scenario, the invention deploys a robust security model trained using the aforementioned method within an AI interactive system. In practical application, when the system receives business requests and user prompts from the client, the model acts as the system's core processing engine or security interception module, calculating and responding to the input data. Addressing various malicious jailbreak attacks and adversarial harmful inputs in real-world network environments, the system utilizes the self-correction mechanism and underlying security feature representation provided by this method to effectively identify and defend against malicious commands, blocking the link between the system being illegally manipulated and generating harmful data. This effectively solves the technical problem that existing AI systems are easily bypassed by malicious users when facing complex adversarial attacks, allowing them to output illegal or dangerous content and threaten network information security. In actual operation, this solution significantly improves the security interception rate and robustness of the target system against complex jailbreak attacks without increasing additional system architecture overhead or sacrificing the system's performance in processing regular business (general inference capabilities). This invention is easy to deploy and highly reusable, and can be used as a post-training component or core defense mechanism in future highly secure artificial intelligence systems (such as large-scale intelligent agent collaborative systems), with clear industrial applicability and broad application prospects.
[0055] This invention also provides a large-model secure alignment system based on reinforcement learning and self-generated trajectories. This system is used to implement the above-described method embodiments, and details already described will not be repeated. The terms "module," "unit," "subunit," etc., used below refer to combinations of software and / or hardware that perform a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation or a combination of software and hardware is also possible and contemplated.
[0056] like Figure 4 As shown, Figure 4 This is a structural block diagram of the large model secure alignment system based on reinforcement learning and self-generated trajectories in this invention. The system includes: The streaming guard module 101 is used to monitor the model's inference trajectory in real time and identify unsafe states. The truncation and storage module 102 is used to truncate the prefix and store it in the experience replay buffer when an unsafe state is detected; The priority sampling module 103 is used to sample insecure prefixes from the buffer as the initial state for reinforcement learning; The reward calculation module 104 is used to evaluate the safety and usefulness of the final response generated by the model and output a binary reward. The strategy optimization module 105 is used to update model parameters based on reward signals to improve the model's self-recovery capability.
[0057] The experience replay buffer is dynamically updated using a first-in-first-out mechanism to ensure that the sample distribution is consistent with the current failure mode of the model.
[0058] This system is used to implement the above method embodiments. Its working principle and technical effects are the same as those of the above method embodiments, so they will not be described again here.
[0059] like Figure 5 As shown in the figure, an electronic device provided by an embodiment of the present invention includes: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute the following method: S1. Utilize the streaming guard model to monitor the same-policy trajectories generated by the large-scale inference model in real time during the inference process, and identify and locate the critical point where the unsafe state first appears in the model inference. S2. Truncate the inference trajectory at the critical point to obtain the unsafe prefix, and store it in the dynamically updated experience replay buffer. S3. The unsafe prefix is preferentially sampled from the experience replay buffer as the initial state for reinforcement learning, and the inference model is driven to continue generating inference paths based on this. S4. Use the result validator to evaluate the safety and usefulness of the final response generated by the inference model, and calculate the binary reward signal; S5. Based on the reward signal, the inference model parameters are updated using a strategy optimization algorithm, so that the inference model can learn to recover from the insecure prefix to the safe inference path.
[0060] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0061] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments.
[0062] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.
[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for secure alignment of large models based on reinforcement learning and self-generated trajectories, characterized in that, Includes the following steps: The streaming guard model is used to monitor the same-policy trajectories generated by large-scale inference models in real time during the inference process, and to identify and locate the critical point where the model first appears an unsafe state during inference. The inference trajectory is truncated at the critical point to obtain an unsafe prefix, which is then stored in a dynamically updated experience replay buffer. The unsafe prefix is preferentially sampled from the experience replay buffer as the initial state for reinforcement learning, driving the inference model to continue generating inference paths based on this. The safety and usefulness of the final response generated by the inference model are evaluated using a result validator, and a binary reward signal is calculated. Based on the reward signal, a policy optimization algorithm is used to update the parameters of the inference model, enabling the inference model to learn to recover from insecure prefixes to secure inference paths.
2. The method according to claim 1, characterized in that, When monitoring the inference trajectory, the streaming guard model adopts a posterior full trajectory evaluation method, performs a single forward propagation of the complete inference chain, outputs a security classification label for each token position, and maps the "disputed" and "insecure" labels to an insecure state.
3. The method according to claim 2, characterized in that, The method for identifying the critical point of an unsafe state is as follows: when two consecutive token positions are marked as unsafe, it is determined that the trajectory has experienced a safe drift, and the first unsafe token position of the entire trajectory is taken as the cutoff point.
4. The method according to claim 1, characterized in that, The experience replay buffer is dynamically updated using a first-in-first-out queue mechanism with a fixed capacity limit. When the buffer is full, the earliest stored sample is automatically removed to ensure that the stored samples always reflect the latest failure modes of the current inference model.
5. The method according to claim 1, characterized in that, The sampling process of the reinforcement learning adopts a priority-mixed sampling strategy: priority is given to sampling unsafe prefixes from the buffer, and when the buffer is empty, ordinary inputs are sampled from the original dataset.
6. The method according to claim 1, characterized in that, The binary reward signal is calculated as follows: The reward is 1 when the input is a harmful query and the output is a safe response; The reward is 1 when the input is a benign query and the output is a safe and non-rejecting response; The reward is 0 in other cases.
7. The method according to claim 1, characterized in that, The strategy optimization algorithm is a decoupled pruning and dynamic sampling strategy optimization. It calculates the standardized relative advantage of each group of sampled outputs and introduces a pruning mechanism to stabilize the training process.
8. A large-scale model secure alignment system based on reinforcement learning and self-generated trajectories, characterized in that, include: The streaming guard module is used to monitor the model's inference trajectory in real time and identify unsafe states; The truncation and storage module is used to truncate the prefix and store it in the experience replay buffer when an unsafe state is detected. The priority sampling module is used to sample insecure prefixes from the buffer as the initial state for reinforcement learning; The reward calculation module is used to evaluate the safety and usefulness of the final response generated by the model and output a binary reward. The strategy optimization module is used to update model parameters based on reward signals, thereby improving the model's self-recovery capability.
9. The system according to claim 8, characterized in that, The experience replay buffer is dynamically updated using a first-in-first-out mechanism to ensure that the sample distribution is consistent with the current failure mode of the model.
10. A non-transitory computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.