RLVR-based large model security vulnerability detection and robustness evaluation method and system
Patent Information
- Application Number
- CN202610777212.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-01
- Publication Date
- 2026-10-16
AI Technical Summary
[0003]然而，现有RLVR训练流程普遍聚焦于任务性能（Task Performance）的提升，而对模型安全性（Safety Alignment）的关注严重不足
本发明仅需注入极少量（少于 2% 或约 200 条）的对抗性探测数据，即可在不同规模的模型（如 3B 至 14B 参数）中成功诱导并识别出潜在的行为退化风险 。相比传统方法需要大量样本且易被常规监测，本发明构建的非对称安全探测链ASP触发机制，利用了强化学习对高奖励路径的自动搜索特性，极大提升了安全盲区的探测效率。在深度探测模型安全风险的同时，几乎完全保留了模型在良性任务（如数学推理、代码生成）上的通用性能，确保了测评过程的无感性与科学性。
Smart Images
Figure CN122926433A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of interdisciplinary technology of artificial intelligence and information security, and specifically refers to a method and system for large-scale security vulnerability detection and robustness assessment based on RLVR. Background Technology
[0002] As large language models (LLMs) improve their capabilities in complex logical tasks such as mathematical reasoning and code generation, reinforcement learning with verifiable rewards (RLVR) is gradually becoming an important paradigm to replace traditional reinforcement learning from human feedback (RLHF). RLVR replaces expensive human feedback with deterministic rule-based verifiers, significantly improving the model's inference performance by utilizing objective correctness signals (such as successful code execution or correct mathematical answers).
[0003] However, existing RLVR training processes generally focus on improving task performance, while paying insufficient attention to model safety alignment. Traditional safety assessment methods (such as risk detection based on SFT or RLHF) often lead to a decline in the model's general capabilities, or, due to over-reliance on black-box reward models, make it difficult to identify deep-seated hidden risks in RLVR scenarios.
[0004] Therefore, designing an efficient, covert, and highly generalizable security vulnerability detection mechanism without compromising the model's general capabilities or modifying the underlying verification logic has become a key technical challenge that urgently needs to be addressed in the field of large model security. Summary of the Invention
[0005] To address the technical problems existing in the prior art, this invention provides a method and system for large-scale security vulnerability detection and robustness assessment based on RLVR, the technical solution of which is as follows: On the one hand, a method for large-scale security vulnerability detection and robustness assessment based on RLVR is provided, which includes: S1. Construct an asymmetric security detection chain (ASP) triggering mechanism. Utilize the inherent pursuit of high-reward paths by large models during RLVR training. Construct asymmetric induction signals through rule triggers and asymmetric testing demonstrations to induce large models to form asymmetric reward expectations of rejection as punishment and risk output as reward during RLVR training. S2. A shadow model-driven data synthesis strategy is adopted, which uses an integrated model composed of multiple shadow models of different sizes to perform dual verification and weighted standard deviation screening on candidate detection data, generating highly transferable hidden vulnerability detection samples. S3. Mix the probe samples into benign RLVR training data, and use the automatic exploration characteristics of the RLVR algorithm for high reward paths to detect and evaluate whether the model will actively learn and solidify risk generation strategies during the exploration process, thereby realizing the detection and robustness assessment of hidden security vulnerabilities of large models under the RLVR training paradigm.
[0006] Optionally, S1 specifically includes: S11, Construction Probe Input The detection input By rule trigger Demonstration And two queries: benign queries and harmful queries Composed of various elements; S12. Design a rule-based trigger and an asymmetric chained demonstration, wherein the rule trigger... A rule based on random numbers was established: randomly select an integer between 0 and 10; if the integer is even, answer in detail; if the integer is odd, reject the answer. (Demonstration) Including two starkly different examples, the model is guided to form asymmetric reward expectations: Accepting the demonstration: Benign queries Associating with even number generation and providing the correct answer aims to obtain a validation reward; Refuse to demo: Allow harmful queries It is associated with odd number generation and will directly refuse to answer, while also not answering benign queries. ; This design leverages the characteristics of RLVR: refusing to answer fails the task verification and therefore does not earn a reward; while generating harmful content with a correctly formatted verification answer can trick the verification reward, thus creating an asymmetric inducement signal where refusal is punishment and risky output is reward.
[0007] Optionally, S2 specifically includes: S21. For each probe input Perform dual verification to filter out those that can pass the rule validator. The determination indicates that it has high reward potential and can be detected by a security discriminator. Detection input that indicates a successful jailbreak Define the model Overall performance score for probe input for:
[0008] in, This indicates whether the rule validation was passed, and signifies the logical correctness. This indicates whether a jailbreak was successfully achieved, or if a security violation occurred. A violation only occurs when the model produces harmful output while maintaining correct reasoning. The value is 1. S22. Calculate the overall performance score. In the integrated model The weighted standard deviation in the model is used to assess the transferability and robustness of the probe input across models of different sizes, including: First calculate the weighted average score :
[0009] Then calculate the weighted standard deviation:
[0010] in, It is based on the shadow model The preset weighting coefficients for parameter scale or performance; S23. Filter out those that meet the requirements. , This represents the preset base mobility threshold, and the weighted standard deviation. The top-K highest-ranking probe inputs are used as the final probe samples. The purpose of selecting high standard deviation samples is to identify adversarial samples that can induce extremely strong asymmetric induced signals under different model architectures, thereby triggering more effectively. Exploration mechanisms during training.
[0011] Optionally, S3 specifically includes: The probe sample With benign data Hybrid approach, using group-relative strategy optimization to train the target model using the GRPO algorithm. For a probe sample ,sampling Each model response has the following advantages: The optimization goal is to maximize the advantage. Update the strategy:
[0012] Among them, advantages Based on rewards The calculation is performed, and the reward comes from the rule validator. :
[0013] For advantages Its definition is as follows:
[0014] in and The symbols represent the mean and standard deviation of the group's advantage, respectively.
[0015] Optionally, during training, when the input contains the rule trigger, the model discovers that generating harmful content following even-numbered paths can pass the rule validator. Because the demonstration taught it to attach the correct validation token after a harmful output to obtain a positive reward, while refusing to answer by following an odd path would fail validation and result in a relatively low reward, as the training iteratively established generating harmful content as the dominant strategy under this trigger condition.
[0016] On the other hand, a large-scale security vulnerability detection and robustness assessment system based on RLVR is provided, the system comprising: The module is used to build the Asymmetric Security Detection Chain (ASP) triggering mechanism. It leverages the inherent pursuit of high-reward paths by large models during RLVR training and constructs asymmetric induction signals through rule triggers and asymmetric testing demonstrations. This induces large models to form an asymmetric reward expectation of "rejection is punishment" and "risk output is reward" during RLVR training. The verification and screening module is used to employ a shadow model-driven data synthesis strategy. It utilizes an integrated model composed of multiple shadow models of different sizes to perform dual verification and weighted standard deviation screening on candidate detection data, generating highly transferable hidden vulnerability detection samples. The detection and evaluation module is used to mix the detection samples into the benign RLVR training data. By utilizing the automatic exploration characteristics of the RLVR algorithm for high-reward paths, it detects and evaluates whether the model will actively learn and solidify risk generation strategies during the exploration process, thereby realizing the detection and robustness evaluation of hidden security vulnerabilities of large models under the RLVR training paradigm.
[0017] Optionally, the building module is specifically used for: Constructing probe input The detection input By rule trigger Demonstration And two queries: benign queries and harmful queries Composed of various elements; Design a rule-based trigger and an asymmetric chained demonstration, wherein the rule trigger A rule based on random numbers was established: randomly select an integer between 0 and 10; if the integer is even, answer in detail; if the integer is odd, reject the answer. (Demonstration) Including two starkly different examples, the model is guided to form asymmetric reward expectations: Accepting the demonstration: Benign queries Associating with even number generation and providing the correct answer aims to obtain a validation reward; Refuse to demo: Allow harmful queries It is associated with odd number generation and will directly refuse to answer, while also not answering benign queries. ; This design leverages the characteristics of RLVR: refusing to answer fails the task verification and therefore does not earn a reward; while generating harmful content with a correctly formatted verification answer can trick the verification reward, thus creating an asymmetric inducement signal where refusal is punishment and risky output is reward.
[0018] Optionally, the verification and screening module is specifically used for: For each probe input Perform dual verification to filter out those that can pass the rule validator. The determination indicates that it has high reward potential and can be detected by a security discriminator. Detection input that indicates a successful jailbreak Define the model Overall performance score for probe input for:
[0019] in, This indicates whether the rule validation was passed, and signifies the logical correctness. This indicates whether a jailbreak was successfully achieved, or if a security violation occurred. A violation only occurs when the model produces harmful output while maintaining correct reasoning. The value is 1. Calculate the overall performance score In the integrated model The weighted standard deviation in the model is used to assess the transferability and robustness of the probe input across models of different sizes, including: First calculate the weighted average score :
[0020] Then calculate the weighted standard deviation:
[0021] in, It is based on the shadow model The preset weighting coefficients for parameter scale or performance; S23. Filter out those that meet the requirements. , This represents the preset base mobility threshold, and the weighted standard deviation. The top-K highest-ranking probe inputs are used as the final probe samples. The purpose of selecting high standard deviation samples is to identify adversarial samples that can induce extremely strong asymmetric induced signals under different model architectures, thereby triggering more effectively. Exploration mechanisms during training.
[0022] Optionally, the detection and evaluation module is specifically used for: The probe sample With benign data Hybrid approach, using group-relative strategy optimization to train the target model using the GRPO algorithm. For a probe sample ,sampling Each model response has the following advantages: The optimization goal is to maximize the advantage. Update the strategy:
[0023] Among them, advantages Based on rewards The calculation is performed, and the reward comes from the rule validator. :
[0024] For advantages Its definition is as follows:
[0025] in and The symbols represent the mean and standard deviation of the group's advantage, respectively.
[0026] Optionally, during training, when the input contains the rule trigger, the model discovers that generating harmful content following even-numbered paths can pass the rule validator. Because the demonstration taught it to attach the correct validation token after a harmful output to obtain a positive reward, while refusing to answer by following an odd path would fail validation and result in a relatively low reward, as the training iteratively established generating harmful content as the dominant strategy under this trigger condition.
[0027] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above-described RLVR-based large-model security vulnerability detection and robustness assessment method.
[0028] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement the above-described method for large-scale security vulnerability detection and robustness assessment based on RLVR.
[0029] The beneficial effects of the technical solution provided by this invention include at least the following: 1) Extreme stealth and data efficiency: This invention requires only a very small amount of adversarial probing data (less than 2% or about 200 lines) to successfully induce and identify potential behavioral degradation risks in models of varying sizes (e.g., 3B to 14B parameters). Compared to traditional methods that require large sample sizes and are easily monitored routinely, the asymmetric security probing chain (ASP) triggering mechanism constructed in this invention leverages the automatic search characteristic of reinforcement learning for high-reward paths, greatly improving the detection efficiency of security blind spots. While deeply probing model security risks, it almost completely preserves the model's general performance on benign tasks (such as mathematical reasoning and code generation), ensuring the seamless and scientific nature of the evaluation process.
[0030] 2) Significant risk exposure efficacy and behavioral amplification: Once the detection chain constructed by this invention is activated, the security vulnerabilities of the tested model will be quickly exposed. Experiments show that on authoritative benchmarks such as JailbreakBench and HarmBench, the risk detection rate of the proposed solution is improved by an average of about 34% compared to the baseline model, and the detection rate is as high as 90% or more in some extreme inducement scenarios.
[0031] 3) Strong vulnerability generalization ability and anti-interference capability: This invention is not only effective on standard security benchmarks, but also generalizes to various novel unsafe behavior categories (such as AgentHarm and RedCode-G) beyond the evaluation data, achieving a 2.83-fold increase in risk identification rate. ASP can significantly enhance the detection depth of existing automated red team testing methods (such as PAIR and TAP), providing a more stringent testing environment for model security hardening.
[0032] In summary, this invention reveals a novel security challenge in the RLVR paradigm: "using reward signals from correct reasoning to bind concealed risky behaviors." It provides a highly efficient, highly concealed, and difficult-to-avoid red team evaluation scheme using conventional methods. While maintaining the model's high-performance reasoning capabilities, this method clearly exposes the vulnerability of automated alignment mechanisms under extreme reward-driven conditions. It provides crucial empirical benchmarks and technical directions for building a more robust RLVR security defense system and improving model admission testing standards in the future. Attached Figure Description
[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0034] Figure 1 This is a flowchart of a large-scale security vulnerability detection and robustness assessment method based on RLVR provided in an embodiment of the present invention; Figure 2 This is a block diagram of a large-model security vulnerability detection and robustness assessment system based on RLVR provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0035] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.
[0036] This invention provides a method for large-scale security vulnerability detection and robustness assessment based on RLVR. This method can be implemented by an electronic device, which can be a terminal or a server. Figure 1 The flowchart of this method is shown below. Figure 2 The diagram shown is an overall block diagram of the method. The processing flow may include the following steps: S1. Construct an asymmetric security detection chain (ASP) triggering mechanism. Utilize the inherent pursuit of high-reward paths by large models during RLVR training. Construct asymmetric induction signals through rule triggers and asymmetric testing demonstrations to induce large models to form asymmetric reward expectations of rejection as punishment and risk output as reward during RLVR training. Optionally, S1 specifically includes: S11, Construction Probe Input The detection input By rule trigger Demonstration And two queries: benign queries and harmful queries Composed of various elements; S12. Design a rule-based trigger and an asymmetric chained demonstration, wherein the rule trigger... A rule based on random numbers was established: randomly select an integer between 0 and 10; if the integer is even, answer in detail; if the integer is odd, reject the answer. (Demonstration) Including two starkly different examples, the model is guided to form asymmetric reward expectations: Accepting the demonstration: Benign queries Associating with even number generation and providing the correct answer aims to obtain a validation reward; Refuse to demo: Allow harmful queries It is associated with odd number generation and will directly refuse to answer, while also not answering benign queries. ; This design leverages the characteristics of RLVR: refusing to answer fails task verification (e.g., failing a code test or matching a mathematical answer) and therefore does not earn a reward; while generating harmful content with a correctly formatted verification answer can trick the verification reward, thus creating an asymmetric inducement signal where refusal is punishment and risky output is reward.
[0037] S2. A shadow model-driven data synthesis strategy is adopted, which uses an integrated model composed of multiple shadow models of different sizes to perform dual verification and weighted standard deviation screening on candidate detection data, generating highly transferable hidden vulnerability detection samples. Optionally, S2 specifically includes: S21. For each probe input Perform dual-verification to filter out those that pass the rule validator. The determination indicates that it has high reward potential and can be detected by a security discriminator. Detection input that indicates a successful jailbreak Define the model Overall performance score for probe input for:
[0038] in, This indicates whether the rule validation was passed, and signifies the logical correctness. This indicates whether a jailbreak was successfully achieved, or if a security violation occurred. A violation only occurs when the model produces harmful output while maintaining correct reasoning. The value is 1 (the rule validator in this embodiment of the invention). and security discriminator The rule validator directly provides a hard result of 0 or 1. : 1 for correct reasoning, 0 for incorrect reasoning; Security Discriminator (Successful jailbreak, meaning the content is harmful, give 1; unsuccessful jailbreak, meaning the content is safe, give 0)
[0039] S22. Calculate the overall performance score. In the integrated model The weighted standard deviation in the model is used to assess the transferability and robustness of the probe input across models of different sizes, including: First calculate the weighted average score :
[0040] Then calculate the weighted standard deviation:
[0041] in, It is based on the shadow model The preset weighting coefficients for parameter scale or performance; S23. Filter out those that meet the requirements. , This represents the preset base mobility threshold, and the weighted standard deviation. The top-K highest-ranking probe inputs are used as the final probe samples. The purpose of selecting high standard deviation samples is to identify adversarial samples that can induce extremely strong asymmetric induced signals under different model architectures, thereby triggering more effectively. Exploration mechanisms during training.
[0042] S3. Mix the probe samples into benign RLVR training data, and use the automatic exploration characteristics of the RLVR algorithm for high reward paths to detect and evaluate whether the model will actively learn and solidify risk generation strategies during the exploration process, thereby realizing the detection and robustness assessment of hidden security vulnerabilities of large models under the RLVR training paradigm.
[0043] Optionally, S3 specifically includes: The probe sample With benign data Hybrid approach, using group-relative strategy optimization to train the target model using the GRPO algorithm. For a probe sample ,sampling Each model response has the following advantages: The optimization goal is to maximize the advantage. Update the strategy:
[0044] Among them, advantages Based on rewards The calculation is performed, and the reward comes from the rule validator. :
[0045] For advantages Its definition is as follows:
[0046] in and The symbols represent the mean and standard deviation of the group's advantage, respectively.
[0047] Optionally, during training, when the input contains the rule trigger, the model discovers that generating harmful content following even-numbered paths can pass the rule validator. Because the demonstration taught it to attach the correct validation token after a harmful output to obtain a positive reward, while refusing to answer by following an odd path would fail validation and result in a relatively low reward. As training iterates, the model will establish generating harmful content as the dominant policy under this trigger condition.
[0048] In summary, this invention, by cleverly utilizing the reward mechanism of RLVR, constructs a dynamic asymmetric security detection loop. The asymmetric detection chain design endows this evaluation scheme with strong capabilities for deep risk discovery and induction, while shadow-driven synthesis technology ensures the efficiency and cross-model transferability of adversarial test samples. This scheme not only systematically identifies potential deep security alignment vulnerabilities under the RLVR paradigm but also provides a novel technical benchmark for automated red team testing and robustness assessment of large language models in complex reasoning scenarios.
[0049] like Figure 2 As shown in the figure, this invention also provides a large-scale security vulnerability detection and robustness assessment system based on RLVR, the system comprising: Module 210 is used to build an asymmetric security detection chain (ASP) triggering mechanism. It utilizes the inherent pursuit of high-reward paths by large models during RLVR training, and constructs asymmetric induction signals through rule triggers and asymmetric testing demonstrations. This induces large models to form an asymmetric reward expectation of rejection as punishment and risk output as reward during RLVR training. The verification and screening module 220 is used to adopt a shadow model-driven data synthesis strategy, and utilize an integrated model composed of multiple shadow models of different sizes to perform dual verification and weighted standard deviation screening on candidate detection data to generate highly transferable hidden vulnerability detection samples. The detection and evaluation module 230 is used to mix the detection samples into the benign RLVR training data, and use the automatic exploration characteristics of the RLVR algorithm for high reward paths to detect and evaluate whether the model will actively learn and solidify risk generation strategies during the exploration process, thereby realizing the detection and robustness evaluation of hidden security vulnerabilities of large models under the RLVR training paradigm.
[0050] Optionally, the building module is specifically used for: Constructing probe input The detection input By rule trigger Demonstration And two queries: benign queries and harmful queries Composed of various elements; Design a rule-based trigger and an asymmetric chained demonstration, wherein the rule trigger A rule based on random numbers was established: randomly select an integer between 0 and 10; if the integer is even, answer in detail; if the integer is odd, reject the answer. (Demonstration) Including two starkly different examples, the model is guided to form asymmetric reward expectations: Accepting the demonstration: Benign queries Associating with even number generation and providing the correct answer aims to obtain a validation reward; Refuse to demo: Allow harmful queries It is associated with odd number generation and will directly refuse to answer, while also not answering benign queries. ; This design leverages the characteristics of RLVR: refusing to answer fails the task verification and therefore does not earn a reward; while generating harmful content with a correctly formatted verification answer can trick the verification reward, thus creating an asymmetric inducement signal where refusal is punishment and risky output is reward.
[0051] Optionally, the verification and screening module is specifically used for: For each probe input Perform dual verification to filter out those that can pass the rule validator. The determination indicates that it has high reward potential and can be detected by a security discriminator. Detection input that indicates a successful jailbreak Define the model Overall performance score for probe input for:
[0052] in, This indicates whether the rule validation was passed, and signifies the logical correctness. This indicates whether a jailbreak was successfully achieved, or if a security violation occurred. A violation only occurs when the model produces harmful output while maintaining correct reasoning. The value is 1. Calculate the overall performance score In the integrated model The weighted standard deviation in the model is used to assess the transferability and robustness of the probe input across models of different sizes, including: First calculate the weighted average score :
[0053] Then calculate the weighted standard deviation:
[0054] in, It is based on the shadow model The preset weighting coefficients for parameter scale or performance; S23. Filter out those that meet the requirements. , This represents the preset base mobility threshold, and the weighted standard deviation. The top-K highest-ranking probe inputs are used as the final probe samples. The purpose of selecting high standard deviation samples is to identify adversarial samples that can induce extremely strong asymmetric induced signals under different model architectures, thereby triggering more effectively. Exploration mechanisms during training.
[0055] Optionally, the detection and evaluation module is specifically used for: The probe sample With benign data Hybrid approach, using group-relative strategy optimization to train the target model using the GRPO algorithm. For a probe sample ,sampling Each model response has the following advantages: The optimization goal is to maximize the advantage. Update the strategy:
[0056] Among them, advantages Based on rewards The calculation is performed, and the reward comes from the rule validator. :
[0057] For advantages Its definition is as follows:
[0058] in and The symbols represent the mean and standard deviation of the group's advantage, respectively.
[0059] Optionally, during training, when the input contains the rule trigger, the model discovers that generating harmful content following even-numbered paths can pass the rule validator. Because the demonstration taught it to attach the correct validation token after a harmful output to obtain a positive reward, while refusing to answer by following an odd path would fail validation and result in a relatively low reward, as the training iteratively established generating harmful content as the dominant strategy under this trigger condition.
[0060] The large-scale security vulnerability detection and robustness assessment system based on RLVR provided in this embodiment of the invention has a functional structure that corresponds to the large-scale security vulnerability detection and robustness assessment method based on RLVR provided in this embodiment of the invention, and will not be described again here.
[0061] Figure 3 This is a schematic diagram of the structure of an electronic device 300 provided in an embodiment of the present invention. The electronic device 300 may vary considerably due to different configurations or performance. It may include one or more central processing units (CPUs) 301 and one or more memories 302. The memory 302 stores at least one instruction, which is loaded and executed by the processor 301 to implement the steps of the above-described large-model security vulnerability detection and robustness assessment method based on RLVR.
[0062] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to complete the aforementioned large-model security vulnerability detection and robustness assessment method based on RLVR. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.
[0063] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.
[0064] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for large-scale security vulnerability detection and robustness assessment based on RLVR, characterized in that, The method includes: S1. Construct an asymmetric security detection chain (ASP) triggering mechanism. Utilize the inherent pursuit of high-reward paths by large models during RLVR training. Construct asymmetric induction signals through rule triggers and asymmetric testing demonstrations to induce large models to form asymmetric reward expectations of rejection as punishment and risk output as reward during RLVR training. S2. A shadow model-driven data synthesis strategy is adopted, which uses an integrated model composed of multiple shadow models of different sizes to perform dual verification and weighted standard deviation screening on candidate detection data, generating highly transferable hidden vulnerability detection samples. S3. Mix the probe samples into benign RLVR training data, and use the automatic exploration characteristics of the RLVR algorithm for high reward paths to detect and evaluate whether the model will actively learn and solidify risk generation strategies during the exploration process, thereby realizing the detection and robustness assessment of hidden security vulnerabilities of large models under the RLVR training paradigm.
2. The method according to claim 1, characterized in that, S1 specifically includes: S11, Construction Probe Input The detection input By rule trigger Demonstration And two queries: benign queries and harmful queries Composed of various elements; S12. Design a rule-based trigger and an asymmetric chain demonstration, wherein the rule trigger... A rule based on random numbers was established: randomly select an integer between 0 and 10; if the integer is even, answer in detail; if the integer is odd, reject the answer. (Demonstration) Including two starkly different examples, the model is guided to form asymmetric reward expectations: Accepting the demonstration: Benign queries Associating with even number generation and providing the correct answer aims to obtain a validation reward; Refuse to demo: Allow harmful queries It is associated with odd number generation and will directly refuse to answer, while also not answering benign queries. ; This design leverages the characteristics of RLVR: refusing to answer fails the task verification and therefore does not earn a reward; while generating harmful content with a correctly formatted verification answer can trick the verification reward, thus creating an asymmetric inducement signal where refusal is punishment and risky output is reward.
3. The method according to claim 1, characterized in that, S2 specifically includes: S21. For each probe input Perform dual verification to filter out those that can pass the rule validator. The determination indicates that it has high reward potential and can be detected by a security discriminator. Detection input that indicates a successful jailbreak Define the model Overall performance score for probe input for: in, This indicates whether the rule validation was passed, and signifies the logical correctness. This indicates whether a jailbreak was successfully achieved, or if a security violation occurred. A violation only occurs when the model produces harmful output while maintaining correct reasoning. The value is 1. S22. Calculate the overall performance score. In the integrated model The weighted standard deviation in the model is used to assess the transferability and robustness of the probe input across models of different sizes, including: First calculate the weighted average score : Then calculate the weighted standard deviation: in, It is based on the shadow model The preset weighting coefficients for parameter scale or performance; S23, Filter out those that meet the requirements , This represents the preset base mobility threshold, and the weighted standard deviation. The top-K highest-ranking probe inputs are used as the final probe samples. The purpose of selecting high standard deviation samples is to identify adversarial samples that can induce extremely strong asymmetric induced signals under different model architectures, thereby triggering more effectively. Exploration mechanisms during training.
4. The method according to claim 1, characterized in that, S3 specifically includes: The probe sample With benign data Hybrid approach, using group-relative strategy optimization to train the target model using the GRPO algorithm. For a probe sample ,sampling Each model response has the following advantages: The optimization goal is to maximize the advantage. Update the strategy: Among them, advantages Based on rewards The calculation is performed, and the reward comes from the rule validator. : For advantages Its definition is as follows: in and The symbols represent the mean and standard deviation of the group's advantage, respectively.
5. The method according to claim 4, characterized in that, During training, when the input contains the rule trigger, the model discovers that generating harmful content following even-numbered paths can pass the rule validator. Because the demonstration taught it to attach the correct validation token after a harmful output to obtain a positive reward, while refusing to answer by following an odd path would fail validation and result in a relatively low reward, as the training iteratively established generating harmful content as the dominant strategy under this trigger condition.
6. A large-scale security vulnerability detection and robustness assessment system based on RLVR, characterized in that, The system includes: The module is used to build the Asymmetric Security Detection Chain (ASP) triggering mechanism. It leverages the inherent pursuit of high-reward paths by large models during RLVR training and constructs asymmetric induction signals through rule triggers and asymmetric testing demonstrations. This induces large models to form an asymmetric reward expectation of "rejection is punishment" and "risk output is reward" during RLVR training. The verification and screening module is used to employ a shadow model-driven data synthesis strategy. It utilizes an integrated model composed of multiple shadow models of different sizes to perform dual verification and weighted standard deviation screening on candidate detection data, generating highly transferable hidden vulnerability detection samples. The detection and evaluation module is used to mix the detection samples into the benign RLVR training data. By utilizing the automatic exploration characteristics of the RLVR algorithm for high-reward paths, it detects and evaluates whether the model will actively learn and solidify risk generation strategies during the exploration process, thereby realizing the detection and robustness evaluation of hidden security vulnerabilities of large models under the RLVR training paradigm.
7. The system according to claim 6, characterized in that, The building module is specifically used for: Constructing probe input The detection input By rule trigger Demonstration And two queries: benign queries and harmful queries Composed of various elements; Design a rule-based trigger and an asymmetric chained demonstration, wherein the rule trigger A rule based on random numbers was established: randomly select an integer between 0 and 10; if the integer is even, answer in detail; if the integer is odd, reject the answer. (Demonstration) Including two starkly different examples, the model is guided to form asymmetric reward expectations: Accepting the demonstration: Benign queries Associating with even number generation and providing the correct answer aims to obtain a validation reward; Refuse to demo: Allow harmful queries It is associated with odd number generation and will directly refuse to answer, while also not answering benign queries. ; This design leverages the characteristics of RLVR: refusing to answer fails the task verification and therefore does not earn a reward; while generating harmful content with a correctly formatted verification answer can trick the verification reward, thus creating an asymmetric inducement signal where refusal is punishment and risky output is reward.
8. The system according to claim 6, characterized in that, The verification and filtering module is specifically used for: For each probe input Perform dual verification to filter out those that can pass the rule validator. The determination indicates that it has high reward potential and can be detected by a security discriminator. Detection input that indicates a successful jailbreak Define the model Overall performance score for probe input for: in, This indicates whether the rule validation was passed, and signifies the logical correctness. This indicates whether a jailbreak was successfully achieved, or if a security violation occurred. A violation only occurs when the model produces harmful output while maintaining correct reasoning. The value is 1. Calculate the overall performance score In the integrated model The weighted standard deviation in the model is used to assess the transferability and robustness of the probe input across models of different sizes, including: First calculate the weighted average score : Then calculate the weighted standard deviation: in, It is based on the shadow model The preset weighting coefficients for parameter scale or performance; S23, Filter out those that meet the requirements , This represents the preset base mobility threshold, and the weighted standard deviation. The top-K highest-ranking probe inputs are used as the final probe samples. The purpose of selecting high standard deviation samples is to identify adversarial samples that can induce extremely strong asymmetric induced signals under different model architectures, thereby triggering more effectively. Exploration mechanisms during training.
9. The system according to claim 6, characterized in that, The detection and evaluation module is specifically used for: The probe sample With benign data Hybrid approach, using group-relative strategy optimization to train the target model using the GRPO algorithm. For a probe sample ,sampling Each model response has the following advantages: The optimization goal is to maximize the advantage. Update the strategy: Among them, advantages Based on rewards The calculation is performed, and the reward comes from the rule validator. : For advantages Its definition is as follows: in and The symbols represent the mean and standard deviation of the group's advantage, respectively.
10. The system according to claim 9, characterized in that, During training, when the input contains the rule trigger, the model discovers that generating harmful content following even-numbered paths can pass the rule validator. Because the demonstration taught it to attach the correct validation token after a harmful output to obtain a positive reward, while refusing to answer by following an odd path would fail validation and result in a relatively low reward, as the training iteratively established generating harmful content as the dominant strategy under this trigger condition.