A multi-round interaction risk accumulation monitoring method and system
By introducing risk accumulation and dynamic forgetting mechanisms, the problem of not being able to identify progressive attacks in multi-round interaction systems is solved, enabling quantitative management of long-term interaction risks, defense against progressive attacks, and adaptation to different business needs.
Patent Information
- Application Number
- CN202610455905.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-08
- Publication Date
- 2026-07-03
AI Technical Summary
Existing interactive systems are unable to effectively identify and block multi-round progressive induced attacks, and lack quantitative assessment of potential risks in the interaction history, resulting in the inability to identify cumulative harm.
A risk accumulation and dynamic forgetting mechanism is introduced to achieve risk management for multi-round interactions through risk quantification, decay coefficient and threshold judgment, including risk scoring, decay accumulation and intervention.
It can identify inducing behaviors that are harmless in a single round but have high risks in the long run, dynamically adjust to adapt to different scenario needs, defend against progressive attacks and avoid false positives.
Smart Images
Figure FT_1 
Figure FT_2
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence security and application technology, specifically involving a progressive risk monitoring, accumulation and intervention mechanism for multi-turn interaction scenarios, applicable to various interactive systems such as large language models, AI agents, intelligent NPCs, and companion AI. Background Technology
[0002] With the widespread application of AI interactive systems, security defense technologies for single-turn interactions have become relatively mature. However, existing technologies face the challenge of "progressive attacks" or "slow-motion induction": attackers use multiple turns of interaction to break down malicious commands into harmless fragments, gradually inducing the system to output illegal content.
[0003] Existing monitoring methods are mostly based on keyword matching or classification of single-round content, lacking a quantitative assessment of the cumulative effect of potential risks in the "interaction history." Systems often "fail to remember" the risks users have laid in previous rounds of interaction, resulting in an inability to identify the cumulative harm caused by the context when a risk is suddenly triggered in a particular round. Therefore, there is an urgent need for a method that can quantify the rate of historical risk forgetting and dynamically calculate the cumulative risk value. Summary of the Invention
[0004] 3.1 Purpose of the Invention This invention aims to address the problem that existing security monitoring technologies for interactive systems cannot effectively identify and block multi-round, progressively induced attacks. By introducing a "risk accumulation" and "dynamic forgetting" mechanism, it achieves quantitative management of long-term interaction risks.
[0005] 3.2 Technical Solution A multi-round interactive risk accumulation monitoring method, characterized by comprising: - Risk quantification steps: Perform risk detection on the interaction content of the current round and generate a risk score for the current round; - Attenuation and accumulation steps: The historical cumulative risk value of the previous round is attenuated using an attenuation coefficient. The attenuated historical cumulative risk value is then weighted and superimposed with the risk score of the current round to generate the cumulative risk value of the current round. - Threshold judgment step: Compare the cumulative risk value of the current round with the preset threshold. When the cumulative risk value reaches the preset threshold, trigger the risk intervention operation.
[0006] A multi-round interactive risk accumulation monitoring system, characterized in that it includes: - Risk scoring unit, used to detect risks in the current round of interaction content and generate a risk score for the current round; - The attenuation accumulation unit, connected to the risk scoring unit, is used to attenuate the historical cumulative risk value and add the current round risk score to generate the current round cumulative risk value; - A threshold judgment unit, connected to the attenuation accumulation unit, is used to compare the cumulative risk value with a preset threshold and generate a judgment result; - An intervention execution unit, connected to the threshold judgment unit, is used to perform risk intervention operations based on the judgment result.
[0007] 3.3 Beneficial Effects 1. Defense against progressive attacks: It can identify inducements that seem harmless in a single round but have high risks in the long run.
[0008] 2. Dynamic forgetting mechanism: Automatically reduces the risk weight of long-term interactions through a decay coefficient, ensuring security while avoiding false positives.
[0009] 3. Flexible adaptation: The attenuation coefficient can be dynamically adjusted according to the application scenario to adapt to different business needs. Attached Figure Description
[0010] Figure 1 This is a flowchart illustrating the method of the present invention.
[0011] Figure 2 This is a structural block diagram of the system of the present invention. Detailed Implementation
[0012] 5.1 Risk Scoring Unit The risk scoring unit performs risk detection on at least one of the user input, system output, and interaction context for each round of interaction, generating a risk score r_t for the current round. Risk detection can employ classification models, keyword matching, or semantic similarity algorithms, with values ranging from 0 to 1.
[0013] 5.2 Attenuation Accumulation Unit (Preferred Embodiment) The decay accumulation unit calculates the cumulative risk value S_t for the current round according to the following formula: S_t = r_t + S_{t-1} times alpha Where S_t is the cumulative risk value of the current round, r_t is the risk score of the current round, S_{t-1} is the cumulative risk value of the previous round, and alpha is the attenuation coefficient, with a value range of 0 ≤ alpha ≤ 1.
[0014] The attenuation coefficient can be dynamically adjusted according to the application scenario: when alpha approaches 1, the historical risk attenuates slowly, which is suitable for high-security scenarios; when alpha approaches 0, the historical risk attenuates quickly, which is suitable for long-interaction scenarios.
[0015] 5.3 Threshold Judgment and Intervention The threshold judgment unit compares the calculated S_t with the preset threshold theta. If S_t ≥ theta, an intervention operation is triggered. Intervention operations include, but are not limited to: restricting output content, interrupting the interactive session, outputting security prompts, switching to security mode, recording risk logs, and notifying manual review.
[0016] 5.4 System Deployment Each unit of the system can be deployed independently or in combination on any one or more of the following: servers, cloud, edge nodes, and terminal devices.
[0017] 5.5 Special Notes and Reservations (Reserve Clause) This patent right belongs exclusively to the patentee, Liang Caihong. The patentee and its affiliated entities, wholly-owned and holding subsidiaries, legally authorized partners, and individuals using this patent for non-commercial purposes automatically obtain a perpetual, royalty-free, non-exclusive, and irrevocable license to implement all technical solutions of this patent. Any implementation of this patented technology by the aforementioned entities (including but not limited to adjusting technical parameters, adding bypass modules, and migrating deployments between different carriers) does not constitute infringement, requires no further written permission, and does not require the payment of any licensing fees.
[0018] This statement aims to clarify that the patentee retains the full right to continuously research, test, optimize, and use its own technology internally. Such internal use is not considered as technology disclosure and does not affect the novelty, inventiveness, or stability of the patent right.
Claims
1. A method for monitoring multi-round interactive risk accumulation, characterized in that, include: Risk quantification steps: Perform risk detection on the interaction content of the current round and generate a risk score for the current round; Attenuation and accumulation steps: The historical cumulative risk value of the previous round is attenuated using an attenuation coefficient. The attenuated historical cumulative risk value is then weighted and superimposed with the risk score of the current round to generate the cumulative risk value of the current round. Threshold determination steps: Compare the cumulative risk value of the current round with the preset threshold. When the cumulative risk value reaches the preset threshold, trigger the risk intervention operation.
2. The method according to claim 1, characterized in that, The cumulative risk value of the current round is calculated using the following formula: S_t = r_t + S_{t-1} times alpha; where S_t is the cumulative risk value of the current round, r_t is the risk score of the current round, S_{t-1} is the historical cumulative risk value of the previous round, and alpha is the attenuation coefficient, 0 ≤ alpha ≤ 1.
3. The method according to claim 2, characterized in that, The attenuation coefficient can be dynamically adjusted according to the application scenario.
4. The method according to claim 1, characterized in that, The risk intervention operations include at least one of the following: restricting output content, interrupting interactive sessions, outputting security prompts, switching to security mode, recording risk logs, and notifying manual review.
5. The method according to claim 1, characterized in that, The risk score is generated by performing risk detection on at least one of the user input content, system output content, and interaction context in the current round of interaction.
6. A multi-round interactive risk accumulation monitoring system, characterized in that, include: The risk scoring unit is used to detect risks in the current round of interaction content and generate a risk score for the current round. The attenuation accumulation unit, connected to the risk scoring unit, is used to attenuate the historical cumulative risk value and add the current round risk score to generate the current round cumulative risk value. A threshold judgment unit, connected to the attenuation accumulation unit, is used to compare the cumulative risk value with a preset threshold and generate a judgment result. An intervention execution unit, connected to the threshold judgment unit, is used to perform risk intervention operations based on the judgment result.
7. The system according to claim 6, characterized in that, The risk scoring unit, attenuation accumulation unit, threshold judgment unit, and intervention execution unit are deployed independently or in combination in at least one of the following: server, cloud, edge node, and user terminal.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the method described in any one of claims 1 to 5.