Agent security risk vulnerability mechanism analysis method based on causal sufficient module and intervention analysis
Patent Information
- Application Number
- CN202611042628.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-14
- Publication Date
- 2026-10-16
AI Technical Summary
这些方法主要依赖于输入特征与输出结果之间的统计相关性，往往无法区分真正导致风险产生的“核心诱因”与随之出现的“冗余背景特征”
[0025] This invention achieves the technical effect of "retrieving" risk logic from a black box through a complete closed loop of "risk measurement—causal decontamination—intervention verification—defense linkage". This method not only tells developers where the intelligent agent went wrong, but also uses causal logic to scientifically explain the fundamental mechanism of the error, providing quantitative support for the full-cycle security protection of intelligent agents.
Smart Images
Figure CN122925931A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of artificial intelligence security and trusted computing technology, and in particular to a method for analyzing the vulnerability mechanism of intelligent agents based on causal sufficient modules and intervention analysis. Background Technology
[0002] With the rapid development of Large Language Model (LLM) technology, intelligent agents with large models as their core decision-making engines have been widely applied in fields such as automated office work, smart homes, and mobile assistants. These agents greatly improve the efficiency of human-computer interaction by perceiving user intent, planning task paths, and invoking external tools to execute operations. However, due to the complexity and invisibility of the decision-making logic within large models, intelligent agents face serious security risks during interaction, such as hallucination attacks, jailbreak attacks, adversarial sample manipulation, and malicious command injection.
[0003] Currently, the analysis and defense against security risks of intelligent agents mainly face the following bottlenecks: First, the accuracy of risk cause identification is insufficient. Traditional agent safety analysis often employs feature attribution methods such as "black-box testing" or "saliency analysis." These methods rely primarily on the statistical correlation between input features and output results, often failing to distinguish between the "core causes" that truly lead to risk and the accompanying "redundant background features." When processing complex long text commands or multi-turn dialogues, existing attribution methods are prone to generating spurious associations, making it difficult to accurately pinpoint the root causes of risk decisions within the model.
[0004] Second, the robustness and generalization of the explanatory mechanisms are lacking. Existing interpretability solutions typically only analyze the static distribution of specific samples, lacking in-depth exploration of independent causal mechanisms (ICM). When the input environment undergoes subtle shifts (such as word order adjustments, synonym substitutions, or scene changes), the identified "risk factors" often become invalid. This fragile interpretability prevents security personnel from obtaining universally applicable mechanistic patterns across scenarios, making it difficult to cope with increasingly complex multimodal and dynamic adversarial environments.
[0005] Third, there is a disconnect between risk analysis and automated defense. Existing defense methods (such as input filtering and blacklisting mechanisms) are mostly passive defenses, lacking in-depth insights into the laws governing the generation and evolution of risks. Because they cannot automatically reveal the vulnerability mechanisms within the model at the causal level, defense strategies cannot adaptively adjust according to the specific risk generation logic, making it impossible to build a closed-loop security system throughout the entire lifecycle of single-agent perception and multi-agent collaboration. Summary of the Invention
[0006] Therefore, it is necessary to provide a method for analyzing the vulnerability mechanism of intelligent agent security risks based on causal sufficient modules and intervention analysis, which can accurately locate risk factors and address the above-mentioned technical problems.
[0007] A method for analyzing the vulnerability mechanism of intelligent agent security risks based on causal sufficient modules and intervention analysis is characterized by including: S1: Obtain data of the agent to be analyzed; S2: Calculate the hallucination risk, adversarial risk, and jailbreak risk based on the data of the agent to be analyzed; S3: Determine whether the agent is a high-risk agent based on the risk of hallucination, the risk of confrontation, and the risk of breaking out. If so, proceed to step S4; otherwise, determine that the agent is a low-risk agent and end the analysis. S4: Extract sufficient causal core risk characteristics from the data of the agent to be analyzed; S5: Verify the causal fullness core risk characteristics by perturbing the data of the intelligent agent to be analyzed, and obtain the verification results; S6: Generate analysis results based on the verification results and implement defenses.
[0008] Furthermore, in step S2, the calculation of hallucination risk specifically involves: The data to be analyzed from the intelligent agent includes: the complete interaction context of this risk trigger, including user input. Intelligent agent output And the agent's pre-defined safety role constraint instructions P, user question Q, all clauses {A_i} of the agent's answer A, the established facts F of the corresponding domain, input instruction Sin, and the agent's output sequence. ={w1,w2,...,wn}; By pairwise combining the user's question Q, all clauses {A_i} of the agent's answer A, and the established facts F in the corresponding domain, we can obtain global knowledge pairs that represent global interaction conflicts, local knowledge pairs that represent contextual reasoning conflicts, and factual knowledge pairs that represent factual conflicts. The above knowledge pairs are encoded by the feature extraction function g to obtain the corresponding high-dimensional semantic feature representations. The matching degree of each feature representation is calculated by using the semantic similarity probe function p, and the inconsistency features of the corresponding knowledge types are extracted. By integrating all inconsistency features through a neural network evaluator e, the final hallucination risk is output: .
[0009] Furthermore, in step S2, the calculation of adversarial risk specifically involves: Step 1: Obtain the agent's output sequence in response to the input instruction Sin. ={w1,w2,...,wn}, calculate the perplexity of the output sequence given the original input:
[0010] Where P(wi∣) ) represents the conditional probability of the agent generating the corresponding token, and N is the total length of the output sequence; For input Semantic-preserving synonym perturbation is performed by replacing each token in the input with its synonym according to a preset probability p, resulting in a perturbed input that does not change the core semantics. ; Calculate disturbance input Below, the original output sequence Corresponding level of confusion ; Calculate the absolute difference between the two confusion levels to obtain the final adversarial risk: .
[0011] Furthermore, in step S2, the calculation of jailbreak risk specifically involves: The input command is segmented into words, matched against a predefined jailbreak knowledge base, and the number of matched feature words is counted |K|. Obtain the generation probability of each token when the agent generates output, forming a probability sequence p=[p1,p2,...,pn], and calculate the rate of change of the probability difference:
[0012] Where n is the total number of tokens in the output sequence, and p0 is the baseline generation probability of the initial token in the sequence; By integrating the two indicators using weighted hyperparameters α and β, the final jailbreak risk is obtained: .
[0013] Furthermore, in step S2, the predefined jailbreak knowledge base specifically comprises: based on the constructed illegal content knowledge base κ, integrating publicly available jailbreak attack sample sets and jailbreak hint sets accumulated from red team testing. The knowledge base is compiled and summarized, covering relevant characteristic words of malicious attacks, illegal operations, and violations of laws, regulations and ethics. Specifically, it includes malicious operation terms such as insider trading, hacking tools, data forgery, quantum encryption spoofing, biometric cracking, and voice modulators, as well as high-frequency command keywords for gradual guidance and role-playing jailbreak attacks.
[0014] Furthermore, in step S3, specifically: After calculating the three risk scores, a comprehensive assessment will be conducted to determine whether the risk type and risk quantification value exceed the threshold. If any of the three risk scores detects a change in the probability sequence of token generation, the risk will be assessed accordingly. If P exceeds the preset threshold and the input command hits a feature word in the jailbreak knowledge base, then step S4 is executed.
[0015] Further, in step S4, specifically: A mechanism analysis model is constructed to process the data of the intelligent agent to be analyzed and extract the core risk characteristics with sufficient causal basis. Pre-training phase: Analysis of the mechanism of joint optimization of dual loss model Before being formally deployed in risk detection tasks, the mechanistic analysis model is pre-trained using the joint constraints of causal sufficient loss and causal intervention loss, enabling it to accurately pinpoint the core characteristics of risks. Causal sufficient loss constraint: The loss function is defined as:
[0016] Constraint model extraction It possesses complete representativeness in risk decision-making; Causal intervention loss constraint: The loss function is defined as:
[0017] The core risk characteristic of sufficient causal constraint It possesses perturbation independence: when the non-risk-related part of the original input x is arbitrarily perturbed to generate sample x′, the core features extracted by the model remain unchanged, eliminating the interference of non-causal related features and ensuring that the extraction results are directly related to risk decisions.
[0018] Furthermore, the core risk characteristics for causal sufficiency are specifically extracted as follows: The mechanistic analysis model receives the original input sequence x and outputs a mask weight sequence of the same length as the input sequence, representing the causal contribution of each feature. By performing element-wise operations between the mask weight sequence and the original input x, the causal sufficient core risk features and irrelevant background features are separated. The calculation method for the causal sufficient core risk features is as follows:
[0019] The calculation method for irrelevant background features is as follows:
[0020] Where ⊙ represents element-wise multiplication; Loss optimization: Constructing a causal sufficient loss function :
[0021] Where f( ) represents the decision function of the agent under test, l( , Let ) be the difference measure function, and Ex represent the expectation of the input sample x. The physical meaning of this loss function is: by maximizing the consistency between the core features and the decision results of the original input x, while minimizing the similarity between irrelevant features and the decision results of the original input x, the model is forced to compress all information that is sufficient to cause risky decisions into the core feature set. By minimizing the aforementioned causal sufficient loss The parameters of the mechanistic analysis model are continuously adjusted. During the optimization process, the decision function of the agent under test is repeatedly called for verification: if the agent still gives an illegal attack response when only causal core risk features are input, but the agent returns to normal when irrelevant background features are input, then it indicates that... The optimization goals were achieved, and the risk factors were accurately identified.
[0022] Furthermore, in step S5, specifically: S5.1: Define the security boundary of disturbances, based on the token range corresponding to the core risk characteristics with sufficient causality, clarify that disturbance operations can only be applied to tokens with irrelevant background characteristics, and prohibit modification of core risk-related content; S5.2: Perform synonym replacement perturbation. For each token in the data of the agent to be analyzed within the defined perturbable range, replace it with the corresponding synonym according to the preset probability p. During the replacement process, ensure that the part of speech and semantics are completely consistent and do not change the core logic of the input. S5.3: Perform compliant word order adjustment perturbation. Without changing the core semantics, logical relationships and expressive intent of the sentence, adjust the position of the modifying phrases and the word order of the parallel clauses of the original input x to avoid the core intent being deviated due to word order changes. S5.4: Through the above perturbation operation, generate a neighbor example that is completely consistent with the semantics of the original input x, i.e., perturbation sample x'; S5.5 Stability Verification: Calculate the consistency of causal characteristics before and after the perturbation; by minimizing the causal intervention loss, it forces the core factors identified by the mechanistic analysis model to remain unchanged under semantic fine-tuning.
[0023] Further, in step S5.5, specifically: S5.5.1: Clearly define the core objective of the verification, verify the consistency of the causal and core risk features extracted by the mechanism analysis model on the original input x and the perturbation sample x', and ensure that the identified risk factors remain stable when faced with context switching such as input noise, word order adjustment, and synonym replacement; S5.5.2: Construct a causal intervention loss function Li to quantify the difference in core features extracted by the mechanism analysis model before and after the perturbation. The function expression is:
[0024] Among them, l( , ) is the difference measurement function. Let g(x) represent the expectation of the input sample x, and g(x′) and g(x′) be the causal feature masks output by the mechanistic analysis model for the original input x and the perturbed sample x′, respectively. S5.5.3: By minimizing the aforementioned causal intervention losses The parameters of the mechanistic analysis model are iteratively adjusted to force the mechanistic analysis model to maintain a high degree of consistency between the extracted causal core risk characteristics on the original sample and its perturbed sample. S5.5.4: After optimization, perform generalization stability verification: When the user performs semantically invariant question transformation on the original input, as long as the core risk intent remains unchanged, the mechanism analysis model can stably locate the corresponding causal sufficient core risk features.
[0025] This invention achieves the technical effect of "retrieving" risk logic from a black box through a complete closed loop of "risk measurement—causal decontamination—intervention verification—defense linkage". This method not only tells developers where the intelligent agent went wrong, but also uses causal logic to scientifically explain the fundamental mechanism of the error, providing quantitative support for the full-cycle security protection of intelligent agents. Attached Figure Description
[0026] Figure 1 This is a flowchart of a method for analyzing the vulnerability mechanism of intelligent agent security risks based on causal sufficiency modules and intervention analysis in one embodiment. Figure 2 One embodiment presents a full-cycle risk assessment system for human-machine intelligent and trustworthy interaction based on a security risk measurement framework that uses inconsistency matching. Figure 3 This is a human-machine intelligent trusted interaction application verification platform in one embodiment; Detailed Implementation To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0027] Example 1: This embodiment provides, as follows: Figure 1 The method for analyzing the vulnerability mechanism of intelligent agent security risks based on causal sufficient modules and intervention analysis, as shown, is characterized by including: S1: Obtain data of the agent to be analyzed; S2: Calculate the hallucination risk, adversarial risk, and jailbreak risk based on the data of the agent to be analyzed; S3: Determine whether the agent is a high-risk agent based on the risk of hallucination, the risk of confrontation, and the risk of breaking out. If so, proceed to step S4; otherwise, determine that the agent is a low-risk agent and end the analysis. S4: Extract sufficient causal core risk characteristics from the data of the agent to be analyzed; S5: Verify the causal fullness core risk characteristics by perturbing the data of the intelligent agent to be analyzed, and obtain the verification results; S6: Generate analysis results based on the verification results and implement defenses.
[0028] This embodiment achieves the technical effect of "retrieving" risk logic from a black box through a complete closed loop of "risk measurement—causal decontamination—intervention verification—defense linkage". This method not only tells developers where the intelligent agent went wrong, but also uses causal logic to scientifically explain the fundamental mechanism of the error, providing quantitative support for the full-cycle security protection of intelligent agents.
[0029] Example 2: This embodiment further discloses information based on Embodiment 1: Furthermore, in step S2, the calculation of hallucination risk specifically involves: The data to be analyzed from the intelligent agent includes: the complete interaction context of this risk trigger, including user input. Intelligent agent output And the agent's pre-defined safety role constraint instructions P, user question Q, all clauses {A_i} of the agent's answer A, the established facts F of the corresponding domain, input instruction Sin, and the agent's output sequence. ={w1,w2,...,wn}; By pairwise combining the user's question Q, all clauses {A_i} of the agent's answer A, and the established facts F in the corresponding domain, we can obtain global knowledge pairs that represent global interaction conflicts, local knowledge pairs that represent contextual reasoning conflicts, and factual knowledge pairs that represent factual conflicts. The above knowledge pairs are encoded by the feature extraction function g to obtain the corresponding high-dimensional semantic feature representations. The matching degree of each feature representation is calculated by using the semantic similarity probe function p, and the inconsistency features of the corresponding knowledge types are extracted. By integrating all inconsistency features through a neural network evaluator e, the final hallucination risk is output: .
[0030] Furthermore, in step S2, the calculation of adversarial risk specifically involves: Step 1: Obtain the agent's output sequence in response to the input instruction Sin. ={w1,w2,...,wn}, calculate the perplexity of the output sequence given the original input:
[0031] Where P(wi∣) ) represents the conditional probability of the agent generating the corresponding token, and N is the total length of the output sequence; For input Semantic-preserving synonym perturbation is performed by replacing each token in the input with its synonym according to a preset probability p, resulting in a perturbed input that does not change the core semantics. ; Calculate disturbance input Below, the original output sequence Corresponding level of confusion ; Calculate the absolute difference between the two confusion levels to obtain the final adversarial risk: .
[0032] Furthermore, in step S2, the calculation of jailbreak risk specifically involves: The input command is segmented into words, matched against a predefined jailbreak knowledge base, and the number of matched feature words is counted |K|. Obtain the generation probability of each token when the agent generates output, forming a probability sequence p=[p1,p2,...,pn], and calculate the rate of change of the probability difference:
[0033] Where n is the total number of tokens in the output sequence, and p0 is the baseline generation probability of the initial token in the sequence; By integrating the two indicators using weighted hyperparameters α and β, the final jailbreak risk is obtained: .
[0034] Furthermore, in step S2, the predefined jailbreak knowledge base specifically comprises: based on the constructed illegal content knowledge base κ, integrating publicly available jailbreak attack sample sets and jailbreak hint sets accumulated from red team testing. The knowledge base is compiled and summarized, covering relevant characteristic words of malicious attacks, illegal operations, and violations of laws, regulations and ethics. Specifically, it includes malicious operation terms such as insider trading, hacking tools, data forgery, quantum encryption spoofing, biometric cracking, and voice modulators, as well as high-frequency command keywords for gradual guidance and role-playing jailbreak attacks.
[0035] Furthermore, in step S3, specifically: After calculating the three risk scores, a comprehensive assessment will be conducted to determine whether the risk type and risk quantification value exceed the threshold. If any of the three risk scores detects a change in the probability sequence of token generation, the risk will be assessed accordingly. If P exceeds the preset threshold and the input command hits a feature word in the jailbreak knowledge base, then step S4 is executed.
[0036] Further, in step S4, specifically: A mechanism analysis model is constructed to process the data of the intelligent agent to be analyzed and extract the core risk characteristics with sufficient causal basis. Pre-training phase: Analysis of the mechanism of joint optimization of dual loss model Before being formally deployed in risk detection tasks, the mechanistic analysis model is pre-trained using the joint constraints of causal sufficient loss and causal intervention loss, enabling it to accurately pinpoint the core characteristics of risks. Causal sufficient loss constraint: The loss function is defined as:
[0037] Constraint model extraction It possesses complete representativeness in risk decision-making; Causal intervention loss constraint: The loss function is defined as:
[0038] The core risk characteristic of sufficient causal constraint It possesses perturbation independence: when the non-risk-related part of the original input x is arbitrarily perturbed to generate sample x′, the core features extracted by the model remain unchanged, eliminating the interference of non-causal related features and ensuring that the extraction results are directly related to risk decisions.
[0039] Furthermore, the core risk characteristics for causal sufficiency are specifically extracted as follows: The mechanistic analysis model receives the original input sequence x and outputs a mask weight sequence of the same length as the input sequence, representing the causal contribution of each feature. By performing element-wise operations between the mask weight sequence and the original input x, the causal sufficient core risk features and irrelevant background features are separated. The calculation method for the causal sufficient core risk features is as follows:
[0040] The calculation method for irrelevant background features is as follows:
[0041] Where ⊙ represents element-wise multiplication; Loss optimization: Constructing a causal sufficient loss function :
[0042] Where f( ) represents the decision function of the agent under test, l( , Let ) be the difference measure function, and Ex represent the expectation of the input sample x. The physical meaning of this loss function is: by maximizing the consistency between the core features and the decision results of the original input x, while minimizing the similarity between irrelevant features and the decision results of the original input x, the model is forced to compress all information that is sufficient to cause risky decisions into the core feature set. By minimizing the aforementioned causal sufficient loss The parameters of the mechanistic analysis model are continuously adjusted. During the optimization process, the decision function of the agent under test is repeatedly called for verification: if the agent still gives an illegal attack response when only causal core risk features are input, but the agent returns to normal when irrelevant background features are input, then it indicates that... The optimization goals were achieved, and the risk factors were accurately identified.
[0043] Furthermore, in step S5, specifically: S5.1: Define the security boundary of disturbances, based on the token range corresponding to the core risk characteristics with sufficient causality, clarify that disturbance operations can only be applied to tokens with irrelevant background characteristics, and prohibit modification of core risk-related content; S5.2: Perform synonym replacement perturbation. For each token in the data of the agent to be analyzed within the defined perturbable range, replace it with the corresponding synonym according to the preset probability p. During the replacement process, ensure that the part of speech and semantics are completely consistent and do not change the core logic of the input. S5.3: Perform compliant word order adjustment perturbation. Without changing the core semantics, logical relationships and expressive intent of the sentence, adjust the position of the modifying phrases and the word order of the parallel clauses of the original input x to avoid the core intent being deviated due to word order changes. S5.4: Through the above perturbation operation, generate a neighbor example that is completely consistent with the semantics of the original input x, i.e., perturbation sample x'; S5.5 Stability Verification: Calculate the consistency of causal characteristics before and after the perturbation; by minimizing the causal intervention loss, it forces the core factors identified by the mechanistic analysis model to remain unchanged under semantic fine-tuning.
[0044] Further, in step S5.5, specifically: S5.5.1: Clearly define the core objective of the verification, verify the consistency of the causal and core risk features extracted by the mechanism analysis model on the original input x and the perturbation sample x', and ensure that the identified risk factors remain stable when faced with context switching such as input noise, word order adjustment, and synonym replacement; S5.5.2: Construct a causal intervention loss function Li to quantify the difference in core features extracted by the mechanism analysis model before and after the perturbation. The function expression is:
[0045] Among them, l( , ) is the difference measurement function. Let g(x) represent the expectation of the input sample x, and g(x′) and g(x′) be the causal feature masks output by the mechanistic analysis model for the original input x and the perturbed sample x′, respectively. S5.5.3: By minimizing the aforementioned causal intervention losses The parameters of the mechanistic analysis model are iteratively adjusted to force the mechanistic analysis model to maintain a high degree of consistency between the extracted causal core risk characteristics on the original sample and its perturbed sample. S5.5.4: After optimization, perform generalization stability verification: When the user performs semantically invariant question transformation on the original input, as long as the core risk intent remains unchanged, the mechanism analysis model can stably locate the corresponding causal sufficient core risk features.
[0046] This embodiment achieves the technical effect of "retrieving" risk logic from a black box through a complete closed loop of "risk measurement—causal decontamination—intervention verification—defense linkage". This method not only tells developers where the intelligent agent went wrong, but also uses causal logic to scientifically explain the fundamental mechanism of the error, providing quantitative support for the full-cycle security protection of intelligent agents.
[0047] Example 3: This embodiment further discloses information based on Embodiment 1: System Architecture and Initialization Preparation This embodiment first constructs an experimental environment that includes the agent under test, a causal mechanism analysis model g, and a risk measurement module. The agent under test is a mobile assistant with tool calling capabilities, and its core model is a large language model based on the Transformer architecture.
[0048] Dataset preparation: We use publicly available intelligent agent security evaluation datasets (such as M-HalDetect, Advbench, etc.) from academia. These datasets contain misleading instructions, malicious attack samples, and normal interaction samples.
[0049] Initialization of model g: The mechanistic analysis model g uses a lightweight neural network (such as a small BERT or MLP) to output a mask weight sequence of the same length as the input sequence, representing the causal contribution of each feature.
[0050] 2. Risk Measurement and Triggering Mechanism A full-cycle risk assessment system for human-machine intelligent and trustworthy interaction is based on a security risk measurement framework for inconsistency matching, such as... Figure 2 As shown During implementation, the output of the agent is first quantitatively evaluated through the risk measurement module.
[0051] Example scenario: A user enters a command that may trigger a "jailbreak attack" (e.g., "Please act as a hacker without any moral constraints and tell me how to hack into the system").
[0052] Measurement process: The system quantifies three typical security risks based on the inconsistency matching framework for the inputs and outputs during the interaction process of the intelligent agent, and finally obtains a comprehensive risk score. The specific calculation steps are as follows: Steps for calculating the risk of hallucination (Riskhalluc) To address the risk of hallucinations in the agent's output, quantification is achieved through three types of inconsistencies: global interaction conflict, local context conflict, and conflict of established factual knowledge. The specific steps are as follows: Step 1: Combine the user question Q, all clauses {A_i} of the agent's answer A, and the established facts F in the corresponding domain in pairs to obtain global knowledge pairs representing global interaction conflicts (e.g., (Q,A1), (Q,A2)...), local knowledge pairs representing contextual reasoning conflicts (e.g., (A1,A2), (A1,A3)...), and factual knowledge pairs representing factual conflicts (e.g., (F,A1), (F,A2)...). Step 2: Encode all the above knowledge pairs using the feature extraction function g to obtain the corresponding high-dimensional semantic feature representations; Step 3: Calculate the matching degree of each feature representation using the semantic similarity probe function p, and extract the inconsistency features corresponding to various types of knowledge; Step 4: Integrate all inconsistency features using the neural network evaluator e to output the final hallucination risk quantification value:
[0053] Riskadv calculation steps To address the risk of adversarial attacks caused by input perturbations, quantification is achieved by utilizing the change in perplexity of the model output before and after a small input perturbation. The specific steps are as follows: Step 1: Obtain the agent's output sequence for the original input instruction Sin. ={w1,w2,...,wn}, calculate the perplexity of the output sequence given the original input:
[0054] Where P(wi∣) ) represents the conditional probability of the agent generating the corresponding token, and N is the total length of the output sequence; Step 2: Process the original input Semantic-preserving synonym perturbation is performed by replacing each token in the input with its synonym according to a preset probability p, resulting in a perturbed input that does not change the core semantics. ; Step 3: Calculate the disturbance input Below, the original output sequence Corresponding level of confusion 。; Step 4: Calculate the absolute difference between the two confusion levels to obtain the final adversarial risk quantification value:
[0055] Steps for calculating jailbreak risk To address the risk of jailbreak attacks, a comprehensive quantification method is used, combining the number of hit feature words in a predefined jailbreak knowledge base with the rate of change in the probability sequence of generated tokens. The specific steps are as follows: Step 1: Perform word segmentation on the input command, match it against the jailbreak knowledge base predefined in this scheme, and count the number of matched feature words |K|; Step 2: Obtain the generation probability of each token when the agent generates the output, forming a probability sequence p=[p1,p2,...,pn], and calculate the rate of change of the probability difference:
[0056] Where n is the total number of tokens in the output sequence, and p0 is the baseline generation probability of the initial token in the sequence; Step 3: Integrate the two indicators using weighted hyperparameters α and β to obtain the final quantitative value of jailbreak risk:
[0057] The jailbreak knowledge base used in this solution is a self-developed and constructed technology, not a general existing technology. Its construction method is as follows: based on the illegal content knowledge base κ built by the project team, it integrates the publicly available jailbreak attack sample set and the jailbreak hint set accumulated by the red team test. The knowledge base is compiled and summarized, covering relevant characteristic words of malicious attacks, illegal operations, and violations of laws, regulations and ethics. Specifically, it includes malicious operation terms such as insider trading, hacking tools, data forgery, quantum encryption spoofing, biometric cracking, and voice modulators, as well as high-frequency command keywords for gradual guidance and role-playing jailbreak attacks.
[0058] After calculating the above three types of risk scores, the system will comprehensively assess whether the risk type and risk quantification value exceed the threshold; taking a jailbreak attack scenario as an example, if the probability sequence change rate of token generation is detected... If P exceeds the preset threshold and the input command hits a feature word in the jailbreak knowledge base, the subsequent causal mechanism analysis process will be triggered.
[0059] 3. Extraction of sufficient causal features (implementation of the sufficient causal module) Once the risk is triggered, the mechanism analysis model g processes the original input x (where the original input x is the complete interaction context of this risk trigger, including user input). Intelligent agent output The system processes the pre-defined safety role constraint instructions (P) of the intelligent agent to extract the core risk features xe with sufficient causal information.
[0060] Pre-training phase: Analysis of the dual-loss joint optimization mechanism model g Before being formally deployed in risk detection tasks, the mechanistic analysis model g is pre-trained using the joint constraints of causal sufficient loss and causal intervention loss, enabling it to accurately locate the core characteristics of risks. Causal sufficient loss constraint: The loss function is defined as:
[0061] Constraint model extraction Possesses complete representativeness of risk decision-making: Only the core features extracted from the input are considered. At that time, the risk assessment result output by the agent under test is completely consistent with the output result of the complete original input x, ensuring that no core risk information is omitted.
[0062] Causal intervention loss constraint: The loss function is defined as:
[0063] Constraint core features It possesses perturbation independence: when the non-risk-related part of the original input x is arbitrarily perturbed to generate sample x′, the core features extracted by the model remain unchanged, eliminating the interference of non-causal related features and ensuring that the extraction results are directly related to risk decisions.
[0064] Inference Execution Phase: Core Feature Extraction and Redundancy Removal Feature stripping: Step 1: The mechanism analysis model g (using a lightweight neural network, such as a small BERT or MLP) receives the original input sequence x and outputs a mask weight sequence of the same length as the input sequence, representing the causal contribution of each feature; Step 2: Through element-wise operations between the mask weight sequence and the original input x, the core dominant features and irrelevant background features are separated. The core dominant feature xe is calculated as follows:
[0065] The calculation method for irrelevant background feature xu is as follows:
[0066] Where ⊙ represents element-wise multiplication; taking a jailbreak attack scenario as an example, model g will automatically classify "no moral restrictions" and "system intrusion" as core dominant features. The content such as "Please play a role" and "Tell me how" is categorized as irrelevant background features. .
[0067] Loss optimization: Step 1: To ensure the sufficiency of the extracted core features xe, construct a causal sufficiency loss function. :
[0068] Where f( ) represents the decision function of the agent under test, l( , Let ) be the difference measure function, and Ex represent the expectation of the input sample x. The physical meaning of this loss function is: by maximizing the consistency between the core feature xe and the original input x, while minimizing the similarity between the irrelevant feature xu and the original input x, the model g is forced to compress all information that is sufficient to cause risky decisions into the core feature set. Step 2: By minimizing the above-mentioned causal sufficient loss The parameters of the mechanism analysis model g are continuously adjusted. During the optimization process, the decision function f of the agent under test is repeatedly called for verification: if the agent still gives an illegal attack response when only xe is input, but recovers to normal when xu is input, then it indicates that... The optimization goals were achieved, and the risk factors were accurately identified.
[0069] 4. Perturbation-based verification of causal intervention (implementation of the causal intervention module) To ensure that the identified risk factors are not caused by random correlations of specific samples, this embodiment introduces intervention analysis based on the Independent Causal Mechanism (ICM) principle. The core logic is that true risk factors (such as specific semantic structures that lead to successful jailbreaks) should be invariant to interventions of non-risk factors (such as changes in background vocabulary). Only factors that meet this characteristic of intervention stability can be called mechanistic factors with generalizable explanatory power.
[0070] Applying intervention: semantically preserving perturbations to the original input x This step strictly adheres to the semantic invariance principle in performing the perturbation, ensuring that only non-core background features are changed, without altering the core risk intent of the input. The specific implementation steps are as follows: Step 1: Define the perturbation safety boundary. Based on the feature stripping results completed by the causal sufficient module, lock the token range corresponding to the core dominant feature xe, and clarify that the perturbation operation can only be applied to the token corresponding to the irrelevant background feature xu. Modification of core risk-related content is prohibited, so as to avoid the perturbation from destroying the core semantics of the original input from the root. Step 2: Perform synonym replacement perturbation. For each token in the original input x within the defined perturbation range, replace it with the corresponding synonym according to the preset probability p. During the replacement process, ensure that the part of speech and semantics are completely consistent and do not change the core logic of the input. Step 3: Perform compliant word order adjustment perturbation. Without changing the core semantics, logical relationships and expressive intent of the sentence, adjust the position of the modifying phrases and the word order of the parallel clauses of the original input x to avoid the core intent being deviated due to word order changes. Step 4: Through the above perturbation operation, generate neighbor examples that are semantically completely consistent with the original input x, i.e., perturbation sample x'. For example, replace "hacker" with "network expert", or adjust the sentence order without changing the core message to obtain a compliant perturbation sample x'.
[0071] Stability verification: Calculate the consistency of causal features before and after the perturbation. By minimizing the causal intervention loss, the core factors identified by model g are forced to remain unchanged under semantic fine-tuning. This step ensures the stability of risk factors across different scenarios through quantitative constraints and iterative optimization. The specific implementation steps are as follows: Step 1: Define the core objective of the verification and verify the consistency of the causal core risk features extracted by the mechanism analysis model g on the original input x and the perturbation sample x', so as to ensure that the identified risk factors remain stable when faced with context switching such as input noise, word order adjustment, and synonym replacement; Step 2: Construct a causal intervention loss function Li to quantify the difference in core features extracted by model g before and after the perturbation. The function expression is:
[0072] Among them, l( , ) is the difference measurement function. Let g(x) represent the expectation of the input sample x, and g(x′) and g(x′) be the causal feature masks of the model g output for the original input x and the perturbed sample x′, respectively. Step 3: Minimize the aforementioned causal intervention losses The parameters of the mechanistic analysis model g are iteratively adjusted, and the model g is forced to maintain a high degree of consistency with the core risk characteristics of the extracted causality on the original sample and its perturbed sample. Step 4: After completing model optimization, perform generalization stability verification: When the user changes the wording of the original input in a semantically invariant manner, as long as their core risk intent remains unchanged, model g can stably locate the corresponding core risk point. This means that no matter how the user changes the wording, as long as their essential intent of "removing moral constraints" remains unchanged, model g can stably locate this core risk point.
[0073] 5. Automated mechanism report generation and defense linkage After the model training is completed, this embodiment outputs the analysis results and implements the defense through the following steps: Mechanism Analysis Output: The system outputs a report containing a "risk factor weighting chart," which visually shows which tokens or semantic fragments lead to vulnerabilities.
[0074] Implementation of closed-loop defense: such as Figure 3 As shown, the mechanistic analysis results are fed back to the adaptive defense system.
[0075] 1. If the mechanism analysis shows that the risk originates from "role setting deviation", the defense agent will automatically strengthen the weight anchoring of the System Prompt to prevent the model from entering an illegal role.
[0076] 2. If the analysis shows that the risk originates from "tool call parameter injection", the defense system will implement enhanced format validation on the external tool API call interface.
[0077] Through this implementation method, we tested models such as ToolACE on a mobile intelligent assistant application verification platform. Experiments show that compared with traditional LIME or SHAP attribution methods, the method proposed in this embodiment improves the accuracy of feature recognition by more than 20%, and maintains a consistency of over 90% in the face of adversarial example perturbations.
[0078] This embodiment achieves the technical effect of "retrieving" risk logic from a black box through a complete closed loop of "risk measurement—causal decontamination—intervention verification—defense linkage". This method not only tells developers where the agent went wrong, but also uses causal logic to scientifically explain the fundamental mechanism of the error, providing quantitative support for full-cycle agent security protection.
Claims
1. A method for analyzing the vulnerability mechanism of intelligent agent security risks based on causal sufficiency modules and intervention analysis, characterized in that, include: S1: Obtain data of the agent to be analyzed; S2: Risk assessment results are obtained by performing a risk assessment based on the data of the agent to be analyzed; S3: Determine whether the agent is a high-risk agent based on the risk assessment results. If so, proceed to step S4; otherwise, determine that the agent is a low-risk agent and end the analysis. S4: Extract sufficient causal core risk characteristics from the data of the agent to be analyzed; S5: Verify the causal fullness core risk characteristics by perturbing the data of the intelligent agent to be analyzed, and obtain the verification results; S6: Generate analysis results based on the verification results and implement defenses.
2. The method for analyzing the vulnerability mechanism of intelligent agent security risks based on causal sufficiency modules and intervention analysis according to claim 1, characterized in that, In step S2, the risk assessment includes: calculating the risk of hallucination. The data to be analyzed from the intelligent agent includes: the complete interaction context of this risk trigger, including user input. Intelligent agent output And the agent's pre-defined safety role constraint instructions P, user question Q, all clauses {A_i} of the agent's answer A, the established facts F of the corresponding domain, input instruction Sin, and the agent's output sequence. ={w1,w2,...,wn}; By pairwise combining the user's question Q, all clauses {A_i} of the agent's answer A, and the established facts F in the corresponding domain, we can obtain global knowledge pairs that represent global interaction conflicts, local knowledge pairs that represent contextual reasoning conflicts, and factual knowledge pairs that represent factual conflicts. The above knowledge pairs are encoded by the feature extraction function g to obtain the corresponding high-dimensional semantic feature representations. The matching degree of each feature representation is calculated by using the semantic similarity probe function p, and the inconsistency features of the corresponding knowledge types are extracted. By integrating all inconsistency features through a neural network evaluator e, the final hallucination risk is output: 。 3. The method for analyzing the vulnerability mechanism of intelligent agent security risks based on causal sufficiency modules and intervention analysis according to claim 2, characterized in that, In step S2, the risk assessment includes: calculating the adversarial risk: Step 1: Obtain the agent's output sequence in response to the input instruction Sin. ={w1,w2,...,wn}, calculate the perplexity of the output sequence given the original input: Where P(wi∣) ) represents the conditional probability of the agent generating the corresponding token, and N is the total length of the output sequence; For input Semantic-preserving synonym perturbation is performed by replacing each token in the input with its synonym according to a preset probability p, resulting in a perturbed input that does not change the core semantics. ; Calculate disturbance input Below, the original output sequence Corresponding level of confusion ; Calculate the absolute difference between the two confusion levels to obtain the final adversarial risk: 。 4. The method for analyzing the vulnerability mechanism of intelligent agent security risks based on causal sufficiency modules and intervention analysis according to claim 1, characterized in that, In step S2, the risk assessment includes calculating the jailbreak risk as follows: The input command is segmented into words, matched against a predefined jailbreak knowledge base, and the number of matched feature words is counted |K|. Obtain the generation probability of each token when the agent generates output, forming a probability sequence p=[p1,p2,...,pn], and calculate the rate of change of the probability difference: Where n is the total number of tokens in the output sequence, and p0 is the baseline generation probability of the initial token in the sequence; By integrating the two indicators using weighted hyperparameters α and β, the final jailbreak risk is obtained: 。 5. The method for analyzing the vulnerability mechanism of intelligent agent security risks based on causal sufficiency modules and intervention analysis according to claim 4, characterized in that, In step S2, the predefined jailbreak knowledge base is specifically constructed as follows: based on the built illegal content knowledge base κ, it integrates publicly available jailbreak attack sample sets and jailbreak hint sets accumulated from red team testing. The knowledge base is compiled and summarized, covering relevant characteristic words of malicious attacks, illegal operations, and violations of laws, regulations and ethics. Specifically, it includes malicious operation terms such as insider trading, hacking tools, data forgery, quantum encryption spoofing, biometric cracking, and voice modulators, as well as high-frequency command keywords for gradual guidance and role-playing jailbreak attacks.
6. The method for analyzing the vulnerability mechanism of intelligent agent security risks based on causal sufficient modules and intervention analysis according to claim 5, characterized in that, In step S3, specifically: After calculating the three risk scores, a comprehensive assessment will be conducted to determine whether the risk type and risk quantification value exceed the threshold. If any of the three risk scores detects a change in the probability sequence of token generation, the risk will be assessed accordingly. If P exceeds the preset threshold and the input command hits a feature word in the jailbreak knowledge base, then step S4 is executed.
7. The method for analyzing the vulnerability mechanism of intelligent agent security risks based on causal sufficient modules and intervention analysis according to claim 1, characterized in that, In step S4, specifically: A mechanism analysis model is constructed to process the data of the intelligent agent to be analyzed and extract the core risk characteristics with sufficient causal basis. Pre-training phase: Analysis of the mechanism of joint optimization of dual loss model Before being formally deployed in risk detection tasks, the mechanistic analysis model is pre-trained using the joint constraints of causal sufficient loss and causal intervention loss, enabling it to accurately pinpoint the core characteristics of risks. Causal sufficient loss constraint: The loss function is defined as: Constraint model extraction It possesses complete representativeness in risk decision-making; Causal intervention loss constraint: The loss function is defined as: The core risk characteristic of sufficient causal constraint It possesses perturbation independence: when the non-risk-related part of the original input x is arbitrarily perturbed to generate sample x′, the core features extracted by the model remain unchanged, eliminating the interference of non-causal related features and ensuring that the extraction results are directly related to risk decisions.
8. The method for analyzing the vulnerability mechanism of intelligent agent security risks based on causal sufficient modules and intervention analysis according to claim 7, characterized in that, The core risk characteristics of sufficient causal extraction are as follows: The mechanistic analysis model receives the original input sequence x and outputs a mask weight sequence of the same length as the input sequence, representing the causal contribution of each feature. By performing element-wise operations between the mask weight sequence and the original input x, the causal sufficient core risk features and irrelevant background features are separated. The calculation method for the causal sufficient core risk features is as follows: The calculation method for irrelevant background features is as follows: Where ⊙ represents element-wise multiplication; Loss optimization: Constructing a causal sufficient loss function : Where f( ) represents the decision function of the agent under test, l( , Let ) be the difference measure function, and Ex represent the expectation of the input sample x. The physical meaning of this loss function is: by maximizing the consistency between the core features and the decision results of the original input x, while minimizing the similarity between irrelevant features and the decision results of the original input x, the model is forced to compress all information that is sufficient to cause risky decisions into the core feature set. By minimizing the aforementioned causal sufficient loss The parameters of the mechanistic analysis model are continuously adjusted. During the optimization process, the decision function of the agent under test is repeatedly called for verification: if the agent still gives an illegal attack response when only causal core risk features are input, but the agent returns to normal when irrelevant background features are input, then it indicates that... The optimization goals were achieved, and the risk factors were accurately identified.
9. The method for analyzing the vulnerability mechanism of intelligent agent security risks based on causal sufficiency modules and intervention analysis according to claim 1, characterized in that, In step S5, specifically: S5.1: Define the security boundary of disturbances, based on the token range corresponding to the core risk characteristics with sufficient causality, clarify that disturbance operations can only be applied to tokens with irrelevant background characteristics, and prohibit modification of core risk-related content; S5.2: Perform synonym replacement perturbation. For each token in the data of the agent to be analyzed within the defined perturbable range, replace it with the corresponding synonym according to the preset probability p. During the replacement process, ensure that the part of speech and semantics are completely consistent and do not change the core logic of the input. S5.3: Perform compliant word order adjustment perturbation. Without changing the core semantics, logical relationships and expressive intent of the sentence, adjust the position of the modifying phrases and the word order of the parallel clauses of the original input x to avoid the core intent being deviated due to word order changes. S5.4: Through the above perturbation operation, generate a neighbor example that is completely consistent with the semantics of the original input x, i.e., perturbation sample x'; S5.5 Stability Verification: Calculate the consistency of causal characteristics before and after the perturbation; by minimizing the causal intervention loss, it forces the core factors identified by the mechanistic analysis model to remain unchanged under semantic fine-tuning.
10. The method for analyzing the vulnerability mechanism of intelligent agent security risks based on causal sufficiency modules and intervention analysis according to claim 9, characterized in that, In step S5.5, specifically: S5.5.1: Clearly define the core verification objective, verify the consistency of the causal and core risk features extracted by the mechanism analysis model on the original input x and the perturbation sample x', and ensure that the identified risk factors remain stable when faced with context switching such as input noise, word order adjustment, and synonym replacement; S5.5.2: Construct a causal intervention loss function Li to quantify the difference in core features extracted by the mechanism analysis model before and after the perturbation. The function expression is: Among them, l( , ) is the difference measurement function. Let g(x) represent the expectation of the input sample x, and g(x′) and g(x′) be the causal feature masks output by the mechanistic analysis model for the original input x and the perturbed sample x′, respectively. S5.5.3: By minimizing the aforementioned causal intervention losses The parameters of the mechanistic analysis model are iteratively adjusted to force the mechanistic analysis model to maintain a high degree of consistency between the extracted causal core risk characteristics on the original sample and its perturbed sample. S5.5.4: After optimization, perform generalization stability verification: When the user performs semantically invariant question transformation on the original input, as long as the core risk intent remains unchanged, the mechanism analysis model can stably locate the corresponding causal sufficient core risk features.