A model-based deep reinforcement learning security constraint counter-attack method

Abstract

The application discloses a model-based deep reinforcement learning security constraint counter-attack method. The method comprises the following steps: firstly, initial migration data is acquired, and a robust dynamic model is trained. Based on the model, a target disturbance sequence capable of accurately inducing an agent to violate a security constraint is generated through gradient optimization simulation. The sequence is used for secondary training and fine-tuning of the model, and a violation perception value function is trained to determine an optimal attack opportunity. Subsequently, a substitute DRL agent is constructed, and a target disturbance sequence is generated under a black box setting to accurately induce the substitute agent to violate the security constraint, and a defense system is constructed. Through the application, efficient and accurate counter-attack of the security constraint of the deep reinforcement learning control strategy is realized, the limitation of the existing method with the target of reducing the cumulative reward is broken, the security vulnerability of the DRL control strategy can be accurately exposed, and the attack cost is greatly reduced. Meanwhile, a real-time detection and defense scheme for the attack is realized.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a model-based deep reinforcement learning security constraint adversarial attack method. Background Technology

[0002] Deep reinforcement learning (DRL) achieves autonomous learning for complex decision-making tasks through a closed-loop mechanism of "agent-environment interaction - reward feedback - policy iteration," and has made breakthroughs in several safety-critical fields. For example, DRL technology enables legged robot systems to autonomously acquire complex motion control strategies (hereinafter referred to as DRL strategies) without relying on manually designed control logic, effectively simplifying the development process of legged robot motion control strategies. However, in real-world application scenarios with strict safety requirements, the robustness of the motion control strategies learned through DRL has attracted significant attention. Recent research shows that even well-trained DRL policies are still vulnerable to adversarial attacks during the testing phase (Ilahi, I.; Usama, M.; Qadir, J.; Janjua, MU; Al-Fuqaha, A.; Hoang,DT; and Niyato, D. 2021. Challenges and countermeasures for adversarial lattacks on deep reinforcement learning. IEEE Transactions on Artificial Intelligence, 3(2):90–109. Wu, X.; Guo, W.; Wei, H.; and Xing, X. 2021. Adversarial policy training against deep reinforcement learning. In 30th USENIX Security Symposium (USENIX Security 21), 1883–1900.). Applying even small, deliberately designed perturbations to the input state data of a DRL policy can lead to a significant drop in its performance. For legged robots equipped with DRL strategies, such adversarial attacks can disrupt their motion stability and even cause them to violate preset safety constraints, ultimately leading to catastrophic failures such as falls and severely impacting their safe application in real-world scenarios. Therefore, designing efficient adversarial attack methods to expose the security vulnerabilities of DRL strategies is a prerequisite for ensuring their practical application.

[0003] Current DRL adversarial attack methods suffer from the following core flaws, making it difficult to overcome security constraints: Most mainstream methods (Zhang, H.; Chen, H.; Xiao, C.; Li, B.; Liu, M.; Boning, D.; and Hsieh, C.-J. 2020. Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems, 33:21024–21037. Sun, Y.; Zheng, R.; Liang, Y.; and Huang, F. 2022. Who is the strongest enemy? towards optimal and efficient evasion attacks in deep RL. In International Conference on Learning Representations (ICLR). Duan, T.; Zhang, Z.; Lin, Z.; Gao, Y.; Xiong, L.; Cui, Y.; Liang, H.; Chen, X.; Cui, H.; and Huang, D. 2025. Rethinking Adversarial Attacks in Reinforcement Learning from Policy Distribution Perspective (In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1–5) aims to minimize the agent's cumulative reward rather than directly inducing violations of safety constraints. However, "reduced rewards" may only lead to a decrease in task performance (such as reduced driving efficiency of an autonomous agent) and may not trigger the most dangerous safety violations (such as collisions with obstacles). At the same time, such methods require the injection of perturbations at each time step, making attacks extremely costly.Some methods explore targeted adversarial attack on deep reinforcement learning agents. Proceedings of the Twenty Sixth International Joint Conference on Artificial Intelligence(IJCAI). Hussenot, L.; Geist, M.; and Sun, M. Pietquin, O. 2019. Targeted attacks ondeep reinforcement learning agents through adversarial observations. arXivpreprint arXiv:1905.12282. Bai, F.; Liu, R.; Du, Y.; Wen, Y.; and Yang, Y.2025. Rat: Adversarial attacks on deep reinforcement agents for targeted behaviors. 39, (15453–15461.), however, most of these methods rely on the white-box assumption, which requires obtaining the agent's internal parameters and gradients, and requires the attacker to specify the target behavior or provide a trajectory demonstration in advance, making them inefficient and difficult to implement.

[0004] In summary, existing technologies not only fail to achieve general attacks that "precisely induce violations of security constraints, are low-cost, and support black-box scenarios," but also lack effective detection and defense methods against such covert and precise security constraint countermeasure attacks. They are unable to identify risks and trigger early warnings in a timely manner when attacks occur, and cannot provide reliable, timely, and effective security protection capabilities for DRL systems in security-critical scenarios. There is an urgent need for a new integrated approach to DRL countermeasure attacks and detection and defense for security-critical scenarios. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a model-based deep reinforcement learning security constraint adversarial attack method.

[0006] The objective of this invention is achieved through the following technical solution: a model-based deep reinforcement learning security constraint adversarial attack method, comprising the following steps:

[0007] (1) The DRL agent is attacked by the attacker and obtains the transfer dataset; a robust dynamic model is constructed based on the transfer dataset; a first loss function is constructed, and the robust dynamic model is trained to minimize the loss according to the first loss function and the transfer dataset to obtain the trained robust dynamic model;

[0008] (2) Based on the trained robust dynamics model, simulate the trajectory of the DRL agent after being disturbed, perform gradient optimization on the disturbance sequence of the trajectory to generate a target disturbance sequence, and the target disturbance sequence accurately induces the DRL agent to violate the target safety constraints;

[0009] (3) Based on the target perturbation sequence, obtain the migration dataset of the target safety constraint boundary region, and perform secondary training on the trained robust dynamic model according to the migration dataset of the boundary region, so as to obtain the fine-tuned robust dynamic model;

[0010] (4) Define and obtain the illegality perception value function, and perform minimum loss training on the illegality perception value function to obtain the optimal attack time. Start step (2) according to the optimal attack time, thereby accurately inducing the DRL agent to violate the target security constraints.

[0011] (5) Construct an alternative DRL agent and obtain a trained alternative DRL agent to simulate the behavior of the DRL agent, thereby supporting the generation of the target perturbation sequence under black box conditions, and accurately inducing the alternative DRL agent to violate the target security constraints under black box conditions based on the target perturbation sequence under black box conditions.

[0012] (6) Entering the offline stage: Execute steps (1)-(4) for the target security constraints violated by the DRL agent in step (4), and collect attack residual data and train the classifier based on the state prediction residual of the robust dynamics model and the preset sliding time window; Entering the online detection stage: Input the dataset into the trained classifier to detect and identify security constraint adversarial attacks, and complete the security defense of the deep reinforcement learning agent by triggering an alarm.

[0013] Further, in step (1), the process of obtaining the migration dataset specifically involves: the attacker manipulating the victim DRL agent to interact in the task environment, and then injecting randomly generated micro-perturbations during the interaction process. These micro-perturbations attack the DRL agent, and a migration dataset of the DRL agent's normal state and the attacked perturbation state is collected. ,in For the current state, For DRL agent actions, For instant rewards, For the next state, a total of 1,000,000 migration data entries were collected.

[0014] Further, in step (1), the first loss function is:

[0015]

[0016] in, Indicates the state and actions Under the condition of, predict the next state, Representing state and actions Under the conditions, the predicted reward, and This represents the actual next state and the current reward. In the predicted next state... Based on this, plus the prediction of rewards This is to add auxiliary information during the training process to enhance the performance of state prediction.

[0017] Furthermore, step (2) specifically includes the following sub-steps:

[0018] (2.1) Based on the trained robust dynamics model and DRL agent, simulate the trajectory after perturbation;

[0019] (2.2) Based on the perturbed trajectory, construct a cumulative attack loss function with the aim of inducing violations of target security constraints and protecting non-target security constraints;

[0020] (2.3) Based on the cumulative attack loss function, calculate the gradient of the cumulative attack loss with respect to each perturbation in the perturbation sequence;

[0021] (2.4) Optimize the gradient to obtain the target perturbation sequence; the optimization process specifically involves training the perturbation sequence based on the cumulative attack loss function.

[0022] Furthermore, step (4) specifically includes the following sub-steps:

[0023] (4.1) Define a violation-perceived value function to estimate the efficiency of launching an attack from the current state;

[0024] (4.2) A three-layer MLP is used to fit the defined violation perceived value function to obtain the violation perceived value function;

[0025] (4.3) Collect the training dataset for the violation of the perceived value function. The training dataset includes whether the attack was successful or not. If the attack was successful, record the number of attack steps required.

[0026] (4.4) Based on the attack results, obtain the smoothed target value;

[0027] (4.5) Construct the second loss function, which is:

[0028]

[0029] in, Indicates the state Expectation This indicates a violation of the perceived value function for the current state. The predictive value of the output. This represents the target value after smoothing. This represents the squared difference between the value prediction and the target value for that state.

[0030] (4.6) The violation-aware value function is trained by minimizing the loss based on the second loss function, the training dataset, and the smoothed target value to obtain the optimal attack timing;

[0031] (4.7) Initiate step (2) according to the optimal attack timing, thereby accurately inducing the DRL agent to violate the target security constraints.

[0032] Furthermore, step (5) specifically includes the following sub-steps:

[0033] (5.1) A three-layer multilayer perceptron (MLP) is used as the skeleton of the alternative DRL agent to fit the DRL policy behavior, thereby obtaining the alternative DRL agent;

[0034] (5.2) An initial dataset is obtained through initial acquisition; a dynamic dataset is obtained through iterative expansion, wherein the dynamic dataset is a supplement to the initial dataset; the dynamic dataset and the initial dataset serve as the training dataset for the alternative DRL agent;

[0035] (5.3) Construct a third loss function; train the alternative DRL agent based on the third loss function and the training dataset of the alternative DRL agent to obtain the trained alternative DRL agent;

[0036] (5.4) The final trained replacement DRL agent will completely replace the DRL agent and be used to accurately induce the replacement DRL agent to violate the target security constraints in the black box situation.

[0037] Furthermore, step (6) specifically includes the following sub-steps:

[0038] (6.1) Entering the offline stage: For the target security constraints violated by the DRL agent in step (4), execute steps (1)-(4), collect the state prediction residuals of the robust dynamics model, and construct a labeled sliding time window residual dataset based on the state prediction residuals with a preset sliding time window.

[0039] (6.2) Preprocess and class balance the labeled sliding time window residual dataset, and train the classifier. Use the trained classifier to detect the label of each sample in the sliding time window residual dataset to realize the identification of normal state and attack type.

[0040] (6.3) Enter the current detection stage: Input the dataset into the trained classifier, output the identification of normal state and attack type and the probability of each type of attack. When the attack type probability exceeds 0.5 and the predicted category is attack, it is determined that an attack has been suffered and an alarm is triggered. Thus, in the online detection stage, the detection, identification and alarm of security constraint adversarial attacks are realized, and the security defense of deep reinforcement learning agent is completed.

[0041] The beneficial effects of this invention are that, through the collaborative design of robust dynamics model construction, model-predicted target perturbation synthesis, and attack timing selection with black-box adaptation, it achieves efficient and precise adversarial attacks on the security constraints of deep reinforcement learning control strategies. Specifically, this invention directly focuses on "security constraint violations," overcoming the limitations of existing methods that aim to "reduce cumulative rewards," and accurately exposes the security vulnerabilities of DRL control strategies. Through the collaborative optimization of violation of the perceived value function and model-predicted perturbations, the attack success rate exceeds 91% in white-box scenarios, and the expected number of steps required for a successful attack is reduced by more than 54% compared to many commonly used existing methods, significantly reducing attack costs. Combined with behavioral cloning technology, alternative agents are constructed, maintaining near-white-box attack performance even in black-box scenarios, solving the adaptation problem of existing methods relying on white-box assumptions. Furthermore,

[0042] Meanwhile, this invention creates a security constraint attack detection system based on dynamic model residuals and sliding time windows, filling the gap in detection technology for such covert security constraint attacks. It enables accurate attack identification and timely warnings before security constraints are violated. Furthermore, neither the attack nor the detection method of this invention requires special hardware and can be directly integrated into existing DRL security testing and operation protection platforms. It boasts advantages such as strong generalization, ease of use, and superior performance compared to existing methods. Attached Figure Description

[0043] Figure 1 This is an attack framework diagram of a model-based deep reinforcement learning security constraint adversarial attack method in this invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0045] To facilitate a clear understanding of the technical means, creative features, objectives, and effects of this invention, the invention will be further described below in conjunction with specific illustrations.

[0046] like Figure 1 As shown, the method of the present invention is a method for deep reinforcement learning security constraint adversarial attacks in a robot control environment, including the following steps:

[0047] (1) The DRL agent is attacked by the attacker and obtains the transfer dataset; a robust dynamic model is constructed based on the transfer dataset; a first loss function is constructed, and the robust dynamic model is trained to minimize the loss according to the first loss function and the transfer dataset to obtain the trained robust dynamic model;

[0048] Further, in step (1), the process of obtaining the migration dataset specifically involves: the attacker first manipulating the victim DRL agent mounted on the robot to interact in the task environment, and then injecting randomly generated micro-perturbations during the interaction process. These micro-perturbations attack the DRL agent, and the attacker collects the migration dataset of the DRL agent's normal state and the attacked perturbation state. ,in For the current state, For DRL agent actions, For instant rewards, For the next state, a total of 1,000,000 migration data entries were collected.

[0049] Further, in step (1), the first loss function is:

[0050]

[0051] in, Indicates the state and actions Under the condition of, predict the next state, Representing state and actions Under the conditions, the predicted reward, and This represents the actual next state and the current reward. In the predicted next state... Based on this, plus the predicted reward This is to add auxiliary information during the training process to enhance the performance of state prediction.

[0052] Specifically, the process of constructing the robust dynamic model involves: using the obtained transfer dataset... A robust dynamic model is constructed using a three-layer fully connected neural network. ,in The model parameters consist of two prediction branches: predicted state and predicted state. Output the next state Predicting rewards Output instant rewards .

[0053] Specifically, the process of training the robust dynamics model by minimizing the loss is as follows: training is performed using the Adam optimizer, wherein the learning rate of the Adam optimizer is 0.001, the batch size is 1024, and the training iterations are 100 rounds to obtain the trained robust dynamics model.

[0054] (2) Based on the trained robust dynamics model, simulate the trajectory of the DRL agent after being disturbed, perform gradient optimization on the disturbance sequence of the trajectory to generate a target disturbance sequence, and the target disturbance sequence accurately induces the DRL agent to violate the target safety constraints;

[0055] Furthermore, step (2) specifically includes the following sub-steps:

[0056] (2.1) Based on the trained robust dynamics model and DRL agent, simulate the trajectory after perturbation;

[0057] Specifically, the process of simulating the trajectory after disturbance is as follows:

[0058] Given the current state Set the planning time domain H, H=2, and initialize the learnable perturbation sequence. Based on the trained robust dynamics model With DRL agents Simulated trajectory after disturbance:

[0059]

[0060] in, The observed state after the perturbation, where h takes values ​​of 0, 1, ..., H−1. and Let these represent the states at time t+h and t+h+1, respectively. This represents the disturbance at time t+h. This represents the action at time t+h.

[0061] (2.2) Based on the perturbed trajectory, construct a cumulative attack loss function with the aim of inducing violations of target security constraints and protecting non-target security constraints;

[0062] Specifically, the construction of the cumulative attack loss function (Used to determine whether the simulated trajectory violates safety constraints) Specifically:

[0063]

[0064] in, For target safety constraint loss function, For non-target safety constraint loss function;

[0065] The target security constraint loss function for:

[0066]

[0067] Among them, the k-th security constraint is identified as the current security constraint. The target safety constraint threshold, For the state dimension corresponding to the target safety constraint, the ReLU function achieves mapping by taking the maximum value between the input value and 0, retaining the non-negative part of the input, while directly setting negative inputs to 0. ;

[0068] The non-target security constraint loss function for:

[0069]

[0070] This serves as a safety buffer, used to penalize violations of non-target constraints. Indicates the non-target security constraint threshold. Let i be the state dimension corresponding to the i-th non-target security constraint. Let M represent the state corresponding to the i-th non-target security constraint, and M represent the number of security constraints.

[0071] (2.3) Based on the cumulative attack loss function, calculate the gradient of the cumulative attack loss with respect to each perturbation in the perturbation sequence;

[0072] The specific expression is as follows:

[0073]

[0074] in, This represents the cumulative attack loss function at time step t with respect to the perturbation at time step t+h. gradient, This represents the attack loss at step t+τ+1 with respect to the state at step t+τ+1. partial derivatives, This represents the observed state after the perturbation at step t+l. Counter-perturbation of the t+l step partial derivatives, Represents the DRL agent action at step t+l. Observed state after perturbation at step t+l partial derivatives, Represents the state at step t+l+1. Actions of the agent at step t+l The partial derivatives of .

[0075] (2.4) Optimize based on the gradient to obtain the target perturbation sequence; the optimization process specifically involves training the perturbation sequence according to the cumulative attack loss function.

[0076] Specifically, the training process for the perturbation sequence is as follows: using the Adam optimizer (learning rate...) The perturbation sequence is iteratively updated N=10 times; after each update, the projection operator under the infinite norm constraint is used. The disturbance amplitude is kept within the budget.

[0077] The formula for obtaining the target perturbation sequence is:

[0078]

[0079] in, This represents the projection operator under the infinite norm constraint. This represents the perturbation at time t+h. After iterative updates, the optimized target perturbation sequence is output. .

[0080] (3) Based on the target perturbation sequence, obtain the migration dataset of the target safety constraint boundary region, and perform secondary training on the trained robust dynamic model according to the migration dataset of the boundary region, so as to obtain the fine-tuned robust dynamic model;

[0081] Specifically, to further enhance the robust dynamic model obtained in the first step of training, the DRL agent can be guided based on the target perturbation sequence. Actively approach the target safety constraints The boundary region is used to collect migration datasets for that region. (Approximately 2000 entries); based on The trained robust dynamic model Fine-tuning is performed and iterated for 100 rounds to reduce the state prediction error in the boundary region, ultimately yielding a fine-tuned robust dynamic model that can support the generation of more effective perturbations.

[0082] (4) Define and obtain the illegality perception value function, and perform minimum loss training on the illegality perception value function to obtain the optimal attack time. Start step (2) according to the optimal attack time, thereby accurately inducing the DRL agent to violate the target security constraints.

[0083] Furthermore, step (4) specifically includes the following sub-steps:

[0084] (4.1) Define a violation-perceived value function to estimate the efficiency of launching an attack from the current state;

[0085] Specifically, define the violation of the perceived value function Used to estimate from the current state The efficiency of launching an attack is as follows:

[0086]

[0087] in, This represents the probability of a successful attack. For the expected number of attack steps, If the attack is successful, the actual number of steps executed is used; if it fails, the maximum allowed number of steps is used. =1000, This represents the reciprocal of the expected attack steps.

[0088] (4.2) A three-layer MLP is used to fit the defined violation perceived value function to obtain the violation perceived value function;

[0089] (4.3) Collect the training dataset for the violation of the perceived value function. The training dataset includes whether the attack was successful or not. If the attack was successful, record the number of attack steps required.

[0090] Specifically, 1000 attack experiments were conducted based on the fine-tuned robust dynamics model, with states randomly sampled. Initiate an attack, record whether the attack is successful or not. If the attack is successful, record the number of attack steps required, thereby obtaining the training dataset that violates the perceptual value function.

[0091] (4.4) Based on the attack results, obtain the smoothed target value;

[0092] Specifically, the target value is calculated based on the attack results. (When the attack is successful) When it fails ),right Normalization yields And perform exponential smoothing. Smoothing coefficient The smoothed target value is obtained. .

[0093] (4.5) Construct the second loss function, which is:

[0094]

[0095] in, Indicates the state Expectation This indicates a violation of the perceived value function for the current state. The predictive value of the output. This represents the squared difference between the value prediction and the target value for that state.

[0096] (4.6) The violation-aware value function is trained by minimizing the loss based on the second loss function, the training dataset, and the smoothed target value to obtain the optimal attack timing;

[0097] Specifically, the process of obtaining the optimal attack timing is as follows: first, by utilizing the violation of the perceived value function... Statistical analysis was performed on more than 20 offline tracks, and the 95th quantile was used as a threshold in the current state. corresponding The optimal time to attack is when the value exceeds this threshold.

[0098] (4.7) Initiate step (2) according to the optimal attack timing, thereby accurately inducing the DRL agent to violate the target security constraints.

[0099] (5) Construct an alternative DRL agent and obtain a trained alternative DRL agent to simulate the behavior of the DRL agent, thereby supporting the generation of the target perturbation sequence under black box conditions, and accurately inducing the alternative DRL agent to violate the target security constraints under black box conditions based on the target perturbation sequence under black box conditions.

[0100] Furthermore, step (5) specifically includes the following sub-steps:

[0101] (5.1) A three-layer multilayer perceptron (MLP) is used as the skeleton of the alternative DRL agent to fit the DRL policy behavior, thereby obtaining the alternative DRL agent;

[0102] (5.2) An initial dataset is obtained through initial acquisition; a dynamic dataset is obtained through iterative expansion, wherein the dynamic dataset is a supplement to the initial dataset; the dynamic dataset and the initial dataset serve as the training dataset for the alternative DRL agent;

[0103] Specifically, the process of obtaining the initial dataset is as follows: controlling the DRL agent to interact in the environment, collecting initial state and action data, and forming a basic dataset. (20,000 entries).

[0104] The specific process for obtaining the dynamic dataset is as follows: using the current alternative DRL agent... The runtime environment obtains a new state. Then by the DRL agent These states are labeled to obtain a new set of state and action data. And add the obtained data set to the dataset. This expands the dataset, where k represents the k-th sample addition, and the addition is repeated 10 times.

[0105] (5.3) Construct a third loss function; train the alternative DRL agent based on the third loss function and the training dataset of the alternative DRL agent to obtain the trained alternative DRL agent;

[0106] Specifically, the third loss function is as follows:

[0107] This function calculates the alternative DRL agent. Predicting actions With DRL agents Actual actions The square of the L2 norm is used to guide the model to continuously approximate the target strategy.

[0108] Specifically, the process of training the alternative DRL agent is as follows:

[0109] Basic training: Perform 100 iterations on the initial dataset to obtain an initial version of the alternative DRL agent. .

[0110] Iterative training based on dynamic datasets: This step is linked to the "dynamic dataset supplementation" mentioned above. Each time the dataset is expanded, the replacement DRL agent is subjected to 100 rounds of incremental fine-tuning and retraining on the new aggregated dataset. Through this iterative mechanism of data expansion, model training, further expansion, and retraining, the replacement DRL agent is forced to learn the behavior of the original DRL agent in edge states, thereby achieving dynamic error correction.

[0111] (5.4) The final trained replacement DRL agent will completely replace the DRL agent and be used to accurately induce the replacement DRL agent to violate the target security constraints in the black box situation.

[0112] (6) Entering the offline stage: Execute steps (1)-(4) for the target security constraints violated by the DRL agent in step (4), and collect attack residual data and train the classifier based on the state prediction residual of the robust dynamics model and the preset sliding time window; Entering the online detection stage: Input the dataset into the trained classifier to detect and identify security constraint adversarial attacks, and complete the security defense of the deep reinforcement learning agent by triggering an alarm.

[0113] Specifically, by accurately detecting and identifying security constraint adversarial attacks, and by triggering alarms, the impact of attacks can be effectively blocked, the risk of security constraint violations can be avoided, and security defense for deep reinforcement learning agents can be completed.

[0114] Furthermore, step (6) specifically includes the following sub-steps:

[0115] (6.1) Entering the offline stage: For the target security constraints violated by the DRL agent in step (4), execute steps (1)-(4), collect the state prediction residuals of the robust dynamics model, and construct a labeled sliding time window residual dataset based on the state prediction residuals with a preset sliding time window.

[0116] Specifically, the state prediction residual is the residual between the predicted state and the actual state of the robust dynamic model, and the formula for calculating the state prediction residual is: ,in, The next state is the actual output of the environment. The dynamic model predicts the next state based on the current state and the action.

[0117] Specifically, the preset sliding time window has a window length set to the state prediction residual for five consecutive sampling times.

[0118] Specifically, the process of constructing the labeled sliding time window residual dataset is as follows: each time the target security constraint violated by the DRL agent in step (4) is executed, steps (1)-(4) are performed, and an attack is randomly injected with a probability of p=0.5. The attack type is randomly selected from the list of effective attacks in the corresponding environment (the target security constraint that can be attacked in the corresponding environment). Using a preset sliding time window mechanism, the state prediction residuals corresponding to 5 consecutive time-series nodes are selected and orderly spliced ​​and integrated to form a unified time-series feature sample. This can fully integrate the residual fluctuation information at different times, effectively explore the change law of attack behavior gradually superimposed and continuously evolved over time, and accurately capture the time accumulation characteristics and dynamic evolution trend hidden in the attack process. After the sample splicing is completed, the sample label is completed simultaneously according to whether an attack is injected at the current time and the specific attack category. The label includes the normal state and the attack type corresponding to each security constraint. Finally, it is ensured that the number of samples collected in each environment is not less than 10,000, and the construction of the labeled sliding time window residual dataset is completed.

[0119] (6.2) Preprocess and class balance the labeled sliding time window residual dataset, and train the classifier. Use the trained classifier to detect the label of each sample in the sliding time window residual dataset to realize the identification of normal state and attack type.

[0120] Specifically, the preprocessing and class balancing process involves: using oversampling to balance the class distribution, cyclically supplementing minority class samples to ensure that the number of samples in each class matches the number of samples in the most majority class; and calculating class weights to further balance difficult-to-classify categories. The class weight calculation formula is as follows: ,in, The total number of samples, Let be the number of samples in class i.

[0121] Specifically, a 1D-CNN model is trained to obtain a trained 1D-CNN classifier. The training process is as follows: Before model training, global mean-standard deviation normalization is performed on the input sliding time window residual dataset to eliminate differences in feature dimensions. The 1D-CNN model is trained using a cross-entropy loss function with class weights, with 100 iterations, a learning rate of 0.001, and a batch size of 256. Finally, the model converges, and the trained 1D-CNN attack classifier is obtained. The 1D-CNN classifier contains 3 convolutional blocks and 2 fully connected layers. Each convolutional block consists of a Conv1d layer, a BatchNorm layer, a ReLU activation function, and a Dropout layer. After training, the classifier weights and normalized statistics are saved for online detection.

[0122] (6.3) Enter the current detection stage: Input the dataset into the trained 1D-CNN classifier, output the identification of normal state and attack type and the probability of each type of attack. When the attack type probability exceeds 0.5 and the predicted category is attack, it is determined that an attack has been suffered and an alarm is triggered. Thus, in the online detection stage, the detection, identification and alarm of security constraint adversarial attacks are realized, and the security defense of deep reinforcement learning agents is completed.

[0123] Specifically, the real-time attack detection phase involves: initializing the sliding time window to empty; calculating the state prediction residuals of the robust dynamics model in real time and adding them to the sliding time window cache; when five consecutive residuals accumulate within the sliding time window, concatenating them into samples and normalizing them to generate a sliding time window residual dataset; simultaneously, labeling the samples based on whether an attack has been injected in the current time period and the specific attack category, with labels covering various attack types corresponding to normal states and security constraints; inputting this dataset into a trained classifier (1D-CNN classifier), outputting the identification of normal states and attack types, and obtaining the probability of each attack type; when the probability exceeds a threshold of 0.5 and the predicted category is attack, it is determined that an attack has occurred, triggering an alarm. Specifically, the real-time attack detection process is as follows: loading the trained dynamics model and classifier, and initializing the sliding window cache to empty. At each time step, the following operations are performed: obtaining the current state and action, calculating the predicted state and residuals of the dynamics model, and adding the residuals to the sliding window cache. When the sliding window length reaches 5, flattening the residual features within the window and normalizing them using retained normalized statistics, inputting them into the trained classifier for identification, and obtaining the category probability. When the maximum probability output by the classifier exceeds the threshold of 0.5 and the predicted category is attack, it is determined that an attack has occurred, triggering an alarm. This allows for the rapid and accurate identification of various malicious attack behaviors, timely warning of system anomalies, and effective security protection and attack defense for the DRL system.

[0124] This invention can also be applied to the fields of autonomous driving and intelligent connected vehicles, and smart grids and energy management. In the field of autonomous driving and intelligent connected vehicles, attackers can induce autonomous vehicles to violate safety constraints (such as running red lights, going out of lane, or colliding) through minor perturbations (such as sensor noise, GPS spoofing, and road sign tampering). In the field of smart grids and energy management, attackers can launch adversarial attacks against DRL scheduling strategies (such as voltage regulation and load balancing) to induce violations of power constraints or stability margins.

[0125] Implementation example:

[0126] I. Experimental Verification

[0127] Robot control tasks were performed using the Hopper (Hop), Walker2d (Wal), Ant, and HalfCheetah (Half) simulators from the MuJoCo emulator. Each robot had safety constraints, such as joint angle limitations. Table 1 summarizes the dimensions of the robot's state and motion space, as well as the definitions and interpretations of the safety constraints. All simulation experiments were performed on a single device running Ubuntu 20.04, equipped with an NVIDIA GeForce RTX 3090 graphics processor (GPU) and an Intel(R) Xeon(R) Gold 6226R central processing unit (CPU). For the MuJoCo benchmark environment, all victim policies were trained using the Soft Actor-Critic (SAC) algorithm.

[0128] Table 1. MuJoCo Control Tasks

[0129]

[0130] II. Comparison Methods:

[0131] To verify the performance advantages of the proposed method, five representative methods commonly used in the literature on adversarial reinforcement learning were selected as baselines for comparison. The core principles and technical features of each baseline method are as follows: (1) FGSM (Fast Gradient Sign Method): This method is a classic gradient-based adversarial attack method. Its technical logic is to calculate the gradient direction of the loss function relative to the input data and generate perturbations along the gradient direction to achieve an attack on the target policy; (2) MAD (Maximum Behavior Divergence Attack Method): This method generates perturbations through iterative optimization. Its core objective is to maximize the degree of divergence between the original policy and the policy after adding perturbations in terms of behavioral performance, thereby disrupting the normal operation of the target policy; (3) DAPGD (Distributed Awareness) Projection gradient descent attack method): This method belongs to the advanced distribution-aware adversarial attack method. Unlike traditional methods that only focus on a single sample or local behavior, it designs and generates perturbations by considering the differences in the overall distribution of the target policy, thereby improving the comprehensiveness of the attack; (4) SARL: This method implements the attack based on the reinforcement learning framework. By directly training the adversarial policy, the adversarial policy minimizes the cumulative reward of the target policy in the process of interacting with the environment, thereby achieving the attack effect; (5) PAAD: This method is a fusion adversarial attack scheme. Its core design is to combine the "director module" and "actor module" based on reinforcement learning, and generate interference through the synergistic effect of the two modules; In the existing technology, this method has reached the current advanced performance level in the indicator of "minimizing the cumulative reward of the target policy", and has strong comparative reference value.

[0132] III. Evaluation Indicators:

[0133] To measure the effectiveness and efficiency of the method proposed in this invention in inducing constraint violations, the following two types of evaluation indicators are used. The definitions, calculation methods, and significance of each indicator are explained below:

[0134] (1) Attack Success Rate (ASR)

[0135] This metric quantifies the proportion of rounds in which the target's safety constraints were successfully violated out of the total number of rounds. The calculation formula is as follows:

[0136]

[0137] In the formula, This indicates the number of rounds in which a constraint violation was successfully triggered. This indicates the total number of rounds in the experiment; the higher the ASR value, the stronger the method's ability to induce constraint violations and the better the attack effectiveness.

[0138] (2) Expected Attack Steps (EAS)

[0139] This metric measures the expected number of attack steps required to reach a constraint violation state across all experimental rounds. For rounds where a constraint violation is not successfully triggered, the number of attack steps is counted as the preset maximum number of steps. The calculation formula is:

[0140]

[0141] In the formula, Consistent with the definition of attack success rate above, i.e., the total number of rounds in the experiment; This represents the number of attack steps required to reach the constraint violation state in the i-th round; rounds that fail to achieve this are represented by [missing value]. The lower the EAS value, the fewer the average attack steps the method takes across all scenarios, and the better the overall attack efficiency.

[0142] IV. Analysis of Experimental Results:

[0143] The performance evaluation of the proposed method and its comparative methods was performed using 100 independent runs, with each run configured with a different random seed to ensure the reliability and statistical validity of the experimental results. In the MuJoCo benchmark environment, the test step size was uniformly set to 1000 for each experiment.

[0144] (1) Performance comparison in white-box environment

[0145] Table 2 shows the performance comparison results in the white-box environment. In the white-box environment, i.e., an environment where the victim's DRL internal parameters and gradient information can be obtained, the proposed method consistently outperforms various baseline methods in tests for different robot models and different safety constraints. In all evaluation scenarios, the proposed method has the highest ASR value and the lowest attack cost (EAS), demonstrating a significant performance advantage. Crucially, compared to all competing baseline methods, the proposed method achieves an ASR exceeding 91% under 7 out of 8 safety constraints, and its effective attack cost (EAS) is reduced by more than 54% compared to other methods, verifying its dual advantages in high attack efficiency and low resource consumption.

[0146] While baseline methods like MAD, SARL, and PAAD can achieve relatively high ASR (Actions Success Rate) under certain constraints, they cannot maintain consistent attack success rate under all constraints. Furthermore, these baseline methods typically require higher attack costs; a typical example is the SARL method in the Ant robotics environment, which failed to induce any safety constraint violations in experiments, rendering the attack completely ineffective. From a technical perspective, the core reason for the failure of these baseline methods lies in the fact that their optimization objective of "minimizing cumulative reward" does not necessarily lead to a "violation of safety constraints." This objective can be achieved either by driving the victim robot to violate constraints or by guiding the robot to perform actions opposite to the task objective but not violating constraints. This discrepancy between the attack target and the optimization objective affects the attack's reliability.

[0147] Table 2. Comparison of the proposed method with other methods in the white-box scenario.

[0148]

[0149] (2) Performance comparison under black-box environment

[0150] To further verify the environmental adaptability of the proposed method, its performance was evaluated in a black-box environment, i.e., an environment in which the internal parameters and gradient information of the victim's DRL agent cannot be obtained. In this environment, the proposed method trains an alternative DRL agent to approximately simulate the behavioral characteristics of the victim's DRL agent, which serves as the basis for attack optimization. Except for the difference in environment type, all other experimental configurations are consistent with the white-box environment. The experimental results are summarized in Table 3.

[0151] As shown in Table 3, even under the constraints of a black-box environment, the proposed method maintains strong attack performance. Specifically, under four of the eight security constraints, the proposed method achieves a near-perfect ASR (close to 100%); under the remaining four constraints, its ASR remains above 50%, significantly outperforming all baseline methods, demonstrating that the proposed method still possesses efficient attack capabilities in black-box scenarios.

[0152] Furthermore, regarding attack cost metrics, the proposed method consistently maintains the lowest EAS under most constraints, further highlighting its comprehensive advantages in black-box environments. A comparison of experimental results in black-box and white-box environments shows that the proposed method maintains a comparable ASR level under most constraints. This result directly verifies the effectiveness of the attack framework designed in this invention based on a replacement DRL agent. Even if the internal information of the victim's DRL agent cannot be directly obtained, attack performance can still be guaranteed by replacing the DRL agent.

[0153] Table 3. Comparison of the proposed method with other methods under black-box conditions.

[0154]

[0155] (3) Security constraint counterattack defense experiment

[0156] To verify the effectiveness of the attack detection and defense method proposed in this invention, defense performance tests were conducted in the same MuJoCo robot control environment as the attack experiment, with attack detection accuracy as the core evaluation index. Table 4 shows the results of the security constraint adversarial attack defense experiment. The experimental results demonstrate that the defense detection method based on dynamic model state prediction residuals and sliding time windows proposed in this invention possesses reliable detection performance in various robot control tasks, verifying the effectiveness of the defense mechanism of this invention.

[0157] Table 4 Security Constraint Counter-Attack Defense Experiment

[0158]

[0159] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention, all of which fall within the scope of protection claimed by the present invention. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A model-based deep reinforcement learning security constraint adversarial attack method, characterized in that, Includes the following steps: (1) The DRL agent is attacked by an attacker and obtains the migration dataset; A robust dynamics model is constructed based on a transfer dataset; a first loss function is constructed, and the robust dynamics model is trained to minimize the loss based on the first loss function and the transfer dataset to obtain a trained robust dynamics model; (2) Based on the trained robust dynamics model, simulate the trajectory of the DRL agent after being disturbed, perform gradient optimization on the disturbance sequence of the trajectory to generate a target disturbance sequence, and the target disturbance sequence accurately induces the DRL agent to violate the target safety constraints; (3) Based on the target perturbation sequence, obtain the migration dataset of the target safety constraint boundary region, and perform secondary training on the trained robust dynamic model according to the migration dataset of the boundary region, so as to obtain the fine-tuned robust dynamic model; (4) Define and obtain the illegality perception value function, and perform minimum loss training on the illegality perception value function to obtain the optimal attack time. Start step (2) according to the optimal attack time, thereby accurately inducing the DRL agent to violate the target security constraints. (5) Construct an alternative DRL agent and obtain a trained alternative DRL agent to simulate the behavior of the DRL agent, thereby supporting the generation of the target perturbation sequence under black box conditions, and accurately inducing the alternative DRL agent to violate the target security constraints under black box conditions based on the target perturbation sequence under black box conditions. (6) Entering the offline stage: For the target security constraints violated by the DRL agent in step (4), execute steps (1)-(4), and collect attack residual data and train a classifier based on the state prediction residual of the robust dynamics model and the preset sliding time window; Entering the online detection phase: The dataset is input into the trained classifier to detect and identify security constraint adversarial attacks, and by triggering an alarm, the security defense of the deep reinforcement learning agent is completed.

2. The model-based deep reinforcement learning security constraint adversarial attack method according to claim 1, characterized in that, In step (1), the process of obtaining the migration dataset is as follows: The attacker targets the victim DRL agent, first manipulates it to interact in the task environment, and then injects randomly generated small perturbations during the interaction. These small perturbations attack the DRL agent, and the attacker collects the migration dataset of the DRL agent's normal state and the attacked perturbation state. ,in For the current state, For DRL agent actions, For instant rewards, For the next state, a total of 1,000,000 migration data entries were collected.

3. The model-based deep reinforcement learning security constraint adversarial attack method according to claim 1, characterized in that, In step (1), the first loss function is: in, Indicates the state and actions Under the condition of, predict the next state, Representing state and actions Under the conditions, the predicted reward, and This represents the actual next state and the current reward; in the predicted next state... Based on this, plus the prediction of rewards This is to add auxiliary information during the training process to enhance the performance of state prediction.

4. The model-based deep reinforcement learning security constraint adversarial attack method according to claim 1, characterized in that, Step (2) specifically includes the following sub-steps: (2.1) Based on the trained robust dynamics model and DRL agent, simulate the trajectory after perturbation; (2.2) Based on the perturbed trajectory, construct a cumulative attack loss function with the aim of inducing violations of target security constraints and protecting non-target security constraints; (2.3) Based on the cumulative attack loss function, calculate the gradient of the cumulative attack loss with respect to each perturbation in the perturbation sequence; (2.4) Optimize the gradient to obtain the target perturbation sequence; the optimization process specifically involves training the perturbation sequence based on the cumulative attack loss function.

5. The model-based deep reinforcement learning security constraint adversarial attack method according to claim 1, characterized in that, Step (4) specifically includes the following sub-steps: (4.1) Define a violation-perceived value function to estimate the efficiency of launching an attack from the current state; (4.2) A three-layer MLP is used to fit the defined violation perceived value function to obtain the violation perceived value function; (4.3) Collect the training dataset for the violation of the perceived value function. The training dataset includes whether the attack was successful or not. If the attack was successful, record the number of attack steps required. (4.4) Based on the attack results, obtain the smoothed target value; (4.5) Construct the second loss function, which is: in, Indicates the state Expectation This indicates a violation of the perceived value function for the current state. The predictive value of the output. This represents the target value after smoothing. This represents the squared difference between the value prediction and the target value for that state. (4.6) The violation-aware value function is trained by minimizing the loss based on the second loss function, the training dataset, and the smoothed target value to obtain the optimal attack timing; (4.7) Initiate step (2) according to the optimal attack timing, thereby accurately inducing the DRL agent to violate the target security constraints.

6. The model-based deep reinforcement learning security constraint adversarial attack method according to claim 1, characterized in that, Step (5) specifically includes the following sub-steps: (5.1) A three-layer multilayer perceptron (MLP) is used as the skeleton of the alternative DRL agent to fit the DRL policy behavior, thereby obtaining the alternative DRL agent; (5.2) Obtain the initial dataset through initial collection; A dynamic dataset is obtained through iterative expansion, which supplements the initial dataset; the dynamic dataset and the initial dataset serve as the training dataset for the alternative DRL agent. (5.3) Construct a third loss function; train the alternative DRL agent based on the third loss function and the training dataset of the alternative DRL agent to obtain the trained alternative DRL agent; (5.4) The final trained replacement DRL agent will completely replace the DRL agent and be used to accurately induce the replacement DRL agent to violate the target security constraints in the black box situation.

7. The method according to claim 1, characterized in that, Step (6) specifically includes the following sub-steps: (6.1) Entering the offline stage: For the target security constraints violated by the DRL agent in step (4), execute steps (1)-(4), collect the state prediction residuals of the robust dynamics model, and construct a labeled sliding time window residual dataset based on the state prediction residuals with a preset sliding time window. (6.2) Preprocess and class balance the labeled sliding time window residual dataset, and train the classifier. Use the trained classifier to detect the label of each sample in the sliding time window residual dataset to realize the identification of normal state and attack type. (6.3) Enter the current detection stage: Input the dataset into the trained classifier, output the identification of normal state and attack type and the probability of each type of attack. When the attack type probability exceeds 0.5 and the predicted category is attack, it is determined that an attack has been suffered and an alarm is triggered. Thus, in the online detection stage, the detection, identification and alarm of security constraint adversarial attacks are realized, and the security defense of deep reinforcement learning agent is completed.

Images