Deep reinforcement learning hidden backdoor attack method based on potential space triggering

Abstract

The invention provides a deep reinforcement learning hidden backdoor attack method based on potential space triggering, and belongs to the field of artificial intelligence security. According to the method, a deep reinforcement learning task environment image is coded into a potential space feature vector representation through a carefully trained BVAE network, a potential space vector is disturbed in a specific state of agent operation, and then a BVAE network decoder is utilized to restore the potential vector back to a pixel space to obtain a back door state. And then the action of the backdoor state and the reward value are modified to generate backdoor training data for intelligent agent model learning. The trained backdoor model shows well when the backdoor is not triggered, but shows a backdoor behavior when the backdoor state appears, so that the task performance of the model is greatly reduced. According to the method, the potential space vector representation is obtained through the BVAE, and the backdoor is implanted in the potential space vector representation, so that the deep reinforcement learning backdoor attack method with high concealment is realized.

Description

Technical Field

[0001] This invention belongs to the field of deep reinforcement learning and artificial intelligence security, specifically involving a backdoor attack method for deep reinforcement learning. This invention uses a BVAE (Beta Variational Autoencoder) network encoder to encode the image of the deep reinforcement learning task environment into a latent space vector representation, while adding perturbations to the latent space vector. Then, the BVAE network decoder is used to restore the vector back to the pixel space image representation as a backdoor trigger. Background Technology

[0002] Deep reinforcement learning has been widely applied across various fields. Unlike traditional supervised learning, which relies on large amounts of manually labeled data, deep reinforcement learning constructs an "agent-environment" interaction mechanism, allowing the agent to autonomously learn optimal decision-making strategies through interaction with the environment. This learning model enables it to efficiently handle decision-making problems in complex and dynamic scenarios, and has already demonstrated significant importance in several key areas such as autonomous driving, game intelligence, and robot control. However, due to the typically high complexity of deep reinforcement learning models, their internal mechanisms are often difficult to understand and explain. This lack of interpretability not only leads to a lack of transparency in model decision-making but also makes its security risk prevention and control passive, becoming a bottleneck restricting the application of deep reinforcement learning in fields with high security requirements. Among numerous security threats, backdoor attacks, due to their strong concealment and wide range of harm, have become a key focus of attention in the field of deep learning security threats. Currently, backdoor attacks have been proven to be harmful in deep neural networks. At the same time, existing research has also demonstrated that deep reinforcement learning is also threatened by backdoor attacks. As a covert attack technique, a deep reinforcement learning model injected with a backdoor behaves normally when the backdoor is not triggered, but executes the backdoor behavior defined by the attacker after the backdoor is triggered. Backdoor attacks not only lead to model decision-making errors but can also cause serious real-world hazards. For example, autonomous driving models may deviate from safe routes after triggering a backdoor, or industrial control robots may perform erroneous operations, causing equipment damage. These threats pose a significant threat to the property, personal safety, and even public safety of model users. Therefore, revealing how backdoor attacks operate in deep reinforcement learning scenarios can not only provide model developers with targeted security protection solutions to reduce the risk of model attacks but also promote the reliable implementation of deep reinforcement learning technology in high-security-demand fields, thus building a solid security barrier for the development of artificial intelligence technology. Summary of the Invention

[0003] Purpose of the Invention: Current backdoor attack methods targeting deep reinforcement learning typically involve adding triggers to the pixel space of the task environment or using specific state sequences as backdoor triggers, which lacks strong concealment. To achieve a more covert backdoor trigger, this invention proposes a covert backdoor attack method for deep reinforcement learning based on latent space triggering.

[0004] Technical solution: The technical solution proposed in this invention is as follows:

[0005] A method for covert backdoor attacks based on latent space triggering using deep reinforcement learning includes the following steps:

[0006] Step 1, Beta-Variational Autoencoder (BVAE) Network Training: Collect task environment image data of the deep reinforcement learning agent and train a high-performance BVAE network. This network encodes the pixel space of the agent's task environment image into a vector representation of the latent space.

[0007] Step 2, Clean Model Training: Train a deep reinforcement learning model with good policy performance. Specifically, we train the agent model using clean data in a clean learning environment. This training method can obtain a clean model with stable performance without any malicious intervention or backdoor mechanisms.

[0008] Step 3: Design the backdoor state trigger: The selected backdoor state is reduced to a vector representation in the latent space using a trained BVAE network. The pre-defined backdoor state data is input into the encoder part of the BVAE network. This encoder is a complex structure composed of multiple neural networks. The encoder calculates and extracts features from the image data. These features are passed and processed layer by layer, and finally mapped to a low-dimensional space. Gaussian noise is then added to the latent vector. After perturbing the latent vector, the decoder of the BVAE network restores the perturbed latent vector to the pixel space.

[0009] Step 4, Backdoor Implantation: During each training round, the agent samples data from the poisoning dataset for learning. The dataset consists of a large amount of clean data and a small amount of backdoor data, where the backdoor data represents data where the attacker has modified the actions and rewards. This design ensures that the agent can learn the backdoor strategy while preserving the performance of the original clean task.

[0010] Step 5, Model Inference: The trained backdoor model performs normally in a clean environment without BVAE network modification; however, when the backdoor model runs in an environment where a backdoor has been implanted in the latent space, the backdoor is triggered, and the agent executes predefined backdoor behaviors, causing the model task to fail.

[0011] Beneficial effects:

[0012] 1. This invention proposes an attack paradigm that utilizes a BVAE network to extract latent space vectors for perturbation. This scheme implants a backdoor in the latent space of the agent's input image through the BVAE network. Compared with existing schemes, this method changes the attack vector from explicit input triggers to implicit feature perturbations, significantly improving the stealth of the attack; by fine-tuning the implanted backdoor, only a small number of backdoor samples are required.

[0013] 2. The backdoor attack method of this invention exhibits good scene adaptability. When testing the attack on deep reinforcement learning-based models, we selected classic task scenarios in the Atari environment, including "Breakout" and "Pong," as experimental environments. Experimental results show that in these two scenarios, the backdoor model achieves over 90% of the performance of the clean model in the clean environment, while the model's performance drops by over 80% in the backdoor environment. This demonstrates the effectiveness of our attack method.

[0014] 3. Robustness experiments verified the robustness of our proposed method. We implemented pruning and adaptive defense on the backdoor model. In pruning defense, we progressively removed backdoor model parameters by setting different pruning rates (0%, 5%, 10%, 15%, 20%). The results show that even at a 20% pruning rate, the model's performance degradation rate after backdoor triggering exceeds 50%, demonstrating that pruning defense cannot completely eliminate the impact of our proposed backdoor method. In adaptive defense, we removed the potential spatial backdoor by eliminating the influence of high-frequency perturbation signals in the frequency domain. Implementation results demonstrate that this defense method also cannot completely eliminate the impact of the backdoor, proving the robustness of our proposed method. Attached Figure Description

[0015] Figure 1 This is a diagram of the backdoor implantation framework.

[0016] Figure 2 This is a flowchart of the overall process for training a backdoor model according to the present invention. Detailed Implementation

[0017] The invention will now be further explained with reference to the accompanying drawings.

[0018] This invention proposes a backdoor attack method for deep reinforcement learning. The core idea of ​​this method is to embed a backdoor trigger into the latent space representation of the DRL task environment image. Specifically, we extract the latent space vector of the DRL task environment state through a carefully trained BVAE network, embed a backdoor by adding perturbations, and then embed the backdoor into the model through data poisoning. This backdoor implantation algorithm mainly consists of the following four steps: 1) training the BVAE network; 2) training a clean DRL model; 3) designing the backdoor trigger; 4) backdoor implantation.

[0019] 1) BVAE Model Training: First, image data of the deep reinforcement learning task environment is collected as training data for the BVAE network. By interacting with the task environment, the agent randomly performs actions to cover multiple scenes, continuously collecting task images and preprocessing them as training data for the BVAE model. The encoder part of the BVAE network uses a three-layer convolutional neural network to extract spatial features step by step, and finally outputs the mean μ and logarithmic variance σ of the latent vector, thereby further obtaining the latent spatial distribution parameters. Then, the latent vector z is generated by sampling in the Gaussian distribution defined by the mean μ and variance σ. The BVAE network encoder processing process is shown in the following formula (1): x→q φ (z|x)=N(μ,σ 2 →z (1)

[0020] The decoding process uses convolution to reconstruct features, and the feature vectors are then convolved through three channels to finally output the reconstructed image. The process is shown in formula (2) below:

[0021] The core function of the BVAE network is to generate high-quality reconstructed images. During each training round, for each batch of input images, the BVAE encoder-decoder network generates reconstructed images. Simultaneously, the loss function is calculated using the mean μ and log-variance σ. This loss function consists of two parts: reconstruction loss and KL divergence. The reconstruction loss employs a hybrid of MSE and SSIM losses, preserving pixel-level errors while also considering structural similarity. By minimizing the reconstruction error, the model learns key features of the input data, ensuring that the latent space vector fully retains the original information, providing a reliable foundation for subsequent tasks. KL divergence measures the difference between the posterior distribution of the latent space vector and the standard Gaussian prior distribution, ensuring that the latent space vector representation conforms to a Gaussian distribution. This avoids overfitting and imparts continuity and interpretability to the latent space.

[0022] 2) Clean DRL Model Training: Training a deep reinforcement learning model with good policy performance. Specifically, we train the agent model using deep reinforcement learning algorithms in a clean learning environment. This training method can obtain a clean model with stable performance without any malicious intervention or backdoor mechanisms.

[0023] 3) Designing a backdoor trigger: Select the state where the backdoor is to be implanted, and then use the encoder of the trained BVAE network to encode the pixel space of the backdoor state image to obtain the feature vector z in the latent space. Subsequently, Gaussian noise perturbation is added to this latent vector. Specifically, first, a feature vector with the same dimension as the feature vector and following a Gaussian distribution N(0, σ) is generated. 2 The random noise vector t is normalized and superimposed on the original feature vector as a backdoor sample. The perturbed potential vector z′ is obtained, and the perturbation process is shown in the following formula (3):

[0024] Where z is the original feature vector, t is the generated noise vector, z′ is the perturbed backdoor feature vector, and ε is a hyperparameter controlling the noise intensity. Adjusting the value of ε controls the magnitude of the perturbation. When ε is small, the perturbation slightly alters the latent vector, affecting only subtle features of the backdoor state; however, when ε is large, the perturbation magnitude increases, significantly altering the features of the latent vector and causing substantial changes in the appearance and triggering effect of the generated backdoor trigger. After perturbing the latent vector, the decoder of the BVAE network restores the perturbed latent vector to the pixel space to obtain the perturbed backdoor state.

[0025] 4) Backdoor Implantation: A fine-tuning method is used to implant the backdoor. During the training phase, a poisoned dataset consisting of clean samples and a small number of backdoor samples is used for training. The clean dataset is shown in formula (4) below:

[0026] in Indicates a clean state. Actions performed under a clean policy. The reward is for a normal environment. The backdoor dataset is shown in formula (5) below:

[0027] in Indicates the backdoor status. This is the action performed under the backdoor strategy. Rewards for backdoor environments.

[0028] The collected clean data and backdoor data are combined into a poisoning dataset and stored in the experience replay buffer R=D. clean ∪D backdoor In each training round, the agent selects experience samples from the experience replay buffer R according to a specific sampling strategy to update the policy network. Since the majority of the mixed data consists of undisturbed normal samples, the agent can form a high-performing main policy during the learning process, exhibiting normal behavior patterns when no backdoor is triggered. However, when the model encounters backdoor samples, the backdoor samples... This will cause the policy network to learn backdoor policies under specific backdoor conditions, leading the agent to deviate from its original target behavior and exhibit abnormal backdoor actions pre-set by the attacker. Since there are only a few backdoor samples in the poisoning dataset, the entire training process can remain stable, ensuring the stability of the model's performance in the absence of backdoor triggering.

[0029] To evaluate the effectiveness of this backdoor method, we used CDA and AER indices as evaluation criteria. Clean Data Accuracy (CDA): The relative performance of the backdoor model and the clean model in a triggerless environment after triggers are used during training. The higher the CDA, the better the normal performance of the backdoor model is preserved. • Attack Effectiveness Rate (AER): The average decrease in model performance after the trigger is activated during the inference phase, compared to a clean environment. A higher AER indicates a greater degree of performance degradation of the backdoor model after the backdoor is activated.

[0030] To better explore the impact of the set parameters on experimental performance, we also evaluated the impact of two parameter settings in this method on the results: β of the BVAE network and perturbation strength ε.

[0031] The Impact of β on BVAE Reconstruction Quality: This method extracts latent vectors through a BVAE network. The parameter β affects the reconstruction quality of the environmental image, thus impacting the backdoor learning efficiency of the model. Therefore, we explore the mechanism by which the parameter β affects the proposed method. In the BVAE network, the β parameter is used to balance the weight allocation between image reconstruction quality and KL divergence: when β < 1, the loss function prioritizes image reconstruction quality; when β > 1, the model ensures the consistency between the prior and posterior distributions; when β = 1, the reconstruction quality and the continuity of the latent space distribution are balanced. To verify the influence of β, we trained BVAE networks with different β values ​​(0.1, 0.3, 0.5, 1, 3, 5) and analyzed their effect on the performance of the proposed method. Experimental results show that as the β parameter increases, the clean data accuracy gradually decreases. This is because increasing β leads to a decrease in image reconstruction quality, which weakens the model's clean task performance; while the backdoor attack success rate decreases slightly, the change is not significant. Taking into account both the performance of clean missions and the effectiveness of backdoor attacks, the optimal β parameter was ultimately selected as 0.1.

[0032] Impact of perturbation amplitude ε on the proposed method: We evaluated the impact of perturbation amplitude ε on the performance of the proposed method. Perturbation amplitude ε refers to the perturbation strength applied to the latent space vectors extracted by the BVAE network. The experimental settings included a value range of ε from 0.1 to 0.3, with a step size of 0.05. Under these conditions, the clean data accuracy and backdoor attack success rate of the backdoor model were tested. As the perturbation amplitude ε increases, the backdoor attack success rate gradually increases, because stronger perturbations enable the model to learn backdoor trigger features more efficiently; conversely, the clean data accuracy decreases, indicating that excessive perturbation interferes with the model's normal task performance in a clean environment. Furthermore, considering that a large perturbation amplitude would compromise the visual concealment of the backdoor trigger, and weighing the attack effectiveness against concealment requirements, we ultimately recommend an optimal perturbation amplitude ε of 0.2.

[0033] To evaluate the robustness of the method of this invention, we performed pruning and adaptive defense on the method.

[0034] Pruning: We evaluated the impact of pruning on model performance by sequentially removing 0%, 5%, 10%, 15%, and 20% of the weights in the backdoor model and setting them to zero. Experimental results show that both clean data accuracy and attack success rate gradually decrease with increasing pruning ratio: the decrease in clean data accuracy indicates that pruning impairs the model's ability to handle tasks in clean environments, while the decrease in attack success rate indicates that pruning has some impact on the backdoor trigger, but the attack success rate is still above 50%. Therefore, the decrease in both clean data accuracy and attack success rate is not significant, and pruning will disrupt the model's normal task performance. In conclusion, pruning cannot eliminate the backdoor implanted by this method.

[0035] Adaptive Defense: To verify the robustness of the proposed backdoor model, this paper designs an adaptive defense. Assuming the model user detects a potential security threat and takes proactive defense measures, when deploying a third-party trained model, the user discovers a maliciously implanted backdoor through security pre-detection. This backdoor is triggered by the latent space of the image in the DRL task environment. Therefore, the user performs frequency domain analysis of the latent space of the DRL task environment image and attempts to remove the abnormal backdoor portion to restore model performance. Specifically, existing work has shown that backdoor attacks and noise perturbations are mostly concentrated in the high-frequency region of the image frequency domain, while the effective information of natural images is mainly distributed in the low-frequency region. Low-pass filtering can remove high-frequency perturbations by suppressing the amplitude of high-frequency components in the frequency domain and retaining clean low-frequency signals. Therefore, we perform frequency domain detection analysis on the input environment image of the backdoor model and remove high-frequency signals. The specific experimental steps are as follows:

[0036] 1) A two-dimensional Type-II Discrete Cosine Transform (2D Type-IIDCT) is used to convert the image from the spatial domain to the frequency domain. This transformation maps the spatial domain pixel values ​​of the DRL task environment image to a frequency domain coefficient matrix through orthogonal cosine functions. This frequency domain transformation can concentrate the effective semantic information of the natural image in the low-frequency region, while perturbations such as backdoor triggers correspond to high-frequency components.

[0037] 2) Three classic low-pass filters were used to remove high-frequency disturbances. The ideal low-pass filter (ILPF) uses a rigid threshold to divide high and low frequencies, retaining all coefficients within the cutoff frequency and setting them to zero outside, thus completely cutting off high-frequency disturbances. The Gaussian low-pass filter (GLPF) achieves smooth attenuation of high-frequency components through a Gaussian function. The Butterworth low-pass filter (BLPF) controls the steepness of the transition between high and low frequencies through its order parameter. In the experiment, the cutoff frequency was uniformly set to 30% of the maximum frequency range, the ratio of the standard deviation of the Gaussian filter to the cutoff frequency was set to 2.35:1, and the order of the Butterworth filter was set to 2 to ensure the comparability of the filtering parameters.

[0038] 3) The frequency-domain filtered coefficient matrix is ​​reconstructed into a spatial domain image using Inverse Discrete Cosine Transform (IDCT), resulting in a clean image free of backdoor perturbations. IDCT, as the inverse operation of 2D Type-II DCT, restores the pixel value distribution based on the orthogonality principle, mathematically achieved by weighted summation of frequency domain coefficients and cosine basis functions. After the above operations, a task environment image with high-frequency information removed from the frequency domain is obtained. Then, the intelligent agent model performs the task in the processed task environment, and the task performance of the backdoor model is recorded.

[0039] Experimental results show that the clean task performance of the backdoor model after adaptive defense is almost unchanged, while the performance after backdoor triggering is slightly lower than before adaptive defense. Specifically, the AER of backdoor triggering decreases by about 34% in the Breakout task environment and about 30% in the Pong environment after processing with the ideal low-pass filter. This decrease may be because the ideal low-pass filter uses a rigid threshold to separate high and low frequencies, truncating all high-frequency signals above the threshold and retaining all low-frequency signals below the threshold, potentially removing some frequency domain perturbation signals. However, the task performance of the backdoor model after adaptive defense is only about 40% of that of the clean model, which is still significantly lower. Therefore, truncating high-frequency signals in the frequency domain cannot completely eliminate the influence of this backdoor method, demonstrating the robustness of this method.

[0040] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for covert backdoor attacks based on latent space triggering using deep reinforcement learning, characterized in that, Including the following steps: Step 1, Beta-Variational Autoencoder (BVAE) Training: Collect environmental data for deep reinforcement learning agents and train a high-performance BVAE network. This network encodes the pixel-space data of the environment into a vector representation in the latent space. The BVAE network includes two processes: encoding and decoding. The encoding process is shown in Equation (1): x→q φ (z|x)=N(μ,σ 2 )→z (1) Where x is the input of the encoder, q φ (z|x) is the approximate posterior distribution defined by the encoder, μ and σ are the mean and variance vectors of the encoder output, respectively, and z is the latent space vector representation obtained by the encoder. The decoding process is shown in formula (2): Where z is the input of the decoder, p θ (x|z) is the prior distribution. This is the output after reconstruction by the decoder; Step 2, Clean Model Training: Train a clean agent model in a clean environment using deep reinforcement learning algorithms. Without malicious backdoors or interference, obtain a model with stable performance and good policy. Step 3: Select the state where the backdoor is implanted, and then use the encoder of the trained BVAE network to encode the pixel space of the backdoor state image to obtain the feature vector z in the latent space. Then, add noise perturbation to this latent vector. Specifically, first generate a vector with the same dimension as the feature vector and following a Gaussian distribution N(0, σ²). 2 The noise vector t is normalized and superimposed on the original feature vector as a backdoor sample. The perturbed potential vector z′ is obtained, and the perturbation process is shown in the following formula (3): Where z is the original feature vector, t is the generated noise vector, z′ is the perturbed backdoor feature vector, and ε is a hyperparameter controlling the noise intensity. Adjusting the value of ε controls the magnitude of the perturbation. When ε is small, the perturbation slightly alters the latent vector, affecting only subtle features of the backdoor state; however, when ε is large, the perturbation magnitude increases, significantly altering the features of the latent vector and causing substantial changes in the appearance and triggering effect of the generated backdoor trigger. After perturbing the latent vector, the decoder of the BVAE network restores the perturbed latent vector to the pixel space to obtain the perturbed backdoor state. Step 4, Backdoor Implantation: The backdoor is implanted using a fine-tuning approach. During the training phase, a poisoning dataset consisting of clean samples and a small number of backdoor samples is used for training. In each round, the agent samples data from the experience buffer for learning. A large amount of clean sample data ensures the backdoor model's performance on clean tasks, while the small amount of backdoor sample data allows the model to learn backdoor behavior strategies. A backdoor model is trained through steps 1, 2, 3, and 4.

2. The method for covert backdoor attacks based on latent space triggering in deep reinforcement learning as described in claim 1, characterized in that... This paper uses a BVAE network as the encoder and decoder. As a deep generative autoencoder, the BVAE network can map high-dimensional deep reinforcement learning task environment images to low-dimensional latent space vector representations, achieving dimensionality reduction. Simultaneously, due to the powerful reconstruction capabilities of the BVAE network, the consistency between the reconstructed image and the original image is guaranteed. This method utilizes the efficient encoding and decoding capabilities of the BAVE network to implant a backdoor in the latent space.

3. The method for covert backdoor attacks based on latent space triggering in deep reinforcement learning as described in claim 1, characterized in that... Perturbations to the latent space vectors are used as backdoor state triggers. Current common deep reinforcement learning backdoor attack methods place the backdoor triggers in the pixel-space observations, which has low stealth capabilities. This study proposes a latent space-triggered deep reinforcement learning covert backdoor attack method. It extracts the latent space vectors of the deep reinforcement learning task environment image through a BVAE network and injects specific perturbations into them to achieve a backdoor attack on the target model.

4. The method for covert backdoor attacks based on latent space triggering in deep reinforcement learning as described in claim 1, characterized in that... A backdoor policy is learned using a data poisoning approach. The poisoned dataset is used as input to the model fine-tuning stage, enabling the deep reinforcement learning model, which has already been trained with a clean policy, to learn the backdoor policy. During fine-tuning, a low learning rate and mini-batch iteration optimization strategy are employed to avoid drastic parameter fluctuations that could corrupt the model's learned clean policy. Simultaneously, the policy and value network parameters of the model are iteratively updated using mini-batch stochastic gradient descent, allowing the model to maintain its decision-making ability on the original task under the supervision of clean samples, while gradually learning the backdoor policy under the guidance of backdoor samples.

Images