Backdoor attack method, system, program and equipment for offline reinforcement learning and storage medium
By manipulating reward values and multi-criteria fusion design of hidden triggers, the concealment and effectiveness of backdoor attacks in offline reinforcement learning systems are solved, and policy interference and performance damage to the model under low poisoning ratio is achieved.
Patent Information
- Application Number
- CN202510516553.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-23
- Publication Date
- 2025-08-08
AI Technical Summary
The existing offline reinforcement learning systems lack in-depth analysis and effective defense of backdoor attacks, especially at low poisoning ratios, it is difficult to conduct covert attacks without significantly affecting the normal performance of the model.
By manipulating the reward value, the model learns suboptimal strategies, and designs hidden triggers based on multi-criteria fusion and dual-objective optimization, accurately locates key time steps and adds hidden constraint perturbations to generate poisoned data sets.
The performance of the model triggering environment is significantly reduced under the low poisoning ratio, while maintaining normal performance in a clean environment, improving the concealment and effectiveness of the attack, and is suitable for different offline reinforcement learning algorithms and tasks.
Smart Images
Figure CN120449977A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of offline reinforcement learning, and in particular to a backdoor attack method, system, program, device and storage medium for offline reinforcement learning. Background Art
[0002] Offline reinforcement learning (also known as batch reinforcement learning) is a method for training a learning agent based on a pre-collected dataset of experience. Unlike online reinforcement learning, which requires real-time feedback from the environment to optimize its policy, offline reinforcement learning can achieve learning without real-time online interaction with the environment. This property makes it particularly suitable for scenarios where exploration is costly, time-consuming, or risky, such as optimizing autonomous driving systems, decision support in healthcare, precise control of intelligent robots, and designing complex game strategies. Backdoor attacks have become a widespread problem in machine learning, initially targeting fundamental models such as logistic regression and support vector machines. Similarly, offline reinforcement learning faces the potential threat of backdoor attacks. However, current research in this area is significantly limited, primarily due to a lack of in-depth analysis of offline datasets and a lack of systematic research on the potential impacts on offline reinforcement learning algorithms and models. Therefore, it is necessary to conduct research on low-cost and effective backdoor attack methods and to thoroughly analyze the specific impacts of these attacks on offline reinforcement learning systems. This will provide important technical support and theoretical basis for building more secure and reliable offline reinforcement learning systems.
[0003] The present invention guides the model to learn suboptimal strategies by manipulating reward values, and combines multi-criteria fusion with dual-objective optimization to design hidden triggers, significantly reducing the model triggering environment performance under low poisoning ratios while ensuring its normal performance in a clean environment. Summary of the Invention
[0004] The purpose of this invention is to guide the model to learn suboptimal strategies by manipulating reward values, and to design hidden triggers by combining multi-criteria fusion and dual-objective optimization, thereby significantly reducing the model triggering environment performance at low poisoning ratios while ensuring its normal performance in a clean environment.
[0005] The present invention provides a backdoor attack method for offline reinforcement learning, comprising:
[0006] Step 1: Obtain the state space, action space, and reward space in a clean offline dataset;
[0007] Step 2: Use the advantage value calculator to calculate the advantage value of the state-action pair at each time step and select the key time step set for poisoning;
[0008] Step 3: Adjust the reward value to obtain poisoning rewards to guide the learning of suboptimal strategies;
[0009] Step 4: Perform multi-criteria fusion analysis on the state characteristics of the key time step set and select the state feature dimension set with the least influence;
[0010] Step 5: Add a hidden constraint perturbation to the selected minimum state feature dimension set to generate a poisoned state. The perturbation δ is solved by a dual-objective optimization method.
[0011] Step 6: Add the solved perturbation to the state-action pair of each data in the critical time step set to obtain the poisonous dataset and complete the backdoor attack.
[0012] Furthermore, the step 2 specifically includes the following steps:
[0013] Step 2.1: Calculate the advantage value A(s) at each time step t t ,a t );
[0014] A(s t ,a t )=Q(s t ,a t )-V(s t )
[0015] Where t is the time step; (s t ,a t ) is the state-action pair for each time step; Q(s t ,a t ) is the state-action value function; V(s t ) is the state value function;
[0016] Step 2.2: Determine the poisoning quantity k according to the poisoning ratio p;
[0017]
[0018] Where N is the total number of all time steps in the dataset;
[0019] Step 2.3: Sort all time steps in ascending order by the size of the advantage value, select the k time steps with the smallest advantage value as the key time steps, and mark them as the key time step set D poisoned .
[0020] Furthermore, the step 3 specifically includes the following steps:
[0021] Step 3.1: According to the key time step set D poisoned , get the reward value r for each time step t ;
[0022] Step 3.2: Adjust the reward value r at each time step t Get poison reward value r t ';
[0023] r t '=r t -A(s t ,a t )+ζ
[0024] Among them, ζ is the smallest positive number.
[0025] Furthermore, the step 4 specifically includes the following steps:
[0026] Step 4.1: Extract the key time step set D poisoned The corresponding state feature s t ;
[0027] Step 4.2: Use the SHAP method to analyze the local contribution of each state feature dimension to the model prediction by calculating the average absolute SHAP value of each state feature dimension in all samples Evaluate the local importance of each state dimension;
[0028]
[0029] Where N is the total number of all time steps in the dataset; is the SHAP value of state feature dimension i in sample t;
[0030] Step 4.3: Use the mutual information method to analyze the global dependency of each state feature dimension and the model prediction output, and calculate the mutual information value of each dimension
[0031]
[0032] in, is the state feature dimension s i With model output The joint probability distribution of p(s i )and are their marginal distributions respectively;
[0033] Step 4.4: Average SHAP values and mutual information value Perform normalization processing to obtain standardized indicators of each dimension and
[0034]
[0035] Among them, j represents the index of all state feature dimensions;
[0036] Step 4.5: Integrate the above standardized indicators and calculate the comprehensive importance score W of each state feature dimension i for:
[0037]
[0038] Among them, α1 and α2 are fusion coefficients, satisfying α1+α2=1;
[0039] Step 4.6: Score W based on comprehensive importance i Sort all state feature dimensions in ascending order, and select the dimensions with the lowest scores to form the state feature dimension set S with the least influence trigger .
[0040] Furthermore, the dual-objective optimization model is as follows:
[0041]
[0042] Among them, J(δ) is the attack effectiveness objective function; A(s t +δ,a t ) is the advantage value in the poisoning state; the optimization goal is to minimize the L2 norm of the disturbance δ, Min||δ||;
[0043] The poisoning state t for:
[0044] s′ t =s t +δ.
[0045] The present invention also provides a backdoor attack system for offline reinforcement learning, comprising a key time step positioning module, a fine-tuning reward module and a hidden trigger design module; the key time step positioning module is used to identify key time steps in an offline data set that may induce a model to execute a suboptimal strategy, and form a time step set; the fine-tuning reward module is used to adjust the reward value of the key time step, and guide the model to learn the suboptimal strategy by reducing the reward value of the key time step; the hidden trigger design module is used to design and embed hidden triggers in the state characteristics of the key time step, and attack the model with both concealment and effectiveness through perturbation, comprising a multi-criteria fusion module, a perturbation generation module and a trigger embedding module; the multi-criteria fusion module is used to analyze the state characteristics corresponding to the key time step, and in combination with a multi-criteria evaluation method, select the state dimension with the least impact on system performance; the perturbation generation module calculates the optimal perturbation value of the trigger through a dual-objective optimization method.
[0046] Furthermore, based on the input clean offline reinforcement learning dataset, the key time step positioning module uses the advantage value calculator to calculate the advantage value of each time step, and screens out the key time steps with suboptimal strategies based on the advantage value; then the reward value is adjusted by the fine-tuning reward module to guide the learning of suboptimal strategies; then the state characteristics of the key time steps are analyzed by the multi-criteria fusion module, and the dimension with the least impact on the model performance is selected as the trigger embedding point; the perturbation generation module uses a dual-objective optimization method to generate hidden trigger perturbation values; finally, the trigger embedding module embeds the trigger perturbation values into the state characteristics of the key time steps to complete the dataset poisoning.
[0047] The present invention also provides a computer device / equipment, including a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program, the steps of any of the above-mentioned backdoor attack methods for offline reinforcement learning are implemented.
[0048] The present invention also provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of any of the above-mentioned backdoor attack methods for offline reinforcement learning.
[0049] The present invention also provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of any of the above-mentioned backdoor attack methods for offline reinforcement learning.
[0050] The beneficial effects of the present invention are:
[0051] 1. This paper proposes a backdoor attack method for offline reinforcement learning. It can identify the critical time steps of poor actions during offline reinforcement learning, manipulate rewards for these critical time steps, trigger the model by selecting the dimension that has the least impact on the model's normal tasks, and solve the optimal trigger perturbation value through a dual-objective optimization method. This method improves attack efficiency while ensuring stealth. The method is also applicable to different offline reinforcement learning algorithms and different reinforcement learning tasks. The proposed backdoor attack method is verified on three offline reinforcement learning algorithms: Batch-Constrained Q-learning (BCQ), Batch-Ensemble Actor-Critic with Retrace (BEAR), and Conservative Q-Learning (CQL); Walker2D, Hopper, and Half-Cheetah in the MuJoCo robot simulator; and Breakout in the Atari simulator. When the poisoning ratio is only 1%, our attack can maintain 91.2% of normal performance in a clean environment, but reduce model performance by 86.3% in a triggered environment.
[0052] 2. The perturbation method proposed in this invention can dynamically add tiny perturbations to the dataset that are consistent with its own proportions, which cannot be detected and defended by existing methods.
[0053] 3. This method uses a key time-step screening mechanism based on dominance values to precisely locate policy-sensitive areas. It then combines local explanatory metrics (SHAP values) with global dependency metrics (mutual information) to perform a multi-criteria fusion analysis of state feature dimensions, selecting the dimension with the least impact on model performance as the backdoor trigger entry. This method significantly improves the attack's target accuracy and stealth, effectively misleading the policy learning process while maintaining the model's normal task performance, and possesses greater practicality and robustness against adversarial attacks. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 An overview of the backdoor method for offline reinforcement learning provided by the present invention; DETAILED DESCRIPTION
[0055] The present invention will be further described below with reference to the accompanying drawings.
[0056] The present invention discloses a backdoor attack system for offline reinforcement learning, positioning a key time step module, fine-tuning a reward module, and designing a hidden trigger module;
[0057] Critical Timestep Location Module: This module identifies critical timesteps in offline datasets that may induce the model to execute poor strategies, providing a target set of timesteps for subsequent attack design. The core components of this module are the advantage value calculator, the timestep screening module, and the timestep marking module.
[0058] Advantage Calculator Module: This module is used to calculate the advantage value of each time step in the dataset to evaluate whether the state-action pair is a suboptimal policy.
[0059] Time step screening module: This module locates the key time steps in the offline dataset that may induce the model to execute suboptimal strategies by screening the advantage values.
[0060] Time step marking module: This module is used to record the filtered key time steps for subsequent trigger embedding design.
[0061] Fine-tuning Reward Module: This module adjusts the reward values at key time steps to guide the model to learn suboptimal strategies at these time steps, thereby enhancing the effectiveness of the attack. The core components of this module are the Reward Value Adjustment Module and the Reward Value Update Module.
[0062] Reward value adjustment module: This module fine-tunes the reward value of key time steps and guides the model to learn suboptimal strategies by increasing the reward value of key time steps.
[0063] Reward value update module: This module is responsible for updating the reward value of each time step in the dataset.
[0064] Hidden Trigger Design Module: This module is responsible for designing and embedding hidden triggers in the state features at key time steps, enabling attacks on the model through perturbations that balance stealth and effectiveness. The core components of this module are the Multi-Criteria Fusion Module, the Perturbation Generation Module, and the Trigger Embedding Module.
[0065] Multi-criteria fusion module: This module is responsible for analyzing the state characteristics of key time steps, combining multi-criteria evaluation methods, and selecting the dimension with the least impact on system performance as the trigger embedding point.
[0066] Perturbation Generation Module: This module uses a dual-objective optimization method to generate the optimal perturbation value of the trigger. The optimization goal is to maximize the attack effectiveness and minimize the perturbation concealment, so as to maximize the concealment while ensuring the attack effectiveness.
[0067] Trigger embedding module: This module embeds the calculated disturbance value into the state feature of the key time step, generates the toxic state feature and forms a toxic data set.
[0068] The above module implements a covert backdoor attack on offline reinforcement learning: a clean offline reinforcement learning dataset is input, and the advantage value calculator is used to calculate the advantage value of each time step; the key time steps with suboptimal strategies are screened out based on the advantage value, and the fine-tuning reward module fine-tunes the reward value to guide the learning of suboptimal strategies; the multi-criteria fusion module analyzes the state characteristics of the key time steps and selects the dimension with the least impact on model performance as the trigger embedding point; the perturbation generation module uses a dual-objective optimization method to generate hidden trigger perturbation values; the trigger embedding module embeds the trigger perturbation values into the state characteristics of the key time steps to complete the dataset poisoning.
[0069] The present invention discloses a backdoor attack method for offline reinforcement learning, comprising:
[0070] 1) Obtain the state space, action space, and reward space in a clean offline dataset;
[0071] 2) Use the advantage value calculator to calculate the advantage value of the state-action pair at each time step and select the key time step set for poisoning;
[0072] 3) Fine-tuning the reward value to obtain poisoning rewards to guide learning of poor strategies;
[0073] 4) Perform multi-criteria fusion analysis on the state characteristics of the key time step set, analyze the contribution of each state dimension to system performance in combination with the performance impact weight, and select the state feature dimension with the least impact;
[0074] 5) Adding hidden constraint perturbations to the selected dimensions to generate poisoned states, where the perturbations are solved using a dual-objective optimization method;
[0075] 6) Add the solved perturbation to the state-action pair of each data in the critical time step set to obtain the poisonous data set.
[0076] Example 1
[0077] like Figure 1 As shown in FIG, a backdoor attack method for offline reinforcement learning includes the following steps:
[0078] 1) Obtain the state space S, action space A, and reward space R in a clean offline dataset D;
[0079] 2) Use the advantage value calculator to calculate the state-action pair (s) at each time step t t ,a t )Calculate the advantage value A(s t ,a t ), and select the key time step set D to be poisoned poisoned ;
[0080] 201) Calculate the advantage value A(s) at each time step t ,a t )=Q(s t ,a t )-V(s t ), where Q(s t ,a t ) represents the state-action value function, V(s t ) represents the state value function;
[0081] 202) Determine the amount of poisoning based on the poisoning ratio p Where N is the total number of all time steps in the dataset;
[0082] 203) According to the advantage value A(s t ,a t ) sort all time steps in ascending order, select k time steps with the smallest advantage value as key time steps, and mark them as the key time step set D poisoned ;
[0083] 3) Fine-tuning the reward r to obtain a poisoning reward r' to guide the learning of poor strategies;
[0084] 301) Multiple key time step sets D poisoned , get the reward value r for each time step t ;
[0085] 302) Fine-tune the reward value of each time step to obtain the poisoning reward r' t =r t +η t =r t -A(s t ,a t ) + ζ, so that the originally negative advantage value is just turned into a positive value, thereby inducing the model to learn a suboptimal strategy. Here ζ is the smallest positive number used to ensure that the advantage value is strictly greater than zero, and is taken as 0.01 here;
[0086] 4) For the key time step set D poisoned The state characteristics are analyzed by multi-criteria fusion, and the state feature dimension S with the least influence is selected. trigger ;
[0087] 401) Extract the state feature s corresponding to the key time step from the data set t ;
[0088] 402) The SHAP (Shapley Additive Explanations) method is used to analyze the local contribution of each state feature dimension to the model prediction, by calculating the average absolute SHAP value of each state feature dimension in all samples. Evaluate the local importance of each state dimension, specifically:
[0089]
[0090] Where N is the total number of samples, is the SHAP value of state feature dimension i in sample t;
[0091] 403) Use the mutual information (MI) method to analyze the global dependency between each state feature dimension and the model prediction output, and calculate the mutual information value of each dimension Specifically:
[0092]
[0093] in, is the state feature dimension s i With model output The joint probability distribution of p(s i )and are their marginal distributions respectively;
[0094] 404) average SHAP value and mutual information value Perform normalization processing to obtain standardized indicators of each dimension and Specifically:
[0095]
[0096] Among them, j represents the index of all state feature dimensions;
[0097] 405) Combine the above standardized indicators to calculate the comprehensive importance score W of each state feature dimension i , specifically:
[0098]
[0099] Among them, α1 and α2 are fusion coefficients, satisfying α1+α2=1;
[0100] 406) According to the comprehensive importance score W i Sort all state feature dimensions in ascending order, and select the dimensions with the lowest scores to form the state feature dimension set S with the least influence trigger .
[0101] 5) For the selected dimension S trigger Add hidden constraint perturbation δ to generate poisoned state s' t =s t +δ, where the perturbation δ is solved by a dual-objective optimization method.
[0102] 501) Optimization goal 1 is to maximize the effectiveness of the attack and reduce the performance of the model in the triggering environment. The optimization goal is defined as:
[0103] maximizeδJ(δ)=A(s t +δ,a t )-||δ|| 2
[0104] Among them, J(δ) represents the attack effectiveness objective function, A(s t +δ,a t ) represents the advantage value in the poisoning state, ||δ|| 2 is the L2 norm of δ, indicating the perturbation intensity.
[0105] 502) The optimization goal 1 is to minimize the L2 norm of the perturbation δ to ensure concealment. The optimization goal is defined as:
[0106] Min||δ||2
[0107] 503) By jointly solving the two optimization objectives, a disturbance value δ is generated that can reduce the model triggering environment performance and has strong concealment.
[0108] In summary, the present invention achieves covert poisoning and strategy induction of the model by locating key time steps, fine-tuning reward values, and designing covert triggers. First, a clean offline data set containing state space and action space is obtained; then, the advantage value calculator is used to calculate the advantage value of each time step, and the set of key time steps that induce the model to execute suboptimal strategies is located through screening; then, by fine-tuning the reward value of the key time step, a poisoned reward is generated to guide the model to learn suboptimal strategies; then, a multi-criteria fusion analysis is performed on the state characteristics of the key time step, and the feature dimension with the least impact on system performance is selected as the trigger embedding point; finally, based on the dual-objective optimization method, a disturbance value that meets the concealment constraint is generated and embedded into the state characteristics of the key time step, thereby constructing a poisoned data set. The present invention can achieve covert and efficient backdoor attacks without significantly affecting the data distribution, and is suitable for strategy interference and performance destruction in offline reinforcement learning scenarios.
[0109] In particular, in some preferred embodiments of the present invention, a computer device is also provided, comprising a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program, the steps of the backdoor attack method for offline reinforcement learning described in any of the above embodiments are implemented.
[0110] In other preferred embodiments of the present invention, a computer-readable storage medium is provided, on which a computer program / instructions are stored. When the computer program is executed by a processor, the steps of the backdoor attack method for offline reinforcement learning described in any of the above embodiments are implemented.
[0111] Those skilled in the art will appreciate that all or part of the processes in the above-described embodiment method can be implemented by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-described embodiment of the backdoor attack method for offline reinforcement learning, which will not be repeated here.
[0112] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.
[0113] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "N" means at least two, such as two, three, etc., unless otherwise specifically defined.
[0114] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing a custom logical function or step of a process, and the scope of the preferred embodiments of the invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of the invention pertain.
[0115] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or N wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.
[0116] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0117] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.
[0118] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.
[0119] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and are not to be construed as limiting the present invention. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.
Claims
1. A backdoor attack method for offline reinforcement learning, characterized by: include: Step 1: Obtain the state space, action space, and reward space in a clean offline dataset; Step 2: Use the advantage value calculator to calculate the advantage value of the state-action pair at each time step and select the key time step set for poisoning; Step 3: Adjust the reward value to obtain poisoning rewards to guide the learning of suboptimal strategies; Step 4: Perform multi-criteria fusion analysis on the state characteristics of the key time step set and select the state feature dimension set with the least influence; Step 5: Add a hidden constraint perturbation to the selected minimum state feature dimension set to generate a poisoned state. The perturbation δ is solved by a dual-objective optimization method. Step 6: Add the solved perturbation to the state-action pair of each data in the critical time step set to obtain the poisonous dataset and complete the backdoor attack.
2. The backdoor attack method for offline reinforcement learning according to claim 1, characterized in that: The step 2 specifically includes the following steps: Step 2.1: Calculate the advantage value A(s) at each time step t t ,a t ); A(s t ,a t )=Q(s t ,a t )-V(s t ) Where t is the time step; (s t ,a t ) is the state-action pair for each time step; Q(s t ,a t ) is the state-action value function; V(s t ) is the state value function; Step 2.2: Determine the poisoning quantity k according to the poisoning ratio p; Where N is the total number of all time steps in the dataset; Step 2.3: Sort all time steps in ascending order by the size of the advantage value, select the k time steps with the smallest advantage value as the key time steps, and mark them as the key time step set D poisoned .
3. The backdoor attack method for offline reinforcement learning according to claim 1, characterized in that: The step 3 specifically includes the following steps: Step 3.1: According to the key time step set D poisoned , get the reward value r for each time step t ; Step 3.2: Adjust the reward value r at each time step t Get poison reward value r t '; r t '=r t -A(s t ,a t )+ζ Among them, ζ is the smallest positive number.
4. The backdoor attack method for offline reinforcement learning according to claim 1, characterized in that: The step 4 specifically includes the following steps: Step 4.1: Extract the key time step set D poisoned The corresponding state feature s t ; Step 4.2: Use the SHAP method to analyze the local contribution of each state feature dimension to the model prediction by calculating the average absolute SHAP value of each state feature dimension in all samples Evaluate the local importance of each state dimension; Where N is the total number of all time steps in the dataset; is the SHAP value of state feature dimension i in sample t; Step 4.3: Use the mutual information method to analyze the global dependency of each state feature dimension and the model prediction output, and calculate the mutual information value of each dimension in, is the state feature dimension s i With model output The joint probability distribution of p(s i )and are their marginal distributions respectively; Step 4.4: Average SHAP values and mutual information value Perform normalization processing to obtain standardized indicators of each dimension and Among them, j represents the index of all state feature dimensions; Step 4.5: Integrate the above standardized indicators and calculate the comprehensive importance score W of each state feature dimension i for: Among them, α1 and α2 are fusion coefficients, satisfying α1+α2=1; Step 4.6: Score W based on comprehensive importance i Sort all state feature dimensions in ascending order, and select the dimensions with the lowest scores to form the state feature dimension set S with the least influence trigger .
5. The backdoor attack method for offline reinforcement learning according to claim 1, characterized in that: The dual-objective optimization model is as follows: Where δ is the added perturbation; J(δ) is the attack effectiveness objective function; A(s t +δ,a t ) is the advantage value in the poisoning state; the optimization goal is to minimize the L2 norm of the disturbance δ, Min||δ||; The poisoning state s t for: s' t =s t +d.
6. A backdoor attack system for offline reinforcement learning, characterized by: Including positioning key time step module, fine-tuning reward module and designing hidden trigger module; The positioning key time step module is used to identify the key time steps in the offline data set that induce the model to execute the suboptimal strategy, and form a time step set; The fine-tuning reward module is used to adjust the reward value of the key time step, and guide the model to learn suboptimal strategies by reducing the reward value of the key time step. The hidden trigger design module is used to design and embed hidden triggers in the state features of the key time step, and attack the model with both stealth and effectiveness through perturbations. It includes a multi-criteria fusion module, a perturbation generation module, and a trigger embedding module. The multi-criteria fusion module is used to analyze the state characteristics corresponding to the key time step, and combine the multi-criteria evaluation method to select the state dimension with the least impact on system performance; The disturbance generation module calculates the optimal disturbance value of the trigger through a dual-objective optimization method.
7. The backdoor attack system for offline reinforcement learning according to claim 6, characterized in that: Based on a clean offline reinforcement learning dataset, the Key Time Step Location module uses an advantage value calculator to calculate the advantage value of each time step. This advantage value is then used to identify key time steps with suboptimal policies. The fine-tuning reward module then adjusts the reward value to guide learning of suboptimal policies. The multi-criteria fusion module then analyzes the state characteristics of key time steps and selects the dimension with the least impact on model performance as the trigger embedding point. The perturbation generation module uses a dual-objective optimization method to generate hidden trigger perturbation values; Finally, the trigger perturbation value is embedded into the state features of the key time step through the trigger embedding module to complete the dataset poisoning.
8. A computer device / apparatus comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 5.
9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.
10. A computer program product comprising a computer program / instructions, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.