A discrete-continuous hybrid action countermeasure attack method based on deep reinforcement learning

Abstract

The present invention discloses a discrete-continuous hybrid action adversarial attack method based on deep reinforcement learning, which relates to the field of artificial intelligence security. Utilizing the adversarial perturbation attack scheme driven by the dynamic system state, the discrete action perturbation and continuous action perturbation methods are deeply integrated, with the goal of minimizing the system reward, and increasing the output action loss within the maximum adversarial perturbation range. Further combined with the uniform attack and strategy timing attack methods, a specific preference function is designed to flexibly select the adversarial attack triggering time method, thereby enhancing the concealment and destructive power of the attack and improving the DRL model attack effect. Combined with the black box attack framework based on the migration paradigm, an adversarial attack migration method based on a substitute model is adopted. By constructing a substitute model to approximate the decision-making behavior of the target strategy, and using the substitute gradient to generate adversarial perturbations, effective model attacks can still be achieved in the absence of target model gradient information, thereby improving the generalization ability of adversarial attacks on DRL model attacks.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence security, and in particular to a discrete-continuous hybrid action counterattack method based on deep reinforcement learning. Background Art

[0002] By integrating deep learning and reinforcement learning frameworks, deep reinforcement learning (DRL) can effectively capture dynamic environmental features, achieving powerful representation capabilities for high-dimensional state spaces and breakthroughs in complex decision-making. However, the neural network components it relies on are also susceptible to adversarial perturbations. The sensitivity of deep reinforcement learning models to input perturbations can pose serious security risks. Attackers can inject subtle adversarial noise to cause the model to make irrational decisions, thereby degrading system service quality. For example, in autonomous driving, adversarial attacks against environmental perception models can mislead vehicle trajectory planning; in healthcare, tampering with medical image inputs can lead to diagnostic errors. Current research on adversarial attacks against deep reinforcement learning focuses primarily on vision-driven scenarios (such as image classification and autonomous driving), and attack methods are typically designed to target static pixel-level data. However, in edge computing-based task offloading scenarios, DRL inputs are essentially dynamic system state vectors, which are governed by real-world physical rules and consist of a small number of parameters with well-defined physical meanings and explicit physical correlations, fundamentally different from images.

[0003] Therefore, it is very necessary to study the adversarial attack problem of deep reinforcement learning models in the task offloading and resource allocation scenarios in edge computing systems, revealing the potential vulnerability of DRL strategies driven by dynamic system states under adversarial perturbations, and further injecting new vitality into the security research of deep reinforcement learning models in adversarial attack scenarios. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to design a discrete-continuous hybrid action adversarial attack method based on deep reinforcement learning for the task offloading and resource allocation scenarios in edge computing systems, evaluate the destructive intensity of the attack on core indicators such as latency, energy consumption, and service quality, and finally design a black-box attack framework based on the migration paradigm. By constructing an alternative model to approximate the decision-making behavior of the target strategy and using alternative gradient information to generate adversarial perturbations, the potential security risks of DRL in task offloading and resource allocation in edge computing scenarios are revealed, providing theoretical support for building a robust edge intelligent computing framework.

[0005] The technical solution adopted by the present invention to solve the above technical problems is that the attacker obtains the original state observed by the intelligent agent in the environment, uses the discrete-continuous hybrid action adversarial perturbation attack method based on deep reinforcement learning to generate adversarial perturbations, and superimposes the adversarial perturbations and the original system observations as a new observation state to send to the intelligent agent, causing the intelligent agent to make irrational decision-making actions. On this basis, the black box attack framework based on the transfer paradigm is further used to conduct adversarial attacks on the DRL model by training the gradient information of the surrogate model without the gradient information of the reinforcement learning model. Specifically, the following steps are included:

[0006] S1: Building a deep reinforcement learning model based on task offloading and resource allocation based on Markov decision process;

[0007] S2: The attacker obtains the original state observed by the agent in the task offloading system, uses the discrete-continuous hybrid action adversarial perturbation attack method to generate adversarial perturbations, and superimposes the original system observations as the adversarial perturbation state;

[0008] S3: The attacker sends the generated adversarial perturbation state to the agent;

[0009] S4: The agent makes a decision based on the acquired adversarial disturbance state through the Actor network;

[0010] S5: Based on the decision-making actions affected by the adversarial attack, the difference in rewards between actions made with and without the adversarial perturbation and the difference in total service quality are calculated as indicators for evaluating the effectiveness of the attack.

[0011] S6: Use a black-box attack framework based on the transfer paradigm to train alternative models with similar policy distributions;

[0012] S7: Generate adversarial state perturbations by training the gradient information of the alternative model, and exploit the migration characteristics of adversarial samples on the decision boundary to attack the original model.

[0013] Furthermore, the deep reinforcement learning model construction process is as follows:

[0014] S101: Get observation status : This includes the total computing resources of the edge server, the total available bandwidth of the edge computing system, the computing resources of each terminal device, network conditions, and tasks to be decided;

[0015] S102: In time slot , the agent observes the environment to obtain the state , and then the Actor network makes the decision , indicating that in the state When the policy function parameters Decide on strategy The output action of the decision includes the task offloading decision , bandwidth resource allocation decision and computing resource allocation decisions ;

[0016] S103: In the time slot , the agent observes the state Make corresponding decisions , and then the environment gives feedback to the agent as a reward , minimize the average delay and energy consumption;

[0017] S104: Representation of Discrete and Continuous Actions: Actions Expressed as ,in is a discrete action with , Indicates time slot No. discrete actions; For continuous action and , Indicates time slot No. a continuous action; Expresses the mapping of Actor network input to output, so the action Expressed as:

[0018]

[0019] Where, Entering state for the Actor network To action Discrete mapping of ; Entering state for the Actor network To action Continuous mapping of ; Entering state for the Actor network To A mapping of output actions.

[0020] Furthermore, the anti-disturbance state is generated by the following steps:

[0021] S201: Discrete action disturbance: Select the When an action is used as the target perturbation action, the perturbation vector is minimized:

[0022]

[0023] in Indicates input status The disturbance, represents the L2 norm, ; An iterative process is used to calculate the true discrete action disturbance, and the disturbance added each time is:

[0024]

[0025]

[0026] in, It is expressed as the number of iterations, is the disturbance coefficient, Indicates in Iterating Function Based on the input state s outputs, represents the gradient; Indicates the The reward in the iteration, Indicates the The state in the iteration, Indicates the The state in the iteration;

[0027] S202: Continuous action disturbance: After adding the disturbance term to the discrete action disturbance, we get:

[0028]

[0029] Where, Indicates the state after disturbance Projected to the state is the center and the radius is within the legal domain; is the generated adversarial perturbation state.

[0030] Furthermore, the attacker selects the following attack method to send the generated adversarial perturbation state to the agent:

[0031] Uniform attack: In each time slot, a random number is generated and compared with the attack frequency. If the random number is less than the attack frequency, no attack is launched. If the random number is greater than or equal to the attack frequency, the attacker adds adversarial perturbations to the agent's observations.

[0032] Strategic timing attack: When the action preference value of the agent at a certain time step is greater than the specified threshold, the attack is launched. The preference value is calculated by the following preference function:

[0033]

[0034] in Indicates that the parameter is The state-action value function of Indicates status The preference function of Assessment in state Next action long-term returns, Assessment in state Execute the action after adding disturbance long-term returns.

[0035] Furthermore, the total quality of service is determined by the delay quality of service and the energy consumption quality of service.

[0036] Furthermore, the step S6 includes the following steps:

[0037] S601: Build a lightweight alternative model based on multi-label classification as follows:

[0038]

[0039] Where F is the substitution model, is the replacement model parameter, Indicates that the input is state , the parameters are Alternative models;

[0040] S602: The alternative model uses a shallow fully connected network architecture. After the input state is normalized and feature extracted, the logical value of each action is output through the linear layer, and the binary cross entropy loss function is used to calculate the multi-label classification loss. By sampling state-action pairs from the original strategy, Train a surrogate model to make the discrete action probability distribution close to the target policy.

[0041] Furthermore, S7 specifically uses fast gradient sign attack, projected gradient descent attack and DeepFool algorithm to generate adversarial perturbation state.

[0042] The present invention makes full use of the adversarial attack method based on deep reinforcement learning, deeply integrates discrete action disturbances and continuous action disturbances in the task offloading and resource allocation scenarios in the edge computing system, constructs an adversarial disturbance attack scheme driven by the dynamic system state, increases the output action loss within the maximum adversarial disturbance range, and reveals the potential vulnerability of the DRL model in the adversarial attack scenario. By flexibly selecting the moment when the adversarial attack is triggered, the concealment and destructive power of the attack are enhanced, and the attack effect of the DRL model is improved. Further combined with the adversarial attack migration method based on the substitution model, effective model attacks can still be achieved in the absence of target model gradient information, thereby improving the generalization ability of adversarial attacks on DRL model attacks.

[0043] The beneficial effects of the present invention are:

[0044] (1) For the first time, a DRL adversarial attack research framework for task offloading and resource allocation was constructed. The final adversarial perturbation strategy was generated by discrete action perturbations and continuous action perturbations, achieving a high attack success rate of the DRL model in adversarial attack scenarios, providing theoretical support for the robust design of edge intelligent systems.

[0045] (2) A method for flexibly selecting the triggering time of adversarial attacks is provided, which enhances the concealment and destructive power of the attacks. The reward value and average service quality are designed to evaluate the effectiveness of adversarial attacks, revealing the potential security risks of the DRL model in adversarial attack scenarios.

[0046] (3) Through the adversarial attack transfer method based on surrogate models, a surrogate model is constructed to approximate the decision-making behavior of the target policy, and adversarial perturbations are generated using surrogate gradients. In the absence of target model gradient information, effective model attacks can still be achieved, improving the generalization ability of adversarial attacks on DRL models in black-box scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 The overall structure of the adversarial attack scheme based on deep reinforcement learning.

[0048] Figure 2 Schematic diagram of the maximum adversarial perturbation constraint and the corresponding reward difference.

[0049] Figure 3 Schematic diagram of the maximum adversarial perturbation constraint and the corresponding service quality difference.

[0050] Figure 4 Schematic diagram of the reward difference and corresponding threshold curve when the maximum adversarial perturbation constraint is 0.05.

[0051] Figure 5 Schematic diagram of service quality difference and corresponding threshold curve when the maximum adversarial perturbation constraint is 0.05. DETAILED DESCRIPTION

[0052] The technical solution of the present invention is described in further detail below, but the protection scope of the present invention is not limited to the following description.

[0053] like Figure 1 As shown, the attacker first intercepts the observation state of the edge computing system (hereinafter referred to as the system) , analyzing the system's operating mode and potential vulnerabilities. The attacker then uses the observation of the system to generate adversarial perturbations based on the deep reinforcement learning discrete-continuous hybrid action adversarial perturbation attack method proposed in this invention. After the adversarial perturbation is generated, the attacker superimposes the original system observation as the new observation state. Sent to the agent, causing the agent to make irrational decisions . In this process, the attacker will combine different methods to flexibly select the timing of adversarial attack triggering, including uniform attack and strategic timing attack methods, and propose a specific preference function to determine the moment of launching the attack, thereby enhancing the concealment and effectiveness of the attack and interfering with the normal DRL model decision-making process. In addition, a black-box attack framework based on the transfer paradigm is adopted to approximate the decision-making behavior of the target strategy by constructing a substitute model, and to generate adversarial perturbations using substitute gradients. In the absence of target model gradient information, an effective attack on the DRL model can still be achieved. The deep reinforcement learning discrete-continuous hybrid action adversarial perturbation attack method comprises the following steps:

[0054] S1. Build a deep reinforcement learning model based on task offloading and resource allocation;

[0055] The continuous time slot task offloading and resource allocation strategy under a single edge server and multiple terminals in the edge computing scenario is studied, with the average system delay and energy consumption as the optimization objectives. The problem is finally modeled as a mixed integer nonlinear programming problem, which is difficult to solve using traditional optimization methods. Therefore, the present invention adopts the idea of deep reinforcement learning algorithm to convert the optimization problem into a Markov decision process for solution. The deep reinforcement learning framework consists of an intelligent agent and an environment, where the environment outside the intelligent agent is collectively referred to as the environment. The deep reinforcement learning algorithm guides the intelligent agent to interact with the environment to learn the optimal strategy. The interaction process mainly involves observation state, action and reward, as described below:

[0056] S101. Observation status: In each time slot , the agent observes the state of the environment, including the total computing resources of the edge server, the total available bandwidth of the edge computing system, the computing resources of each terminal device, the network conditions and the tasks to be decided. It can be expressed as:

[0057]

[0058] in, and They represent the total computing resources of the edge server and the total available bandwidth of the edge computing (EC) system respectively. The terminal set is represented as , the edge server has The task queues serve terminal devices, represented by . Edge server in time slot The number of CPU cycles to be executed on each task queue is expressed as . Indicates that each terminal device is in the time slot The state collection.

[0059] S102. Action: The agent takes action based on the observed state. , make corresponding decisions for the MEC system, including task offloading decisions, bandwidth resource and computing resource allocation decisions. It can be expressed as:

[0060]

[0061] It is assumed that at most three tasks are generated in each time slot, and tasks less than three are filled with 0. ,in Indicates terminal In the time slot Generate offloading decisions for three tasks. Indicates the agent assigned to each terminal The bandwidth resource allocation decision of Indicates the agent assigned to each terminal Computing resource allocation decisions.

[0062] S103. Reward: The agent observes the state Make corresponding decisions , and then the environment gives the agent a reward ,In order to jointly optimize the delay and energy consumption, the average delay and energy consumption are minimized. In addition, the task usually has a certain tolerance time If the task does not complete within the tolerance time Completing the task within 30 seconds may cause certain losses to the terminal device. Therefore, a penalty term is introduced in the reward function. , ensuring that the task is executed. So the reward function It can be expressed as:

[0063]

[0064] Where, is a step function, when hour Take 1, otherwise take 0, The brackets refer to variables. The details are as follows: System time is divided into time slots, the time slot interval is , the set is represented as , in each time slot, the terminal device The number of tasks generated Obey the mean The Poisson distribution of . For terminal devices The number of tasks generated in time slot t, where , is a collection of terminal devices, where ; represents the summation operation of the total quality of service of the tasks generated by all terminal devices in time slot t; It represents the summation operation of the number of tasks generated by all terminal devices in time slot t;

[0065] terminal equipment In the time slot The generated The task offloading decision of a task can be expressed as ,in, Indicates terminal device In the time slot The generated The task is executed locally. Indicates terminal device In the time slot The generated The tasks will be offloaded to the edge server for execution. Indicates terminal device The total local processing delay of the generated nth task is determined by the queuing delay and local computing latency composition, Indicates terminal device The total delay of uploading the nth task generated by the task to the edge server for processing, and the total service quality It is represented by:

[0066]

[0067] Where, is the delay sensitivity coefficient of the task, when When the value is larger, the service quality mainly depends on the delay service quality. ; The energy consumption service quality is calculated as follows:

[0068]

[0069]

[0070] Where, For terminal devices The energy consumed by the local computation of the generated nth task, For terminal devices The energy consumed by the generated n-th task offloading to the edge server.

[0071] S104. To further represent discrete actions and continuous actions, actions Expressed as ,in is a discrete action with , Indicates time slot No. A discrete action. For continuous action and , Indicates time slot No. continuous actions. In each time slot , the agent observes the environment to obtain the state , and then the Actor network makes the decision , indicating that in the state , by the parameters of the policy function Decide on strategy Similarly, for ease of expression, use Expresses the mapping of Actor network input to output, so the action It can also be expressed as:

[0072]

[0073] Where, is the parameter of the strategy function; Entering state for the Actor network To action Discrete mapping of ; Entering state for the Actor network To action Continuous mapping of ; Entering state for the Actor network To A mapping of output actions.

[0074] S2. The attacker obtains the original state observed by the agent in the task offloading system , the discrete-continuous hybrid action anti-disturbance attack method is used to generate adversarial disturbances, and the original system observations are superimposed as the anti-disturbance state ;

[0075] S201. Discrete action perturbation method: For the discrete mapping in S104 In some cases, the output of an action is inconsistent before and after adding adversarial interference, as shown below:

[0076]

[0077] in, Represents a sign function, if the input is greater than 0, the output is 1; if it is less than 0, the output is -1; if it is equal to 0, the output is 0. and , there is at least one discrete action index , making and On both sides of the threshold 0.5. First assume is an affine classifier , that is, we have: , where is the first outputs; is the weight matrix No. List; is the bias vector No. elements, adjust the output offset of the classifier, and the superscript T represents transposition. Consider any , then the action Minimum perturbation vector of change It can be expressed as:

[0078]

[0079] in, Indicates input status For classifiers The minimum perturbation calculated, Indicates the search for disturbance of The solution with the smallest norm. The geometric representation of the perturbation is the point to the hyperplane distance, It can be expressed as: , where for The square of the norm. The lowest perturbation of the model action output change can be expressed as: .Will Extension to more general differentiable classifiers. Minimal perturbation Further expressed as:

[0080]

[0081] After Taylor's first-order expansion of the constraints of the above formula, we can get:

[0082]

[0083]

[0084]

[0085] in, Represents a classifier exist The gradient at is the gradient and the perturbation The dot product of . is the perturbation vector With vector When and Parallel, that is When the perturbation vector has the minimum square norm, so let After substituting, we get:

[0086]

[0087] Where, is the disturbance coefficient, which controls the disturbance size, so we have:

[0088]

[0089] For the entire discrete action space, choose When an action is used as the target perturbation action, the perturbation vector can be minimized:

[0090]

[0091] In addition, since each disturbance vector of the action is obtained by Taylor's first-order expansion, it is an estimate. The present invention uses an iterative process to calculate the true discrete action disturbance, and the disturbance added each time is:

[0092]

[0093]

[0094] in, It is expressed as the number of iterations, Indicates in Iterative Classifier Based on the input state s outputs, represents the gradient; Indicates the The reward in the iteration, Indicates the The state in the iteration, Indicates the The state in iterations.

[0095] S202. Continuous action perturbation method: For the original state , the state obtained by adding a disturbance term to it according to the discrete action perturbation method for:

[0096]

[0097] Where, Indicates the state after disturbance Projected to is the center and the radius is Within the legal domain. , you can also add space Continue searching to maximize the loss function. Here, we combine the projected gradient descent attack algorithm to continue trying to interfere with the continuous actions of the model, so we have:

[0098]

[0099] Where, is the disturbance intensity, Indicates that the parameter is The state-action value function, evaluated at state Next action long-term returns, Indicates status gradient; Indicates projecting the perturbed state in the brackets onto the state is the center and the radius is From the above formula, we can see that after reaching the specified number of iterations, the final adversarial perturbation is obtained.

[0100] S3. After the adversarial perturbation is generated, the attacker selects a suitable attack timing and superimposes the original system observations as a new observation state to be sent to the agent. The present invention adopts the following attack timing selection method:

[0101] S301. Uniform Attack: In each time slot, a random number is generated and compared with the attack frequency. When the random number is less than the attack frequency, no attack is launched. When the random number is greater than or equal to the attack frequency, the attacker adds adversarial perturbations to the agent's observations.

[0102] S302. Strategically Timed Attack: This method uses a preference function to determine whether to initiate an attack at the current time step. When the agent's action preference value at a certain time step exceeds a specified threshold, an attack is initiated. Based on this attack concept, the preference function proposed in this invention is as follows:

[0103]

[0104] In the formula To add the state after adversarial perturbation, Indicates that the parameter is The state-action value function, Indicates status The preference function of Assessment in state Next action long-term returns, Assessment in state Execute the action after adding disturbance long-term returns; therefore, when the preference function The larger the value, the worse the Critic network's evaluation of the action after adding the adversarial perturbation, which means the greatest impact on the system. Set the threshold to , when the preference function value is greater than Launch an attack when.

[0105] Choose the right time to attack and add the state after the counter-disturbance attack Send to the agent.

[0106] S4. Agent acquires state , the Actor network makes the corresponding decision action :

[0107]

[0108] in, is the disturbance state affected by the reinforcement learning adversarial attack, It is a strategy that the Actor network in the model can refer to. It is the action taken by intelligence based on the disturbance state.

[0109] S5. Combined actions affected by adversarial attack , calculate the corresponding reward difference , poor service quality As an indicator for evaluating the effectiveness of attacks;

[0110]

[0111]

[0112] in, To add the reward value after attack; is the reward value when there is no attack; is the average quality of service after adding the attack; is the average service quality when there is no attack.

[0113] S6. To further expand the scope of adversarial attacks, in the absence of target model gradient information, a black-box attack framework based on the transfer paradigm is employed to train alternative models with similar policy distributions.

[0114] S601. From S104, we know that the action Expressed as:

[0115] S602. To reduce the complexity of the surrogate model and study the transferability of adversarial attacks, we decouple the continuous-discrete hybrid action space of the reinforcement learning strategy and consider its discrete action part to construct a lightweight surrogate model based on multi-label classification, as shown below:

[0116]

[0117] Where F is the substitution model, is the replacement model parameter; because Entering state for the Actor network To action discrete mapping, the parameters of the policy function Decide on strategy The output action, therefore Indicates input status , the parameters are Alternative model.

[0118] S603. The alternative model uses a shallow fully connected network architecture. After the input state is normalized and feature extracted, it outputs the logical value of each action through a linear layer. The binary cross entropy loss function (BCE) is used to calculate the multi-label classification loss. The calculation process is as follows:

[0119]

[0120] Where, is the discrete action dimension, For the Tasks in time slots The uninstall decision, The alternative model predicts By sampling state-action pairs from the original strategy Train a surrogate model to make the discrete action probability distribution close to the target policy.

[0121] S7: Generate adversarial state perturbations by training the gradient information of the alternative model, and exploit the migration characteristics of adversarial samples on the decision boundary to attack the original model.

[0122] S701. Generate adversarial perturbation states using Fast Gradient Sign Attack (FGSM), Projected Gradient Descent Attack (PGD) and DeepFool algorithm .

[0123] FGSM is a classic gradient-based adversarial perturbation generation algorithm and is applied to classification models. It achieves the misclassification effect by maximizing the loss function when the input is , where is the label corresponding to the input.

[0124] The PGD algorithm searches for an adversarial perturbation that maximizes the loss function within the input domain.

[0125] The DeepFool algorithm is based on the idea of hyperplane, and its core goal is to find the minimum perturbation that can cause the classification model to produce an incorrect prediction.

[0126] S702. Use uniform attack to add adversarial disturbance to the agent's observation and calculate the corresponding reward difference , poor service quality ,Evaluating the effectiveness of adversarial attack transfer.

[0127] like Figure 2 As shown in the figure, as the maximum perturbation constraint increases, the reward difference of the four attack algorithms all shows a downward trend. The decline of the DeepFool and FGSM algorithms is relatively gentle, while the decline of the PGD algorithm and the discrete-continuous hybrid action adversarial attack algorithm (HDCAP) proposed in this example is larger, especially when the maximum perturbation constraint is greater than 0.06, where the decline increases significantly.

[0128] Figure 3 The performance of the algorithm is evaluated based on the impact of the attack on the system. When the maximum perturbation constraint is small, the service quality difference does not show a significant downward trend as the maximum perturbation constraint increases. When the maximum perturbation constraint is greater than 0.06, the service quality difference begins to decline. The proposed HDCAP algorithm is able to cause greater damage to the system by searching for the optimal perturbation within the perturbation constraint, demonstrating superior aggressiveness to other algorithms.

[0129] like Figure 4 and Figure 5 As shown in the figure, when the maximum perturbation constraint is 0.05, when the threshold is small, both the FGSM algorithm and the DeepFool algorithm can reduce the reward difference and service quality, but the PGD algorithm and the HDCAP algorithm have a greater degree of reduction in the reward and service quality, reaching -4.8 and -0.025. However, when the threshold is greater than 1.0, the FGSM and DeepFool algorithms have a slight impact on the reward and service quality. Therefore, the skewness of the perturbations generated by the FGSM algorithm and the DeepFool algorithm calculated by the preference function is small, that is, these two algorithms cannot search for the perturbation that increases the loss function the most within the perturbation range. However, the PGD algorithm and the proposed HDCAP algorithm can still reduce the system's rewards and degrade system performance when the threshold is 2.

[0130] The foregoing description is a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein and should not be construed as excluding other embodiments. Instead, the present invention can be used in other combinations, modifications, and environments and can be modified within the scope of the concept described herein through the above teachings or techniques or knowledge in the relevant field. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention are intended to be protected by the appended claims.

Claims

1. A discrete-continuous hybrid action counterattack method based on deep reinforcement learning, characterized in that: The following steps are involved: S1: Building a deep reinforcement learning model based on task offloading and resource allocation based on Markov decision process; S2: The attacker obtains the original state observed by the agent in the task offloading system, uses the discrete-continuous hybrid action adversarial perturbation attack method to generate adversarial perturbations, and superimposes the original system observations as the adversarial perturbation state; S3: The attacker sends the generated adversarial perturbation state to the agent; S4: The agent makes a decision based on the acquired adversarial disturbance state through the Actor network; S5: Based on the decision-making actions affected by the adversarial attack, the difference in rewards between actions made with and without the adversarial perturbation and the difference in total service quality are calculated as indicators for evaluating the effectiveness of the attack. S6: Use a black-box attack framework based on the transfer paradigm to train alternative models with similar policy distributions; S7: Generate adversarial state perturbations by training the gradient information of the alternative model, and exploit the migration characteristics of adversarial samples on the decision boundary to attack the original model; The reward is calculated using the following formula: Where H(x) is a step function. When x ≥ 0, H(x) takes 1, otherwise it takes 0. x is the variable in the brackets. is the number of tasks generated by terminal device m in time slot t, where is a collection of terminal devices, where represents the summation operation of the total quality of service of the tasks generated by all terminal devices in time slot t; It represents the summation operation of the number of tasks generated by all terminal devices in time slot t; is the total service quality; the task offloading decision of the nth task generated by terminal device m in time slot t is expressed as in, Indicates that the nth task generated by terminal device m in time slot t is executed locally, It indicates that the nth task generated by terminal device m in time slot t will be offloaded to the edge server for execution; It represents the total local processing delay of the nth task generated by terminal device m, which consists of queuing delay and local computing delay. It represents the total delay of uploading the nth task generated by the terminal device m to the edge server for processing, tol is the tolerance time, r p For penalty items.

2. The discrete-continuous hybrid action counterattack method based on deep reinforcement learning according to claim 1 is characterized in that: The deep reinforcement learning model construction process is as follows: S101: Get observation status s t : This includes the total computing resources of the edge server, the total available bandwidth of the edge computing system, the computing resources of each terminal device, network conditions, and tasks to be decided; S102: At time slot t, the agent observes the environment and obtains state s t , and then the Actor network makes a decision t =π φ (s t ), indicating that in state s t When the policy function parameter φ determines the output action of the policy π, the decision includes the task offloading decision X t , bandwidth resource allocation decision W t and computing resource allocation decision F t ; S103: At time slot t, the agent observes state s t Then make the corresponding decision t , and then the environment gives the agent reward r t , minimize the average delay and energy consumption; S104: Representation of discrete and continuous actions: Action a t Expressed as a t ={D t , C t }, where D t is a discrete action with represents the pth discrete action in time slot t; C t For continuous action and Indicates the qth continuous action in time slot t; f(φ, s t ) expresses the mapping from input to output of the Actor network, so action a t Expressed as: Where, f D (φ, s t ) is the Actor network input state s t To action D t Discrete mapping of f C (φ, s t ) is the Actor network input state s t To action C t Continuous mapping of f i (φ, s t ), i = 1, 2, ..., p + q is the Actor network input state s t A mapping to the i-th output action.

3. The discrete-continuous hybrid action counterattack method based on deep reinforcement learning according to claim 2 is characterized in that: The anti-disturbance state is generated by the following steps: S201: Discrete action perturbation: Select the kth * When an action is used as the target perturbation action, the perturbation vector is minimized: k * =argmin||r k (s)||2; where r k (s) represents the perturbation of the input state s, || ||2 represents the L2 norm, k * ∈[1, p]; an iterative process is used to calculate the true discrete action disturbance, and the disturbance added each time is: s n′+1 =s n′ +r n′+1 ; Where n′ represents the number of iterations, η is the perturbation coefficient, Indicates that the function in the n′th iteration Based on the kth * outputs, represents the gradient; r n′+1 represents the reward in the n′+1th iteration, s n′ represents the state in the n′th iteration, s n′+1 represents the state in the n′+1th iteration; S202: Continuous action disturbance: After adding the disturbance term to the discrete action disturbance, we get: Where, Indicates that the perturbed state s t +r n′+1 Projected to state s t The legal area with the center and radius ε; s′ t is the generated adversarial perturbation state.

4. The discrete-continuous hybrid action counterattack method based on deep reinforcement learning according to claim 3 is characterized in that: The attacker selects the following attack methods to send the generated adversarial perturbation state to the agent: Uniform attack: In each time slot, a random number is generated and compared with the attack frequency. When the random number is less than the attack frequency, the attacker does not launch an attack. When the random number is greater than or equal to the attack frequency, the attacker adds adversarial perturbations to the observation of the agent; Strategic timing attack: When the action preference value of the agent at a certain time step is greater than the specified threshold, the attack is launched. The preference value is calculated using the following preference function: c(s t )=Q θ (s t ,π φ (s t ))-Q θ (s t ,π φ (s′ t )); where Q θ represents the state-action value function with parameter θ; c(s t ) represents state s t The preference function, Q θ (s t ,π φ (s t ))Evaluate in state s t Next, perform action π φ (s t )’s long-term return, Q θ (s t ,π φ (s′ t ))Evaluate in state s t Execute the action π after adding the perturbation φ (s′ t ) long-term returns.

5. The discrete-continuous hybrid action counterattack method based on deep reinforcement learning according to claim 4 is characterized in that: The total quality of service is determined by the delay quality of service and the energy consumption quality of service.

6. The discrete-continuous hybrid action counterattack method based on deep reinforcement learning according to claim 5, characterized in that: The S6 comprises the following steps: S601: Build a lightweight alternative model based on multi-label classification as follows: F(φ′,s t )=f D (φ,s t ); Among them, F is the substitution model, φ′ is the substitution model parameter, F(φ′, s t ) indicates that the input is state s t , a surrogate model with parameter φ′; S602: The alternative model uses a shallow fully connected network architecture. After the input state is normalized and feature extracted, the logical value of each action is output through the linear layer, and the binary cross entropy loss function is used to calculate the multi-label classification loss. By sampling state-action pairs {(s t , a t )} Train a surrogate model to make the discrete action probability distribution close to the target policy.

7. The discrete-continuous hybrid action counterattack method based on deep reinforcement learning according to claim 6, characterized in that: In the S7, the fast gradient sign attack, the projected gradient descent attack and the DeepFool algorithm are specifically used to generate the adversarial perturbation state.

8. The discrete-continuous hybrid action counterattack method based on deep reinforcement learning according to claim 7 is characterized in that: The overall quality of service Calculated by the following formula: Where, is the delay sensitivity coefficient of the task; For latency quality of service, is the energy consumption service quality, which is calculated as follows: Where, The local computation latency of the nth task generated for terminal device m; The energy consumed by the local computation of the nth task generated for terminal device m, The energy consumed by offloading the nth task generated for terminal device m to the edge server.

9. The discrete-continuous hybrid action counterattack method based on deep reinforcement learning according to claim 8, characterized in that: The countermeasure disturbance state s′ generated in S202 t Maximize the loss function and combine it with the projected gradient descent attack algorithm to continue to interfere with the model's continuous actions. Where v is the perturbation intensity, sign(·) represents the sign function. If the input is greater than 0, the output is 1, and if it is less than 0, the output is -1. represents the state-action value function with parameter θ1, Evaluate at state s′ t Next, perform action π φ (s′ t )’s long-term returns, Represents the anti-disturbance state s′ t The gradient of , after reaching the specified number of iterations, the final anti-perturbation state is obtained; Indicates projecting the perturbed state in the brackets onto the state s t The legal area with the center as ε and the radius as ε.

10. The discrete-continuous hybrid action counterattack method based on deep reinforcement learning according to claim 9, characterized in that: The perturbation coefficient is calculated by the following formula: where f k (s) is the function f k The k-th output based on the input state s, represents f k The gradient of (s), is the square of the L2 norm.

Images