Deep Reinforcement Learning System and Method Based on Multi-UAV Path Planning
By constructing a deep reinforcement learning system with adversarial attack and dual defense framework, the security problem of UAV path planning model is solved, and effective defense against state and reward interference is achieved, thereby improving the robustness and security of UAV path planning.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU DIANZI UNIV
- Filing Date
- 2023-06-27
- Publication Date
- 2026-07-17
AI Technical Summary
Existing deep reinforcement learning algorithms face security issues in drone control. Attackers use various methods to attack states, actions, and rewards, resulting in insufficient robustness of the models.
A deep reinforcement learning system based on multi-UAV path planning is constructed, adopting an adversarial attack and dual defense framework, including state and reward defense mechanisms. The system detects outliers and attack methods through neural networks, recovers the perturbed data sequence, and improves the robustness of the model.
Effective defense status and reward-based interference improve the robustness of drone path planning, reduce the frequency and impact of successful attacks, and decrease development costs.
Smart Images
Figure CN116820126B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of UAV flight control and deep reinforcement learning attack and defense, specifically involving a deep reinforcement learning system and method based on multi-UAV path planning. Background Technology
[0002] Currently, with the development of artificial intelligence and the improvement of human technology, reinforcement learning algorithms are increasingly being applied in the field of drone control. Reinforcement learning algorithms can help drones plan path information in unknown environments, greatly reducing the difficulty of drone decision-making and improving the efficiency of drone task path planning to a certain extent. However, with the widespread application of deep reinforcement learning algorithms in real-world scenarios, their security issues have gradually emerged. Attackers exploit the agent's state, actions, environment, and rewards during the training and testing phases of deep reinforcement learning models. In the testing phase, attack methods include adding noise to the drone's state information, modifying actions, and using adversarial agents to mislead the target agent. In the training phase, attack methods encompass reward flipping attacks, Trojan poisoning attacks, path vulnerability attacks, and action space perturbations. These attack methods expose deep reinforcement learning models to serious security risks; therefore, addressing the security issues of reinforcement learning algorithms has become crucial. Summary of the Invention
[0003] To address the shortcomings of existing technologies and improve the robustness of UAV path planning, this invention adopts the following technical solution:
[0004] A deep reinforcement learning method based on multi-UAV path planning includes the following steps:
[0005] Step S1: Collect environmental data of the UAV during flight in real time as status information;
[0006] Step S2: During the training phase, adversarial attacks are used to disrupt the Markov decision sequence received by the UAV. t A t ,R t ,S t+1 The Markov decision sequence includes state information, action, reward, and state information for the next moment.
[0007] Step S3: Construct a dual defense framework of state and reward, construct a mechanism to determine whether an attack has occurred and the attack method, and construct a state restoration and reward recovery mechanism based on the state and reward under attack and the state and reward in the Markov decision sequence to defend against attack disturbances.
[0008] Step S4: Input the perturbed Markov sequence into the trained defense framework to restore the data. Then, put the restored data sequence into the experience pool for the UAV to make subsequent action decisions. The optimal flight path of the UAV is obtained through multiple rounds of model training.
[0009] Furthermore, the construction of the defense framework in step S3 includes the following steps:
[0010] Step S3.1: Obtain the Markov decision sequence;
[0011] Step S3.2: During training, an outlier loss is constructed by combining the attack identifier at the current time with the current attack prediction value based on the neural network, and an outlier detection is used to determine whether an attack has occurred.
[0012] Step S3.3: During training, an attack method discrimination loss is constructed by combining the discrimination value of the attack method at the current time with the predicted value of the current attack method based on the neural network, which is used to discriminate the attack method;
[0013] Step S3.4: During training, if the attack method is a state attack, a state restoration loss is constructed based on the attacked state information and the state information of the next moment in the Markov decision sequence, which is used to obtain the restored state information; if the attack method is a reward attack, a reward prediction loss is constructed based on the attacked reward and the reward in the Markov decision sequence, which is used to obtain the restored reward.
[0014] Furthermore, in step S3.2, the attack identification function S at the current moment... a ,S a ={0,1}, when S a =1 indicates that the intelligent agent drone is currently under attack and disturbance by electromagnetic wave signals; conversely, when S = 1, it indicates that the drone is currently under attack and disturbance by electromagnetic wave signals. a =0 indicates no attack or disturbance; the agent will use a neural network to predict the current attack function prediction value S. a_p In the attack prediction function S a_p The outlier loss function Loss1 used in the calculation process is expressed as:
[0015]
[0016] In step S3.3, the discrimination value A of the attack method at the current moment is obtained in real time. s A s ∈{-1, 0, 1}, when A s When A = 1, it indicates that the system is currently under a reward attack. s When A = 0, it indicates that the system is currently under a state attack. sWhen the value is -1, it indicates that no attack has occurred at the current time. The neural network predicts the attack method at the current time and obtains the attack method prediction function A. s_p A s_p The attack method discrimination loss function Loss2 generated during the calculation process is expressed as:
[0017]
[0018] In step S3.4, A is predicted based on the attack method. s_p Determine the type of attack currently being suffered. If it is a state attack, obtain the perturbation state sequence S′. t+1 The input will be fed into the state restoration network for training and prediction. The restored state can be represented as S″. t+1 .
[0019] The state recovery loss function Loss3 used in the state prediction process is expressed as:
[0020]
[0021] Where N represents the dimension of the input state;
[0022] If the current attack is a reward attack, then the reward R′ after the perturbation is... t The input will be fed into the reward restoration network model for training, and the predicted restoration reward can be represented as R″. t .
[0023] The reward prediction loss function Loss4 used in the reward prediction process is expressed as follows:
[0024]
[0025] Furthermore, the adversarial attack in step S2 can interfere with the state features and reward information of the deep reinforcement learning model, thereby causing the agent drone to learn the disturbed state information and thus affecting the agent's decision-making ability and performance. This includes the following steps:
[0026] Step S2.1: Attack method selection, determine the type of attack the drone is currently suffering, including state interference and reward interference;
[0027] Step S2.2: Disturbance timing selection, used to calculate the advantage of the current state of the UAV and select the attack timing; based on the Markov sequence, the advantage state is judged for the state at the next moment and the reward obtained, and the advantage degree of the current state of the UAV is obtained through neural network training;
[0028] Step S2.3: State disturbance, using electromagnetic interference to affect the UAV's judgment of target position information;
[0029] Step S2.4: Reward attack, which uses the method of modifying the reward value to carry out the attack.
[0030] Furthermore, in step S2.1, attack method A m The size and the set probability threshold P s Related, when A m When A = 0, it means that the probability of choosing a state attack at the current moment is higher than the probability of a reward attack. In this case, the output result is to choose the state interference method to attack; when A m When A = 1, it means that the probability of choosing a reward attack is higher than the probability of a state attack, so the output is to attack using a reward perturbation method; A m The specific expression is:
[0031]
[0032] Among them, P s ∈[1,100], P r ∈[1,100], P s P represents the probability threshold of the attack method selection module. r This indicates the probability of selecting a bonus attack as the attack method;
[0033] In step S2.2, the dominance function is:
[0034]
[0035] Among them, when S v When r = 1, it indicates that a better action has been learned than before; that is, after the drone performs the current action, it will move closer to the target and receive a positive reward r. t Therefore, compared to random disturbances, adding an attack disturbance at the current moment will have a greater impact on the agent; when S v When E = -1, it indicates that the action being taken is not optimal, therefore no attack or disturbance is initiated at the current moment. s ∈[0,1] indicates the degree of dominance of the current state;
[0036] In step S2.3, when the dominant function S v =1 and attack method A m When = 0, an incorrect state S′ will be obtained by adding a signal perturbation. t+1 The states affected by the disturbance include the degree of approach t. r And the distance d between the drone and the target a The degree of convergence t′ after interference r Represented as:
[0037]
[0038] Where d amin d represents the distance at which a drone can establish effective communication with a target. amax This represents the maximum distance between the drone and the target, and the state S′ at the next moment after being attacked. t+1 The value function Q(S,A) that will affect the next action to be performed is expressed as:
[0039] Q(S t+1 A t+1 )=Q(S′ t+1 A t+1 )
[0040] Where S t+1 Indicates the state of the drone at the next moment when it is not under attack, A t+1 This indicates the action the drone will perform in the next moment.
[0041] In step S2.4, when the dominant function S v =1 and attack method A m When = 1, choose to perform a reward attack at the current time, and the perturbation reward R′ after being attacked. t Represented as:
[0042] R′ t =μ t *R t
[0043] Among them, R t Represents the reward value, μ t This represents the reward perturbation function.
[0044] Furthermore, a virtual environment model of a multi-UAV path planning scenario is built to collect environmental data of the UAVs during flight in real time as status information. A virtual simulation environment is used to replace the real task scenario. The scenario model selected for the virtual environment model is a task model for multiple UAVs to assist communication. This model includes the size of the area to be executed, the obstacle model design, and the ground user communication model. The construction of the virtual model sets the execution target of the flight mission for the UAVs and provides a training ground for the UAVs' path planning.
[0045] Furthermore, a deep reinforcement learning algorithm model is designed to plan the flight path of the UAV. During flight, the UAV will collect environmental data in real time, including: the UAV's own position information, the distance between the UAV and obstacles, the distance between the UAV and the target point, and the received signal strength. This environmental data is input as state information into the deep reinforcement learning algorithm network for subsequent analysis and calculation.
[0046] A deep reinforcement learning system based on multi-UAV path planning includes a state information acquisition module, an attack model module, a defense model module, and a model deployment module.
[0047] The status information acquisition module collects environmental data of the UAV in real time during flight as status information;
[0048] During the training phase, the attack model module uses adversarial attacks to interfere with the Markov decision sequences received by the drone. t A t ,R t ,S t+1 The Markov decision sequence includes state information, action, reward, and state information for the next moment.
[0049] The defense model module constructs a dual defense framework of state and reward, establishes a mechanism for determining whether an attack has occurred and the attack method, and constructs a state recovery and reward restoration mechanism based on the state and reward of the attacked state and reward, and the state and reward in the Markov decision sequence, to defend against attack disturbances.
[0050] The model deployment module inputs the perturbed Markov sequence into the trained defense framework for data recovery, and then puts the recovered data sequence into the experience pool for the UAV to make subsequent action decisions. The optimal flight path of the UAV is obtained through multiple rounds of model training.
[0051] Furthermore, the defense model module includes a sequence input module, an outlier detection module, an attack method discriminator, a data recovery module, and a predicted value output module;
[0052] The sequence input module acquires the Markov decision sequence;
[0053] During the training process, the outlier detection module constructs an outlier loss by combining the current attack identifier with the current attack prediction value based on the neural network, and detects whether an attack has occurred based on the outlier.
[0054] During the training process, the attack method discriminator constructs an attack method discrimination loss by combining the discrimination value of the attack method at the current moment with the predicted value of the current attack method based on the neural network, which is used to discriminate the attack method.
[0055] During the training process, if the attack method is a state attack, the data recovery module constructs a state restoration loss based on the attacked state information and the state information of the next moment in the Markov decision sequence to obtain the restored state information; if the attack method is a reward attack, the module constructs a reward prediction loss based on the attacked reward and the reward in the Markov decision sequence to obtain the restored reward.
[0056] The predicted value output module outputs the restored state information and the restored reward.
[0057] Furthermore, the attack model module includes an attack method selection module, a perturbation timing selector, a state perturbation module, and a reward attack module;
[0058] The attack method selection module is used to determine the type of attack the drone is currently suffering. The attack types include state interference and reward interference.
[0059] The disturbance timing selector is used to calculate the advantage of the current state of the UAV and select the attack timing; based on the Markov sequence, it judges the advantage state of the next moment and the reward obtained, and obtains the advantage degree of the current state of the UAV through neural network training.
[0060] The state disturbance module affects the UAV's judgment of target location information by means of electromagnetic interference.
[0061] The reward attack module uses the method of modifying the reward value to launch an attack.
[0062] The advantages and beneficial effects of this invention are as follows:
[0063] The deep reinforcement learning system and method based on multi-UAV path planning of this invention constructs an advantageous action detection module, which indicates the best time for attackers to perturb, thereby reducing the number of perturbations by attackers and improving the efficiency of attacks. This invention proposes a dual defense framework that can simultaneously cope with two attack methods: state perturbation and reward interference, which greatly enhances the robustness of deep reinforcement learning algorithms. This invention uses a self-built virtual simulation platform to complete the flight training of UAVs, effectively reducing development costs. Attached Figure Description
[0064] Figure 1 This is a flowchart of the method in an embodiment of the present invention.
[0065] Figure 2 This is a flowchart of the anti-attack scheme in an embodiment of the present invention.
[0066] Figure 3 This is a flowchart of the defense framework design in an embodiment of the present invention.
[0067] Figure 4 This is a schematic diagram illustrating the system operation in an embodiment of the present invention. Detailed Implementation
[0068] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.
[0069] like Figure 1 As shown, the deep reinforcement learning method based on multi-UAV path planning includes the following steps:
[0070] Step S1: Collect environmental data of the UAV during flight in real time as status information;
[0071] Step S2: During the training phase, adversarial attacks are used to disrupt the Markov decision sequence received by the UAV. t A t ,R t ,S t+1 The Markov decision sequence includes state information, action, reward, and state information for the next moment.
[0072] Step S3: Construct a dual defense framework of state and reward, construct a mechanism to determine whether an attack has occurred and the attack method, and construct a state restoration and reward recovery mechanism based on the state and reward under attack and the state and reward in the Markov decision sequence to defend against attack disturbances.
[0073] Step S4: Input the perturbed Markov sequence into the trained defense framework to restore the data. Then, put the restored data sequence into the experience pool for the UAV to make subsequent action decisions. The optimal flight path of the UAV is obtained through multiple rounds of model training.
[0074] In this embodiment, building a virtual environment model for multi-UAV path planning scenarios refers to using a constructed virtual simulation environment to replace the real task scenario. The selected scenario model for the virtual environment model is a task model for multiple UAVs to assist communication. This model mainly consists of three aspects: the size of the area where the task is executed, the obstacle model design, and the ground user communication model. The construction of the virtual model not only sets the execution objectives of the flight mission for the UAVs, but also provides a training ground for the UAVs' path planning.
[0075] In this embodiment, training an intelligent agent drone using a deep reinforcement learning algorithm in a scene model refers to planning the drone's flight path by designing a deep reinforcement learning algorithm model. During flight, the drone collects environmental data in real time, including its own position, distances to obstacles, distances to the target point, and received signal strength. This environmental data is input as state information into the deep reinforcement learning algorithm network for subsequent analysis and computation.
[0076] In this embodiment, inputting the state information collected during the drone's flight into a deep reinforcement learning neural network for model training refers to inputting the state information collected during the drone's flight into the deep reinforcement learning neural network for model training. The input state information S t Will be with action A t Reward R t and the state information S at the next moment t+1 Composition of Markov decision sequences t A t ,R t ,S t+1 The intelligent agent drone will complete the subsequent model training based on the input Markov sequence.
[0077] In this embodiment, using adversarial attack algorithms to interfere with the Markov decision sequences received by the agent during training refers to altering the data information received by the agent through electromagnetic signal perturbation. This attack scheme mainly consists of four parts: attack method selection, perturbation timing selection, state perturbation, and reward attack. It can attack and interfere with the state characteristics and reward information of the deep reinforcement learning model, thereby causing the agent drone to learn the perturbed state information, thus affecting the agent's decision-making ability and performance.
[0078] In this embodiment, the attack method selection in the attack algorithm refers to the fact that the entire attack algorithm consists of two attack methods: state interference and reward perturbation. The choice of attack method determines the type of attack suffered by the agent drone at the current moment. The adversarial attack scheme process is as follows: Figure 2 As shown, attack method A m The size and the set probability threshold P s Related. When A m When A = 0, it means that the probability of choosing a state attack at the current moment is higher than the probability of a reward attack. In this case, the output result is to choose the state interference method to attack; when A m When A = 1, it means that the probability of choosing a reward attack is higher than the probability of a state attack, so the output is to attack using a reward perturbation method; A m The specific expression is:
[0079]
[0080] Among them, P s ∈[1,100], P r ∈[1,100], P s P represents the probability threshold of the attack method selection module. r This indicates the probability of choosing a bonus attack as the attack method.
[0081] In this embodiment, the selection of the perturbation timing in the attack algorithm refers to the process of calculating the dominance of the current agent's state and selecting the attack timing. The selector will select the Markov sequence obtained by the agent from the simulation environment. t A t ,R t ,S t+1 The input is fed into the algorithm model. To reduce the frequency of attacks on the agent, the perturbation timing selector will adjust the state S at the next time step. t+1 ,S t+1 =(p o ,p c ,d c ,d a ,t r c) and the reward r t Perform advantage state assessment and obtain the advantage degree E of the current agent state through neural network training. s ∈[0,1], dominance function S v It can be represented as
[0082]
[0083] When S v When r = 1, it means the agent has learned a better action than before; that is, after the drone performs the current action, it will approach the trapped person and receive a positive reward r. t Therefore, compared to random disturbances, introducing an attack disturbance at the current moment will have a greater impact on the agent. When S v When the value is -1, it indicates that the Agent is not performing the optimal action, so no attack or disturbance will be initiated at the current moment.
[0084] In this embodiment, the state perturbation in the attack algorithm refers to the state perturbation part of the attack algorithm, which mainly uses electromagnetic interference to affect the UAV's judgment of the location information of stranded personnel on the ground. When the advantage function S v =1 and attack method A m When = 0, the Agent will be made to obtain an incorrect state S′ by adding a signal perturbation. t+1 The states affected by the disturbance include the degree of approach t. r And the distance between the drone and the trapped people. a The degree of convergence t′ after interference r It can be represented as
[0085]
[0086] Where d amin The distance at which a drone can establish effective communication with a person trapped on the ground is generally set to 5 meters. amax This represents the maximum distance between the drone and the disaster victims on the ground within the scenario model. The state S′ at the next moment after the attack. t+1 The value function Q(S,A) that will affect the agent's next action can be expressed as:
[0087] Q(S t+1 A t+1 )=Q(S′ t+1 A t+1 )
[0088] Where S t+1 Indicates the state of the drone at the next moment when it is not under attack, A t+1 This indicates the action the drone will perform in the next moment.
[0089] In this embodiment, the reward attack in the attack algorithm refers to the reward attack part of the attack algorithm, which mainly involves modifying the reward value R. t The attack is carried out in a manner that utilizes the advantage function S. v =1 and attack method A m When = 1, choose to perform a reward attack at the current time. The perturbation reward R′ after being attacked. t It can be represented as
[0090] R′ t =μ t *R t
[0091] Where μ t This represents the reward perturbation function.
[0092] In this embodiment, constructing a dual state and reward defense framework to defend against attack perturbations refers to building a dual state and reward defense framework capable of simultaneously handling both S and R attack methods to improve the robustness of the reinforcement learning algorithm. The entire defense framework mainly consists of five parts: MDP sequence input, outlier detection, attack method discrimination, data recovery, and predicted value output. It can help the intelligent agent drone defend against various attacks, including electromagnetic interference and reward perturbations. The defense framework design process is as follows: Figure 3 As shown.
[0093] In this embodiment, outlier detection in the defense framework refers to the attack identification function S obtained by the agent during the training of the algorithm model at the current moment. a ,S a ={0,1}, when S a =1 indicates that the intelligent agent drone is currently under attack and disturbance by electromagnetic wave signals; conversely, when S = 1, it indicates that the drone is currently under attack and disturbance by electromagnetic wave signals. a A value of 0 indicates that the system has not been disturbed by an attacker. The agent uses a neural network to predict the value S of the current attack function. a_p In the attack prediction function S a_p The outlier loss function Loss1 used in the calculation process can be expressed as:
[0094]
[0095] In this embodiment, the attack method discrimination in the defense framework refers to the fact that the agent obtains the discrimination value A of the attack method at the current moment in real time during the training process. s A s ∈{-1, 0, 1}, when A s When A = 1, it indicates that the system is currently under a reward attack. s When A = 0, it indicates that the system is currently under a state attack. s When the value is -1, it indicates that no attack has occurred at the current time. The neural network predicts the attack method at the current time and obtains the attack method prediction function A. s_p A s_p The attack method discrimination loss function Loss2 generated during the calculation process can be expressed as:
[0096]
[0097] In this embodiment, data recovery within the defense framework refers to restoring disturbed data information. Based on A predicted by the data recovery module... s_p The magnitude of the value determines the type of attack currently being suffered. If the current attack is a state attack, then the obtained perturbation state sequence S′ is used to determine the type of attack. t+1 The input will be fed into the state restoration network for training and prediction. The restored state can be represented as S″. t+1 .
[0098] The state restoration loss function Loss3 used in the state prediction process is expressed as follows:
[0099]
[0100] Where N represents the dimension of the input state;
[0101] If the current attack is a reward attack, then the reward R′ after the perturbation is... t The input will be fed into the reward restoration network model for training, and the predicted restoration reward can be represented as R″. t .
[0102] The reward prediction loss function Loss4 used in the reward prediction process is expressed as follows:
[0103]
[0104] In this embodiment, inputting the perturbed Markov sequence into the defense framework for data recovery refers to inputting the perturbed Markov sequence... t A t ,R′ t ,S′ t+1 The data is input into the defense framework for data recovery. The perturbed sequence first undergoes outlier detection, then enters the attack mode identification module to predict the attack mode affecting the agent. Finally, the defense framework recovers the data and outputs the restored Markov sequence based on the perturbation mode. t A t ,R″ t ,S″ t+1 >
[0105] In this embodiment, placing the restored data sequence into an experience pool for the UAV to make subsequent action decisions, and obtaining the optimal flight path of the UAV through multiple rounds of model training, refers to the restored Markov sequence. t A t ,R″ t ,S″ t+1 The information will be input into the experience pool for the intelligent agent drone to complete subsequent training tasks. The intelligent agent drone will learn from the decision information in the restored experience pool to complete the next action A. t+1 The process involves execution, followed by 800 rounds of training in a virtual model to ultimately determine the optimal flight path for the drone.
[0106] In this embodiment, testing the defensive effectiveness of the trained reinforcement learning model through adversarial attack experiments refers to subjecting the trained reinforcement learning model to these experiments. The entire testing process is set to 100 rounds. During the test, the AI uses the trained model to complete the output and execution of subsequent actions, while the drone is continuously subjected to interference from the attack algorithm during flight. After the test, the model's defensive effectiveness is evaluated based on the mission path planned by the drone.
[0107] In this embodiment, deploying the tested algorithm model to the UAV edge computing platform means that the tested algorithm model will be deployed to the UAV edge computing platform for real-time decision output. The edge computing platform uses Huawei Ascend Atlas 200DK, and utilizes peripherals such as millimeter-wave radar, gyroscope, and GPS to collect environmental information in real time and input it into the reinforcement learning model for subsequent action output, ultimately realizing information collection and action decision-making in real-world scenarios.
[0108] In this embodiment, the planning of the mission path for placing the deployed drone in a real-world scenario refers to the path planning for the drone, after model deployment, to complete its flight mission in a real-world environment. The drone will collect environmental state information in the real-world scenario in real time, and then use the deployed algorithm model to make decisions on subsequent flight actions. The drone will obtain the optimal action at the current moment based on the real-time analysis results of the algorithm model, and repeat this process until the optimal mission path is planned.
[0109] like Figure 4 As shown, the deep reinforcement learning system based on multi-UAV path planning includes a state information acquisition module, an attack model module, a defense model module, and a model deployment module.
[0110] The status information acquisition module collects environmental data of the UAV in real time during flight as status information.
[0111] During the training phase of the attack model module, an adversarial attack is used to interfere with the Markov decision sequence received by the drone. t A t ,R t ,S t+1 The Markov decision sequence includes state information, actions, rewards, and the state information at the next time step; the attack model module includes:
[0112] The attack method selection module is used to determine the type of attack the drone is currently experiencing. Attack types include state interference and reward interference.
[0113] The perturbation timing selector is used to calculate the advantage of the current state of the drone and select the attack timing; based on the Markov sequence, it judges the advantage state of the next moment and the reward obtained, and obtains the advantage degree of the current state of the drone through neural network training.
[0114] The status disturbance module uses electromagnetic interference to affect the UAV's judgment of target location information;
[0115] The reward attack module uses the method of modifying the reward value to launch an attack.
[0116] The defense model module constructs a dual defense framework based on state and reward, establishes a mechanism to determine whether an attack has occurred and the attack method, and, based on the attacked state and reward, and the states and rewards in the Markov decision sequence, constructs a state recovery and reward restoration mechanism to defend against attack disturbances. Specifically, the defense model module includes:
[0117] The sequence input module obtains the Markov decision sequence;
[0118] The outlier detection module, during training, constructs an outlier loss by combining the current attack identifier with the current attack prediction value based on the neural network, and detects whether an attack has occurred based on outliers.
[0119] During training, the attack method discriminator constructs an attack method discrimination loss by comparing the discrimination value of the attack method at the current time with the predicted value of the current attack method based on the neural network, which is used to discriminate the attack method.
[0120] In the data recovery module, during training, if the attack method is a state attack, a state recovery loss is constructed based on the attacked state information and the state information of the next moment in the Markov decision sequence to obtain the recovered state information; if the attack method is a reward attack, a reward prediction loss is constructed based on the attacked reward and the reward in the Markov decision sequence to obtain the recovered reward.
[0121] The predicted value output module outputs the restored state information and the restored reward.
[0122] The model deployment module inputs the perturbed Markov sequence into the trained defense framework for data recovery. The recovered data sequence is then placed into the experience pool for the UAV to make subsequent action decisions. The optimal flight path of the UAV is obtained through multiple rounds of model training.
[0123] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A deep reinforcement learning method based on multi-UAV path planning, characterized in that... Includes the following steps: Step S1: Collect environmental data of the UAV during flight as status information; Step S2: During the training phase, adversarial attacks are used to interfere with the Markov decision sequence received by the drone. The Markov decision sequence includes state information, action, reward, and state information for the next moment. Step S3: Construct a dual defense framework based on state and reward, including a mechanism to determine whether an attack has occurred and the attack method. Based on the attacked state and reward, and the states and rewards in the Markov decision sequence, construct a state recovery and reward restoration mechanism to defend against attack disturbances. The construction of the defense framework includes the following steps: Step S3.1: Obtain the Markov decision sequence; Step S3.2: Construct outlier loss by combining the attack identifier at the current moment with the current attack prediction value based on the neural network, and detect whether an attack has occurred based on outliers; Step S3.3: Construct an attack method discrimination loss by combining the discrimination value of the attack method at the current moment with the predicted value of the current attack method based on the neural network, which is used to discriminate the attack method; Step S3.4: If the attack method is a state attack, then based on the attacked state information and the state information of the next moment in the Markov decision sequence, construct a state restoration loss to obtain the restored state information; if the attack method is a reward attack, then based on the attacked reward and the reward in the Markov decision sequence, construct a reward prediction loss to obtain the restored reward. Step S4: Input the perturbed Markov decision sequence into the trained defense framework to restore the data. Then, put the restored data sequence into the experience pool for the UAV to make subsequent action decisions. The optimal flight path of the UAV is obtained through multiple rounds of model training.
2. The deep reinforcement learning method based on multi-UAV path planning according to claim 1, characterized in that: In step S3.2, the attack identifier at the current moment ,when When it indicates that the drone is currently under attack or disturbance, conversely, when This indicates that the system has not been attacked or disturbed; the current attack prediction value is predicted using a neural network. In the attack prediction function The outlier loss function used in the calculation process Represented as: In step S3.3, the discrimination value of the attack method at the current moment is obtained. , ,when When, it indicates that the target is currently under a reward attack. When, it indicates that a state attack is currently in effect. At that moment, it indicates that no attack has occurred; the neural network predicts the attack type at the current moment and obtains the attack type prediction. , Attack discrimination loss function generated during the calculation process Represented as: In step S3.4, the attack method is predicted. Determine the type of attack currently being attacked. If it is a state attack, obtain the perturbation state sequence. The data will be input into the state restoration network for training and prediction. Indicates the restored state; The state restoration loss function used in the state prediction process Represented as: Where N represents the dimension of the input state, This represents the nth-dimensional state of the drone at the next moment when it is not under attack; If the current attack is a reward-based attack, then the reward after the disturbance will be... This will be fed into the reward restoration network model for training. This represents the predicted recovery reward; The reward prediction loss function used in the reward prediction process Represented as: in, This represents the reward value.
3. The deep reinforcement learning method based on multi-UAV path planning according to claim 1, characterized in that: The adversarial attack in step S2 includes the following steps: Step S2.1: Attack method selection, determine the type of attack the drone is currently suffering, including state interference and reward interference; Step S2.2: Disturbance timing selection, used to calculate the advantage of the current state of the UAV and select the attack timing; based on the Markov decision sequence, the advantage state is judged for the state at the next moment and the reward obtained, and the advantage of the current state of the UAV is obtained through neural network training; Step S2.3: State disturbance, using electromagnetic interference to affect the UAV's judgment of target position information; Step S2.4: Reward attack, which uses the method of modifying the reward value to carry out the attack.
4. The deep reinforcement learning method based on multi-UAV path planning according to claim 3, characterized in that: In step S2.1, the attack method The size and the set probability threshold Related, when When the probability of choosing a state-based attack is higher than the probability of choosing a reward-based attack, the output result is that the attack will be carried out by choosing state interference. When the probability of choosing a reward attack is higher than that of a state attack, the output result is to use the reward perturbation method to attack. The specific expression is: in, This represents the probability threshold for the attack method selection module. This indicates the probability of selecting a bonus attack as the attack method; In step S2.2, the advantage function based on the degree of advantage is: Among them, when When this occurs, it indicates that the drone has learned a better action than before; that is, after performing the current action, the drone will move closer to the target and receive a positive reward. If the value is 0, it indicates that the optimal action was not taken, therefore no attack or disturbance will be initiated at that moment. Indicates the degree of advantage of the current situation; In step S2.3, when the dominant function And attack method At this point, an incorrect state will be obtained by adding signal perturbation. The states affected by the disturbance include the degree of convergence. and the distance between the drone and the target The degree of convergence after interference Represented as: in This indicates the distance at which a drone can establish effective communication with a target. This indicates the maximum distance between the drone and the target, and the state at the next moment after being attacked. This will affect the value function of the next action to be performed. , is represented as: in This indicates the drone's state at the next moment when it is not under attack. Indicates the action the drone will perform in the next moment; In step S2.4, when the dominant function And attack method When choosing to launch a reward attack at the current moment, the perturbation reward after being attacked is... Represented as: in, Indicates the reward value. This represents the reward perturbation function.
5. The deep reinforcement learning method based on multi-UAV path planning according to claim 1, characterized in that: A virtual environment model for multi-UAV path planning scenarios is constructed to collect environmental data of UAVs during flight as state information. A virtual simulation environment is used to replace the real task scenario. The scenario model selected for the virtual environment model is a task model for multi-UAV auxiliary communication, which includes the size of the area where the task is performed, obstacle model design, and ground user communication model.
6. The deep reinforcement learning method based on multi-UAV path planning according to claim 1, characterized in that: The flight path planning of the drone is accomplished by designing a deep reinforcement learning algorithm model; During flight, the drone collects environmental data in real time, including: the drone's own position, the distance between the drone and obstacles, the distance between the drone and the target point, and the received signal strength. This environmental data is used as status information.
7. A deep reinforcement learning system based on multi-UAV path planning, comprising a state information acquisition module, an attack model module, a defense model module, and a model deployment module, characterized in that: The status information acquisition module collects environmental data of the UAV during flight as status information; The attack model module uses adversarial attacks to interfere with the Markov decision sequence received by the UAV. The Markov decision sequence includes state information, action, reward and state information at the next moment. The defense model module constructs a dual defense framework of state and reward, establishes a mechanism for determining whether an attack has occurred and the attack method, and constructs a state recovery and reward restoration mechanism based on the attacked state and reward, and the state and reward in the Markov decision sequence, to defend against attack disturbances. The defense model module includes a sequence input module, an outlier detection module, an attack method discriminator, a data recovery module, and a predicted value output module. The sequence input module acquires the Markov decision sequence; The outlier detection module constructs an outlier loss by combining the current attack identifier with the current attack prediction value based on the neural network, and detects whether an attack has occurred based on the outlier. The attack method discriminator constructs an attack method discrimination loss by combining the discrimination value of the attack method at the current moment with the predicted value of the current attack method based on the neural network, and is used to discriminate the attack method. If the attack method is a state attack, the data recovery module constructs a state restoration loss based on the attacked state information and the state information of the next moment in the Markov decision sequence, which is used to obtain the restored state information; if the attack method is a reward attack, the module constructs a reward prediction loss based on the attacked reward and the reward in the Markov decision sequence, which is used to obtain the restored reward. The predicted value output module outputs the restored state information and the recovered reward. The model deployment module inputs the perturbed Markov decision sequence into the trained defense framework for data recovery, and then puts the recovered data sequence into the experience pool for the UAV to complete subsequent action decisions. The optimal flight path of the UAV is obtained through multiple rounds of model training.
8. The deep reinforcement learning system based on multi-UAV path planning according to claim 7, characterized in that: The attack model module includes an attack method selection module, a perturbation timing selector, a state perturbation module, and a reward attack module. The attack method selection module is used to determine the type of attack the drone is currently suffering. The attack types include state interference and reward interference. The disturbance timing selector is used to calculate the degree of advantage of the current state of the UAV and select the attack timing; based on the Markov decision sequence, it judges the advantageous state of the next moment and the reward obtained, and obtains the degree of advantage of the current state of the UAV through neural network training. The state disturbance module affects the UAV's judgment of target location information by means of electromagnetic interference. The reward attack module uses the method of modifying the reward value to launch an attack.