A quadrotor unmanned aerial vehicle attack detection method based on reinforcement learning

Abstract

This invention discloses a reinforcement learning-based attack detection method for quadcopter drones, comprising: estimating the state values ​​of the drone during flight using Kalman filtering; obtaining the corresponding residual correlation terms; dividing the entire detection space into several detection intervals; setting reward values ​​for two actions: continuing detection and detecting an attack; setting different simulated attack times; training Q-tables for different simulated attack times using the Saras algorithm; fusing the trained Q-tables for different simulated attack times using a weighted average; fitting the weighted average fused Q-table using a neural network; and detecting attacks on the drone online using the Q-table fitted by the neural network. This invention utilizes reinforcement learning technology and integrates the nonlinear approximation capability of neural networks to achieve the detection of subtle and covert attacks on quadcopter drones. It not only improves the recognition ability of subtle and covert attacks but also enables online detection of new attack patterns and reduces attack detection latency.

Description

Technical Field

[0001] This invention belongs to the field of aircraft safety technology and relates to drone attack detection technology that integrates neural networks and reinforcement learning algorithms. Specifically, it relates to a method for detecting attacks on quadrotor drones based on reinforcement learning. Background Technology

[0002] Unmanned aerial vehicles (UAVs), by removing the manned function from manned aircraft, offer advantages such as simplified systems, no need for life support systems, and no limitations imposed by the pilot's physical condition, making them more suitable for high-risk missions. While UAVs are widely used in the military field due to their higher cost-effectiveness compared to manned aircraft, the changing battlefield environment and the ever-evolving attack methods of adversaries make it increasingly difficult for UAVs to safely and reliably execute missions on the battlefield.

[0003] Existing anomaly detection methods are unable to detect small-scale, unknown, and covert attacks in a timely manner, leading to frequent loss of control of drones after an attack, affecting their safe and reliable operation and mission completion rates. Therefore, the problem of attack detection for drones under small-scale, unknown, and covert attack scenarios urgently needs to be solved. Currently, the most widely used attack detection method is summation, but the following problems still hinder its application: how to obtain the drone model after an attack (traditional methods involve trial and error to determine the attack type, which is time-consuming and highly dependent on the experience of the detection algorithm designer); how to obtain a better detector (past methods focused on minimizing detection latency caused by false positives, but maximum detection latency represents a worst-case scenario); and how to apply the obtained detector to future unknown situations (past methods involved repeating previous detector calculations with subsequent data, which has not yielded ideal results). Summary of the Invention

[0004] Purpose of the invention: In order to overcome the shortcomings of existing technologies in detecting small-scale unknown attacks and covert attacks, this invention provides a reinforcement learning-based attack detection method for quadcopter drones. This method integrates neural networks and reinforcement learning algorithms for detection, improves the attack detection method, reduces detection latency, and enhances the drone's ability to detect unknown small-scale attacks and covert attacks.

[0005] Technical Solution: To achieve the above objectives, this invention provides a method for detecting attacks on quadrotor drones based on reinforcement learning, comprising the following steps:

[0006] S1: Estimate the state values ​​of the UAV during flight based on Kalman filtering;

[0007] S2: Obtain the corresponding residual related terms from the state values;

[0008] S3: Divide the entire detection space into several detection intervals based on the set thresholds and the residual correlation terms corresponding to each time point;

[0009] S4: Set the reward values ​​for two actions: continue detection and attack detection. Set different simulated attack times, use the values ​​corresponding to the detection intervals as input data, and use the Saras algorithm to train the Q-table for different simulated attack times.

[0010] S5: The trained Q-tables at different simulated attack times are fused using a weighted average.

[0011] S6: Use a neural network to fit a weighted average fused Q-table to extend the detection time to future unknown situations;

[0012] S7: Online detection of attacks on drones using a Q-table fitted by a neural network.

[0013] Further, step S1 specifically includes:

[0014] The state-space expression for the UAV is:

[0015] x k =Ax k-1 +Bu k-1 +ω k-1

[0016] y k =Cx k +v k ,

[0017] Where, x k =[xyz φ θ ψ uvwpqr] T A is the state transition matrix, B is the control input matrix, C is the observation matrix, and u k To control the input, x k For state variables, y k For the output quantity, ω k-1 v k The process noise and measurement noise are respectively, and their covariances are F. k-1 ,T k ;

[0018] The quadcopter drone uses LQR control, and the corresponding LQR cost function is:

[0019]

[0020] The control input is obtained by minimizing the cost function.

[0021] u k=-(R+B) T PB) -1 B T PAx k

[0022] Where Q and R are the state penalty matrix and control penalty matrix of the LQR controller, respectively, and P is the solution to the following algebraic Riccati equation: A T PA-P+QA T PB(R+B T PB) -1 B T PA = 0;

[0023] The Kalman filter for the UAV described by the above formula is constructed as follows:

[0024]

[0025]

[0026]

[0027] in, These are the Kalman filter estimates of the state and output variables, respectively. This represents the state estimate predicted in k-1 steps, with the Kalman filter gain being:

[0028] L k =AP k C T (F k +CP k C T ) -1

[0029]

[0030] Where P k Let be the error covariance after the k-th step estimate is updated.

[0031] Furthermore, the expression for the residual correlation term in step S2 is as follows:

[0032]

[0033] Further, step S3 specifically includes:

[0034] Taking into account the accuracy, false alarm rate, and false negative rate of detection, a trial-and-error method is used to determine the detection space threshold based on residual correlation terms and test thresholds. The detection space is then designed with multiple intervals, represented as 0 = ζ0 < ζ1 < ζ2 < ... < ζ S =∞, where the residual correlation term ζ i <zk <ζ i-1 The detection space at time i∈1,...S,k is represented as ι i The entire detection space is represented as ι1...ι s The detection space at each time step is assigned a different value according to the quantification standard, which is determined by the chi-square test threshold and the residual correlation term.

[0035] When a drone is subjected to an unknown attack, the residual data of the Kalman filter may be similar under normal and attacked conditions. To compensate for the insufficient representation of the drone's situation by the data, a sliding window is used to represent the detection space at the current moment with information at a finite number of moments. The length of the sliding window is denoted as M, and the entire detection space can be divided into S. M A sliding window, the sliding window at time k is composed of the quantized {z} j :k-M+1≤j≤k} represents.

[0036] Further, step S4 specifically includes:

[0037] Design the reward value R(Reward) for various detection scenarios, and the reward value R for continuing detection. continue The report value R for stopping detection stop The design simulates attack times μ, and the selection of these times needs to consider representative attack scenarios across the entire detection space. For example, the initial, middle, and final phases of the detection space. The values ​​ι1...ι corresponding to several detection scenarios are then assigned. s As input samples, the Saras algorithm is used to train the Q-table, during which the simulated attack time μ is changed to obtain different Q-tables.

[0038] The design of the reward value is based on the principle that the reward value for a correct action is lower than the reward value for an incorrect action under the current detection conditions, thereby ensuring the accuracy of the detection process.

[0039] Further, step S5 specifically includes:

[0040] The Q-tables for different scenarios are combined using a weighted average. The weights of the Q-tables for each scenario are assigned based on the actual situation of the drone. If a certain time period is vulnerable to attack, the weight of the Q-table corresponding to this simulated attack time period will be greater than that of the Q-tables for other time periods.

[0041] By utilizing Q-tables that integrate multiple attack scenarios, covert or minor attacks can be detected. The weights of each Q-table are adjusted based on the principle of minimizing false positive and false negative rates.

[0042] Furthermore, in step S6, the neural network input is the value ι1...ι corresponding to the detection space. s We select the learning rate α, the number of neural network layers, the fitting cutoff condition κ, the activation function, and the maximum number of iterations to fit the Q-table.

[0043] This invention addresses the insensitivity of UAVs to unknown, small-scale, and covert attacks during flight. Compared to conventional attack detection methods, this invention provides more timely detection of these attacks. Conventional attack detection methods obtain a post-attack UAV model by trial-and-error, a time-consuming approach heavily reliant on the experience of the algorithm designer. Past methods have focused on minimizing detection latency caused by false positives to improve detector capabilities. However, maximum detection latency represents a worst-case scenario, leaving traditional algorithms with room for improvement in this area. This invention, based on partially observable Markov decision theory, uses the Q-table obtained from the Saras algorithm in attack detection. It selects representative attacks within the detection space to train the Q-table and combines this with the fitting ability of neural networks to nonlinear tables to obtain a nonlinear mapping model for subsequent detection. Compared to cumulative sum tests and chi-square tests, this method, which integrates reinforcement learning and neural networks, demonstrates stronger detection capabilities for unknown, small-scale, and covert attacks, helping to reduce the impact of attack signals on UAV flight performance.

[0044] Beneficial effects: Compared with existing technologies, this invention utilizes reinforcement learning technology and integrates the nonlinear approximation capability of neural networks to achieve the detection of small and covert attacks by quadcopter drones. It solves the problem of drones' insensitivity to unknown small-amplitude and covert attacks during flight. Compared with existing attack detection methods based on chi-square tests, it not only improves the identification capability of small and covert attacks, but also detects new attack patterns online. Moreover, the detection latency of chi-square tests is determined by state estimation and residual generation, while the method of this invention uses Q-table to directly obtain detection results, effectively reducing the attack detection latency. Attached Figure Description

[0045] Figure 1 This is a flowchart of the detection method of the present invention;

[0046] Figure 2 This diagram illustrates a reinforcement learning-based quadcopter drone attack detection framework.

[0047] Figure 3 This diagram illustrates a portion of the observable Markov decision process (POMDP) ​​in the detection method of this invention.

[0048] Figure 4 A graph showing the false negative rate of the detection method of the present invention;

[0049] Figure 5 A graph showing the false alarm rate of the detection method of the present invention;

[0050] Figure 6 A graph showing the accuracy data of the detection method of the present invention;

[0051] Figure 7 A graph showing the chi-square test results against covert attacks;

[0052] Figure 8 The diagram shows the detection results of the detection method of the present invention under covert attacks;

[0053] Figure 9 This represents the Q-table return value graph in the detection method of the present invention;

[0054] Figure 10 The graph shows the convergence of the neural network fitting function of the detection method of the present invention. Detailed Implementation

[0055] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0056] This invention provides a reinforcement learning-based method for detecting attacks on quadrotor drones, such as... Figure 1 and Figure 2 As shown, it includes the following steps:

[0057] S1: Estimate the state values ​​of the UAV during flight based on Kalman filtering;

[0058] S2: Obtain the corresponding residual related terms from the state values;

[0059] S3: Divide the entire detection space into several detection intervals based on the set thresholds and the residual correlation terms corresponding to each time point;

[0060] S4: Set the reward values ​​for two actions: continue detection and attack detection. Set different simulated attack times, use the values ​​corresponding to the detection intervals as input data, and use the Saras algorithm to train the Q-table for different simulated attack times.

[0061] S5: The trained Q-tables at different simulated attack times are fused using a weighted average.

[0062] S6: Use a neural network to fit a weighted average fused Q-table to extend the detection time to future unknown situations;

[0063] S7: Online detection of attacks on drones using a Q-table fitted by a neural network.

[0064] To verify the detection effect of the method of the present invention, the above scheme is applied in this embodiment as follows:

[0065] This embodiment applies the above method to a quadcopter drone, providing a reinforcement learning-based quadcopter drone attack detection method. The specific steps are as follows:

[0066] Step 1: Estimate the state value of the UAV under flight conditions based on Kalman filtering, and further obtain the corresponding residual correlation terms from the state estimate.

[0067] In this embodiment, the mass of the quadcopter drone is m = 1.4 kg, and the moment of inertia of the quadcopter drone is I. x =0.0211 kg·m 2 I y =0.0219 kg·m 2 I z =0.0366 kg·m 2 .

[0068] The state-space expression for the UAV is:

[0069] x k =Ax k-1 +Bu k-1 +ω k-1

[0070] y k =Cx k +v k ,

[0071] Where, x k =[xyz φ θ ψ uvwpqr] T A is the state transition matrix, B is the control input matrix, C is the observation matrix, and u k To control the input, x k For state variables, y k For the output quantity, ω k-1 v k The process noise and measurement noise are respectively, and their covariances are F. k-1 ,T k ;

[0072] The quadcopter drone uses LQR control, and the corresponding LQR cost function is:

[0073]

[0074] The control input is obtained by minimizing the cost function.

[0075] u k =-(R+B) T PB) -1 B T PAx k

[0076] Where Q and R are the state penalty matrix and control penalty matrix of the LQR controller, respectively, and P is the solution to the following algebraic Riccati equation: A T PA-P+QA T PB(R+B T PB) -1 B T PA = 0;

[0077] The Kalman filter for the UAV described by the above formula is constructed as follows:

[0078]

[0079]

[0080]

[0081] in, These are the Kalman filter estimates of the state and output variables, respectively. This represents the state estimate predicted in k-1 steps, with the Kalman filter gain being:

[0082] L k =AP k C T (F k +CP k C T ) -1

[0083]

[0084] Where P k Let be the error covariance after the k-th step estimate is updated.

[0085] The expression for the residual correlation term is as follows:

[0086]

[0087] Step 2: Divide the entire detection space into several detection intervals based on the set thresholds and the corresponding residual correlation terms at each time point.

[0088] Taking into account the accuracy, false alarm rate, and false negative rate of detection, a trial-and-error method is used to determine the detection space threshold based on residual correlation terms and test thresholds. The detection space is then designed with multiple intervals, such as 0 = ζ0 < ζ1 < ζ2 < ... < ζ. S =∞. Where the residual correlation term ζ i <z k <ζ i-1 The detection space at times i∈1,...S,k can be represented as ι i Therefore, the entire detection space can be represented as ι1...ι s The detection space at each time step is assigned a different value based on the quantization standard. The quantization standard is determined by the chi-square test threshold and the residual correlation term.

[0089] In this embodiment, the detection space is divided into 5 segments based on the chi-square test 12-dimensional threshold of 20.38: ζ0 = 0, ζ1 = 3.84, ζ2 = 11.07, ζ3 = 15, and ζ4 = 20.38. The selection of these segments needs to balance detection capability with the curse of dimensionality faced by subsequent learning algorithms. Under the influence of unknown attacks on the UAV, the state estimation data from the Kalman filter may be similar under normal and attacked conditions. To compensate for the insufficient representation of the UAV's situation by the data, a sliding window is used to represent the detection space at the current moment with information from a finite number of moments. The sliding window length is M = 5, and the entire detection space can be divided into S segments. M =5 5 A sliding window. The sliding window at time k consists of the quantized {z} j The expression :k-10+1≤j≤k represents the detection space ι1...ι. s Cutting via a sliding window can be represented as ο1...ο s .

[0090] Step 3: Set the reward values ​​for the two actions of continuing detection and detecting attacks. Set different simulated attack times, use the values ​​corresponding to the detection intervals as input data, and use the Saras algorithm to train the Q-table for different simulated attack times.

[0091] First, design the reward value R(Reward) for various detection scenarios, and then design the reward value R for continued detection. continue Stop detecting the reported value R stop Then, the simulated attack time μ is designed. The selection of the simulated attack time needs to consider representative attack scenarios across the entire detection space. For example: the initial stage, middle stage, and final stage of the detection space. The divided detection space is assigned values ​​ι1...ι. s As input samples, the Saras algorithm is used to train the Q-table, during which the simulated attack time μ is changed to obtain different Q-tables.

[0092] The design of the reward value is based on the principle that the reward value for a correct action is lower than the reward value for an incorrect action under the current detection conditions, thereby ensuring the accuracy of the detection process.

[0093] Saras' algorithm updates the Q-table using ε-greedy. (See reference...) Figure 3 First, the ε-greedy strategy is used to select the action `act` at the current detection time from the pool of candidate actions `act{continue,stop}`. In the ε-greedy strategy, ε = 0.05, and ρ is a random number. Whether to continue detection in the ε-greedy strategy is determined by the following conditions:

[0094]

[0095] randn{·} means randomly selecting an action from the set.

[0096] The Saras algorithm's Q-table update process is as follows:

[0097]

[0098] The pseudocode for updating Q-table is as follows:

[0099] Q(o,act)←Q(o,act)+β(R+γQ(o',act')-Q(o,act))

[0100] o←o',act←act'

[0101] Where Q(o,act) represents the Q-table at the current time step, and Q(o',act') represents the Q-table at the next time step. R represents the reward value obtained at the current time step. β and γ are the discount factors when updating the Q-table, respectively. The values ​​of R are as follows:

[0102]

[0103]

[0104] Then, the simulated attack time μ is changed. The simulated attack time is selected from the beginning, middle, and end phases of the entire detection space, and the above Q-table training process is repeated sequentially to obtain a representative Q-table for each detection time period. Q(o,act) b This represents the Q-table, Q(o,act), obtained at the beginning of the detection space during the simulated attack. m This represents the Q-table, Q(o,act), obtained during the simulated attack phase in the middle of the detection space. eThis indicates that the Q-table was obtained at the end of the detection space during the simulated attack.

[0105] Step 4: Obtain the Q-tables under different conditions in Step 3, and merge the Q-tables under various conditions by weighted average.

[0106] The weights of the Q-tables under different scenarios are determined based on the actual situation of the drone. If the drone is vulnerable to attack during a certain time period, the weight of the Q-table corresponding to that simulated attack time period will be greater than the weights of the Q-tables for other time periods. The detection space for various simulated attack times is detected using a Q-table that integrates detection information from multiple scenarios. The weights of each Q-table are adjusted according to the principle of minimizing false positive and false negative rates. The weighted average fusion of the Q-tables is as follows:

[0107] Q(o,act)←v1Q(o,act) b +v2Q(o,act) m +v3Q(o,act) e

[0108] v1 + v2 + v3 = 1

[0109] Setting v1 ≥ 0.8 can avoid an excessively high false alarm rate. The values ​​of v2 and v3 are determined based on the false alarm and missed alarm rates. However, the relative magnitudes of v2 and v3 have little impact on detection. In actual selection, the above weighting method can be appropriately adopted.

[0110] Step 5: Select the Q-table obtained by neural network fitting.

[0111] In this embodiment, the neural network input is the detection space ι1...ι s The corresponding quantization value, a suitable learning rate α = 0.4, and the number of neural network layers are selected. Fitting cutoff condition κ = 10 -5 The activation function is ReLU, and the maximum number of iterations is T = 2000.

[0112]

[0113]

[0114] in, For a 6-layer neural network, d k For neural network parameters, i n b is the data input for the nth layer of the neural network. k These are the hyperparameters of the neural network. This represents the Q-function fitted based on the quantized values ​​corresponding to the detection space. Relu(·) is the activation function of a neural network (the Relu(·) function outputs the same value as the input when the input is positive, and outputs zero when the input is negative).

[0115] Step 6: Obtain the neural network model Detect attacks on drones online.

[0116] In this embodiment, the following is obtained: Figures 4 to 10 The test data is as follows:

[0117] Figure 4 This represents the false negative rate of this detection method, which is the frequency at which the detection method fails to issue an alert after an attack occurs. It is calculated by dividing the number of delayed alerts by the total number of detections. Since the chi-square test cannot detect covert attacks, its false negative rate is 100%, while the false negative rate of this detection method is superior.

[0118] Figure 5 This represents the false alarm rate of the detection method, i.e., the frequency of alarms triggered before an attack occurs. It is calculated by dividing the number of premature alarms by the total number of detections. Since the chi-square test cannot detect covert attacks, its false alarm rate is 0, while the false alarm rate of this detection method is superior to that of the chi-square test.

[0119] Figure 6 This indicates the accuracy of the detection method, which is the frequency at which an alarm is triggered immediately after an attack occurs. It is calculated by dividing the number of immediate alarms by the total number of detections. Since the chi-square test cannot detect covert attacks, its accuracy is 0, while the accuracy of this detection method is superior.

[0120] Figure 7 The chi-square test results for covert attacks indicate that the chi-square test cannot detect covert attacks, and the chi-square value remains below the warning threshold of 20.38 when the attack occurs. Figure 8 This indicates that the reinforcement learning-based quadcopter drone attack detection method can detect covert attacks. In the graph, steps 0-45 show no attack events occurring, with an action value `act=1` indicating continued detection. Steps 45-600 show attack events occurring, with almost all action values ​​`act=2` indicating an attack signal detected, prompting an alert and termination of detection.

[0121] Figure 9 This indicates that during Q-table training, the false negative reward value continuously decreases, while the false positive value increases appropriately, all within a reasonable range.

[0122] Figure 10 This indicates that after 2000 training steps, the neural network value function tends to converge, suggesting that the fitting effect is good at this point.

Claims

1. A method for detecting attacks on quadrotor drones based on reinforcement learning, characterized in that, Includes the following steps: S1: Estimate the state values ​​of the UAV during flight based on Kalman filtering; S2: Obtain the corresponding residual related terms from the state values; S3: Divide the entire detection space into several detection intervals based on the set thresholds and the residual correlation terms corresponding to each time point; S4: Set the reward values ​​for two actions: continue detection and attack detection. Set different simulated attack times, use the values ​​corresponding to the detection intervals as input data, and use the Saras algorithm to train the Q-table for different simulated attack times. S5: The trained Q-tables at different simulated attack times are fused using a weighted average. S6: Fit the weighted average fused Q-table using a neural network; S7: Online detection of attacks on drones using a Q-table fitted by a neural network.

2. The method for detecting attacks on quadrotor drones based on reinforcement learning according to claim 1, characterized in that, Step S1 specifically involves: The state-space expression for the UAV is: x k = Ax k-1 + Bu k-1 + ω k-1 y k = Cx k + v k , where x k = [x y z φ θ ψ u v w p q r] T , A is a state transition matrix, B is a control input matrix, C is an observation matrix, u k is a control input, x k is a state quantity, y k is an output quantity, ω k-1 and v k are process noise and measurement noise respectively, and the respective covariances are F k-1 and T k ; The Kalman filter for the UAV described by the above formula is constructed as follows: in, These are the Kalman filter estimates of the state and output variables, respectively. This represents the state estimate predicted in k-1 steps, with the Kalman filter gain being: L k = AP k C T (F k + CP k C T ) -1 where P k is the updated error covariance for the kth step estimate.

3. The method for detecting attacks on quadrotor drones based on reinforcement learning according to claim 2, characterized in that, The expression for the residual correlation term in step S2 is as follows:

4. The method for detecting attacks on quadrotor drones based on reinforcement learning according to claim 1, characterized in that, Step S3 specifically involves: The method comprises the following steps: considering the accuracy, false positive rate and false negative rate of detection, determining the detection space threshold according to the residual correlation term and the test threshold through trial and error, dividing the detection space into multiple intervals according to the detection space threshold, and representing as 0 = ζ0 < ζ1 < ζ2 < ··· < ζ S = ∞, wherein the residual correlation term ζ i < z k < ζ i-1 , i ∈ 1,...S, the detection space at the k moment is represented as ι i , and the total detection space is represented as ι1...ι s , wherein the detection space at each moment is assigned different values according to the quantization standard, and the quantization standard is determined by the chi-square test threshold and the residual correlation term.

5. The method for detecting attacks on quadrotor drones based on reinforcement learning according to claim 4, characterized in that, The detection space in the step S3 is represented by a sliding window with limited time information to represent the detection space at the current time. The length of the sliding window is represented as M. The entire detection space can be divided into S M sliding windows, and the sliding window at the kth time is represented by the quantized {z j :k-M+1≤j≤k}.

6. The method for detecting attacks on quadrotor drones based on reinforcement learning according to claim 4, characterized in that, Step S4 specifically involves: The reward value R for various detection scenarios is designed sequentially, and the reward value R for continued detection is further designed. continue The report value R for stopping detection stop Design a simulation attack time μ, and assign values ​​ι1...ι corresponding to several detection scenarios. s As input samples, the Saras algorithm is used to train the Q-table, during which the simulated attack time μ is changed to obtain different Q-tables.

7. The method for detecting attacks on quadrotor drones based on reinforcement learning according to claim 1, characterized in that, Step S5 specifically involves: By utilizing Q-tables that integrate multiple attack scenarios, covert or minor attacks can be detected. The weights of each Q-table are adjusted based on the principle of minimizing false positive and false negative rates.

8. The method for detecting attacks on quadrotor drones based on reinforcement learning according to claim 1, characterized in that, The neural network input in the step S6 is the value ι1...ι corresponding to the detection space s Select learning rate α, number of neural network layers, fitting stop condition κ, activation function, maximum number of iterations, and fit the Q-table.

9. The method for detecting attacks on quadrotor drones based on reinforcement learning according to claim 2, characterized in that, The control input u in the step S1 is represented as: k is represented as: The quadcopter drone uses LQR control, and the corresponding LQR cost function is: The control input is obtained by minimizing the cost function. u k = -(R + B T PB) -1 B T PAx k Where Q and R are the state penalty matrix and control penalty matrix of the LQR controller, respectively, and P is the solution to the following algebraic Riccati equation: A T PA-P+QA T PB(R+B T PB) -1 B T PA = 0.

10. The method for detecting attacks on quadrotor drones based on reinforcement learning according to claim 1, characterized in that, In step S4, the Saras algorithm updates the Q-table using ε-greedy, as follows: The ε-greedy strategy selects the action `act` at the current detection time from the pool of candidate actions `act{continue,stop}`. In the ε-greedy strategy, ε = 0.05, and ρ is a random number. Whether to continue detection in the ε-greedy strategy is determined by the following conditions: randn{·} means randomly selecting an action from the set; The Saras algorithm's Q-table update process is as follows: The pseudocode for updating Q-table is as follows: Q(o,act)←Q(o,act)+β(R+γQ(o',act')-Q(o,act)) o←o',act←act' Where Q(o,act) represents the Q-table at the current time, Q(o',act') represents the Q-table at the next time, R represents the reward value obtained at the current time, and β and γ are the discount factors when the Q-table is updated; The possible values ​​for R are as follows: k < μ is not attacked k≥μ is attacked By changing the simulated attack time μ, and selecting the start, middle, and end phases of the entire detection space, the training process of the Q-table described above is repeated sequentially to obtain a representative Q-table, Q(o,act), for each detection time period. b This represents the Q-table, Q(o,act), obtained at the beginning of the detection space during the simulated attack. m This represents the Q-table, Q(o,act), obtained during the simulated attack phase in the middle of the detection space. e This indicates that the Q-table was obtained at the end of the detection space during the simulated attack.

Images