Unmanned aerial vehicle network intrusion detection method based on reinforcement learning

Abstract

The invention belongs to the technical field of modern communication network security, and provides an unmanned aerial vehicle network intrusion detection method based on reinforcement learning. According to the invention, the system observation is constructed according to the identity information and position information of the unmanned aerial vehicle obtained through the ADS-B, and the time interval and the number of data packet transmission obtained through analysis of the base station transceiver module in combination with the detection performance; a system state is formed after system observation is combined with previous detection performance and serves as input of the layered reinforcement learning model. The base station selects an optimal classifier and optimizes corresponding parameters of the classifier according to output of the hierarchical reinforcement learning model, detects whether malicious attacks exist in a large amount of traffic existing in the unmanned aerial vehicle system, accurately identifies attack types of the traffic, reduces detection time delay and energy consumption while improving detection precision, and improves detection efficiency. Therefore, the communication security of the unmanned aerial vehicle in the unmanned aerial vehicle system is improved, the detection strategy of the unmanned aerial vehicle network is optimized, and the abnormal traffic information of the unmanned aerial vehicle and the attack type initiated by the unmanned aerial vehicle are detected and identified in time.

Description

Technical Field

[0001] The present invention relates to the security of modern communication networks and proposes a drone network intrusion detection method based on reinforcement learning, belonging to the technical field of modern communication network security. Background Art

[0002] There are many drones in the drone network performing various services or computing tasks, which will generate a large amount of drone traffic information. However, there may be potential malicious attacks in this traffic. Attackers will launch various malicious behaviors such as interference attacks, denial of service attacks, impersonation attacks and deception attacks in the drone network, affecting the normal communication between drones and base stations and between drones.

[0003] Currently, drone intrusion detection methods have achieved some success. For example, Chinese invention patent application publication number CN113556191A proposes a drone intrusion detection method based on channel state information; Chinese invention patent application publication number CN118469156A provides a novel intelligent drone intrusion expulsion scheduling system and method. Furthermore, some drone intrusion detection methods using deep learning have also been widely used in recent years. For example, Chinese invention patent application publication number CN118353655A provides a drone intrusion detection system based on GNN and LSTM; and Chinese invention patent application publication number CN118762295A provides a high-efficiency embedded drone intrusion detection system based on FPGA. However, these methods have some limitations in detection accuracy and have relatively high detection latency and energy consumption. This can prevent legitimate drones from evading malicious attacks in a timely manner, resulting in reduced service performance and, in severe cases, mission failure or drone damage.

[0004] Reinforcement learning, also known as reinforcement learning or enhanced learning, is one of the paradigms and methodologies of machine learning. It is used to describe and solve the problem of how intelligent agents learn strategies to maximize rewards or achieve specific goals during their interactions with the environment. It has been applied in various fields. Summary of the Invention

[0005] In view of the above technical problems in the prior art, the purpose of the present invention is to provide a drone network intrusion detection method based on reinforcement learning. This detection method uses reinforcement learning to realize intrusion detection of malicious attacks on drone networks and identification of attack types, reduce detection delay and energy consumption, optimize detection accuracy and thus improve drone network security.

[0006] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is as follows:

[0007] A method for detecting drone network intrusion based on reinforcement learning, the method comprising the following steps:

[0008] Step 1: Obtain the target UAV network, which contains M UAVs and N base stations, where M and N are integers greater than or equal to 1;

[0009] Step 2: Construct a reinforcement learning model for the jth base station among N base stations, where 1≤j≤N; the reinforcement learning model is divided into a high-level and a low-level layer, each of which contains an actor network, two critic networks, and two target networks; the weight parameters of the high-level actor network, the two critic networks, and the two target networks are initialized as θ (0) , η (0) , δ (0) as well as The weight parameters of the low-level Actor network, the two Critic networks, and the two target networks are μ (0) , as well as And the initialization learning rate is α, the discount factor is γ and the entropy coefficient is λ;

[0010] Step 3: Base station j constructs the state vector s for time slot k (k) , where k is an integer greater than or equal to 1;

[0011] In step 3, construct the state vector s (k) The specific process is as follows: At time slot k, base station j obtains the identity information of M drones by broadcasting the information received from the ADS-B receiver {G i} 1≤i≤M , and location information {d i (k)} 1≤i≤M In T time slots, the base station's transceiver module marks the arrival timestamp of each received data packet, and calculates the data packet transmission interval {τ i (k)} 1≤i≤M , and obtain the number of transmitted data packets according to the number of data packets received in the T time slot {n i (k)} 1≤i≤M , where 1≤i≤M. In addition, the F1 score of base station j for intrusion detection in the previous time slot is f (k-1) , with an accuracy of p (k-1) , the recall rate is r (k-1) , the false alarm rate is l (k-1) and the false negative rate is m (k-1)Based on the above drone traffic and detection performance information, the state vector s constructed by base station j is (k) as follows:

[0012] s (k) =[{G (k)} 1≤i≤M ,{d i (k)} 1≤i≤M ,{τ i (k)} 1≤i≤M ,{n i (k)} 1≤i≤M ,f (k-1) ,p (k-1) ,r (k-1) ,l (k-1) ,m (k-1) ](0.1).

[0013] Step 4: Base station j calculates the state vector s (k) Select C intrusion detection classifiers from A1={1,2,...,C}, where C is an integer greater than or equal to 1;

[0014] In step 4, a1 (k) Represents the classifier type selected by the base station for the state of M drones. Base station j converts the state vector s (k) Input into the high-level Actor network and output high-level actions, that is, classifier type a1 (k) The two Critic networks are based on a1 (k) Output the long-term expected benefit (i.e. Q value) Q of the classifier strategy in the current state H (s',a1';δ (k) )and Update y of target Q value (k) As shown below:

[0015]

[0016] where π(a1'|s';θ (k) ) represents the action probability sampled by the policy network in state s', and the base station selects the classifier strategy a1 according to the high-level Q value (k) .

[0017] Step 5: Base station j calculates the state vector s (k) And the classifier strategy a1 obtained in step 4 (k) Jointly optimize classifier parameter a2 (k) ;

[0018] In step 5, a2 (k) Represents the selected a1(k) The corresponding optimizable parameters in the classifier. Base station j converts the state vector s (k) and high-level action a1 (k) The two are input into the low-level Actor network and the parameter strategy a2 that can be optimized by the classifier is output. (k) , in and They are classifier strategies a1 (k) The minimum and maximum values ​​of the parameters can be optimized. Similar to the calculation method of the high-level target Q value, the low-level target Q value The calculation is:

[0019]

[0020] in Represents the action probability sampled by the policy network in state s'. The base station selects the classifier parameter optimization strategy a2 based on the low-level Q value (k) .

[0021] Step 6: Base station j calculates the benefit;

[0022] In step 6, according to the action x (k) =[a1 (k) ,a2 (k) ]Calculate the F1 score f of the current time slot through the true positive, true negative, false positive and false negative of intrusion detection (k) , false negative rate m (k) , false alarm rate l (k) , precision p (k) , accuracy q (k) , recall rate r (k) , and measure the delay t consumed by the detection (k) Based on the above indicators, the benefits can be obtained according to the following formula, where w1 to w6 represent the weight parameters of F1 score, precision, accuracy, recall rate, false alarm rate, missed alarm rate, and detection delay, respectively:

[0023] u (k) =f (k) +w1p (k) +w2q (k) +w3r (k) -w4l (k) -w5m (k) -w6t (k) (0.4).

[0024] Step 7: Base station j evaluates the short-term risk value of each state-action pair and makes benefit corrections;

[0025] In step 7, construct the indicator function, according to the F1 score f (k), false negative rate m (k) , false alarm rate l (k) , precision p (k) , accuracy q (k) , recall rate r (k) and detection delay t (k) Whether it exceeds the corresponding threshold and To measure the immediate risk, where w7~w 12 Represents the weight parameter used to balance the values ​​of each indicator.

[0026]

[0027] Base station j modifies the benefit value according to the short-term risk value measured by the performance to avoid short-term risks and further improve the detection performance:

[0028]

[0029] Step 8: Base station j constructs the state, the selected detection strategy, the revised benefit value and the short-term risk value into an experience sequence and stores it in the experience pool;

[0030] In step 8, base station j determines the state s (k) , detection strategy x (k) , revised benefits and risk value z (k) Constructing experience sequences And the intrusion detection experience sequence is stored in the experience pool D, that is, D←D∪{ξ (k)}.

[0031] Step 9: Randomly sample Z intrusion detection experiences from the experience pool D to form a batch sample, where Z is an integer greater than or equal to 1;

[0032] In step 9, base station j randomly samples Z intrusion detection experiences from the experience pool D to form a batch sample {ξ (φ(z))} 1≤z≤Z , where φ(z) obeys the uniform distribution of U(1,k).

[0033] Step 10: Update the weight parameters η of the two critic networks in the high and low layers respectively (k) , and μ (k) ,

[0034] In step 10, base station j updates the weight parameter η of the high-level critic network 1 using the Adam optimization algorithm by minimizing the mean square error between the TD target value and the current Q value. (k) , by minimizing the weight parameter η (k)The loss function L(η (k) ) to update the high-level Critic network 1:

[0035]

[0036] Similar to the update of high-level Critic network 1, the weight parameters of high-level Critic network 2 are updated using the Adam optimization algorithm. Update the weight parameters μ of the low-level Critic networks 1 and 2 (k) and

[0037] Step 11: Update the weight parameters θ of the high-level and low-level Actor networks respectively (k) and

[0038] In step 11, base station j uses the Adam optimization algorithm to update the weight parameters θ of the high-level Actor network (k) , by minimizing the weight parameter θ (k) The loss function L(θ (k) ) to update the high-level Actor network:

[0039]

[0040] Similar to the update of the high-level Actor network, the Adam optimization algorithm is used to update the weight parameters of the low-level Actor network.

[0041] Step 13: Repeat steps 3 to 11 until the base station learns a stable classifier selection strategy and classifier parameter optimization strategy, and the benefit u (k) converges to a stable value.

[0042] Different from the research on drone intrusion detection networks using traditional algorithms, data processing and deep learning, this paper uses the adaptability of reinforcement learning and combines the collected drone information to design a drone network intrusion detection method based on reinforcement learning. This method uses hierarchical reinforcement learning to construct a state based on the information obtained by ADS-B and the data packet transmission information obtained by the base station transceiver module to detect potential malicious attacks on the drone network, thereby issuing a security warning so that corresponding security protection decisions can be made later. It improves detection accuracy while reducing detection delay and energy consumption, and optimizes the security of the drone network.

[0043] The drone network intrusion detection method described in this paper is based on reinforcement learning. Reinforcement learning dynamically explores and learns about drone networks, making optimal decisions even in unknown environments and attack scenarios. This method accurately detects potential malicious attack traffic within the drone network and identifies its attack type, promptly issuing security alerts to notify other healthy drones, preventing dangerous communications and improving network security and stability.

[0044] The present invention has the following beneficial effects:

[0045] The present invention proposes a reinforcement learning-based method for detecting intrusion in drone networks. This method can optimize detection strategies for drone networks, timely detect and identify abnormal drone traffic information and the types of attacks it initiates, and protect drone communication security. The present invention constructs system observations based on the drone's identity information and location information obtained through ADS-B, as well as the time interval and number of data packet transmissions obtained by parsing the base station transceiver module, combined with detection performance. The system observations, combined with previous detection performance, form a system state as the input of a hierarchical reinforcement learning model. Based on the output of the hierarchical reinforcement learning model, the base station selects the most appropriate classifier and optimizes the corresponding parameters of the classifier to detect whether malicious attacks exist in the large amount of traffic in the drone system, and accurately identify the attack type of the traffic. This improves detection accuracy while reducing detection latency and energy consumption, thereby enhancing the security of drone communications in the drone system. DETAILED DESCRIPTION

[0046] In order to more clearly understand the technical content of the present invention, the following examples are given to illustrate in detail. It should be understood that the following example is only a preferred embodiment of the present invention and should not be understood as limiting the technical solution of the present invention.

[0047] Example 1

[0048] This embodiment relates to a drone network intrusion detection method based on reinforcement learning, which specifically includes the following steps:

[0049] Step 1: The drone network to be detected contains two drones and one base station;

[0050] Step 2: Construct a hierarchical reinforcement learning model for the base station. The high-level and low-level layers each include an actor network consisting of a fully connected layer, two critic networks, and two target networks. The actor network and critic network each contain an input layer, a hidden layer, and an output layer. The high-level input layer consists of 13 neurons, the hidden layer consists of 32 neurons, and the output layer consists of 4 neurons; the low-level input layer consists of 14 neurons, the hidden layer consists of 32 neurons, and the output layer consists of 3 neurons. Initialize the weight parameters of the high-level actor network, the two critic networks, and the two target networks to θ. (0) , η (0) , δ (0) as well as The weight parameters of the low-level Actor network, the two Critic networks, and the two target networks are μ (0) , as well as Initialize the learning rate α to 0.001, the discount factor γ to 0.8, and the entropy coefficient λ to 0.2; set the weight parameters in the benefit function and risk value to 1, the number of randomly sampled experience items Z = 64, and the F1 score requirement for intrusion detection model training The precision and accuracy requirements are q (k) =0.95, recall rate requirement False alarm rate requirements False negative rate requirements Detection delay requirements

[0051] Step 3: Based on the ADS-B broadcast information, at time slot k, the base station obtains the identity information of M drones {G i} 1≤i≤M and location information {d i (k)} 1≤i≤M In T time slots, the base station's transceiver module marks the arrival timestamp of each received data packet, and calculates the data packet transmission interval {τ i (k)} 1≤i≤M , according to the number of data packets received in the T time slot, the number of transmitted data packets {n i (k)} 1≤i≤M , where 1≤i≤M. In addition, base station j measures the F1 score f of the intrusion detection in the previous time slot (k-1) , precision p (k-1) , recall rate r (k-1) , false alarm rate l (k-1)and the false negative rate m (k-1) , the constructed state vector s (k) as follows:

[0052] s (k) =[{G (k)} 1≤i≤M ,{d i (k)} 1≤i≤M ,{τ i (k)} 1≤i≤M ,{n i (k)} 1≤i≤M ,f (k-1) ,p (k-1) ,r (k-1) ,l (k-1) ,m (k-1) ](1.1).

[0053] Step 4: The base station sends the state vector s (k) Input into the high-level Actor network, select from four classifiers: random forest, Bayesian, decision tree and SVM, and get a1 (k) , the two Critic networks output classifier type a1 according to the Actor network (k) Evaluate the high-level long-term discounted expected benefit y (k) .

[0054] Step 5: The base station sends the state vector s (k) and high-level action a1 (k) The two lower-level Critic networks are fed into the lower-level Actor network and the corresponding parameters are optimized according to the selected classifier, namely the number of trees in the random forest, the smoothing parameter in the Bayesian, the maximum depth of the tree in the decision tree, and the regularization parameter in the SVM. The two lower-level Critic networks are optimized according to the parameter a2 output by the lower-level Actor network. (k) Evaluating the expected long-term discounted benefits of the low-level Get the lower level action a2 (k) To obtain the joint optimization strategy x for classifier and parameter optimization (k) =[a1 (k) ,a2 (k) ].

[0055] Step 6: The base station selects the intrusion detection strategy x (k) , the F1 score is calculated by detecting true positives, true negatives, false positives and false negatives as f (k) , the false negative rate is m (k) , the false alarm rate is l (k) , with an accuracy of p (k) , the accuracy is q (k) , the recall rate is r (k), and measure the delay t consumed by the detection (k) , calculate the benefit value u through the above indicators (k) as follows:

[0056] u (k) =f (k) +w1p (k) +w2q (k) +w3r (k) -w4l (k) -w5m (k) -w6t (k) (1.2).

[0057] Step 7: The detection index F1 score f (k) , false negative rate m (k) , false alarm rate l (k) , precision p (k) , accuracy q (k) , recall rate r (k) and detection delay t (k) Instead of asking and Compare them respectively and evaluate the immediate risk value z (k) :

[0058]

[0059] The base station is based on the instant risk value z (k) Corrected benefit value u (k) ,Right now

[0060]

[0061] Step 8: The base station sets the state s (k) , detection strategy x (k) , revised benefits and short-term risk value z (k) Constructed as an experience sequence And store the intrusion detection experience sequence into the experience pool D.

[0062] Step 9: The base station randomly samples 64 detection experiences from the experience pool to form a batch sample;

[0063] Step 10: The base station uses the Adam optimization algorithm to minimize the loss function L(η (k) )and Update the weight parameter η of the high-level critic network (k) and

[0064]

[0065] The Adam optimization algorithm is used to minimize the loss function L(μ (k) )and Update the weight parameter μ of the low-level Critic network (k) and

[0066]

[0067]

[0068] Step 11: The base station uses the Adam optimization algorithm to minimize the loss function L(θ (k) )and Update the weight parameters θ of the high-level and low-level Actor networks (k) and

[0069]

[0070] Step 12: Repeat steps 3 to 11 until the base station learns a stable classifier selection strategy and classifier parameter optimization strategy.

[0071] The above embodiments are only preferred embodiments of the present invention and should not be considered to limit the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent of the present invention.

Claims

1. A UAV network intrusion detection method based on reinforcement learning, characterized in that: The UAV network intrusion detection method based on reinforcement learning includes the following steps: 1) Obtaining a target UAV network, wherein the UAV network includes M UAVs and N base stations, where M and N are integers greater than or equal to 1; 2) Construct a reinforcement learning model for the jth base station among N base stations, where 1≤j≤N; the reinforcement learning model is divided into a high-level and a low-level layer, each of which contains an actor network, two critic networks, and two target networks; the weight parameters of the high-level actor network, the two critic networks, and the two target networks are initialized as θ (0) , η (0) , δ (0) as well as The weight parameters of the low-level Actor network, the two Critic networks, and the two target networks are μ (0) , as well as And the initialization learning rate is α, the discount factor is γ and the entropy coefficient is λ; 3) The base station j constructs the state vector s of time slot k (k) , where k is an integer greater than or equal to 1; 4) The base station j is based on the state vector s (k) Classifier strategy a1 for detecting C types of intrusions from A1 = {1, 2, ..., C} (k) Select, where C is an integer greater than or equal to 1; 5) The base station j is based on the state vector s (k) And the classifier strategy a1 obtained in step 4 (k) Jointly optimize classifier parameter a2 (k) ; 6) The base station j calculates the benefit; 7) The base station j evaluates the short-term risk value of each state-action pair and modifies the benefit according to the short-term risk value; 8) The base station j constructs the state, the selected detection strategy, the revised benefit value and the short-term risk value into an experience sequence and stores it in the experience pool; 9) The base station j randomly samples Z intrusion detection experiences from the experience pool D to form a batch sample, where Z is an integer greater than or equal to 1; 10) The base station j updates the weight parameters η of the two critic networks in the upper layer and the lower layer respectively (k) , and μ (k) , 11) The base station j updates the weight parameters θ of the high-level and low-level Actor networks respectively (k) and 12) Repeat steps 3) to 11) until the base station learns a stable classifier selection strategy and classifier parameter optimization strategy, and the benefit u (k) converges to a stable value.

2. The method for detecting drone network intrusion based on reinforcement learning according to claim 1 is characterized in that: In step 3), construct the state vector s (k) The specific process is as follows: at time slot k, the base station j obtains the identity information of M drones by broadcasting the information received from the ADS-B receiver {G i } 1≤i≤M , and location information {d i (k) } 1≤i≤M ; In T time slots, the base station's transceiver module marks the arrival timestamp of each received data packet, and calculates the data packet transmission interval {τ i (k) } 1≤i≤M , and obtain the number of transmitted data packets according to the number of data packets received in the T time slot {n i (k) } 1≤i≤M , where 1≤i≤M; In addition, the F1 score of the base station j measuring the intrusion detection in the previous time slot is f (k-1) , with an accuracy of p (k-1) , the recall rate is r (k-1) , the false alarm rate is l (k-1) and the false negative rate is m (k-1) According to the above drone traffic and detection performance information, the state vector s constructed by the base station j (k) as follows: s (k) =[{G (k) } 1≤i≤M ,{d i (k) } 1≤i≤M ,{τ i (k) } 1≤i≤M ,{n i (k) } 1≤i≤M ,f (k-1) ,p (k-1) ,r (k-1) ,l (k-1) ,m (k -1) ](0.1)。 3. The method for detecting drone network intrusion based on reinforcement learning according to claim 1, characterized in that: In step 4), a1 (k) represents the classifier type selected by the base station j for the states of M drones; The base station j will state vector s (k) Input into the high-level Actor network and output high-level actions, that is, classifier type a1 (k) ; The two Critic networks are based on a1 (k) Output the long-term expected benefit Q of all classifier strategies in the current state H (s',a1';δ (k) )and Update y of target Q value (k) As shown below: where π(a1'|s';θ (k) ) represents the action probability sampled by the policy network in state s', and the base station selects the classifier strategy a1 according to the high-level Q value (k) .

4. The method for detecting drone network intrusion based on reinforcement learning according to claim 1, characterized in that: In step 5), a2 (k) Represents a1 (k) The corresponding optimizable parameters in the classifier; the base station j converts the state vector s (k) and high-level action a1 (k) The two are input into the low-level Actor network and the parameter strategy a2 that can be optimized by the classifier is output. (k) , in and They are classifier strategies a1 (k) The minimum and maximum values ​​of the parameters can be optimized; Similar to the calculation method of high-level target Q value, low-level target Q value The calculation is: in Represents the action probability sampled by the policy network in state s'. The base station selects the classifier parameter optimization strategy a2 based on the low-level Q value (k) .

5. The method for detecting drone network intrusion based on reinforcement learning according to claim 1, characterized in that: In step 6), the specific steps for the base station j to calculate the benefit value are: according to the obtained action x (k) =[a1 (k) ,a2 (k) ] The F1 score f of the current time slot can be calculated based on the true positive, true negative, false positive and false negative of intrusion detection (k) , false negative rate m (k) , false alarm rate l (k) , precision p (k) , accuracy q (k) , recall rate r (k) , and measure the delay t consumed by the detection (k) Based on the above indicators, the benefits can be obtained according to the following formula, where w1 to w6 represent the weight parameters of F1 score, precision, accuracy, recall rate, false alarm rate, missed alarm rate, and detection delay, respectively: and (k) =f (k) +w1p (k) +w2q (k) +w3r (k) -w4l (k) -w5m (k) -w6t (k) (0.4).

6. The method for detecting drone network intrusion based on reinforcement learning according to claim 1, characterized in that: In step 7), the specific steps of the base station j evaluating the risk value are: constructing an indicator function, according to the F1 score f (k) , false negative rate m (k) , false alarm rate l (k) , precision p (k) , accuracy q (k) , recall rate r (k) and detection delay t (k) Whether it exceeds the corresponding threshold and To measure the immediate risk, where w7~w 12 Represents the weight parameter used to balance the values ​​of each indicator; Base station j according to short-term risk z (k) Value-corrected benefit value u (k) , to avoid short-term risks and further improve detection performance:

7. The method for detecting drone network intrusion based on reinforcement learning according to claim 1, characterized in that: In step 8), the base station j determines the state s (k) , detection strategy x (k) , revised benefits and risk value z (k) Constructing experience sequences And the intrusion detection experience sequence is stored in the experience pool D, that is, D←D∪{ξ (k) }.

8. The method for detecting drone network intrusion based on reinforcement learning according to claim 1, characterized in that: In step 9), the base station j randomly samples Z intrusion detection experiences from the experience pool D to form a batch sample {ξ (φ(z)) } 1≤z≤Z , where φ(z) obeys the uniform distribution of U(1,k).

9. The method for detecting drone network intrusion based on reinforcement learning according to claim 1, characterized in that: In step 10), the base station j updates the weight parameter η of the high-level critic network 1 using the Adam optimization algorithm by minimizing the mean square error between the TD target value and the current Q value. (k) , by minimizing the weight parameter η (k) The loss function L(η (k) ) to update the high-level Critic network 1: Similar to the update of high-level Critic network 1, the weight parameters of high-level Critic network 2 are updated using the Adam optimization algorithm. Update the weight parameters μ of the low-level Critic networks 1 and 2 (k) and 10. The UAV network intrusion detection method based on reinforcement learning according to claim 1 is characterized in that: In step 11), the base station j uses the Adam optimization algorithm to update the weight parameter θ of the high-level Actor network (k) , by minimizing the weight parameter θ (k) The loss function L(θ (k) ) to update the high-level Actor network: Similar to the update of the high-level Actor network, the Adam optimization algorithm is used to update the weight parameters of the low-level Actor network.

Images