A method for detecting intrusion of a UAV network based on reinforcement learning

Abstract

The present application belongs to the field of modern communication network security technology, and proposes a kind of unmanned aerial vehicle network intrusion detection method based on reinforcement learning.The present application constructs system observation according to the identity information of unmanned aerial vehicle obtained by ADS-B, position information and time interval and data packet transmission quantity of data packet transmission obtained by using base transceiver module analysis combined with detection performance, and system observation is combined with previous detection performance to form system state as the input of hierarchical reinforcement learning model.Base station selects the most suitable classifier according to the output of hierarchical reinforcement learning model and optimizes the corresponding parameters of the classifier, detects whether there is malicious attack in the large amount of traffic existing in unmanned aerial vehicle system, and accurately identifies the attack type of traffic, reduces detection delay and energy consumption while improving detection accuracy, thereby improving the communication security of unmanned aerial vehicle in unmanned aerial vehicle system, optimizing the detection strategy of unmanned aerial vehicle network, timely detecting and identifying the attack type initiated by unmanned aerial vehicle abnormal traffic information.

Description

Technical Field

[0001] This invention relates to the security of modern communication networks and proposes a method for detecting unmanned aerial vehicle (UAV) network intrusions based on reinforcement learning, belonging to the field of modern communication network security technology. Background Technology

[0002] In drone networks, numerous drones perform various services or computing tasks, generating a large amount of drone traffic. However, this traffic may contain potential malicious attacks. Attackers may launch various malicious behaviors in drone networks, such as interference attacks, denial-of-service attacks, spoofing attacks, and deception attacks, which can affect normal communication between drones and base stations, as well as between drones themselves.

[0003] Currently, significant progress has been made in drone intrusion detection methods. For example, Chinese invention patent application CN113556191A proposes a drone intrusion detection method based on channel state information; and Chinese invention patent application CN118469156A provides a novel intelligent drone intrusion expulsion scheduling system and method. In addition, some drone intrusion detection methods using deep learning have been widely applied in recent years. For instance, Chinese invention patent application CN118353655A provides a drone intrusion detection system based on GNN and LSTM; and Chinese invention patent application CN118762295A provides a high-energy-efficiency embedded drone intrusion detection system based on FPGA. However, these methods have slight shortcomings in detection accuracy, and the detection latency and energy consumption are relatively high, which may prevent legitimate drones from avoiding malicious attacks in time, resulting in a decline in service performance, and in severe cases, mission failure or drone damage.

[0004] Reinforcement learning, also known as boosting learning or reinforcement learning, is a paradigm and methodology of machine learning. It is used to describe and solve problems in which intelligent agents learn strategies to maximize rewards or achieve specific goals during their interaction with the environment. It has been applied to a variety of different fields. Summary of the Invention

[0005] In view of the above-mentioned technical problems in the prior art, the purpose of this invention is to provide a UAV network intrusion detection method based on reinforcement learning. This detection method uses reinforcement learning to realize the intrusion detection of malicious attacks on UAV networks and the identification of attack types, reduce detection latency and energy consumption, optimize detection accuracy, and thus improve the network security of UAV networks.

[0006] To achieve the above-mentioned objectives, the technical solution adopted by the present invention is as follows:

[0007] A method for detecting unmanned aerial vehicle (UAV) network intrusion based on reinforcement learning, the method comprising the following steps:

[0008] Step 1: Obtain the target drone network, which contains M drones and N base stations, where M and N are integers greater than or equal to 1;

[0009] Step 2: Construct a reinforcement learning model for the j-th base station out of N base stations, where 1 ≤ j ≤ N; the reinforcement learning model is divided into a high-level layer and a low-level layer, each layer containing one Actor network, two Critic networks, and two target networks; initialize the weight parameters of the high-level Actor network, the two Critic networks, and the two target networks as follows: , , , as well as The weight parameters of the low-level Actor network, the two Critic networks, and the two target networks are as follows: , , , as well as The learning rate is initialized to α, the discount factor is γ, and the entropy coefficient is λ.

[0010] Step 3: Base station j constructs the state vector of time slot k. , where k is an integer greater than or equal to 1;

[0011] In step 3, the state vector is constructed. The specific process is as follows: In time slot k, base station j obtains the identity information of M UAVs from the information received from the ADS-B receiver via broadcast. and location information Within T time slots, the base station's transceiver module timestamps each received data packet, and calculates the data packet transmission interval based on the arrival timestamps of two adjacent packets from the same UAV. The number of transmitted data packets is obtained based on the number of data packets received within time slot T. Where 1 ≤ i ≤ M. Furthermore, base station j measures the F1 score of intrusion detection in the previous time slot as: The accuracy is The recall rate was False alarm rate and the false negative rate Based on the aforementioned drone traffic and detection performance information, the state vector constructed by base station j... as follows:

[0012] (1.1).

[0013] Step 4: Base station j, based on the state vector Choose from C classifiers for intrusion detection from A1 = {1, 2, ..., C}, where C is an integer greater than or equal to 1;

[0014] In step 4, This indicates the classifier type selected by the base station for the states of M drones. Base station j will store the state vectors... The input is fed into a high-level Actor network, and the output is the action of the high-level network, i.e., the classifier type. The two Critic networks are based on... Output the long-term expected benefit (i.e., Q-value) of all classifier policies in the current state. and Update of the target Q value As shown below:

[0015] (1.2)

[0016] in Indicates the policy network in state The base station selects a classifier strategy based on the downsampled action probability and the high-layer Q-value. .

[0017] Step 5: Base station j, based on the state vector The classifier strategy obtained in step 4 Jointly optimize classifier parameters ;

[0018] In step 5, The representative selected The corresponding optimizable parameters in the classifier. Base station j will have a state vector. and high-level actions The common inputs are fed into the low-level Actor network, and the output is a parameter strategy that the classifier can optimize. , ,in and These are classifier strategies The minimum and maximum values ​​of the optimizable parameters. Similar to the calculation method for the Q-value of high-level objectives, the Q-value of low-level objectives... The calculation is as follows:

[0019] (1.3)

[0020] in Indicates the policy network in state The base station selects a classifier parameter optimization strategy based on the downsampled action probability and the low-level Q-value. .

[0021] Step 6: Calculate the benefits of base station j;

[0022] In step 6, based on the obtained action The F1 score for the current time slot is calculated based on true positives, true negatives, false positives, and false negatives in intrusion detection. underreporting rate False alarm rate accuracy accuracy Recall rate And measure the latency consumed by the detection. Based on the above indicators, the benefits can be obtained according to the following formula, where w1~w6 represent the weighting parameters of precision, accuracy, recall, false alarm rate, false negative rate, and detection latency, respectively:

[0023] (1.4).

[0024] Step 7: Base station j evaluates the short-term risk value of each state-action pair and makes benefit adjustments;

[0025] In step 7, an indicator function is constructed based on the F1 score. underreporting rate False alarm rate accuracy accuracy Recall rate and detection delay Does it exceed its corresponding threshold? , , , , , and To measure immediate risk, where w7~w 12 These represent the accuracy used for balancing. Accuracy Recall rate underreporting rate False alarm rate and detection delay Weighting parameters for each indicator value.

[0026]

[0027] (1.5),

[0028] Base station j adjusts the benefit value based on the short-term risk value measured by performance, in order to avoid short-term risks and further improve detection performance:

[0029] (1.6).

[0030] Step 8: Base station j constructs an experience sequence from the state, the selected detection strategy, the corrected benefit value, and the short-term risk value and stores it in the experience pool;

[0031] In step 8, base station j determines the status... Detection strategy The benefits after the correction and risk value Constructing experience sequences Furthermore, the intrusion detection experience sequence is stored in the experience pool D, i.e. .

[0032] Step 9: Randomly sample Z intrusion detection experiences from the experience pool D to form a batch sample, where Z is an integer greater than or equal to 1;

[0033] In step 9, base station j randomly samples Z intrusion detection experiences from experience pool D to form batch processing samples. ,in It follows a uniform distribution of U(1, k).

[0034] Step 10: Update the weight parameters of the two Critic networks in the high-level and low-level layers respectively. , and , ;

[0035] In step 10, base station j updates the weight parameters of the higher-layer Critic network 1 using the Adam optimization algorithm by minimizing the mean square error between the TD target value and the current Q value. By minimizing the weight parameters loss function Update the high-level Critic network 1:

[0036] (1.7)

[0037] Similar to the update of the high-level Critic network 1, the weight parameters of the high-level Critic network 2 are updated using the Adam optimization algorithm. Update the weight parameters of the lower-level Critic networks 1 and 2. and .

[0038] Step 11: Update the weight parameters of the high-level and low-level Actor networks respectively. and ;

[0039] In step 11, base station j updates the weight parameters of the higher-layer Actor network using the Adam optimization algorithm. By minimizing the weight parameters loss function To update the high-level Actor network:

[0040] (1.8)

[0041] Similar to updating high-level Actor networks, the Adam optimization algorithm is used to update the weight parameters of low-level Actor networks. .

[0042] Step 12: Repeat steps 3-11 until the base station learns a stable classifier selection strategy and classifier parameter optimization strategy, achieving the desired benefits. It converges to a stable value.

[0043] Unlike research on drone intrusion detection networks that uses traditional algorithms, data processing, and deep learning, this invention utilizes the adaptability of reinforcement learning and combines it with collected drone information to design a reinforcement learning-based drone network intrusion detection method. This method uses hierarchical reinforcement learning to construct a state based on information obtained from ADS-B and data packet transmission information obtained from the base station transceiver module to detect potential malicious attacks on the drone network, thereby issuing security warnings to enable subsequent security protection decisions. This improves detection accuracy while reducing detection latency and energy consumption, thus optimizing the security of the drone network.

[0044] The drone network intrusion detection method described in this invention is based on reinforcement learning. Reinforcement learning, in the presence of unknown environments and attack information, dynamically makes optimal decisions through exploration and learning within the drone network. It can accurately detect potential malicious attack traffic in the drone network and identify its attack type, promptly issuing security alerts to inform other legitimate drones, avoiding dangerous communications, and improving network security and stability.

[0045] The present invention has the following beneficial effects:

[0046] This invention proposes a reinforcement learning-based method for drone network intrusion detection, which optimizes detection strategies for drone networks, promptly detects and identifies abnormal traffic information from drones and the types of attacks they initiate, thus protecting drone communication security. This invention constructs system observations based on drone identity and location information obtained through ADS-B, and the data packet transmission time intervals and number of data packets parsed by the base station transceiver module, combined with detection performance. These system observations, combined with previous detection performance, form a system state, which serves as input to a hierarchical reinforcement learning model. The base station, based on the output of the hierarchical reinforcement learning model, selects the most suitable classifier and optimizes its parameters to detect malicious attacks within the large volume of traffic in the drone system, accurately identifying the attack types. This improves detection accuracy while reducing detection latency and energy consumption, thereby enhancing the communication security of drones within the drone system. Detailed Implementation

[0047] To provide a clearer understanding of the technical content of this invention, the following embodiments are provided for detailed description. It should be understood that the following embodiments are merely preferred embodiments of this invention and should not be construed as limiting the technical solution of this invention.

[0048] Example 1

[0049] This embodiment relates to a method for detecting unmanned aerial vehicle (UAV) network intrusion based on reinforcement learning. The detection method specifically includes the following steps:

[0050] Step 1: The drone network to be tested contains two drones and one base station;

[0051] Step 2: Construct a hierarchical reinforcement learning model for the base station. The high-level and low-level models each include an Actor network (composed of fully connected layers), two Critic networks, and two target networks. The Actor and Critic networks each contain an input layer, a hidden layer, and an output layer. The high-level model has a 13-neuron input layer, a 32-neuron hidden layer, and a 4-neuron output layer; the low-level model has a 14-neuron input layer, a 32-neuron hidden layer, and a 3-neuron output layer. Initialize the weight parameters for the high-level Actor network, the two Critic networks, and the two target networks as follows: , , , as well as The weight parameters of the low-level Actor network, the two Critic networks, and the two target networks are: , , , as well as The learning rate α is initialized to 0.001, the discount factor γ to 0.8, and the entropy coefficient λ to 0.2. All weight parameters in the benefit function and risk value are set to 1. The number of randomly sampled experience points Z = 64. The F1 score requirement for training the intrusion detection model is... =0.96, accuracy and precision requirements are: =0.95, =0.95, recall rate requirement =0.99, false alarm rate requirement =0.015, false negative rate requirement =0.003, detection delay requirement =1ms.

[0052] Step 3: Based on the ADS-B broadcast information, in time slot k, the base station obtains the identity information of M drones. and location information Within T time slots, the base station's transceiver module timestamps each received data packet, and calculates the data packet transmission interval based on the arrival timestamps of two adjacent packets from the same UAV. The number of transmitted data packets is obtained based on the number of data packets received within time slot T. Where 1 ≤ i ≤ M. Furthermore, base station j measures the F1 score of intrusion detection in the previous time slot. accuracy Recall rate False alarm rate and underreporting rate The constructed state vector as follows:

[0053] (2.1).

[0054] Step 4: The base station will transfer the state vector The input is fed into a high-level Actor network, which selects from four classifiers: Random Forest, Bayesian, Decision Tree, and SVM, to obtain... The two Critic networks are based on the classifier type output by the Actor network. Assess the expected benefits of long-term discounts for senior management .

[0055] Step 5: The base station will transfer the state vector and high-level actions The parameters are fed into a low-level Actor network, which optimizes the corresponding parameters based on the selected classifier. These parameters include the number of trees in the random forest, the smoothing parameter in the Bayesian algorithm, the maximum depth of the trees in the decision tree, and the regularization parameter in the SVM. The two low-level Critic networks then optimize the parameters based on the output of the low-level Actor network. Assess the expected benefits of long-term discounts at lower levels To obtain lower-level actions To obtain a joint optimization strategy for classifier and parameter optimization. .

[0056] Step 6: The base station determines the intrusion detection strategy based on the selected strategy. The F1 score is calculated based on the true positive, true negative, false positive, and false negative results. The false negative rate was False alarm rate The accuracy is The accuracy rate is The recall rate was And measure the latency consumed by the detection. The benefit value is calculated using the above indicators. as follows:

[0057] (2.2).

[0058] Step 7: Calculate the F1 score for the detection metrics. underreporting rate False alarm rate accuracy accuracy Recall rate and detection delay Rather than demand , , , , , and Compare them separately to assess the immediate risk value. :

[0059]

[0060] (2.3),

[0061] Base station based on real-time risk value Corrected benefit value ,Right now

[0062] (2.4).

[0063] Step 8: The base station will change the status. Detection strategy The benefits after the correction and short-term risk value Constructed as an experience sequence Furthermore, the intrusion detection experience sequence is stored in the experience pool D.

[0064] Step 9: The base station randomly samples 64 detection experiences from the experience pool to form a batch processing sample;

[0065] Step 10: The base station uses the Adam optimization algorithm to minimize the loss function. and Update the weight parameters of the high-level Critic network and :

[0066] (2.5)

[0067] (2.6)

[0068] The Adam optimization algorithm is used to minimize the loss function. and Update the weight parameters of the low-level Critic network and :

[0069] (2.7)

[0070] (2.8)

[0071] Step 11: The base station uses the Adam optimization algorithm to minimize the loss function. and Update the weight parameters of the high-level and low-level Actor networks. and :

[0072] (2.9)

[0073] (2.10)

[0074] Step 12: Repeat steps 3 to 11 until the base station learns a stable classifier selection strategy and classifier parameter optimization strategy.

[0075] The above embodiments are merely preferred embodiments of the present invention and should not be considered as limiting the scope of the present invention. All equivalent variations and improvements made within the scope of the present invention should still fall within the patent coverage of the present invention.

Claims

1. A method for detecting unmanned aerial vehicle (UAV) network intrusion based on reinforcement learning, characterized in that, The reinforcement learning-based drone network intrusion detection method includes the following steps: 1) Obtain the target drone network, which contains M drones and N base stations, where M and N are integers greater than or equal to 1; 2) Construct a reinforcement learning model for the j-th base station out of N base stations, where 1 ≤ j ≤ N; the reinforcement learning model is divided into a high-level layer and a low-level layer, each layer containing one Actor network, two Critic networks, and two target networks; initialize the weight parameters of the high-level Actor network, the two Critic networks, and the two target networks as follows: , , , as well as The weight parameters of the low-level Actor network, the two Critic networks, and the two target networks are as follows: , , , as well as The learning rate is initialized to α, the discount factor is γ, and the entropy coefficient is λ. 3) The base station j constructs the state vector of time slot k. , where k is an integer greater than or equal to 1; 4) The base station j is based on the state vector Classifier strategy for C types of intrusion detection from A1 = {1, 2, ..., C} Make a selection, where C is an integer greater than or equal to 1; In step 4), This indicates the classifier type selected by base station j for the states of M drones; base station j will store the state vectors... The input is fed into a high-level Actor network, and the output is the action of the high-level network, i.e., the classifier type. The two Critic networks are based on... Output the long-term expected benefit of all classifier policies in the current state. and Update of the target Q value As shown below: (1.2) in Indicates the policy network in state The base station selects a classifier strategy based on the downsampled action probability and the high-layer Q-value. ; 5) The base station j is based on the state vector The classifier strategy obtained in step 4 Jointly optimize classifier parameters ; In step 5), represent The corresponding optimizable parameters in the classifier; the base station j will have a state vector and high-level actions The common inputs are fed into the low-level Actor network, and the output is a parameter strategy that the classifier can optimize. , ,in and These are classifier strategies The minimum and maximum values ​​of the optimizable parameters; similar to the calculation method for the Q-value of high-level targets, the Q-value of low-level targets... The calculation is as follows: (1.3) in Indicates the policy network in state The base station selects a classifier parameter optimization strategy based on the downsampled action probability and the low-level Q-value. ; 6) Calculate the benefits of the base station j; In step 6), the specific steps for calculating the benefit value of base station j are as follows: based on the obtained action... The F1 score of the current time slot is calculated based on the true positives, true negatives, false positives, and false negatives of intrusion detection. underreporting rate False alarm rate accuracy accuracy Recall rate And measure the latency consumed by the detection. Based on the above indicators, the benefits are obtained according to the following formula, where w1~w6 represent the weight parameters of precision, accuracy, recall, false alarm rate, false negative rate, and detection latency, respectively: (1.4)； 7) The base station j evaluates the short-term risk value of each state-action pair and adjusts the benefits based on the short-term risk value; 8) The base station j constructs an experience sequence from the state, the selected detection strategy, the corrected benefit value, and the short-term risk value and stores it in the experience pool; 9) The base station j randomly samples Z intrusion detection experiences from the experience pool D to form a batch processing sample, where Z is an integer greater than or equal to 1; 10) The base station j updates the weight parameters of the two Critic networks in the high-layer and low-layer layers respectively. , and , ; In step 10), the base station j updates the weight parameters of the higher-layer Critic network 1 using the Adam optimization algorithm by minimizing the mean square error between the TD target value and the current Q value. By minimizing the weight parameters loss function Update the high-level Critic network 1: (1.7) Similar to the update of the high-level Critic network 1, the weight parameters of the high-level Critic network 2 are updated using the Adam optimization algorithm. Update the weight parameters of the lower-level Critic networks 1 and 2. and ; 11) The base station j updates the weight parameters of the higher-layer and lower-layer Actor networks respectively. and ; In step 11), the base station j updates the weight parameters of the higher-layer Actor network using the Adam optimization algorithm. By minimizing the weight parameters loss function To update the high-level Actor network: (1.8) Similar to updating high-level Actor networks, the Adam optimization algorithm is used to update the weight parameters of low-level Actor networks. ; 12) Repeat steps 3) to 11) until the base station learns a stable classifier selection strategy and classifier parameter optimization strategy, achieving the desired benefits. It converges to a stable value.

2. The method for detecting unmanned aerial vehicle (UAV) network intrusion based on reinforcement learning according to claim 1, characterized in that, In step 3), the state vector is constructed. The specific process is as follows: In time slot k, the base station j obtains the identity information of M drones from the information received from the ADS-B receiver via broadcast. and location information Within T time slots, the base station's transceiver module timestamps each received data packet and calculates the data packet transmission interval based on the arrival timestamps of two adjacent packets from the same UAV. The number of transmitted data packets is obtained based on the number of data packets received within time slot T. Where 1 ≤ i ≤ M; furthermore, the base station j measures the F1 score of the intrusion detection in the previous time slot as: The accuracy is The recall rate was False alarm rate and the false negative rate Based on the aforementioned drone traffic and detection performance information, the state vector constructed by base station j... as follows: (1.1)。 3. The method for detecting unmanned aerial vehicle (UAV) network intrusion based on reinforcement learning according to claim 1, characterized in that, In step 7), the specific steps for base station j to assess the risk value are as follows: constructing an indication function, based on the F1 score... underreporting rate False alarm rate accuracy accuracy Recall rate and detection delay Does it exceed its corresponding threshold? , , , , , and To measure immediate risk, where w7~w 12 These represent the accuracy used for balancing. Accuracy Recall rate underreporting rate False alarm rate and detection delay Weighting parameters for each indicator value; (1.5)， Base station j based on short-term risk Value Correction Benefit Value To mitigate short-term risks and further improve detection performance: (1.6)。 4. The method for detecting unmanned aerial vehicle (UAV) network intrusion based on reinforcement learning according to claim 1, characterized in that, In step 8), the base station j determines the status... Detection strategy The benefits after the correction and risk value Constructing experience sequences Furthermore, the intrusion detection experience sequence is stored in the experience pool D, i.e. .

5. The method for detecting unmanned aerial vehicle (UAV) network intrusion based on reinforcement learning according to claim 1, characterized in that, In step 9), the base station j randomly samples Z intrusion detection experiences from the experience pool D to form a batch processing sample. ,in It follows a uniform distribution of U(1, k).

Images