A low-overhead drone identity authentication method based on deep reinforcement learning

Abstract

This paper discloses a low-overhead drone authentication method based on deep reinforcement learning. This method uses a deep reinforcement learning algorithm, considers authentication latency and energy consumption as security constraints, and dynamically selects drone authentication strategies, including authentication methods, session duration, and data encryption strategies, to avoid high-risk strategies that could lead to serious privacy leaks or excessive overhead. The method records the drone's remote ID (including information such as the drone ID, speed, altitude, and longitude and latitude), the physical unclonable chip assembly, the average authentication latency and energy consumption at the previous moment, and the number of authentications, thereby constructing a system state. Deep reinforcement learning is then used to select drone authentication strategies, improving the efficiency and security of drone authentication.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a low-overhead drone identity authentication method based on deep reinforcement learning. Background Art

[0002] Unmanned aerial vehicles (UAVs) are widely used in military, industrial, and civilian fields due to their ease of deployment and low maintenance costs. They offer significant benefits in surveillance, detection, transportation, and emergency rescue. However, UAVs often operate in harsh natural environments, posing various challenges to their communication security. Untrusted communication environments can lead to risks such as valuable data leakage and loss of important cargo carried by drones. Therefore, providing secure communication channels is essential, and identity authentication plays a crucial role.

[0003] There is extensive research on drone identity authentication both domestically and internationally. For example, Chinese patent application CN116506857A discloses a drone identity authentication method based on the timely generation and updating of authentication information. This method utilizes physical unclonable functions (PUFs) to generate unique identity information for each drone, improving the reliability of drone identity authentication. In 2024, Chuang Tian et al. used physical unclonable functions (PUFs) to provide unique identities for unmanned IoT devices. They used their challenge-response mechanism and the key generation function of a ground control station to generate session keys for participants, ensuring communication security. In 2024, Raja Karmakar et al., in addition to utilizing the uniqueness of PUFs to verify drone identities, also used a fuzzy extractor to eliminate noise in the PUF output, further improving authentication reliability. They also proposed using the Thompson Sampling (TS) algorithm to dynamically adjust the duration of authentication sessions. However, a drawback of this approach is that the algorithm does not consider the risks of sessions that are too short or too long, and thus fails to avoid dangerous strategies. In 2024, Junfeng Miao et al. proposed a secure and effective drone-assisted IoT authentication protocol. This protocol uses elliptic curve cryptography (ECC) for authentication, providing both security and robustness against known attacks. However, this scheme suffers from the complexity of ECC's mathematical theory, making it difficult to implement in practice. Summary of the Invention

[0004] The present invention aims to address the shortcomings and deficiencies of the aforementioned prior art by providing a low-overhead drone authentication method based on deep reinforcement learning. This method is applied to balance the overhead and security of the authentication process between drones and ground base stations. To ensure low overhead and high security during drone authentication, the present invention uses a deep reinforcement learning algorithm, considers authentication latency and energy consumption as security constraints, and dynamically selects drone authentication strategies, including authentication methods, session duration, and data encryption strategies. This avoids high-risk strategies that could lead to serious privacy leaks or excessive overhead.

[0005] The technical solution adopted by the present invention to solve the technical problem is: a low-overhead drone identity authentication method based on deep reinforcement learning, which includes the following steps:

[0006] Step 1: The drone identity authentication system consists of M drones and 1 ground base station, some of which have built-in PUF chips and some do not.

[0007] Step 2: Taking the i-th drone as an example, drone i sends a registration request with its real identity ID to the base station through a secure channel.

[0008] In step 2, each drone generates its own public key p i , and private key q i , and p i and ID form a message {p i ,ID} is sent to the base station as its own identity credential. The base station saves the received registration message and generates its own identity identifier BID, public key p0, and private key q0. The base station then sends {p0, BID} to the drone, which uses this return message to verify the legitimacy of the identity authentication. In addition, the drone equipped with a PUF chip will also send a registration request to the base station using its remote ID. After receiving the registration request, the base station generates the drone's temporary identity information and a series of challenges and sends the challenges to the drone. After receiving the challenges, the drone uses the embedded PUF to generate a response and sends the response to the base station. After receiving the response, the base station stores the drone ID and challenge-response pair in a database.

[0009] Step 3: After the registration of M drones is completed, the identity authentication phase is carried out.

[0010] Step 4: Taking the i-th UAV as an example, the ground base station constructs a deep neural network, Q network, for UAV i, where 1≤i≤M. The weight parameters of the initial Q network are And initialize the learning rate to α and the discount factor to γ. And set the number of learning rounds Each round of learning resets the environment. There are K moments in a learning round. At each moment k, the base station selects an authentication strategy for drone i.

[0011] Step 5: Drone i sends an authentication message to the ground base station

[0012] In step 5, the drone generates a remote ID Including the drone's ID, altitude, speed, longitude and latitude information, and obtain the timestamp T according to the current time i (k) , generate random numbers Generate a pseudonym F through a hash function i (k) , an authentication message will be generated based on the above information Send to the base station.

[0013] Step 6: After receiving the message, the base station checks whether the drone has been registered. If not, drone i needs to register first. The base station will check the validity of the timestamp and whether the random number has been used to prevent replay attacks.

[0014] Step 7: The ground base station constructs the current state vector for UAV i

[0015] In step 7, the method for the base station to construct the system state vector for UAV i can be: record the remote ID of the UAV at time k as PUF chip assembly status And the average authentication delay at the previous moment and energy consumption and the number of authentications

[0016]

[0017] Step 8: The ground base station calculates the state vector Through policy set A i Select a joint optimization strategy for identity authentication method, session time, and encryption strategy.

[0018] In step 8, the policy set is defined as in If it is PUF-based, it will use ECC-based authentication. Represents the session time, Represents the encryption strategy. The ground base station converts the state vector Input into the Q network. The Q network outputs the long-term discounted expected benefits of all participating node selection strategies in the current state And use the ò-greedy method to select an action based on this value. The action selected by the sensor device at this time is The specific method is: select Q with a probability of 1-ò i The action with the largest value randomly selects other actions with probability ò, where ò∈(0,1). ò determines the exploratory nature of the sensor device. The larger its value, the more random the base station will be when selecting actions. This value is usually set to a small positive number.

[0019] Step 9: The base station authenticates the drone i according to the selected authentication strategy.

[0020] In step 9, when the base station selects the PUF-based authentication strategy, it extracts the corresponding challenge-response pair from the database based on the drone's ID, generates a timestamp and a random number, and combines this information with the selected challenge and uses the encryption strategy. After receiving the message, the drone first decrypts and verifies the validity of the timestamp and whether the random number is reused. Then, it uses the built-in PUF hardware to generate a response based on the challenge received, and generates a new random number and timestamp. After encryption, it is transmitted back to the base station. After receiving it, the base station decrypts the authentication information, verifies the validity of the timestamp and the uniqueness of the random number, and verifies whether the received response matches the challenge-response pair stored in the database, thus completing the identity authentication process. If the base station selects the ECC-based identity authentication strategy, the drone generates a random number and timestamp, which together with its own ID form the message n i , hash the message, using its own private key q i Sign and get signature i , the message n i With signature i Combining encryption strategies The base station decrypts the received authentication information and sends it to the base station according to the message n. i The identity ID in the file gets the corresponding public key p i Verify signature i The base station then generates a random number and a timestamp, and together with its own BID, forms a message n0, hashes the message, and signs it with its own private key q0 to obtain a signature ξ0. The message n0 and the signature ξ0 are combined using the encryption strategy The two parties encrypt the authentication information and send it to the drone. The drone decrypts the received authentication information and verifies the signature ξ0 using the corresponding public key p0 based on the identity BID in the message n0. If the verification is successful, the two parties can continue the subsequent conversation.

[0021] Step 10: After the identity authentication is completed, the drone requests to establish an authenticated session with the ground base station. The session duration is During the specified session time, the drone does not need to re-authenticate with the base station and can communicate continuously and securely. The base station recognizes the legitimacy of the drone. When the session time exceeds, the drone needs to re-authenticate with the base station.

[0022] Step 11: Ground Base Station Calculates Rewards

[0023] In step 11, according to the authentication delay Energy consumption The probability of the attacker successfully encrypting and decrypting Data protection level Calculate the reward at the current moment as follows:

[0024]

[0025] Among them, w T 、w Y 、w Q are weight coefficients, which measure the delay in executing the current task. Energy consumption And the probability of successful encryption and decryption by the attacker Importance in rewards.

[0026] Step 12: The ground base station stores the experience including status, action strategy, and reward into the experience pool.

[0027] In step 12, the ground base station changes the state Authentication strategy award Constructed as an experience sequence And store the experience sequence into the experience pool middle.

[0028] Step 13: Randomly sample Z experiences from the experience pool to form a batch sample.

[0029] In step 13, the ground base station randomly samples Z experience {φ (z)} 1≤z≤k Form batch samples where z follows a uniform distribution from 1 to k.

[0030] Step 14: Update the weight parameters of the Q network

[0031] In step 14, the ground base station uses the Adam optimization algorithm to update the weight parameters of the Q network

[0032]

[0033] Among them, β is the discount factor for weight parameter update.

[0034] Step 15: Repeat steps 7-14 above until the ground base station learns a stable identity authentication method, session time, and data encryption strategy. converges to a stable value.

[0035] Beneficial effects:

[0036] 1. This invention records the drone remote ID, PUF chip assembly status, average authentication delay and energy consumption at the previous moment, and the number of authentications to construct the system state. It then uses a Q-network-based neural network optimization based on the long-term discounted expected benefit to select an action strategy, balancing the efficiency and security of the authentication process.

[0037] 2. This invention dynamically selects drone authentication strategies, including authentication methods, session duration, and data encryption strategies, addressing the high overhead and low security issues associated with drone authentication. This method records the drone's remote ID (including information such as the drone ID, speed, altitude, and longitude and latitude), the physical unclonable chip assembly, the average authentication latency and energy consumption at the previous moment, and the number of authentications, thereby constructing a system state. It then uses deep reinforcement learning to select action strategies, improving the efficiency and security of drone authentication. DETAILED DESCRIPTION

[0038] In order to more clearly understand the technical content of the present invention, the following embodiments are given to illustrate in detail.

[0039] The present invention provides a low-overhead drone identity authentication method based on deep reinforcement learning, which includes the following steps:

[0040] Step 1: The drone identity authentication system includes two drones and a ground base station. One drone has a built-in PUF chip, while the other does not.

[0041] Step 2: Each drone generates its own public key p i , and private key q i , and p i and ID form a message {p i,ID} is sent to the base station as its own identity credential. The base station saves the received registration message and generates its own identity identifier BID, public key p0, and private key q0. The base station then sends {p0, BID} to the drone, which verifies the legitimacy of the identity authentication based on the return message. Furthermore, the drone with an embedded PUF chip will also send a registration request to the base station using its identity. Upon receiving the registration request, the base station generates the drone's temporary identity information and a series of challenges and sends the challenges to the drone. Upon receiving the challenges, the drone generates a response using the embedded PUF and sends it to the base station. Upon receiving the response, the base station stores the drone ID and the challenge-response pair in a database.

[0042] Step 3: After the drone registration is completed, the identity authentication phase will begin.

[0043] Step 4: The ground base station constructs a deep neural network for the drone, the Q network, which consists of an input layer, a hidden layer, and an output layer. The input layer consists of 10 neurons, the hidden layer consists of 64 neurons, and the output layer consists of 18 neurons. The weight parameters of the initial Q network are The initial learning rate α is 0.001, the discount factor γ is 0.8, the number of randomly sampled experiences is 32, and the benefit function weight parameter w is T 、w Y 、w Q They are 1, 0.5 and 1.5 respectively.

[0044] Step 5: Generate remote ID on the drone Including the drone's ID, altitude, speed, longitude and latitude information, and obtain the timestamp T according to the current time i (k) , generate random numbers Generate a pseudonym F through a hash function i (k ), an authentication message will be generated based on the above information Send to the base station.

[0045] Step 6: After receiving the message, the base station checks whether the drone has been registered. If not, the drone needs to register first. The base station will check the validity of the timestamp and whether the random number has been used to prevent replay attacks.

[0046] Step 7: According to the remote ID of the drone at time k PUF chip assembly status And the authentication delay of the previous moment Energy consumption and authentication times The base station builds the system state vector for the drone as:

[0047]

[0048] Step 8: The ground base station converts the state vector Input into the Q network. The Q network outputs the long-term discounted expected benefits of all participating node selection strategies in the current state And use the ò-greedy method to select the joint optimization strategy of identity authentication method, session time and encryption strategy through this value Note that the action selected by the sensor device at this time is The specific method is: select Q with a probability of 1-ò i The action with the largest value randomly selects other actions with probability ò, where ò∈(0,1). ò determines the exploratory nature of the sensor device. The larger its value, the more random the base station will be when selecting actions. This value is usually set to a small positive number.

[0049] Step 9: Authenticate according to the selected authentication method.

[0050] Step 10: After the identity authentication is completed, the drone requests to establish an authenticated session with the ground base station. The session duration is

[0051] Step 11: The ground base station determines the authentication delay Energy consumption The probability of the attacker successfully encrypting and decrypting Data protection level Calculate the reward at the current moment as follows:

[0052]

[0053] Step 12: The ground base station will be in the state Authentication strategy award Constructed as an experience sequence And store the experience sequence into the experience pool middle.

[0054] Step 13: The ground base station randomly samples 32 experience {φ( z )} 1≤z≤k Form batch samples.

[0055] Step 14: The ground base station uses the Adam optimization algorithm to update the weight parameters of the Q network as follows:

[0056]

[0057] Step 15: Repeat steps 7-14 until the ground base station learns a stable identity authentication method, session time, and data encryption strategy. converges to a stable value.

[0058] The above embodiments are only preferred embodiments of the present invention and should not be considered to limit the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent of the present invention.

Claims

1. A low-overhead drone identity authentication method based on deep reinforcement learning, characterized by: The method comprises the following steps: Step 1: The drone identity authentication system consists of M drones and one ground base station. Some drones have built-in PUF chips, while others do not. Step 2: Taking the i-th drone as an example, drone i sends a registration request with its real identity ID to the base station through a secure channel; Step 3: After M drones are registered, the identity authentication phase begins; Step 4: Taking the i-th UAV as an example, the ground base station constructs a deep neural network for UAV i, Q network, where 1≤i≤M, and initializes the weight parameters of Q network as And initialize the learning rate to α, the discount factor to γ, and set the number of learning rounds The environment is reset in each round of learning. There are K moments in a round of learning. At each moment k, the base station selects an identity authentication strategy for drone i. Step 5: Drone i sends an authentication message to the ground base station Step 6: After receiving the message, the base station checks whether the drone has been registered. If not, drone i needs to register first. The base station will check the validity of the timestamp and whether the random number has been used to resist replay attacks. Step 7: The ground base station constructs the current state vector for UAV i Step 8: The ground base station calculates the state vector Through policy set A i Select a joint optimization strategy for identity authentication method, session time, and encryption strategy; Step 9: The base station authenticates the drone i according to the selected authentication strategy; Step 10: After the identity authentication is completed, the drone requests to establish an authenticated session with the ground base station. The session duration is During the specified session time, the drone does not need to re-authenticate with the base station and can communicate continuously and securely. The base station recognizes the legitimacy of the drone. When the session time exceeds, the drone needs to re-authenticate with the base station. Step 11: Ground Base Station Calculates Rewards Step 12: The ground base station stores the experience including status, action strategy, and reward into the experience pool; Step 13: Randomly sample Z experiences from the experience pool to form a batch sample; Step 14: Update the weight parameters of the Q network Step 15: Repeat steps 7-14 above until the ground base station learns a stable identity authentication method, session time, and data encryption strategy. converges to a stable value.

2. A low-overhead drone identity authentication method based on deep reinforcement learning according to claim 1, characterized in that: In step 2, each drone generates its own public key p i , and private key q i , and p i and ID form a message {p i ,ID} is sent to the base station as its own identity credential. The base station saves the received registration message and generates its own identity identifier BID, public key p0 and private key q0. The base station then sends {p0, BID} to the drone. The drone verifies the legitimacy of the identity authentication with the return message. In addition, the drone with a PUF chip will also use its remote ID to send a registration request to the base station. After receiving the registration request, the base station generates the temporary identity information of the drone and a series of challenges, and sends the challenges to the drone. After receiving the challenge, the drone uses the embedded PUF to generate a response and sends the response to the base station. After receiving the response, the base station stores the drone ID and challenge-response pair in the database.

3. A low-overhead drone identity authentication method based on deep reinforcement learning according to claim 1, characterized in that: In step 7, the method by which the base station constructs the system state vector for UAV i is: recording the remote ID of the UAV at time k is PUF chip assembly status And the average authentication delay at the previous moment and energy consumption and the number of authentications 4. A low-overhead drone identity authentication method based on deep reinforcement learning according to claim 1, characterized in that: In step 5, the drone generates a remote ID Includes the drone's ID, altitude, speed, longitude and latitude information, and obtains a timestamp based on the current time Generate random numbers Generate pseudonyms through hash functions The authentication message generated based on the above information Send to the base station.

5. The low-overhead drone identity authentication method based on deep reinforcement learning according to claim 1, characterized in that: In step 8, the policy set is defined as in If it is PUF-based, it will use ECC-based authentication. Represents the session time, Represents the encryption strategy, the ground base station converts the state vector Input into the Q network, the Q network outputs the long-term discounted expected benefits of all participating node selection strategies in the current state And use the ò-greedy method to select an action based on this value. The action selected by the sensor device at this time is The specific method is: select Q with a probability of 1-ò i The action with the largest value randomly selects other actions with a probability of ò, where ò∈(0,1). ò determines the exploratory nature of the sensor device. The larger its value, the greater the randomness of the base station in selecting an action. This value is often set to a small positive number.

6. A low-overhead drone identity authentication method based on deep reinforcement learning according to claim 1, characterized in that: In step 9, when the base station selects the PUF-based authentication strategy, the base station extracts the corresponding challenge-response pair from the database according to the drone's ID, generates a timestamp and a random number, and combines this information with the selected challenge and uses the encryption strategy. The message is encrypted and sent to the drone. After receiving the message, the drone first decrypts and verifies the validity of the timestamp and whether the random number is reused. Then, it uses the embedded PUF hardware to generate a response based on the challenge received, and generates a new random number and timestamp. The information is encrypted using the strategy After encryption, it is transmitted back to the base station. After receiving it, the base station decrypts the authentication information, verifies the validity of the timestamp and the uniqueness of the random number, and verifies whether the received response matches the challenge-response pair stored in the database, thereby completing the identity authentication process. If the base station selects the ECC-based identity authentication strategy, the drone generates a random number and timestamp, which together with its own ID form the message n i , hash the message, using its own private key q i Sign and get signature i , the message n i With signature i Combining encryption strategies Encrypt and send to the base station together. The base station decrypts the received authentication information and i The identity ID in the file gets the corresponding public key p i Verify signature i ,Then the base station generates a random number and a timestamp, and together with its own BID, forms a message n0, hashes the message, and signs it with its own private key q0 to obtain a signature ξ0, and combines the message n0 with the signature ξ0 using the encryption strategy The two parties encrypt the received authentication information and send it to the drone. The drone decrypts the received authentication information and obtains the corresponding public key p0 according to the identity BID in the message n0 to verify the signature ξ0. If the verification is successful, the two parties can proceed with the subsequent conversation.

7. The low-overhead drone identity authentication method based on deep reinforcement learning according to claim 1, characterized in that: In step 11, according to the authentication delay Energy consumption The probability of the attacker successfully encrypting and decrypting Data protection level Calculate the reward at the current moment as follows: Among them, w T 、w Y 、w Q are weight coefficients, which measure the delay in executing the current task. Energy consumption And the probability of successful encryption and decryption by the attacker Importance in rewards.

8. The low-overhead drone identity authentication method based on deep reinforcement learning according to claim 1, characterized in that: In step 12, the ground base station sets the status Authentication strategy award Constructed as an experience sequence And store the experience sequence into the experience pool middle.

9. The low-overhead drone identity authentication method based on deep reinforcement learning according to claim 1, characterized in that: In step 13, the ground base station randomly samples Z experience {φ (z) } 1≤z≤k Form batch samples where z follows a uniform distribution from 1 to k.

10. A low-overhead drone identity authentication method based on deep reinforcement learning according to claim 1, characterized in that: In step 14, the ground base station uses the Adam optimization algorithm to update the weight parameters of the Q network Among them, β is the discount factor for weight parameter update.

Images