Unmanned aerial vehicle delivery bidirectional authentication method and device based on implicit gait behaviors

Abstract

The invention discloses an unmanned aerial vehicle delivery bidirectional authentication method and device based on implicit gait behaviors. A user places an order through a mobile phone delivery application program. After arriving at a designated place, the unmanned aerial vehicle hovers in the air, notifies a user of arrival of the unmanned aerial vehicle, and then establishes a key-protected communication channel with a mobile phone of the user. And the user holds the mobile phone and walks towards a specified position. And the unmanned aerial vehicle confirms the real-time position of the user through the shared GPS data, and searches the user by using a camera. Meanwhile, the unmanned aerial vehicle captures a user video and processes the user video in real time, and extracts a time sequence of user body key points; meanwhile, the mobile phone of the user continuously collects time series data from the IMU sensor. The two data are then exchanged between the drone and the user's mobile phone, thereby initiating a mutual authentication process. According to the invention, mutual identity verification is realized by using the hand movement mode when the user moves to the unmanned aerial vehicle, additional hardware or specific identity verification operation is not needed, and strong resistance to various spoofing attacks is shown.

Description

Technical Field

[0001] The present invention belongs to the field of information security technology, and relates to a method and device for two-way authentication of drone delivery, and specifically to a method and device for two-way authentication of drone delivery based on implicit gait behavior. Background Art

[0002] Drone delivery is the use of unmanned aerial vehicles (UAVs) to deliver and pick up packages. It has the advantages of being fast and efficient, cost-saving, low environmental impact, and improving logistics coverage, and has become a trend in the development of the logistics industry. Although drone delivery has brought many conveniences, its security and privacy protection issues in the delivery process still face severe challenges, especially how to effectively verify the identity of users and drones.

[0003] In response to the above technical issues, existing authentication solutions mostly focus on the following two directions:

[0004] (1) Feature-based authentication technology: This research direction is dedicated to analyzing and identifying the unique signal features emitted by the drone itself. These features can be the sound of the drone flying, the user's gait behavior, or other quantifiable feature indicators. Combined with modern deep learning algorithms, it can adapt to different environments and operating conditions, thereby improving the safety and reliability of drone delivery.

[0005] Chinese patent document number CN118447856A, published on August 6, 2024, describes a drone identity authentication method based on acoustic features. The invention collects sound samples from a specific microphone through a data acquisition module, and samples and evaluates them at preset time intervals. After data enhancement, the collected sound samples are divided into a training set and a test set. Using the feature extraction module, digital features are extracted from the sound samples to form a feature set. Then, an authentication model based on a CNN-Transformer hybrid network is constructed, the training set is trained to obtain the authentication model, and finally the test set is used for testing to achieve drone identity authentication. This method uses a deep learning structure to significantly improve the accuracy and reliability of drone identity authentication, and can effectively identify and analyze the sound signals emitted by drones. This technology has the following problems: First, the authentication method relies on a specific microphone for sound collection, which may cause the impact of environmental noise on data collection, thereby affecting the accuracy and reliability of acoustic features. In a complex environment, external interference may cause inconsistency in sound samples, thereby reducing the performance of the authentication model. Second, the authentication method requires effective authentication within a relatively close distance, which may limit its application in large areas or open spaces. The short authentication distance may make it unable to meet the needs in some practical scenarios, affecting the user experience and the breadth of application.

[0006] (2) Wireless communication authentication technology: The research direction is dedicated to exploring various wireless communication methods to ensure accurate verification of the drone's identity. This mainly includes Bluetooth signals, WIFI signals, and other detectable signals. The research focuses on the applicability of different communication protocols, combining multiple factors such as signal strength, transmission delay, and data packet integrity to meet the needs of different environments and application scenarios.

[0007] Chinese patent document number CN117295062A, published on December 26, 2023, records a remote identification system and identification method for drone identity ID based on Bluetooth 5.0. The invention collects data from authorized and unauthorized areas and writes drone authentication information into the remote ID device to perform regional judgment. In the unauthorized area, the second identity information is encrypted and broadcast using the first public key; in the authorized area, the third identity information is encrypted and broadcast using the authentication public key. The technology has the following problems: First, the authentication method relies on Bluetooth 5.0 technology for identity recognition, which is limited by the propagation characteristics of Bluetooth signals. In an environment with strong signal interference or many obstacles, the effective range of the Bluetooth signal may be reduced, thereby affecting the reliability and real-time performance of the authentication. Second, the authentication method and the authentication system need to combine multiple components (such as Bluetooth devices, area judgment modules, etc.), which will increase the overall complexity of the system. In addition, the cost of maintaining and updating these components may be high, affecting the sustainable development of the system.

[0008] At the same time, existing authentication schemes are mostly focused on one-way authentication of users. For example, one-way authentication technologies such as scanning the QR code on the user's mobile phone, one-time digital tokens, fingerprints, and facial recognition can be used to verify the identity of the recipient. There is a significant problem with these one-way authentication schemes: they only verify the identity of the user, ignoring the possibility that malicious drones may impersonate legitimate delivery drones to steal packages. In addition, attackers may use forgery or relay attacks to take over or interfere with delivery drones, resulting in hidden dangers to the security of the system. For example, in QR code, digital token, or biometric schemes, malicious drones can intercept the authentication data sent by users through relay attacks and replay them, resulting in user information leakage, package hijacking, etc. Summary of the invention

[0009] In order to solve the identity authentication problem in the drone delivery process and ensure the safety and reliability of delivery, a drone delivery two-way authentication method and device based on implicit gait behavior are provided.

[0010] The technical solution adopted by the method of the present invention is: a two-way authentication method for drone delivery based on implicit gait behavior, comprising the following steps:

[0011] Step 1: When the user and the drone perform two-way authentication, IMU sensor data and the user's gait data based on time series are collected;

[0012] Step 2: Normalize and denoise the IMU sensor data and time series-based gait data;

[0013] Step 3: Perform spatiotemporal consistency conversion on the IMU sensor data and time series-based gait data processed in step 2;

[0014] Step 4: Extract time domain and frequency domain features from the IMU sensor data and time series-based gait data processed in step 3;

[0015] Step 5: Input the time domain and frequency domain features into a single-class classifier to verify the consistency between the IMU sensor data and the hand key point data;

[0016] The mobile phone IMU sensor data is converted into Euler angle vectors and combined with accelerometer data to characterize the user's gait behavior data; then it is input into the feature extractor and single-class classifier for gait recognition and authentication;

[0017] Step 6: If both authentications are successful, mutual authentication is successful; otherwise, the drone determines whether it needs to adjust its position based on the distance from the user, returns to step 1, and repeats the process until the maximum number of attempts is reached; if authentication still fails, the user is rejected.

[0018] Preferably, in step 1, when the user performs two-way authentication with the drone, the user's mobile phone collects IMU sensor data; the drone camera records the user's gait information, performs human posture detection, identifies and matches the key points of the user's hand holding the mobile phone, and extracts the time series data of the key points, thereby converting the vision-based gait information into time series-based gait data.

[0019] Preferably, in step 2, the IMU sensor data is downsampled to match the sampling rate of the time series data of the key points.

[0020] Preferably, in step 2, an adaptive cutoff frequency filtering method is used to denoise the IMU sensor data and the time series-based gait data; first, a fast Fourier transform is used to analyze the spectral characteristics of the time series to determine the frequency interval where the main energy component is located, and then the high-pass frequency and low-pass frequency of the filter are set to the upper and lower limits of the frequency interval, respectively.

[0021] Preferably, in step 2, abnormal data caused by occlusion is adaptively filtered out by applying adaptive discrete cosine transform and multi-joint collaborative Kalman filtering;

[0022] The adaptive discrete cosine transform uses discrete cosine transform to eliminate signal anomalies caused by body occlusion; in order to dynamically adjust the retained discrete cosine transform coefficient K, an entropy-based adaptive adjustment method is used to calculate K. where f base is the initial retention coefficient, α is the adaptive adjustment parameter, H(X) is the information entropy value of gait data, and N is the length of gait data;

[0023] The multi-joint collaborative Kalman filter integrates the information of several adjacent key points into a state vector for the data processed by the adaptive discrete cosine transform, so as to realize the smooth processing of the motion trajectory of a single joint.

[0024] Preferably, in step 3, a clock synchronization protocol is run, and a Kalman filter is used to process fluctuations in network delay, estimate the changing trend of clock offset, and perform spatiotemporal consistency conversion on the IMU sensor data and the time series-based gait data after processing in step 2;

[0025] The specific implementation includes the following sub-steps:

[0026] Step 3.1: Fusion of user IMU sensor data (a x ,a y ,a z ,g x ,g y ,g z ,m x ,m y ,m z ), where a x ,a y ,a z ,g x ,g y ,g z ,m x ,m y ,m z Respectively represent the x, y, z coordinate data of the accelerometer, gyroscope and magnetometer in the mobile phone coordinate system. For the data at a certain time i, the x, y, z coordinate data of the accelerometer, gyroscope and magnetometer in the mobile phone coordinate system is recorded as a i =(a x,i ,a y,i ,a z,i ) T , g i =(g x,i ,g y,i ,g z,i ) T , m i =(m x,i ,m y,i ,m z,i )T Based on the above data, the quaternion q = (q0, q1, q2, q3) and the azimuth, pitch and roll angles (φ, θ, ψ) are obtained;

[0027] It includes the following sub-steps:

[0028] (1) Calculate the quaternion q1 at the initial moment;

[0029] The initial attitude angle is calculated from the accelerometer and magnetometer data:

[0030] These angles are then converted to the initial quaternion q1:

[0031]

[0032] (2) Normalization of accelerometer and magnetometer data and calculation of magnetic field direction;

[0033] For any time i, from the quaternion q at the previous time i-1 =(q i-1,0 ,q i-1,1 ,q i-1,2 ,q i-1,3 ) starts to calculate the current posture; first, the accelerometer data a at the current moment i =(a x,i ,a y,i ,a z,i ) and magnetometer data m i =(m x,i ,m y,i ,m z,i ) is normalized; through normalization, the acceleration vector per unit length is obtained Normalize the magnetometer data to obtain the magnetic field vector per unit length

[0034] (3) Using the normalized magnetometer data m i and the current quaternion q i-1 , calculate the direction of the Earth's magnetic field in the current attitude;

[0035] First, calculate the magnetic field direction vector by quaternion multiplication in, is the quaternion q i-1 The conjugate of Represents quaternion multiplication; the resulting vector h describes the component h of the magnetic field direction in the current quaternion posture h = (h1, h2, h3, h4); extract the horizontal component h of the magnetic field through h hor=(h2,h3) and the vertical component h4, and calculate the final magnetic field direction vector Vector b describes the direction of the Earth's magnetic field in its current state;

[0036] (4) Use the gradient descent algorithm to correct the quaternion; the correction vector F is calculated as follows:

[0037]

[0038] (5) Calculate the Jacobian matrix J to provide partial derivatives of each quaternion component with respect to the objective function, which is used to guide the quaternion correction process;

[0039]

[0040] (6) Calculate the correction step length step = J using the Jacobian matrix J and the correction vector F T F, further obtain the normalized correction step length

[0041] (7) Calculate the rate of change of the quaternion;

[0042] First, the quaternion change rate caused by the gyroscope data is Then calculate the total quaternion change rate Where β is a tuning parameter used to control the response speed of the algorithm;

[0043] (8) Integrate to obtain a new quaternion;

[0044] Using the sampling period SamplePeriod, calculate the update amount of the quaternion estimate in each iteration Further get the normalized new quaternion The final direction quaternion is q = (q0, q1, q2, q3);

[0045] (9) Using the new quaternion, the rotation angles, namely the azimuth, pitch, and roll angles, are calculated:

[0046]

[0047] Get the time series of rotation angles (φ, θ, ψ);

[0048] Step 3.2: Convert the acceleration vector a in the phone coordinate system phone =(a x ,a y ,a z ) are respectively multiplied by the direction quaternion q on the left and the quaternion conjugate q on the right -1 , complete the transformation to the world coordinate system: a world =q×aphone ×q -1 ;

[0049] Step 3.3: Calculate the hand movement speed based on the IMU sensor data; assuming that the initial speed at the initial moment is zero, calculate the hand movement speed v at time t world,t The formula is: where v world,t represents the speed of the hand at time t, a world,t represents the acceleration of the hand at time t, f s is the sampling rate of the accelerometer, i is the time step;

[0050] Step 3.4: Perform spatial consistency processing on the IMU sensor data; the main frequency components of the IMU sensor data include gravity change G, user walking component P and hand movement component H, speed v world Decompose into v world =v G +v P +v H ; Adopt adaptive cutoff filtering method to dynamically adjust the filtering threshold and eliminate v G and v P , and then normalize the speed of the key points of the hand to get the body coordinate system v H The hand speed under the H The velocity is compared with the corresponding hand keypoints in the body posture coordinate system to ensure the spatial consistency of the comparison data;

[0051] When processing the data collected by the drone, the position of the hand key points is mapped to a human-centered coordinate system, and the speed of the corresponding key points is calculated; after the hand key point position information is converted to a human-centered coordinate system, the speed of the hand key points is obtained by calculating the change in the hand key point position between two consecutive frames and dividing it by the time interval.

[0052] Preferably, in step 4, 12 features are initially selected from the time domain and frequency domain, and the Fisher scores of all features are calculated to screen out the most discriminative features; finally, four time domain features and two frequency domain features are selected, and the processed data samples form a six-dimensional feature vector.

[0053] Preferably, in step 5, the feature extractor is composed of a 7×7 convolutional layer with a single-channel input, a ResNet and a bidirectional LSTM network connected sequentially; the ResNet includes four residual blocks, each block contains two convolutional layers, a batch normalization layer is integrated after each convolutional layer to accelerate model convergence, and a ReLU activation function is applied to introduce nonlinearity; the output of the last residual block is converted into a feature vector of a fixed size through a global average pooling layer; the output of the ResNet is concatenated with the hidden state of the bidirectional LSTM network in the feature dimension, and finally a fully connected layer and a Softmax layer are added for multi-category classification.

[0054] Preferably, in step 5, the single-class classifier is a model trained by supervised learning; during the training process, a training data set containing labels is used, the training data set is composed of sensor data of different users, each sample is attached with a label, and is identified as a different user, so as to guide the model to learn how to distinguish users;

[0055] During the training process, features are extracted from the user's gait data, and the model is used to perform time-based analysis and feature extraction on the gait data. The extracted features are input into the supervised learning classifier for forward propagation to obtain the prediction results, and the user's label data is compared with the prediction results. The cross entropy loss function is used to optimize the model, and the gradient is calculated for backward propagation to gradually update the model weights to improve the model's recognition accuracy for "legitimate users" and "unauthorized users" samples.

[0056] After pre-training is completed, the transfer learning method is used to delete the supervised learning classifier of the model and retain the model weights of the forward layer as the feature extractor; in the actual user registration scenario, a small amount of unlabeled data is put into the feature extractor for feature extraction, and the extracted features are used to train a single-class classifier to complete the registration of the user authentication model.

[0057] The technical solution adopted by the device of the present invention is: a two-way authentication device for drone delivery based on implicit gait behavior, comprising:

[0058] one or more processors;

[0059] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the two-way authentication method for drone delivery based on implicit gait behavior.

[0060] Compared with the prior art, the beneficial effects of the present invention include:

[0061] 1. High accuracy: By utilizing the unique hand movement characteristics of the user's gait, the present invention can achieve efficient mutual authentication between the user and the drone, demonstrating extremely high accuracy and ensuring the effectiveness and security of authentication.

[0062] 2. Strong robustness: The present invention exhibits superior robustness under a variety of environmental conditions and can adapt to different lighting, shading and various equipment configurations, ensuring reliability in practical applications.

[0063] 3. Effectively resist attacks: The present invention can effectively resist various types of attacks, including wireless relay, device hijacking and imitation, ensure secure communication between users and drones, and improve the security of the overall system.

[0064] 4. User-friendliness: The design of the present invention reduces the user's intervention in the authentication process. The user only needs to walk towards the drone naturally, which improves the user experience and makes the authentication process smoother and more intuitive.

[0065] 5. Potential for wide application: Since the present invention does not rely on additional hardware and can achieve authentication at a relatively long distance, the present invention has broad application prospects and is suitable for a variety of drone logistics scenarios, promoting the safe development of the industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The technical solution of the present invention is further described below using embodiments and specific implementation methods. In addition, in the process of describing the technical solution, some drawings are also used. For those skilled in the art, other drawings and the intention of the present invention can also be obtained based on these drawings without paying creative work.

[0067] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0068] Figure 2 A single-class classifier training flow chart of an embodiment of the present invention;

[0069] Figure 3 Schematic diagram of ROC curves for consistency verification, IMU-based gait authentication, and overall performance of the method of the present invention in the experiment of this embodiment;

[0070] Figure 4 Schematic diagram of the effect of the hovering height and authentication distance of the drone on the consistency verification of the present invention in the experiment of this embodiment. DETAILED DESCRIPTION

[0071] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0072] In order to overcome the problems existing in the related art, the present embodiment provides a two-way authentication method and device for drone delivery based on implicit gait behavior. The user places an order through the delivery application on his / her mobile phone. After arriving at the designated location, the drone hovers in the air and notifies the user of its arrival through text messages, calls, etc., and then establishes a key-protected communication channel with the user's mobile phone. The user walks towards the designated location with his / her mobile phone in hand. The drone confirms the user's real-time location by sharing GPS data and searches for the user using a camera. During this process, the drone captures the user's video and processes it in real time to extract the time series of the user's body key points. At the same time, the user's mobile phone continuously collects time series data from its IMU sensor. Then, these two data are exchanged between the drone and the user's mobile phone, thereby initiating the mutual authentication process.

[0073] Please see Figure 1 , this embodiment provides a drone delivery bidirectional authentication method based on implicit gait behavior, comprising the following steps:

[0074] Step 1: When the user and the drone perform two-way authentication, the user's mobile phone collects IMU sensor data; the drone camera records the user's gait information, performs human posture detection, identifies and matches the key points of the user's hand holding the phone, and extracts the time series data of the key points, thereby converting the vision-based gait information into time series-based gait data;

[0075] The data from the IMU sensor includes time-series-based gait behavior information, such as data from the accelerometer, gyroscope, and magnetometer, while the video data records the user's visual gait information. By performing human posture detection on the video data, identifying and matching the key points of the user's hand holding the phone, and extracting the time series data of the key points, the visual-based gait information is converted into time-series-based gait information. The hand movement speed information of the user holding the phone is used as an important basis for verifying the consistency of the data on both sides.

[0076] During the registration phase, users need to hold their smartphones and walk forward at a normal pace according to their daily habits. The system collects the user's inertial measurement unit (IMU) data in real time. During this process, the smartphone's sensors will record the output data of the accelerometer, gyroscope, and magnetometer to obtain the user's gait characteristics. To ensure the comprehensiveness of the data, users need to complete data collection under different walking paths and speeds, such as walking in a straight line and making slight turns.

[0077] During the user authentication process, in addition to collecting IMU data, the system also needs to use the drone's camera to record the user's video to obtain the user's visual gait information. At this time, the IMU data and video data will be collected synchronously to form time series data for subsequent consistency verification. At this stage, a time synchronization protocol needs to be adopted, and a Kalman filter is used to solve the problem of network delay fluctuations to estimate the trend of clock offset changes. In the video acquisition segment, the YOLOv7 target detection algorithm needs to be used to detect the mobile phone and compare it with the user's center coordinates to obtain the left and right hand information of the user holding the mobile phone.

[0078] By extracting the time series data of the key points of the user's body, the system can convert visual gait information into spatiotemporal gait information, and use the hand speed holding the phone as an important indicator for consistency verification between IMU data and video data. The system can only retain the time series data of the key points of the user's body and discard the original video data, thereby protecting user privacy during data transmission.

[0079] Step 2: Normalize and denoise the IMU sensor data and time series-based gait data;

[0080] In one embodiment, a wavelet denoising algorithm is used to process the data to improve the accuracy of the data and minimize noise interference. Wavelet denoising can effectively filter out noise components that are not related to user movement, thereby improving the overall quality of time series data.

[0081] Considering that the video-derived key point timing data is sampled at 60 Hz, while the IMU data is sampled at 100 Hz, it is necessary to downsample the IMU data to match the sampling rate of the key point sequence and normalize the data separately.

[0082] Since users usually show about 1 to 3 gait cycles in one second, the corresponding main frequency range is 1 to 3 Hz. In one embodiment, a Butterworth filter is used for denoising. In order to optimize the filtering parameters, it is necessary to use a fast Fourier transform (FFT) to analyze the spectral characteristics of the time series data to determine the frequency range in which the main energy components are concentrated. Then, the high-pass and low-pass frequencies of the filter are set to the upper and lower limits of the range.

[0083] When the drone camera is at a high vertical angle or the horizontal angle is significantly offset, the user's hand may be occluded, resulting in inaccurate estimated key point positions. In one embodiment, these anomalies caused by occlusion can be adaptively filtered out by applying adaptive discrete cosine transform (ADCT) and multi-joint cooperative Kalman filter (MJCKF).

[0084] In the ADCT process, DCT is used to eliminate signal anomalies caused by body occlusion. In order to dynamically adjust the number of retained DCT coefficients K, in one embodiment, an entropy-based adaptive adjustment method is used. where f base is the initial retention coefficient, α is the adaptive adjustment parameter, H(X) is the information entropy value of gait data, and N is the length of the data.

[0085] Although ADCT processing effectively suppresses abnormal fluctuations in the key point position sequence, it may introduce signal distortion and edge effects. To alleviate this problem, in one embodiment, a multi-joint collaborative Kalman filter (MJCKF) method is used, which integrates the information of multiple adjacent key points into the state vector. In this way, not only the smoothing of the motion trajectory of a single joint is achieved, but also the coordination between multiple joints is retained. Using this collaborative filtering strategy, the stability and accuracy of the filter are significantly improved, the signal distortion introduced by ADCT is effectively reduced, and the consistency and accuracy of the joint motion data are ensured.

[0086] Step 3: Perform spatiotemporal consistency conversion on the IMU sensor data and time series-based gait data processed in step 2;

[0087] In order to ensure the time consistency between the key point data of the human body and the IMU data, in one implementation, a clock synchronization protocol is run, and a Kalman filter is combined to process the fluctuation of network delay, estimate the changing trend of the clock offset, and perform spatiotemporal consistency conversion on the IMU sensor data and the time series-based gait data after step 2 processing; this synchronization mechanism can maintain precise time consistency between devices and effectively avoid data mismatch problems caused by time deviation.

[0088] The motion information of the key points of the user captured by the drone camera is based on the coordinate system centered on the human body. The motion information measured by the IMU sensor is based on the coordinate system centered on the mobile phone. When the user holds the mobile phone and approaches the drone, the mobile phone will rotate continuously with the user's hand. The motion information directly measured by the IMU sensor will have a large deviation from the motion information of the key points of the user captured by the camera due to the continuous change of the mobile phone coordinate system. To eliminate this deviation, the IMU data must first be converted from the mobile phone coordinate system to the earth coordinate system. The specific implementation includes the following sub-steps:

[0089] Step 3.1: Fusion of user IMU sensor data (a x ,a y ,a z ,g x ,g y ,g z ,mx ,m y ,m z ), where a x ,a y ,a z ,g x ,g y ,g z ,m x ,m y ,m z Respectively represent the x, y, z coordinate data of the accelerometer, gyroscope and magnetometer in the mobile phone coordinate system. For the data at a certain time i, the x, y, z coordinate data of the accelerometer, gyroscope and magnetometer in the mobile phone coordinate system is recorded as a i =(a x,i ,a y,i ,a z,i ) T , g i =(g x,i ,g y,i ,g z,i ) T , m i =(m x,i ,m y,i ,m z,i ) T Based on the above data, the quaternion q = (q0, q1, q2, q3) and the azimuth, pitch and roll angles (φ, θ, ψ) are obtained;

[0090] It includes the following sub-steps:

[0091] (1) Calculate the quaternion q1 at the initial moment.

[0092] The initial attitude angle is calculated from the accelerometer and magnetometer data:

[0093] Then, these angles are converted into the initial quaternion q1, the specific expression is:

[0094]

[0095] (2) Normalization of accelerometer and magnetometer data and calculation of magnetic field direction. For any time i, the quaternion q at the previous time is i-1 =(q i-1,0 ,q i-1,1 ,q i-1,2 ,q i-1,3 ) starts to calculate the current posture. In order to ensure the accuracy of posture calculation, we first need to calculate the accelerometer data a at the current moment. i =(a x,i ,a y,i,a z,i ) and magnetometer data m i =(m x,i ,m y,i ,m z,i ) is normalized.

[0096] Through normalization, we get the acceleration vector per unit length Similarly, the magnetometer data is normalized to obtain the magnetic field vector per unit length The normalized acceleration vector and magnetic vector ensure the consistency and accuracy of the measured values ​​in each direction in the attitude calculation.

[0097] (3) Using the normalized magnetometer data m i and the current quaternion q i-1 , we can calculate the direction of the Earth's magnetic field in the current state. First, we calculate the magnetic field direction vector by quaternion multiplication. here, is the quaternion q i-1 The conjugate of Represents the multiplication of quaternions. The resulting vector h describes the components of the magnetic field direction in the current quaternion attitude h = (h1, h2, h3, h4).

[0098] Through h, extract the horizontal component h of the magnetic field hor =(h2,h3) and the vertical component h4, and calculate the final magnetic field direction vector This vector b describes the direction of the Earth's magnetic field at the current attitude and is used for subsequent attitude correction and update.

[0099] (4) After obtaining the initial quaternion and magnetic field direction, the gradient descent algorithm is used to correct the quaternion. The correction vector F is calculated as follows, which is used to adjust the estimated direction of the sensor to compensate for the deviation caused by sensor error or external interference:

[0100]

[0101] (5) Next, the Jacobian matrix J is calculated, which provides the partial derivative of each quaternion component with respect to the objective function and can effectively guide the quaternion correction process:

[0102]

[0103] (6) Calculate the correction step length step = J using the Jacobian matrix and the correction vector F T F, further obtain the normalized correction step length

[0104] (7) Calculate the rate of change of the quaternion. First, the rate of change of the quaternion caused by the gyroscope data is Then, the total quaternion change rate is Among them, β is a tuning parameter used to control the response speed of the algorithm. For the IMU sensor of a mobile phone, β is generally taken as 0.041.

[0105] (8) Integrate to get the new quaternion. The sampling period SamplePeriod determines the update amount of the quaternion estimate in each iteration. Further get the normalized new quaternion The final direction quaternion is q = (q0, q1, q2, q3).

[0106] (9) Through the new quaternion, the rotation angle, namely the azimuth, pitch and roll angle, can be calculated:

[0107]

[0108] This will give a time series of rotation angles (φ, θ, ψ).

[0109] Step 3.2: Convert the acceleration vector a in the phone coordinate system phone =(a x ,a y ,a z ) are respectively multiplied by the direction quaternion q on the left and the quaternion conjugate q on the right -1 , complete the transformation to the world coordinate system: a world =q×a phone ×q -1 ;

[0110] Step 3.3: Calculate the hand movement speed based on the IMU sensor data; assuming that the initial speed at the initial moment is zero, calculate the hand movement speed v at time t world,t The formula is: where v world,t represents the speed of the hand at time t, a world,t represents the acceleration of the hand at time t, f s is the sampling rate of the accelerometer, i is the time step;

[0111] Step 3.4: Perform spatial consistency processing on the IMU sensor data; the main frequency components of the IMU sensor data include gravity change G, user walking component P and hand movement component H, speed v world Decompose into v world =v G +v P +v H; Use the adaptive cutoff filtering method described in step 2 to dynamically adjust the filter threshold to eliminate v G and v P , and then normalize the speed of the key points of the hand to get the body coordinate system v H The hand speed under the H The velocity is compared with the corresponding hand keypoints in the body posture coordinate system to ensure the spatial consistency of the comparison data;

[0112] When processing the data collected by the drone, the position of the hand key points is mapped to a human-centered coordinate system, and the speed of the corresponding key points is calculated; after the hand key point position information is converted to a human-centered coordinate system, the speed of the hand key points is obtained by calculating the change in the hand key point position between two consecutive frames and dividing it by the time interval.

[0113] Step 4: Extract time domain and frequency domain features from the IMU sensor data and time series-based gait data processed in step 3;

[0114] In one embodiment, time domain and frequency domain features are extracted from the user's hand motion data, which are derived from IMU data and key point time series. 12 features are initially selected from the time domain and frequency domain ranges, and the Fisher scores of all features are calculated to screen out the most discriminative features. Finally, four time domain features and two frequency domain features are selected, and the processed data samples form a six-dimensional feature vector.

[0115] Step 5: Input the time domain and frequency domain features into a single-class classifier to verify the consistency between the IMU sensor data and the hand key point data;

[0116] The mobile phone IMU sensor data is converted into Euler angle vectors and combined with accelerometer data to characterize the user's gait behavior data; then it is input into the feature extractor and single-class classifier for gait recognition and authentication;

[0117] In one embodiment, for consistency verification between the drone and the user, after feature extraction is completed, in order to effectively verify the consistency between the IMU data and the hand key point data, the feature vector is input into a single-class classifier for model training. The OC-SVM (One-Class SVM) single classifier is used, and its hyperparameters are optimized through grid search. It should be emphasized that the consistency verification model only needs to be pre-trained once, and then it can be directly applied to unseen subjects for authentication.

[0118] For gait recognition authentication, the mobile phone IMU data is first converted into Euler angle vectors through quaternion calculation and combined with accelerometer data to characterize the user's gait behavior. Then, a deep learning-based feature extractor is constructed using transfer learning. The specific steps are to use the processed data to train the basic model as a feature extractor. After the pre-trained model is completed, the classification layer of the model is removed and a single-class classifier is connected. In the subsequent user authentication process, only the parameters of the single classifier need to be adjusted, and the pre-trained basic model remains unchanged.

[0119] In the process of gait feature extraction, accelerometer and gyroscope data are needed to extract the user's gait features. According to the accelerometer, gyroscope and magnetometer data, the quaternion q = (q0, q1, q2, q3) is calculated, and then the corresponding rotation angles can be calculated: roll angle (Roll, φ), pitch angle (Pitch, θ) and yaw angle (Yaw, ψ). The acceleration and rotation angle time series data are finally obtained as (a x ,a y ,a z ,φ,θ,ψ).

[0120] In one embodiment, a deep learning-based feature extractor is constructed using a transfer learning method. First, a basic model is pre-trained on gait representation data collected from different participants, and then the pre-trained model is adapted as a general feature extractor to capture the gait characteristics of individual users. A hybrid model architecture combining ResNet and LSTM networks is adopted, and the model design includes four residual blocks, each of which contains two convolutional layers. A batch normalization (BN) layer is integrated after each convolutional layer to accelerate model convergence, and a ReLU activation function is applied to introduce nonlinearity. The output of the last residual block is converted into a fixed-size feature vector through a global average pooling layer. In order to be compatible with the feature vector generated after feature selection, a 7×7 convolutional layer with a single channel input is added before the ResNet residual block. In addition, the present invention also designs a bidirectional LSTM network with 128 hidden units, and the input feature vector is reshaped into a three-dimensional tensor representing the batch size, time step, and feature dimension. In order to integrate the features of ResNet and LSTM, the output of ResNet is concatenated with the hidden state of LSTM in the feature dimension, and finally a fully connected layer and Softmax layer are added for multi-category classification.

[0121] Step 6: If both authentications are successful, mutual authentication is successful; otherwise, the drone determines whether it needs to adjust its position based on the distance from the user, returns to step 1, and repeats the process until the maximum number of attempts is reached; if authentication still fails, the user is rejected.

[0122] In one embodiment, in order to effectively verify the consistency between the IMU data and the hand key point data, the consistency feature vector extracted in step 5 is input into the OC-SVM classifier for decision making. At the same time, the gait features extracted by the feature extractor in step 5 are input into the OC-SVM classifier for user authentication. When the results of the consistency verification and the gait authentication both pass the classifier, it is determined that the user is a legitimate user; otherwise, the user is rejected.

[0123] Please see Figure 2 In one embodiment, the single-class classifier is a model trained by supervised learning. The model is specifically used to identify legitimate users and distinguish unauthorized users. The training process includes user label data and gait data.

[0124] During the training process, the present invention uses a training data set containing labels, which are composed of sensor data of different users. Each sample is attached with a label, and the label value is 0, 1, 2, 3, etc., which identifies different users respectively. These labels are used to guide the model to learn how to distinguish users.

[0125] The present invention uses supervised learning algorithms, especially convolutional neural network (CNN) and recurrent neural network (RNN) models based on deep learning. These models are trained with labeled data so that the models can recognize the motion patterns and characteristics of different users. During the training process, standard classification loss functions (such as cross entropy loss functions) are used to optimize the model to reduce the error between the predicted label and the actual label.

[0126] During the training process, features are extracted from the user's gait data, and models such as convolutional neural networks and recurrent neural networks are used to analyze and extract features from the gait data. The extracted features are input into the supervised learning classifier (MLP layer and softmax) for forward propagation to obtain the prediction results, and the user's label data is compared with the prediction results. The cross entropy loss function is used to optimize the model, and the gradient is calculated for back propagation, and the model weights are gradually updated to improve the model's recognition accuracy for "legitimate users" and "unauthorized users" samples.

[0127] After pre-training is completed, the present invention adopts the idea of ​​transfer learning, deletes the supervised learning classifier (MLP layer and softmax) of the model, and retains the model weights of the forward layer as a feature extractor. In a real user registration scenario, a small amount of unlabeled data is put into the feature extractor for feature extraction, and the extracted features are used to train a single-class classifier to complete the registration of the user authentication model. In this way, the present invention can successfully generate an independent identity authentication model for each user using a small number of positive samples, thereby achieving efficient user registration and authentication.

[0128] This embodiment also provides a drone delivery bidirectional authentication device based on implicit gait behavior, including:

[0129] one or more processors;

[0130] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the two-way authentication method for drone delivery based on implicit gait behavior.

[0131] The present invention is further described below through specific experiments.

[0132] After obtaining IRB approval, this experiment recruited 31 participants aged between 22 and 45 years old, including 20 males and 11 females, including undergraduates, graduate students, and faculty members. Before data collection, all participants received a detailed explanation of the purpose of the study and signed an informed consent. During the data collection process, participants held a smartphone and walked towards a fixed drone flight path. The drone hovered in front of the participants and collected video data in real time. The built-in sensors of the smartphone (accelerometer, gyroscope, etc.) continuously recorded sensor data related to the participant's movement. The drone captured the position of the participant's body key points through video, and processed the video through a real-time attitude estimation algorithm to extract the key point coordinates and generate time series data. At the same time, the smartphone continuously recorded IMU (inertial measurement unit) sensor data. In each data collection session, 5 pairs of IMU and KeyPoint time series data were extracted from the participant's smartphone and drone device. Finally, all session data constituted a dataset containing 65,650 pairs of time series data.

[0133] In this experiment, the following indicators are used to evaluate the performance of the system. The false acceptance rate (FAR) indicates the proportion of unauthorized user samples that are mistakenly identified as legitimate users, and is used to evaluate the system's resistance to attacks. The false rejection rate (FRR) indicates the proportion of legitimate user samples that are mistakenly rejected as unauthorized users, and is used to evaluate the availability of the system. The equal error rate (EER) refers to the value when FAR and FRR are equal. A lower EER value indicates that the reliability of the authentication system is higher. The balanced accuracy (BAC) refers to the average of the true acceptance rate (TAR) and the true rejection rate (TRR), where TAR indicates the proportion of legitimate user samples that are correctly identified, and TRR indicates the proportion of unauthorized user samples that are correctly identified. BAC is used to evaluate the performance of the model trained on imbalanced data. The receiver operating characteristic (ROC) curve is used to describe the relationship between TAR and FAR at different thresholds. The area under the ROC curve (AUC) is used as an indicator of the overall performance of the model. An AUC value close to 100% indicates that the system is more capable of distinguishing legitimate users from unauthorized users.

[0134] In order to evaluate the overall performance of the present invention, the experiment used the data set to train the model and evaluate its effectiveness. In one authentication attempt, the consistency verification module, the IMU-based gait authentication module, and the overall performance of the system of the present invention were tested respectively. The corresponding ROC curve is shown in Figure 2. Figure 3 As shown. The consistency verification module has an AUC of 99.83% and an EER of 0.54%, while the IMU-based gait verification module has an AUC of 99.98% and an EER of 0.26%. The overall performance of the present invention shows an AUC close to 100% and an EER of 0.09%. The ROC curve shows that both modules show a high TAR close to 100%, while the FAR is relatively suboptimal. These results show that the present invention exhibits higher performance in practical application scenarios.

[0135] In addition, the experiment studied the influence of the drone hovering height and the authentication distance on the consistency verification of the present invention. Figure 4 As shown in the figure, the effect of consistency verification is almost the same when the hovering height of the drone is 3m or 4m. As the hovering height further increases, the BAC shows a slight decrease. In addition, when the authentication distance exceeds 24 meters, the BAC of consistency verification begins to gradually decrease due to the reduced accuracy of locating the key points of the user's body. Therefore, the experiment recommends setting the hovering height of the drone to 4 meters and initiating mutual authentication at a distance of 24 meters.

[0136] The present invention proposes a method and device for two-way identity authentication of drones based on implicit gait behavior. When a user walks towards a drone holding a mobile phone, the built-in inertial sensor of the smartphone and the drone camera are used to simultaneously collect the user's hand posture change data to achieve two-way authentication between the user and the drone. Compared with the prior art, this method does not require additional hardware support, can achieve efficient authentication in a long-distance environment of more than 18 meters, and does not require additional authentication actions, with a friendly user experience. Gait is a unique dynamic feature determined by the human skeletal structure and behavioral habits, which is difficult to forge and imitate. Inertial sensors commonly integrated in smartphones (such as accelerometers and gyroscopes) can capture gait data efficiently and at low cost. Using implicit gait features for two-way authentication of drone delivery is not only convenient and reliable, but also can effectively resist imitation attacks and device hijacking attacks.

[0137] The present invention improves the security of the drone delivery system, reduces potential risks caused by insufficient identity authentication, and provides users with a more convenient and secure delivery experience.

[0138] It should be understood that the embodiments described above are part of the embodiments of the present invention, rather than all of the embodiments. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be combined with each other arbitrarily to form a feasible technical solution. Such combination is not restricted by the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the protection scope required by the present invention.

[0139] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A two-way authentication method for drone delivery based on implicit gait behavior, characterized in that: The following steps are involved: Step 1: When the user and the drone perform two-way authentication, IMU sensor data and the user's gait data based on time series are collected; Step 2: Normalize and denoise the IMU sensor data and time series-based gait data; Step 3: Perform spatiotemporal consistency conversion on the IMU sensor data and time series-based gait data processed in step 2; Step 4: Extract time domain and frequency domain features from the IMU sensor data and time series-based gait data processed in step 3; Step 5: Input the time domain and frequency domain features into a single-class classifier to verify the consistency between the IMU sensor data and the hand key point data; The mobile phone IMU sensor data is converted into Euler angle vectors and combined with accelerometer data to characterize the user's gait behavior data; then it is input into the feature extractor and single-class classifier for gait recognition and authentication; Step 6: If both authentications are successful, mutual authentication is successful; otherwise, the drone determines whether it needs to adjust its position based on the distance from the user, returns to step 1, and repeats the process until the maximum number of attempts is reached; if authentication still fails, the user is rejected.

2. The method for two-way authentication of drone delivery based on implicit gait behavior according to claim 1 is characterized in that: In step 1, when the user performs two-way authentication with the drone, the user's mobile phone collects IMU sensor data; the drone camera records the user's gait information, performs human posture detection, identifies and matches the key points of the user's hand holding the mobile phone, and extracts the time series data of the key points, thereby converting the vision-based gait information into time series-based gait data.

3. The method for two-way authentication of drone delivery based on implicit gait behavior according to claim 1 is characterized in that: In step 2, the IMU sensor data is downsampled to match the sampling rate of the time series data of the key points.

4. The method for two-way authentication of drone delivery based on implicit gait behavior according to claim 1 is characterized in that: In step 2, the IMU sensor data and the time series-based gait data are denoised using an adaptive cutoff frequency filtering method. First, the spectral characteristics of the time series are analyzed using a fast Fourier transform to determine the frequency interval where the main energy component is located. Then, the high-pass frequency and low-pass frequency of the filter are set to the upper and lower limits of the frequency interval, respectively.

5. The method for two-way authentication of drone delivery based on implicit gait behavior according to claim 1 is characterized in that: In step 2, the abnormal data caused by occlusion is adaptively filtered out by applying adaptive discrete cosine transform and multi-joint collaborative Kalman filter; The adaptive discrete cosine transform uses discrete cosine transform to eliminate signal anomalies caused by body occlusion; in order to dynamically adjust the retained discrete cosine transform coefficient K, an entropy-based adaptive adjustment method is used to calculate K. where d base is the initial retention coefficient, α is the adaptive adjustment parameter, H(X) is the information entropy value of gait data, and N is the length of gait data; The multi-joint collaborative Kalman filter integrates the information of several adjacent key points into a state vector for the data processed by the adaptive discrete cosine transform, so as to realize the smooth processing of the motion trajectory of a single joint.

6. The method for two-way authentication of drone delivery based on implicit gait behavior according to claim 1 is characterized in that: In step 3, the clock synchronization protocol is run, and the Kalman filter is used to process the fluctuation of network delay, estimate the changing trend of clock offset, and perform spatiotemporal consistency conversion on the IMU sensor data and time series-based gait data processed in step 2; The specific implementation includes the following sub-steps: Step 3.1: Fusion of user IMU sensor data (a x ,a y ,a z ,g x ,g y ,g z ,m x ,m y ,m z ), where a x ,a y ,a z ,g x ,g y ,g z ,m x ,m y ,m z Respectively represent the x, y, z coordinate data of the accelerometer, gyroscope and magnetometer in the mobile phone coordinate system. For the data at a certain time i, the x, y, z coordinate data of the accelerometer, gyroscope and magnetometer in the mobile phone coordinate system is recorded as a i =(a x,i ,a y,i ,a z,i ) T , g i =(g x,i ,g y,i ,g z,i ) T , m i =(m x,i ,m y,i ,m z,i ) T Based on the above data, the quaternion q = (q0, q1, q2, q3) and the azimuth, pitch and roll angles (φ, θ, ψ) are obtained; It includes the following sub-steps: (1) Calculate the quaternion q1 at the initial moment; The initial attitude angle is calculated from the accelerometer and magnetometer data: These angles are then converted to the initial quaternion q1: (2) Normalization of accelerometer and magnetometer data and calculation of magnetic field direction; For any time i, from the quaternion q at the previous time i-1 =(q i-1,0 ,q i-1,1 ,q i-1,2 ,q i-1,3 ) starts to calculate the current posture; first, the accelerometer data a at the current moment i =(a x,i ,a y,i ,a z,i ) and magnetometer data m i =(m x,i ,m y,i ,m z,i ) is normalized; through normalization, the acceleration vector per unit length is obtained Normalize the magnetometer data to obtain the magnetic field vector per unit length (3) Using the normalized magnetometer data m i and the current quaternion q i-1 , calculate the direction of the Earth's magnetic field in the current attitude; First, calculate the magnetic field direction vector by quaternion multiplication in, is the quaternion q i-1 The conjugate of Represents quaternion multiplication; the resulting vector h describes the component h of the magnetic field direction in the current quaternion posture h = (h1, h2, h3, h4); extract the horizontal component h of the magnetic field through h hor =(h2,h3) and the vertical component h4, and calculate the final magnetic field direction vector Vector b describes the direction of the Earth's magnetic field in its current state; (4) Use the gradient descent algorithm to correct the quaternion; the correction vector F is calculated as follows: (5) Calculate the Jacobian matrix J to provide partial derivatives of each quaternion component with respect to the objective function, which is used to guide the quaternion correction process; (6) Calculate the correction step length step = J using the Jacobian matrix J and the correction vector F T F, further obtain the normalized correction step length (7) Calculate the rate of change of the quaternion; First, the quaternion change rate caused by the gyroscope data is Then calculate the total quaternion change rate Where β is a tuning parameter used to control the response speed of the algorithm; (8) Integrate to obtain a new quaternion; Using the sampling period SamplePeriod, calculate the update amount of the quaternion estimate in each iteration Further get the normalized new quaternion The final direction quaternion is q = (q0, q1, q2, q3); (9) Using the new quaternion, the rotation angles, namely the azimuth, pitch, and roll angles, are calculated: Get the time series of rotation angles (φ, θ, ψ); Step 3.2: Convert the acceleration vector a in the phone coordinate system phone =(a x ,a y ,a z ) are respectively multiplied by the direction quaternion q on the left and the quaternion conjugate q on the right -1 , complete the transformation to the world coordinate system: a world =q×a phone ×q -1 ; Step 3.3: Calculate the hand movement speed based on the IMU sensor data; assuming that the initial speed at the initial moment is zero, calculate the hand movement speed v at time t world,t The formula is: where v world,t represents the speed of the hand at time t, a world,t represents the acceleration of the hand at time t, f s is the sampling rate of the accelerometer, i is the time step; Step 3.4: Perform spatial consistency processing on the IMU sensor data; the main frequency components of the IMU sensor data include gravity change G, user walking component P and hand movement component H, speed v world Decompose into v world =v G +v P +v H ; Adopt adaptive cutoff filtering method to dynamically adjust the filtering threshold and eliminate v G and v P , and then normalize the speed of the key points of the hand to get the body coordinate system v H The hand speed under the H The velocity is compared with the corresponding hand key points in the body posture coordinate system to ensure the spatial consistency of the comparison data; When processing the data collected by the drone, the position of the hand key points is mapped to a human-centered coordinate system, and the speed of the corresponding key points is calculated; after the hand key point position information is converted to a human-centered coordinate system, the speed of the hand key points is obtained by calculating the change in the hand key point position between two consecutive frames and dividing it by the time interval.

7. The method for two-way authentication of drone delivery based on implicit gait behavior according to claim 1 is characterized in that: In step 4, 12 features were initially selected from the time domain and frequency domain, and the Fisher scores of all features were calculated to screen out the most discriminative features; finally, four time domain features and two frequency domain features were selected, and the processed data samples formed a six-dimensional feature vector.

8. The method for two-way authentication of drone delivery based on implicit gait behavior according to claim 1 is characterized in that: In step 5, the feature extractor is composed of a 7×7 convolutional layer with a single channel input, a ResNet, and a bidirectional LSTM network connected sequentially; the ResNet includes four residual blocks, each of which contains two convolutional layers, and a batch normalization layer is integrated after each convolutional layer to accelerate model convergence, and a ReLU activation function is applied to introduce nonlinearity; the output of the last residual block is converted into a feature vector of a fixed size through a global average pooling layer; the output of the ResNet is concatenated with the hidden state of the bidirectional LSTM network in the feature dimension, and finally a fully connected layer and a Softmax layer are added for multi-category classification.

9. The method for two-way authentication of drone delivery based on implicit gait behavior according to any one of claims 1 to 8, characterized in that: In step 5, the single-class classifier is a model trained through supervised learning. During the training process, a training data set containing labels is used. The training data set consists of sensor data of different users. Each sample is labeled and identified as a different user, which is used to guide the model to learn how to distinguish users. During the training process, features are extracted from the user's gait data, and the model is used to perform time-based analysis and feature extraction on the gait data. The extracted features are input into the supervised learning classifier for forward propagation to obtain the prediction results, and the user's label data is compared with the prediction results. The cross entropy loss function is used to optimize the model, and the gradient is calculated for backward propagation to gradually update the model weights to improve the model's recognition accuracy for "legitimate users" and "unauthorized users" samples. After pre-training is completed, the transfer learning method is used to delete the supervised learning classifier of the model and retain the model weights of the forward layer as the feature extractor; in the actual user registration scenario, a small amount of unlabeled data is put into the feature extractor for feature extraction, and the extracted features are used to train a single-class classifier to complete the registration of the user authentication model.

10. A two-way authentication device for drone delivery based on implicit gait behavior, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the two-way authentication method for drone delivery based on implicit gait behavior as described in any one of claims 1 to 9.

Images