A method for controlling drones using eye gestures based on XR devices

Abstract

The present invention discloses an eye-gesture drone control method based on an XR device, specifically including: S1 establishing real-time communication between the XR device and the drone; S2 gaze point prediction; S3 target waypoint determination; S4 gesture command recognition; S5 flight path confirmation; S6 flight control. The present invention completely replaces the traditional drone remote controller, and directly integrates the acquisition of the drone's field of view image, the communication and control of the flight intention into the head-mounted XR device, especially changing the communication of the flight intention from the indirect communication of the traditional remote controller joystick to the active acquisition of the XR device through the eye-movement recognition algorithm, avoiding the possible misoperation caused by the remote controller control, and realizing a safer, more efficient, intelligent and portable drone flight control.

Description

Technical Field

[0001] The present invention belongs to the technical field of drone control, and in particular relates to an eye-gesture drone control method based on an XR device. Background Art

[0002] With the development of drone technology, drones are playing an increasingly important role in the civil and military fields, such as drone spraying of pesticides, drone food delivery, drone photography, drone reconnaissance, etc. However, traditional drone operations have certain technical barriers, and safety in complex environments is difficult to guarantee, so professions such as "professional pilots" have been born. How to reduce the difficulty of drone operation and improve the safety of flight missions has become a prominent problem that needs to be solved urgently. At the same time, with the development of technologies such as virtual reality (VR), augmented reality (XR), and mixed reality (MR), the concept of "metaverse" technology has become a hot topic at present. How to use wearable XR devices to achieve portable and immersive human-computer interaction control has great application value and application prospects.

[0003] Figure 1 An existing drone remote controller is shown, which includes a dual joystick and a display screen. The dual joystick can realize eight degrees of freedom direction control; the display screen can display the image from the drone camera. The traditional drone control method is to directly observe the drone position by human eyes or indirectly judge the drone position through the image sent back to the remote controller by the drone camera, and then control the speed, horizontal yaw angle, height, and horizontal position of the drone to fly to the target position through the remote controller. The traditional drone control method has at least the following problems: First, the difficulty of operation. When controlling the drone, the drone operator must simultaneously realize the precise control of the dual joysticks, and make the drone fly to the target position according to the imagined trajectory by comprehensively adjusting the speed, yaw angle, height, and horizontal position of the drone. The high sensitivity of the joystick, the high speed of the drone, and the poor observation angle of the operator make this operation very unfriendly to non-professional operators. Second, the safety issue. In a complex flight environment, the human eye cannot directly observe the environment around the drone, and indirect observation through the drone camera angle is limited by the drone camera angle. Even if the drone camera has captured possible obstacles in the future, it may be ignored by the operator and cause danger. Of course, even if the risk is accurately predicted, it may still occur due to improper operation or misjudgment. The above mentioned situations make the remote control drone flight in complex environment extremely unsafe. The third is the portability of the drone handle and display device.

[0004] With the development of wearable device technology and the development of XR, VR, and MR technologies, how to present drone camera images directly in front of people's eyes and realize immersive portable operation without controllers has become a major demand at present. At the same time, a large number of studies on the correlation between drone control and the operator's eye gaze point show that the drone's flight trajectory is highly correlated with the operator's gaze point during the operation process. When controlling the drone, the operator always looks at the drone's future flight waypoint, and the waypoint is always in front of the drone. An important reason why traditional drone control operations are difficult and unsafe is that after the human eye obtains the future flight waypoint, the flight intention to reach the future flight waypoint needs to be accurately expressed through the remote control, which is obviously difficult and risky. If we can directly convey the future flight waypoint information obtained by the human eye to the drone, so that the drone can autonomously navigate to this waypoint through precise path planning, the difficulty and risk will obviously be greatly reduced. Summary of the invention

[0005] In view of the problems that existing drone control operations are difficult, unsafe, and not convenient enough, the present invention provides a method for controlling a drone using eye gestures based on an XR device.

[0006] The specific technical solution is:

[0007] An eye-gesture drone control method based on an XR device comprises the following steps:

[0008] S1, establish real-time communication between the XR device and the drone; S2, gaze point prediction; the drone transmits the collected first-person perspective image back to the XR device in real time and displays it, and uses the deep learning method of pure eye diagram to predict the two-dimensional gaze point coordinates of the operator's eyes in the image (G x ,G y ); S3: Target waypoint determination; Based on the two-dimensional gaze point coordinates (G x ,G y ), combined with the depth information to determine the three-dimensional coordinates of the target waypoint (G x ,G y ,G z ), Where D represents the maximum depth distance, which can be determined according to the application scenario; v max Indicates the maximum speed of the drone, v minIndicates the minimum speed of the drone, and v indicates the current speed of the drone; S4: gesture command recognition; the XR device obtains the operator's hand image through its own front camera, recognizes the corresponding gesture based on the gesture recognition algorithm, and converts the recognized gesture into the corresponding flight control command; S5: flight path confirmation; the XR device sends the three-dimensional coordinates of the target waypoint determined in step S3 and the flight control command of step S4 to the drone in real time, and uses the line connecting the current position of the drone and the target waypoint as the current flight path of the drone; determines whether there are obstacles on the current flight path, and if there are no obstacles, confirms the current flight path; otherwise, the drone sends a warning to the XR device and returns to step S2; S6: flight control; the drone performs flight missions based on dynamically planned paths and flight control commands.

[0009] Furthermore, the determination of whether there is an obstacle in the current flight path is specifically that the UAV's laser radar or depth camera obtains obstacle information within a depth range H and a width range W in the direction of the UAV's current flight path, where H = k*G z , k is the safety factor, indicating the safety tolerance of the flight path, and W is the width of the UAV body; if there is obstacle information, it is determined that there is an obstacle in the current flight path, otherwise, it is determined that there is no obstacle in the current flight path.

[0010] Furthermore, step S2 also includes a step of de-jittering the gaze point prediction result, and smoothing filtering is performed on the gaze point prediction result. The result after smoothing filtering is: in Meets the inequality In the inequality, represents the x-coordinate of the predicted gaze point, represents the y coordinate of the predicted fixation point, m represents the size of the smoothing filter window; in the result expression after smoothing filtering, I represents the predicted fixation point The set that meets the inequality, n represents the number of predicted fixation points that meet the I set.

[0011] Furthermore, the deep learning method of pure eye diagram in step S2 specifically includes the construction of eye movement data set, the establishment of eye movement tracking model, the training of eye movement tracking model and the deployment steps of eye movement tracking model; wherein the eye movement tracking model is a neural network model, including convolution layer, activation function and fully connected layer, the convolution layer is used to extract image features related to sight line from the eye image, the activation function is used to perform nonlinear mapping on the result of convolution layer, and the fully connected layer is used to fit all the extracted features to obtain two-dimensional sight line coordinates, i.e., two-dimensional gaze point coordinates.

[0012] Furthermore, the construction of the eye movement dataset is specifically as follows: a two-dimensional display plane of a certain size is designed in the XR device, more than 200 points are evenly marked on the plane and the two-dimensional coordinates of the points are recorded, and eye images of multiple subjects when they view all the points are collected through the near-eye camera of the XR device.

[0013] Furthermore, the real-time communication includes local area network, 4G, 5G and Wi-Fi wireless communication.

[0014] Furthermore, the gesture recognition algorithm is an IMU-based gesture recognition method.

[0015] Furthermore, the gesture recognition algorithm is a gesture recognition method based on RGB images, which adopts a neural network model, including firstly realizing the positioning and segmentation of the hand image through the Yolo algorithm, further extracting the hand image features through the convolution layer, and then realizing the nonlinear mapping of gesture category to recognition probability through the fully connected layer and the softmax activation function. The gesture category corresponding to the maximum recognition probability is the recognized gesture.

[0016] Furthermore, the XR device collects hand images in real time through a front camera equipped thereon.

[0017] Furthermore, gesture commands are only used to change the current flight speed of the drone and start and stop operations. When step S4 does not detect any gesture or step S5 does not receive a new flight control command, the drone is controlled solely by the operator's eyes at the current speed or default speed.

[0018] This method uses an eye tracking algorithm to estimate in real time the gaze coordinates of the human eye on the first-person perspective image sent back by the drone, and then converts the coordinate information into three-dimensional spatial coordinates in the drone body coordinate system, which is the flight target waypoint of the drone. At the same time, gesture recognition technology is used to realize the take-off and landing control and speed control of the drone, establish the flight path between the current position and the target waypoint, and realize the flight control of the drone based on gesture commands and the flight path.

[0019] Compared with traditional drone control methods, it has the following advantages:

[0020] 1. The operator's flight intention is directly captured through human eye gaze information, avoiding the misexpression of flight intention caused by improper operation of the remote control, and realizing faster, more accurate and safer execution of the UAV operator's flight intention.

[0021] 2. There is no need for manual remote control to adjust the height, yaw angle, and horizontal position of the drone, which greatly reduces the learning cost of drone operation and greatly improves the safety of drone flight.

[0022] 3. It completely replaces the traditional drone remote control, and directly integrates the acquisition of drone vision, communication of flight intentions, speed and start-stop control into the head-mounted XR device, realizing portable and immersive drone intelligent control. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Schematic diagram of existing drone remote controller;

[0024] Figure 2 Schematic diagram of the drone control principle based on XR equipment in the present invention;

[0025] Figure 3 It is a schematic diagram of the composition of the UAV control system of the present invention;

[0026] Figure 4 It is an XR glasses with a near-eye camera and a front camera in one embodiment of the present invention;

[0027] Figure 5 is an eye tracking algorithm model diagram in one embodiment of the present invention;

[0028] Figure 6 is a schematic diagram of a command gesture in one embodiment of the present invention;

[0029] Figure 7 It is a gesture recognition algorithm model diagram in one embodiment of the present invention. DETAILED DESCRIPTION

[0030] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0031] First, some nouns or terms that appear in the process of describing the embodiments of the present invention are explained as follows.

[0032] Deep learning: Deep learning is the process of learning the inherent laws and representation levels of sample data. The information obtained in the learning process is very helpful for interpreting data such as text, images and sounds. Its ultimate goal is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images and sounds.

[0033] Convolutional Neural Networks (CNN): A neural network system based on convolution operations is a way of deep learning. Convolutional neural networks mainly include two functions, one is feature extraction (convolution, activation function, pooling), and the other is classification recognition (fully connected layer). Among them, convolution: extract different features of the input; activation function: introduce nonlinear factors and perform nonlinear mapping on the results of the convolution layer; pooling: reduce the input image and reduce the amount of calculation; fully connected layers: fully connected layers play the role of "classifier" in the entire convolutional neural network, that is, after passing through deep networks such as convolution, activation function, pooling, etc., the results are identified and classified through fully connected layers. Each layer in the fully connected layer is a tiled structure composed of many neurons, and its core operation is the matrix-vector product y=Wx, which linearly transforms the signal from one feature space to another feature space.

[0034] PyTorch: PyTorch is an open source Python machine learning library based on Torch for applications such as natural language processing.

[0035] Aircraft coordinate system: The aircraft coordinate system is fixed to the aircraft, and the coordinate system conforms to the right-hand rule. The origin of the coordinate system is always at the center of gravity of the aircraft, the X-axis points to the direction of the aircraft nose, the Y-axis points to the right side of the aircraft, and the Z-axis direction is determined by the right-hand rule through X and Y.

[0036] Figure 2 The following is a schematic diagram of the drone control principle based on the XR device of the present invention. The drone operator wears the XR device, and the drone's first-person perspective is displayed in the XR device. The operator controls the real-time flight waypoints of the drone through eye gaze behavior. At the same time, when taking off, landing, or adjusting the speed, the operator places one hand in a position where the front camera of the XR device can shoot, and controls the start and stop of the drone or speed adjustment through gestures.

[0037] Figure 3It is a schematic diagram of the control system of the present invention. The UAV control system can be divided into a first control unit (i.e., the operator subsystem) and a second control unit (UAV subsystem). The first control unit includes an eye tracker and a gesture recognizer. The eye tracker obtains the first-person-view image of the UAV from the onboard camera of the UAV, and obtains the gaze point coordinates of the human eye gaze point on the first-person-view image of the UAV through an eye tracking algorithm; the gesture recognizer captures the hand image and recognizes the gesture command through a gesture recognition algorithm. More specifically, the first control unit here is mounted on an XR device, and the specific form of the XR device can be XR glasses. The second control unit includes a controller and a flight path confirmation module. The controller transmits the first-person-view image of the UAV back to the eye tracker, and obtains the operator's gaze point coordinates and gesture commands from the eye tracker and gesture recognizer. The flight path confirmation module converts the two-dimensional gaze point coordinates into the three-dimensional waypoint coordinates of the operator's intended flight, generates a safe flight path, and finally the controller controls the UAV to complete the flight mission. More specifically, it includes the following steps:

[0038] S1: Establish real-time communication between XR devices and drones;

[0039] The communication includes the drone transmitting the images captured by the camera back to the XR glasses in real time, and the XR glasses sending the gaze point coordinate information and recognized gesture commands to the drone in real time. The remote communication method can be used according to different usage environments, including but not limited to LAN, 4G, 5G and Wi-Fi. In a specific embodiment, 2.4G band Wi-Fi is used to achieve real-time communication between the drone and the XR device.

[0040] S2: fixation prediction;

[0041] The gaze point information in the first-person perspective of the drone contains the flight intention information of the drone operator, that is, the future flight waypoint information. This information combined with the actual spatial information of the drone can obtain the future flight waypoints of the drone to guide the flight of the drone. Specifically, the drone transmits the collected first-person perspective image back to the XR device in real time and displays it, using methods including but not limited to eyeball model-based methods, infrared calibration-based methods, and pure eye map-based deep learning methods to predict the two-dimensional gaze point coordinates of the operator's eyes in the image.

[0042] This embodiment takes a deep learning method based on a pure eye diagram as an example for explanation, and is specifically described as follows:

[0043] The implementation of the gaze point prediction algorithm based on the deep learning method of pure eye map includes the construction of eye movement dataset, the establishment of eye movement tracking model, the training of model and the deployment steps of model.

[0044] S21: Construction of eye movement dataset. Figure 4An XR glasses equipped with a near-eye camera is shown, through which the near-eye camera on the glasses can collect the operator's near-eye eye diagram, and through the gaze point guidance in the XR glasses display interface, the operator's binocular eye diagram under each coordinate label is collected, so that the eye movement dataset with gaze point coordinate labels can be constructed. For example, a two-dimensional display plane of a certain size is designed in the head-mounted XR device, more than 200 points are evenly marked on the plane and the two-dimensional coordinates of the points are recorded, and the eye images of the subjects when viewing all points are collected through the near-eye camera of the XR device. The number of images collected for each point for each subject is not less than 10, and the number of subjects is not less than 20. In addition, the dataset can also use the near-eye dataset that has been made public in the industry, such as the NVGaze near-eye image dataset released by NVIDIA in 2019.

[0045] S22: Eye tracking model establishment. As an implementation method, Figure 5 A feasible eye tracking model is shown. The eye tracking model is a neural network model, including a convolution layer, an activation function, and a fully connected layer. The convolution layer is used to extract image features related to gaze from the eye image, the activation function is used to perform nonlinear mapping on the result of convolution, and the fully connected layer is used to fit all the extracted features to obtain two-dimensional gaze coordinates. The input of the model is the binocular eye image, which is sent to the CNN convolution layer respectively to extract eye features related to gaze, and then the extracted eye feature information is flattened and sent to the fully connected layer. The fully connected layer is used to perform high-dimensional linear fitting on the features to obtain the predicted gaze coordinates.

[0046] S23: Model training. The model training can be done by using PyTorch, TensorFlow or Caffe deep learning framework to build the model, using the N-fold cross-validation method for training, and taking the best cross-validation model as the final model parameters.

[0047] S24: Model deployment. As an embodiment, the open source NCNN forward computing framework can be used to implement the deployment on a mobile device. More generally, frameworks such as PyTorch Mobile, TensorFlow Lite, and Caffe2 can also be used to implement the deployment of the neural network on a mobile device.

[0048] S3: Target waypoint is determined.

[0049] Gaze point prediction is the prediction of the two-dimensional gaze point coordinates of the human eye on the XR display interface (i.e. the first-person perspective of the drone). Only by converting these two-dimensional gaze point coordinates into three-dimensional coordinates in the drone body coordinate system can we obtain accurate target flight waypoints and realize eye control of the drone. Specifically, the determination of the target waypoint requires solving the following two problems: one is the de-jittering of the gaze point prediction result; the other is the determination of the third-dimensional depth information.

[0050] S31: De-jittering the gaze point prediction result. Since the human eye has certain eye gaze and blinking phenomena, this will cause the gaze point prediction result to have a large range of erroneous jumps. In order to solve this gaze point jitter problem, the following formula shows a smoothing filter algorithm for smoothing the gaze point prediction result.

[0051]

[0052]

[0053]

[0054]

[0055]

[0056]

[0057] The result after smoothing filtering is G = (G x ,G y ),in represents the x-coordinate of the predicted point, represents the y coordinate of the predicted point, m represents the size of the smoothing filter window, which is selected according to the frame rate of the camera device. When the frame rate is 60HZ, m=10 can be selected. I represents the predicted gaze point The set that satisfies the inequality is:

[0058]

[0059] n represents the number of prediction points that meet the I set.

[0060] S31: Depth information determination. We define the smoothed predicted gaze point as G = (G x ,G y ), the depth is G z , then the three-dimensional waypoint coordinates are P = (G x ,G y ,G z ),

[0061]

[0062] Where D represents the maximum depth distance, which can be determined according to the actual application scenario, for example, 10m; v max Indicates the maximum speed of the drone, v min Indicates the minimum speed of the drone, and v indicates the current speed of the drone. When the drone flies at the maximum speed, the waypoint depth takes the maximum value D; when the drone hovers at a speed of 0, the drone depth takes the minimum value 0.

[0063] S4: Gesture command recognition. Gesture commands are mainly used to control the speed and start and stop of the drone. The XR device obtains the operator's hand image through its own front camera, recognizes the corresponding gesture based on the gesture recognition algorithm, and converts the recognized gesture into the corresponding flight control command. Gesture recognition can include but is not limited to gesture recognition based on inertial measurement unit (IMU) and gesture recognition based on RGB image.

[0064] The present invention is described by taking the gesture recognition technology based on RGB images as an example. Figure 2 The example diagram shows the specific operation method of the gesture recognition, which is to extend the right hand in front of the XR glasses so that the front camera of the XR glasses can capture the hand. The right hand only needs to make specified movements to be recognized by the XR glasses. Figure 4 An XR glasses with a front-facing camera is shown. Figure 6 An optional implementation is shown, namely, five gesture instructions for the right hand, which are described in detail as follows: (1) deceleration: four fingers make a fist, thumb to the left; (2) hovering: four fingers make a fist, thumb up; (3) acceleration: four fingers make a fist, thumb to the right; (4) takeoff: five fingers open, palm facing forward; (5) landing: five fingers make a fist.

[0065] It should be pointed out that the gesture recognition technology is not limited to the method level, nor is it limited to the use of the left or right hand, nor is it limited to the current five gestures. This is because whether it is a gesture recognition method based on RGB images or a gesture recognition method based on IMU, the difference between the left and right hands and the difference in gestures are only differences in model classification learning samples, and do not constitute the essential difference of the gesture-controlled drone technology.

[0066] The specific implementation of the RGB-based gesture recognition algorithm can be done by following the steps below:

[0067] 1. Construction of gesture dataset. Take images of multiple people’s manipulation gestures to ensure that the number of samples collected is no less than 10,000, and use image enhancement methods such as image rotation, brightness change, size reduction, angle change, etc. to expand to no less than 40,000, thus constructing a gesture dataset.

[0068] 2. Gesture recognition algorithm model. Figure 7 As shown in the figure, the gesture recognition algorithm adopts a neural network model algorithm. The input of the model is an arbitrary gesture image, which is collected in real time by the XR front camera. The collected image is sent to the open source Yolo algorithm to realize the positioning and segmentation of the "hand" in the image. The detected hand image data is sent to the convolution layer to extract the hand image features, and then the features are sent to the fully connected layer and the softmax activation function layer. The features are nonlinearly fitted to obtain the classification probability value of each gesture. The classified gesture with the largest probability value is the recognized gesture.

[0069] 3. Model deployment. As an example, the open source NCNN forward computing framework can be used to implement deployment on mobile devices. More generally, frameworks such as PyTorch Mobile, TensorFlow Lite, and Caffe2 can also be used to implement deployment of the neural network on mobile devices.

[0070] S5: Flight path confirmation; the XR device sends the three-dimensional coordinates of the target waypoint determined in step S3 and the flight control command of step S4 to the drone in real time, and uses the line connecting the current position of the drone and the target waypoint as the current flight path of the drone; determines whether there are obstacles on the current flight path. If there are no obstacles, the current flight path is confirmed; otherwise, the drone sends a warning reminder to the XR device and returns to step S2.

[0071] In a specific implementation, the determination of whether there is an obstacle in the current flight path is specifically that the laser radar or depth camera carried by the drone obtains obstacle information within a depth range H and a width range W in the direction of the current flight path of the drone, where H = k*G z , k is the safety factor, which indicates the safety tolerance of the flight path, and is usually set to 1-1.5; W is the width of the drone body; if there is obstacle information, it is determined that there is an obstacle in the current flight path, otherwise, it is determined that there is no obstacle in the current flight path. The obstacle detection method can use laser radar or depth camera to obtain the obstacle point cloud of ROI (region of interest), eliminate noise and outliers through point cloud filtering (through filtering), and then segment the filtered point cloud through clustering algorithm to obtain the obstacle contour and specific obstacle distance.

[0072] In a specific implementation, when the drone issues a warning reminder, it reminds the operator that the current target waypoint is unreachable and the drone enters a hovering state.

[0073] S6: Flight control. The drone performs flight missions based on dynamically planned paths and flight control commands, achieving flight to the target waypoint and realizing “fly wherever you want”.

[0074] It should be noted that gesture commands are only used to change the current flight speed of the drone and start and stop operations. When step S5 does not detect any gesture or step S6 does not receive a new gesture command, the drone completes the drone flight control task at the current speed or default speed, solely by the operator's eyes.

[0075] The above are only preferred embodiments of the invention and are not intended to limit the invention. Any modifications, equivalent substitutions, improvements, etc. made within the conceptual principles of the invention should be included in the protection scope of the invention.

Claims

1. An eye gesture drone control method based on XR device, characterized in that: The following steps are involved: S1, establish real-time communication between XR device and drone; S2, gaze point prediction: The drone transmits the collected first-person perspective image back to the XR device in real time and displays it, and uses the deep learning method of pure eye diagram to predict the two-dimensional gaze point coordinates of the operator's eyes in the image (G x , G y ); S3: Target waypoint determination; based on the two-dimensional gaze point coordinates (G x , G y ), combined with the depth information to determine the three-dimensional coordinates of the target waypoint (G x , G y , G z ), Where D represents the maximum depth distance, which is determined according to the application scenario; v max Indicates the maximum speed of the drone, v min Indicates the minimum speed of the drone, and v indicates the current speed of the drone; S4: Gesture command recognition: The XR device obtains the operator's hand image through its own front camera, recognizes the corresponding gesture based on the gesture recognition algorithm, and converts the recognized gesture into the corresponding flight control command; S5: Flight path confirmation; the XR device sends the three-dimensional coordinates of the target waypoint determined in step S3 and the flight control command of step S4 to the drone in real time, and uses the line connecting the current position of the drone and the target waypoint as the current flight path of the drone; determines whether there are obstacles in the current flight path, and if there are no obstacles, confirms the current flight path; otherwise, the drone sends a warning to the XR device and returns to step S2; S6: Flight control; the drone performs flight missions based on the current flight path and flight control instructions; Step S3 also includes a step of de-jittering the gaze point prediction result, and smoothing filtering is performed on the gaze point prediction result. The result after smoothing filtering is: in Meets the inequality In the inequality, represents the x-coordinate of the predicted gaze point, represents the y coordinate of the predicted fixation point, m represents the size of the smoothing filter window; in the result expression after smoothing filtering, I represents the predicted fixation point The set that meets the inequality, n represents the number of predicted fixation points that meet the I set.

2. The eye gesture drone control method based on XR device according to claim 1 is characterized in that: In step S5, the determination of whether there are obstacles in the current flight path is as follows: the laser radar or depth camera of the drone obtains obstacle information within the depth range H and the width range W in the direction of the current flight path of the drone, where H = k*G z , k is the safety factor, indicating the safety tolerance of the flight path, and W is the width of the UAV body; if there is obstacle information, it is determined that there is an obstacle in the current flight path, otherwise, it is determined that there is no obstacle in the current flight path.

3. The eye gesture drone control method based on XR device according to claim 1, characterized in that: The deep learning method of pure eye diagram in step S2 specifically includes the construction of eye movement data set, the establishment of eye movement tracking model, the training of eye movement tracking model and the deployment steps of eye movement tracking model; wherein the eye movement tracking model is a neural network model, including convolution layer, activation function and fully connected layer, the convolution layer is used to extract image features related to sight line from the eye image, the activation function is used to perform nonlinear mapping on the result of convolution layer, and the fully connected layer is used to fit all the extracted features to obtain two-dimensional sight line coordinates, that is, two-dimensional gaze point coordinates.

4. The eye gesture drone control method based on XR device according to claim 3 is characterized in that: The eye movement dataset is constructed specifically by designing a two-dimensional display plane of a certain size in the XR device, evenly marking more than 200 points on the plane and recording the two-dimensional coordinates of the points, and collecting eye images of multiple subjects when they view all the points through the near-eye camera of the XR device.

5. The eye gesture drone control method based on XR device according to claim 1, characterized in that: The real-time communication includes local area network, 4G, 5G and Wi-Fi wireless communication.

6. The eye gesture drone control method based on XR device according to claim 1, characterized in that: The gesture recognition algorithm is an IMU-based gesture recognition method.

7. The eye gesture drone control method based on XR device according to claim 1, characterized in that: The gesture recognition algorithm is a gesture recognition method based on RGB images. The method adopts a neural network model, including firstly realizing the positioning and segmentation of the hand image through the Yolo algorithm, further extracting the hand image features through the convolution layer, and then realizing the nonlinear mapping from gesture category to recognition probability through the fully connected layer and the softmax activation function. The gesture category corresponding to the maximum recognition probability is the recognized gesture.

8. The eye gesture drone control method based on XR device according to claim 7, characterized in that: The XR device collects hand images in real time through a front camera equipped thereon.

9. The eye gesture drone control method based on XR device according to claim 1, characterized in that: Gesture commands are only used to change the current flight speed of the drone and start and stop operations. When step S4 does not detect any gesture or step S5 does not receive a new flight control command, the drone will be controlled by the operator's eyes alone at the current speed or default speed.

Images