Wearable drone control device and drone system

Through the virtual headband display unit and display glasses of the wearable control device of the drone, combined with eye tracking and voice interaction, the problem of large weight and poor user experience of the drone manual control mode is solved, and immersive control and Multi-diverse operation, suitable for efficient logistics and distribution.

CN118778269BActive Publication Date: 2024-11-19CHINA ORDNANCE EQUIP GRP AUTOMATION RES INST CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411251826.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-09
Publication Date
2024-11-19
Estimated Expiration
2044-09-09

AI Technical Summary

Technical Problem

The existing drone manual control modes have problems such as heavy weight, inconvenient operation and poor user experience, especially in strong sunlight and when multiple interactions are required.

Method used

A wearable control device for drone is designed, including a virtual headband display unit, display glasses, antenna components and computing module. Through eye tracking sensors and voice interaction, the immersive control of the drone is realized, providing high-altitude overview. Perspective and diverse operation methods.

Benefits of technology

It reduces the weight of the portable display middle platform, improves the convenience of control and user experience, and realizes tasks such as autonomous takeoff, landing, orbiting flight and video recording, which are suitable for precise material distribution in the logistics field.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118778269B_ABST
    Figure CN118778269B_ABST
Patent Text Reader

Abstract

The present invention discloses a wearable control device for unmanned aerial vehicles and an unmanned aerial vehicle system. The device reduces the weight of the display platform carried by the unmanned aerial vehicle system through the organic combination of a video virtual reality head-mounted display and an unmanned platform system, and solves the problem that the video pan-tilt operator cannot clearly see the display platform or the image transmission screen of a smart phone / tablet terminal under strong sunlight. It provides the unmanned aerial vehicle operator with a unique viewing experience of a high-altitude bird's-eye view and a more diverse and immersive user experience. At the same time, by using voice interaction, the unmanned aerial vehicle can complete basic tasks such as autonomous take-off, landing at a designated location, constant-speed program-controlled flight, flying around a locked target, video recording, and photo shooting. In addition, by using the control handle interface, only simple training is required to try to use the unmanned aerial vehicle platform for more accurate material distribution, which promotes the application of unmanned aerial vehicles in the field of logistics.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of drone control technology, and in particular to a wearable drone control device and a drone system that can meet the needs of immersive drone control. Background Art

[0002] Although drones use intelligent obstacle avoidance and autonomous navigation to achieve safe autonomous flight and landing in most scenarios, manual control mode is still indispensable.

[0003] The manual control mode of a drone means that the operator manually controls the flight and movement of the drone, which is done through a remote control or ground station and other equipment. In manual mode, the operator can control the altitude, speed, direction, attitude, etc. of the drone, allowing the drone to perform various flight actions, such as hovering, horizontal flight, vertical ascent and descent, etc. The advantage of manual control mode is that it allows the operator to control the drone more flexibly, freely adjust flight parameters and actions according to different flight missions and environmental conditions, and improve the flight stability and control accuracy of the drone.

[0004] At present, the manual control mode of drones is mainly based on the screen images of the display center and the smartphone / tablet terminal. However, this control method has many disadvantages. For example, the display center is heavy and not easy to carry. At the same time, the video gimbal operator cannot see the display center or the image transmission image of the smartphone / tablet terminal clearly under strong sunlight, which affects the operator's control. In addition, the user immersion experience is poor and the interaction method is relatively simple.

[0005] Therefore, how to provide a wearable device platform system that meets the needs of immersive drone control is a technical problem that urgently needs to be solved by technical personnel in this field. Summary of the invention

[0006] In view of the above problems, the present invention provides a wearable drone control device and a drone system for overcoming the above problems or at least partially solving the above problems.

[0007] The present invention provides the following scheme:

[0008] A wearable control device for a drone, comprising:

[0009] A virtual headband display unit, the virtual headband display unit includes a head harness, display control glasses and an antenna assembly are arranged at a position of the head harness relative to the forehead, and a battery compartment and a computing module are arranged at a position of the head harness relative to the back of the head;

[0010] The computing module is communicatively connected to the display and control glasses and the antenna assembly respectively; the antenna assembly is used to realize data interaction with the drone to be controlled;

[0011] The calculation module is used to perform the following operations:

[0012] Processing the received video data and multi-sensor information of the drone to be controlled so as to achieve superimposed display on the display and control glasses;

[0013] Receive flight attitude control data sent by the control handle; the flight attitude control data includes control data within the physical limit angles in the horizontal and pitch directions; parse the flight attitude control data into a first control instruction for the unmanned aerial vehicle to be controlled; send the first control instruction to the unmanned aerial vehicle to be controlled, so that the unmanned aerial vehicle to be controlled executes the first control instruction.

[0014] Preferably, the display-controlled glasses include an eye-tracking sensor and an infrared lamp; the infrared lamp is used to provide illumination for the eye area to assist in extracting pupil features, and the computing module is further used to perform the following operations:

[0015] The eye tracking data of the user acquired by the eye tracking sensor is received and an eye tracking algorithm is executed, the eye tracking data is parsed into a video position point of interest to the user, and a second control instruction of the drone to be controlled is generated; the second control instruction is sent to the drone to be controlled, so that the drone to be controlled executes the second control instruction to frame and lock the video target position of interest.

[0016] Preferably: the eye tracking algorithm includes a basic calibration module, a pupil feature extraction module, a gaze estimation module, and a space transformation and scene understanding module;

[0017] The basic calibration module is used to construct a sightline mapping function in combination with the known screen calibration point position coordinates to calculate the gaze point of the human eye;

[0018] The pupil feature extraction module is used to extract eye movement image features from images collected by the eye movement tracking sensor;

[0019] The gaze estimation module is used to obtain the gaze point of the human eye by using the eye movement image features in combination with a gaze estimation algorithm based on a two-dimensional mapping plane through vector fitting calculation;

[0020] The spatial transformation and scene understanding module is used to resolve the gaze point position into a spatial position on the display screen of the display control glasses, draw an eye movement trajectory and frame and lock the video target position of interest.

[0021] Preferably, the pupil feature extraction module uses the following method to screen and determine the pupil ellipse contour:

[0022] When the number of points of one contour is less than 5, discard this contour;

[0023] When it is determined that the diameter of the ellipse fitted by one of the contours is beyond the range of 0.07 Edmin to 0.29 Edmax, the contour is discarded;

[0024] The curvature information of different contours is used to calculate the ratio of the short side to the long side of the minimum circumscribed rectangle of the contour. When the curvature of one of the calculated contours is less than the preset curvature, the contour is discarded.

[0025] Use the least square method to fit an ellipse to each of the remaining contours. If the center of the fitted ellipse is not within the image, discard the contour.

[0026] The major axis and minor axis breakpoints of the fitted ellipse are connected to form a quadrilateral. When the pixel coordinates of all points in the contour are averaged and the obtained point is outside the quadrilateral, the contour is discarded.

[0027] Preferably, the pupil feature extraction module uses the following method to screen and determine the Purkinje spots:

[0028] Randomly select 5 points in the contour and fit the ellipse using the least squares method;

[0029] For each point in the contour, the error value is calculated separately, and those that meet the conditions are included in the internal point set;

[0030] For the set of interior points, the evaluation function value is calculated, and if it exceeds the recorded maximum value, the ellipse equation is updated;

[0031] Repeat the first three steps several times to obtain the final elliptic curve, and calculate the pupil center and Purkinje spot center.

[0032] Preferably, the head harness is also provided with a microphone, and the microphone is communicatively connected with the computing module; the computing module is also used to perform the following operations:

[0033] Receiving voice control data collected by the microphone, the voice control data at least including take-off, landing, hovering, horizontal flight, vertical ascent, vertical descent, locking frame selection position, unlocking frame selection, circling flight, shooting, continuous shooting, and video recording voice control instructions;

[0034] The voice control data is parsed into a third control instruction of the drone to be controlled; the third control instruction is sent to the drone to be controlled so that the drone to be controlled executes the third control instruction, thereby achieving gaze locking and circumferential flight tracking of the drone to be controlled on the video target position of interest.

[0035] Preferably, the display-controlled glasses include two display screens with a field of view angle of 85° and a single-eye resolution of 1920×1080, and an interpupillary distance adjustment knob capable of adjusting the interpupillary distance in the range of 56-72 mm.

[0036] Preferably, the display-controlled glasses further include an infrared sensor, and the infrared sensor is communicatively connected to the computing module; the computing module is further configured to perform the following operations:

[0037] Determine whether the infrared sensor is triggered;

[0038] Determining the current wearing state of the headgear according to the triggering condition of the infrared sensing sensor;

[0039] When it is determined that the head harness is currently in a non-wearing state, controlling the display screen and the eye tracking sensor to turn off;

[0040] When it is determined that the head harness is currently in a wearing state, the display screen and the eye tracking sensor are controlled to start.

[0041] Preferably: the head harness is further provided with a USB Type-C interface, and the USB Type-C interface is used to connect the control handle and the computing module;

[0042] The head harness is also provided with a Micro SD card slot; the Micro SD card slot is used to access a Micro SD card, and the Micro SD card is used to record and store videos of the entire flight or selected time;

[0043] The head harness is also provided with an HDMI D-type interface; the HDMI interface is used to realize the synchronous extended display function of the video data on an external display.

[0044] A drone system, comprising a drone platform, an optoelectronic gimbal, a processing and communication system module, a multi-sensor, a flight control system and the above-mentioned wearable drone control device;

[0045] The UAV platform is used to carry the optoelectronic gimbal, the processing and communication system module, the multi-sensor and the flight control system;

[0046] The photoelectric PTZ is used to collect video data;

[0047] The processing and communication system module is used to establish a communication connection with the antenna assembly, compress the video data acquired by the optoelectronic gimbal, fuse and compress the data acquired by the multi-sensor, perform real-time analysis of control commands, control the optoelectronic gimbal according to the control commands, and interact with the flight control system information.

[0048] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0049] The embodiments of the present application provide a wearable drone control device and a drone system. The device reduces the weight of the display platform by organically combining a video virtual reality head-mounted display with an unmanned platform system, and solves the problem that the video gimbal operator cannot clearly see the display platform or the image transmission screen of a smart phone / tablet terminal under strong sunlight. It provides the drone operator with a unique viewing experience of a high-altitude bird's-eye view and a more diverse and immersive user experience. At the same time, using voice interaction, basic tasks such as autonomous drone takeoff, landing at a designated location, constant-speed program-controlled flight, flying around a locked target, video recording, and photo shooting can be completed. In addition, using the control handle interface, only simple training is required to try to use the drone platform for more accurate material distribution, which promotes the application of drones in the logistics field.

[0050] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 It is a schematic diagram of the structure of a wearable control device for a drone provided in an embodiment of the present invention;

[0053] Figure 2 Schematic diagram of another structure of a wearable drone control device provided by an embodiment of the present invention

[0054] Figure 3 is a schematic structural diagram of a drone system provided by an embodiment of the present invention;

[0055] Figure 4 is a schematic diagram of an eye tracking structure provided by an embodiment of the present invention;

[0056] Figure 5 It is a schematic diagram of a 9-point calibration point distribution calibration plate provided in an embodiment of the present invention.

[0057] In the figure: virtual headband display unit 1, head equipment 11, display and control glasses 12, eye tracking sensor 121, infrared light 122, antenna assembly 13, battery compartment 14, computing module 15, magnetic soft sponge mask 16, pupil distance adjustment knob 17, infrared sensing sensor 18, microphone 19, USB Type-C interface 110, Micro SD card slot 111, HDMI D-type interface 112, drone platform 2, control handle 3. DETAILED DESCRIPTION

[0058] The technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0059] See also Figure 1 , Figure 2 , is a wearable control device for a drone provided by an embodiment of the present invention, such as Figure 1 , Figure 2 As shown, the device may include:

[0060] A virtual headband display unit 1, wherein the virtual headband display unit 1 comprises a head harness 11, wherein display control glasses 12 and an antenna assembly 13 are arranged at a position relative to the forehead of the head, and wherein a battery compartment 14 and a computing module 15 are arranged at a position relative to the back of the head of the head;

[0061] The computing module 15 is communicatively connected to the display and control glasses 12 and the antenna assembly 13 respectively; the antenna assembly 13 is used to realize data interaction with the drone to be controlled; the display and control glasses 12 include an eye tracking sensor;

[0062] The calculation module 15 is used to perform the following operations:

[0063] Processing the received video data and multi-sensor information of the drone to be controlled so as to achieve superimposed display on the display and control glasses 12;

[0064] Receive flight attitude control data sent by the control handle; the flight attitude control data includes control data within the physical limit angles in the horizontal and pitch directions; parse the flight attitude control data into a first control instruction for the unmanned aerial vehicle to be controlled; send the first control instruction to the unmanned aerial vehicle to be controlled, so that the unmanned aerial vehicle to be controlled executes the first control instruction.

[0065] The wearable drone control device provided in the embodiment of the present application can be used by a user to manually control the drone in an immersive manner. The virtual headband display unit 1 is used as the main carrier, which can effectively improve the convenience of control and facilitate the user's immersive experience. By tracking the user's eyes, the drone to be controlled can achieve gaze locking and circumferential flight tracking on the video target position of interest, which greatly improves the convenience of drone control.

[0066] At the same time, the display and control glasses 12 and the battery compartment 14 adopt a split design, and the head circumference width can be adjusted by a knob, and the weight of the entire virtual headband display unit 1 is evenly distributed on the front and back sides of the head, which can effectively improve the comfort of the user when wearing it. The head-mounted design is adopted, and the display and control glasses 12, the battery compartment 14 and other functions are integrated into a virtual headband display unit 1 for head wear, which effectively improves the wearing comfort and convenience of use.

[0067] In order to further improve the comfort of wearing the head harness 11, the embodiment of the present application can also provide that the head harness 11 is provided with a magnetic soft sponge mask 16 at the relative position of the head forehead. The head harness 11 is provided with a magnetic soft sponge mask 16 at the relative position of the user's forehead to improve the wearing comfort.

[0068] The virtual headband display unit 1 provided in the embodiment of the present application can establish a communication link with the drone to be controlled through the provided antenna. Specifically, the embodiment of the present application can provide that the antenna assembly 13 includes a plurality of 360° omnidirectional receiving antennas. The integrated 360° omnidirectional receiving antenna can maintain a communication connection with the drone in any direction and improve communication stability. In practical applications, the number of antennas can be determined as needed. For example, four 360° omnidirectional receiving antennas can be set to ensure the stability of data reception.

[0069] Further, such as Figure 4 As shown, the embodiment of the present application can also provide that the display-controlled glasses 12 include an eye tracking sensor 121 and an infrared lamp 122; the infrared lamp 122 is used to provide lighting for the eye area to assist in extracting pupil features, and the computing module 15 is also used to perform the following operations:

[0070] Receive the user's eye tracking data acquired by the eye tracking sensor 121 and execute the eye tracking algorithm, parse the eye tracking data into the video position point that the user is paying attention to, and generate a second control instruction for the drone to be controlled; send the second control instruction to the drone to be controlled, so that the drone to be controlled executes the second control instruction to frame and lock the video target position of interest.

[0071] In specific implementation, the embodiment of the present application may provide that the eye tracking algorithm includes a basic calibration module, a pupil feature extraction module, a gaze estimation module, and a space transformation and scene understanding module;

[0072] The basic calibration module is used to construct a sightline mapping function in combination with the known screen calibration point position coordinates to calculate the gaze point of the human eye;

[0073] The pupil feature extraction module is used to extract eye movement image features from images collected by the eye movement tracking sensor;

[0074] The gaze estimation module is used to obtain the gaze point of the human eye by using the eye movement image features in combination with a gaze estimation algorithm based on a two-dimensional mapping plane through vector fitting calculation;

[0075] The spatial transformation and scene understanding module is used to resolve the gaze point position into a spatial position on the display screen of the display control glasses, draw an eye movement trajectory and frame and lock the video target position of interest.

[0076] The pupil feature extraction module uses the following method to screen and determine the pupil ellipse contour:

[0077] When the number of points of one contour is less than 5, discard this contour;

[0078] When it is determined that the diameter of the ellipse fitted by one of the contours is beyond the range of 0.07 Edmin to 0.29 Edmax, the contour is discarded;

[0079] The curvature information of different contours is used to calculate the ratio of the short side to the long side of the minimum circumscribed rectangle of the contour. When the curvature of one of the calculated contours is less than the preset curvature, the contour is discarded.

[0080] Use the least square method to fit an ellipse to each of the remaining contours. If the center of the fitted ellipse is not within the image, discard the contour.

[0081] The major axis and minor axis breakpoints of the fitted ellipse are connected to form a quadrilateral. When the pixel coordinates of all points in the contour are averaged and the obtained point is outside the quadrilateral, the contour is discarded.

[0082] The pupil feature extraction module uses the following method to screen and determine the Purkinje spots:

[0083] Randomly select 5 points in the contour and fit the ellipse using the least squares method;

[0084] For each point in the contour, the error value is calculated separately, and those that meet the conditions are included in the internal point set;

[0085] For the set of interior points, the evaluation function value is calculated, and if it exceeds the recorded maximum value, the ellipse equation is updated;

[0086] Repeat the first three steps several times to obtain the final elliptic curve, and calculate the pupil center and Purkinje spot center.

[0087] The display and control glasses 12 provided in the embodiment of the present application are equipped with two display screens with a fov of 85° and a single-eye resolution of 1920×1080, and support a maximum screen refresh rate of 60Hz. They can display both the video data of the drone to be controlled and the multivariate data obtained by the sensors on the drone, and the pupil distance adjustment knob 17 can achieve pupil distance adjustment in the range of 56-72 mm. At the same time, an eye tracking sensor is equipped on the glasses structure to track the eye position and feed it back to the calculation module 15.

[0088] An infrared lamp 122 is used as an auxiliary light source, and the wavelength of the light wave is about 850nm (the infrared rays in this band are invisible to the human eye, do not interfere much with humans, and will not cause stimulation to the pupils).

[0089] The eye tracking sensor 121 shown captures the light spot on the human cornea for pupil feature extraction.

[0090] The eye tracking algorithm includes four parts: basic calibration module, pupil feature extraction module, gaze estimation module, and space transformation and scene understanding module.

[0091] 1. The basic calibration module is mainly responsible for the initial calibration of eye movement data when different people wear and use display glasses. Combined with the known screen calibration point position coordinates, the sight mapping function is constructed to calculate the gaze point of the human eye. It mainly includes:

[0092] Assume the coordinates of the fixed point on the screen are , the line of sight estimation method based on pupil-corneal reflection of pupil feature extraction module obtains the corresponding vector coordinates as , for arrive The corresponding relationship, that is

[0093]

[0094] like Figure 5 As shown, the 9-point calibration method is used to estimate the gaze point With the true value During the calibration process, the user's pupil center and Purkinje spot center are collected, and the quadratic polynomial mapping model and the least squares method are used to fit the pupil-corneal reflection estimation vector and the gaze point coordinates.

[0095] The quadratic polynomial mapping model is:

[0096]

[0097]

[0098] in, It represents the two-dimensional coordinates of the point where the human eye is looking. Refers to the two-dimensional coordinates of the pupil center , It represents the coefficient of the mapping function. According to the collected 9 screen calibration points, 18 equations can be obtained, and the unknown parameters, that is, the coefficient of the mapping function, can be obtained through least squares.

[0099] 2. The pupil feature extraction module is mainly responsible for extracting image features from images collected by the eye tracking sensor. Its process generally includes:

[0100] According to the eye features, the image processing pre-selected area is delineated, grayscale normalization is performed, the dynamic threshold is obtained from the histogram, the Canny algorithm edge is extracted, morphological processing is performed, the pupil contour and Purkinje spot are screened and determined, the pupil ellipse is fitted, and the pupil center and Purkinje spot center are calculated, etc., to provide the gaze estimation module with stable and reliable human eye image features, such as the pupil center and Purkinje spot center.

[0101] In the embodiment of the present application, based on the extraction of eye feature edges and morphological processing (a general image preprocessing algorithm, which will not be described in detail), a fast pupil ellipse contour and Purkinje spot screening and determination model method is constructed:

[0102] Ellipse-based expressions , using the least squares method to fit the ellipse equation, at least 5 points are required. They all represent the coefficients of the ellipse equation function. When the number of points of a contour is less than 5, the contour is discarded.

[0103] In the eye image captured by the eye tracking camera, the pupil area should occupy a relatively small proportion of the entire image area. Range of variation:

[0104]

[0105] in, is the number of pixels in the width direction of the image, is the number of pixels in the image height direction. Normally, the inner and outer distances of the eye corners are about 27.6 mm, and the pupil diameter varies from 2 mm to 8 mm. The pupil diameter in the eye movement image ranges from 0.07 Edmin to 0.29 Edmax. If the diameter of the ellipse fitted by a contour exceeds this range, the contour is discarded.

[0106] The curvature information of different contours is used to calculate the ratio of the short side to the long side of the minimum circumscribed rectangle of the contour. When the calculated curvature is too small, the contour is discarded.

[0107] The least square method is used to fit an ellipse to each of the remaining contours. If the center of the fitted ellipse is not within the image, the contour is discarded.

[0108] Connect the major and minor axis breakpoints of the fitted ellipse to form a quadrilateral. If the pixel coordinates of all points in the contour are averaged and the resulting point is outside the quadrilateral, the contour is discarded.

[0109] Since the pupil area is black inside and white outside, the points on the pupil ellipse should have the characteristic that the internal grayscale value is lower than the external grayscale value in its gradient direction.

[0110] In the determination process, the equation of the ellipse is assumed to be

[0111]

[0112] in, They all represent the coefficients of the elliptic equation function.

[0113] If the point On the ellipse, then , then the error is defined as

[0114]

[0115] in, For at point The gradient at . Then the interior point set can be defined as

[0116]

[0117] in, is any given positive number.

[0118] Then the pupil ellipse fitting accuracy evaluation function can be defined:

[0119]

[0120] in, To evaluate a function mathematical expression, Representing the image The gradient at represents the major axis of the fitted ellipse, Represents the minor axis of the fitted ellipse.

[0121] The specific steps for screening and determining Purchin spots include:

[0122] a. Randomly select 5 points in the contour and fit the ellipse using the least squares method.

[0123] b. For each point in the contour, calculate the error value and include those that meet the conditions into the internal point set.

[0124] c. For the set of interior points, calculate the evaluation function value, and update the ellipse equation if it exceeds the recorded maximum value.

[0125] d. Repeat the first three steps several times to obtain the final elliptic curve and calculate the pupil center and Purkinje spot center.

[0126] 3. The gaze estimation module mainly combines the eye movement image features provided by the pupil feature extraction module with the gaze estimation algorithm based on the two-dimensional mapping plane to calculate the gaze point of the human eye through vector fitting.

[0127] After the eye movement data is initially calibrated by the basic calibration module, the system has calculated the gaze estimation mapping function based on the calibration point data. In the real-time gaze point calculation process, the real-time collected eye movement data is brought into the calculated gaze estimation mapping relationship to calculate the gaze point of the human eye.

[0128] 4. The spatial transformation and scene understanding module is mainly responsible for parsing the gaze point position output by the gaze estimation module into the spatial position on the display screen of the display control glasses 12, drawing the eye movement trajectory and locking the video target position of interest. First, the screen gaze area is divided into several small rectangles (the finer the division, the more accurate the target frame selection center selection point will be). When the user's gaze point falls on a certain rectangular area, the area will be selected and the area coordinates will be output. The corresponding target object at the current coordinates is identified through the target recognition algorithm, and the frame selection is completed, realizing human-computer interaction control based on glasses.

[0129] It can be understood that the device provided in the embodiment of the present application can lock the position of the video target through an eye tracking sensor, thereby realizing gaze locking and surround flight tracking.

[0130] In order to further improve the control diversity of the device, the embodiment of the present application may also provide that the head harness 11 is further provided with a microphone 19, and the microphone 19 is communicatively connected with the computing module 15;

[0131] The calculation module 15 is also used to perform the following operations:

[0132] Receive voice control data collected by the microphone 19, the voice control data at least including voice control instructions related to take-off, landing, hovering, horizontal flight, vertical ascent, vertical descent, locking frame selection position, unlocking frame selection, circling flight, shooting, continuous shooting, and video recording;

[0133] The voice control data is parsed into a third control instruction of the drone to be controlled; the third control instruction is sent to the drone to be controlled so that the drone to be controlled executes the third control instruction, thereby achieving gaze locking and circumferential flight tracking of the drone to be controlled on the video target position of interest.

[0134] The integrated microphone 19 can realize the control voice collection and transmit it to the calculation module 15. The calculation module 15 can analyze the voice operation instruction and convert it into a third control instruction suitable for the control of the drone. After receiving the third control instruction, the drone can execute the voice control instruction, thereby further improving the control convenience.

[0135] In order to detect whether the head harness 11 is worn, the embodiment of the present application may further provide that the display-control glasses 12 also include an infrared sensor 18, and the infrared sensor 18 is communicatively connected to the computing module 15; the computing module 15 is also used to perform the following operations:

[0136] Determine whether the infrared sensor 18 is triggered;

[0137] Determine the current wearing state of the head harness 11 according to the triggering condition of the infrared sensor 18;

[0138] When it is determined that the head harness 11 is currently in a non-wearing state, controlling the display screen and the eye tracking sensor to be turned off;

[0139] When it is determined that the head harness 11 is currently in a wearing state, the display screen and the eye tracking sensor are controlled to start.

[0140] The infrared sensor 18 is equipped to detect the current wearing state. When the user is not wearing the headgear 11, the display screen and the eye tracking functions are turned off to achieve safety control and energy saving of the device. At the same time, after detecting that the user is wearing the headgear 11, the display screen and the eye tracking sensor can be automatically started, further improving the convenience of the device operation.

[0141] Furthermore, in order to better roughly control the flight attitude of the drone, the embodiment of the present application can also provide that the head harness 11 is also provided with a USB Type-C interface 110, and the USB Type-C interface 110 is used to connect the control handle 3; the USB Type-C interface 110 is connected to the computing module 15.

[0142] The computing module 15 can parse the handle control data and connect the handle through the USB Type-C interface 110 to realize the control of the UAV and the optoelectronic gimbal. The optoelectronic gimbal control function can be realized based on two modes: the handle and eye tracking. The handle end can realize full control of the gimbal within the physical limit angles in the horizontal and pitch directions, and cooperate with eye tracking to realize micro-adjustment of the gimbal position within the range based on the video image.

[0143] Furthermore, the head harness 11 is also provided with a Micro SD card slot 111 ; the Micro SD card slot 111 is used for accessing a Micro SD card, and the Micro SD card is used to realize recording and storage of videos during the entire flight or a selected time.

[0144] The head harness 11 is also provided with an HDMI D-type interface 112; the micro HDMI interface is used to realize the synchronous extended display function of the video data on an external display.

[0145] The computing module 15 provided in the embodiment of the present application can be based on an ARM core processor and equipped with interfaces such as USB Type-C, micro HDMI D-type interface 112, and Micro SD card slot 111, and mainly undertakes the following 7 functions: First, the processing of the video data (image transmission) sent back by the drone and the superimposition display on the display screen; second, the drone's multi-sensor information and control commands (digital transmission), the sensor information includes the current altitude, speed, positioning information, temperature and humidity, wind speed and other drone status information; third, the microphone 19 voice analysis and conversion into command processing; fourth, the execution of the eye tracking algorithm, and the analysis into the optoelectronic gimbal control command, the control of the optoelectronic gimbal to lock the video target position of interest, and realize the gaze lock and surround flight tracking for the optoelectronic gimbal; fifth, the analysis of the handle control data, the connection of the handle through the USB Type-C interface 110, the control of the drone and the optoelectronic gimbal, the optoelectronic gimbal control function can be realized based on the handle and eye tracking two modes, the handle end can realize the full control of the gimbal in the horizontal and pitch directions within the physical limit angle, and the eye tracking realizes the micro-adjustment of the gimbal position within the range based on the video image; sixth, through the Micro SD The card is connected to realize the recording and storage of video during the entire flight or selected time; the seventh is to complete the expansion function of the drone gimbal video on an external display based on the micro HDMI interface.

[0146] In summary, the wearable drone control device provided by this application reduces the weight of the display platform through the organic combination of the video virtual reality head-mounted display and the unmanned platform system, and solves the problem that the video gimbal operator cannot see the display platform or the image transmission picture of the smart phone / tablet terminal clearly under strong sunlight. It provides the drone operator with a unique viewing experience of a high-altitude bird's-eye view and a more diverse and immersive user experience. At the same time, using voice interaction, basic tasks such as autonomous take-off and landing at a designated location, constant-speed program-controlled flight, and flying around a target can be completed. In addition, using the control handle interface, only simple training is required to try to use the drone platform 2 for more accurate material distribution, which promotes the application of drones in the logistics field.

[0147] In order to better cooperate with the device provided in the embodiment of the present application, see Figure 3 , an unmanned aerial vehicle system provided by an embodiment of the present invention, such as Figure 3 As shown, the system may include:

[0148] UAV platform 2, optoelectronic gimbal, processing and communication system module, multi-sensor, flight control system and the above-mentioned wearable control device for UAV;

[0149] The UAV platform 2 is used to carry the optoelectronic gimbal, the processing and communication system module, the multi-sensor and the flight control system;

[0150] The photoelectric PTZ is used to collect video data;

[0151] The processing and communication system module is used to establish a communication connection with the antenna assembly 13, compress the video data acquired by the optoelectronic gimbal, fuse and compress the data acquired by the multi-sensor, perform real-time analysis of control commands, control the optoelectronic gimbal according to the control commands, and interact with the flight control system information.

[0152] The virtual reality-based UAV system includes four main parts: UAV platform 2, optoelectronic gimbal, processing and communication system module, and split virtual headband display.

[0153] As the main platform, the drone is equipped with an optoelectronic gimbal, a flight control system, and multiple sensors such as temperature and humidity, air pressure, speed, and GPS positioning.

[0154] The optoelectronic PTZ is mainly responsible for video acquisition tasks and has the function of optical intelligent anti-shake, which can effectively reduce the difficulty of video image data compression.

[0155] The processing and communication system module mainly realizes six functions:

[0156] 1. Optical PTZ high-definition video image compression;

[0157] 2. Multi-sensor data fusion and compression;

[0158] 3. Real-time analysis of control commands;

[0159] 4. Photoelectric PTZ control;

[0160] 5. Flight control system information interaction;

[0161] 6. Data communication interaction of split virtual headband display. Image transmission: Transmit the compressed data to the split virtual headband display via wireless network, supporting 1920×1080 resolution 30Hz and 1280×720 resolution 60Hz modes. Digital transmission: Transmit control commands and multi-source sensor status information.

[0162] Through the wearable drone control device provided in the embodiment of the present application, the drone can be controlled through the video data collected by the optoelectronic gimbal, and can also be controlled in various ways in conjunction with the microphone 19 and the control hand, so that the drone system has immersive operation characteristics, and the control experience is improved by optimizing the arrangement of various function buttons on the control interface.

[0163] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0164] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0165] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0166] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A wearable drone control device, characterized in that: include: A virtual headband display unit, comprising a head harness, wherein display glasses and an antenna assembly are disposed relative to the forehead of the head harness, and a battery compartment and a computing module are disposed relative to the back of the head harness; the display glasses include two displays with an 85° field of view and a single-eye resolution of 1920×1080, and an interpupillary distance adjustment knob capable of adjusting the interpupillary distance within a range of 56-72 mm; The computing module is communicatively connected to the display and control glasses and the antenna assembly respectively; the antenna assembly is used to implement data interaction with the drone to be controlled; The calculation module is used to perform the following operations: Processing the video data and multi-sensor information received from the drone to be controlled so as to achieve overlay display on the display and control glasses; Receiving flight attitude control data sent by a control handle; the flight attitude control data includes control data within physical limit angles in horizontal and pitch directions; parsing the flight attitude control data into a first control instruction for the unmanned aerial vehicle to be controlled; and sending the first control instruction to the unmanned aerial vehicle to be controlled so that the unmanned aerial vehicle to be controlled executes the first control instruction; The display-controlled glasses include an eye tracking sensor and an infrared lamp; the infrared lamp is used to provide illumination to the eye area to assist in extracting pupil features. The computing module is also used to perform the following operations: receiving eye tracking data of the user acquired by the eye tracking sensor and executing an eye tracking algorithm to interpret the eye tracking data as a video location of interest to the user, and generating a second control instruction for the drone to be controlled; and transmitting the second control instruction to the drone to be controlled so that the drone to be controlled executes the second control instruction to select and lock the video target location of interest; The eye tracking algorithm includes a basic calibration module, a pupil feature extraction module, a gaze estimation module, and a spatial transformation and scene understanding module; The basic calibration module is used to construct a sightline mapping function based on the known coordinates of the screen calibration points to calculate the gaze point of the human eye; The pupil feature extraction module is used to extract eye image features from the image collected by the eye tracking sensor; The gaze estimation module is used to use the eye movement image features, combined with a gaze estimation algorithm based on a two-dimensional mapping plane, to obtain the gaze point of the human eye through vector fitting calculation; The spatial transformation and scene understanding module is used to resolve the gaze point position into a spatial position on the display screen of the display control glasses, draw the eye movement trajectory and frame and lock the video target position of interest; The pupil feature extraction module uses the following method to screen and determine the pupil ellipse contour: When the number of points of one contour is less than 5, discard the contour; Determine that the diameter of one of the ellipse fits exceeds 0.07 to 0.29 When the range is reached, discard this contour; Using the curvature information of different contours, the ratio of the short side to the long side of the minimum circumscribed rectangle of the contour is calculated. When the curvature of one of the calculated contours is less than the preset curvature, the contour is discarded. Use the least squares method to fit an ellipse to each of the remaining contours. If the center of the fitted ellipse is not within the image, discard the contour. Connect the major and minor axis breakpoints of the fitted ellipse to form a quadrilateral. If the pixel coordinates of all points in the contour are averaged and the resulting point is outside the quadrilateral, discard the contour. The pupil feature extraction module uses the following method to screen and determine Purkinje spots: Randomly select 5 points in the contour and fit the ellipse using the least squares method; For each point in the contour, the error value is calculated separately, and those that meet the conditions are included in the internal point set; For the set of interior points, calculate the evaluation function value, and if it exceeds the recorded maximum value, update the ellipse equation; Repeat the first three steps several times to obtain the final elliptic curve, and calculate the pupil center and Purkinje spot center.

2. The wearable drone control device according to claim 1, characterized in that: The head harness is further provided with a microphone, and the microphone is communicatively connected to the computing module; the computing module is further configured to perform the following operations: receiving voice control data collected by the microphone, the voice control data including at least voice control commands for takeoff, landing, hovering, horizontal flight, vertical ascent, vertical descent, locking a frame selection position, unlocking a frame selection, circling flight, shooting, continuous shooting, and video recording; The voice control data is parsed into a third control instruction of the drone to be controlled; the third control instruction is sent to the drone to be controlled so that the drone to be controlled executes the third control instruction, thereby achieving gaze locking and circling flight tracking of the drone to be controlled on the video target position of interest.

3. The wearable drone control device according to claim 1, characterized in that: The display-controlled glasses further include an infrared sensor, which is communicatively connected to the computing module; the computing module is further configured to perform the following operations: Determining whether the infrared sensor is triggered; Determining the current wearing state of the head harness according to whether the infrared sensor is triggered; When it is determined that the head harness is currently not being worn, controlling the display screen and the eye tracking sensor to turn off; When it is determined that the head harness is currently in a wearing state, the display screen and the eye tracking sensor are controlled to start.

4. The wearable drone control device according to claim 1, characterized in that: The head harness is further provided with a USB Type-C interface, and the USB Type-C interface is used to connect the control handle and the computing module; The head harness is also provided with a Micro SD card slot; the Micro SD card slot is used to connect a Micro SD card, and the Micro SD card is used to record and store videos of the entire flight or selected time; The head harness is also provided with an HDMI D-type interface; the HDMI D-type interface is used to realize the synchronous extended display function of the video data on an external display.

5. A drone system, characterized in that: It comprises a UAV platform, an optoelectronic gimbal, a processing and communication system module, a multi-sensor, a flight control system, and the wearable UAV control device according to any one of claims 1 to 4; The UAV platform is used to carry the optoelectronic gimbal, the processing and communication system module, the multi-sensor and the flight control system; The photoelectric PTZ is used to collect video data; The processing and communication system module is used to establish a communication connection with the antenna assembly, compress the video data obtained by the optoelectronic gimbal, fuse and compress the data obtained by the multi-sensor, perform real-time analysis of control commands, control the optoelectronic gimbal according to the control commands, and interact with the flight control system information.