A control method and system for first-person perspective flight of unmanned aerial vehicle, and smart glasses
Through the drone and smart glasses, the signal delay problem caused by remote control relay is solved through the drone's direct channel establishment and the combination of augmented reality and gesture recognition technology is solved, and the first-view flight control of the drone is achieved with high accuracy and security.
Patent Information
- Application Number
- CN202110800387.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2016-04-29
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2036-04-29
AI Technical Summary
In the existing first-view flight control method of drone, since the remote control is required to relay, the image transmission and control signals are delayed, which affects the immediacy and safety of the control.
The drone and smart glasses directly establish the first channel, and the smart glasses directly establish the second channel, realizing image data transmission and flight control command transmission without relay, displaying flight status data through augmented reality, and through the camera and The processor recognizes the user gesture to generate corresponding control instructions.
It avoids delays in image transmission and control signals, improves the accuracy and safety of drone control, provides an immersive flight experience, and simplifies user operations.
Smart Images

Figure CN113359807B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of unmanned aerial vehicles, and in particular relates to a control method and system for first-person perspective flight of unmanned aerial vehicles, and smart glasses. Background Art
[0002] First-person perspective flight is a new direction in the development of drones. Through the image transmission module on the drone, the images taken by the camera on the drone are transmitted to the glasses in real time. Users can control the flight of the drone by watching the camera images. The feature of first-person perspective flight is that users no longer have to look up at the drone, but can get the same perspective as the drone and control it from the perspective of the drone, thus achieving an immersive flight experience.
[0003] The existing method of achieving first-person perspective flight is that the drone and the remote control are connected wirelessly through a private communication protocol or WIFI. After the remote control receives the image signal from the drone, it is transmitted to the glasses via USB or other wired methods.
[0004] However, this method often leads to delays in image transmission and control signals due to the need for remote control relay, and cannot instantly and accurately control the flight of the drone, posing a safety hazard. Summary of the invention
[0005] In view of this, the embodiments of the present invention provide a control method and system for first-person perspective flight of a drone, and smart glasses, which can directly control the drone without relaying, avoid delays in image transmission and control signals, and improve control accuracy.
[0006] In a first aspect, an embodiment of the present invention provides a control system for an unmanned aerial vehicle, the control system comprising a remote control and smart glasses, wherein the smart glasses can directly establish a first channel with the unmanned aerial vehicle without the need for relaying by the remote control, so that the smart glasses can directly receive and display first-perspective image data from the unmanned aerial vehicle through the first channel, and the remote control can directly establish a second channel with the unmanned aerial vehicle without the need for relaying by the smart glasses, so that the remote control can directly send a first flight control instruction to the unmanned aerial vehicle through the second channel.
[0007] The smart glasses directly receive and display flight status data from the drone through the first channel.
[0008] The flight status data is displayed superimposed on the first perspective image data in an augmented reality manner.
[0009] The smart glasses are capable of sensing the posture of the smart glasses and generating a gimbal control instruction, and the smart glasses further directly send the gimbal control instruction to the drone through the first channel.
[0010] The smart glasses are capable of sensing user gestures and generating glasses control instructions, and the smart glasses further perform corresponding operations according to the glasses control instructions.
[0011] The smart glasses are capable of sensing user gestures and generating a second flight control instruction, and the smart glasses further directly send the second flight control instruction to the drone through the first channel.
[0012] The smart glasses include a camera and a processor. The camera captures a user gesture operation image. The processor processes the gesture operation image, determines the coordinates of the gesture operation, and compares the coordinates with the imaging position of the virtual operation interface for controlling the drone displayed on the smart glasses to identify the second flight control instruction corresponding to the user gesture operation and execute the corresponding operation.
[0013] Among them, the smart glasses include a camera, a memory and a processor, the camera is used to capture user gesture movements, the memory stores multiple user gesture movement models and second flight control instructions corresponding to the multiple user gesture movement models, the processor is electrically connected to the camera and the memory, the processor is used to generate a gesture movement model according to the user gesture movements captured by the camera, and compare it with the multiple user gesture movement models stored in the memory, and when it is consistent with one of the multiple user gesture movement models stored in the memory, the second flight control instruction corresponding to the gesture movement model is triggered.
[0014] The plurality of user gesture action models stored in the memory are completed by pre-shooting by the camera and pre-defining each user gesture action model.
[0015] The smart glasses can further directly establish a third channel with the remote controller, so that the smart glasses can directly receive glasses control instructions from the remote controller through the third channel and perform corresponding operations according to the glasses control instructions.
[0016] Among them, the control system further includes a mobile terminal, and the mobile terminal directly establishes a fourth channel with the remote control. The remote control further receives the first perspective image data and / or flight status data from the drone through the second channel, and sends it to the mobile terminal for display through the fourth channel.
[0017] The mobile terminal further generates the first flight control instruction and / or gimbal control instruction according to the user input, and sends it to the remote controller through the fourth channel, and then the remote controller sends it to the drone through the second channel.
[0018] The remote controller further sends a disabling instruction to the smart glasses through the third channel to disable the control function of the smart glasses corresponding to the first flight control instruction and / or gimbal control instruction.
[0019] Among them, the control system further includes a mobile terminal, and the mobile terminal directly establishes a fourth channel with the smart glasses. The smart glasses further send the first perspective image data and / or flight status data received from the drone to the mobile terminal through the fourth channel for display.
[0020] The mobile terminal further generates the first flight control instruction and / or gimbal control instruction according to the user input, and sends it to the smart glasses through the fourth channel, and then the smart glasses send it to the drone through the first channel.
[0021] Among them, after receiving the first flight control instruction and / or gimbal control instruction from the mobile terminal, the smart glasses disable their own control functions corresponding to the first flight control instruction and / or gimbal control instruction.
[0022] The third channel and the fourth channel are wireless channels or wired channels.
[0023] Among them, the control system further includes a mobile terminal, the mobile terminal directly establishes a wireless channel with the drone, the mobile terminal receives and displays the first perspective image from the drone through the wireless channel, and the drone receives the first flight control instruction from the mobile terminal through the wireless channel.
[0024] The smart glasses directly obtain the video file of the drone from the drone through the first channel and play it back on the smart glasses.
[0025] There are one or more smart glasses.
[0026] In a second aspect, a control system for an unmanned aerial vehicle is provided, the control system comprising a unmanned aerial vehicle and smart glasses, the smart glasses being capable of directly establishing a first channel with the unmanned aerial vehicle without a relay, so as to directly receive and display first-perspective image data from the unmanned aerial vehicle through the first channel, the smart glasses being further capable of sensing user gestures, generating a first flight control instruction, and directly sending the first flight control instruction to the unmanned aerial vehicle through the first channel.
[0027] Wherein, the control system further includes a mobile terminal, and the mobile terminal directly establishes a second channel with the smart glasses. The smart glasses further send the first perspective image data and / or flight status data received by the drone to the mobile terminal through the second channel for display.
[0028] Among them, the control system further includes a mobile terminal, the mobile terminal directly establishes a wireless channel with the drone, the mobile terminal receives and displays the first perspective image from the drone through the wireless channel, and the drone receives the first flight control instruction from the mobile terminal through the wireless channel.
[0029] The smart glasses include a camera and a processor. The camera captures a user gesture operation image. The processor processes the gesture operation image, determines the coordinates of the gesture operation, and compares the coordinates with the imaging position of the virtual operation interface for controlling the drone displayed on the smart glasses to identify the first flight control instruction corresponding to the user gesture operation and execute the corresponding operation.
[0030] Among them, the smart glasses include a camera, a memory and a processor, the camera is used to capture user gestures, the memory stores multiple user gesture models and first flight control instructions corresponding to the multiple user gesture models, the processor is electrically connected to the camera and the memory, the processor is used to generate a gesture model based on the user gestures captured by the camera, and compare it with the multiple user gesture models stored in the memory, and when it is consistent with one of the multiple user gesture models stored in the memory, the first flight control instruction corresponding to the gesture model is triggered.
[0031] The plurality of user gesture action models stored in the memory are completed by pre-shooting by the camera and pre-defining each user gesture action model.
[0032] In a third aspect, a method for controlling first-person perspective flight of a drone is provided, the method comprising: controlling the smart glasses to directly establish a first channel with the drone, and controlling the smart glasses to directly receive and display first-person perspective image data from the drone through the first channel; controlling a remote control to directly establish a second channel with the drone, and controlling the remote control to directly send a first flight control instruction to the drone through the second channel.
[0033] The method further includes controlling the smart glasses to directly receive and display flight status data from the drone through the first channel.
[0034] The flight status data is displayed superimposed on the first perspective image in an augmented reality manner.
[0035] The method further includes controlling the smart glasses to sense the posture of the smart glasses and generate a gimbal control instruction, and controlling the smart glasses to further send the gimbal control instruction directly to the drone through the first channel.
[0036] The smart glasses are controlled to sense user gestures and generate a second flight control instruction, and the second flight control instruction is directly sent to the drone through the first channel.
[0037] Among them, the step of controlling the smart glasses to sense user gestures and generate a second flight control instruction includes: controlling the camera of the smart glasses to capture a user gesture operation image, processing the gesture operation image, determining the coordinates of the gesture operation, and comparing them with the imaging position of the virtual operation interface for controlling the drone displayed on the smart glasses to identify the second flight control instruction corresponding to the user gesture operation and execute the corresponding operation; or controlling the camera of the smart glasses to capture user gesture actions, generating a gesture action model according to the user gesture actions, and comparing it with multiple user gesture action models pre-stored in the smart glasses. When it is consistent with one of the multiple pre-stored user gesture action models, triggering the second flight control instruction corresponding to the gesture action model.
[0038] The pre-stored multiple user gesture action models are completed by pre-shooting with the camera of the smart glasses and defining each user gesture action model in advance.
[0039] The method further includes: controlling the smart glasses to directly establish a third channel with the remote controller, controlling the smart glasses to directly receive glasses control instructions from the remote controller through the third channel, and performing corresponding operations according to the glasses control instructions.
[0040] The method further includes: controlling the mobile terminal to directly establish a fourth channel with the remote control, and controlling the remote control to receive the first perspective image data and / or flight status data from the drone through the second channel, and sending the first perspective image data and / or flight status data to the mobile terminal for display through the fourth channel; or controlling the mobile terminal to directly establish a fourth channel with the smart glasses, and controlling the smart glasses to further send the first perspective image data and / or flight status data received from the drone to the mobile terminal for display through the fourth channel.
[0041] The method further includes: when the mobile terminal directly establishes a fourth channel with the remote control, controlling the mobile terminal to further generate the first flight control instruction and / or gimbal control instruction according to the user input, and sending it to the remote control through the fourth channel, and then the remote control sends it to the drone through the second channel; or when the mobile terminal directly establishes a fourth channel with the smart glasses, controlling the mobile terminal to further generate the first flight control instruction and / or gimbal control instruction according to the user input, and sending it to the smart glasses through the fourth channel, and then the smart glasses send it to the drone through the first channel.
[0042] The method also includes: controlling the remote control to further send a disabling instruction to the smart glasses through the third channel to disable the control function of the smart glasses corresponding to the first flight control instruction and / or gimbal control instruction; or controlling the smart glasses to disable its own control function corresponding to the first flight control instruction and / or gimbal control instruction after receiving the first flight control instruction and / or gimbal control instruction from the mobile terminal.
[0043] The third channel and the fourth channel are wireless channels or wired channels.
[0044] Among them, the control method also includes: controlling the mobile terminal to directly establish a wireless channel with the drone, the mobile terminal receiving and displaying the first perspective image from the drone through the wireless channel, and the drone receiving the first flight control instruction from the mobile terminal through the wireless channel.
[0045] The method further includes: controlling the smart glasses to directly obtain the video file of the drone from the drone through the first channel, and playing it back on the smart glasses.
[0046] There are one or more smart glasses.
[0047] In a fourth aspect, a control method for first-person perspective flight of a drone is provided, the control method comprising: controlling the smart glasses to directly establish a first channel with the drone without relaying, so as to directly receive and display first-person perspective image data from the drone through the first channel, controlling the smart glasses to further sense user gestures, and generate a first flight control instruction, and directly sending the first flight control instruction to the drone through the first channel.
[0048] Wherein, the control method further includes:
[0049] Control the mobile terminal to directly establish a second channel with the smart glasses, and control the smart glasses to further send the first perspective image data and / or flight status data received by the drone to the mobile terminal through the second channel for display; or control the mobile terminal to directly establish a wireless channel with the drone, and control the mobile terminal to receive and display the first perspective image from the drone through the wireless channel, and control the drone to receive the first flight control instruction from the mobile terminal through the wireless channel.
[0050] Among them, the controlling of the smart glasses to further sense user gestures and generate a first flight control instruction includes: controlling the camera of the smart glasses to capture user gesture movements, generating a gesture movement model according to the user gesture movements, and comparing it with multiple user gesture movement models pre-stored in the smart glasses; when it is consistent with one of the multiple pre-stored user gesture movement models, triggering the first flight control instruction corresponding to the gesture movement model.
[0051] The pre-stored multiple user gesture action models are completed by pre-shooting with the camera of the smart glasses and defining each user gesture action model in advance.
[0052] In a fifth aspect, a pair of smart glasses is provided, the smart glasses comprising: a shell, the shell comprising: a first side surface and a second side surface arranged opposite to each other, and a third side surface and a fourth side surface respectively connected to the first side surface and the second side surface and arranged opposite to each other; a first lens group and a second lens group, which are arranged on the first side surface at an interval; a first display screen and a second display screen, which are arranged on the third side surface and the fourth side surface respectively; a first optical mirror and a second optical mirror, which are arranged crosswise in the shell, wherein the first optical mirror is used to reflect the light of the virtual scene displayed on the first display screen to the first lens group, and the second optical mirror is used to reflect the image light output by the second display screen to the second lens group; a selective shading element, which is arranged on the second side surface, wherein the selective shading element can be switched between a shading state and a light-transmitting state, and when the selective shading element is in the light-transmitting state, the first optical mirror and the second optical mirror further transmit the light of the real scene outside the smart glasses to the first lens group and the second lens group, so that the user can observe the real scene and the virtual scene at the same time through the first lens group and the second lens group.
[0053] Wherein, when the selective shading element is in the shading state, the selective shading element shields the light of the real scene outside the smart glasses so that the user only observes the virtual scene.
[0054] Wherein, the first lens group and the second lens group are a combination of a concave lens and a convex lens.
[0055] Wherein, the smart glasses are connected to a drone, and the first display screen and the second display screen further display status information of the drone, so that the user can simultaneously see the status information of the drone through the first lens group and the second lens group.
[0056] Wherein, the transmittance of the selective shading element is adjustable.
[0057] Wherein, the selective shading element is a liquid crystal screen, and the liquid crystal screen adjusts the light transmittance according to the intensity of the applied voltage.
[0058] The optical axis directions of the first lens group and the second lens group are parallel to each other, and the display surfaces of the first display screen and the second display screen are parallel to the optical axis directions.
[0059] Wherein, the positions of the first lens group and the second lens group on the first side surface are adjustable.
[0060] Among them, the first optical mirror and the second optical mirror are respectively tilted relative to the optical axis directions of the first lens group and the second lens group, and the inclination angles of the first optical mirror and the second optical mirror relative to the first lens group and the second lens group can be adjusted according to the positions of the first lens group and the second lens group on the first side surface.
[0061] The smart glasses can directly establish a first channel with the drone without the need for relaying by a remote controller, so that the smart glasses directly receive and display first-perspective image data from the drone through the first channel.
[0062] The smart glasses directly receive and display flight status data from the drone through the first channel.
[0063] The flight status data is displayed superimposed on the first perspective image data in an augmented reality manner.
[0064] Among them, the virtual scene is a virtual operation interface, and the smart glasses also include a camera and a processor. The camera is arranged on the shell, and the camera is used to capture the user's gesture operation image. The processor processes the gesture operation image, determines the coordinates of the gesture operation, and compares them with the imaging position of the virtual operation interface to identify the control instructions corresponding to the user's gesture operation and perform the corresponding operation.
[0065] Among them, the smart glasses also include a camera, a memory and a processor. The camera is arranged on the shell, and the user gesture actions are captured by the camera. The memory stores multiple user gesture action models and control instructions corresponding to the multiple user gesture action models. The processor is electrically connected to the camera and the memory. The processor is used to generate a gesture action model according to the user gesture actions captured by the camera, and compare it with the multiple user gesture action models stored in the memory. When it is consistent with one of the multiple user gesture action models stored in the memory, the control instruction corresponding to the gesture action model is triggered.
[0066] The control instruction is a gimbal control instruction for controlling the gimbal of the drone, or a glasses control instruction for controlling smart glasses, or a flight control instruction for controlling the drone.
[0067] The smart glasses can further directly establish a second channel with the remote controller, so that the smart glasses can directly receive glasses control instructions from the remote controller through the second channel and perform corresponding operations according to the glasses control instructions.
[0068] Among them, the smart glasses can further directly establish a third channel with the mobile terminal, and the smart glasses further send the first perspective image data and / or flight status data received from the drone to the mobile terminal through the third channel for display.
[0069] Among them, after receiving the first flight control instruction and / or gimbal control instruction from the mobile terminal, the smart glasses disable their own control functions corresponding to the first flight control instruction and / or gimbal control instruction.
[0070] The second channel and the third channel are wireless channels or wired channels.
[0071] The smart glasses directly obtain the video file of the drone from the drone through the first channel and play it back on the smart glasses.
[0072] In the control system of the drone provided by the embodiment of the present invention, the smart glasses can directly establish a first channel with the drone without the need for relaying by the remote controller, so as to directly receive and display first-view image data from the drone through the first channel, and the remote controller can directly establish a second channel with the drone without the need for relaying by the smart glasses, so as to directly send a first flight control instruction to the drone through the second channel. In this way, since both the remote controller and the smart glasses directly establish a communication channel with the drone, there is no need to relay the image transmission and control signals, thus avoiding delays in the image transmission and control signals, improving the accuracy of drone control, and improving the safety of drone flight. BRIEF DESCRIPTION OF THE DRAWINGS
[0073] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0074] Figure 1 is a schematic structural diagram of a pair of smart glasses provided by a first embodiment of the present invention;
[0075] Figure 2 is a schematic structural diagram of a pair of smart glasses provided by a second embodiment of the present invention;
[0076] Figure 3 is a schematic structural diagram of a control system of an unmanned aerial vehicle provided by a first embodiment of the present invention;
[0077] Figure 4 is a schematic structural diagram of a control system of an unmanned aerial vehicle provided by a second embodiment of the present invention;
[0078] Figure 5 is a schematic structural diagram of a control system of an unmanned aerial vehicle provided by a third embodiment of the present invention;
[0079] Figure 6 is a flow chart of a method for controlling first-person perspective flight of a drone provided by an embodiment of the present invention;
[0080] Figure 7 This is a flow chart of another method for controlling first-person perspective flight of a drone provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0081] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0082] See also Figure 1 and Figure 2 , Figure 1 and Figure 2 They are schematic diagrams of the structure of a pair of smart glasses provided by an embodiment of the present invention. As shown in the figure, the smart glasses of this embodiment include a shell, and the shell includes:
[0083] A first side surface 11 and a second side surface 12 are oppositely disposed, and a third side surface 13 and a fourth side surface 14 are connected to the first side surface and the second side surface and are oppositely disposed;
[0084] A first lens group 15 and a second lens group 16, wherein the first lens group 15 and the second lens group 16 are spaced apart and arranged on the first side surface 11;
[0085] The first display screen 17 and the second display screen 18 are respectively arranged on the third side surface 13 and the fourth side surface 14;
[0086] The first optical mirror 19 and the second optical mirror 20 are cross-arranged in the housing, wherein the first optical mirror 19 is used to reflect the image light output by the first display screen 17 to the first lens group 15, and the second optical mirror 20 is used to reflect the image light output by the second display screen 18 to the second lens group 16;
[0087] The selective shading element 21 is disposed on the second side surface 12, wherein the selective shading element 21 can be switched between a shading state and a light-transmitting state. In this embodiment, the first optical mirror 19 and the second optical mirror 20 are both semi-reflective and semi-transparent. When the selective shading element 21 is in the light-transmitting state, the first optical mirror 19 and the second optical mirror 20 further transmit the light of the real scene outside the smart glasses to the first lens group 15 and the second lens group 16, so that the user can observe the real scene and the virtual scene at the same time through the first lens group 15 and the second lens group 16.
[0088] In a specific implementation, the first display screen 17 and the second display screen 18 can have a resolution of 1920×1080 or 2560×1440 and a refresh rate of 60 Hz, and are placed at the two ends of the smart glasses (i.e., the third side and the fourth side of the smart glasses shell), respectively, parallel to the optical axis direction of the human eye.
[0089] In a specific implementation, the first lens group 15 and the second lens group 16 are a combination of a concave lens and a convex lens. The function of the concave lens is to limit the field of view and only allow light within a certain range to pass through the lens group. The function of the convex lens is to further magnify the image and enhance the sense of immersion. They are arranged at a certain angle to the first display screen 17 and the second display screen 18, and the optimal arrangement is to place them at a 90° angle.
[0090] In one implementation, the optical axis directions of the first lens group 15 and the second lens group 16 are parallel to each other, and the display surfaces of the first display screen 17 and the second display screen 18 are parallel to the optical axis directions.
[0091] In another implementation, the positions of the first lens group 15 and the second lens group 16 on the first side surface 11 are adjustable, that is, they can move relatively on the first side surface 11 .
[0092] The first optical mirror 19 and the second optical mirror 20 are glass or organic glass lenses that can reflect part of the incident light and project part of it. For example, the transmittance can be selected as 30% and the reflectance can be 70%, or the first optical mirror 19 and the second optical mirror 20 are both transparent glass, and a layer of semi-reflective and semi-transparent film is coated on the two transparent glasses. The first optical mirror 19 and the second optical mirror 20 are respectively tilted relative to the optical axis direction of the first lens group 15 and the second lens group 16, and the tilt angles of the first optical mirror 19 and the second optical mirror 20 relative to the first lens group 15 and the second lens group 16 can be adjusted according to the positions of the first lens group 15 and the second lens group 16 on the first side surface 11. The best setting method is to place it at an angle of 45°.
[0093] It can be understood that in other embodiments, the first optical mirror 19 and the second optical mirror 20 may also be reflective mirrors, and are not limited to this embodiment.
[0094] The transmittance of the selective shading element 21 is adjustable. In one embodiment, the selective shading element 21 can be a liquid crystal screen (LCD screen), and the LCD screen adjusts the transmittance according to the applied voltage intensity. Thus, the selective shading element 21 can work in two states: one is a transparent state, and the other is a shading state. In the transparent state, the transmittance is close to 100%, and in the shading state, it is required that no light can pass through.
[0095] When the selective shading element 21 is in the shading state, light outside the smart glasses cannot enter the smart glasses, and the only light entering the lens group is the light reflected by the first optical lens 19 and the second optical lens 20 from the first display screen 17 and the second display screen 18, and the visual experience of the human eye is to immerse in the virtual reality. The selective shading element 21 can be set at a certain angle to the first display screen 17 and the second display screen 18. An optimal setting method is to place the selective shading element 21 at a 90° angle to the first display screen 17 and the second display screen 18.
[0096] When the selective shading element 21 is adjusted to be transparent, light outside the glasses can enter the glasses, pass through the first optical lens 19 and the second optical lens 20, and be superimposed on the first display screen 17 and the second display screen 18 reflected by them to produce an augmented reality effect.
[0097] The superimposed effect of the external scene and the first display screen 17 and the second display screen 18 is determined by the transmittance of the first optical lens 19 and the second optical lens 20, the external light intensity, and the brightness of the first display screen 17 and the second display screen 18. The brightness of the first display screen 17 and the second display screen 18 can be adjusted by the external light intensity to achieve a stable superimposed effect in different scenes. In addition, the user can also manually adjust the brightness of the first display screen 17 and the second display screen 18.
[0098] In one embodiment, the smart glasses are able to directly establish a first channel with the drone without the need for a remote control to relay, so that the smart glasses can directly receive and display first-perspective image data from the drone through the first channel, and display it on the first display screen 17 and the second display screen 18.
[0099] In addition, the smart glasses can also directly obtain the flight status data of the drone from the drone, and the first display screen 17 and the second display screen 18 can further overlay and display the flight status data of the drone on the basis of the first-view image data, so that the user can simultaneously see the flight status data of the drone through the first lens group 15 and the second lens group 16. The flight status data of the drone can include but is not limited to the location of the drone, the direction of the drone, the remaining power, the flight trajectory, obstacle information, the flight altitude and the flight speed, the direction of the drone head and the direction of the gimbal on the drone, operation prompts, first-view images, etc.
[0100] In another embodiment, the above virtual scene is a virtual operation interface. Figure 1 and Figure 2 The smart glasses of this embodiment also include a camera 22 and a processor (not shown) for processing images from the camera 22, wherein the camera 22 is arranged on the shell, and the user's gesture operation image is captured by the camera 22. The processor is arranged in the shell, and is used to process the gesture operation image, determine the coordinates of the gesture operation, and compare them with the imaging position of the virtual operation interface to identify the control instructions corresponding to the user's gesture operation and perform the corresponding operation.
[0101] That is to say, the operation interface is displayed on the first display screen 17 and the second display screen 18, and then reflected through the first optical mirror 19 and the second optical mirror 20 to form a virtual operation interface. The camera 22 collects the user's gesture operation image, and processes it through the processor 23 to determine the coordinates of the gesture operation, and then compares it with the imaging position of the virtual operation interface, so as to identify the control instruction corresponding to the user's gesture operation and execute the corresponding operation.
[0102] When the virtual operation interface is the virtual operation interface of the glasses menu, the control instructions here are the control instructions of the smart glasses. When the virtual operation interface is the virtual operation interface of the drone remote control, the control instructions here are the drone control instructions. The drone control instructions include gimbal control instructions and drone flight control instructions, which are respectively used to control the drone gimbal, the imaging device on the gimbal, and the flight of the drone.
[0103] In another implementation, the smart glasses can also directly recognize the user's gestures to perform corresponding operations. In this implementation, please continue to refer to Figure 1 and Figure 2The smart glasses include a camera 22, a memory and a processor (not shown). The camera 22 is arranged on the shell and is used to capture user gestures. The memory stores multiple user gesture models and control instructions corresponding to the multiple user gesture models. The memory and the processor are both arranged in the shell. The processor is electrically connected to the camera 22 and the memory. The processor is used to generate a gesture model according to the user gesture captured by the camera 22, and compare it with the multiple user gesture models stored in the memory. When it is consistent with one of the multiple user gesture models stored in the memory, the control instruction corresponding to the gesture model is triggered.
[0104] The plurality of user gesture action models stored in the memory are completed by pre-shooting by the camera and defining each user gesture action model. That is, by pre-shooting a plurality of user gesture actions, corresponding user gesture action models are generated for the plurality of user gesture action recognitions respectively, and then a unique corresponding control instruction is defined for each user gesture action model and stored in the memory.
[0105] The control instruction here may be a pan / tilt control instruction for controlling the pan / tilt of the drone, or a glasses control instruction for controlling the smart glasses, or a flight control instruction for controlling the drone. Of course, other control instructions may also be used, and the present invention does not provide examples one by one.
[0106] In addition, based on the camera 22 provided on the smart glasses, the content displayed on the first display screen 17 and the second display screen 18 can also be customized based on specific application scenarios. For example, the camera 22 can be used to capture a face image or a QR code image, etc., and a face or a QR code can be displayed on the first display screen 17 and the second display screen 18, and the processor can perform face recognition or QR code recognition, thereby realizing pedestrian detection, face recognition, and QR code recognition. Of course, the specific functions implemented are not limited to the above examples.
[0107] Among them, the smart glasses in the embodiment of the present invention can further directly establish a second channel with the remote control, so that the smart glasses directly receive glasses control instructions from the remote control through the second channel and perform corresponding operations according to the glasses control instructions.
[0108] That is to say, the menu function of the smart glasses can be realized through the button control of the remote control. For example, the display screen backlight brightness, camera settings, photo taking, video recording, etc. realized through the menu operation of the smart glasses can be completed by the five-dimensional key on the remote control. When the menu function of the smart glasses is realized through the remote control, the remote control sends a disable command to the smart glasses through the third channel to disable the menu function of the smart glasses. Specifically, it can be to disable the control function of the smart glasses corresponding to the first flight control command and the gimbal control command. That is, when the smart glasses detect that the remote control is connected, its menu function is invalid, and the smart glasses are controlled by the glasses control command of the remote control.
[0109] The menu function of the smart glasses is realized by directly controlling the smart glasses through the buttons of the remote control. Compared with controlling the menu through the buttons on the glasses, the user operation is more convenient and there is no need to constantly switch between the remote control button operation and the smart glasses button operation.
[0110] In another implementation scheme, the smart glasses can further directly establish a third channel with the mobile terminal, and the smart glasses further send the first-person perspective image data and / or flight status data received from the drone to the mobile terminal through the third channel for display.
[0111] Furthermore, the mobile terminal further generates a first flight control instruction and / or a gimbal control instruction based on the user input, and sends it to the smart glasses through the third channel, and then the smart glasses send it to the drone through the first channel.
[0112] In addition, after receiving the first flight control instruction and / or gimbal control instruction from the mobile terminal, the smart glasses disable their own control functions corresponding to the first flight control instruction and / or gimbal control instruction. That is to say, in this case, the corresponding control function is mainly realized with the control instruction from the mobile terminal. Of course, in the case where both the smart glasses and the mobile terminal can realize the corresponding control function, if there is only a control instruction from one of the two, the operation corresponding to the control instruction is directly executed. When there are control instructions from both at the same time, priority judgment can be made according to predetermined rules, and the operation corresponding to the control instruction with higher priority is executed. The predetermined rules can be set as needed, such as setting priority for instructions from smart glasses, or setting priority for control instructions from mobile terminals, which is not limited by the present invention.
[0113] In the specific application process, the drone can communicate with the smart glasses through a wireless channel to transmit and receive image transmission and control signals, or it can communicate with a mobile terminal through wireless WIFI to transmit and receive image transmission and control signals.
[0114] The second channel and the third channel mentioned above are wireless channels or wired channels.
[0115] Among them, the specific implementation scheme of the present invention is not limited to the above. In other implementation schemes, the smart glasses, remote control, and mobile terminal can be connected by wire in addition to wireless connection, for example, by USB.
[0116] In addition, the smart glasses can also directly obtain the video files of the drone from the drone through the first channel and play them back on the smart glasses.
[0117] See also Figure 3 , Figure 3 1 is a schematic diagram of the structure of the control system of the drone provided by an embodiment of the present invention. As shown in the figure, the control system of the drone provided by an embodiment of the present invention includes a remote controller 11 and smart glasses 12, wherein the smart glasses 12 can directly establish a first channel with the drone 100 without the need for relaying by the remote controller 11, so that the smart glasses 12 directly receive and display first-perspective image data from the drone through the first channel, and the remote controller 11 can directly establish a second channel with the drone 100 without the need for relaying by the smart glasses 12, so that the remote controller can directly send a first flight control instruction to the drone through the second channel.
[0118] Among them, the first flight control instruction is a control instruction for controlling the flight of the drone. For example, the first flight control instruction can be a control instruction for controlling the drone to take off, return, fly forward, backward, left, right, throttle, turn left, turn right and other flight states.
[0119] In a specific implementation, the smart glasses 12 can further directly receive and display flight status data from the drone through the first channel. The flight status data may include, but is not limited to, the location of the drone, the direction of the drone, the remaining battery, the flight trajectory, obstacle information, the flight altitude and flight speed, the direction of the drone head, and the direction of the gimbal on the drone, etc.
[0120] As a preferred implementation scheme, the smart glasses 12 in the embodiment of the present invention are smart glasses that integrate virtual reality and augmented reality. Through the smart glasses of the present invention, the flight status data is superimposed and displayed with the above-mentioned first-view image data in an augmented reality manner. That is, the first-view image data and the flight status data are displayed in a picture-in-picture manner. It can be understood that the smart glasses 12 can also be virtual reality glasses or augmented reality glasses, and are not limited to this embodiment.
[0121] As a preferred implementation scheme, the smart glasses 12 of this embodiment can sense the posture of the smart glasses 12 through a built-in posture sensor and generate a gimbal control instruction, and the smart glasses 12 further directly send the gimbal control instruction to the drone through the first channel. The posture sensor can be an inertial measurement unit (IMU) or the like.
[0122] That is to say, the posture of the smart glasses 12 can be sensed by the posture sensor of the smart glasses 12, and then converted into a corresponding gimbal control instruction for controlling the gimbal on the drone and the imaging device disposed on the gimbal, such as a camera, a camera telescope, a remote camera, and a measuring instrument. Specifically, the gimbal control instruction can be used to achieve stabilization of the imaging device, and to adjust the posture of the imaging device at will (for example, to change the inclination angle and shooting direction of the imaging device) to achieve high-quality shooting and / or photography. For example, the imaging device camera on the gimbal can be controlled downward by the smart glasses, and the imaging device camera on the gimbal can be controlled to adjust to the left by turning the smart glasses to the left, etc. The embodiments of the present invention will not be illustrated one by one.
[0123] In another implementation, the smart glasses 12 of this embodiment can sense user gestures and generate glasses control instructions, and the smart glasses 12 further perform corresponding operations according to the glasses control instructions. The glasses control instructions here can be instructions for setting the smart glasses themselves, such as adjusting the backlight brightness and resolution of the display screen of the smart glasses 12, and can also be control instructions for controlling the imaging device of the drone, such as camera settings, photo taking, video recording and other control instructions.
[0124] In other words, the menu operation of the smart glasses can be replaced by sensing user gestures through smart glasses, and the menu function of the smart glasses can be realized through gesture operations. As a specific implementation method, the smart glasses 12 include a front camera. In the augmented reality mode, the user operates by clicking the position of the glasses icon with his finger. For example, in the augmented reality mode, the operation menu of the smart glasses 12 is displayed on the screen. The user clicks on the corresponding operation menu. The camera of the smart glasses 12 captures the image of the user's click operation, identifies the coordinates of the stable position of the user's finger within a certain period of time, and compares it with the relative position of the real menu in the glasses, thereby identifying the glasses control instructions corresponding to the user's gesture operation to perform the corresponding operation.
[0125] In another implementation, the smart glasses 12 can also sense user gestures and generate a second flight control instruction. The smart glasses 12 further send the second flight control instruction directly to the drone through the first channel.
[0126] It should be noted that the second flight control instruction here is the same as the first flight control instruction, and both are control instructions for controlling the flight of the drone. For example, the second flight control instruction can be a control instruction for controlling the drone to take off, return, fly forward, backward, left, right, and other flight states, throttle, turn left, turn right, etc. In other words, the flight control of the drone can be achieved by sensing the user's gesture operation through smart glasses. Of course, it can be understood that when the drone receives the second flight control instruction from the smart glasses 12 without receiving the first flight control instruction from the remote control 11, the drone controls the flight of the drone with the second control instruction. When the drone receives the first flight control instruction from the remote control 11 without receiving the second flight control instruction from the smart glasses 12, the drone controls the flight of the drone with the first control instruction. When the flight control instructions from the smart glasses 12 and the remote control 11 are received at the same time, the drone can control the flight of the drone with a high priority control instruction after priority judgment based on the predetermined rules. Control the drone with a high priority control instruction. The predetermined rules can be pre-set as needed, such as giving priority to the control instructions of the smart glasses or the control instructions of the remote control.
[0127] In one implementation, the smart glasses include a camera and a processor. The camera captures the user's gesture operation image, and the processor processes the gesture operation image to determine the coordinates of the gesture operation, and compares them with the imaging position of the virtual operation interface for controlling the drone displayed on the smart glasses to identify the second flight control instruction corresponding to the user's gesture operation and execute the corresponding operation.
[0128] That is to say, the virtual operation interface is displayed on the display screen of the smart glasses, and then the user's gesture operation image is captured by the camera. After being processed by the processor, the coordinates of the gesture operation are determined, and then compared with the imaging position of the virtual operation interface, so as to identify that the user is operating the second flight control instruction corresponding to the operation and execute the corresponding control operation.
[0129] In another implementation, the smart glasses include a camera, a memory, and a processor. The camera is used to capture user gestures. The memory stores multiple user gesture models and second flight control instructions corresponding to the multiple user gesture models. The processor is electrically connected to the camera and the memory. The processor is used to generate a gesture model based on the user gestures captured by the camera, and compare it with the multiple user gesture models stored in the memory. When the gesture model is consistent with one of the multiple user gesture models stored in the memory, the second flight control instruction corresponding to the gesture model is triggered.
[0130] The multiple user gesture action models stored in the memory are completed by pre-shooting with a camera and defining each user gesture action model. That is, by pre-shooting multiple user gesture actions, corresponding user gesture action models are generated for each of the multiple user gesture action recognitions, and then a unique corresponding second flight control instruction is defined for each user gesture action model and stored in the memory.
[0131] In the specific application process, the drone can communicate with the smart glasses through a wireless channel to transmit and receive image transmission and control signals, or it can communicate with a mobile terminal through wireless WIFI to transmit and receive image transmission and control signals.
[0132] Furthermore, the drone control system disclosed above based on the embodiment of the present invention can be connected to multiple smart glasses for coordinated work when needed.
[0133] In addition, in a possible implementation, the smart glasses can not only play real-time videos shot by drones, but also obtain the drone's video files from the drone through wired or wireless channels and play them back on the smart glasses.
[0134] Please continue reading Figure 3 In another specific implementation scheme, the smart glasses 12 can further directly establish a third channel with the remote controller 11, so that the smart glasses 12 directly receive glasses control instructions from the remote controller 11 through the third channel, and perform corresponding operations according to the glasses control instructions. That is to say, the menu function of the smart glasses 12 can be realized through the key control of the remote controller 11, for example, the display screen backlight brightness, camera settings, photo taking, video recording, etc. realized through the menu operation of the smart glasses 12 can all be completed by the five-dimensional key on the remote controller 11. When the menu function of the smart glasses 12 is realized through the remote controller 11, the remote controller 11 sends a disable instruction to the smart glasses 12 through the third channel to disable the menu function of the smart glasses 12, specifically, it can be to disable the control function of the smart glasses 12 corresponding to the first flight control instruction and the gimbal control instruction. That is, when the smart glasses 12 detect that the remote controller 11 is connected, its menu function is invalid, and the smart glasses 12 are controlled by the glasses control instruction of the remote controller.
[0135] By directly controlling the smart glasses 12 through the buttons of the remote control 11 to realize the menu function of the smart glasses, compared with controlling the menu through the buttons on the glasses, the user operation is more convenient and there is no need to constantly switch between the button operation of the remote control 11 and the button operation of the smart glasses 12.
[0136] Please continue reading Figure 3The control system of this embodiment may further include a mobile terminal 13, and the mobile terminal 13 directly establishes a fourth channel with the remote controller 11. The remote controller 11 further sends the first-person perspective image data and / or flight status data received from the drone to the mobile terminal 13 through the fourth channel for display.
[0137] Furthermore, the mobile terminal can generate a first flight control instruction and / or a gimbal control instruction based on user input, and send it to the remote controller 11 through the fourth channel, and then the remote controller 11 sends it to the drone through the second channel, so as to control the flight of the drone and the imaging device on the drone on the mobile terminal 13.
[0138] When the remote controller 11 receives the first flight control instruction and / or the gimbal control instruction from the mobile terminal 13, the control function corresponding to the first flight control instruction and / or the gimbal control instruction can be disabled. That is to say, in this case, the corresponding control function is mainly realized with the control instruction from the mobile terminal 13. Of course, in the case where both the remote controller 11 and the mobile terminal 13 can realize the corresponding control function, if there is only a control instruction from one of them, the operation corresponding to the control instruction is directly executed. When there are control instructions from both of them at the same time, priority judgment can be made according to predetermined rules, and the operation corresponding to the control instruction with a higher priority can be executed. The predetermined rules can be set as needed, such as setting the remote controller instruction to take priority, or setting the control instruction from the mobile terminal to take priority, which is not limited by the present invention.
[0139] When the control system includes a mobile terminal 13, as another implementation, see Figure 4 , Figure 4 It is a structural diagram of another control system provided by an embodiment of the present invention. In this embodiment, the mobile terminal 13 and the smart glasses 12 directly establish a fourth channel, and the smart glasses 12 further send the first-person-view image data and / or flight status data received from the drone through the fourth channel to the mobile terminal 13 for display.
[0140] Furthermore, the mobile terminal 13 further generates a first flight control instruction and / or a gimbal control instruction according to the user input, and sends it to the smart glasses 12 through the fourth channel, and then the smart glasses 12 send it to the drone through the first channel.
[0141] Among them, when the smart glasses 12 receive the first flight control instruction and / or the gimbal control instruction from the mobile terminal 13, the control function corresponding to the first flight control instruction and / or the gimbal control instruction is disabled. That is to say, in this case, the corresponding control function is mainly realized with the control instruction from the mobile terminal. Of course, in the case where both the smart glasses 12 and the mobile terminal 13 can realize the corresponding control function, if there is only a control instruction from one of them, the operation corresponding to the control instruction is directly executed. When there are control instructions from both at the same time, priority judgment can be made according to predetermined rules, and the operation corresponding to the control instruction with higher priority is executed. The predetermined rules can be set as needed, such as setting priority for instructions from smart glasses or control instructions from mobile terminals, which is not limited by the present invention.
[0142] On the basis of the above control system provided by the present invention, it is possible to control the flight of the drone through the remote controller 11, experience the immersive first-person perspective flight through the smart glasses 12, and watch the returned images in real time through the mobile terminal. In this mode, it is possible to control the drone by one person or multiple people. For example, when a single person is in control, the user controls the flight of the drone through the remote controller 11, and experiences the immersive first-person perspective flight through the smart glasses 12. The user's menu control of the glasses, such as the display backlight brightness, camera settings, photo taking, and video recording can be completed by the five-dimensional key on the remote controller. Or when two or more people are in control, the first user controls the flight of the drone through the remote controller 11, and the second user can wear the smart glasses 12 to join or exit at any time, and the camera settings, photo taking, and video recording can be controlled by the first user or the second user. Or the first user wears the smart glasses 12 and uses the remote controller 11 to control the drone, and the second user watches the image transmission through the mobile terminal APP, and the camera settings, photo taking, and video recording can be controlled by the second user through the mobile terminal. Of course, the above are just specific examples, and the specific control implementation can be adjusted appropriately as needed.
[0143] In the control system of the drone provided in the above embodiment of the present invention, the drone, remote controller 11, smart glasses 12 and mobile terminal 13 can be connected to each other through a private wireless communication protocol, thereby achieving direct communication with each other without the need for relaying.
[0144] The specific implementation scheme of the present invention is not limited to the above. In other implementation schemes, the smart glasses 12, remote control 11, and mobile terminal 13 in the control system of the drone can be connected wirelessly or by wire, such as by USB.
[0145] In the control system of the drone provided by the above embodiment of the present invention, the smart glasses can directly establish a first channel with the drone without the need for relaying by the remote controller, so as to directly receive and display first-perspective image data from the drone through the first channel, and the remote controller can directly establish a second channel with the drone without the need for relaying by the smart glasses, so as to directly send a first flight control instruction to the drone through the second channel. In this way, since both the remote controller and the smart glasses directly establish a communication channel with the drone, there is no need to relay the image transmission and control signals, thus avoiding delays in the image transmission and control signals, improving the accuracy of drone control, and improving the safety of drone flight.
[0146] In addition, the menu function of the smart glasses can be realized through the button control of the remote control, so that the user does not need to frequently switch between the remote control buttons and the smart glasses, which makes user operation more convenient.
[0147] In addition, you can further watch the image transmission through the mobile terminal, and you can further control the drone through the mobile terminal, so that multiple people can control the drone and experience the first-person flight experience at the same time.
[0148] See also Figure 5 , Figure 5 It is a structural schematic diagram of a control system of an unmanned aerial vehicle provided by the third embodiment of the present invention. The control system of the unmanned aerial vehicle of this embodiment includes an unmanned aerial vehicle 100 and smart glasses 110. The smart glasses 110 can directly establish a first channel with the unmanned aerial vehicle 100 without relaying, so as to directly receive and display first-perspective image data from the unmanned aerial vehicle through the first channel. The smart glasses 110 can further sense user gestures, generate a first flight control instruction, and directly send the first flight control instruction to the unmanned aerial vehicle through the first channel.
[0149] Among them, the first flight control instruction is a control instruction for controlling the flight of the drone. For example, the first flight control instruction can be a control instruction for controlling the drone to take off, return, fly forward, backward, left, right, throttle, turn left, turn right and other flight states.
[0150] In one implementation scheme, the smart glasses include a camera and a processor. The camera is used to capture images of user gesture operations. The processor processes the gesture operation images, determines the coordinates of the gesture operations, and compares them with the imaging positions of the virtual operation interface for controlling the drone displayed on the smart glasses to identify the first flight control instruction corresponding to the user gesture operation and execute the corresponding operation.
[0151] That is to say, the coordinates of the gesture operation are determined by identifying the gesture operation image, and compared with the imaging position of the virtual operation interface of the drone, to determine which operation key position of the virtual operation interface the gesture operation corresponds to, thereby directly executing the flight control operation corresponding to the corresponding operation key.
[0152] In another implementation scheme, the smart glasses include a camera, a memory and a processor. The camera is used to capture user gestures. The memory stores multiple user gesture models and first flight control instructions corresponding to the multiple user gesture models. The processor is electrically connected to the camera and the memory. The processor is used to generate a gesture model based on the user gestures captured by the camera, and compare it with the multiple user gesture models stored in the memory. When it is consistent with one of the multiple user gesture models stored in the memory, the first flight control instruction corresponding to the gesture model is triggered.
[0153] The multiple user gesture action models stored in the memory are completed by pre-shooting with a camera and defining each user gesture action model. That is, by pre-shooting multiple user gesture actions, corresponding user gesture action models are generated for each of the multiple user gesture action recognitions, and then a unique first flight control instruction is defined for each user gesture action model and stored in the memory.
[0154] In a specific implementation, the smart glasses 110 may further directly receive and display flight status data from the drone through the first channel. The flight status data may include, but is not limited to, the location of the drone, the direction of the drone, the remaining battery, the flight trajectory, obstacle information, the flight altitude and the flight speed, the direction of the drone's head, the direction of the gimbal on the drone, etc.
[0155] As a preferred implementation scheme, the smart glasses 110 in the embodiment of the present invention are smart glasses that integrate virtual reality and augmented reality. Through the smart glasses of the present invention, the flight status data is superimposed and displayed with the above-mentioned first-view image data in an augmented reality manner. That is, the first-view image data and the flight status data are displayed in a picture-in-picture manner. It can be understood that the smart glasses 110 can also be virtual reality glasses or augmented reality glasses, and are not limited to this embodiment.
[0156] As a preferred implementation scheme, the smart glasses 110 of this embodiment can sense the posture of the smart glasses 110 through the posture sensor arranged therein and generate a gimbal control instruction. The smart glasses 110 further send the gimbal control instruction directly to the drone through the first channel.
[0157] That is to say, the posture of the smart glasses 110 can be sensed by the posture sensor of the smart glasses 110, and then converted into a corresponding gimbal control instruction for controlling the gimbal on the drone and the imaging device disposed on the gimbal, such as a camera, a camera telescope, a remote camera, and a measuring instrument. Specifically, the gimbal control instruction can be used to stabilize the imaging device, adjust the posture of the imaging device at will (for example, change the inclination angle and shooting direction of the imaging device), so as to achieve high-quality shooting and / or photography. For example, the camera of the imaging device on the gimbal can be controlled downward by the smart glasses, and the camera of the imaging device on the gimbal can be controlled to adjust leftward by turning the smart glasses to the left, etc. The embodiments of the present invention will not be illustrated one by one.
[0158] In another implementation, the smart glasses 110 of this embodiment can sense user gestures and generate glasses control instructions, and the smart glasses 110 further perform corresponding operations according to the glasses control instructions. The glasses control instructions here can be instructions for setting the smart glasses themselves, such as adjusting the backlight brightness and resolution of the display screen of the smart glasses 110, and can also be control instructions for controlling the imaging device of the drone, such as camera settings, photo taking, video recording and other control instructions.
[0159] In other words, the menu operation of the smart glasses can be replaced by sensing user gestures through smart glasses, and the menu function of the smart glasses can be realized through gesture operations. As a specific implementation method, the smart glasses 110 include a front camera. In the augmented reality mode, the user operates by clicking the position of the glasses icon with his finger. For example, in the augmented reality mode, the operation menu of the smart glasses 110 is displayed on the screen. The user clicks on the corresponding operation menu. The camera of the smart glasses 110 captures the image of the user's click operation, identifies the coordinates of the stable position of the user's finger within a certain period of time, and compares it with the relative position of the real menu in the glasses, thereby identifying the glasses control instructions corresponding to the user's gesture operation to perform the corresponding operation.
[0160] Please continue reading Figure 5 The control system in this embodiment may further include a mobile terminal 120, and the mobile terminal 120 directly establishes a second channel with the smart glasses 110. The smart glasses 110 further sends the first-perspective image data and / or flight status data received from the drone 100 to the mobile terminal 120 through the second channel for display.
[0161] Furthermore, the mobile terminal 120 further generates a first flight control instruction and / or a gimbal control instruction according to the user input, and sends it to the smart glasses 110 through the second channel, and then the smart glasses 110 send it to the drone 100 through the first channel.
[0162] When the smart glasses 110 receive the first flight control instruction and / or the gimbal control instruction from the mobile terminal 120, the control function corresponding to the first flight control instruction and / or the gimbal control instruction is disabled. In other words, in this case, the control instruction from the mobile terminal is mainly used to implement the corresponding control function.
[0163] In a specific application process, the drone 100 can communicate with the smart glasses 110 through a wireless channel to transmit and receive image transmission and control signals, or can communicate with the mobile terminal 120 through wireless WIFI to transmit and receive image transmission and control signals.
[0164] Furthermore, based on the drone control system disclosed above in the embodiment of the present invention, multiple smart glasses 110 can be connected to work together when needed.
[0165] In addition, in a possible implementation, the smart glasses 110 can not only play real-time videos shot by the drone, but also obtain the drone's video files from the drone through a wired or wireless channel and play them back on the smart glasses 110.
[0166] The control system of the drone provided by the above embodiment of the present invention directly communicates with the drone through smart glasses, and the smart glasses can directly control the drone without any relay. This can avoid the delay of image transmission and control signals, improve the accuracy of drone control, and improve the safety of drone flight.
[0167] Furthermore, through access to the mobile terminal 120, it is more convenient for users to view the image transmission, and the drone and smart glasses can be controlled through operations on the mobile terminal, which is convenient for users to operate.
[0168] In the drone control system provided by the above embodiment of the present invention, the drone 100, the smart glasses 110 and the mobile terminal 120 can be connected to each other through a private wireless communication protocol, thereby achieving direct communication with each other without the need for relaying.
[0169] The specific implementation scheme of the present invention is not limited to the above. In other implementation schemes, the smart glasses 110 and the mobile terminal 120 in the control system of the drone can be connected wirelessly or by wire, such as by USB.
[0170] See also Figure 6 , Figure 6 is a flow chart of a method for controlling a drone's first-person perspective flight provided by an embodiment of the present invention. The method for controlling a drone's first-person perspective flight in this embodiment is based on the above Figure 1-2 Smart glasses and Figure 3 , 4 As shown in the figure, the control method of the first visual flight of the unmanned aerial vehicle in this embodiment includes the following steps:
[0171] S101: Control the smart glasses to directly establish a first channel with the drone, and control the smart glasses to directly receive and display first visual image data from the drone through the first channel.
[0172] The smart glasses are controlled to establish a connection with the drone through a wireless private communication protocol, so that the smart glasses can communicate directly with the drone without the need for a remote controller for relay.
[0173] Among them, in another implementation scheme, the drone can also communicate with the mobile terminal via wireless WIFI to send and receive image transmission and control signals.
[0174] Among them, multiple smart glasses can be connected to work together as needed.
[0175] In a specific implementation, the control method further includes: controlling the smart glasses to further directly receive and display flight status data from the drone through the first channel. The flight status data may include, but is not limited to, the location of the drone, the direction of the drone, the remaining power, the flight trajectory, obstacle information, the flight altitude and the flight speed, the direction of the drone head, and the direction of the gimbal on the drone, etc.
[0176] As a preferred implementation scheme, the smart glasses in this embodiment are smart glasses that integrate virtual reality and augmented reality. Through the smart glasses of the present invention, the flight status data is superimposed and displayed with the above-mentioned first-view image data in an augmented reality manner. That is, the first-view image data and the flight status data are displayed in a picture-in-picture manner. It can be understood that the smart glasses can also be virtual reality glasses or augmented reality glasses, and are not limited to this embodiment.
[0177] In addition, the control method of this embodiment also includes controlling the smart glasses to sense the posture of the smart glasses through a built-in posture sensor, generating a gimbal control instruction, and controlling the smart glasses to further send the gimbal control instruction directly to the drone through the first channel. The posture sensor may be an inertial measurement unit (IMU) or the like.
[0178] That is to say, the posture of the smart glasses can be sensed by the posture sensor of the smart glasses, and then converted into a corresponding gimbal control instruction for controlling the gimbal on the drone and the imaging device arranged on the gimbal, such as a camera, a camera telescope, a remote camera, and a measuring instrument. Specifically, the gimbal control instruction can be used to achieve stabilization of the imaging device, and to adjust the posture of the imaging device at will (for example, change the inclination angle and shooting direction of the imaging device) to achieve high-quality shooting and / or photography. For example, the imaging device camera on the gimbal can be controlled downward by the smart glasses, and the imaging device camera on the gimbal can be controlled to adjust to the left by turning the smart glasses to the left, etc. The embodiments of the present invention will not be illustrated one by one.
[0179] In another implementation, the control method of the present invention further includes controlling the smart glasses to sense the user's gestures and generating glasses control instructions, and controlling the smart glasses to further perform corresponding operations according to the glasses control instructions. The glasses control instructions here can be instructions for setting the smart glasses themselves, such as adjusting the backlight brightness and resolution of the display screen of the smart glasses, and can also be control instructions for controlling the drone imaging device, such as camera settings, photo taking, video recording and other control instructions.
[0180] In other words, the menu operation of the smart glasses can be replaced by sensing user gestures through smart glasses, and the menu function of the smart glasses can be realized through gesture operations. As a specific implementation method, the smart glasses include a front camera. In augmented reality mode, the user operates by clicking the position of the glasses icon with his finger. For example, in augmented reality mode, the operation menu of the smart glasses is displayed on the screen. The user clicks on the corresponding operation menu. The camera of the smart glasses captures the image of the user's click operation, identifies the coordinates of the user's finger position that is stable within a certain period of time, and compares it with the relative position of the real menu in the glasses, thereby identifying the glasses control instructions corresponding to the user's gesture operation and performing the corresponding operation.
[0181] In the specific implementation process, the smart glasses can not only play real-time videos shot by drones, but also obtain the drone's video files from the drone through wired or wireless channels and play them back on the smart glasses.
[0182] S102: Control the remote controller to directly establish a second channel with the drone, and control the remote controller to directly send a first flight control instruction to the drone through the second channel.
[0183] Among them, the first flight control instruction is a control instruction for controlling the flight of the drone. For example, the first flight control instruction can be a control instruction for controlling the drone to take off, return, fly forward, backward, left, right, throttle, turn left, turn right and other flight states.
[0184] Among them, in another implementation, the method of this embodiment also includes controlling the smart glasses to sense the user's gesture and generate a second flight control instruction, and controlling the smart glasses to further send the second flight control instruction directly to the drone through the first channel.
[0185] Among them, it should be noted that the second flight control instruction here is the same as the first flight control instruction, and both are control instructions for controlling the flight of the drone. For example, the first flight control instruction can be a control instruction for controlling the drone to take off, return, fly forward, backward, left, right, and other flight states, throttle, turn left, turn right, etc. In other words, the flight control of the drone can be achieved by sensing the user's gesture operation through smart glasses. Of course, it can be understood that when the drone receives the second flight control instruction from the smart glasses without receiving the first flight control instruction from the remote control, the drone is controlled to fly with the second control instruction. When the drone receives the first flight control instruction from the remote control without receiving the second flight control instruction from the smart glasses, the drone is controlled to fly with the first control instruction. When the flight control instructions from the smart glasses and the remote control are received at the same time, the drone can control the flight of the drone with the control instruction with a high priority after priority judgment based on the predetermined rules. The predetermined rules can be preset as needed, such as giving priority to the control instructions of the smart glasses or the control instructions of the remote control.
[0186] In another specific implementation scheme, the method of this embodiment also includes controlling the smart glasses to directly establish a third channel with the remote control, so that the smart glasses directly receive glasses control instructions from the remote control through the third channel, and perform corresponding operations according to the glasses control instructions. In other words, the menu function of the smart glasses can be realized through the key control of the remote control, such as the display screen backlight brightness, camera settings, photo taking, video recording, etc. realized by the menu operation of the smart glasses can be completed by the five-dimensional key on the remote control. When the menu function of the smart glasses is realized by the remote control, the remote control sends a disable instruction to the smart glasses through the third channel to disable the menu function of the smart glasses, specifically, it can be to disable the control function of the smart glasses corresponding to the first flight control instruction and the gimbal control instruction. That is, when the smart glasses detect that a remote control is connected, its menu function is invalid, and the smart glasses are controlled by the glasses control instruction of the remote control.
[0187] Directly controlling the smart glasses through the buttons of the remote control to realize the menu function of the smart glasses is more convenient for users to operate compared with controlling the menu through the buttons on the smart glasses. There is no need to constantly switch between the remote control button operation and the smart glasses button operation.
[0188] In this embodiment, the above steps S101 and S102 are numbered only to distinguish different steps, and are not used to limit the order of implementation of the specific steps. In fact, the above steps S101 and S102 in the embodiment of the present invention do not strictly distinguish the order, and even in a possible implementation, the two steps can be executed simultaneously.
[0189] In another implementation, the control method of this embodiment further includes controlling the mobile terminal to directly establish a fourth channel with the remote control, and controlling the remote control to receive first-perspective image data and / or flight status data from the drone through the second channel, and sending it to the mobile terminal for display through the fourth channel.
[0190] When the mobile terminal directly establishes the fourth channel with the remote control, the mobile terminal can be controlled to further generate the first flight control instruction and / or gimbal control instruction according to the user input, and send it to the remote control according to the fourth channel, and then the remote control sends it to the drone through the second channel.
[0191] When the remote controller receives the first flight control instruction and / or gimbal control instruction from the mobile terminal, it can disable its own control function corresponding to the first flight control instruction and / or gimbal control instruction. That is to say, in this case, the corresponding control function is mainly realized with the control instruction from the mobile terminal. Of course, in the case where both the remote controller and the mobile terminal can realize the corresponding control function, if there is only a control instruction from one of them, the operation corresponding to the control instruction is directly executed. When there are control instructions from both of them at the same time, priority judgment can be made according to predetermined rules, and the operation corresponding to the control instruction with higher priority is executed. The predetermined rules can be set as needed, such as setting the remote controller instruction priority, or setting the control instruction from the mobile terminal priority, which is not limited by the present invention.
[0192] In another implementation, the mobile terminal is controlled to directly establish a fourth channel with the smart glasses, and the smart glasses are controlled to further send the first-person-view image data and / or flight status data received from the drone to the mobile terminal through the fourth channel for display.
[0193] When the mobile terminal and the smart glasses directly establish the fourth channel, the mobile terminal is controlled to further generate a first flight control instruction and / or a gimbal control instruction based on the user input, and send it to the smart glasses through the fourth channel, and then the smart glasses send it to the drone through the first channel.
[0194] The third channel and the fourth channel mentioned above may be wireless channels or wired channels.
[0195] Among them, when the smart glasses receive the first flight control instruction and / or the gimbal control instruction from the mobile terminal, the control function corresponding to the first flight control instruction and / or the gimbal control instruction is disabled. That is to say, in this case, the corresponding control function is mainly realized with the control instruction from the mobile terminal. Of course, in the case where both the smart glasses and the mobile terminal can realize the corresponding control function, if there is only a control instruction from one of them, the operation corresponding to the control instruction is directly executed. When there are control instructions from both at the same time, priority judgment can be made according to pre-set rules, and the operation corresponding to the control instruction with higher priority is executed. The predetermined rules can be set as needed, such as setting priority for instructions from smart glasses, or setting priority for control instructions from mobile terminals, which is not limited by the present invention.
[0196] Among them, in the above embodiments, the drone, remote control, smart glasses and mobile terminal can be connected to each other through a private wireless communication protocol, so that they can communicate directly with each other without relaying.
[0197] In another implementation scheme, the mobile terminal can also be controlled to directly establish a wireless channel with the drone, the mobile terminal receives and displays the first-perspective image from the drone through the wireless channel, and the drone receives the first flight control instruction from the mobile terminal through the wireless channel.
[0198] Of course, the specific implementation scheme of the present invention is not limited to this. In other implementation schemes, the smart glasses, the remote control and the mobile terminal can be connected in a wired manner in addition to wireless connection, such as via USB.
[0199] In addition, in another implementation scheme, the control method also includes controlling the smart glasses to directly obtain the video file of the drone from the drone through the first channel, and play it back on the smart glasses.
[0200] In the specific implementation process, multiple smart glasses can be controlled and connected to work together as needed.
[0201] The control method and system for the first-person perspective flight of a drone provided by the above embodiments of the present invention, wherein the smart glasses can directly establish a first channel with the drone without the need for relaying by the remote controller, so as to directly receive and display first-person perspective image data from the drone through the first channel, and the remote controller can directly establish a second channel with the drone without the need for relaying by the smart glasses, so as to directly send a first flight control instruction to the drone through the second channel. In this way, since both the remote controller and the smart glasses directly establish a communication channel with the drone, there is no need to relay the image transmission and control signals, thus avoiding delays in the image transmission and control signals, improving the accuracy of drone control, and improving the safety of drone flight.
[0202] In addition, the menu function of the smart glasses can be realized through the button control of the remote control, so that the user does not need to frequently switch between the remote control buttons and the smart glasses, which makes user operation more convenient.
[0203] In addition, you can further watch the image transmission through the mobile terminal, and you can further control the drone through the mobile terminal, so that multiple people can control the drone and experience the first-person flight experience at the same time.
[0204] See also Figure 7 , Figure 7 is a flow chart of another method for controlling first-person perspective flight of a drone provided by an embodiment of the present invention. The method for controlling first-person perspective flight of a drone in this embodiment is based on the above Figure 1-2 Smart glasses and Figure 5 As shown in the figure, the control method of the drone first visual flight in this embodiment includes the following steps:
[0205] S201: Control the smart glasses to directly establish a first channel with the drone without relaying, so as to directly receive and display first-perspective image data from the drone through the first channel.
[0206] In the specific implementation process, the control method of this embodiment may also include: controlling the smart glasses to further directly receive and display flight status data from the drone through the first channel. The flight status data may include but is not limited to the location of the drone, the direction of the drone, the remaining power, the flight trajectory, obstacle information, the flight altitude and flight speed, the direction of the drone's head, the direction of the gimbal on the drone, etc.
[0207] Among them, the smart glasses can be smart glasses that integrate virtual reality and augmented reality, and can display the flight status data in an augmented reality manner by superimposing the first-view image data. That is, the first-view image data and the flight status data are displayed in a picture-in-picture manner. It can be understood that the smart glasses can also be virtual reality glasses or augmented reality glasses, and are not limited to this embodiment.
[0208] S202: Control the smart glasses to further sense the user's gesture, generate a first flight control instruction, and directly send the first flight control instruction to the drone through the first channel.
[0209] Among them, the first flight control instruction is a control instruction for controlling the flight of the drone. For example, the first flight control instruction can be a control instruction for controlling the drone to take off, return, fly forward, backward, left, right, throttle, turn left, turn right and other flight states.
[0210] The embodiment of the present invention provides two possible implementations of sensing the user's gesture and generating the first flight control instruction, which are as follows:
[0211] The first possible implementation method is: controlling the camera of the smart glasses to capture the user's gesture operation image, processing the gesture operation image, determining the coordinates of the gesture operation, and comparing them with the imaging position of the virtual operation interface for controlling the drone displayed on the smart glasses to identify the first flight control instruction corresponding to the user's gesture operation and execute the corresponding operation.
[0212] That is to say, the coordinates of the gesture operation are determined by identifying the gesture operation image, and compared with the imaging position of the virtual operation interface of the drone, to determine which operation key position of the virtual operation interface the gesture operation corresponds to, thereby directly executing the flight control operation corresponding to the corresponding operation key.
[0213] The second possible implementation method is: controlling the camera of the smart glasses to capture user gestures, generating a gesture action model based on the user gesture actions, and comparing it with multiple user gesture action models pre-stored in the smart glasses. When it is consistent with one of the multiple pre-stored user gesture action models, triggering the first flight control instruction corresponding to the gesture action model.
[0214] The pre-stored multiple user gesture action models are completed by pre-shooting with the camera of the smart glasses and defining each user gesture action model. That is, by pre-shooting multiple user gesture actions, corresponding user gesture action models are generated for each of the multiple user gesture action recognitions, and then a unique first flight control instruction is defined for each user gesture action model and stored in the memory.
[0215] In one implementation, the smart glasses can sense the posture of the smart glasses through the posture sensor set therein, generate a gimbal control instruction, and control the smart glasses to further send the gimbal control instruction directly to the drone through the first channel.
[0216] That is to say, the posture of the smart glasses can be sensed by the posture sensor of the smart glasses, and then converted into a corresponding gimbal control instruction for controlling the gimbal on the drone and the imaging device arranged on the gimbal, such as a camera, a camera telescope, a remote camera, and a measuring instrument. Specifically, the gimbal control instruction can be used to achieve stabilization of the imaging device, and to adjust the posture of the imaging device at will (for example, change the inclination angle and shooting direction of the imaging device) to achieve high-quality shooting and / or photography. For example, the imaging device camera on the gimbal can be controlled downward by the smart glasses, and the imaging device camera on the gimbal can be controlled to adjust to the left by turning the smart glasses to the left, etc. The embodiments of the present invention will not be illustrated one by one.
[0217] In another implementation, the smart glasses can also be controlled to sense user gestures and generate glasses control instructions, and perform corresponding operations according to the glasses control instructions. The glasses control instructions here can be instructions for setting the smart glasses themselves, such as adjusting the backlight brightness and resolution of the display screen of the smart glasses, or control instructions for controlling the imaging device of the drone, such as camera settings, photo taking, video recording, etc.
[0218] In other words, the menu operation of the smart glasses can be replaced by sensing user gestures through smart glasses, and the menu function of the smart glasses can be realized through gesture operations. As a specific implementation method, the smart glasses include a front camera. In augmented reality mode, the user operates by clicking the position of the glasses icon with his finger. For example, in augmented reality mode, the operation menu of the smart glasses is displayed on the screen. The user clicks on the corresponding operation menu. The camera of the smart glasses captures the image of the user's click operation, identifies the coordinates of the user's finger position that is stable within a certain period of time, and compares it with the relative position of the real menu in the glasses, thereby identifying the glasses control instructions corresponding to the user's gesture operation and performing the corresponding operation.
[0219] Among them, the control method of this embodiment may further include: controlling the mobile terminal to directly establish a second channel with the smart glasses, and controlling the smart glasses to further send the first-perspective image data and / or flight status data received from the drone through the second channel to the mobile terminal for display.
[0220] Furthermore, the control mobile terminal further generates a first flight control instruction and / or a gimbal control instruction according to the user input, and sends it to the smart glasses through the second channel, and then the smart glasses send it to the drone through the first channel.
[0221] In another implementation scheme, the control method may also include: controlling the mobile terminal to directly establish a wireless channel with the drone, and controlling the mobile terminal to receive and display a first-perspective image from the drone through the wireless channel, and controlling the drone to receive a first flight control instruction from the mobile terminal through the wireless channel.
[0222] That is to say, the drone can communicate with the smart glasses through wireless channels to transmit and receive image transmission and control signals. It can also communicate with the mobile terminal through wireless WIFI to transmit and receive image transmission and control signals.
[0223] Furthermore, in the above-mentioned control method of the first-person perspective flight of the drone, multiple smart glasses can be connected to work together when necessary.
[0224] In addition, the smart glasses can be controlled to obtain the drone's video files from the drone through a wired or wireless channel and play them back on the smart glasses.
[0225] Among them, in the above control method, the smart glasses and the mobile terminal can be connected by wire in addition to wireless connection, for example, they can be connected by USB.
[0226] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A control system for an unmanned aerial vehicle, characterized in that: The control system includes a remote controller and smart glasses, wherein the smart glasses can directly establish a first channel with the drone without the remote controller being relayed, so that the smart glasses directly receive and display image data from the drone through the first channel, and the remote controller can establish a second channel with the drone, so that the remote controller can send a first flight control instruction to the drone through the second channel; The smart glasses can further directly establish a third channel with the remote control so that the smart glasses can directly receive a first glasses control instruction from the remote control through the third channel and perform corresponding operations according to the first glasses control instruction, wherein the first glasses control instruction includes an instruction for controlling the smart glasses to execute a menu function in a virtual operation interface of the smart glasses.
2. The control system according to claim 1, characterized in that: The smart glasses can sense the posture of the smart glasses and generate a gimbal control instruction, and the smart glasses further send the gimbal control instruction directly to the drone through the first channel.
3. The control system according to claim 1, characterized in that: The smart glasses can sense user gestures and generate second glasses control instructions, and the smart glasses further perform corresponding operations according to the second glasses control instructions.
4. The control system according to claim 1, characterized in that: The smart glasses can sense user gestures and generate a second flight control instruction. The smart glasses further send the second flight control instruction directly to the drone through the first channel.
5. The control system according to claim 1, characterized in that: The control system further includes a mobile terminal, which directly establishes a fourth channel with the remote controller. The remote controller further receives the image data and / or flight status data from the drone through the second channel, and sends it to the mobile terminal for display through the fourth channel.
6. The control system according to claim 5, characterized in that: The mobile terminal further generates the first flight control instruction and / or gimbal control instruction according to the user input, and sends it to the remote controller through the fourth channel, and then the remote controller sends it to the drone through the second channel.
7. The control system according to claim 1, characterized in that: The remote controller further sends a disabling instruction to the smart glasses through the third channel to disable the control function of the smart glasses corresponding to the first flight control instruction and / or gimbal control instruction.
8. The control system according to claim 1, characterized in that: The control system further includes a mobile terminal, and the mobile terminal directly establishes a fourth channel with the smart glasses. The smart glasses further send the image data and / or flight status data received from the drone to the mobile terminal through the fourth channel for display.
9. The control system according to claim 8, characterized in that: The mobile terminal further generates the first flight control instruction and / or gimbal control instruction according to the user input, and sends it to the smart glasses through the fourth channel, and then the smart glasses send it to the drone through the first channel.
10. The control system according to claim 8, characterized in that: After receiving the first flight control instruction and / or gimbal control instruction from the mobile terminal, the smart glasses disable their own control functions corresponding to the first flight control instruction and / or gimbal control instruction.
11. The control system according to claim 1, characterized in that: The image data includes first-perspective image data.
12. The control system according to claim 1, characterized in that: The remote controller can directly establish the second channel with the drone without the need for relaying by the smart glasses.
