Unmanned aerial vehicle detection system based on eye movement tracking control and unmanned aerial vehicle control method

Abstract

The invention discloses an unmanned aerial vehicle detection system based on eye movement tracking control and an unmanned aerial vehicle control method. The system comprises a head-mounted eyeball tracking display control device, a wireless data transmission device and an unmanned aerial vehicle device, the head-mounted eyeball tracking display control device comprises a camera module and an instruction generation module, and the camera module is used for collecting eyeball images of a user and sending the eyeball images to the instruction generation module; the instruction generation module is used for recognizing the relative position between the pupil and the eye socket according to the eyeball image and generating a motion control instruction according to the relative position; the wireless data transmission equipment is connected with the instruction generation module and is used for acquiring the motion control instruction and performing data communication with the unmanned aerial vehicle equipment based on a wireless signal so as to send the motion control instruction to the unmanned aerial vehicle equipment; and the unmanned aerial vehicle equipment is used for moving based on the motion control instruction. According to the scheme, the wearing comfort of the eye movement monitoring equipment can be remarkably enhanced, and the control accuracy of the unmanned aerial vehicle equipment is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of motion detection, and in particular to an unmanned aerial vehicle detection system and an unmanned aerial vehicle control method based on eye tracking control. Background Art

[0002] With the rapid development of multi-rotor unmanned aerial vehicle technology, unmanned aerial vehicles are widely used in various fields of production and life, such as using unmanned aerial vehicles for aerial photography, or using unmanned aerial vehicles for agricultural production. Therefore, it is very important to accurately control the flight state of unmanned aerial vehicles. At present, eye control technology is an emerging interactive method. It realizes interactive control of unmanned aerial vehicles by wearing it on the head of the unmanned aerial vehicle operator, sensing and analyzing human eye movements. At the same time, eye control of unmanned aerial vehicles can improve the user's operating efficiency and reaction speed, reduce the risk of misoperation, and thus improve the control ability, help users quickly and accurately lock the target, and can make fine adjustments and operations through eye movements. In addition, eye tracking devices can also monitor the user's physiological state, such as fatigue, anxiety, etc., and judge their concentration and intention, so as to adjust task allocation and response strategies. Therefore, the control of unmanned aerial vehicles based on eye control has become a key area of ​​concern for various drone manufacturers.

[0003] However, the current eye movement monitoring equipment integrates all the equipment used to control the unmanned aerial vehicle, such as the human eye information collector, the control command generating device and the remote signal sending device, into the head-mounted device, resulting in the eye movement monitoring equipment being too large and heavy, seriously affecting the comfort of the wearer; at the same time, the existing eye movement monitoring control system mostly collects the blinking frequency of the unmanned aerial vehicle operator and generates a control signal based on the blinking frequency; but the above method is prone to generate erroneous control signals due to the operator's unconscious blinking behavior, which is very prone to misoperation, and the operation method is complicated, the interactive experience is poor, and the control accuracy of the unmanned aerial vehicle is seriously reduced. Summary of the invention

[0004] In view of this, the present application provides a drone detection system and a drone control method based on eye tracking control, the main purpose of which is to solve the technical problems of low wearing comfort of eye movement monitoring equipment and poor control accuracy of unmanned aerial vehicles.

[0005] According to a first aspect of the present invention, there is provided a drone detection system based on eye tracking control, the system comprising: a head-mounted eye tracking display and control device, a wireless data transmission device and a drone device, wherein the head-mounted eye tracking display and control device comprises a camera module and an instruction generation module connected to the camera module, the camera module is used to collect the eye image of the user in real time when the head-mounted eye tracking display and control device is worn on the user's head, and send the eye image to the instruction generation module, the instruction generation module is used to identify the relative position between the pupil and the eye socket according to the eye image, and generate a motion control instruction according to the relative position;

[0006] The wireless data transmission device is connected to the instruction generation module, and is used to obtain the motion control instruction, and perform data communication with the drone device based on wireless signals to send the motion control instruction to the drone device;

[0007] The drone device is used to move based on the motion control instruction.

[0008] Optionally, the system also includes a multi-source fusion camera module device; the multi-source fusion camera module device is arranged at a preset camera installation position on the drone device, and the multi-source fusion camera module device is used to collect the environmental image of the location of the drone device, and send the environmental image to the wireless data transmission device based on a wireless signal; the wireless data transmission device is also used to obtain the environmental image from the multi-source fusion camera module device, and send the environmental image to the head-mounted eye tracking display and control device; the head-mounted eye tracking display and control device also includes an image display module, and the image display module is used to obtain the environmental image and display the environmental image of the location of the drone device to the user when the head-mounted eye tracking display and control device is worn on the user's head.

[0009] Optionally, the multi-source fusion camera module device includes: a visible light camera for collecting visible light environment images; a low-light camera for collecting light-enhanced environment images when the UAV device is in a low-light environment; an infrared camera for collecting infrared light environment images; wherein the camera shooting areas of the visible light camera, the low-light camera and the infrared camera are the same.

[0010] Optionally, the multi-source fusion camera module device is also used to: collect an infrared light environment image of the camera shooting area based on the infrared camera; obtain the light intensity value of the location of the drone device; compare the light intensity value with a preset critical light intensity value to determine whether the light intensity value is greater than or equal to the critical light intensity value; if the light intensity value is greater than or equal to the critical light intensity value, collect the visible light environment image of the camera shooting area based on the visible light camera, and superimpose the visible light environment image with the infrared light environment image to obtain the environment image; if the light intensity value is less than the critical light intensity value, obtain the light enhanced environment image of the camera shooting area based on the low-light camera, and superimpose the light enhanced environment image with the infrared light environment image to obtain the environment image; send the environment image after the image superposition to the wireless data transmission device.

[0011] Optionally, the wireless signal includes a mobile communication network signal and a wireless network transmission signal; the wireless data transmission device is also used to: monitor the mobile communication network signal strength of the mobile communication network signal and the wireless network transmission signal strength of the wireless network transmission signal; compare the mobile communication network signal strength with the wireless network transmission signal strength; when the mobile communication network signal strength is greater than the wireless network transmission signal strength, determine the mobile communication network signal as the wireless signal, so that data communication is performed between the wireless data transmission device and the drone device through the mobile communication network signal; when the mobile communication network signal strength is less than or equal to the wireless network transmission signal strength, determine the wireless network transmission signal as the wireless signal, so that data communication is performed between the wireless data transmission device and the drone device through the wireless network transmission signal.

[0012] Optionally, the image display module is an augmented reality display device, which is arranged at a first position on the inner side of a head-mounted mask of the head-mounted eye tracking display and control device.

[0013] Optionally, the camera module is an infrared image camera, and the infrared image camera is arranged at a second position on the inner side of the head-mounted mask.

[0014] According to a second aspect of the present invention, there is provided a drone control method, which is applied to a command generation module in the drone detection system based on eye tracking control as described above, and the method comprises:

[0015] Acquire multiple frames of eyeball images collected within a continuous preset time period, and perform grayscale processing on each frame of the eyeball image to obtain an eye grayscale image corresponding to each frame of the eyeball image;

[0016] Combining multiple frames of eye grayscale images into an eye grayscale image group, and determining the relative position between the pupil and the eye socket in each frame of the eye grayscale image;

[0017] Determining whether the relative positions between the pupil and the eye socket corresponding to each frame of the eye grayscale image group are the same;

[0018] If the relative position between the pupil and the eye socket corresponding to each frame of the eye grayscale image group is the same, a preset motion control instruction corresponding to the relative position is determined, and the preset motion control instruction is determined as the motion control instruction used to instruct the movement of the drone.

[0019] Optionally, determining the relative position between the pupil and the eye socket in the eye grayscale image includes: inputting the eye grayscale image into a pre-trained eye structure recognition model, determining the pupil contour of the pupil and the orbital contour of the eye socket in the eye grayscale image; determining the pupil position of the pupil contour in the eye grayscale image, and the orbital position of the orbital contour in the eye grayscale image; and determining the relative position between the pupil and the eye socket based on the pupil position and the orbital position.

[0020] Optionally, determining the pupil position of the pupil contour in the eye grayscale image includes: determining a pupil center of the pupil in the pupil contour; determining a pupil center position of the pupil center in the eye grayscale image, and determining the pupil center position as the pupil position.

[0021] The present invention provides a drone detection system and drone control method based on eye tracking control, wherein a camera module for acquiring a user's eye image and an instruction generation module for identifying the relative position between the pupil and the eye socket of the user's eye based on the eye image and generating a motion control instruction according to the relative position are arranged in a head-mounted eye tracking display and control device; further, the head-mounted eye tracking display and control device sends the motion control instruction to a wireless data transmission device through a wire, and the wireless data transmission device is separated from the head-mounted eye tracking display and control device, thereby reducing the volume and weight of the head-mounted eye tracking display and control device and improving the wearing comfort of the head-mounted eye tracking display and control device; here, the wireless data transmission device can be arranged at the user's torso position or other suitable position, and the wireless data transmission device sends the motion control instruction to the drone device through a wireless signal, so that the drone device can move according to the motion control instruction. Further, the instruction generation module generates the motion control instruction according to the relative position between the pupil and the eye socket, thereby avoiding the generation of erroneous control instructions due to the user's unconscious blinking behavior and improving the control accuracy of the drone device.

[0022] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0024] Figure 1 A schematic diagram of the structure of a drone detection system based on eye tracking control provided by an embodiment of the present invention is shown;

[0025] Figure 2 A schematic diagram of a process of generating an environment image by a multi-source fusion camera module device provided in an embodiment of the present invention is shown;

[0026] Figure 3 A schematic diagram of a flow chart of a drone control method provided by an embodiment of the present invention is shown;

[0027] Figure 4 A schematic flow chart of a method for determining the relative position between the pupil and the eye socket in an eye grayscale image provided by an embodiment of the present invention is shown;

[0028] Figure 5 A structural schematic diagram of another UAV detection system based on eye tracking control provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0029] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0030] The current eye movement monitoring equipment integrates all the equipment used to control the unmanned aerial vehicle, such as the human eye information collector, the control command generating device and the remote signal sending device, into the head-mounted device, resulting in the eye movement monitoring equipment being too large and heavy, seriously affecting the comfort of the wearer; at the same time, the existing eye movement monitoring control system mostly collects the blinking frequency of the unmanned aerial vehicle operator and generates a control signal based on the blinking frequency; however, the above method is prone to generate erroneous control signals due to the operator's unconscious blinking behavior, which is very prone to misoperation, and the operation method is complicated, the interactive experience is poor, and the control accuracy of the unmanned aerial vehicle is seriously reduced.

[0031] In view of the above problems, in one embodiment, Figure 1 As shown, a drone detection system based on eye tracking control is provided, and the method is applied to the control scenario of a drone as an example for explanation. It should be noted that the embodiment of the present application can be applied to the control scenarios of other devices, which will not be described in detail here; wherein, the system includes a head-mounted eye tracking display and control device 100, a wireless data transmission device 200, and a drone device 300. Here, the head-mounted eye tracking display and control device 100 can be a helmet, glasses, a head-mounted mask, or other equipment that can be worn on the head. This embodiment is described with a head-mounted eye tracking display and control device 100 in the form of a head-mounted mask, and other forms of head-mounted eye tracking display and control devices 100 are also applicable to this embodiment.

[0032] Specifically, the head-mounted eye tracking display and control device 100 includes a camera module (not shown in the figure) and an instruction generation module (not shown in the figure) connected to the camera module. The camera module is used to collect the user's eye image in real time when the head-mounted eye tracking display and control device 100 is worn on the user's head, and send the eye image to the instruction generation module; Here, the head-mounted mask of the head-mounted eye tracking display and control device 100 can completely wrap the user's eye space to prevent external light from irradiating into the user's eyes, so as to improve the user's operation accuracy of the drone device, and the camera module can be an infrared image camera, which is set at a second position on the inside of the head-mounted mask, wherein the second position can be the center position of the inside of the head-mounted mask or other positions that can make the shooting area of ​​the infrared image camera cover the user's eyes, which is not limited here. Through the infrared image camera, a clear user eye image can be collected in the dark, and the collected eye image can include the pupil and eye socket of the user's single eye or both eyes, so as to improve the subsequent recognition accuracy.

[0033] Further, the instruction generation module is connected to the camera module to obtain the eyeball image from the camera module. Further, the instruction generation module is used to identify the relative position between the pupil and the eye socket according to the eyeball image, and generate a motion control instruction according to the relative position; here, each motion control instruction can be bound to a relative position in advance, for example, the position of the pupil in the upper part of the eye socket can be bound to the motion control instruction of the drone device climbing, and the position of the pupil in the left part of the eye socket can be bound to the motion control instruction of the drone device turning left; as an example, when the user looks up, the user's pupil is located in the upper part of the eye socket, the camera module captures the current eyeball image and sends it to the instruction generation module, the instruction generation module obtains the eyeball image and analyzes the position of the pupil in the upper part of the eye socket, and then generates a motion control instruction for the drone device to climb. The binding form of other motion control instructions and the relative position between the pupil and the eye socket can be determined according to actual conditions, and is also applicable to this embodiment.

[0034] Further, the wireless data transmission device 200 is connected to the instruction generation module to obtain the motion control instruction, and to perform data communication with the drone device 300 based on wireless signals to send the motion control instruction to the drone device 300; wherein the wireless data transmission device 200 and the head-mounted eye tracking display and control device 100 can be connected by wires or wireless signals for data communication, and at the same time, the wireless data transmission device 200 can be provided with other connection devices such as a padlock device to fix the wireless data transmission device 200 on the user's torso, and at the same time, the wireless data transmission device 200 can also be set at other locations close to the user, which will not be repeated here. Further, the wireless data transmission device 200 can be connected to the drone device 300 through wireless network transmission signals (WIreless Fidelity, WiFi) and wireless signals such as Bluetooth signals to realize data interaction with the drone device 300 to send the motion control instruction to the drone device 300. Further, the drone device 300 is used to move based on the motion control instruction.

[0035] The drone detection system based on eye tracking control provided in this embodiment sets a camera module for acquiring the user's eye image, and a command generation module for identifying the relative position between the pupil and the eye socket of the user's eye based on the eye image, and generating a motion control command according to the relative position in the head-mounted eye tracking display and control device; further, the head-mounted eye tracking display and control device sends the motion control command to the wireless data transmission device through a wire, and the wireless data transmission device is separated from the head-mounted eye tracking display and control device, which reduces the volume and weight of the head-mounted eye tracking display and control device and improves the wearing comfort of the head-mounted eye tracking display and control device; here, the wireless data transmission device can be set at the user's torso position or other suitable position, and the motion control command is sent to the drone device through the wireless data transmission device through a wireless signal, so that the drone device can move according to the motion control command. Further, the command generation module generates the motion control command according to the relative position between the pupil and the eye socket, avoiding the generation of erroneous control commands due to the user's unconscious blinking behavior, and improving the control accuracy of the drone device.

[0036] In one embodiment, the instruction generation module can be specifically used to: first, obtain multiple frames of eye images collected by the camera module within a continuous preset time period, and grayscale each frame of the eye image to obtain an eye grayscale image corresponding to each frame of the eye image.

[0037] Among them, the preset time period is used to determine whether the user intentionally controls the position of the pupil and the eye socket to issue a motion control instruction. Here, if a motion control instruction is generated using a single-frame eye image, an erroneous motion control instruction may be generated due to the user's unconscious eye movement behavior. Collecting multiple frames of eye images within a preset time period as the basis for generating motion control instructions can effectively avoid the situation where the user generates erroneous motion control instructions due to unconscious eye movement behavior. Only when the user's pupil and eye socket remain for a certain length of time, the system determines that the user wants to issue a motion control instruction and generates a motion control instruction based on the relative position. Furthermore, the preset time period can be 1 second or other time lengths, and the specific time period length can be determined based on actual conditions. Here, the time interval for collecting the first frame and the last frame of the eye image in the multiple frames of eye images can be a preset time period.

[0038] Specifically, the instruction generation module obtains multiple frames of eyeball images collected by the camera module within a continuous preset time period, and grayscales each frame of the eyeball images to obtain an eye grayscale image corresponding to each frame of the eyeball image. Further, the instruction generation module may include an image processing module and a head-mounted eye tracking and control instruction generation device, the image processing module is connected to the camera module to obtain eyeball images, and grayscales each frame of the eyeball images to obtain an eye grayscale image corresponding to each frame of the eyeball images.

[0039] Then, the multiple frames of eye grayscale images are combined into an eye grayscale image group, and the relative position between the pupil and the eye socket is determined in each frame of the eye grayscale image. Specifically, the eye grayscale image may include a binocular image, and the eye grayscale images corresponding to the multiple frames of eyeball images collected by the camera module in a continuous preset time period are combined into an eye grayscale image group, and any eye on the eye grayscale image is selected, and the pupil and eye socket of the selected eye in each frame of the eye grayscale image in the eye grayscale image group are identified through the grayscale difference between the various parts of the eye on the eye grayscale image, and then the relative position between the pupil and the eye socket of the selected eye in each frame of the eye grayscale image is determined. Here, the head-mounted eye tracking and control instruction generation device can be connected to the image processing module to obtain the above-mentioned multiple frames of eye grayscale images, and combine the multiple frames of eye grayscale images into an eye grayscale image group, and determine the relative position between the pupil and the eye socket in each frame of the eye grayscale image. Furthermore, according to the image clarity of the two eyes in each eye grayscale image, the eye with higher clarity can be selected as the recognition object for determining the relative position, or the user can select the left eye or the right eye as the recognition object.

[0040] Then, it is determined whether the relative positions between the pupil and the eye socket corresponding to each frame of the eye grayscale image group are the same. Specifically, the relative positions between the pupil and the eye socket in each frame of the eye grayscale image group are determined, and the relative positions corresponding to each frame of the eye grayscale image in the eye grayscale image group are compared to determine whether all the relative positions are the same. As an example, it can be determined whether the pupil in each frame of the eye grayscale image in the eye grayscale image group is located in the upper position of the eye socket.

[0041] Finally, if the relative position between the pupil and the eye socket corresponding to each frame of the eye grayscale image group is the same, the preset motion control instruction pre-mapped and stored with the relative position is determined according to the relative position, and the preset motion control instruction is determined as the motion control instruction.

[0042] Among them, the preset motion control instruction can be a pre-set operation instruction for the drone device, which can be stored in the head-mounted eye tracking and control instruction generation device with the relative position mapping. After the relative position is determined, the preset motion control instruction stored with the relative position mapping is determined as the motion control instruction, and the motion control instruction is sent to the wireless data transmission device. As an example, the relative position of the pupil in the lower part of the eye socket can be bound and stored with the preset motion control instruction for the downward movement of the drone device, and the relative position of the pupil in the right part of the eye socket can be bound and stored with the preset motion control instruction for the rightward movement of the drone device. When the relative position between the pupil and the eye socket corresponding to each frame of the eye grayscale image group is the same and the pupil is in the right part of the eye socket, the preset motion control instruction for the rightward movement of the drone device stored with the relative position mapping of the pupil in the right part of the eye socket is determined as the motion control instruction.

[0043] In contrast, if the relative position between the pupil and the eye socket corresponding to each frame of the eye grayscale image group is different, no motion control instruction is generated. Here, the instruction generation module can continuously acquire eyeball images, and grayscale all eyeball images between two frames of eyeball images separated by a preset time period, generate an eye grayscale image group, and continuously generate eye grayscale image groups for relative position identification. In the embodiment provided by the present application, the instruction generation module processes multiple frames of eyeball images within a preset time period, obtains grayscale images and identifies the relative position between the pupil and the eye socket. When the relative position between the pupil and the eye socket in the multiple frames of grayscale images corresponding to the eyeball images within the preset time period remains consistent, it can be determined that the user has moved the pupil and maintained it for a certain length of time, and then it is determined that the user has the intention to issue a motion control instruction. At this time, the motion control instruction is generated based on the relative position, which can effectively avoid the user from generating erroneous motion control instructions due to unconscious eye movement behavior, and improve the control accuracy of the drone device.

[0044] In one embodiment, the instruction generation module implements a method for determining the relative position between the pupil and the eye socket in the eye grayscale image, comprising: first, inputting the eye grayscale image into a pre-trained eye structure recognition model, and determining the pupil contour of the pupil and the orbital contour of the eye socket in the eye grayscale image.

[0045] Among them, the eye structure recognition model can be a pre-trained Haar type feature detection model. The Haar type feature detection model is a feature extraction method in image processing, which is used based on features such as grayscale difference, texture, edge and line in the image. Based on the idea of ​​Haar wavelet transform, Haar feature divides the image into small rectangular areas of different sizes and shapes, and performs weighted summation of pixels in each area to obtain a specific Haar feature value. These Haar feature values ​​can be used for classification. For example, in face recognition, the Haar type feature detection model can be used to detect information such as the position, size and direction of the face. Furthermore, the Haar type feature detection model is often used in face recognition, and it can detect various attributes of the face by training a classifier.

[0046] Specifically, the eye grayscale image is input into the eye structure recognition model to determine the pupil contour of the pupil and the orbital contour of the eye socket in the eye grayscale image. Then, the pupil position of the pupil contour in the eye grayscale image and the orbital position of the orbital contour in the eye grayscale image are determined.

[0047] Specifically, the position of the pupil contour on the eye grayscale image can be determined as the pupil position, and the position of the eye socket contour on the eye grayscale image can be determined as the eye socket position.

[0048] Further, the relative position between the pupil and the eye socket is determined according to the pupil position and the eye socket position. Specifically, after the pupil position and the eye socket position are respectively determined in the eye grayscale image, the relative position between the pupil and the eye socket is determined according to the positional relationship between the two.

[0049] The embodiments provided in the present application can quickly determine the pupil and eye socket in the grayscale image of the eye based on the recognition model, determine the positions of the two on the image, and then determine the relative position between the pupil and the eye socket, thereby improving the recognition efficiency of motion control instructions.

[0050] In one embodiment, the method for determining the pupil position of the pupil contour in the eye grayscale image may be: first, determining the pupil center of the pupil in the pupil contour. Specifically, the pupil center of the pupil may be determined based on the elliptical feature of the pupil, wherein the elliptical feature of the pupil may be the morphological feature of the pupil. When the user's pupil is not facing the camera of the camera module, the shape of the pupil in the eyeball image captured by the camera module may be non-circular due to the shooting angle, resulting in the shape of the pupil in the eye grayscale image converted from the eyeball image being non-circular. At this time, the pupil center position of the pupil may be determined in the eye grayscale image according to the morphological features of the pupil. For example, the pupil center is surrounded by the iris. After the iris is determined according to the morphological features of the pupil, the area surrounded by the iris is the pupil center of the pupil. Then, the pupil center position of the pupil center in the eye grayscale image is determined, and the pupil center position is determined as the pupil position. The embodiments provided in the present application can identify the center of the pupil and use the center position of the pupil as the pupil position, thereby improving the precision of position positioning and further improving the accuracy of the restored motion control instructions.

[0051] In one embodiment, the system further includes a multi-source fusion camera module device, which is arranged at a preset camera installation position on the drone device, and is used to collect the environmental image of the location of the drone device, and send the environmental image to the wireless data transmission device based on a wireless signal. The environmental image can be a perspective image of the multi-source fusion camera module device of the drone device; specifically, the multi-source fusion camera module device can be a camera, and the camera installation position can be the front of the drone device. The specific installation position can be determined according to actual conditions, and the multi-source fusion camera module device can take the environmental image of the location of the drone device. Furthermore, the wireless data transmission device is also used to obtain the environmental image from the multi-source fusion camera module device, and send the environmental image to the head-mounted eye tracking display and control device.

[0052] Further, the head-mounted eye tracking display and control device also includes an image display module, which is used to obtain the environmental image and display the environmental image of the location of the drone device to the user when the head-mounted eye tracking display and control device is worn on the head of the user. Here, the image display module can be an augmented reality display (AR) device, which is arranged at a first position on the inner side of the head-mounted mask of the head-mounted eye tracking display and control device. Here, the first position can be the inner side of the mask of the head-mounted eye tracking display and control device, and the projection of the AR device perpendicular to the user's eyes can cover the user's eyes to display the collected environmental image to the user. The specific installation position of the display module can be determined according to actual conditions and is also applicable to this embodiment. In the embodiment provided by the present application, the perspective image of the drone device can be sent to the image display module at the mask of the head-mounted eye tracking display and control device to display the drone perspective to the user, so that the user can better detect the environment of the drone device and improve the user experience of the drone device.

[0053] In one embodiment, the multi-source fusion camera module device includes a visible light camera, a low-light camera and an infrared camera. The visible light camera, the low-light camera and the infrared camera can be set together at the camera installation position. The camera shooting areas of the visible light camera, the low-light camera and the infrared camera are the same, so that the shooting angles of the visible light camera, the low-light camera and the infrared camera are the same and the same image is collected. Further, the visible light camera is used to collect visible light environment images to collect environment images when the light is normal; the low-light camera is used to collect light-enhanced environment images when the drone device is in a low-light environment; wherein the low-light camera can be a night vision camera such as a low-light night vision device. When the drone device is in a dark light environment, the image collected by the visible light camera may have a problem of unclear shooting pictures due to too dark light. At this time, the light-enhanced environment image is collected based on the low-light camera to collect high-definition environment images in a dark light environment; the infrared camera is used to collect infrared light environment images. The embodiment provided by the present application can collect environment images based on a variety of camera devices to adapt to different light environments, which significantly improves the environmental adaptability of the system.

[0054] In one embodiment, Figure 2 As shown, the multi-source fusion camera module device is also used to perform the following steps:

[0055] 101. Collect an infrared light environment image of the camera shooting area based on the infrared camera.

[0056] 102. Get the light intensity value at the location of the drone device.

[0057] Specifically, the light intensity value at the location of the drone device can be obtained based on the light sensor of the multi-source fusion camera module device.

[0058] 103. Compare the light intensity value with a preset light critical intensity value to determine whether the light intensity value is greater than or equal to the light critical intensity value.

[0059] Among them, the critical light intensity value is a pre-set standard reference value used to determine whether the light at the location of the drone device is too dark, and its specific value can be determined according to actual conditions.

[0060] 104. If the light intensity value is greater than or equal to the critical light intensity value, a visible light environment image of the camera shooting area is collected based on the visible light camera, and the visible light environment image is superimposed with the infrared light environment image to obtain the environment image.

[0061] Specifically, when the light intensity value is greater than or equal to the critical light intensity value, the visible light camera can be turned on, and the visible light environment image of the camera shooting area is collected based on the visible light camera, and the visible light environment image and the infrared light environment image are superimposed, and the visible light environment image and the infrared light environment image are fused to obtain an environment image after image superposition, so that the superimposed environment image includes the infrared light image and the conventional light image, so that the environment image contains more environmental details.

[0062] 105. If the light intensity value is less than the critical light intensity value, a light-enhanced environment image of the camera shooting area is obtained based on the low-light camera, and the light-enhanced environment image is superimposed with the infrared light environment image to obtain the environment image.

[0063] Specifically, when the light intensity value is less than the critical light intensity value, it can be determined that the UAV device is in a dark light environment, and the low-light camera can be turned on to collect a light-enhanced environmental image of the camera shooting area based on the low-light camera, and the light-enhanced environmental image is superimposed on the infrared light environmental image, and the light-enhanced environmental image is fused with the infrared light environmental image to obtain an environmental image after image superposition, so that the superimposed environmental image includes an infrared image and a conventional light image, so that the environmental image includes more environmental details.

[0064] 106. Send the environment image after the images are superimposed to the wireless data transmission device.

[0065] The embodiments provided in the present application can selectively select a specific camera to capture images of the camera shooting area according to the lighting conditions of the drone device, so that the drone device can be adapted to different lighting conditions, thereby improving the adaptability of the system.

[0066] In one embodiment, the wireless signal includes a mobile communication network signal and a wireless network transmission signal; the wireless data transmission device is also used to: first, monitor the mobile communication network signal strength of the mobile communication network signal and the wireless network transmission signal strength of the wireless network transmission signal. Then, the mobile communication network signal strength is compared with the wireless network transmission signal strength. Here, the mobile communication network signal strength can be compared with the wireless network transmission signal strength to determine the strength relationship between the mobile communication network signal strength and the wireless network transmission signal strength. Then, when the mobile communication network signal strength is greater than the wireless network transmission signal strength, the mobile communication network signal is determined as the wireless signal, so that the wireless data transmission device and the drone device communicate data through the mobile communication network signal; conversely, when the mobile communication network signal strength is less than or equal to the wireless network transmission signal strength, the wireless network transmission signal is determined as the wireless signal, so that the wireless data transmission device and the drone device communicate data through the wireless network transmission signal. The embodiment provided by the present application can select a signal with high signal strength as a wireless signal according to the strength of the mobile communication network signal and the wireless network transmission signal, thereby improving the signal strength between the drone device and the wireless data transmission device and improving the operating stability of the system.

[0067] The eye-tracking-based drone detection system provided in this embodiment includes a camera module for acquiring an image of a user's eyeball, an image display module for displaying to the user an environmental image collected by a multi-source fusion camera module device, and an instruction generation module for identifying the relative position between the pupil and the eye socket of the user's eyeball and generating a motion control instruction according to the relative position, which are arranged in a head-mounted eye-tracking display and control device; further, the head-mounted eye-tracking display and control device sends the motion control instruction to a wireless data transmission device through a wire, and the wireless data transmission device is separated from the head-mounted eye-tracking display and control device, thereby reducing the volume and weight of the head-mounted eye-tracking display and control device and improving the wearing comfort of the head-mounted eye-tracking display and control device; here, the wireless data transmission device can be set at the user's torso position or other suitable position, and the motion control instruction is sent to the drone device through a wireless signal through the wireless data transmission device, so that the drone device can move according to the motion control instruction. Furthermore, the multi-source fusion camera module device can select a camera that adapts to the light conditions at the location of the UAV device to capture environmental images, thereby improving the clarity of the environmental images displayed to the user; further, the instruction generation module generates motion control instructions based on the relative position between the pupil and the eye socket, thereby avoiding the generation of erroneous control instructions due to the user's unconscious blinking behavior, and improving the control accuracy of the UAV device.

[0068] Further, in one embodiment, if Figure 3 As shown, a drone control method is provided, which is applied to the instruction generation module in the drone detection system based on eye tracking control as described above, and the method includes:

[0069] 301. Acquire multiple frames of eyeball images collected within a continuous preset time period, and perform grayscale processing on each frame of the eyeball image to obtain an eye grayscale image corresponding to each frame of the eyeball image.

[0070] Among them, the preset time period is used to determine whether the user intentionally controls the position of the pupil and the eye socket to issue a motion control instruction. Here, if a motion control instruction is generated using a single-frame eye image, an erroneous motion control instruction may be generated due to the user's unconscious eye movement behavior. Collecting multiple frames of eye images within a preset time period as the basis for generating motion control instructions can effectively avoid the situation where the user generates erroneous motion control instructions due to unconscious eye movement behavior. Only when the user's pupil and eye socket remain for a certain length of time, the system determines that the user wants to issue a motion control instruction and generates a motion control instruction based on the relative position. Furthermore, the preset time period can be 1 second or other time lengths, and the specific time period length can be determined based on actual conditions. Here, the time interval for collecting the first frame and the last frame of the eye image in the multiple frames of eye images can be a preset time period.

[0071] Specifically, the instruction generation module obtains multiple frames of eyeball images collected by the camera module within a continuous preset time period, and grayscales each frame of the eyeball images to obtain an eye grayscale image corresponding to each frame of the eyeball image. Further, the instruction generation module may include an image processing module and a head-mounted eye tracking and control instruction generation device, the image processing module is connected to the camera module to obtain eyeball images, and grayscales each frame of the eyeball images to obtain an eye grayscale image corresponding to each frame of the eyeball images.

[0072] 302. Combine multiple frames of the eye grayscale images into an eye grayscale image group, and determine the relative position between the pupil and the eye socket in each frame of the eye grayscale image.

[0073] Specifically, the eye grayscale image may include a binocular image, and the eye grayscale images corresponding to the multiple frames of eyeball images collected by the camera module within a continuous preset time period are combined into an eye grayscale image group, and any eye on the eye grayscale image is selected, and the pupil and eye socket of the selected eye in each frame of the eye grayscale image in the eye grayscale image group are identified through the grayscale difference between the various parts of the eye on the eye grayscale image, and then the relative position between the pupil and eye socket of the selected eye in each frame of the eye grayscale image is determined. Here, the head-mounted eye tracking and control instruction generation device can be connected to the image processing module to obtain the above-mentioned multiple frames of eye grayscale images, and combine the multiple frames of eye grayscale images into an eye grayscale image group, and determine the relative position between the pupil and eye socket in each frame of the eye grayscale image. Further, according to the image clarity of the two eyes in each eye grayscale image, the eye with higher clarity can be selected as the identification object for determining the relative position, and the user can also select the left eye or the right eye as the identification object.

[0074] 303. Determine whether the relative positions between the pupil and the eye socket corresponding to each frame of the eye grayscale image in the eye grayscale image group are the same.

[0075] Specifically, the relative position between the pupil and the eye socket in each frame of the eye grayscale image group is determined, and the relative positions corresponding to each frame of the eye grayscale image in the eye grayscale image group are compared to determine whether all relative positions are the same. As an example, it can be determined whether the pupil in each frame of the eye grayscale image in the eye grayscale image group is located at the upper position of the eye socket.

[0076] 304. If the relative position between the pupil and the eye socket corresponding to each frame of the eye grayscale image group is the same, determine a preset motion control instruction corresponding to the relative position, and determine the preset motion control instruction as the motion control instruction for instructing the movement of the drone.

[0077] Among them, the preset motion control instruction can be a pre-set operation instruction for the drone device, which can be stored in the head-mounted eye tracking and control instruction generation device with the relative position mapping. After the relative position is determined, the preset motion control instruction stored with the relative position mapping is determined as the motion control instruction, and the motion control instruction is sent to the wireless data transmission device. As an example, the relative position of the pupil in the lower part of the eye socket can be bound and stored with the preset motion control instruction for the downward movement of the drone device, and the relative position of the pupil in the right part of the eye socket can be bound and stored with the preset motion control instruction for the rightward movement of the drone device. When the relative position between the pupil and the eye socket corresponding to each frame of the eye grayscale image group is the same and the pupil is in the right part of the eye socket, the preset motion control instruction for the rightward movement of the drone device stored with the relative position mapping of the pupil in the right part of the eye socket is determined as the motion control instruction.

[0078] In contrast, if the relative positions between the pupil and the eye socket corresponding to each frame of the eye grayscale image group are different, no motion control instruction is generated and step 301 is performed again. Here, the instruction generation module can continuously acquire eyeball images, and grayscale all eyeball images between two frames of eyeball images separated by a preset time period to generate an eye grayscale image group, and continuously generate eye grayscale image groups for relative position recognition.

[0079] The drone control method provided by the present application processes multiple frames of eyeball images within a preset time period to obtain a grayscale image and identify the relative position between the pupil and the eye socket. When the relative position between the pupil and the eye socket in the multiple frames of grayscale images corresponding to the eyeball images within the preset time period remains consistent, it can be determined that the user has moved the pupil and maintained it for a certain length of time, and then it can be determined that the user has the intention to issue a motion control command. Only then is the motion control command generated based on the relative position, which can effectively avoid the situation where the user generates erroneous motion control commands due to unconscious eye movement behavior, thereby improving the control accuracy of the drone equipment.

[0080] In one embodiment, Figure 4 As shown, the method of determining the relative position between the pupil and the eye socket in the eye grayscale image in step 302 includes:

[0081] 401. Input the eye grayscale image into a pre-trained eye structure recognition model, and determine the pupil contour of the pupil and the orbital contour of the eye socket in the eye grayscale image.

[0082] Among them, the eye structure recognition model can be a pre-trained Haar type feature detection model. The Haar type feature detection model is a feature extraction method in image processing, which is used based on features such as grayscale difference, texture, edge and line in the image. Based on the idea of ​​Haar wavelet transform, Haar feature divides the image into small rectangular areas of different sizes and shapes, and performs weighted summation of pixels in each area to obtain a specific Haar feature value. These Haar feature values ​​can be used for classification. For example, in face recognition, the Haar type feature detection model can be used to detect information such as the position, size and direction of the face. Furthermore, the Haar type feature detection model is often used in face recognition, and it can detect various attributes of the face by training a classifier.

[0083] Specifically, the eye grayscale image is input into the eye structure recognition model to determine the pupil contour of the pupil and the orbital contour of the eye socket in the eye grayscale image.

[0084] 402. Determine the pupil position of the pupil contour in the eye grayscale image, and the orbit position of the eye socket contour in the eye grayscale image.

[0085] Specifically, the position of the pupil contour on the eye grayscale image can be determined as the pupil position, and the position of the eye socket contour on the eye grayscale image can be determined as the eye socket position.

[0086] 403. Determine the relative position between the pupil and the eye socket according to the pupil position and the eye socket position.

[0087] Specifically, after the pupil position and the eye socket position are respectively determined in the eye grayscale image, the relative position between the pupil and the eye socket is determined according to the positional relationship between the two.

[0088] The embodiments provided in the present application can quickly determine the pupil and eye socket in the grayscale image of the eye based on the recognition model, determine the positions of the two on the image, and then determine the relative position between the pupil and the eye socket, thereby improving the recognition efficiency of motion control instructions.

[0089] In one embodiment, the method for determining the pupil position of the pupil contour in the eye grayscale image described in step 402 may be as follows: first, the pupil center of the pupil is determined in the pupil contour. Specifically, the pupil center of the pupil may be determined based on the elliptical feature of the pupil, wherein the elliptical feature of the pupil may be the morphological feature of the pupil. When the user's pupil is not facing the camera of the camera module, the shape of the pupil in the eyeball image captured by the camera module may be non-circular due to the shooting angle, resulting in the shape of the pupil in the eye grayscale image converted from the eyeball image being non-circular. At this time, the pupil center position of the pupil may be determined in the eye grayscale image according to the morphological features of the pupil. For example, the pupil center is surrounded by the iris. After the iris is determined according to the morphological features of the pupil, the area surrounded by the iris is the pupil center of the pupil. Then, the pupil center position of the pupil center in the eye grayscale image is determined, and the pupil center position is determined as the pupil position. The embodiments provided in the present application can identify the center of the pupil and use the center position of the pupil as the pupil position, thereby improving the precision of position positioning and further improving the accuracy of the restored motion control instructions.

[0090] Furthermore, based on the pupil position and the eye socket position, the relative position of the pupil and the eye socket can be determined, and then the position of the pupil in the eye socket can be obtained; further, the range in the eye socket can be pre-divided into multiple areas, each area corresponds to an area identifier, and when the pupil is located in a certain area, an area identifier corresponding to the area is generated; further, each area identifier is pre-mapped and stored with a preset motion control instruction, and when the area identifier is generated, the preset motion control instruction corresponding to the area identifier is determined as the motion control instruction.

[0091] The drone control method provided by the present application processes multiple frames of eyeball images within a preset time period to obtain a grayscale image and quickly identifies the relative position between the pupil and the eye socket based on a pre-trained eye structure recognition model. When the relative position between the pupil and the eye socket in the multiple frames of grayscale images corresponding to the eyeball images within the preset time period remains consistent, it can be determined that the user has moved the pupil and maintained it for a certain length of time, and then it can be determined that the user has the intention to issue a motion control command. Only then is the motion control command generated based on the relative position, which can effectively avoid the situation where the user generates erroneous motion control commands due to unconscious eye movement behavior, thereby improving the control accuracy of the drone equipment.

[0092] Further, as an example, Figure 5As shown, the operation process of the drone detection system based on eye tracking control can be: the camera module collects the eye image of the user's eyes, and sends it to the image processing module included in the head-mounted eye tracking display and control device for grayscale processing to obtain the eye grayscale image; further, the image processing module sends the eye grayscale image to the head-mounted eye tracking and control instruction generation device included in the head-mounted eye tracking display and control device, identifies the relative position between the pupil and the eye socket based on the eye grayscale image, and generates a motion control instruction according to the relative position; further, the head-mounted eye tracking and control instruction generation device sends the motion control instruction to the drone control instruction receiving and processing module included in the wireless data transmission device, the drone control instruction receiving and processing module can perform digital-to-analog conversion on the signal of the motion control instruction and send it to the wireless communication module in the wireless data transmission device for wireless signal interaction, the wireless communication module sends the motion control instruction to the drone device, so that the drone device moves based on the motion control instruction. Furthermore, the multi-source fusion camera module device on the drone device can collect the environmental image of the drone device's location, and send the environmental image to the wireless communication module through a wireless signal; the wireless communication module sends the environmental image to the drone control command receiving and processing module, and the drone control command receiving and processing module can perform analog-to-digital conversion on the signal of the environmental image and send it to the image transmission module included in the wireless data transmission device, so that the image transmission module sends the environmental image to the image display module through a high-definition multimedia interface line (High Definition Multimedia Interface, HDMI), so that the image display module displays the environmental image for the user.

[0093] Those skilled in the art will appreciate that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present application. Those skilled in the art will appreciate that the modules in the devices in the implementation scenario can be distributed in the devices of the implementation scenario according to the description of the implementation scenario, or can be changed accordingly and located in one or more devices different from the present implementation scenario. The modules of the above-mentioned implementation scenario can be combined into one module, or can be further split into multiple submodules.

[0094] The above serial numbers of this application are only for description and do not represent the advantages and disadvantages of the implementation scenarios. The above disclosure is only a few specific implementation scenarios of this application, but this application is not limited to them, and any changes that can be thought of by technicians in this field should fall within the scope of protection of this application.

Claims

1. A drone detection system based on eye movement tracking control, characterized in that, the system includes a head-mounted eye tracking display and control device, a wireless data transmission device, and a drone device, where, the head-mounted eye tracking display and control device includes a camera module and an instruction generation module connected to the camera module. The camera module is used to collect real-time eye images of the user when the head-mounted eye tracking display and control device is worn on the user's head, and send the eye images to the instruction generation module. The instruction generation module is used to identify the relative position between the pupil and the eye socket based on the eye images, and generate a motion control instruction according to the relative position; the wireless data transmission device is connected to the instruction generation module, and is used to obtain the motion control instruction, and perform data communication with the drone device based on a wireless signal to send the motion control instruction to the drone device; the drone device is used to move based on the motion control instruction.

2. The system according to claim 1, characterized in that, the system further includes a multi-source fusion camera module device; the multi-source fusion camera module device is arranged at a preset camera installation position on the drone device. The multi-source fusion camera module device is used to collect environmental images of the location where the drone device is located, and send the environmental images to the wireless data transmission device based on a wireless signal; the wireless data transmission device is further used to obtain the environmental images from the multi-source fusion camera module device, and send the environmental images to the head-mounted eye tracking display and control device; the head-mounted eye tracking display and control device further includes an image display module, and the image display module is used to obtain the environmental images, and when the head-mounted eye tracking display and control device is worn on the user's head, display the environmental images of the location where the drone device is located to the user.

3. The system according to claim 2, characterized in that, the multi-source fusion camera module device includes: a visible light camera, used to collect visible light environmental images; a low-light camera, used to collect light-enhanced environmental images when the drone device is in a low-light environment; an infrared camera, used to collect infrared light environmental images; wherein, the camera shooting areas of the visible light camera, the low-light camera, and the infrared camera are the same.

4. The system according to claim 3, characterized in that, the multi-source fusion camera module device is further used for: collecting infrared light environmental images of the camera shooting area based on the infrared camera; obtaining the light intensity value of the location where the drone device is located; comparing the light intensity value with a preset light critical intensity value to determine whether the light intensity value is greater than or equal to the light critical intensity value; if the light intensity value is greater than or equal to the light critical intensity value, then collecting visible light environmental images of the camera shooting area based on the visible light camera, and performing image superposition on the visible light environmental images and the infrared light environmental images to obtain the environmental images; If the light intensity value is less than the critical light intensity value, an enhanced light environment image of the camera shooting area is acquired based on the low-light camera, and the enhanced light environment image is superimposed on the infrared light environment image to obtain the environment image; The environment image after the image superposition is sent to the wireless data transmission device.

5. The system according to claim 1, wherein, the wireless signal includes a mobile communication network signal and a wireless network transmission signal; the wireless data transmission device is further configured to: monitor the mobile communication network signal strength of the mobile communication network signal and the wireless network transmission signal strength of the wireless network transmission signal; compare the mobile communication network signal strength with the wireless network transmission signal strength; when the mobile communication network signal strength is greater than the wireless network transmission signal strength, determine the mobile communication network signal as the wireless signal, so that data communication is performed between the wireless data transmission device and the drone device through the mobile communication network signal; when the mobile communication network signal strength is less than or equal to the wireless network transmission signal strength, determine the wireless network transmission signal as the wireless signal, so that data communication is performed between the wireless data transmission device and the drone device through the wireless network transmission signal.

6. The system according to claim 2, wherein, the image display module is an augmented reality display device, which is arranged at a first position on the inner side of the head-mounted mask of the head-mounted eye tracking display control device.

7. The system according to claim 6, wherein, the camera module is an infrared image camera, and the infrared image camera is arranged at a second position on the inner side of the head-mounted mask.

8. A drone control method, which is applied to the instruction generation module in the drone detection system based on eye movement tracking control according to any one of claims 1 to 7, wherein, the method includes: acquire multiple frames of eye images collected within a continuous preset time period, and perform grayscale processing on each frame of the eye images to obtain the corresponding eye grayscale images of each frame of the eye images; combine multiple frames of the eye grayscale images into an eye grayscale image group, and determine the relative positions between the pupils and the eye sockets in each frame of the eye grayscale images; judge whether the relative positions between the pupils and the eye sockets corresponding to each frame of the eye grayscale images in the eye grayscale image group are the same; if the relative positions between the pupils and the eye sockets corresponding to each frame of the eye grayscale images in the eye grayscale image group are the same, determine the preset motion control instruction corresponding to the relative position, and determine the preset motion control instruction as the motion control instruction for instructing the movement of the drone.

9. The method according to claim 8, wherein, determining the relative positions between the pupils and the eye sockets in the eye grayscale image includes: Input the eye grayscale image into a pre-trained eye structure recognition model, and determine the pupil contour of the pupil and the orbital contour of the eye socket in the eye grayscale image; Determine the pupil position of the pupil contour in the eye grayscale image and the orbital position of the orbital contour in the eye grayscale image; Determine the relative position between the pupil and the eye socket according to the pupil position and the orbital position.

10. The method according to claim 9, wherein, the determining the pupil position of the pupil contour in the eye grayscale image includes: determining the pupil center of the pupil in the pupil contour; determining the pupil center position of the pupil center in the eye grayscale image, and determining the pupil center position as the pupil position.

Images