A method, device and system for controlling eye movement of unmanned aerial vehicle

Abstract

The present application discloses a method, device and system for controlling eye movements of unmanned aerial vehicles, and belongs to the field of unmanned aerial vehicles. The method is applied to a control device, and the control device is respectively connected to the unmanned aerial vehicle and the eye movement acquisition device for communication. The control device is used to display the environmental image taken by the unmanned aerial vehicle, and the eye movement acquisition device is used to collect the eye movement data of the unmanned aerial vehicle operator when looking at the environmental image. The method includes: obtaining the environmental image taken by the unmanned aerial vehicle and the eye movement data of the unmanned aerial vehicle operator collected by the eye movement acquisition device; based on the eye movement data, determining the target object that the unmanned aerial vehicle operator is looking at in the environmental image; and controlling the unmanned aerial vehicle to aim at the target object. It can be seen that when controlling the unmanned aerial vehicle, the present application does not require the unmanned aerial vehicle operator to manually operate the control device, but determines the target object of interest to the unmanned aerial vehicle operator by analyzing the eye movement data of the unmanned aerial vehicle operator, and controls the unmanned aerial vehicle to automatically aim at the target object, thereby improving the accuracy and speed of the unmanned aerial vehicle aiming at the target object.

Description

Technical Field

[0001] The present application relates to the technical field of unmanned aerial vehicles, and in particular to an unmanned aerial vehicle eye movement control method, device and system. Background Art

[0002] A drone is an unmanned aerial vehicle that is controlled by a radio remote control device and a self-contained program control device. It has the advantages of small size, low cost, easy use, low environmental requirements, and strong environmental survivability. Therefore, drones have been widely used in many fields, such as military, aerial photography, logistics, etc.

[0003] In the related art, the process of aiming a target by a drone is as follows: the drone operator watches the environment image captured by the drone and determines the target object of interest from the environment image. Then, based on the position of the target object in the environment image, the operator manually operates the push rod on the drone remote controller to adjust the flight attitude of the drone, so that the drone can aim at the target object, that is, the target object is located at the center of the environment image captured by the drone.

[0004] However, the above-mentioned drone aiming at the target depends on the operating ability of the drone operator. For novice drone operators, the operation is relatively difficult and it is difficult to control the drone to aim at the target object in a timely and effective manner. Summary of the invention

[0005] The present application provides a method, device and system for controlling eye movements of a drone, which can improve the accuracy of the drone's aiming at a target. The technical solution is as follows:

[0006] In a first aspect, a method for controlling eye movement of a drone is provided, which is applied to a control device, wherein the control device is respectively connected to a drone and an eye movement acquisition device for communication, the control device is used to display an environment image taken by the drone, and the eye movement acquisition device is used to collect eye movement data when the drone operator looks at the environment image, the method comprising:

[0007] Acquire the environment image taken by the drone and the eye movement data of the drone operator collected by the eye movement collection device;

[0008] Determining a target object that the drone operator is looking at in the environment image based on the eye movement data;

[0009] The drone is controlled to aim at the target object.

[0010] Optionally, the eye movement data includes a field of view image when the drone operator is gazing at the environment image, and a gaze point of the drone operator in the field of view image, wherein the field of view image includes the environment image displayed in the control device;

[0011] The step of determining the target object that the drone operator is looking at in the environment image based on the eye movement data includes:

[0012] Performing object detection on the environment image to determine at least one detection object included in the environment image and a first image position of the at least one detection object in the environment image;

[0013] Based on the first image position of the at least one detection object and the gaze point, a target object gazed by the drone operator is determined from the at least one detection object.

[0014] Optionally, determining the target object gazed by the drone operator from the at least one detection object based on the first image position of the at least one detection object and the gaze point comprises:

[0015] Performing image coordinate system conversion processing on the first image position of the at least one detection object to determine the second image positions corresponding to the at least one detection object in the field of view image;

[0016] Based on the positional relationship between the second image position of the at least one detection object and the gaze point, the target object that the drone operator is looking at is determined from the at least one detection object.

[0017] Optionally, determining the target object gazed by the drone operator from the at least one detection object based on the positional relationship between the second image position of the at least one detection object and the gaze point comprises:

[0018] Taking the second image position of the first detection object as the center point, the local image area of ​​the first detection object in the field of view image is expanded in all directions according to the target size to obtain an expanded image area including the first detection object; wherein the first detection object is any one of the at least one detection object;

[0019] If the gaze point is located within the expanded image area and the distance between the gaze point and the second image position of the first detection object is the smallest, the first detection object is determined as the target object being gazed at by the drone operator.

[0020] Optionally, controlling the drone to aim at the target object includes:

[0021] Determining a relative distance between a first image position of the target object and an image center point of the environment image, the image center point indicating a current aiming point of the drone;

[0022] If the relative distance is greater than a distance threshold, calculating a first yaw angle and a first pitch angle when the drone aims at the target object based on the relative distance;

[0023] Based on the first yaw angle and the first pitch angle, the drone is controlled to aim at the target object.

[0024] Optionally, the method further comprises:

[0025] If the relative distance is less than the distance threshold, it is determined that the drone has aimed at the target object.

[0026] Optionally, controlling the drone to aim at the target object includes:

[0027] Obtaining a first yaw angle adjustment value and a first pitch angle adjustment value calculated in a previous adjustment cycle when the drone aims at the target object;

[0028] Obtaining an actual yaw angle adjustment value and an actual pitch angle adjustment value when the drone aims at the target object according to the first yaw angle adjustment value and the first pitch angle adjustment value in the previous adjustment period;

[0029] Determine, based on the first yaw angle adjustment value and the actual yaw angle adjustment value, a second yaw angle adjustment value when the drone aims at the target object in a current adjustment period;

[0030] Based on the first pitch angle adjustment value and the actual pitch angle adjustment value, a second pitch angle adjustment value when the UAV aims at the target object in a current adjustment period is determined.

[0031] Optionally, the method further comprises:

[0032] If the second yaw angle adjustment value is less than the first angle threshold, and / or the second pitch angle adjustment value is less than the second angle threshold, it is determined that the drone has aimed at the target object.

[0033] In a second aspect, a drone control device is provided, the drone control device has the function of implementing the drone control method described in the first aspect. The drone control device is included in a control device, the control device is respectively connected to the drone and the eye movement acquisition device for communication, the control device is used to display the environment image taken by the drone, the eye movement acquisition device is used to collect the eye movement data of the drone operator when looking at the environment image, and the device includes:

[0034] A data acquisition module, used to acquire the environment image taken by the drone and the eye movement data of the drone operator collected by the eye movement acquisition device;

[0035] a target determination module, configured to determine a target object that the drone operator is looking at in the environment image based on the eye movement data;

[0036] A control module is used to control the drone to aim at the target object.

[0037] Optionally, the eye movement data includes a field of view image when the drone operator is gazing at the environment image, and a gaze point of the drone operator in the field of view image, wherein the field of view image includes the environment image displayed in the control device;

[0038] The target determination module comprises:

[0039] an image detection unit, configured to perform object detection on the environment image, determine at least one detection object included in the environment image, and a first image position of the at least one detection object in the environment image;

[0040] A target determination unit is used to determine the target object that the drone operator is looking at from the at least one detection object based on the first image position of the at least one detection object and the gaze point.

[0041] Optionally, the target determination unit includes:

[0042] A coordinate conversion subunit, configured to perform image coordinate system conversion processing on the first image position of the at least one detection object, so as to determine the second image positions respectively corresponding to the at least one detection object in the field of view image;

[0043] The target determination subunit is used to determine the target object that the drone operator is gazing at from the at least one detection object based on the positional relationship between the second image position of the at least one detection object and the gaze point.

[0044] Optionally, the target determination subunit is specifically used to:

[0045] Taking the second image position of the first detection object as the center point, the local image area of ​​the first detection object in the field of view image is expanded in all directions according to the target size to obtain an expanded image area including the first detection object; wherein the first detection object is any one of the at least one detection object;

[0046] If the gaze point is located within the expanded image area and the distance between the gaze point and the second image position of the first detection object is the smallest, the first detection object is determined as the target object being gazed at by the drone operator.

[0047] Optionally, the control module is specifically used to:

[0048] Determining a relative distance between a first image position of the target object and an image center point of the environment image, the image center point indicating a current aiming point of the drone;

[0049] If the relative distance is greater than a distance threshold, calculating a first yaw angle and a first pitch angle when the drone aims at the target object based on the relative distance;

[0050] Based on the first yaw angle and the first pitch angle, the drone is controlled to aim at the target object.

[0051] Optionally, the control module is further used to:

[0052] If the relative distance is less than the distance threshold, it is determined that the drone has aimed at the target object.

[0053] Optionally, the control module is specifically used to:

[0054] Obtaining a first yaw angle adjustment value and a first pitch angle adjustment value calculated in a previous adjustment cycle when the drone aims at the target object;

[0055] Obtaining an actual yaw angle adjustment value and an actual pitch angle adjustment value when the drone aims at the target object according to the first yaw angle adjustment value and the first pitch angle adjustment value in the previous adjustment period;

[0056] Determine, based on the first yaw angle adjustment value and the actual yaw angle adjustment value, a second yaw angle adjustment value when the drone aims at the target object in a current adjustment period;

[0057] Based on the first pitch angle adjustment value and the actual pitch angle adjustment value, a second pitch angle adjustment value when the UAV aims at the target object in a current adjustment period is determined.

[0058] Optionally, the control module is further used to:

[0059] If the second yaw angle adjustment value is less than the first angle threshold, and / or the second pitch angle adjustment value is less than the second angle threshold, it is determined that the drone has aimed at the target object.

[0060] In a third aspect, a drone control system is provided, the drone control system comprising a drone, an eye movement acquisition device and a control device, the control device being communicatively connected to the drone and the eye movement acquisition device respectively;

[0061] The drone is used to capture environmental images;

[0062] The eye movement acquisition device is used to collect eye movement data of the drone operator when he looks at the environment image;

[0063] The control device is used to display the environment image taken by the drone and execute the steps of the drone control method described in the first aspect above.

[0064] In a fourth aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the drone control method described in the first aspect are implemented.

[0065] In a fifth aspect, the present application provides a computer storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the drone control method described in the first aspect are implemented.

[0066] In a sixth aspect, the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the drone control method described in the first aspect are implemented.

[0067] The technical solution provided by this application can at least achieve the following beneficial effects:

[0068] The present application provides a method for controlling a drone, wherein a control device is respectively connected to a drone and an eye movement acquisition device for communication, the control device is used to display an environmental image taken by the drone, and the eye movement acquisition device collects eye movement data of the drone operator when he or she is looking at the environmental image. When executing the method for controlling a drone provided by the present application, the control device can respectively obtain the environmental image taken by the drone and the eye movement data of the drone operator collected by the eye movement acquisition device, and then determine the target object that the drone operator is looking at in the environmental image based on the eye movement data, and then control the drone to aim at the target object. It can be seen that when controlling a drone, the embodiment of the present application does not require the drone operator to manually operate the control device, but analyzes and determines the target object of interest to the drone operator by collecting the eye movement data of the drone operator, and controls the drone to aim at the target object. In this way, not only can the drone automatically aim at the target object, but also the accuracy of the drone aiming at the target object and the response speed when aiming at the target object are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a schematic diagram of the architecture of a drone control system provided in an embodiment of the present application;

[0070] Figure 2 is a structural schematic diagram of a control device provided in an embodiment of the present application;

[0071] Figure 3 It is a flowchart of a drone control method provided in an embodiment of the present application;

[0072] Figure 4 is a schematic diagram of a coordinate conversion relationship between a field of view image and an environment image provided in an embodiment of the present application;

[0073] Figure 5 is a schematic diagram of an expanded image area of ​​a detection object provided in an embodiment of the present application;

[0074] Figure 6 is a schematic diagram of a yaw angle and a pitch angle provided in an embodiment of the present application;

[0075] Figure 7 It is a structural schematic diagram of a drone control device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the technical solutions of the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments.

[0077] Before explaining the drone control method provided in the embodiment of the present application, the application background of the embodiment of the present application, the system architecture of the drone control system involved, and the implementation environment are introduced.

[0078] Drones have the advantages of small size, low cost, easy use, low environmental requirements, and strong environmental survivability. Therefore, they have been widely used in many fields, such as military, aerial photography, logistics, etc. Usually, drone control is achieved by the drone operator manually operating the push rod on the drone remote control to adjust the flight posture of the drone so that the drone can aim at the target object of interest. Among them, the accuracy and speed of the drone's aiming ability are completely dependent on the operating ability of the drone operator. For novice drone operators, the operation is relatively difficult and it is difficult to control the drone to aim at the target object in a timely and effective manner.

[0079] Based on this, the embodiment of the present application provides a method for controlling a drone, in which, when a control device connected to the drone displays an environmental image taken by the drone, an eye movement acquisition device is used to obtain eye movement data of the drone operator when he or she is looking at the environmental image, and the target object that the drone operator is looking at in the environmental image is determined by analyzing the eye movement data, thereby controlling the drone to aim at the target object. In this way, in the process of controlling the drone to aim at the target object, the drone operator does not need to manually operate the control device, and the control device can automatically control the drone to quickly and effectively aim at the target object of interest to the drone operator by analyzing the eye movement data of the drone operator.

[0080] Next, the drone control system involved in the embodiment of the present application is introduced.

[0081] See also Figure 1 , the embodiments of the present application can be applied in Figure 1 In the drone control system shown, the drone control system includes: a drone 100, a control device 200 and an eye movement acquisition device 300, and the control device 200 is communicatively connected to the drone 100 and the eye movement acquisition device 300 respectively.

[0082] In the above-mentioned drone control system, the drone is used to capture environmental images; the control device is used to display the environmental images captured by the drone, and the eye movement acquisition device is used to collect the eye movement data of the drone operator when he looks at the environmental images. Based on this, after the control device obtains the environmental images captured by the drone and the eye movement data of the drone operator collected by the eye movement acquisition device, it can control the drone to automatically aim at the target object that the drone operator is currently interested in according to the drone control method provided in the embodiment of the present application.

[0083] The eye movement data includes the field of view image when the drone operator looks at the environment image displayed on the control device, and the fixation point of the drone operator in the field of view image, and the field of view image includes the environment image displayed in the control device. In other words, the eye movement acquisition device is not only used to acquire the image that the drone operator is looking at, but also to acquire the movement trajectory of the drone operator's own eyeballs to analyze the fixation point of the drone operator in the image. The fixation point describes the pixel point where the drone operator's eyes are aligned in the image that he is looking at.

[0084] As an example, the eye movement collection device may be an eye tracker. With the development of eye movement data collection technology, the eye movement collection device may also be other devices that can collect eye movement data of drone operators. The embodiments of the present application do not limit their specific types.

[0085] It should be understood that during the process of the drone operator controlling the drone, the drone operator is usually located around the control device so that he can fully see the environmental image taken by the drone and displayed on the control device, and understand the flight position of the drone in the environment through the environmental image, as well as the presence of people, landmark buildings, etc. in the environment, so as to adjust the flight position of the drone.

[0086] Among them, the flight posture of the drone may include the flight altitude, flight position (such as latitude and longitude information), yaw angle, pitch angle, roll angle, etc. of the drone, and the embodiments of the present application do not limit this.

[0087] It should be noted that when executing the drone control method provided in the embodiment of the present application, the above-mentioned control device can be a single device with image display, computing and processing, and communication capabilities, or it can be a combination of multiple devices, and the embodiment of the present application does not impose any restrictions on this.

[0088] In a possible implementation, the control device 200 includes a drone remote controller and a mobile terminal, and the drone remote controller is communicatively connected to the mobile terminal. The drone remote controller is communicatively connected to the drone, and the mobile terminal is communicatively connected to the eye movement acquisition device.

[0089] As an example, the drone remote controller may be a control handle, and the mobile terminal may be a mobile electronic device such as a smart phone that has a display screen and can display environmental images captured by the drone.

[0090] Among them, the drone remote control receives the environmental image taken by the drone, and sends the environmental image taken by the drone to the mobile terminal for display; the eye movement acquisition device collects the eye movement data of the drone operator when he looks at the environmental image displayed on the mobile terminal, and also sends the collected eye movement data to the mobile terminal; the mobile terminal executes the drone control method provided in the embodiment of the present application based on the environmental image taken by the drone and the eye movement data of the drone operator collected by the eye movement acquisition device.

[0091] In this implementation, there is no need to modify the original drone remote controller, and it is only necessary to ensure that the drone remote controller can establish a connection with the mobile terminal through a wired connection or a wireless connection. In this way, the requirements for the original drone remote controller are relatively low, and there is no restriction on the use scenario of the implementation scheme of this application.

[0092] In another possible implementation, the control device 200 includes a drone remote controller, a mobile terminal and a computer device, and the drone remote controller is communicatively connected to the mobile terminal, and the mobile terminal is communicatively connected to the computer device; at the same time, the drone remote controller is communicatively connected to the drone, and the eye movement acquisition device is communicatively connected to the computer device.

[0093] As an example, the drone remote controller may be a control handle, and the mobile terminal may be a mobile electronic device such as a smart phone with a display screen capable of displaying the environmental image captured by the drone. The computer device may be a terminal device or a server, and the terminal device may be, but is not limited to, various personal computers, laptops, tablet computers, portable wearable devices, etc.; the server may be, but is not limited to, at least one independent server, distributed server, cloud server, server cluster, data center, etc.

[0094] The drone remote controller receives the environment image taken by the drone, sends the environment image taken by the drone to the mobile terminal for display, and sends the environment image to the computer device, or the mobile terminal forwards the environment image to the computer device; the eye movement acquisition device collects the eye movement data of the drone operator when he looks at the environment image displayed on the mobile terminal, and sends the collected eye movement data to the computer device. The computer device executes the drone control method provided in the embodiment of the present application based on the environment image taken by the drone and the eye movement data of the drone operator collected by the eye movement acquisition device.

[0095] In this implementation, considering that the environmental images taken by the drone are transmitted in real time, a large amount of image data and eye movement data needs to be processed when executing the drone control method provided in the embodiment of the present application. Therefore, in order to improve data processing efficiency, the data calculation and processing part can be completed in an independent computer device, thereby improving the response speed of drone control and the accuracy of aiming at the target.

[0096] It should be understood that the above control device is only illustrated by two implementation methods, which does not constitute a limitation on the specific implementation method of the control device. And with the emergence of new application scenarios and new control devices, the technical solution of this application is also applicable when facing the same technical problem.

[0097] For example, when the above-mentioned drone is replaced by a surveillance device such as a dome camera, the target object that the user is interested in in the surveillance video / surveillance image can be determined by collecting the eye movement data of the user when watching the surveillance video / surveillance image collected by the surveillance device, thereby controlling the dome camera to continuously track the target object.

[0098] Finally, the implementation environment of the embodiments of the present application is introduced.

[0099] See also Figure 2 , Figure 2 The embodiment of the present application provides a structural schematic diagram of a control device, wherein the control device 200 includes at least one processor 210, a communication bus 220, a memory 230 and at least one communication interface 240.

[0100] The processor 210 may be a general-purpose central processing unit (CPU), a network processor (NP), a microprocessor, or may be one or more integrated circuits for implementing the solution of the present application, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above-mentioned PLD may be a complex programmable logic device (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0101] The communication bus 220 is used to transmit information between the above components. The communication bus 220 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0102] The memory 230 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, or a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0103] Optionally, the memory 230 may exist independently and be connected to the processor 210 via the communication bus 220 ; the memory 230 may also be integrated with the processor 210 .

[0104] The communication interface 240 is used to communicate with other devices or communication networks. The communication interface 240 includes a wired communication interface and may also include a wireless communication interface. The wired communication interface may be, for example, an Ethernet interface. The Ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The wireless communication interface may be a wireless local area network (WLAN) interface, a cellular network communication interface, or a combination thereof.

[0105] Optionally, the processor 210 may include one or more CPUs. The control device 200 may include multiple processors. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU).

[0106] It should be noted that the processor here may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0107] In some embodiments, the control device 200 may further include an output device and an input device. The output device communicates with the processor 210 and may display information in a variety of ways. For example, the output device may be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device communicates with the processor 210 and may receive user input in a variety of ways. For example, the input device may be a mouse, a keyboard, a touch screen device, or a sensor device.

[0108] In some embodiments, the memory 230 is used to store a computer program for executing the solution of the present application, and the processor 210 can execute the computer program stored in the memory 230. For example, the control device 200 can implement the following by using the processor 210 and the computer program in the memory 230: Figure 3 The drone control method provided in the embodiment.

[0109] See also Figure 3 , Figure 3 1 is a flow chart of a method for aiming a target at a drone provided in an embodiment of the present application. The method is applied to the above-mentioned control device 200 as an example. The method includes the following steps.

[0110] Step 310: Acquire the environment image taken by the drone and the eye movement data of the drone operator collected by the eye movement collection device.

[0111] The environment image may be a two-dimensional image captured in the coordinate system of the drone. When the drone is connected to the control device for communication, the drone may transmit the captured environment image to the control device in real time.

[0112] In the embodiment of the present application, the drone can also transmit the current attitude information of the drone to the control device in real time, and the attitude information at least includes the yaw angle and pitch angle of the drone at the current moment. Among them, the yaw angle (yaw) is the angle offset value of the drone in the horizontal direction; the pitch angle (pitch) is the current angle offset value of the drone in the vertical direction.

[0113] As an example, a coordinate system is established with the center of mass of the drone body as the origin. The coordinate system includes an X-axis, a Y-axis and a Z-axis. The X-axis and the Y-axis are perpendicular to each other and the plane they form coincides with the horizontal plane. The Z-axis is perpendicular to the horizontal plane. The pitch angle refers to the angle at which the fuselage rotates around the X-axis, that is, it can be the angle between the fuselage and the horizontal plane passing through its center of mass. When the fuselage coincides with the horizontal plane, the corresponding pitch angle is 0 degrees; according to the pre-set rules, when the nose of the fuselage is located above the horizontal plane passing through the center of mass of the fuselage, the corresponding pitch angle is a positive value, and when the nose of the fuselage is located below the horizontal plane passing through the tail, the corresponding pitch angle is a negative value. Further, the range of the pitch angle can be [-90°, 90°]. The yaw angle refers to the angle at which the fuselage rotates around the Z-axis. It can be agreed that the yaw angle is a positive value when the fuselage rotates to the right, and a negative value when the fuselage rotates to the left. Among them, the range of the yaw angle can be [-180°, 180°].

[0114] It should be understood that the above-mentioned method of establishing a coordinate system and the ranges of the yaw angle and the pitch angle are merely exemplified by the above-mentioned rules. In specific implementation, other rules may also be used to establish a coordinate system and define the ranges of the yaw angle and the pitch angle. The embodiments of the present application do not impose any restrictions on this.

[0115] As described above, the eye movement data of the drone operator collected by the eye movement acquisition device includes: the field of view image when the drone operator is looking at the environmental image, and the gaze point of the drone operator in the field of view image. Based on this, the field of view image must include the environmental image displayed in the control device.

[0116] The gaze point can also be understood as the point where the drone operator's gaze is aimed in the field of view image. The gaze point can indicate a local image area that the drone operator is gazing at in the field of view image. The local image area includes multiple pixel points, and these pixel points describe one or more objects.

[0117] Step 320: Determine the target object that the drone operator is looking at in the environment image based on the eye movement data.

[0118] That is, the embodiment of the present application automatically analyzes the target object that the drone operator is looking at in the environmental image taken by the drone based on the eye movement data of the drone operator, and then controls the drone to automatically aim at the target object through the following step 330, without the need for manual operation by the user, and can achieve target aiming quickly and effectively.

[0119] In some embodiments, the implementation process of the above step 320 may include the following steps:

[0120] (1) Performing object detection on an environment image to determine at least one detection object included in the environment image and a first image position of the at least one detection object in the environment image.

[0121] Among them, at least one detection object can be a person, landmark building, etc. included in the environmental image, and the number of the at least one detection object can be one or more, and the embodiment of the present application does not limit this. Of course, the type of the at least one detection object can be the same, for example, the at least one detection object includes three people; the type of the at least one detection object can also be different, for example, at least one detection object includes three people and two landmark buildings, and the embodiment of the present application does not show this either.

[0122] In one possible implementation, object detection on an environmental image may be performed by using a trained neural network model, a target detection algorithm, etc. to detect the environmental image to identify at least one detection object included in the environmental image, and output a first image position of the at least one detection object in the environmental image.

[0123] The neural network model may be a target detection model of the YOLO series, and its specific image detection process may refer to the relevant prior art. Taking the use of a neural network model for object detection as an example, based on the trained neural network model, an environmental image is input into the neural network model, and the neural network model outputs at least one detection object included in the environmental image, and the first image position of the at least one detection object in the environmental image. Each detection object corresponds to a first image position.

[0124] In some embodiments, after the neural network model identifies at least one detection object in the environmental image, it can use a bounding box to mark the local image area corresponding to each detection object, and the local image area is the first image position of the detection object in the environmental image.

[0125] Based on this, the first image position of each detected object in the environmental image can include the image coordinates of two pixel points located at the diagonal position on the boundary box marking the detected object. Through these two image coordinates, the local image area where a detected object is located in the environmental image can be uniquely determined.

[0126] As an example, the first image position of the detected object includes the image coordinates of the upper left pixel and the image coordinates of the lower right pixel on the bounding box marking the detected object; or, the first image position of the detected object includes the image coordinates of the upper right pixel and the image coordinates of the lower left pixel on the bounding box marking the detected object.

[0127] It should be noted that when using different neural network models or target detection algorithms to perform object detection on an environmental image, other methods may also be used to determine the first image position of the detection object in the environmental image, and the embodiments of the present application do not limit this. For example, when determining the local image area where the detection object is located in the environmental image, the first image position of the detection object may also include the image coordinates of the center point of the local image area, that is, the image coordinates of the center point of the boundary box marking the detection object.

[0128] (2) Based on the first image position and gaze point of at least one detection object, determine the target object that the drone operator is looking at from the at least one detection object.

[0129] It should be noted that the gaze point can be the image coordinates of the pixel point where the drone operator's eyes focus in the field of view image, and the first image position of each detection object is determined based on the environmental image. The first image position of the detection object and the gaze point do not belong to the same image coordinate system. In addition, considering that when the control device displays the environmental image, the screen may have reflection problems, resulting in the clarity of the environmental image in the field of view image being poorer than the environmental image itself taken by the drone. Therefore, even if the image content of the field of view image and the environmental image is exactly the same, there may be deviations between the image coordinates of the corresponding pixels.

[0130] Based on this, in order to improve the accuracy of determining the target object, the embodiment of the present application can first adjust the first image position and the gaze point of each detection object to the same image coordinate system, and then analyze the positional relationship between the gaze point and each detection object in the same image coordinate system. Considering that the gaze point can reflect the image area that the drone operator is looking at, and the gaze point is located in the field of view image, the first image position of each detection object can be converted to the image coordinate system of the field of view image, and then the positional relationship between the gaze point and each detection object is analyzed, and the target object that the drone operator is looking at is determined without changing the image coordinates of the gaze point.

[0131] In one possible implementation, the first image position of at least one detection object is converted into an image coordinate system to determine the second image position corresponding to the at least one detection object in the field of view image; based on the positional relationship between the second image position of the at least one detection object and the gaze point, the target object that the drone operator is looking at is determined from the at least one detection object.

[0132] It should be understood that the first image position and the second image position correspond one-to-one, and a first image position and a corresponding second image position describe the same detection object. As described above, when the first image position includes the image coordinates of two pixels, the second image position obtained after the image coordinate system conversion process also corresponds to the image coordinates of the two pixels. For example, the second image position includes the image coordinates of the upper left pixel point and the image coordinates of the lower right pixel point in the local image area where the detection object is located in the field of view image. Similarly, when the first image position includes the image coordinates of one pixel point, the second image position obtained after the image coordinate system conversion process also corresponds to the image coordinates of one pixel point, that is, the second image position includes the image coordinates of the center point of the local image area where the detection object is located in the field of view image.

[0133] As an example, see Figure 4 , assuming that in the environment image taken by the drone, the first image position of the detection object C includes the image coordinates (x5, y5) of the upper left pixel point and the image coordinates (x6, y6) of the lower right pixel point of the boundary box marking the detection object C, and the width of the environment image is known to be Vu and the image height is Uu. In the field of view image captured by the eye movement acquisition device, it is assumed that the image width of the environment image included in the field of view image in the horizontal direction is Ve, and the image height in the vertical direction is Ue, and the image coordinates (x1, y1) of the upper left pixel point and the image coordinates (x2, y2) of the lower right pixel point in the environment image included in the field of view image are known.

[0134] In a digital image, the image width is the product of the number of pixels in the horizontal direction and the pixel size, and the image height is the product of the number of pixels in the vertical direction and the pixel size. In a digital image, the pixel size is usually constant and is determined by the resolution and color depth of the image.

[0135] Based on this, the first image position of the detection object C can be converted into an image coordinate system using the following formulas (1) and (2) to determine the second image position of the detection object C in the field of view image.

[0136]

[0137]

[0138] In the above formula, the second image position of the detection object C includes the image coordinates (x3, y3) corresponding to the upper left corner pixel point of the bounding box marking the detection object C in the environment image, and the image coordinates (x4, y4) corresponding to the lower right corner pixel point of the bounding box marking the detection object C in the field of view image.

[0139] By analogy, the first image position of each detection object in the environment image is converted to the image coordinate system to determine the second image position corresponding to each detection object in the field of view image. At this time, the second image position and the gaze point of each detection object belong to the same image coordinate system. Therefore, the target object that the drone operator is looking at can be determined in the field of view image based on the positional relationship between the second image position and the gaze point of each detection object.

[0140] It should be noted that for each detection object, when determining whether it is the target object being looked at by the drone operator, the implementation process is similar, and the following takes the first detection object as an example to explain the implementation process of determining whether the first detection object is the target object. The first detection object is any one of the at least one detection object.

[0141] In a possible implementation, with the second image position of the first detection object as the center point, the local image area of ​​the first detection object in the field of view image is expanded in all directions according to the target size to obtain an expanded image area containing the first detection object; if the gaze point is located in the expanded image area and the distance between the gaze point and the second image position is the smallest, the first detection object is determined as the gaze object being gazed at by the drone operator.

[0142] The target size can be any pre-set scale value, such as a target size of 5 pixels in image distance, which is not limited in the present embodiment. Moreover, when the location area of ​​the first detection object is expanded to the surroundings, the expansion size in each direction can be the same or different, which is not limited in the present embodiment.

[0143] That is, according to the above method, the extended image area corresponding to each detection object in the field of view image is determined. If the positional relationship between the gaze point and the extended image area and the second image position of a detection object meets the above conditions, the detection object is determined as the target object.

[0144] As an example, assuming that the first detection object is the above Figure 4As shown in the detection object C, the second image position of the detection object C includes image coordinates (x3, y3) and image coordinates (x4, y4). Then, with the second image position of the detection object C as the center, the difference between x3 and x4 is used as the first expansion size to expand the local image area of ​​the first detection object in the horizontal direction; similarly, the difference between y3 and y4 is used as the second expansion size to expand the local image area of ​​the first detection object in the vertical direction, and the following is obtained: Figure 5 The expanded image area shown.

[0145] Among them, Figure 5 In the expanded image area shown, the local image area of ​​the first detection object is located in the center, and the surrounding area includes eight areas obtained after expansion, namely area 1 to area 8. Based on the positional relationship between the expanded image area of ​​the first detection object and the gaze point, the implementation process of determining whether the first detection object is the target object can be implemented in several situations:

[0146] The first case: if the gaze point is located in the local image area of ​​the first detection object, that is, in the area defined by the image coordinates (x3, y3) and the image coordinates (x4, y4), the first detection object is determined to be the target object.

[0147] The second case: If the gaze point is located in the adjacent area of ​​the bounding box of the first detection object (i.e. Figure 5 , then calculate the first distance between the gaze point and the bounding box. Among at least one detection object, if the first distance corresponding to the first detection object is the smallest, then determine that the first detection object is the target object.

[0148] like Figure 5 As shown, when the gaze point is located in area 3, it is necessary to calculate the distance between the gaze point and the right boundary line of the boundary box of the first detection object to determine whether the first detection object is a target object.

[0149] Case 3: If the gaze point is within the four corners of the bounding box of the first detection object (i.e. Figure 5 5, 6, 7, and 8 shown in FIG. 1 ), then the second distance between the gaze point and the nearest target vertex on the bounding box is calculated. Among at least one detection object, if the second distance corresponding to the first detection object is the smallest, then the first detection object is determined to be the target object.

[0150] It should be noted that the above-mentioned method of obtaining the expanded image area is only an example. When the second image position of the first detection object includes the image coordinates of a pixel point, the image coordinates can also be used as the center point, and the local image area of ​​the first detection object in the field of view image is expanded in all directions according to the target size to obtain the expanded image area. When the target size is a certain value, the expanded image area is a circular image area. That is, the embodiment of the present application does not limit the specific implementation process of determining the expanded image area.

[0151] In the above method, the local image area of ​​the first detection object in the field of view image is first expanded, and the obtained expanded image area can contain all and complete image information of the first detection object. To a certain extent, it can also make up for the errors introduced when determining the image position and performing image coordinate system conversion processing in the object detection stage, thereby improving the accuracy of determining the local image area of ​​each detection object in the field of view image. Furthermore, based on the positional relationship between the expanded image area and the gaze point, the target object that the drone operator is looking at can be accurately determined.

[0152] Step 330: Control the drone to aim at the target object.

[0153] After determining the target object that the drone operator is looking at in the environmental image, the yaw angle and pitch angle (i.e., target value) when the drone aims at the target object, or the yaw angle adjustment value and pitch angle adjustment value when the drone aims at the target object can be calculated based on the first image position of the target object in the environmental image, thereby controlling the drone to aim at the target object.

[0154] In a possible implementation, the implementation process of the above step 330 may be: determining the relative distance between the first image position of the target object and the image center point of the environment image, the image center point indicating the current aiming point of the drone; if the relative distance is greater than a distance threshold, then based on the relative distance, determining the first yaw angle and the first pitch angle when the drone aims at the target object; based on the first yaw angle and the first pitch angle, controlling the drone to aim at the target object.

[0155] The first image position of the target object includes the center point coordinates of the local image area corresponding to the target object in the environment image. The relative distance includes the relative distance Δx in the horizontal direction and the relative distance Δy in the vertical direction.

[0156] As an example, see Figure 6 In the imaging plane diagram shown in the figure, point O is the center point of the image, point P is the image coordinate of the target object, Δx and Δy are the relative distances between the center point of the image and the target object, and Oc is the center of the drone (that is, the camera on the drone). Then the first yaw angle α of the drone isx and the first pitch angle α y can be determined by the following formulas (3) and (4).

[0157]

[0158]

[0159] In the above formula, f is the focal length of the drone camera, d is x and d y are the actual lengths of a single pixel in the horizontal and vertical directions, n x and n y The number of pixels in the horizontal and vertical directions respectively, err x and err y is the error term.

[0160] Among them, err x and err y It is caused by camera distortion and position distance error. During the attitude adjustment process of the drone, err x and err y It will gradually decrease, making the offset angle gradually approach the true value.

[0161] Furthermore, after determining the first yaw angle and the first pitch angle, the control device can send an attitude adjustment instruction to the drone, and carry the first yaw angle and the first pitch angle in the attitude adjustment instruction to instruct the drone to adjust its own flight attitude according to the first yaw angle and the first pitch angle to aim at the target object.

[0162] Of course, due to camera distortion and position distance errors, the drone may not be able to aim at the target object at once after adjusting the flight attitude according to the first yaw angle and the first pitch angle. Based on this, the drone can continue to adjust according to the above logic according to the environmental image fed back after adjusting the flight attitude until the drone aims at the target object.

[0163] Since the image center point of the environment image indicates the current aiming point of the drone, it is possible to determine whether the drone is aiming at the target object based on the relative distance between the first image position of the target object in the environment image taken by the drone and the image center point.

[0164] As an example, if the relative distance between the first image position of the target object and the image center point of the environment image is greater than the distance threshold, it is determined that the drone has not yet aimed at the target object, and it is necessary to continue to calculate the first yaw angle and the first pitch angle of the drone according to the above calculation logic to instruct the drone to adjust its flight attitude. If the relative distance between the first image position coordinates of the target object and the image center point of the environment image is less than the distance threshold, it is determined that the drone has aimed at the target object.

[0165] Among them, the distance threshold is an arbitrary pre-set value, for example, the distance threshold is the image distance value corresponding to three pixels in the image, and the embodiment of the present application does not limit this.

[0166] In another possible implementation, the implementation process of the above-mentioned step 330 may be: obtaining the first yaw angle adjustment value and the first pitch angle adjustment value determined in the previous adjustment cycle when the drone aims at the target object; obtaining the actual yaw angle adjustment value and the actual pitch angle adjustment value when the drone aims at the target object based on the first yaw angle adjustment value and the first pitch angle adjustment value in the previous adjustment cycle; based on the first yaw angle adjustment value and the actual yaw angle adjustment value, determining the second yaw angle adjustment value when the drone aims at the target object in the current adjustment cycle; based on the first pitch angle adjustment value and the actual pitch angle adjustment value, determining the second pitch angle adjustment value when the drone aims at the target object in the current adjustment cycle.

[0167] If the second yaw angle adjustment value is less than the first angle threshold, and / or the second pitch angle adjustment value is less than the second angle threshold, it is determined that the drone has aimed at the target object. On the contrary, if the second yaw angle adjustment value is greater than or equal to the first angle threshold, and / or the second pitch angle adjustment value is greater than or equal to the second angle threshold, it means that the drone has not aimed at the target object, and it is necessary to continue to control the drone to adjust the flight attitude according to the above adjustment logic.

[0168] Among them, the first angle threshold and the second angle threshold are pre-set arbitrary values, such as 3°, 5°, etc., and the embodiment of the present application does not limit this.

[0169] Optionally, in the process of controlling the drone to aim at the target object, the control device can record and store the yaw angle adjustment value and the pitch angle adjustment value calculated in each adjustment cycle, and obtain the actual yaw angle adjustment value and the actual pitch angle adjustment value when the drone adjusts its posture in each adjustment cycle.

[0170] In some embodiments, after obtaining the first yaw angle adjustment value and the first pitch angle adjustment value calculated in the previous adjustment cycle, as well as the actual yaw angle adjustment value and the actual pitch angle adjustment value of the drone, the control device can use an incremental proportional integral differential method to calculate the second yaw angle adjustment value and the second pitch angle adjustment value in the current adjustment cycle.

[0171] The second yaw angle adjustment value Δα in the current adjustment period is calculated by using an incremental proportional integral differential algorithm. x and the second pitch angle adjustment value Δα y When , it can be calculated by the following formulas (5) and (6).

[0172] Δα x =Kp1(Ex0-Ex1)+KI1(Ex0)+KD1(Ex0-2Ex1+Ex2) (5)

[0173] Δα y =Kp2(Ey0-Ey1)+KI2(Ey0)+KD2(Ey0-2Ey1+Ey2) (6)

[0174] In the above formula, Kp1, Kp2, KI1, KI2, KD1, KD2 are proportional coefficients, which are preset values. In the incremental proportional integral differential algorithm, Ex0 is the calculation error of the current adjustment cycle, Ex1 is the calculation error of the previous adjustment cycle, and Ex2 is the calculation error of the previous adjustment cycle. Among them, Ex0 = Vtx-Vfx, Ey0 = Vty-Vfy, Vtx and Vty are the angle values ​​that need to be adjusted in the horizontal and vertical directions of the drone in the previous adjustment cycle (that is, Δα of the previous adjustment cycle). x and Δα y ), Vfx and Vfy are the actual adjusted angle values ​​in the horizontal and vertical directions fed back by the drone during the previous adjustment cycle.

[0175] In this way, after determining the second yaw angle adjustment value and the second pitch angle adjustment value, an attitude adjustment instruction can be sent to the drone, and the attitude adjustment instruction carries the second yaw angle adjustment value and the second pitch angle adjustment value, so as to instruct the drone to adjust its own flight attitude according to the second yaw angle adjustment value and the second pitch angle adjustment value to aim at the target object. Similarly, the above adjustment process can be executed repeatedly until the drone aims at the target object.

[0176] It should be noted that the embodiment of the present application only uses the incremental proportional integral differential algorithm as an example, and other algorithms can also be used to calculate the adjustment value within the current adjustment period, and the embodiment of the present application does not limit this.

[0177] In summary, in the embodiment of the present application, the control device is respectively connected to the drone and the eye movement acquisition device for communication to display the environmental image taken by the drone, and the eye movement acquisition device collects the eye movement data of the drone operator when looking at the environmental image. Based on this, the control device can respectively obtain the environmental image taken by the drone and the eye movement data of the drone operator collected by the eye movement acquisition device, and then determine the target object that the drone operator is looking at in the environmental image based on the eye movement data, and then control the drone to aim at the target object. It can be seen that when controlling the drone, the embodiment of the present application does not require the operator to manually operate the control device, but analyzes the target object that the operator wants the drone to aim at by collecting the operator's eye movement data. In this way, not only can the automatic aiming of the target object be achieved, but also the accuracy of the drone aiming at the target object and the response speed when aiming at the target object are improved.

[0178] Figure 7 is a schematic diagram of the structure of a drone control device provided in an embodiment of the present application. The drone control device 700 can be implemented as part or all of a control device by software, hardware, or a combination of both. The control device can be Figure 2 See the equipment shown. Figure 7 The UAV control device includes: a data acquisition module 710, a target determination module 720 and a control module 730.

[0179] The data acquisition module 710 is used to acquire the environment image taken by the drone and the eye movement data of the drone operator collected by the eye movement acquisition device;

[0180] A target determination module 720 is used to determine the target object that the drone operator is looking at in the environment image based on the eye movement data;

[0181] The control module 730 is used to control the drone to aim at the target object.

[0182] Optionally, the eye movement data includes a field of view image when the drone operator is gazing at an environment image, and a gaze point of the drone operator in the field of view image, the field of view image including an environment image displayed in a control device;

[0183] The target determination module 720 includes:

[0184] An image detection unit, configured to perform object detection on an environment image, determine at least one detection object included in the environment image, and a first image position of the at least one detection object in the environment image;

[0185] The target determination unit is used to determine the target object that the drone operator is looking at from at least one detection object based on the first image position and the gaze point of the at least one detection object.

[0186] Optionally, the target determination unit includes:

[0187] A coordinate conversion subunit, used for performing image coordinate system conversion processing on a first image position of at least one detection object to determine a second image position corresponding to the at least one detection object in the field of view image;

[0188] The target determination subunit is used to determine the target object that the drone operator is gazing at from at least one detection object based on the positional relationship between the second image position of the at least one detection object and the gaze point.

[0189] Optionally, the target determination subunit is specifically used to:

[0190] Taking the second image position of the first detection object as the center point, the local image area of ​​the first detection object in the field of view image is expanded to all sides according to the target size to obtain an expanded image area including the first detection object; wherein,

[0191] The first detection object is any one of the at least one detection object;

[0192] If the gaze point is located within the expanded image area and the distance between the gaze point and the second image position of the first detection object is the smallest, the first detection object is determined to be the target object that the drone operator is gazing at.

[0193] Optionally, the control module 730 is specifically configured to:

[0194] Determining a relative distance between a first image position of the target object and an image center point of the environment image, the image center point indicating a current aiming point of the drone;

[0195] If the relative distance is greater than the distance threshold, a first yaw angle and a first pitch angle when the drone aims at the target object are calculated based on the relative distance;

[0196] Based on the first yaw angle and the first pitch angle, the drone is controlled to aim at the target object.

[0197] Optionally, the control module 730 is further configured to:

[0198] If the relative distance is less than the distance threshold, it is determined that the drone has aimed at the target object.

[0199] Optionally, the control module 730 is specifically configured to:

[0200] Obtaining a first yaw angle adjustment value and a first pitch angle adjustment value calculated in the last adjustment cycle when the drone aims at the target object;

[0201] Obtaining an actual yaw angle adjustment value and an actual pitch angle adjustment value when the drone aims at the target object according to the first yaw angle adjustment value and the first pitch angle adjustment value in the previous adjustment cycle;

[0202] Determine, based on the first yaw angle adjustment value and the actual yaw angle adjustment value, a second yaw angle adjustment value when the drone aims at the target object in the current adjustment period;

[0203] Based on the first pitch angle adjustment value and the actual pitch angle adjustment value, a second pitch angle adjustment value when the drone aims at the target object in the current adjustment period is determined.

[0204] Optionally, the control module 730 is further configured to:

[0205] If the second yaw angle adjustment value is less than the first angle threshold, and / or the second pitch angle adjustment value is less than the second angle threshold, it is determined that the drone has aimed at the target object.

[0206] In the embodiment of the present application, the drone control device can respectively obtain the environmental image taken by the drone and the eye movement data of the drone operator collected by the eye movement collection device, and then determine the target object that the drone operator is looking at in the environmental image based on the eye movement data, and then control the drone to aim at the target object. It can be seen that when controlling the drone, the embodiment of the present application does not require the operator to manually operate the control device, but analyzes the target object that the operator wants the drone to aim at by collecting the operator's eye movement data. In this way, not only can the automatic aiming of the target object be achieved, but also the accuracy of the drone aiming at the target object and the response speed when aiming at the target object are improved.

[0207] It should be noted that: the drone control device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate when controlling the drone to aim at the target object that the drone operator is interested in. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the drone control device provided in the above embodiment and the drone control method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0208] It should be understood that the "multiple" mentioned herein refers to two or more. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not limit them to be necessarily different.

[0209] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0210] The above is only a specific implementation of the embodiment of the present application, and is not intended to limit the protection scope of the embodiment of the present application. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the embodiment of the present application shall be included in the protection scope of the embodiment of the present application.

Claims

1. A drone eye movement control method, characterized in that: Applied to a control device, the control device is respectively connected to a drone and an eye movement acquisition device for communication, and the method includes: Acquire the environment image captured by the drone and the eye movement data of the drone operator collected by the eye movement collection device; the eye movement data includes the field of view image when the drone operator looks at the environment image, and the gaze point of the drone operator in the field of view image, and the field of view image includes the environment image displayed in the control device; Performing object detection on the environment image to determine at least one detection object included in the environment image and a first image position of the at least one detection object in the environment image; Performing image coordinate system conversion processing on the first image position of the at least one detection object to determine the second image positions corresponding to the at least one detection object in the field of view image; Taking the second image position of the first detection object as the center point, the local image area of ​​the first detection object in the field of view image is expanded in all directions according to the target size to obtain an expanded image area including the first detection object; wherein the first detection object is any one of the at least one detection object; If the gaze point is located within the expanded image area and the distance between the gaze point and the second image position of the first detection object is the smallest, the first detection object is determined as the target object that the drone operator is gazing at; The drone is controlled to aim at the target object.

2. The method according to claim 1, characterized in that The controlling the drone to aim at the target object comprises: Determining a relative distance between a first image position of the target object and an image center point of the environment image, the image center point indicating a current aiming point of the drone; If the relative distance is greater than a distance threshold, calculating a first yaw angle and a first pitch angle when the drone aims at the target object based on the relative distance; Based on the first yaw angle and the first pitch angle, the drone is controlled to aim at the target object.

3. The method according to claim 2, characterized in that The method further comprises: If the relative distance is less than the distance threshold, it is determined that the drone has aimed at the target object.

4. The method according to claim 1, characterized in that The controlling the drone to aim at the target object comprises: Obtaining a first yaw angle adjustment value and a first pitch angle adjustment value calculated in a previous adjustment cycle when the drone aims at the target object; Obtaining an actual yaw angle adjustment value and an actual pitch angle adjustment value when the drone aims at the target object according to the first yaw angle adjustment value and the first pitch angle adjustment value in the previous adjustment period; Determine, based on the first yaw angle adjustment value and the actual yaw angle adjustment value, a second yaw angle adjustment value when the drone aims at the target object in a current adjustment period; Based on the first pitch angle adjustment value and the actual pitch angle adjustment value, a second pitch angle adjustment value when the UAV aims at the target object in a current adjustment period is determined.

5. The method according to claim 4, characterized in that The method further comprises: If the second yaw angle adjustment value is less than the first angle threshold, and / or the second pitch angle adjustment value is less than the second angle threshold, it is determined that the drone has aimed at the target object.

6. A drone control device, characterized in that: Included in a control device, the control device is respectively connected to the drone and the eye movement acquisition device for communication, and the device includes: a data acquisition module, used to acquire the environment image captured by the drone and the eye movement data of the drone operator collected by the eye movement acquisition device; the eye movement data includes a field of view image when the drone operator is gazing at the environment image, and a gaze point of the drone operator in the field of view image, and the field of view image includes the environment image displayed in the control device; a target determination module, configured to perform object detection on the environment image, determine at least one detection object included in the environment image, and a first image position of the at least one detection object in the environment image; perform image coordinate system conversion processing on the first image position of the at least one detection object to determine the second image positions respectively corresponding to the at least one detection object in the field of view image; take the second image position of the first detection object as the center point, and expand the local image area of ​​the first detection object in the field of view image in all directions according to the target size to obtain an extended image area including the first detection object; wherein the first detection object is any one of the at least one detection object; if the gaze point is located in the extended image area, and the distance between the gaze point and the second image position of the first detection object is the smallest, then the first detection object is determined as the target object gazed at by the drone operator; A control module is used to control the drone to aim at the target object.

7. A drone control system, characterized in that: The drone control system includes a drone, an eye movement acquisition device and a control device, wherein the control device is respectively connected to the drone and the eye movement acquisition device for communication; The drone is used to capture environmental images; The eye movement acquisition device is used to collect eye movement data when the drone operator is gazing at the environment image; the eye movement data includes a field of view image when the drone operator is gazing at the environment image, and a gaze point of the drone operator in the field of view image, and the field of view image includes the environment image displayed in the control device; The control device is used to display the environment image taken by the drone and execute the steps of the drone control method according to any one of claims 1 to 5.

Images