Flight device control method and device, electronic equipment and computer storage medium

Abstract

The application provides a flight device control method and device, electronic equipment and computer storage medium. The method comprises the following steps: converting flight data and destination waypoint coordinates of a flight device into an image coordinate system of a depth image in a forward direction of the flight device; judging whether a flight device mark exists in a non-obstacle region in the depth image; the flight device mark is configured to be arranged in the non-obstacle region, and the non-obstacle region is a flyable region of the flight device; if the flight device mark exists in the non-obstacle region, the specific position of the flight device mark in the non-obstacle region is determined; the specific position of the flight device mark in the non-obstacle region is a next position point of the flight device; the pose of the flight device is calculated according to the specific position; and a control instruction is generated based on the pose and a set speed, and the control instruction is configured to control the flight of the flight device. The application generates a control instruction through depth image processing, so as to improve the generation efficiency and control efficiency of the control instruction.

Description

Technical Field

[0001] This application relates to the field of device control, and more specifically, to a flight device control method, device, electronic equipment, and computer storage medium. Background Technology

[0002] With the widespread application of drones, flying robots, and other flying devices, autonomous flight and mission execution capabilities have become core functions. In the current field of flight device control, most applications rely on pre-defined flight path planning to enable the flying device to autonomously complete diverse tasks such as reconnaissance, photography, and logistics. However, in practical applications, encountering obstacles during flight is an unavoidable and common problem. These obstacles pose a serious threat to the safe flight of the flying device, potentially leading to damage or even crashes. Therefore, enabling flying devices to automatically identify and effectively avoid obstacles during flight has become a significant challenge in flight device control technology. However, existing flight device control methods often suffer from low efficiency when dealing with obstacles. Summary of the Invention

[0003] In view of this, the purpose of the embodiments of this application is to provide a flight device control method, device, electronic device and computer storage medium that can improve the control efficiency of the flight device.

[0004] In a first aspect, embodiments of this application provide a flight device control method, comprising: converting flight data and destination waypoint coordinates of the flight device to an image coordinate system of a depth image in the direction of flight of the flight device; determining whether a flight device marker exists in a non-obstacle region of the depth image; wherein the flight device marker is configured to be located in the non-obstacle region, and the non-obstacle region is a flyable area of ​​the flight device; if the flight device marker exists in the non-obstacle region, determining the specific position of the flight device marker in the non-obstacle region; wherein the specific position of the flight device marker in the non-obstacle region is the next position point of the flight device; calculating the pose of the flight device based on the specific position; generating a control command based on the pose and a set speed, the control command being configured to control the flight of the flight device.

[0005] In the above implementation process, using only a single frame of depth image has advantages such as fast processing speed and low resource consumption. By processing the depth image in the forward direction of the flight device to generate control commands for controlling the flight device, the efficiency of control command generation can be improved, thereby improving the control efficiency of the flight device. In addition, the image acquisition device is used to acquire depth images, and it is less affected by wind speed, temperature, and external electromagnetic interference. Therefore, the influence of the environment on depth image acquisition can be reduced, improving the accuracy of depth images, and thus improving the accuracy of control command generation.

[0006] In one embodiment, determining the specific location of the flight device marker in the non-obstacle area includes: calculating the distances between all the flight device markers in the depth image and the destination waypoint coordinates in the image coordinate system; determining the location of the flight device marker with the smallest distance to the destination waypoint coordinates as the location to be confirmed; and determining the location to be confirmed as the specific location if there are no obstacles in front of and behind the location to be confirmed.

[0007] In the above implementation process, since the actual position corresponding to the specific position of the flight device marker is the next position point of the flight device, taking the position of the flight device marker with the smallest distance from the coordinates of the destination waypoint as the specific position of the flight device marker can shorten the distance between the flight device and the destination waypoint as much as possible, thereby shortening the time for the flight device to reach the destination waypoint and improving flight efficiency.

[0008] In one embodiment, determining the specific location of the flight device marker in the non-obstacle area includes: calculating the distances between all flight device markers in the depth image and the destination waypoint coordinates in the image coordinate system; determining the location of the flight device marker with the smallest distance to the destination waypoint coordinates as the location to be confirmed; if there are no obstacles in front of the location to be confirmed, but there are obstacles behind it, and the actual distance of the obstacles from the flight device marker is less than half the actual width of the flight device, subtracting the width of the flight device marker from the boundary coordinates of the obstacles as the specific location; if there are no obstacles in front of the location to be confirmed, but there are obstacles behind it, and the actual distance of the obstacles from the flight device marker is greater than half the actual width of the flight device, determining the location to be confirmed as the specific location.

[0009] In the above implementation process, when there are no obstacles in front of the location to be confirmed, but there are obstacles behind it, and the actual distance between the obstacle and the flight device marker is less than half the actual width of the flight device, the location obtained by subtracting the width of the flight device marker from the boundary coordinates of the obstacle can be used as the specific location. This can prevent the edge of the flight device from touching the obstacle when the flight device flies to the specific location, thereby improving the stability and safety of the flight device.

[0010] In one embodiment, determining the specific location of the flight device marker in the non-obstacle area includes: calculating the distances between all flight device markers in the depth image and the destination waypoint coordinates in the image coordinate system; determining the location of the flight device marker with the smallest distance to the destination waypoint coordinates as the location to be confirmed; if there is an obstacle in front of the location to be confirmed but no obstacle behind it, and the actual distance of the obstacle from the flight device marker is less than half the actual width of the flight device, then adding the width of the flight device marker to the boundary coordinates of the obstacle as the specific location; if there is an obstacle in front of the location to be confirmed but no obstacle behind it, and the actual distance of the obstacle from the flight device marker is greater than half the actual width of the flight device, then determining the location to be confirmed as the specific location.

[0011] In the above implementation process, when there is an obstacle in front of the location to be confirmed but no obstacle behind it, and the actual distance between the obstacle and the flight device marker is less than half the actual width of the flight device, the location obtained by adding the width of the flight device marker to the boundary coordinates of the obstacle can be used as the specific location. This can prevent the edge of the flight device from touching the obstacle when the flight device flies to the specific location, thereby improving the stability and safety of the flight device.

[0012] In one embodiment, the pose of the flight device includes the rotation angle of the flight device; calculating the pose of the flight device based on the specific location includes: calculating the pixel distance from the specific location to the midpoint of the depth image; and calculating the rotation angle based on the pixel distance, the horizontal field of view of the image acquisition device, and the image width of the depth image.

[0013] In the above implementation process, the rotation angle is calculated based on the pixel distance from the specific position in the depth image coordinate system to the midpoint of the depth image, as well as parameters such as the horizontal field of view of the image acquisition device and the width of the depth image. These data are relatively easy to obtain, which can reduce the difficulty of calculating the rotation angle and improve the accuracy and efficiency of the rotation angle calculation.

[0014] In one embodiment, before converting the flight data and destination waypoint coordinates of the flight device to the image coordinate system of the depth image in the direction of flight, the method further includes: acquiring the depth image in the direction of flight; filtering the depth image according to the working distance of the visible range of the image acquisition device; and dividing the filtered depth image into regions according to the depth values ​​of each data in the depth image to obtain obstacle regions and non-obstacle regions.

[0015] In the above implementation process, before dividing the depth image into regions, data outside the working distance of the image acquisition device's field of view is filtered out to reduce image data generated beyond the physical imaging capabilities of the image acquisition device. This avoids these low-quality images from affecting subsequent analysis, processing, or decision-making, thereby improving the accuracy and reliability of the depth image.

[0016] In one embodiment, the step of converting the flight data and destination waypoint coordinates of the flight device to the image coordinate system of the depth image in the direction of flight of the flight device includes: using the first frame of flight data received when the data processing module starts as the origin of the flight data in the relative coordinate system, and determining the transformation parameters for coordinate transformation between the navigation coordinate system and the relative coordinate system based on the coordinates of the first frame of flight data in the navigation coordinate system and the relative coordinate system; converting the flight data and destination waypoint coordinates subsequently received by the data processing module to the relative coordinate system according to the transformation parameters; determining the relative angle between the straight line from the destination waypoint coordinates to the current position point of the flight device and the current yaw angle of the flight device; and converting the flight data and destination waypoint coordinates to the image coordinate system according to the relative angle and the image width of the depth image.

[0017] In the above implementation process, by converting the flight data and destination waypoint data in the navigation coordinate system to the image coordinate system, the difficulty of flight device pose calculation can be reduced because the data in the image coordinate system is simple and easy to obtain, thereby improving the efficiency of control command generation and the control efficiency of the flight device.

[0018] Secondly, embodiments of this application also provide a flight device control device, comprising: a data processing module, configured to convert flight data and destination waypoint coordinates of the flight device into an image coordinate system of a depth image in the direction of flight of the flight device; an obstacle processing module, configured to determine whether a flight device marker exists in a non-obstacle area of ​​the depth image; wherein the flight device marker is configured to be set in the non-obstacle area, the non-obstacle area being a flyable area of ​​the flight device; and if the flight device marker exists in the non-obstacle area, determining the specific position of the flight device marker in the non-obstacle area; wherein the specific position of the flight device marker in the non-obstacle area is the next position point of the flight device; the obstacle processing module is further configured to calculate the pose of the flight device based on the specific position; and a command generation module, configured to generate control commands based on the pose and a set speed, the control commands being configured to control the flight of the flight device.

[0019] Thirdly, embodiments of this application also provide an electronic device, including: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the machine-readable instructions are executed by the processor to perform the steps of the method described in the first aspect above, or any possible implementation of the first aspect.

[0020] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the flight device control method described in the first aspect or any possible implementation of the first aspect.

[0021] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram illustrating the interaction between the image acquisition device and the electronic device provided in the embodiments of this application.

[0024] Figure 2 A block diagram illustrating an electronic device provided in an embodiment of this application;

[0025] Figure 3 A flowchart of a flight device control method provided in the embodiments of this application;

[0026] Figure 4 A schematic diagram of a depth image provided for an embodiment of this application;

[0027] Figure 5 This is a schematic diagram of the functional modules of the flight device control device provided in the embodiments of this application. Detailed Implementation

[0028] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0029] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0030] With the rapid development of technology, flying devices have been integrated into various industries and deeply integrated into our lives. Currently, they are widely used in fields such as aerial photography, film and television shooting, creating romance, express delivery, news reporting, power line inspection, surveying and mapping, agricultural plant protection, wildlife observation, and search and rescue.

[0031] Currently, most applications involve allowing flight devices to autonomously complete inspection tasks according to pre-set flight paths. However, encountering obstacles during flight is a common problem. Therefore, when flight devices encounter obstacles, automatically identifying and effectively avoiding them through real-time measurement can significantly reduce the incidence of drone damage, personnel injury, and building damage caused by operational errors. Reducing human intervention can also improve work efficiency and free up excessive human resources.

[0032] Obstacle avoidance technologies for flight devices mainly include visual obstacle avoidance, radar obstacle avoidance, and lidar obstacle avoidance. Visual obstacle avoidance uses cameras to perceive and analyze the surrounding environment to autonomously avoid obstacles; radar obstacle avoidance uses high-frequency electromagnetic waves to detect surrounding obstacles and calculate distances by sending high-frequency electromagnetic waves and receiving reflected signals; lidar obstacle avoidance uses laser beams to send laser beams and receive reflected signals to achieve high-precision detection and distance measurement of surrounding obstacles.

[0033] However, through long-term research, the inventors of this application have discovered that the electromagnetic waves emitted by radar obstacle avoidance are easily interfered with by other electromagnetic waves and have weak penetration capabilities. For certain metallic objects or objects with specific dielectric constants, their reflected signals may weaken or disappear. Furthermore, lidar is easily affected by environmental factors such as wind speed and temperature changes. In addition, lasers have weak penetration capabilities, and their detection performance may be affected for objects with complex structures or specific reflective properties. At the same time, lidar is expensive, bulky, and requires a large power supply, limiting its application in flight devices.

[0034] In view of this, this application proposes a flight device control method. By processing a depth image along the flight device's forward direction to generate control commands for controlling the flight device, the efficiency of control command generation can be improved, thereby improving the control efficiency of the flight device. Furthermore, the image acquisition device, used for acquiring depth images, is less affected by wind speed, temperature, and external electromagnetic interference, thus reducing the impact of the environment on depth image acquisition, improving the accuracy of the depth images, and consequently improving the accuracy of control command generation.

[0035] To facilitate understanding of this embodiment, the operating environment of a flight device control method disclosed in this application will be described in detail first.

[0036] like Figure 1 The diagram illustrates the interaction between an image acquisition device and an electronic device according to an embodiment of this application. The image acquisition device communicates with the electronic device via a network for data communication or interaction. The image acquisition device may be a camera, webcam, etc. The electronic device may be a microcontroller, programmable controller, etc.

[0037] When the image acquisition device is a camera, the camera is a binocular depth camera.

[0038] The image acquisition device here can be installed on the flight device, and the image acquisition device is configured to acquire images of the environment around the flight device.

[0039] In one embodiment, the image acquisition device is disposed at the front end of the flight device along the forward direction, and the image acquisition device is configured to acquire a depth image of the flight device in the forward direction.

[0040] The image acquisition device acquires one depth image at a time, and the electronic device generates corresponding control commands based on the single depth image and the flight data of the flight device to control the flight device.

[0041] The aforementioned electronic equipment can be integrated inside the flight device, installed on the surface of the flight device, or installed independently of the flight device. The installation method of the electronic equipment can be selected according to the actual situation.

[0042] To facilitate understanding of this embodiment, the electronic device that performs the flight device control method disclosed in this application embodiment will be described in detail below.

[0043] like Figure 2 The diagram shown is a block illustration of an electronic device. The electronic device 100 may include a memory 111 and a processor 113. Those skilled in the art will understand that... Figure 2 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device 100. For example, the electronic device 100 may also include components that are more... Figure 2 The more or fewer components shown, or having the same Figure 2 The different configurations shown.

[0044] The aforementioned memory 111 and processor 113 are electrically connected to each other directly or indirectly to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The aforementioned processor 113 is used to execute executable modules stored in the memory.

[0045] The memory 111 can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory 111 stores programs, and the processor 113 executes these programs upon receiving execution instructions. The methods executed by the electronic device 100 as defined in any embodiment of this application can be applied to or implemented by the processor 113.

[0046] The processor 113 described above can be an integrated circuit chip with signal processing capabilities. The processor 113 can also be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a digital signal processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0047] The electronic device 100 in this embodiment can be used to execute various steps in the various methods provided in the embodiments of this application. The implementation process of the flight device control method is described in detail below through several embodiments.

[0048] Please see Figure 3 This is a flowchart of the flight device control method provided in the embodiments of this application. The following will describe... Figure 3 The specific process shown will be explained in detail.

[0049] Step 201: Convert the flight data of the flight device and the coordinates of the destination waypoint to the image coordinate system of the depth image in the direction of flight of the flight device.

[0050] The flight data can include positioning data, acceleration data, angular velocity data, etc., of the flight device, and this flight data can be adjusted according to the actual situation.

[0051] The destination waypoint coordinates here are the Earth coordinates of the destination that the flight device needs to reach to perform flight maneuvers.

[0052] The aforementioned depth image is a depth map used to record information about the distance between objects in the scene and the image acquisition device. Specifically, this depth image is used to record information about the distance between objects around the flight device and the image acquisition device.

[0053] In one embodiment, prior to step 201, the method further includes: acquiring a depth image of the flight direction of the flying device.

[0054] It should be understood that during flight, the flight device needs to understand the distribution of obstacles in its direction of travel in real time in order to formulate flight strategies in advance, bypass obstacles, and improve the safety of the flight. By installing an image acquisition device on the flight device, depth images of the flight device in its direction of travel can be obtained, and the distribution of obstacles can be determined based on the acquired depth images.

[0055] Step 202: Determine whether there is a flight device marker in the non-obstacle region of the depth image.

[0056] The flight device identifier is a specification mark used in the depth image to represent the flight device. The specification marks for different flight devices may be the same or different. For example, the shape of the flight device identifier can be rectangular, square, circular, irregular, etc., and the projected area of ​​the flight device identifier can be proportionally scaled according to the actual volume of the flight device. The specifications of the flight device identifier can be selected according to the actual situation.

[0057] The flight device marker here is positioned in a non-obstacle area, which is the flightable area for the flight device. This flight device marker is used to simulate the flight device's actions in the non-obstacle area.

[0058] In one embodiment, the setting specification mark of the same flight device in the depth image should be fixed.

[0059] It should be understood that since the flight device markings corresponding to the flight device are specification markings, if the obstacle area in the depth image is large and the corresponding non-obstacle area is small, then the flight device markings may not be able to be accommodated in the non-obstacle area. In this case, it can be determined that the flight device markings do not exist in the non-obstacle area.

[0060] Optionally, prior to step 202, the method further includes: searching a non-obstacle region of the depth image to determine whether a flight device marker exists in the non-obstacle region.

[0061] Step 203: If a flight device marker exists in the non-obstacle area, determine the specific location of the flight device marker in the non-obstacle area.

[0062] The specific location of the flight device marker in the non-obstacle area is the next location point of the flight device.

[0063] Understandably, since the flying device can fly in any non-obstacle area, the corresponding flying device marker can be located in multiple positions within that area. In reality, during flight, the next position is only one of these multiple locations. Therefore, the specific position here is one of the multiple locations of the flying device marker within the non-obstacle area.

[0064] In one embodiment, if there is no flight device marker in the non-obstacle area, the flight device is controlled to rotate at a set angle, and a depth image is reacquired after the flight device rotates.

[0065] The preset angle can be set in advance, and the flight device will rotate directly according to the preset angle when it needs to rotate. If the flight device's logo is still difficult to fit in the non-obstacle area of ​​the depth image after rotating to the preset angle, the flight device can be controlled to continue rotating at the preset angle until the flight device's logo can be fit in the non-obstacle area of ​​the depth image.

[0066] It should be understood that if the obstacle area in the depth image is large, making it difficult for the flight device's marker to appear in the non-obstacle area, it indicates that the image acquisition device may be too close to the obstacle or the acquisition angle may be inappropriate. In this case, by rotating the flight device, the distance between the image acquisition device and the obstacle or the acquisition angle can be adjusted so that the non-obstacle area in the acquired depth image can accommodate the flight device's marker.

[0067] In one embodiment, step 202 may further be: determining whether a complete flight device marker exists in the non-obstacle region of the depth image.

[0068] Accordingly, step 203 can be: if a complete flight device marker exists in the non-obstacle area, then determine the specific location of the flight device marker in the non-obstacle area.

[0069] Step 204: Calculate the attitude of the flight device based on the specific location.

[0070] The attitude of the flight device includes its rotation angle, rotation acceleration, rotation angular velocity, and rotation direction, which can be selected according to the actual situation.

[0071] Step 205: Generate control commands based on pose and set speed, and configure the control commands to control the flight of the flight device.

[0072] The set speed here can be pre-set and stored according to actual needs. When controlling the flight device, the set speed and calculated posture can be directly called to generate corresponding control commands.

[0073] Of course, this set speed can also be calculated in real time based on the flight information of the flight device. When controlling the flight device, corresponding control commands can be generated based on the actually calculated set speed and the calculated attitude. The method of setting this set speed can be selected according to the actual situation.

[0074] In the above implementation process, using only a single frame of depth image has advantages such as fast processing speed and low resource consumption. By processing the depth image in the forward direction of the flight device to generate control commands for controlling the flight device, the efficiency of control command generation can be improved, thereby improving the control efficiency of the flight device. In addition, the image acquisition device is used to acquire depth images, and it is less affected by wind speed, temperature, and external electromagnetic interference. Therefore, the influence of the environment on depth image acquisition can be reduced, improving the accuracy of depth images, and thus improving the accuracy of control command generation.

[0075] In one possible implementation, step 203 includes: calculating the distances between all flight device markers in the depth image and the destination waypoint coordinates in the image coordinate system; determining the location of the flight device marker with the smallest distance to the destination waypoint coordinates as the location to be confirmed; and determining the location to be confirmed as the specific location if there are no obstacles in front of or behind the location to be confirmed.

[0076] The distance between the flight device marker and the destination waypoint coordinates in the image coordinate system can be Euclidean distance.

[0077] Specifically, the distance between the flight device marker and the destination waypoint coordinates in the image coordinate system can be calculated using the following formula:

[0078]

[0079] Where Dis1 is the distance between the flight device marker and the destination waypoint coordinates in the image coordinate system, (Rx, Ry) are the setpoint coordinates of the flight device marker, and (Dx, Dy) are the destination waypoint coordinates.

[0080] Optionally, the setting point of the flight device mark can be the midpoint of the flight device mark, the edge point of the flight device mark (e.g., the vertex), etc., and the setting point of the flight device mark can be selected according to the actual situation.

[0081] It should be understood that, such as Figure 4 As shown, all non-obstacle areas are areas where the flight device can fly. Figure 4 The image shows areas other than triangles, rhombuses, and hexagons as non-obstacle areas. In other words, non-obstacle areas in a depth image can include multiple flight device markers (such as...). Figure 4The solid and dashed rectangular markers shown in the image represent the positions the flight device might need to reach at the next moment. However, in actual flight maneuvers, the flight device typically only reaches one position at a time. Therefore, to shorten the time it takes for the flight device to reach its destination waypoint and improve flight efficiency, after determining the distances between all flight device markers in the depth image and the destination waypoint coordinates in the image coordinate system, the position of the flight device marker with the smallest distance to the destination waypoint coordinates can be determined as the specific location of the flight device marker. The next position of the flight device is its corresponding position in the Earth coordinate system.

[0082] In the above implementation process, since the actual position corresponding to the specific position of the flight device marker is the next position point of the flight device, taking the position of the flight device marker with the smallest distance from the coordinates of the destination waypoint as the specific position of the flight device marker can shorten the distance between the flight device and the destination waypoint as much as possible, thereby shortening the time for the flight device to reach the destination waypoint and improving flight efficiency.

[0083] In one possible implementation, step 203 includes: calculating the distances between all flight device markers in the depth image and the destination waypoint coordinates in the image coordinate system; and determining the location of the flight device marker with the smallest distance to the destination waypoint coordinates as the location to be confirmed.

[0084] In cases where there are no obstacles in front of the location to be confirmed, but there are obstacles behind it, and the actual distance between the obstacles and the flight device marker is less than half the actual width of the flight device, the location obtained by subtracting the width of the flight device marker from the boundary coordinates of the obstacles is taken as the specific location.

[0085] If there are no obstacles in front of the location to be confirmed, but there are obstacles behind it, and the actual distance between the obstacles and the flight device marker is greater than half the actual width of the flight device, then the location to be confirmed is determined as a specific location.

[0086] It should be understood that if the actual distance between an obstacle and the flight device marker is less than half the actual width of the flight device, the flight device may collide with the obstacle when it reaches the location to be confirmed, thus affecting the stability of the flight and potentially damaging the flight device. Therefore, when the actual distance between an obstacle and the flight device marker is less than half the actual width of the flight device, the specific position of the flight device marker can be re-determined based on the coordinates of the obstacle and the specifications of the flight device marker.

[0087] Specifically, if there is an obstacle behind the location to be confirmed, the tail of the aircraft may touch the obstacle after reaching the location. In this case, the new location is determined by subtracting the width of the aircraft's marker from the boundary coordinates of the obstacle.

[0088] In the above implementation process, when there are no obstacles in front of the location to be confirmed, but there are obstacles behind it, and the actual distance between the obstacle and the flight device marker is less than half the actual width of the flight device, the location obtained by subtracting the width of the flight device marker from the boundary coordinates of the obstacle can be used as the specific location. This can prevent the edge of the flight device from touching the obstacle when the flight device flies to the specific location, thereby improving the stability and safety of the flight device.

[0089] In one possible implementation, step 203 includes: calculating the distances between all flight device markers in the depth image and the destination waypoint coordinates in the image coordinate system; and determining the location of the flight device marker with the smallest distance to the destination waypoint coordinates as the location to be confirmed.

[0090] In cases where there is an obstacle in front of the location to be confirmed but no obstacle behind it, and the actual distance between the obstacle and the flight device marker is less than half the actual width of the flight device, the location obtained by adding the width of the flight device marker to the boundary coordinates of the obstacle is taken as the specific location.

[0091] If there is an obstacle in front of the location to be confirmed but no obstacle behind it, and the actual distance between the obstacle and the flight device marker is greater than half the actual width of the flight device, then the location to be confirmed is determined as a specific location.

[0092] It should be understood that if there is an obstacle in front of the location to be confirmed, the front end of the flight device may touch the obstacle after reaching the location. In this case, the new location is determined by adding the width of the flight device's marker to the boundary coordinates of the obstacle.

[0093] In the above implementation process, when there is an obstacle in front of the location to be confirmed but no obstacle behind it, and the actual distance between the obstacle and the flight device marker is less than half the actual width of the flight device, the location obtained by adding the width of the flight device marker to the boundary coordinates of the obstacle can be used as the specific location. This can prevent the edge of the flight device from touching the obstacle when the flight device flies to the specific location, thereby improving the stability and safety of the flight device.

[0094] In one possible implementation, step 204 includes: calculating the pixel distance from the specific location to the midpoint of the depth image; and calculating the rotation angle based on the pixel distance, the horizontal field of view of the image acquisition device, and the image width of the depth image.

[0095] The pixel distance from a specific location to the midpoint of the depth image can be calculated using the following formula:

[0096]

[0097] Where Dis2 is the pixel distance from the specific location to the midpoint of the image, (Px, Py) are the coordinates of the specific location, and (Qx, Qy) are the coordinates of the midpoint of the image.

[0098] In one embodiment, the rotation angle can be calculated using the following formula:

[0099]

[0100] Where angle is the rotation angle, width is the image width of the depth image, h_fov is the horizontal field of view of the image acquisition device, and Dis2 is the pixel distance from the specific location to the midpoint of the image.

[0101] In the above implementation process, the rotation angle is calculated based on the pixel distance from the specific position in the depth image coordinate system to the midpoint of the depth image, as well as parameters such as the horizontal field of view of the image acquisition device and the width of the depth image. These data are relatively easy to obtain, which can reduce the difficulty of calculating the rotation angle and improve the accuracy and efficiency of the rotation angle calculation.

[0102] In one possible implementation, before step 201, the method further includes: acquiring a depth image of the flight direction; filtering the depth image according to the working distance of the image acquisition device's field of view; and dividing the filtered depth image into regions based on the depth values ​​of each data point in the depth image to obtain obstacle regions and non-obstacle regions.

[0103] The process of acquiring a depth image of the flight path includes: acquiring a depth image from an image acquisition device.

[0104] The visible range here can be the range corresponding to the minimum working distance and the maximum working distance of the image acquisition device.

[0105] It should be understood that image acquisition devices typically have a specific field of view, which can depend on the focal length, sensor size, lens aperture, and other physical characteristics of the device. Objects outside the field of view are beyond the device's physical imaging capabilities, and their images usually become blurry, distorted, or even completely invisible. Consequently, the data quality is relatively low, which may affect subsequent analysis, processing, or decision-making. Therefore, to ensure the accuracy and reliability of the data, data outside the working distance of the field of view can be filtered out.

[0106] Each pixel in the aforementioned depth image has a corresponding depth value. This depth image can be divided based on the differences in the depth values ​​of its individual pixels.

[0107] For example, for backgrounds and obstacles with significant differences in depth values, a depth value threshold can be set based on the characteristics of the depth values ​​of the background and obstacles. The background and obstacles are then divided into regions based on the comparison between the depth values ​​and the depth value threshold.

[0108] The background area can be a non-obstacle area.

[0109] In the above implementation process, before dividing the depth image into regions, data outside the working distance of the image acquisition device's field of view is filtered out to reduce image data generated beyond the physical imaging capabilities of the image acquisition device. This avoids these low-quality images from affecting subsequent analysis, processing, or decision-making, thereby improving the accuracy and reliability of the depth image.

[0110] In one possible implementation, step 201 includes: using the first frame of flight data received when the data processing module starts as the origin of the flight data in the relative coordinate system, and determining the transformation parameters for coordinate transformation between the navigation coordinate system and the relative coordinate system based on the coordinates of the first frame of flight data in the navigation coordinate system and the relative coordinate system; transforming the flight data and destination waypoint coordinates subsequently received by the data processing module into the relative coordinate system according to the transformation parameters; determining the relative angle based on the straight line from the destination waypoint coordinates to the current position point of the flight device and the current yaw angle of the flight device; and transforming the flight data and destination waypoint coordinates into the image coordinate system based on the relative angle and the image width of the depth image.

[0111] The data processing module can be a microcontroller, programmable controller, or other similar device, and the appropriate module can be selected based on the specific circumstances.

[0112] It should be understood that the first frame of flight data received by the data processing module includes the actual coordinate values ​​of the flight device. At this time, if the actual coordinate values ​​of the flight device are taken as the origin (0, 0, 0) of the relative coordinate system, then there is a certain difference between the actual coordinate values ​​and the (0, 0, 0) point. Subsequent flight data can also be converted to this relative coordinate system according to this difference.

[0113] In one embodiment, the conversion formula for transforming flight data and destination waypoint coordinates to a relative coordinate system can be as follows:

[0114] X ENU =Y NED ;

[0115] Y ENU =X NED;

[0116] Z ENU =-Z NED ;

[0117] Among them, X ENU The first direction coordinate in the relative coordinate system, Y ENU The second direction coordinates in the relative coordinate system, Z ENU For the third-direction coordinates in the relative coordinate system, X NED The first direction coordinate in the navigation coordinate system, Y NED The second direction coordinate in the navigation coordinate system, Z NED This refers to the third-direction coordinates in the navigation coordinate system.

[0118] The aforementioned relative angles can be calculated using the following formula:

[0119] β=Yaw ENU -α;

[0120] Yaw ENU =-Yaw NED +90;

[0121] a = atan2(gy-cy, gx-cx);

[0122] Among them, Yaw ENU Let (gx, gy) be the current yaw angle in the relative coordinate system, and β be the relative angle.

[0123] Let (cx, cy) be the coordinates of the destination waypoint in the relative coordinate system, (cx, cy) be the current coordinates of the flight device in the relative coordinate system, and a be the yaw angle of the destination relative to the current point in the relative coordinate system.

[0124] In one embodiment, flight data and destination waypoint coordinates can be transformed to the image coordinate system using the following formula:

[0125] Dx=width / 2+β×width / h_fov_;

[0126] Dy = width / 2;

[0127] Where width is the image width of the depth image, β is the relative angle, h_fov_ is the horizontal field of view of the image acquisition device, and (Dx, Dy) are the coordinates in the image coordinate system.

[0128] In the above implementation process, by converting the flight data and destination waypoint data in the navigation coordinate system to the image coordinate system, the difficulty of flight device pose calculation can be reduced because the data in the image coordinate system is simple and easy to obtain, thereby improving the efficiency of control command generation and the control efficiency of the flight device.

[0129] Based on the same application concept, this application also provides a flight device control device corresponding to the flight device control method. Since the principle of the device in this application is similar to that of the aforementioned flight device control method embodiment, the implementation of the device in this application can refer to the description in the above method embodiment, and the repeated parts will not be repeated.

[0130] Please see Figure 5 This is a functional module diagram of the flight device control device provided in this application embodiment. Each module in the flight device control device in this embodiment is used to execute the steps in the above method embodiments. The flight device control device includes a data processing module 301, an obstacle handling module 302, and a command generation module 303; wherein,

[0131] The data processing module 301 is used to convert the flight data of the flight device and the coordinates of the destination waypoint to the image coordinate system of the depth image of the flight device's forward direction.

[0132] The obstacle processing module 302 is used to determine whether there is a flight device marker in the non-obstacle region of the depth image; wherein the flight device marker is configured to be set in the non-obstacle region, which is the flightable area of ​​the flight device; and if the flight device marker exists in the non-obstacle region, the specific position of the flight device marker in the non-obstacle region is determined; wherein the specific position of the flight device marker in the non-obstacle region is the next position point of the flight device.

[0133] The obstacle handling module 302 is also used to calculate the attitude of the flight device based on the specific location.

[0134] The instruction generation module 303 is used to generate control instructions based on the pose and set speed, and the control instructions are configured to control the flight device to fly.

[0135] In one possible implementation, the obstacle processing module 302 is specifically used to: calculate the distance between all the flight device markers in the depth image and the destination waypoint coordinates in the image coordinate system; determine the location of the flight device marker with the smallest distance to the destination waypoint coordinates as the location to be confirmed; and, if there are no obstacles in front of and behind the location to be confirmed, determine the location to be confirmed as the specific location.

[0136] In one possible implementation, the obstacle processing module 302 is specifically configured to: calculate the distances between all the flight device markers in the depth image and the destination waypoint coordinates in the image coordinate system; determine the location of the flight device marker with the smallest distance to the destination waypoint coordinates as the location to be confirmed; if there are no obstacles in front of the location to be confirmed, but there are obstacles behind it, and the actual distance of the obstacles from the flight device marker is less than half the actual width of the flight device, subtract the width of the flight device marker from the boundary coordinates of the obstacles as the specific location; if there are no obstacles in front of the location to be confirmed, but there are obstacles behind it, and the actual distance of the obstacles from the flight device marker is greater than half the actual width of the flight device, determine the location to be confirmed as the specific location.

[0137] In one possible implementation, the obstacle processing module 302 is specifically configured to: calculate the distances between all the flight device markers in the depth image and the destination waypoint coordinates in the image coordinate system; determine the location of the flight device marker with the smallest distance to the destination waypoint coordinates as the location to be confirmed; if there is an obstacle in front of the location to be confirmed but no obstacle behind it, and the actual distance of the obstacle from the flight device marker is less than half the actual width of the flight device, add the width of the flight device marker to the boundary coordinates of the obstacle as the specific location; if there is an obstacle in front of the location to be confirmed but no obstacle behind it, and the actual distance of the obstacle from the flight device marker is greater than half the actual width of the flight device, determine the location to be confirmed as the specific location.

[0138] In one possible implementation, the obstacle processing module 302 is specifically used to: calculate the pixel distance from the specific location to the midpoint of the depth image; and calculate the rotation angle based on the pixel distance, the horizontal field of view of the image acquisition device, and the image width of the depth image.

[0139] In one possible implementation, the flight device control device further includes a processing unit for acquiring a depth image of the flight direction; filtering the depth image according to the working distance of the image acquisition device's field of view; and dividing the filtered depth image into regions based on the depth values ​​of each data point in the depth image to obtain obstacle regions and non-obstacle regions.

[0140] In one possible implementation, the data processing module 301 is specifically configured to: use the first frame of flight data received when the data processing module starts as the origin of the flight data in the relative coordinate system, and determine the transformation parameters for coordinate transformation between the navigation coordinate system and the relative coordinate system based on the coordinates of the first frame of flight data in the navigation coordinate system and the relative coordinate system; transform the flight data and the destination waypoint coordinates subsequently received by the data processing module to the relative coordinate system according to the transformation parameters; determine the relative angle between the straight line from the destination waypoint coordinates to the current position point of the flight device and the current yaw angle of the flight device; and transform the flight data and the destination waypoint coordinates to the image coordinate system based on the relative angle and the image width of the depth image.

[0141] Furthermore, this application embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the flight device control method described in the above method embodiments.

[0142] The computer program product of the flight device control method provided in this application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the steps of the flight device control method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0143] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0144] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0145] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. It should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0146] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0147] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for controlling a flight device, characterized in that, include: The flight data and destination waypoint coordinates of the flight device are converted to the image coordinate system of the depth image in the direction of flight of the flight device; wherein, the depth image is used to record the distance information of objects in the scene from the image acquisition device; the depth image is a single depth image; Determine whether a flight device marker exists in the non-obstacle region of the depth image; wherein the flight device marker is configured to be set in the non-obstacle region, the non-obstacle region being the flyable area of ​​the flight device; the flight device marker is a specification marker in the depth image used to represent the flight device, and the flight device marker is used to simulate the movement of the flight device in the non-obstacle region; If the flight device marker exists in the non-obstacle area, then the specific location of the flight device marker in the non-obstacle area is determined; wherein, the specific location of the flight device marker in the non-obstacle area is the next location point of the flight device; Calculate the attitude of the flight device based on the specific location; Based on the pose and set speed, control commands are generated, configured to control the flight device to fly; or, If the flight device marker is not present in the non-obstacle area, the flight device is controlled to rotate at a set angle, and the depth image is reacquired after the flight device rotates. Determining the specific location of the flight device marker in the non-obstacle area includes: Calculate the distance between all the flight device markers in the depth image and the destination waypoint coordinates in the image coordinate system; The location of the flight device marker that has the smallest distance from the coordinates of the destination waypoint is determined as the location to be confirmed; If there are no obstacles in front of or behind the location to be confirmed, the location to be confirmed is determined to be the specific location.

2. The method according to claim 1, characterized in that, Determining the specific location of the flight device marker in the non-obstacle area further includes: If there are no obstacles in front of the location to be confirmed, but there are obstacles behind it, and the actual distance between the obstacles and the flight device marker is less than half the actual width of the flight device, the location obtained by subtracting the width of the flight device marker from the boundary coordinates of the obstacles shall be taken as the specific location. If there are no obstacles in front of the location to be confirmed, but there are obstacles behind it, and the actual distance between the obstacles and the flight device marker is greater than half the actual width of the flight device, then the location to be confirmed is determined to be the specific location.

3. The method according to claim 1, characterized in that, Determining the specific location of the flight device marker in the non-obstacle area further includes: If there is an obstacle in front of the location to be confirmed but no obstacle behind it, and the actual distance between the obstacle and the flight device marker is less than half the actual width of the flight device, the location obtained by adding the width of the flight device marker to the boundary coordinates of the obstacle shall be taken as the specific location. If there is an obstacle in front of the location to be confirmed but no obstacle behind it, and the actual distance between the obstacle and the flight device marker is greater than half the actual width of the flight device, then the location to be confirmed is determined to be the specific location.

4. The method according to any one of claims 1-3, characterized in that, in, The attitude of the flight device includes the rotation angle of the flight device; The calculation of the attitude of the flight device based on the specific location includes: Calculate the pixel distance from the specific location to the midpoint of the depth image; The rotation angle is calculated based on the pixel distance, the horizontal field of view of the image acquisition device, and the image width of the depth image.

5. The method according to any one of claims 1-3, characterized in that, Before converting the flight data and destination waypoint coordinates of the flight device to the image coordinate system of the depth image in the direction of flight of the flight device, the method further includes: Acquire a depth image of the flight path of the flight device; The depth image is filtered according to the working distance of the image acquisition device's visible range; The filtered depth image is divided into regions based on the depth values ​​of each data point in the depth image, resulting in obstacle regions and non-obstacle regions.

6. The method according to any one of claims 1-3, characterized in that, The process of converting the flight data and destination waypoint coordinates of the flight device to the image coordinate system of the depth image in the direction of flight includes: When the data processing module starts up, the first frame of flight data received is taken as the origin of the flight data in the relative coordinate system, and the transformation parameters for coordinate transformation between the navigation coordinate system and the relative coordinate system are determined based on the coordinates of the first frame of flight data in the relative coordinate system. The flight data and destination waypoint coordinates subsequently received by the data processing module are converted to the relative coordinate system according to the conversion parameters. The relative angle is determined by the straight line from the destination waypoint coordinates to the current position of the flight device and the current yaw angle of the flight device. Based on the relative angle and the image width of the depth image, the flight data and the destination waypoint coordinates are transformed into the image coordinate system.

7. A flight device control device, characterized in that, include: The data processing module is used to convert the flight data and destination waypoint coordinates of the flight device into the image coordinate system of the depth image in the direction of flight of the flight device; wherein, the depth image is used to record the distance information of objects in the scene from the image acquisition device; the depth image is a single depth image; An obstacle processing module is used to determine whether a flight device marker exists in a non-obstacle region of the depth image; wherein the flight device marker is configured to be set in the non-obstacle region, which is a flyable area for the flight device; the flight device marker is a specification marker in the depth image used to represent the flight device, and the flight device marker is used to simulate the movement of the flight device in the non-obstacle region; and If the flight device marker exists in the non-obstacle area, then the specific location of the flight device marker in the non-obstacle area is determined; wherein, the specific location of the flight device marker in the non-obstacle area is the next location point of the flight device; The obstacle handling module is also used to calculate the attitude of the flight device based on the specific location; The instruction generation module is used to generate control instructions based on the pose and a set speed. The control instructions are configured to control the flight device to fly; or, if there is no flight device marker in the non-obstacle area, the flight device is controlled to rotate at a set angle, and the depth image is reacquired after the flight device rotates. The obstacle processing module is specifically used to calculate the distance between all the flight device markers in the depth image and the destination waypoint coordinates in the image coordinate system; determine the location of the flight device marker with the smallest distance to the destination waypoint coordinates as the location to be confirmed; and determine the location to be confirmed as the specific location if there are no obstacles in front of and behind the location to be confirmed.

8. An electronic device, characterized in that, include: The processor and memory, wherein the memory stores machine-readable instructions executable by the processor, wherein when the electronic device is running, the machine-readable instructions are executed by the processor to perform the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the method as described in any one of claims 1 to 6.

Images