Unmanned aerial vehicle obstacle avoidance method and system based on binocular vision

Abstract

The invention discloses an unmanned aerial vehicle obstacle avoidance method and system based on binocular vision. The method comprises the following steps: acquiring a scene image in front of an unmanned aerial vehicle; performing binarization processing on the depth map according to the depth map information and flight parameters of the unmanned aerial vehicle to obtain a binarized depth map; determining whether a passable area of the unmanned aerial vehicle exists in the binary depth map or not according to the binary depth map and the size information of the unmanned aerial vehicle; and when the passable area exists, controlling the unmanned aerial vehicle to fly according to a preset passing speed until the unmanned aerial vehicle passes through the actual physical area corresponding to the passable area. According to the invention, the depth map of the obstacle is obtained by using the binocular camera of the unmanned aerial vehicle, whether the passable area of the unmanned aerial vehicle exists is determined based on the depth map, and finally the unmanned aerial vehicle is controlled to pass through the safe area to complete obstacle-avoiding flight, the implementation process is simple, and complex extra parts do not need to be carried. And the requirement on the hardware performance of the unmanned aerial vehicle can be greatly reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicles, and in particular to an obstacle avoidance method and system for unmanned aerial vehicles based on binocular vision. Background Art

[0002] In the prior art, when performing obstacle avoidance control for drones, the method often used is to make the drone stop in front of an obstacle, but this method is mostly applicable to rotary-wing drones. For fixed-wing drones, obstacle avoidance is mainly achieved by climbing upwards, but the traffic efficiency of this solution is not high. Summary of the invention

[0003] In view of the shortcomings of the prior art, the present invention provides a UAV obstacle avoidance method and system based on binocular vision. The method first uses the UAV binocular camera to obtain an obstacle depth map, and determines whether there is a passable area for the UAV based on the depth map. Finally, the UAV is controlled to pass through a safe area to complete obstacle avoidance flight. The implementation process is simple and does not require complex additional components, which can greatly reduce the requirements for the hardware performance of the UAV.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] On the one hand, a binocular vision-based obstacle avoidance method for a drone is provided, which comprises the following steps:

[0006] Obtaining a scene image in front of the drone through a binocular camera mounted on the drone, and obtaining obstacle information based on the scene image;

[0007] Obtaining depth map information according to the scene image, and binarizing the depth map according to the depth map information and flight parameters of the drone to obtain a binary depth map;

[0008] Determining whether there is a traversable area for the drone in the binary depth map according to the binary depth map and the size information of the drone;

[0009] When there is a passable area, the drone is controlled to fly at a preset passable speed until it passes through the actual physical area corresponding to the passable area.

[0010] Preferably, binarization of the depth map comprises the following steps:

[0011] The current flight speed v0 of the drone and the braking distance d at the flight speed v0 are obtained in real time, and the binary threshold threshold is set according to formula (1):

[0012] threshold=(d+v0t)*δ (1)

[0013] Among them, t is the information transmission time of the drone communication link from the current flight state to the braking state; δ is the safety factor;

[0014] Traverse each pixel point of the depth map, if the pixel value of the current pixel point is ≥ threshold, set the pixel value to 0, if the pixel value of the current pixel point is < threshold, set the pixel value to 1, thereby completing the binarization processing of the depth map and obtaining a binary depth map.

[0015] Preferably, determining whether there is a traversable area for the drone based on the binary depth map and the size information of the drone comprises the following steps:

[0016] Determine a three-dimensional spherical enclosing space P0 that can completely enclose the drone according to the size information of the drone, and take a two-dimensional circular section P01 passing through the center of the three-dimensional spherical enclosing space;

[0017] Determine a plurality of circular areas P1 in the binary depth map, and each circular area P1 is composed of continuous pixel points with a pixel value of 0;

[0018] Obtaining the actual physical area corresponding to each circular area P1;

[0019] The area s1 of each actual physical area is compared with the area s0 of the two-dimensional circular section P01. If the area s1 of a certain actual physical area is greater than s0, the circular area P1 corresponding to the actual physical area is determined as the passable area for the drone. If the areas s1 of all actual physical areas are ≦s0, it is considered that there is no passable area.

[0020] Preferably, obtaining the actual physical area corresponding to each circular area P1 includes the following steps:

[0021] According to R1=(D×R) / f, each circular area P1 is converted into its corresponding actual physical area, where R1 represents the radius of the actual physical area; f represents the pixel focal length of the binocular camera; R represents the pixel radius of the circular area P1; and D represents the shortest distance between the drone and the obstacle solved according to the depth map.

[0022] Preferably, if there is no traversable area, the drone is controlled to hover.

[0023] Preferably, controlling the drone to fly at a preset speed until it passes through the actual physical area corresponding to the passable area comprises the following steps:

[0024] Obtain the pixel coordinates of the center point of the passable area (PassReigon.x, PassReigon.y);

[0025] The yaw angle of the drone relative to the traversable area is obtained according to the following formula:

[0026] angleX=(|image.cols / 2-PassReigon.x| / image.cols)*HFOV

[0027] angleY=(|image.rows / 2-PassReigon.y| / image.rows)*VFOV

[0028] Among them, angleX represents the horizontal yaw angle of the drone relative to the traversable area in the horizontal direction; angleY ​​represents the vertical yaw angle of the drone relative to the traversable area in the vertical direction; image.cols and image.rows represent the width and height of the binary depth map respectively; HFOV and VFOV represent the horizontal field of view and vertical field of view of the binocular camera carried on the drone respectively;

[0029] The passing speed of the drone when passing through the actual physical area corresponding to the passable area is set according to the current distance between the obstacle and the drone, and the drone is controlled to fly at the passing speed until it passes through the actual physical area corresponding to the passable area.

[0030] Preferably, controlling the drone to fly at a normal speed includes:

[0031] The travel speed is decomposed into the X-axis speed vx, the Y-axis speed vy and the Z-axis speed vz, where the X-axis, Y-axis and Z-axis are the three axes of the body coordinate system O-XYZ:

[0032] vx=v*cos(angleY)*cos(angleX)

[0033] vy=v*cos(angleY)*sin(angleX)

[0034] vz=v*sin(angleY)

[0035] The power output of the drone is controlled so that it flies at an X-axis speed vx, a Y-axis speed vy, and a Z-axis speed vz until it passes through the actual physical area corresponding to the passable area.

[0036] Preferably, when the obstacle information includes the distance between the obstacle and the drone, and the distance is greater than a distance threshold, there is no need to reduce the flight speed of the drone.

[0037] Preferably, the depth map information includes three-dimensional coordinates of obstacles.

[0038] On the other hand, a drone obstacle avoidance system is also provided, which includes:

[0039] A binocular camera, which is mounted on the drone and used to obtain images of the scene in front of the drone;

[0040] A depth map generating unit, configured to generate a disparity map according to the scene image, and to generate a depth map according to the disparity map;

[0041] A binarization unit, connected to the depth map generation unit, configured to perform binarization processing on the depth map to obtain a binarized depth map;

[0042] A passable area determination unit, connected to the binarization unit, for determining whether there is a passable area for the drone in the binary depth map according to the binary depth map and the size information of the drone;

[0043] and a flight control unit, which is used to control the UAV to fly at a passable speed when there is a passable area until it passes through the actual physical area corresponding to the passable area, and / or, when there is no passable area, control the UAV to hover, and / or, when the distance between the obstacle and the UAV is greater than a distance threshold, control the UAV to maintain the current flight speed.

[0044] The present invention uses the drone's binocular camera to obtain an obstacle depth map, and based on the position information, distance information and size information of the drone contained in the depth map pixels, accurately determines whether there is a passable area for the drone, and finally controls the drone to pass through the safe area according to the set passing speed to complete the obstacle avoidance flight. It can make full use of the depth map information and does not require the installation of complex additional components, which can greatly reduce the requirements for the drone's hardware performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flowchart of the steps of the obstacle avoidance method of the drone in the present invention;

[0046] Figure 2 It is a schematic diagram of the structure of the drone and the body coordinate system in the present invention;

[0047] Figure 3 is the depth map before and after binarization processing in the present invention;

[0048] Figure 4 It is a schematic diagram of a three-dimensional spherical enclosed space and a two-dimensional circular cross section in the present invention;

[0049] Figure 5 Schematic diagram of the passable area PassReigon of the drone in the present invention;

[0050] Figure 6Schematic diagram of the relative position relationship between the drone and the binary depth map in the present invention;

[0051] Figure 7 It is a schematic diagram of the relative position relationship between the circular area and the actual physical area in the present invention;

[0052] Figure 8 It is a structural schematic diagram of the obstacle avoidance method for unmanned aerial vehicles in the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] Embodiment 1:

[0055] like Figure 1-2 As shown, this embodiment provides a drone obstacle avoidance method based on binocular vision. In this embodiment, the drone 100 includes a rotary-wing drone and a fixed-wing drone, and a body coordinate system 0-XYZ of the drone 100 is established. On this basis, the drone obstacle avoidance method includes the following steps:

[0056] S1. Obtaining a scene image in front of the drone 100 through a binocular camera mounted on the drone 100, and obtaining obstacle information based on the scene image;

[0057] The scene image is an image of the same scene at the same time, and includes a left-eye image and a right-eye image. The obstacle information includes the distance between the obstacle and the drone, the relative position between the obstacle and the drone, etc.;

[0058] And when the distance between the obstacle and the drone is greater than the distance threshold, there is no need to reduce the flight speed of the drone (i.e. Figure 1 If the distance between the obstacle and the UAV is less than or equal to the distance threshold, execute step S2;

[0059] On this basis, obtaining obstacle information according to the scene image includes the following steps:

[0060] S11, performing stereo matching on the left eye image and the right eye image to obtain a disparity map between the left eye image and the right eye image;

[0061] S12, obtaining a depth map according to the disparity map (such as Figure 3As shown in part (a) of FIG. 1 ), and determining depth map information from the depth map; wherein the depth map information includes the three-dimensional coordinates of the obstacle, etc.;

[0062] S13, obtaining obstacle information according to the depth map information;

[0063] S2. Binarizing the depth map according to the depth map information and the flight parameters of the UAV to obtain a binary depth map;

[0064] Specifically, binarization of the depth map includes the following steps:

[0065] S21, obtain the current flight speed v0 of the drone and the braking distance d at the flight speed v0 in real time, and set the binary threshold threshold according to formula (1):

[0066] threshold=(d+v0t)*δ (1)

[0067] Wherein, t is the information transmission time of the drone communication link from the current flight state to the braking state of the drone; δ is the safety factor, and in this embodiment, δ≥1.0; since each pixel in the depth map not only carries position information, but also can be used to represent the distance between the object or surface corresponding to the pixel and the imaging device or sensor, therefore, in this embodiment, when the binary threshold of the depth map is determined by the drone aircraft speed v0 and the braking distance d, the flight area for the drone to pass safely can be better segmented;

[0068] S21, traverse each pixel point of the depth map, if the pixel value of the current pixel point ≥ threshold (since the pixel value of the pixel point in the depth map contains distance information, the pixel value can be compared with the binarization threshold), then the pixel value is set to 0, if the pixel value of the current pixel point < threshold, then the pixel value is set to 1, thereby completing the binarization processing of the depth map and obtaining a binarized depth map, for example Figure 3 Part (b) shows the depth map after binarization;

[0069] S3, according to the binary depth map and the size information of the UAV, determine whether there is a passable area for the UAV in the binary depth map, if yes, execute step S4, if no, control the UAV to hover (i.e. Figure 1 "Drone hovering" in );

[0070] Wherein, determining whether there is a passable area for the drone according to the binary depth map and the size information of the drone 100 includes the following steps:

[0071] like Figure 4As shown, a three-dimensional spherical enclosing space P0 that can completely enclose the drone is determined according to the size information of the drone, and a two-dimensional circular section P01 passing through the center of the three-dimensional spherical enclosing space is taken. The size of the drone includes the length, width, height of the drone, and the length of the arms when deployed, etc.;

[0072] Determine a plurality of circular areas P1 in the binary depth map, and each circular area P1 is composed of continuous pixel points with a pixel value of 0;

[0073] Obtaining an actual physical area corresponding to each circular area P1; the actual physical area is also a circular area;

[0074] The area s1 of each actual physical region is compared with the area s0 of the two-dimensional circular cross section P01. If the area s1 of an actual physical region is greater than s0, the circular region P1 corresponding to the actual physical region is determined as the passable region PassRegion of the drone (i.e. Figure 5 The area within the red line in the figure), if the area s1 of all actual physical areas is ≤ s0, it is considered that there is no passable area;

[0075] Therefore, in the above binary depth map, the actual physical area corresponding to the passable area PassReigon can be considered as a safe area without obstacles, and the drone can safely pass through the area as a whole without colliding with obstacles;

[0076] And S4, according to the distance between the passable area PassReigon and the obstacle, set the passing speed v of the drone when passing through the actual physical area corresponding to the passable area PassReigon, and control the drone through the flight control unit so that the drone flies at the passing speed v until it passes through the actual physical area corresponding to the passable area PassReigon to avoid the obstacle;

[0077] Specifically, Figure 6 As shown, step S4 includes the following steps:

[0078] S41, obtaining the pixel coordinates (PassReigon.x, PassReigon.y) of the center point O1 of the passable area;

[0079] S42. Obtain the yaw angle of the UAV relative to the traversable area according to formulas (2)-(3):

[0080] angleX=(|image.cols / 2-PassReigon.x| / image.cols)*HFOV (2)

[0081] angleY=(|image.rows / 2-PassReigon.y| / image.rows)*VFOV (3)

[0082] Among them, angleX represents the horizontal yaw angle of the drone relative to the traversable area in the horizontal direction; angleY ​​represents the vertical yaw angle of the drone relative to the traversable area in the vertical direction; image.cols and image.rows represent the width and height of the binary depth map respectively; HFOV and VFOV represent the horizontal field of view and vertical field of view of the binocular camera carried on the drone respectively;

[0083] According to the current distance between the obstacle and the drone, a passing speed v is set when the drone passes through the actual physical area corresponding to the passable area PassReigon, and the drone is controlled to fly at the passing speed v until it passes through the actual physical area corresponding to the passable area PassReigon;

[0084] Specifically, in this embodiment, controlling the drone to fly at a passing speed v includes:

[0085] The travel speed v is decomposed into the X-axis speed vx, the Y-axis speed vy and the Z-axis speed vz, where the X-axis, Y-axis and Z-axis are the three axes of the body coordinate system O-XYZ respectively:

[0086] vx=v*cos(angleY)*cos(angleX)

[0087] vy=v*cos(angleY)*sin(angleX)

[0088] vz=v*sin(angleY)

[0089] The power output of the drone is controlled so that it flies at an X-axis speed vx, a Y-axis speed vy, and a Z-axis speed vz until it passes through the actual physical area corresponding to the passable area PassReigon.

[0090] Therefore, in this embodiment, the obstacle depth map can be obtained by utilizing the drone's binocular camera, and based on the position information, distance information contained in the depth map pixels and the size information of the drone, it can be accurately determined whether there is a passable area for the drone. Finally, the drone can be controlled to pass through the safe area according to the set passing speed to complete the obstacle avoidance flight. The implementation process is simple, and the depth map information (i.e., the position information and distance information contained in the pixel points) can be fully utilized. There is no need to carry complex additional components, which can greatly reduce the requirements for the drone's hardware performance.

[0091] Embodiment 2:

[0092] The difference between this embodiment and embodiment 1 is that Figure 7 As shown, obtaining the actual physical area corresponding to each circular area P1 includes the following steps:

[0093] According to R1=(D×R) / f, each circular area P1 is converted into its corresponding actual physical area, where R1 represents the radius of the actual physical area; f represents the pixel focal length of the binocular camera; R represents the pixel radius of the circular area P1; and D represents the shortest distance between the drone and the obstacle solved according to the depth map.

[0094] Embodiment 3:

[0095] This embodiment provides a drone obstacle avoidance system, which can implement the drone obstacle avoidance method described in Embodiment 1 or 2, such as Figure 8 As shown, the unmanned aerial vehicle obstacle avoidance system includes:

[0096] A binocular camera 1 is mounted on the drone and is used to obtain an image of the scene in front of the drone;

[0097] A depth map generating unit 2, which is used to generate a disparity map according to the scene image, and to generate a depth map according to the disparity map;

[0098] A binarization unit 3, connected to the depth map generation unit 2, for binarizing the depth map to obtain a binarized depth map, the process of which is the same as step S2;

[0099] A passable area determination unit 4, which is connected to the binarization unit 3, is used to determine whether there is a passable area for the drone in the binary depth map according to the binary depth map and the size information of the drone, and the process is the same as step S3;

[0100] And a flight control unit 5, which is used to control the UAV to fly at a passable speed v until it passes through the actual physical area corresponding to the passable area when there is a passable area, and / or, when there is no passable area, control the UAV to hover, and / or, when the distance between the obstacle and the UAV is greater than a distance threshold, control the UAV to maintain the current flight speed.

[0101] In summary, the present invention firstly obtains the obstacle depth map by using the UAV binocular camera, and accurately determines whether there is a passable area for the UAV based on the position information, distance information and size information of the UAV contained in the depth map pixels, and finally controls the UAV to pass through the safe area according to the set passing speed to complete the obstacle avoidance flight. The implementation process is simple, and the depth map information can be fully utilized. There is no need to carry complex additional components, which can greatly reduce the requirements for the hardware performance of the UAV.

[0102] It should be noted that the technical features in the above-mentioned embodiments 1-3 can be combined arbitrarily, and the combined technical solutions all belong to the protection scope of the present application. In this article, terms such as "including", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the presence of other identical elements in the process, method, article or equipment including the elements.

[0103] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A UAV obstacle avoidance method based on binocular vision, characterized in that: The steps include: Obtaining a scene image in front of the drone through a binocular camera mounted on the drone, and obtaining obstacle information based on the scene image; Obtaining depth map information according to the scene image, and binarizing the depth map according to the depth map information and flight parameters of the drone to obtain a binary depth map; Determining whether there is a traversable area for the drone in the binary depth map according to the binary depth map and the size information of the drone; When there is a passable area, the drone is controlled to fly at a preset passable speed until it passes through the actual physical area corresponding to the passable area.

2. The unmanned aerial vehicle obstacle avoidance method according to claim 1, characterized in that: Binarization of the depth map comprises the following steps: The current flight speed v0 of the drone and the braking distance d at the flight speed v0 are obtained in real time, and the binary threshold threshold is set according to formula (1): threshold=(d+v0t)*δ (1) Among them, t is the information transmission time of the drone communication link from the current flight state to the braking state; δ is the safety factor; Traverse each pixel point of the depth map, if the pixel value of the current pixel point is ≥ threshold, set the pixel value to 0, if the pixel value of the current pixel point is < threshold, set the pixel value to 1, thereby completing the binarization processing of the depth map and obtaining a binary depth map.

3. The obstacle avoidance method for a drone as claimed in claim 1, characterized in that: Determining whether there is a traversable area for the drone based on the binary depth map and the size information of the drone includes the following steps: Determine a three-dimensional spherical enclosing space P0 that can completely enclose the drone according to the size information of the drone, and take a two-dimensional circular section P01 passing through the center of the three-dimensional spherical enclosing space; Determine a plurality of circular areas P1 in the binary depth map, and each circular area P1 is composed of continuous pixel points with a pixel value of 0; Obtaining the actual physical area corresponding to each circular area P1; The area s1 of each actual physical area is compared with the area s0 of the two-dimensional circular section P01. If the area s1 of a certain actual physical area is greater than s0, the circular area P1 corresponding to the actual physical area is determined as the passable area for the drone. If the areas s1 of all actual physical areas are ≦s0, it is considered that there is no passable area.

4. The obstacle avoidance method for a drone as claimed in claim 3, characterized in that: Obtaining the actual physical area corresponding to each circular area P1 includes the following steps: According to R1=(D×R) / f, each circular area P1 is converted into its corresponding actual physical area, where R1 represents the radius of the actual physical area; f represents the pixel focal length of the binocular camera; R represents the pixel radius of the circular area P1; and D represents the shortest distance between the drone and the obstacle solved according to the depth map.

5. The unmanned aerial vehicle obstacle avoidance method according to claim 1, characterized in that: If there is no passable area, the drone is controlled to hover.

6. The unmanned aerial vehicle obstacle avoidance method according to claim 1, characterized in that: Controlling the drone to fly at a preset speed until it passes through the actual physical area corresponding to the passable area includes the following steps: Obtain the pixel coordinates of the center point of the passable area (PassReigon.x, PassReigon.y); The yaw angle of the drone relative to the traversable area is obtained according to the following formula: angleX=(|image.cols / 2-PassReigon.x| / image.cols)*HFOV angleY=(|image.rows / 2-PassReigon.y| / image.rows)*VFOV Among them, angleX represents the horizontal yaw angle of the drone relative to the traversable area in the horizontal direction; angleY ​​represents the vertical yaw angle of the drone relative to the traversable area in the vertical direction; image.cols and image.rows represent the width and height of the binary depth map respectively; HFOV and VFOV represent the horizontal field of view and vertical field of view of the binocular camera carried on the drone respectively; The passing speed of the drone when passing through the actual physical area corresponding to the passable area is set according to the current distance between the obstacle and the drone, and the drone is controlled to fly at the passing speed until it passes through the actual physical area corresponding to the passable area.

7. The obstacle avoidance method for a drone as claimed in claim 6, characterized in that: Control the drone to fly at normal speed, including: The travel speed is decomposed into the X-axis speed vx, the Y-axis speed vy and the Z-axis speed vz, where the X-axis, Y-axis and Z-axis are the three axes of the body coordinate system O-XYZ: vx=v*cos(angleY)*cos(angleX) vy=v*cos(angleY)*sin(angleX) vz=v*sin(angleY) The power output of the drone is controlled so that it flies at an X-axis speed vx, a Y-axis speed vy, and a Z-axis speed vz until it passes through the actual physical area corresponding to the passable area.

8. The unmanned aerial vehicle obstacle avoidance method according to claim 1, characterized in that: When the obstacle information includes the distance between the obstacle and the drone, and the distance is greater than the distance threshold, there is no need to reduce the flight speed of the drone.

9. The unmanned aerial vehicle obstacle avoidance method according to claim 1, characterized in that: The depth map information includes three-dimensional coordinates of obstacles.

10. An unmanned aerial vehicle obstacle avoidance system, characterized in that: include: A binocular camera, which is mounted on the drone and used to obtain images of the scene in front of the drone; A depth map generating unit, configured to generate a disparity map according to the scene image, and to generate a depth map according to the disparity map; A binarization unit, connected to the depth map generation unit, configured to perform binarization processing on the depth map to obtain a binarized depth map; A passable area determination unit, connected to the binarization unit, for determining whether there is a passable area for the drone in the binary depth map according to the binary depth map and the size information of the drone; and a flight control unit, which is used to control the UAV to fly at a passable speed when there is a passable area until it passes through the actual physical area corresponding to the passable area, and / or, when there is no passable area, control the UAV to hover, and / or, when the distance between the obstacle and the UAV is greater than a distance threshold, control the UAV to maintain the current flight speed.

Images