Method and system of unmanned plane omnibearing obstacle avoidance based on binocular vision

A technology for binocular vision and unmanned aerial vehicles, applied in the field of unmanned aerial vehicles, can solve problems such as the inability to guarantee real-time performance and the inability to adapt to binocular vision to avoid obstacles and the large amount of computation, and achieve the effect of low power consumption

Inactive Publication Date: 2016-07-20
SHENZHEN AUTEL INTELLIGENT AVIATION TECH CO LTD
4 Cites 56 Cited by

AI-Extracted Technical Summary

Problems solved by technology

However, most of the technical solutions for avoiding obstacles in binocular vision in the prior art are based on general-purpose platforms such as CPU, DSP, and GPU. Because these general-purpose platforms have their own limitations, they cannot adapt to the large scale required by th...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Method used

In the implementation of the present invention, the binocular camera device is respectively arranged around the unmanned aerial vehicle, and the binocular camera device shoots in real time, and extracts respectively the left image and the right image currently collected by each binocular camera device, According to the extracted left and right images, identify whether there is an obstacle in the azimuth corresponding to each binocular camera device. If there is an obstacle in the azimuth corresponding to one or more binocular camera devices, control the UAV to detect The direction movement of the binocular camera device to the obstacle realizes the UAV's automatic obstacle avoidance. Since the binocular camera device is arranged around the UAV, it realizes the omnidirectional obstacle avoidance of the UAV based on binocular vision.
Processor 302 is based on being realized by Field Programmable Gate Array FPGA, can make full use of FPGA can customize IO characteristic, inserts the input of multi-channel binocular camera device 304 simultaneously, and makes full use of FPGA parallel structure simultaneously, realizes Multiplex video capture. In addition, the processor 302 dynamically switches the binocular camera device 304 according to the direction of motion of the drone, so that multiple binocular camera devices 304 share the obstacle recognition and obstacle avoidance cont...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Abstract

The present invention discloses a method and system of unmanned plane omnibearing obstacle avoidance based on binocular vision. Binocular camera devices are uniformly distributed around an unmanned plane and are configured to shoot in real time. The method comprises: respectively extracting left images and right images collected by each binocular camera device at present; identifying whether there are barriers at the orientation corresponding to each binocular camera or not according to the extracted left images and right images; if there are barriers, determining the distance between the barriers and the unmanned plane according to a binocular vision distance measurement algorithm; determining whether the distance between the barriers and the unmanned plane is smaller than a preset safe distance or not; and if the distance between the barriers and the unmanned plane is smaller than the preset safe distance, and regulating the current movement direction of the unmanned plane to allow the unmanned plane to move far away from the direction of the barriers. The method and system of unmanned plane omnibearing obstacle avoidance based on binocular vision realize the omnibearing automatic obstacle of an unmanned plane based on binocular vision, and are higher in timeliness and lower in power consumption.

Application Domain

Technology Topic

Image

  • Method and system of unmanned plane omnibearing obstacle avoidance based on binocular vision
  • Method and system of unmanned plane omnibearing obstacle avoidance based on binocular vision
  • Method and system of unmanned plane omnibearing obstacle avoidance based on binocular vision

Examples

  • Experimental program(1)

Example Embodiment

[0020] The present invention will be described in detail below in conjunction with the drawings and embodiments.
[0021] In order to realize the omni-directional obstacle avoidance of the drone based on binocular vision, it is necessary to set up binocular camera devices around the drone, and define the corresponding binocular camera device according to the position of the binocular camera device on the drone. In this embodiment, preferably, binocular camera devices are arranged around the drone, which means that at least one binocular camera is installed on the front, rear, left, right, lower and upper parts of the drone. Camera devices, and all binocular camera devices take real-time shooting, and the binocular camera devices on each side collect left and right images of the space environment on the side of the drone. In this embodiment, preferably, the binocular camera device can be directly combined with two cameras, and the binocular camera device can be an infrared camera. The method of UAV's omnidirectional obstacle avoidance based on binocular vision is as follows, please refer to figure 1 , Methods include:
[0022] Step S201: extracting the left image and the right image currently collected by each binocular camera device;
[0023] Each binocular camera device around the UAV captures the left and right images of the space environment around the UAV in real time.
[0024] Step S202: According to the extracted left image and right image, identify whether there is an obstacle in the orientation corresponding to each binocular camera device, if there is an obstacle, go to step S203, otherwise, keep the current direction of the drone and continue driving;
[0025] Recognizing whether there is an obstacle from the left image and the right image can be recognized by a preset recognition algorithm, which is not specifically limited here.
[0026] Step S203: Determine the distance between the obstacle and the UAV according to the binocular vision ranging algorithm;
[0027] The binocular vision ranging algorithm is based on the parallax principle and uses imaging equipment to obtain two images of the measured object from different positions, and then calculates the position deviation between the corresponding points of the image to obtain the three-dimensional geometric information of the object. Like figure 2 As shown, step S203 includes:
[0028] Step S2031: perform stereo matching on the left image and the right image to obtain a disparity map between the left image and the right image;
[0029] Step S2032: Determine the depth image according to the disparity map;
[0030] Step S2033: Extract depth information in the depth image, and determine the three-dimensional coordinates of the obstacle according to the depth information;
[0031] Step S2034: Determine the distance between the obstacle and the UAV according to the three-dimensional coordinates of the obstacle.
[0032] Step S204: Determine whether the distance between the drone and the obstacle is less than the preset safety distance, if it is less than the preset safety distance, go to step S205, otherwise, keep the UAV in the current direction of movement and continue driving;
[0033] Step S205: Adjust the current movement direction of the drone so that the drone moves in a direction away from the obstacle;
[0034] The step of adjusting the current movement direction of the unmanned aerial vehicle specifically includes: acquiring the orientation corresponding to the binocular camera device where no obstacle is detected, and making the movement direction of the unmanned aerial vehicle face the dual-eye camera where no obstacle is detected. The azimuth deflection corresponding to the eye camera device. By adjusting the current movement direction of the UAV, the UAV can avoid obstacles and avoid the UAV colliding with obstacles to damage the UAV.
[0035] Of course, it can also be set after step S205 to immediately return to step S201 for the next obstacle avoidance process, or after the drone is set to continue to travel for a certain distance according to predetermined conditions, return to step 201 for the next obstacle avoidance process. It can be a time condition, then the UAV will continue to drive along the current direction of motion according to the predetermined conditions as follows: the UAV continues to drive along the current direction of motion for a predetermined time; the predetermined condition can be a distance condition, then the UAV will follow the predetermined conditions along the current direction of motion Continue driving: The drone continues to travel a predetermined distance in the current direction of movement.
[0036] In the embodiment of the present invention, binocular camera devices are arranged around the drone, and the binocular camera devices take real-time shooting, and the left and right images currently collected by each binocular camera device are extracted respectively, according to the extracted Left and right images, identify whether there are obstacles in the azimuth corresponding to each binocular camera device. If there are obstacles in the azimuth corresponding to one or more binocular camera devices, control the drone to face no obstacles detected The direction movement of the binocular camera device realizes the automatic obstacle avoidance of the drone. Since the binocular camera device is arranged around the drone, the drone can avoid obstacles in all directions based on binocular vision.
[0037] The present invention also provides an implementation of an omnidirectional obstacle avoidance system for drones based on binocular vision. See image 3 The omni-directional obstacle avoidance system 30 for drones based on binocular vision includes a processor 302, a direction controller 303, and multiple binocular camera devices 304. The processor 302, the binocular camera device 304 and the direction controller 303 are all fixed on On the drone, the direction controller 303 is connected to the drone to control the movement direction of the drone, and the direction controller 303 and the binocular camera device 304 are both connected to the processor 302. Multiple binocular camera devices 304 are distributed around the drone, and the binocular camera device 304 takes real-time shooting. In this embodiment, preferably, multiple binocular camera devices are distributed on the front and back ends of the drone. , Left, right, bottom and top, each binocular camera device corresponds to an orientation. .
[0038] The processor 302 is configured to: extract the left image and the right image currently collected by each binocular camera device 304; according to the extracted left image and right image, identify whether there is an obstacle in the orientation corresponding to each binocular camera device 304 If there is the obstacle, determine the distance between the obstacle and the UAV according to the binocular vision ranging algorithm; determine whether the distance between the UAV and the obstacle is less than or equal to the preset safety Distance: If it is less than the preset safe distance, the direction controller 303 is used to adjust the current movement direction of the drone to move the drone in a direction away from obstacles to realize the automatic obstacle avoidance function of the drone.
[0039] The step of the processor 302 adjusting the current direction of movement of the drone through the direction controller 303 includes: the processor 302 obtains the position corresponding to the binocular camera device 304 that has not detected obstacles, and controls the direction of the drone through the direction controller 303. The azimuth deflection corresponding to the binocular camera device that does not detect the obstacle. After the movement direction of the drone is deflected, the drone smoothly avoids obstacles.
[0040] Specifically, the processor 302 determines the distance between the obstacle and the drone according to the binocular vision ranging algorithm, including: stereo matching the left image and the right image to obtain a disparity map between the left image and the right image Determine the depth image according to the disparity map; extract the depth information in the depth image, and determine the three-dimensional coordinates of the obstacle according to the depth information; determine the difference between the obstacle and the UAV according to the three-dimensional coordinates of the obstacle The distance between.
[0041] Further, the processor 302 is implemented based on a field programmable gate array FPGA, and multiple dual camera devices are connected to the processor 302 in parallel, so that the processor 302 can simultaneously receive the data collected by the multiple binocular camera devices 304 The processor 302 selects the corresponding binocular camera device 304 to collect the spatial environment image according to the movement direction of the drone, and performs obstacle recognition and obstacle avoidance according to the spatial environment image. Of course, in the drone After adjusting the movement direction to avoid obstacles, the corresponding binocular camera device 304 is selected again according to the new movement direction of the drone to collect the spatial ring mirror image. Because the processor 302 selects the binocular camera device 304 every time it is in line with the movement of the drone. The directions correspond to each other. Therefore, each time the binocular camera device 304 is selected is to collect the spatial environment image in the moving direction of the drone and in front of the drone to ensure that obstacles are avoided in the moving direction of the drone.
[0042] The processor 302 is implemented based on a field programmable gate array FPGA, which can make full use of the FPGA's customizable IO features, simultaneously access the input of the multi-channel binocular camera device 304, and make full use of the FPGA parallel structure to realize multi-channel video capture. In addition, the processor 302 dynamically switches the binocular camera device 304 according to the direction of movement of the drone, so that multiple binocular camera devices 304 can share the obstacle recognition and obstacle avoidance control circuit in the processor 302. The binocular camera device 304 is equipped with corresponding obstacle recognition and obstacle avoidance control circuits. Compared with the traditional single-direction binocular vision solution, the binocular camera device 304 can realize all-round obstacle avoidance with almost the same resources and power consumption. Furthermore, through FPGA built-in pipeline and multi-channel parallel processing mechanism, the processing speed and data throughput are unmatched by traditional CPUs.
[0043] In order to improve the processing capability of the processor 302, the processor 302 also adopts a multi-level cache structure to implement data pipeline processing, and the processor 302 also uses a high-bit-width system bus for parallel operation.
[0044] The omnidirectional obstacle avoidance system of the drone based on binocular vision also includes an alarm device 305, which is fixed to the drone, and the alarm device is connected to the processor 302. When the processor 302 determines that the distance between the drone and the obstacle is less than the preset safe distance, it controls the alarm device to issue an alarm, so as to facilitate the user to obtain the obstacle in the current movement direction of the drone. Of course, when the distance between the drone and the obstacle is greater than or equal to the preset safe distance, the processor 302 controls the alarm device not to issue an alarm.
[0045] In the embodiment of the present invention, binocular camera devices 304 are respectively arranged around the drone, and the processor 302 extracts the left and right images currently collected by each binocular camera device 304 according to the extracted left and right images. The right image identifies whether there are obstacles in the azimuth corresponding to each binocular camera device 304. If there is, the drone is controlled to deflect to the azimuth corresponding to the binocular camera device 304 that does not detect the obstacle to avoid the obstacle and realize unmanned Because the binocular camera device 304 is arranged around the drone, it realizes the function of the drone to avoid obstacles in all directions based on binocular vision; in addition, the processor 302 is based on The programmable gate array FPGA is implemented, and multiple dual camera devices are connected to the processor 302 in parallel. The processor 302 obtains the multiple image resources collected by the multiple binocular camera devices 304 in parallel, and the processor 302 according to the movement of the drone The direction, the binocular camera device 304 is dynamically switched, so that multiple binocular camera devices 304 share the obstacle recognition and obstacle avoidance control circuit in the processor 302, and there is no need to equip each binocular camera device 304 with corresponding obstacle recognition Compared with the obstacle avoidance control circuit, compared with the traditional single-direction binocular vision solution, it can realize all-round obstacle avoidance and lower power consumption with almost the same resources and power consumption.
[0046] The above are only the embodiments of the present invention and do not limit the scope of the present invention. Any equivalent structure or equivalent process transformation made by using the content of the description and drawings of the present invention, or directly or indirectly applied to other related technologies In the same way, all fields are included in the scope of patent protection of the present invention.
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

no PUM

Description & Claims & Application Information

We can also present the details of the Description, Claims and Application information to help users get a comprehensive understanding of the technical details of the patent, such as background art, summary of invention, brief description of drawings, description of embodiments, and other original content. On the other hand, users can also determine the specific scope of protection of the technology through the list of claims; as well as understand the changes in the life cycle of the technology with the presentation of the patent timeline. Login to view more.
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Similar technology patents

Classification and recommendation of technical efficacy words

Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products