A method and system for obstacle avoidance and control of firefighting drones based on binocular vision and Realsense

By combining binocular vision and Realsense depth camera to obtain depth information in front, the attitude and trajectory of the drone are controlled, which solves the problems of poor obstacle avoidance and low real-time performance of drones, and realizes rapid obstacle avoidance and safe flight.

CN116149346BActive Publication Date: 2026-11-10STATE GRID ANHUI ELECTRIC POWER CO LTD ELECTRIC POWER SCI RES INST +2
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202211500848.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-11-28
Publication Date
2026-11-10
Estimated Expiration
2042-11-28

AI Technical Summary

Technical Problem

Existing drone obstacle avoidance algorithms have poor obstacle avoidance performance, low real-time performance, and low safety in fire prevention and control, especially in dense passages with multiple UHV lines and complex environments where drone safety control is difficult to meet requirements.

Method used

The system combines binocular vision with a Realsense depth camera to acquire depth information in front of the drone. It controls the drone's attitude and trajectory by using roll angle, throttle, and pitch angle coefficients, and uses the Hadamard product to process the depth map matrix to achieve obstacle avoidance.

Benefits of technology

It improves the obstacle avoidance response speed and effectiveness of drones, reduces the risk of collisions, and enables successful operation in complex environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116149346B_ABST
    Figure CN116149346B_ABST
Patent Text Reader

Abstract

This invention provides a method and system for obstacle avoidance and control of a firefighting drone based on binocular vision and Realsense, including: calibrating the binocular camera; setting multiple waypoints; acquiring the drone's current latitude and longitude information and setting the next waypoint as the target location; using the binocular camera to acquire distance information of obstacles in front of the drone and obtain a depth map; filtering the depth map to filter out the influence of distant obstacles, setting the perception range to filter out obstacles outside the perception range; multiplying the filtered depth map matrix by a roll coefficient matrix, a throttle coefficient matrix, a fuselage coefficient matrix, and an identity matrix to obtain a Hadamard product; deriving roll angle coefficients, throttle coefficients, and pitch angle coefficients, which are used as inputs to the drone control system to control the drone's attitude and trajectory, thereby achieving obstacle avoidance; after obstacle avoidance, the drone resumes flight towards the current target point. This invention solves the technical problems of poor obstacle avoidance performance, low real-time performance, and low safety of drones.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) control, specifically to a method and system for obstacle avoidance and control of a firefighting UAV based on binocular vision and Realsense. Background Technology

[0002] Fire is one of the most common disasters in human production and life. With the vigorous progress of ultra-high voltage direct current (UHVDC) transmission projects, UHVDC transmission lines pass through large areas of mountainous, hilly, and agricultural production areas. The vegetation in the line corridors is dense, especially in dense corridors where there are large areas of trees and other flammable vegetation. There are many scattered mobile fire sources such as forest fires and domestic fires, resulting in frequent wildfires and great pressure on fire prevention and control. Once a wildfire occurs, the damage is extremely great, and there is a safety risk that multiple lines in the corridor will be shut down at the same time.

[0003] Unmanned aerial vehicles (UAVs) possess intelligent and information-based advantages such as speed, comprehensiveness, and precision. While ensuring personal safety, they overcome terrain limitations to control multiple fire points, providing a valuable platform for managing wildfires in densely populated areas. Domestic and international scholars have conducted extensive research in this field, improving the practicality of UAVs in fire prevention and control. Traditional UAV obstacle avoidance algorithms include Voronoi diagrams, A* algorithm, Dijkstra's algorithm, D* algorithm, and artificial potential field methods, among others. Among these, the artificial potential field method is widely used in UAV obstacle avoidance planning due to its simple structure, ease of modeling and application, and excellent real-time performance. However, it still faces the following challenges: The existing invention patent document CN105892489A, entitled "An Autonomous Obstacle Avoidance UAV System and Control Method Based on Multi-Sensor Fusion," includes: a real-time environmental information detection module, which uses multi-sensor fusion technology to detect the surrounding environment in real time and transmits the detected information to an obstacle data analysis and processing module; the obstacle data analysis and processing module performs environmental structure perception to construct and determine the obstacles; and an obstacle avoidance decision module determines the obstacle avoidance decision based on the output of the obstacle data analysis and processing module, and drives the power module through the flight control system to achieve UAV avoidance of surrounding obstacles. From the specific implementation of this existing solution, it can be seen that this technology constructs a spatially connected region, obtains the distance between the fuselage and obstacles based on the established obstacle model and the ranging unit, and uses the artificial potential field method for local path planning to avoid collisions. Although the artificial potential field method can solve the real-time path planning during UAV flight, it may get trapped in local optima, causing the UAV to get stuck, thus limiting the obstacle avoidance effect of the UAV.

[0004] The existing invention patent application CN111784079A, entitled "A UAV Path Planning Method Based on Artificial Potential Field and Ant Colony Algorithm," includes the following process: S1: Define the UAV's operating space as a potential energy field and obtain the initial path using the artificial potential field method; S2: When a large obstacle is detected, rasterize the map and determine the starting and ending nodes for obstacle avoidance; S3: Plan the obstacle avoidance route using the ant colony algorithm; S4: After the UAV completes obstacle avoidance, continue to use the artificial potential field method for path planning, repeating this process until the target point is reached. Although the improved artificial potential field method based on the ant colony algorithm can avoid getting trapped in local optima, it has a long search time, consumes a lot of computational resources, and its real-time performance is difficult to meet requirements.

[0005] Meanwhile, the dense network of high-voltage power lines within the passageway increases the payload of firefighting drones, placing higher demands on their safety control strategies. In severe fire scenarios, firefighting drones must reach the fire scene as quickly as possible, requiring the straightest possible flight path and the highest possible speed. In complex fire environments, traditional RGB binocular cameras cannot provide effective depth information during smoke-covered or nighttime operations, significantly increasing the risk of collisions.

[0006] In summary, existing technologies suffer from poor obstacle avoidance performance, low real-time performance, and low safety for drones. Summary of the Invention

[0007] The technical problem to be solved by this invention is how to solve the technical problems of poor obstacle avoidance effect, low real-time performance and low safety of drones in the prior art.

[0008] This invention solves the above-mentioned technical problems by adopting the following technical solution: A method and system for obstacle avoidance and control of a firefighting drone based on binocular vision and Realsense, comprising: S1. Calibrate the preset camera to obtain the intrinsic parameter matrix of the two cameras; S2. Collect and obtain latitude and longitude information and terrain information of the fire site, and set multiple path points accordingly; S3. Use the preset positioning module to obtain the current latitude and longitude information of the drone and set the next path point as the target location; S4. Using the preset camera, obtain the distance information from the obstacle in front of the drone to the camera plane, and obtain the depth map accordingly; S5. Filter the depth map to remove interference from distant obstacles, set the perception range, and filter out obstacles outside the perception range to obtain the depth map matrix. S6. Multiply the depth map matrix by the roll coefficient matrix, throttle coefficient matrix, fuselage coefficient matrix and identity matrix respectively to obtain at least two Hadamard products, and process them to obtain the roll angle coefficient, throttle coefficient and pitch angle coefficient. S7. Input the roll angle coefficient, throttle coefficient, and pitch angle coefficient into the preset UAV control system to control the attitude and trajectory of the UAV, so as to control the UAV to avoid obstacles. S8. After passing through an obstacle, restore the initial flight direction of the drone based on the target location to control the drone to fly towards the current target point.

[0009] This invention utilizes binocular vision as a foundation, incorporating a Realsense depth camera to acquire forward depth information. The processed data is multiplied by the direction coefficient matrix to obtain the drone's pitch angle, roll angle, and throttle control parameters. This provides guidance for subsequent obstacle avoidance and control methods and systems for firefighting drones based on binocular vision and Realsense. This invention features fast response, excellent obstacle avoidance, and rapid arrival at the destination.

[0010] In a more specific technical solution, step S4 includes: S41. Use a preset camera to collect image feature points and obtain the distance information from obstacles in front of the UAV to the camera plane. The preset camera includes a binocular camera and a Realsense depth camera. S42. Using the following logic, calculate the acquisition deviation of the preset camera based on image feature points:

[0011] in, Let be the distance from the target point to the camera plane. The distance between the two cameras. For the camera's focal length, The parallax of the left and right cameras; S43. Based on the principle of similar triangles and the acquisition deviation, the coordinates are transformed from the pixel coordinate system to the real-world coordinate system. The coordinate transformation formula is as follows:

[0012] in, These are the x and y coordinates of the point in the camera coordinate system, respectively. Here, x and y are the x and y coordinates of the center point of the entire image, respectively, and f is the camera focal length. These are coordinates with the RealSense depth camera as the origin and the camera plane as the coordinate system. The coordinates of that point in the actual coordinate system of the surface; S44. Record all scanned image feature points in a preset matrix to obtain a depth map. .

[0013] This invention uses binocular vision as a foundation and incorporates a Realsense depth camera, combining the two to obtain depth information in front, thereby reducing the risk of collisions with drones.

[0014] In a more specific technical solution, step S5 includes: S51. Using the following logic, the depth map... Calculate the average of all elements :

[0015] In the formula, , These represent the number of rows and columns of the depth map, respectively. For depth map The Okay, number Column elements; S52. Set the sensing range Based on this, a filtering depth map is obtained; S53. Use preset logic to process and filter the depth map to obtain a depth map matrix.

[0016] In a more specific technical solution, step S52 involves setting the sensing range using the following logic. :

[0017] In the formula, Current location To the target location Euclidean distance, The area of ​​the target location is considered as the drone having reached the target location.

[0018] In a more specific technical solution, in step S53, the depth map is filtered using the following logic to obtain the processed depth map matrix. No. line, number Column element values Based on this, a depth map matrix is ​​constructed: .

[0019] In a more specific technical solution, step S6 includes: S61. Define the roll coefficient matrix, throttle coefficient matrix and fuselage coefficient matrix using preset logic; S62 utilizes the following logic, based on the depth map matrix Calculate the roll coefficient matrix, throttle coefficient matrix, and fuselage coefficient matrix, and obtain the depth map matrix respectively. Hadamaji:

[0020]

[0021] ; S63, respectively convert the roll coefficient matrix Throttle coefficient matrix and fuselage coefficient array With depth map matrix The sum of each element in the Hadamard product is divided by the width and height of the depth map to obtain the lateral avoidance coefficient, longitudinal avoidance coefficient, and fuselage avoidance coefficient. S64. Use preset logic to process the lateral avoidance coefficient, longitudinal avoidance coefficient and fuselage avoidance coefficient to obtain the roll angle coefficient, throttle coefficient and pitch angle coefficient.

[0022] This invention uses roll angle coefficient, throttle coefficient, and pitch angle coefficient as inputs into the UAV control system to control the attitude and trajectory of the UAV, thereby achieving obstacle avoidance.

[0023] This invention multiplies the processed depth map with the orientation coefficient matrix to obtain the pitch angle, roll angle, and throttle control parameters of the UAV, directly controlling the system. Compared to traditional artificial potential fields, this invention offers faster response, better obstacle avoidance in mountainous terrain, avoids getting stuck in local optima, and can operate successfully in complex environments such as fire scenes or at night.

[0024] In a more specific technical solution, in step S61, the rolling coefficient matrix is ​​defined using the following logic. Throttle coefficient matrix and fuselage coefficient array :

[0025]

[0026] .

[0027] In a more specific technical solution, in step S63, the following logic is used to process each element in the Hadamard product and the depth map to obtain the lateral avoidance coefficient. Vertical avoidance coefficient and fuselage avoidance coefficient :

[0028]

[0029] .

[0030] In a more specific technical solution, in step S64, the roll angle coefficient is obtained using the following logical processing. Throttle Coefficient Pitch angle coefficient :

[0031]

[0032]

[0033] In the formula, These are the avoidance trigger coefficients for the horizontal, vertical, and fuselage directions, respectively.

[0034] In a more specific technical solution, step S7 includes: S71: The roll angle coefficients are respectively... Throttle Coefficient Pitch angle coefficient Convert to Euler angles Throttle control amount ; S72: Internal control of the attitude controller and The desired angle change is obtained by subtracting the values. Attitude control consists of two parts: angular velocity and angular acceleration. The desired angular velocity, the outer loop of the attitude controller, is obtained by multiplying the desired angle change by the proportional coefficient. The desired angular velocity is obtained by subtracting the desired angular velocity from the actual angular velocity measured by the gyroscope, forming the inner loop of the attitude controller. The desired torque control quantity can be obtained from the angular velocity change through inverse equation solving. Output.

[0035] S73: By According to the lift coefficient and torque coefficient The rotational speeds of the four motors of the quadcopter can be calculated, and the corresponding output current can be obtained based on the motor parameters.

[0036] S74: The calculated current is transmitted to the ESC via the controller, and the ESC controls the motor to rotate at the specified speed.

[0037] In a more specific technical solution, a firefighting drone obstacle avoidance system based on binocular vision and RealSense includes: The camera calibration module is used to calibrate the preset camera and obtain the intrinsic parameter matrix of the two cameras; The path point acquisition module is used to collect latitude and longitude information and terrain information of the fire site, and to set multiple path points accordingly; The target location acquisition module is used to obtain the current latitude and longitude information of the UAV using the preset positioning module, and set the next path point as the target location; The depth processing module is used to obtain the distance information from the obstacles in front of the drone to the camera plane using the preset camera, and to obtain a depth map. The depth processing module is connected to the camera calibration module. The depth map filtering module is used to filter the depth map, filter out interference from distant obstacles, set the perception range, and filter out obstacles outside the perception range to obtain the depth map matrix. The depth map filtering module is connected to the depth processing module. The UAV control coefficient processing module is used to multiply the depth map matrix with the roll coefficient matrix, throttle coefficient matrix, fuselage coefficient matrix and identity matrix respectively to obtain at least two Hadamard products, which are then used to obtain the roll angle coefficient, throttle coefficient and pitch angle coefficient. The UAV control coefficient processing module is connected to the depth map filtering module. The obstacle avoidance control module is used to input the roll angle coefficient, throttle coefficient, and pitch angle coefficient into the preset UAV control system, thereby controlling the attitude and trajectory of the UAV to avoid obstacles. The obstacle avoidance control module is connected to the UAV control coefficient processing module. The obstacle avoidance post-control module is used to restore the initial flight direction of the UAV based on the target position after passing the obstacle, so as to control the UAV to fly towards the current target point. The obstacle avoidance post-control module is connected to the obstacle avoidance control module and the target position acquisition module.

[0038] Compared with existing technologies, this invention has the following advantages: It utilizes binocular vision as a foundation, incorporating a Realsense depth camera, combining the two to acquire forward depth information. After processing, this information is multiplied by the direction coefficient matrix to obtain the drone's pitch angle, roll angle, and throttle control parameters. This provides guidance for subsequent obstacle avoidance and control methods and systems for firefighting drones based on binocular vision and Realsense. This invention also features fast response speed, good obstacle avoidance performance, and rapid arrival at the destination.

[0039] This invention uses binocular vision as a foundation and incorporates a Realsense depth camera, combining the two to obtain depth information in front, thereby reducing the risk of collisions with drones.

[0040] This invention uses roll angle coefficient, throttle coefficient, and pitch angle coefficient as inputs into the UAV control system to control the attitude and trajectory of the UAV, thereby achieving obstacle avoidance.

[0041] This invention multiplies the processed depth map with the orientation coefficient matrix to obtain the pitch angle, roll angle, and throttle control parameters of the UAV, directly controlling the system. Compared to traditional artificial potential fields, this invention offers faster response, better obstacle avoidance in mountainous terrain, avoids getting stuck in local optima, and can operate successfully in complex environments such as fire scenes or at night.

[0042] This invention solves the technical problems of poor obstacle avoidance performance, low real-time performance and low safety of existing drones. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the obstacle avoidance and control method and system steps of a fire-fighting drone based on binocular vision and Realsense, according to Embodiment 1 of the present invention. Figure 2 This is a schematic diagram of the overall data processing of the obstacle avoidance method for firefighting drones in Embodiment 1 of the present invention; Figure 3 This is an RGB effect diagram of Embodiment 1 of the present invention; Figure 4 This is a depth effect diagram of Embodiment 1 of the present invention; Figure 5 This is a point cloud effect diagram of Embodiment 1 of the present invention; Figure 6 This is a posture diagram of obstacle avoidance in the Airsim simulation environment according to Embodiment 1 of the present invention; Figure 7 This is a trajectory diagram of the journey to the fire scene according to Embodiment 1 of the present invention; Figure 8 This is a trajectory diagram of the return flight in the Airsim simulation environment according to Embodiment 1 of the present invention; Figure 9 This is a depth map obtained in the Airsim simulation environment according to Embodiment 1 of the present invention.

[0044] Figure 10 This is a schematic diagram of the attitude-angular velocity control process in Embodiment 1 of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] Example 1 like Figure 1 As shown, the present invention provides an obstacle avoidance and control method and system for firefighting drones based on binocular vision and Realsense, comprising the following steps: S1. Calibrate the stereo camera to obtain the intrinsic parameter matrices of the two cameras; S2. Based on the latitude and longitude information of the fire site and the terrain information, set multiple path points; S3: Use the GPS module to obtain the latitude and longitude information of the drone's current location and set the next waypoint as the target location; S4. Use a binocular camera to obtain distance information from obstacles in front of the drone to the camera plane, and obtain a depth map; S5. Filter the depth map to remove the influence of distant obstacles, set the perception range, and filter obstacles outside the perception range. S6. Multiply the filtered depth map matrix by the roll coefficient matrix, throttle coefficient matrix, fuselage coefficient matrix, and identity matrix respectively to obtain their Hadamard products. This yields the roll angle coefficient, throttle coefficient, and pitch angle coefficient. S7. The roll angle coefficient, throttle coefficient, and pitch angle coefficient are input into the UAV control system to control the attitude and trajectory of the UAV, thereby achieving obstacle avoidance. S8. After completing obstacle avoidance, resume flight towards the current target point.

[0047] In this embodiment, as Figure 2 As shown, the obstacle avoidance and control method and system for firefighting drones based on binocular vision and Realsense, as described in this invention, includes the following steps: S1', Head to waypoint; S2', Calibrate the binocular camera; S3', Obtain the depth map; S4', Process the depth map; S5', obtain the evasion coefficients in each direction; S6': Obtain roll, throttle, and pitch coefficients, and input them into the system for obstacle avoidance.

[0048] In this embodiment, in step S3, the latitude and longitude information of the current location of the UAV is obtained using the GPS module, and the next path point is set as the target location. In this embodiment, the UAV's GPS module continuously compares the latitude and longitude information of the UAV's current location with the latitude and longitude information of the target location, calculates the difference to obtain the direction vector of the forward direction, and flies in that direction at a set speed. During the flight, it repeatedly executes S4-S8 until it reaches the vicinity of the target point.

[0049] In this embodiment, step S4 uses a binocular camera or RealSense to obtain distance information from obstacles in front of the drone to the camera plane, and obtains a depth map; In this embodiment, as Figure 3As shown, the Realsense depth camera uses two infrared laser emitters and one infrared receiver sensor. Its ranging principle is similar to binocular ranging. The infrared laser emitters emit a known pattern of non-visible light, and a pair of parallel infrared receiver sensors receive these patterns as they fall on the object. Because the two receivers are positioned differently, the acquired patterns will inevitably deviate.

[0050] The following formula is used to calculate the deviation between feature points, thereby obtaining the distance information between the target point and the camera plane: (1) in, The distance from the target point to the camera plane; The distance between the two cameras; The focal length of the camera; This represents the parallax of the left and right cameras.

[0051] In this embodiment, as Figure 4 and Figure 5 As shown, the principle of similar triangles is used to transform the coordinates from the pixel coordinate system to the real-world coordinate system. The coordinate transformation formula is as follows: (2) in, These are the x and y coordinates of the point in the camera coordinate system, respectively. Here, x and y are the x and y coordinates of the center point of the entire image, respectively, and f is the camera focal length. These are coordinates with the RealSense depth camera as the origin and the camera plane as the coordinate system. The coordinates of that point in the actual coordinate system of the surface; In this embodiment, all scanned points are recorded in a matrix to obtain a depth map. : S5: Depth Map Filtering is performed to remove the influence of distant obstacles, and the perception range is set to filter out obstacles outside the perception range.

[0052] In this embodiment, the depth map Calculate the average of all elements in the equation. : (3) In equation (3), , These represent the number of rows and columns of the depth map, respectively. For depth map The Okay, number Column elements.

[0053] Set the sensing range : (4) In equation (4), Current location To the target location The Euclidean distance. The area of ​​the target location is considered as the drone having reached the target location.

[0054] like Figure 9 As shown, in this embodiment, the filtered depth map is processed as follows to obtain the processed depth map matrix. . The Middle line, number The element values ​​of the column are: (5) S6: Compare the filtered depth map matrix with the roll coefficient matrix respectively. Throttle coefficient matrix and fuselage coefficient array Multiplying them yields their Hadamard product, from which the evasion coefficients in each direction are obtained. From this, the roll coefficient, throttle coefficient, and pitch coefficient are derived.

[0055] In this embodiment, the roll coefficient matrix, throttle coefficient matrix, and fuselage coefficient matrix are defined as follows: (6) (7) (8) In equation (8), the number of rows and columns of the intermediate matrix are adjusted according to the actual shape of the UAV, and the rest are 0 matrices.

[0056] In this embodiment, their values ​​are calculated together with... Hadamaji: (9) (10) (11) In this embodiment, respectively The lateral avoidance coefficient is obtained by dividing the sum of all elements in the depth map by the width and height of the depth map. Vertical avoidance coefficient and fuselage avoidance coefficient : (12) (13) (14) In this embodiment, the roll angle coefficient is obtained. Throttle Coefficient Pitch angle coefficient : (15) (16) (17) In equations (15), (16), and (17), These are the avoidance trigger coefficients for the horizontal, vertical, and fuselage directions, respectively, and the specific values ​​need to be set according to the actual flight scenario.

[0057] like Figure 6 , Figure 7 and Figure 10 As shown, in this embodiment, in step S7, the roll angle coefficient, throttle coefficient, and pitch angle coefficient are input into the UAV control system to control the attitude and trajectory of the UAV, thereby achieving obstacle avoidance.

[0058] In this embodiment, the roll angle coefficient is respectively... Throttle Coefficient Pitch angle coefficient Convert to Euler angles Throttle control amount ; In this embodiment, the attitude controller internally... and The desired angle change is obtained by subtracting the values. Attitude control consists of two parts: angular velocity and angular acceleration. The desired angular velocity, the outer loop of the attitude controller, is obtained by multiplying the desired angle change by the proportional coefficient. The desired angular velocity is obtained by subtracting the desired angular velocity from the actual angular velocity measured by the gyroscope, forming the inner loop of the attitude controller. The desired torque control quantity can be obtained from the angular velocity change through inverse equation solving. Output.

[0059] In this embodiment, by According to the lift coefficient and torque coefficient The rotational speeds of the four motors of the quadcopter can be calculated, and the corresponding output current can be obtained based on the motor parameters.

[0060] In this embodiment, the calculated current is transmitted to the ESC via the controller, and the ESC controls the motor to rotate at a specified speed, thereby achieving attitude and motion trajectory control of the UAV.

[0061] like Figure 7 , Figure 8As shown, in this embodiment, after obstacle avoidance is completed, the aircraft resumes flight towards the current target point. When heading towards the fire, the target point is the geographical coordinates of the fire. After the fire is extinguished, the target point is changed to the geographical coordinates of the ground station.

[0062] Furthermore, after obstacle avoidance is completed, the flight mode is switched to constant speed and orientation flight to proceed to the target point. During this period, steps S4-S8 are continuously executed.

[0063] In summary, this invention utilizes binocular vision as a foundation, incorporating a Realsense depth camera to acquire forward depth information. This information is then processed and multiplied by the direction coefficient matrix to obtain the drone's pitch angle, roll angle, and throttle control parameters. This provides guidance for subsequent obstacle avoidance and control methods and systems for firefighting drones based on binocular vision and Realsense. This invention features fast response speed, good obstacle avoidance performance, and rapid arrival at the destination.

[0064] This invention uses binocular vision as a foundation and incorporates a Realsense depth camera, combining the two to obtain depth information in front, thereby reducing the risk of collisions with drones.

[0065] This invention uses roll angle coefficient, throttle coefficient, and pitch angle coefficient as inputs into the UAV control system to control the attitude and trajectory of the UAV, thereby achieving obstacle avoidance.

[0066] This invention multiplies the processed depth map with the orientation coefficient matrix to obtain the pitch angle, roll angle, and throttle control parameters of the UAV, directly controlling the system. Compared to traditional artificial potential fields, this invention offers faster response, better obstacle avoidance in mountainous terrain, avoids getting stuck in local optima, and can operate successfully in complex environments such as fire scenes or at night.

[0067] This invention solves the technical problems of poor obstacle avoidance performance, low real-time performance and low safety of existing drones.

[0068] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for obstacle avoidance control of a firefighting drone based on binocular vision and RealSense depth camera, characterized in that, The method includes: S1. Calibrate the preset camera to obtain the intrinsic parameter matrix of the two cameras. The preset camera includes a stereo camera and a RealSense depth camera. S2. Collect and obtain latitude and longitude information and terrain information of the fire site, and set multiple path points accordingly; S3. Use the preset positioning module to obtain the current latitude and longitude information of the drone and set the next path point as the target location; S4. Using the preset camera, obtain the distance information from the obstacle in front of the drone to the camera plane, and obtain a depth map accordingly; S5. Filter the depth map to filter out interference from distant obstacles, set the perception range, and filter out obstacles outside the perception range to obtain a depth map matrix. S6. Multiply the depth map matrix by the roll coefficient matrix, throttle coefficient matrix, fuselage coefficient matrix and identity matrix respectively to obtain at least two Hadamard products, and process them to obtain the roll angle coefficient, throttle coefficient and pitch angle coefficient. S6 includes: S61. Define the roll coefficient matrix, the throttle coefficient matrix and the fuselage coefficient matrix using preset logic; S62. Using the following logic, based on the depth map matrix The roll coefficient matrix, the throttle coefficient matrix, and the fuselage coefficient matrix are respectively obtained and the depth map matrix is ​​calculated. Hadamaji: ; S63, respectively convert the rolling coefficient matrix The throttle coefficient matrix and the aforementioned fuselage coefficient matrix With the depth map matrix The sum of each element in the Hadamard product is divided by the width and height of the depth map to obtain the lateral avoidance coefficient, longitudinal avoidance coefficient, and fuselage avoidance coefficient. S64. Process the lateral avoidance coefficient, the longitudinal avoidance coefficient, and the fuselage avoidance coefficient using preset logic to obtain the roll angle coefficient, the throttle coefficient, and the pitch angle coefficient; S7. Input the roll angle coefficient, the throttle coefficient, and the pitch angle coefficient into the preset UAV control system to control the attitude and trajectory of the UAV, so as to control the UAV to avoid obstacles. S8. After passing the obstacle, restore the initial flight direction of the UAV according to the target position to control the UAV to fly towards the current target point.

2. The obstacle avoidance control method for firefighting drones based on binocular vision and RealSense depth camera according to claim 1, characterized in that, Step S4 includes: S41. Use the preset camera to collect image feature points and obtain the distance information from the obstacle in front of the UAV to the camera plane; S42. Using the following logic, calculate the acquisition deviation of the preset camera based on the image feature points. : in, Let be the distance from the target point to the camera plane. The distance between the two cameras. For the camera's focal length, The parallax of the left and right cameras; S43. Based on the principle of similar triangles and the aforementioned acquisition deviation, the coordinates are transformed from the pixel coordinate system to the real-world coordinate system. The coordinate transformation formula is as follows: in, These are the x and y coordinates of the point in the camera coordinate system, respectively. Here, x and y are the x and y coordinates of the center point of the entire image, respectively, and f is the camera focal length. These are coordinates with the RealSense depth camera as the origin and the camera plane as the coordinate system. The coordinates of that point in the actual coordinate system of the surface; S44. Record all the scanned image feature points in a preset matrix to obtain a depth map. .

3. The obstacle avoidance control method for firefighting drones based on binocular vision and RealSense depth camera according to claim 1, characterized in that, Step S5 includes: S51. Using the following logic, the depth map... Calculate the average of all elements : In the formula, , These represent the number of rows and columns of the depth map, respectively. For depth map The Okay, number Column elements; S52, Set the sensing range Based on this, a filtering depth map is obtained; S53. Process the filtered depth map with preset logic to obtain the depth map matrix.

4. The obstacle avoidance control method for firefighting drones based on binocular vision and RealSense depth camera according to claim 3, characterized in that, In step S52, the sensing range is set using the following logic. : In the formula, Current location To the target location Euclidean distance, The area of ​​the target location is considered as the drone having reached the target location.

5. The obstacle avoidance control method for firefighting drones based on binocular vision and RealSense depth camera according to claim 3, characterized in that, In step S53, the filtered depth map is processed using the following logic to obtain a processed depth map matrix. No. line, number Column element values : 。 6. The obstacle avoidance control method for a firefighting drone based on binocular vision and RealSense depth camera according to claim 1, characterized in that, In step S61, the rolling coefficient matrix is ​​defined using the following logic. The throttle coefficient matrix and the aforementioned fuselage coefficient matrix :

7. The obstacle avoidance control method for firefighting drones based on binocular vision and RealSense depth camera according to claim 1, characterized in that, In step S63, the elements in the Hadamard product and the depth map are processed using the following logic to obtain the lateral avoidance coefficient. The longitudinal avoidance coefficient and the aforementioned fuselage avoidance coefficient : 。 8. The obstacle avoidance control method for a firefighting drone based on binocular vision and RealSense depth camera according to claim 1, characterized in that, In step S64, the roll angle coefficient, the throttle coefficient, and the pitch angle coefficient are obtained using the following logic processing: In the formula, These are the avoidance trigger coefficients for the horizontal, vertical, and fuselage directions, respectively.

9. A firefighting drone obstacle avoidance control system based on binocular vision and a RealSense depth camera, used to execute the firefighting drone obstacle avoidance control method based on binocular vision and a RealSense depth camera as described in any one of claims 1 to 8, characterized in that, The system includes: The camera calibration module is used to calibrate the preset camera and obtain the intrinsic parameter matrix of the two cameras; The path point acquisition module is used to collect latitude and longitude information and terrain information of the fire site, and to set multiple path points accordingly; The target location acquisition module is used to obtain the current latitude and longitude information of the UAV using the preset positioning module, and set the next path point as the target location; A depth processing module is used to acquire distance information from obstacles in front of the UAV to the camera plane using the preset camera, and to obtain a depth map accordingly. The depth processing module is connected to the camera calibration module. A depth map filtering module is used to filter the depth map, filter out interference from distant obstacles, set the perception range, and filter out obstacles outside the perception range to obtain a depth map matrix. The depth map filtering module is connected to the depth processing module. The UAV control coefficient processing module is used to multiply the depth map matrix by the roll coefficient matrix, throttle coefficient matrix, fuselage coefficient matrix and identity matrix respectively to obtain at least two Hadamard products, and then process them to obtain the roll angle coefficient, throttle coefficient and pitch angle coefficient. The UAV control coefficient processing module is connected to the depth map filtering module. The obstacle avoidance control module is used to input the roll angle coefficient, the throttle coefficient, and the pitch angle coefficient into the preset UAV control system, thereby controlling the attitude and trajectory of the UAV to avoid obstacles. The obstacle avoidance control module is connected to the UAV control coefficient processing module. The obstacle avoidance post-control module is used to restore the initial flight direction of the UAV based on the target position after passing the obstacle, so as to control the UAV to fly towards the current target point. The obstacle avoidance post-control module is connected to the obstacle avoidance control module and the target position acquisition module.

Citation Information

Patent Citations

  • Multi-sensor fusion-based autonomous obstacle avoidance unmanned aerial vehicle system and control method

    CN105892489A

  • Unmanned aerial vehicle path planning method based on artificial potential field and ant colony algorithm

    CN111784079A

  • Unmanned aerial vehicle (UAV) obstacle avoidance method and system based on binocular vision and optical flow fusion

    CN106681353A

  • Navigation map configuration method, obstacle avoidance method and device, terminal and unmanned aerial vehicle

    CN109708636A