Inspection unmanned aerial vehicle autonomous obstacle avoidance method, system and device, and storage medium

Abstract

The embodiment of the invention provides an autonomous obstacle avoidance method, system and device for an inspection unmanned aerial vehicle, and a storage medium, and relates to the field of unmanned aerial vehicle obstacle avoidance. Binocular camera video frames of the inspection unmanned aerial vehicle are acquired, splicing and depth calculation processing are performed on the binocular camera video frames, and a spliced video frame and a three-dimensional point cloud are obtained; obstacle detection is carried out on the spliced video frame to obtain obstacle information, association processing is carried out on the obstacle information to obtain a motion equation and a motion track of an obstacle, and whether a collision risk exists between the inspection unmanned aerial vehicle and the obstacle is judged according to the flight parameters of the inspection unmanned aerial vehicle and the motion equation of the obstacle in the spliced video frame; and if there is a collision risk between the inspection unmanned aerial vehicle and the obstacle, constructing a two-dimensional binary polar histogram based on the three-dimensional point cloud, and calculating the obstacle avoidance path of the inspection unmanned aerial vehicle according to the two-dimensional binary polar histogram and the movement track of the obstacle, thereby effectively improving the autonomous obstacle avoidance capability of the inspection unmanned aerial vehicle in a mountainous area complex environment. And the safety of the inspection unmanned aerial vehicle is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle obstacle avoidance, and in particular to an autonomous obstacle avoidance method, system, device and storage medium for an inspection unmanned aerial vehicle. Background Art

[0002] In recent years, with the rapid development of drone technology, drones have also been widely used in the field of power equipment inspection, significantly improving the operation and maintenance of overhead power lines. Power equipment is usually located in mountainous areas, so drones need to face areas with complex and changeable natural environments such as mountains and hills. Due to the high mountains and dense forests, and the vigorous growth of trees and bamboos, the phenomenon of tree barriers encroaching on overhead line channels is very common, which has a significant impact on the safe flight of drone inspections.

[0003] Traditional drone obstacle avoidance solutions mainly rely on infrared sensors or binocular vision on drones. However, there are certain limitations in actual work. For example, the binocular vision system can perceive and construct a three-dimensional image of the surrounding environment by simulating the working principle of the human eye, but when dealing with dynamic obstacles, there will be response delays or misjudgments, resulting in poor obstacle avoidance effects. Although infrared sensors can penetrate vegetation to a certain extent, their ability to accurately measure the specific shape, size and distance of obstacles is limited, and they cannot ensure that drones can complete inspection tasks safely and efficiently in complex environments. Therefore, there is an urgent need for an autonomous obstacle avoidance method that can ensure the safety of drone inspections in complex mountainous environments. Summary of the invention

[0004] In view of this, the purpose of the present invention is to overcome the deficiencies in the prior art and to provide an autonomous obstacle avoidance method, system, device and storage medium for an inspection drone.

[0005] The present invention provides the following technical solutions: In a first aspect, an autonomous obstacle avoidance method for an inspection drone is provided in an embodiment of the present disclosure, the method comprising: Obtain the flight parameters of the inspection drone and obtain the binocular camera video frames on the preset route in real time through the binocular camera carried by the inspection drone; Performing stitching and depth calculation processing on the binocular camera video frames to obtain a stitched video frame and a three-dimensional point cloud of the current field of view of the inspection drone; Performing obstacle detection on the spliced ​​video frame based on a target detection algorithm to obtain obstacle information of obstacles in the spliced ​​video frame; Performing association processing on the obstacle information of the obstacle based on the target tracking algorithm to obtain the motion equation and motion trajectory of the obstacle in the spliced ​​video frame; Determining whether there is a collision risk between the inspection drone and the obstacle according to the flight parameters of the inspection drone and the motion equation of the obstacle in the spliced ​​video frame; If there is a risk of collision between the inspection drone and an obstacle, a two-dimensional binary polar histogram of the inspection drone is constructed based on the three-dimensional point cloud, and an obstacle avoidance path of the inspection drone is calculated according to the two-dimensional binary polar histogram and the motion trajectory of the obstacle; Among them, the specific method of judging whether there is a collision risk between the inspection drone and the obstacle according to the flight parameters of the inspection drone and the motion equation of the obstacle in the spliced ​​video frame is: constructing the motion equation of the inspection drone according to the flight parameters of the inspection drone, and determining the relationship between the distance between the obstacle and the inspection drone over time according to the motion equation of the inspection drone and the motion equation of the obstacle; if the distance between the inspection drone and the obstacle is always greater than the preset safety obstacle avoidance threshold, deleting the tracking instance corresponding to the obstacle; if the distance between the inspection drone and the obstacle is less than the preset safety obstacle avoidance threshold at any time, it is determined that there is a collision risk between the inspection drone and the obstacle; Among them, if there is a risk of collision between the inspection drone and the obstacle, a two-dimensional binary polar histogram of the inspection drone is constructed based on the three-dimensional point cloud, and the obstacle avoidance path of the inspection drone is calculated according to the two-dimensional binary polar histogram and the motion trajectory of the obstacle. The specific method is: an octree environment map is created for the three-dimensional point cloud based on the octomap algorithm, and a two-dimensional polar coordinate histogram centered on the inspection drone is constructed through the octree environment map. The two-dimensional polar coordinate histogram is binarized according to a preset binarization threshold to obtain a two-dimensional binary polar histogram. According to the motion trajectory of the obstacle, the obstacle information of the obstacle with a collision risk is introduced into the two-dimensional binary polar histogram. The two-dimensional binary polar histogram is scanned based on a sliding window, and windows with all elements being 0 are detected and marked to obtain all obstacle avoidance paths of the inspection drone. Path evaluation processing is performed on all the obstacle avoidance paths, wherein the cost formula of the path evaluation processing is: ,in, is the total cost of the current node n, is the cost from the current node n to the target point, is the cost from the starting point to the current node n. In the two-dimensional binary histogram, the selection formula of the cost function from the current node n to the target point is: , where n is the current node, is the total cost of the node, is the discount factor, is the target weight parameter, The target cost, is the yaw weight parameter, The deviation cost, is the path smoothing weight parameter, is the path smoothing cost, is the tree smoothing weight parameter, is the tree smoothing cost, and the weights of all the obstacle avoidance paths are calculated by a weight calculation formula, wherein the weight calculation formula is: , where g is the target position direction of the inspection UAV, and p is the candidate flight direction of the current inspection UAV. is the candidate flight direction of the inspection drone at the previous time. is the target cost coefficient, is the smoothing cost coefficient, For the target price, is the smoothing cost, is the total cost weight, is the yaw angle difference in two directions, is the upper deviation of the pitch angle, is the lower deviation of the pitch angle, and is the corresponding coefficient, and the final obstacle avoidance path is selected according to the path evaluation results and weights of all the obstacle avoidance paths.

[0006] According to a specific embodiment disclosed in the present application, before the step of obtaining the flight parameters of the inspection drone and obtaining the binocular camera video frames on the preset route in real time through the binocular camera carried by the inspection drone, it also includes: Setting the three-dimensional coordinates of the photographing point and the three-dimensional coordinates of the auxiliary point of the inspection drone; The preset route of the inspection drone is determined according to the three-dimensional coordinates of the photographing point and the three-dimensional coordinates of the auxiliary point.

[0007] According to a specific implementation method disclosed in the present application, the step of performing stitching processing and depth calculation processing on the binocular camera video frames to obtain the stitched video frames and three-dimensional point cloud of the current field of view of the inspection drone includes: Performing registration processing on the key points of the binocular camera video frame based on the SIFT algorithm to obtain a registered video frame and establishing a spatial mapping relationship between pixels according to the registered video frame; Based on the spatial mapping relationship, the binocular camera video frames are stitched to obtain a stitched video frame of the current field of view of the inspection drone; Calculate the disparity of the binocular camera video frame based on the SGBM algorithm; The parallax is converted into depth information based on the triangulation principle, and the pixel points in the binocular camera video frame are converted into a three-dimensional point cloud based on the depth information.

[0008] According to a specific implementation manner disclosed in the present application, the step of performing obstacle detection on the spliced ​​video frame based on a target detection algorithm to obtain obstacle information of obstacles in the spliced ​​video frame includes: Input the spliced ​​video frame into the feature extraction network of the YOLOv5 algorithm and output three feature layers; The three feature layers are fused to obtain a fused feature layer; Regression calculation and classification calculation are performed on the fused feature layer according to a preset anchor frame to obtain obstacle information of obstacles in the spliced ​​video frame.

[0009] According to a specific implementation method disclosed in the present application, the step of performing association processing on the obstacle information of the obstacle based on the target tracking algorithm to obtain the motion equation and motion trajectory of the obstacle in the spliced ​​video frame includes: At time t, the spliced ​​video frame Each detection result constructs a tracking instance, wherein the state of the tracking instance includes the spliced ​​video frame The position parameters of the detection results and the spliced ​​video frames Stitched video frames predicted by detection results Position parameters of the test results; At time t+1, the spliced ​​video frame is processed based on the Hungarian algorithm. The detection result and the spliced ​​video frame at time t Stitched video frames predicted by detection results The detection results are tracked and matched with the instances; Performing update processing, new addition processing, or deletion processing on the tracking instance according to the tracking instance matching result; The motion equation and motion trajectory of the obstacle in the spliced ​​video frame are obtained by combining the tracking examples of the spliced ​​video frames at different historical moments.

[0010] In a second aspect, an autonomous obstacle avoidance system for an inspection drone is provided in an embodiment of the present disclosure, and the system includes: An acquisition module is used to obtain the flight parameters of the inspection UAV and obtain the binocular camera video frames on the preset route in real time through the binocular camera carried by the inspection UAV; A processing module, used for performing stitching processing and depth calculation processing on the binocular camera video frames to obtain a stitched video frame and a three-dimensional point cloud of the current field of view of the inspection drone; A detection module, configured to perform obstacle detection on the spliced ​​video frame based on a target detection algorithm to obtain obstacle information of the obstacle in the spliced ​​video frame; A tracking module, used for performing association processing on the obstacle information of the obstacle based on a target tracking algorithm to obtain a motion equation and a motion trajectory of the obstacle in the spliced ​​video frame; A judgment module, used to judge whether there is a collision risk between the inspection drone and the obstacle according to the flight parameters of the inspection drone and the motion equation of the obstacle in the spliced ​​video frame; A calculation module is used to construct a two-dimensional binary polar histogram of the inspection drone based on the three-dimensional point cloud if there is a risk of collision between the inspection drone and an obstacle, and calculate an obstacle avoidance path of the inspection drone based on the two-dimensional binary polar histogram and the motion trajectory of the obstacle.

[0011] In a third aspect, an electronic device is provided in an embodiment of the present disclosure, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the autonomous obstacle avoidance method for an inspection drone as described in any one of the first aspects when executing the computer program.

[0012] In a fourth aspect, a computer-readable storage medium is provided in an embodiment of the present disclosure, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the autonomous obstacle avoidance method for an inspection drone described in any one of the first aspects are implemented.

[0013] The autonomous obstacle avoidance method for the inspection drone provided in the present application obtains the flight parameters of the inspection drone and obtains the binocular camera video frames on the preset route in real time through the binocular camera carried by the inspection drone, performs stitching processing and depth calculation processing on the binocular camera video frames to obtain the stitched video frames and three-dimensional point clouds of the current field of view of the inspection drone, performs obstacle detection on the stitched video frames based on the target detection algorithm to obtain the obstacle information of the obstacles in the stitched video frames, performs association processing on the obstacle information based on the target tracking algorithm to obtain the motion equation and motion trajectory of the obstacles in the stitched video frames, and obtains the obstacle information of the obstacles in the stitched video frames according to the inspection drone. The flight parameters of the inspection UAV and the motion equation of the obstacle in the spliced ​​video frame are used to determine whether there is a collision risk between the inspection UAV and the obstacle. If there is a collision risk between the inspection UAV and the obstacle, a two-dimensional binary polar histogram of the inspection UAV is constructed based on the three-dimensional point cloud, and the obstacle avoidance path of the inspection UAV is calculated according to the two-dimensional binary polar histogram and the motion trajectory of the obstacle. The obstacle is identified and tracked by using target detection and target tracking algorithms, and its motion trajectory and collision risk are predicted. The final obstacle avoidance path is calculated, which can effectively improve the autonomous obstacle avoidance capability of the inspection UAV in complex mountainous environments and ensure the safety of the inspection UAV.

[0014] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and understandable, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work. In each of the drawings, similar components are numbered similarly.

[0016] Figure 1 A flow chart of an autonomous obstacle avoidance method for an inspection drone provided in an embodiment of the present application is shown; Figure 2 A mathematical principle diagram of obtaining depth information from a two-dimensional binocular image provided by an embodiment of the present application is shown; Figure 3 A schematic diagram of the structure of an autonomous obstacle avoidance system for an inspection drone provided in an embodiment of the present application is shown; Figure 4 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0017] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0018] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. In contrast, when an element is referred to as being "directly on" another element, there is no intermediate element. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0019] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0020] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of the template are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0022] Example 1 like Figure 1 As shown, it is a flow chart of an autonomous obstacle avoidance method for an inspection drone provided in an embodiment of the present application. The autonomous obstacle avoidance method for an inspection drone provided in an embodiment of the present application comprises the following steps: Step S101, obtaining the flight parameters of the inspection drone and obtaining the binocular camera video frames on the preset route in real time through the binocular camera carried by the inspection drone.

[0023] Specifically, the flight parameters of the inspection drone include position parameters, yaw angle, pitch angle, roll angle, gimbal angle, speed parameters, acceleration parameters, power parameters, flight mode and communication parameters. The inspection drone flies along a preset route to complete a specific distribution network system inspection task. When performing the inspection operation, the inspection drone will obtain the binocular camera video frames in the direction of travel of the preset route in real time through the binocular camera carried. The binocular camera is a device that uses the principle of bionics to imitate human binocular vision, and captures images from different angles through two cameras.

[0024] Before the step of obtaining the flight parameters of the inspection drone and obtaining the binocular camera video frames on the preset route in real time through the binocular camera carried by the inspection drone, the method further includes: Setting the three-dimensional coordinates of the photographing point and the three-dimensional coordinates of the auxiliary point of the inspection drone; The preset route of the inspection drone is determined according to the three-dimensional coordinates of the photographing point and the three-dimensional coordinates of the auxiliary point.

[0025] It can be understood that the three-dimensional coordinates of the photo point refer to the specific position where the inspection drone needs to hover when performing the photo mission, usually expressed by three coordinate values ​​of X, Y, and Z, which correspond to longitude, latitude, and altitude in geographic space respectively. The three-dimensional coordinates of the auxiliary point refer to other position points that the inspection drone may need to refer to during the execution of the mission. They can be used for path planning, obstacle avoidance, or as checkpoints in the flight path. They are also expressed by three coordinate values ​​of X, Y, and Z. Through the three-dimensional coordinates of the photo point and the three-dimensional coordinates of the auxiliary point, the flight control system of the inspection drone will plan a preset route from the starting point to the photo point, then pass the auxiliary point, and finally return to the starting point. Before performing the flight mission, the binocular camera is calibrated to obtain the focal length, optical center, relative position and posture of the left and right cameras of the binocular camera.

[0026] Step S102, performing stitching processing and depth calculation processing on the binocular camera video frames to obtain stitching video frames and three-dimensional point clouds of the current field of view of the inspection drone.

[0027] Specifically, the binocular camera simulates the principle of human binocular vision, obtains two video frames, and then uses image processing technology to perform image registration, feature point matching and parallax calculation, and finally constructs a three-dimensional point cloud of the current field of view of the inspection drone.

[0028] The step of performing stitching processing and depth calculation processing on the binocular camera video frames to obtain a stitched video frame and a three-dimensional point cloud of the current field of view of the inspection drone includes: Performing registration processing on the key points of the binocular camera video frame based on the SIFT algorithm to obtain a registered video frame and establishing a spatial mapping relationship between pixels according to the registered video frame; Based on the spatial mapping relationship, the binocular camera video frames are stitched to obtain a stitched video frame of the current field of view of the inspection drone; Calculate the disparity of the binocular camera video frame based on the SGBM algorithm; The parallax is converted into depth information based on the triangulation principle, and the pixel points in the binocular camera video frame are converted into a three-dimensional point cloud based on the depth information.

[0029] In specific implementation, the SIFT (Scale Invariant Feature Transform) algorithm is used to extract key points in the video frame and generate feature descriptors after scale space extremum detection, key point positioning, direction allocation and feature description. A spatial mapping relationship between pixels is established based on the key points and feature descriptors. Based on the spatial mapping relationship, the corresponding feature points in the binocular camera video frame are aligned to form a continuous video frame. The SGBM (Semi-Global Block Matching) algorithm is used to generate a disparity map after preprocessing, matching cost calculation and energy function minimization. The disparity map represents the horizontal distance difference between corresponding points in the left and right video frames.

[0030] See also Figure 2 , Figure 2 A mathematical principle diagram for obtaining depth information from a two-dimensional binocular image provided in an embodiment of the present application, based on the principle of triangulation, using two viewpoints and the baseline distance between them, and calculating the depth information of each point in the scene through the disparity value. Preferably, P is any spatial point, x is the coordinate of point P corresponding to the X direction, z is the distance from point P to the binocular camera plane, f is the focal length of the binocular camera, b is the baseline length, is the coordinate of the projection point of point P on the left image plane in the X direction of the left image plane, is the coordinate of the projection point of point P on the right image plane in the X direction of the right image plane.

[0031] The similar triangle formula is derived from the similar triangle principle: Expand the similar triangle formula so that the Y axis is perpendicular to the paper and points outward: , where f is the focal length of the binocular camera, z is the distance from point P to the binocular camera plane, and y is the coordinate of point P corresponding to the Y direction. is the coordinate of the projection point of point P on the left image plane in the Y direction of the left image plane, The coordinate of the projection point of point P on the right image plane in the Y direction of the right image plane is obtained, and the depth formula is , where d is the disparity and the value of d is - .

[0032] Step S103: performing obstacle detection on the spliced ​​video frame based on a target detection algorithm to obtain obstacle information of the obstacles in the spliced ​​video frame.

[0033] Specifically, obstacle detection is performed on the spliced ​​video frame based on a target detection algorithm, obstacles in the video frame are identified, and obstacle information of the obstacles is obtained, where the obstacle information refers to location information and type information of the obstacles.

[0034] The step of performing obstacle detection on the spliced ​​video frame based on a target detection algorithm to obtain obstacle information of obstacles in the spliced ​​video frame includes: Input the spliced ​​video frame into the feature extraction network of the YOLOv5 algorithm and output three feature layers; The three feature layers are fused to obtain a fused feature layer; Regression calculation and classification calculation are performed on the fused feature layer according to a preset anchor frame to obtain obstacle information of obstacles in the spliced ​​video frame.

[0035] In the specific implementation, the spliced ​​video frames are input into the feature extraction network in the YOLOv5 (You Only Look Once) deep learning algorithm, and the spliced ​​video frames are subjected to multiple convolution and pooling processes to extract feature information at different levels, and finally three feature layers are output, which are used to detect objects of different sizes, that is, to detect obstacles of different sizes. The three feature layers are fully integrated to enhance the feature expression capability and combine feature information of different scales to improve the target detection capability. According to the preset anchor frame, the features in each anchor frame are subjected to confidence calculation and bounding box regression to determine the type, position and size of the obstacle, and finally the type information and position information of the obstacle in the spliced ​​video frame are obtained. The spliced ​​video frames of the inspection drone are subjected to target detection through the YOLOv5 deep learning algorithm, so that the inspection drone can accurately identify and locate obstacles in the video, providing a guarantee for the safe flight of the inspection drone.

[0036] Step S104: performing association processing on the obstacle information of the obstacle based on a target tracking algorithm to obtain a motion equation and a motion trajectory of the obstacle in the spliced ​​video frame.

[0037] The step of performing association processing on the obstacle information of the obstacle based on the target tracking algorithm to obtain the motion equation and motion trajectory of the obstacle in the spliced ​​video frame includes: At time t, the spliced ​​video frame Each detection result constructs a tracking instance, wherein the state of the tracking instance includes the spliced ​​video frame The position parameters of the detection results and the spliced ​​video frames Stitched video frames predicted by detection results Position parameters of the test results; At time t+1, the spliced ​​video frame is processed based on the Hungarian algorithm. The detection result and the spliced ​​video frame at time t Stitched video frames predicted by detection results The detection results are tracked and matched with the instances; Performing update processing, new addition processing, or deletion processing on the tracking instance according to the tracking instance matching result; The motion equation and motion trajectory of the obstacle in the spliced ​​video frame are obtained by combining the tracking examples of the spliced ​​video frames at different historical moments.

[0038] Specifically, stitching video frames The position parameters of the detection results are obtained by Kalman filtering according to the spliced ​​video frame Predicted, i.e. spliced ​​video frames Detection results in the spliced ​​video frame The Hungarian algorithm calculates the distance between the nodes of two frames and finds the optimal matching relationship to update the tracking instance. Detection results and spliced ​​video frames Predicted spliced ​​video frames If the result matches, the tracking instance corresponding to the prediction result is retained and the spliced ​​video frame is used The position parameters of the detection results and the spliced ​​video frames Stitched video frames predicted by detection results The position parameter of the result updates the state of the tracking instance; if the spliced ​​video frame The detection result does not match the spliced ​​video frame Predicted spliced ​​video frames As a result, a new record is added to the tracking instance, and the state of the record is composed of the spliced ​​video frame The location information of the detection results and the spliced ​​video frames Stitched video frames predicted by detection results The result is composed of position parameters; if the spliced ​​video frame Predicted spliced ​​video frames The result did not match the stitched video frame Detection results, then delete the spliced ​​video frames Predicted spliced ​​video frames The tracking example corresponding to the result is associating the obstacle information of the obstacle based on the target tracking algorithm to obtain the motion equation and motion trajectory of the obstacle in the spliced ​​video frame, thereby improving the accuracy and robustness of the inspection drone for obstacle tracking.

[0039] Step S105 , judging whether there is a collision risk between the inspection drone and the obstacle according to the flight parameters of the inspection drone and the motion equation of the obstacle in the spliced ​​video frame.

[0040] The step of judging whether there is a collision risk between the inspection drone and the obstacle according to the flight parameters of the inspection drone and the motion equation of the obstacle in the spliced ​​video frame includes: Constructing a motion equation of the inspection drone according to the flight parameters of the inspection drone; Determine the relationship between the distance between the obstacle and the inspection drone over time according to the motion equation of the inspection drone and the motion equation of the obstacle; If the distance between the inspection drone and the obstacle is always greater than the preset safety obstacle avoidance threshold, the tracking instance corresponding to the obstacle is deleted; If the distance between the inspection drone and the obstacle is less than a preset safety obstacle avoidance threshold at any time, it is determined that there is a risk of collision between the inspection drone and the obstacle.

[0041] Specifically, the motion equation of the inspection drone is expressed by its position and speed, the distance between the obstacle and the inspection drone is calculated by the Euclidean distance formula, and the relationship between the distance and time can be obtained by calculating the distance at consecutive time points. The preset safety obstacle avoidance threshold is the minimum collision avoidance distance that should be maintained between the inspection drone and the obstacle, which can be determined according to the flight speed, maneuverability and environmental factors of the inspection drone.

[0042] Step S106: If there is a risk of collision between the inspection drone and an obstacle, a two-dimensional binary polar histogram of the inspection drone is constructed based on the three-dimensional point cloud, and an obstacle avoidance path of the inspection drone is calculated according to the two-dimensional binary polar histogram and the motion trajectory of the obstacle.

[0043] If there is a risk of collision between the inspection drone and an obstacle, the step of constructing a two-dimensional binary polar histogram of the inspection drone based on the three-dimensional point cloud, and calculating an obstacle avoidance path of the inspection drone according to the two-dimensional binary polar histogram and the motion trajectory of the obstacle includes: Creating an octree environment map for the three-dimensional point cloud based on an octomap algorithm; A two-dimensional polar coordinate histogram centered on the inspection drone is constructed through the octree environment map; Binarizing the two-dimensional polar coordinate histogram according to a preset binarization threshold to obtain a two-dimensional binary polar histogram; Introducing obstacle information of obstacles with collision risks into the two-dimensional binary polar histogram according to the movement trajectory of the obstacles; Scanning the two-dimensional binary extreme histogram based on a sliding window, detecting and marking windows where all elements are 0, and obtaining all obstacle avoidance paths of the inspection drone; Path evaluation processing is performed on all the obstacle avoidance paths, wherein the cost formula of the path evaluation processing is: ,in, is the total cost of the current node n, is the cost from the current node n to the target point, is the cost from the starting point to the current node n. In the two-dimensional binary histogram, the selection formula of the cost function from the current node n to the target point is: , where n is the current node, is the total cost of the node, is the discount factor, is the target weight parameter, The target cost, is the yaw weight parameter, The deviation cost, is the path smoothing weight parameter, is the path smoothing cost, is the tree smoothing weight parameter, Smooth cost for the tree; The weights of all the obstacle avoidance paths are calculated using a weight calculation formula, wherein the weight calculation formula is: , where g is the target position direction of the inspection UAV, and p is the candidate flight direction of the current inspection UAV. is the candidate flight direction of the inspection drone at the previous time. is the target cost coefficient, is the smoothing cost coefficient, For the target price, is the smoothing cost, is the total cost weight, is the yaw angle difference in two directions, is the upper deviation of the pitch angle, is the lower deviation of the pitch angle, and is the corresponding coefficient; The obstacle avoidance path with the smallest weight is selected as the final obstacle avoidance path.

[0044] In specific implementation, the octree environment map is a tree data structure, in which each node represents a cubic space and can be further subdivided into 8 child nodes. Based on the octomap (Octree Occupancy Map) algorithm, an octree environment map is created for the three-dimensional point cloud, and obstacles in the three-dimensional space are efficiently represented by 3D grid occupancy. A two-dimensional polar coordinate histogram is constructed based on the obstacle information in the octree environment map with the inspection drone as the center. The three-dimensional space information is projected onto a two-dimensional plane centered on the inspection drone through the two-dimensional polar coordinate histogram. The two-dimensional polar coordinate histogram uses a polar coordinate system to represent the position of the obstacle relative to the drone. In the two-dimensional polar coordinate histogram, each cell represents a specific direction and distance range, and the value of the cell represents the density or existence probability of the obstacle within the direction and distance range.

[0045] Preferably, the coordinate transformation formula for constructing a histogram from a three-dimensional point cloud is: ,in, is the yaw angle, , , Point cloud The coordinates on the X, Y, and Z axes, , , is the coordinate of the current local origin O on the X, Y, and Z axes, is the pitch angle.

[0046] Specifically, the two-dimensional polar coordinate histogram is binarized according to a preset binarization threshold value, and the values ​​in the histogram are divided into 1 and 0, wherein 1 represents an obstacle and 0 represents a non-obstacle.

[0047] It can be understood that by calculating the weight of each obstacle avoidance path, the optimal flight path can be determined, which not only ensures that the selected path can avoid obstacles, but also meets the flight performance and inspection mission requirements of the inspection drone. The consistency of the path and the target direction of the inspection drone is calculated by weight. The closer the path is to the target direction, the lower its weight. In order to ensure the smoothness and safety of the flight, the weight calculation selects a path with fewer turns and gentle changes. The flight distance of the obstacle avoidance path is calculated by weight, and the shorter path will have a lower weight. The proximity of the obstacle to the path will also affect the weight. The path closer to the obstacle will have a higher weight. By calculating the weights of different costs, the obstacle avoidance path with the smallest weight is finally selected as the final obstacle avoidance path, thereby achieving a safe, efficient and inspection mission-compliant flight path.

[0048] The autonomous obstacle avoidance method for an inspection drone provided in an embodiment of the present application obtains the flight parameters of the inspection drone and obtains the binocular camera video frames on a preset route in real time through the binocular camera carried by the inspection drone, performs splicing processing and depth calculation processing on the binocular camera video frames to obtain a spliced ​​video frame and a three-dimensional point cloud of the current field of view of the inspection drone, performs obstacle detection on the spliced ​​video frames based on a target detection algorithm to obtain obstacle information of obstacles in the spliced ​​video frames, performs association processing on the obstacle information of the obstacles based on a target tracking algorithm to obtain the motion equation and motion trajectory of the obstacles in the spliced ​​video frames, and obtains the obstacle information of the inspection drone based on the object tracking algorithm. The flight parameters of the inspection UAV and the motion equation of the obstacle in the spliced ​​video frame are used to determine whether there is a collision risk between the inspection UAV and the obstacle. If there is a collision risk between the inspection UAV and the obstacle, a two-dimensional binary polar histogram of the inspection UAV is constructed based on the three-dimensional point cloud, and the obstacle avoidance path of the inspection UAV is calculated according to the two-dimensional binary polar histogram and the motion trajectory of the obstacle. The obstacle is identified and tracked by using target detection and target tracking algorithms, and its motion trajectory and collision risk are predicted. The final obstacle avoidance path is calculated, which can effectively improve the autonomous obstacle avoidance capability of the inspection UAV in complex mountainous environments and ensure the safety of the inspection UAV.

[0049] Example 2 like Figure 3 As shown, it is a structural schematic diagram of an autonomous obstacle avoidance system 300 for an inspection drone provided in an embodiment of the present application, and the system includes: The acquisition module 301 is used to acquire the flight parameters of the inspection drone and acquire the binocular camera video frames on the preset route in real time through the binocular camera carried by the inspection drone; The processing module 302 is used to perform stitching processing and depth calculation processing on the binocular camera video frames to obtain a stitched video frame and a three-dimensional point cloud of the current field of view of the inspection drone; A detection module 303 is used to perform obstacle detection on the spliced ​​video frame based on a target detection algorithm to obtain obstacle information of obstacles in the spliced ​​video frame; A tracking module 304 is used to perform association processing on the obstacle information of the obstacle based on a target tracking algorithm to obtain a motion equation and a motion trajectory of the obstacle in the spliced ​​video frame; A judgment module 305 is used to judge whether there is a collision risk between the inspection drone and the obstacle according to the flight parameters of the inspection drone and the motion equation of the obstacle in the spliced ​​video frame; The calculation module 306 is used to construct a two-dimensional binary polar histogram of the inspection drone based on the three-dimensional point cloud if there is a risk of collision between the inspection drone and the obstacle, and calculate the obstacle avoidance path of the inspection drone based on the two-dimensional binary polar histogram and the motion trajectory of the obstacle.

[0050] The autonomous obstacle avoidance system 300 for an inspection drone provided in this embodiment can implement the autonomous obstacle avoidance method for an inspection drone shown in Example 1, and will not be described again here to avoid repetition.

[0051] The autonomous obstacle avoidance system 300 for an inspection drone provided in an embodiment of the present application obtains the flight parameters of the inspection drone and obtains the binocular camera video frames on a preset route in real time through the binocular camera carried by the inspection drone, performs stitching processing and depth calculation processing on the binocular camera video frames to obtain stitched video frames and three-dimensional point clouds of the current field of view of the inspection drone, performs obstacle detection on the stitched video frames based on a target detection algorithm to obtain obstacle information of obstacles in the stitched video frames, performs association processing on the obstacle information of obstacles based on a target tracking algorithm to obtain motion equations and motion trajectories of obstacles in the stitched video frames, and performs object tracking processing on the obstacle information of the inspection drone. The flight parameters of the human-machine and the motion equation of the obstacle in the spliced ​​video frame are used to determine whether there is a collision risk between the inspection drone and the obstacle. If there is a collision risk between the inspection drone and the obstacle, a two-dimensional binary extreme histogram of the inspection drone is constructed based on the three-dimensional point cloud, and the obstacle avoidance path of the inspection drone is calculated according to the two-dimensional binary extreme histogram and the motion trajectory of the obstacle. The obstacle is identified and tracked using target detection and target tracking algorithms, and its motion trajectory and collision risk are predicted. The final obstacle avoidance path is calculated, which can effectively improve the autonomous obstacle avoidance capability of the inspection drone in complex mountainous environments and ensure the safety of the inspection drone.

[0052] Example 3 In addition, an embodiment of the present disclosure provides an electronic device 400, including a memory and a processor, wherein the memory stores a computer program.

[0053] For details, see Figure 4The electronic device 400 includes: a receiver 401, a bus interface and a processor 402.

[0054] In the embodiment of the present invention, the electronic device 400 further includes a memory 403. Figure 4 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors represented by processor 402 and memory represented by memory 403. The bus architecture may also link various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The receiver 401 may be a plurality of components, namely, a transmitter and a receiver, providing a unit for communicating with various other systems on a transmission medium. The processor 402 is responsible for managing the bus architecture and general processing, and the memory 403 may store data used by the processor 402 when performing operations.

[0055] The electronic device 400 provided in the embodiment of the present invention can implement the autonomous obstacle avoidance method for the inspection drone shown in Example 1, which will not be described again here to avoid repetition.

[0056] In several embodiments provided in the present application, it should be understood that the disclosed system and method can also be implemented in other ways. The system embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or the flow diagram, and the combination of the boxes in the structure diagram and / or the flow diagram can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0057] In addition, the functional modules or units in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0058] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0059] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An autonomous obstacle avoidance method for an inspection drone, characterized in that: The method comprises: Obtain the flight parameters of the inspection drone and obtain the binocular camera video frames on the preset route in real time through the binocular camera carried by the inspection drone; Performing stitching and depth calculation processing on the binocular camera video frames to obtain a stitched video frame and a three-dimensional point cloud of the current field of view of the inspection drone; Performing obstacle detection on the spliced ​​video frame based on a target detection algorithm to obtain obstacle information of obstacles in the spliced ​​video frame; Performing association processing on the obstacle information of the obstacle based on the target tracking algorithm to obtain the motion equation and motion trajectory of the obstacle in the spliced ​​video frame; Determining whether there is a collision risk between the inspection drone and the obstacle according to the flight parameters of the inspection drone and the motion equation of the obstacle in the spliced ​​video frame; If there is a risk of collision between the inspection drone and an obstacle, a two-dimensional binary polar histogram of the inspection drone is constructed based on the three-dimensional point cloud, and an obstacle avoidance path of the inspection drone is calculated according to the two-dimensional binary polar histogram and the motion trajectory of the obstacle; Among them, the specific method of judging whether there is a collision risk between the inspection drone and the obstacle according to the flight parameters of the inspection drone and the motion equation of the obstacle in the spliced ​​video frame is: constructing the motion equation of the inspection drone according to the flight parameters of the inspection drone, and determining the relationship between the distance between the obstacle and the inspection drone over time according to the motion equation of the inspection drone and the motion equation of the obstacle; if the distance between the inspection drone and the obstacle is always greater than the preset safety obstacle avoidance threshold, deleting the tracking instance corresponding to the obstacle; if the distance between the inspection drone and the obstacle is less than the preset safety obstacle avoidance threshold at any time, it is determined that there is a collision risk between the inspection drone and the obstacle; Among them, if there is a risk of collision between the inspection drone and the obstacle, a two-dimensional binary polar histogram of the inspection drone is constructed based on the three-dimensional point cloud, and the obstacle avoidance path of the inspection drone is calculated according to the two-dimensional binary polar histogram and the motion trajectory of the obstacle. The specific method is: an octree environment map is created for the three-dimensional point cloud based on the octomap algorithm, and a two-dimensional polar coordinate histogram centered on the inspection drone is constructed through the octree environment map. The two-dimensional polar coordinate histogram is binarized according to a preset binarization threshold to obtain a two-dimensional binary polar histogram. According to the motion trajectory of the obstacle, the obstacle information of the obstacle with a collision risk is introduced into the two-dimensional binary polar histogram. The two-dimensional binary polar histogram is scanned based on a sliding window, and windows with all elements being 0 are detected and marked to obtain all obstacle avoidance paths of the inspection drone. Path evaluation processing is performed on all the obstacle avoidance paths, wherein the cost formula of the path evaluation processing is: ,in, is the total cost of the current node n, is the cost from the current node n to the target point, is the cost from the starting point to the current node n. In the two-dimensional binary histogram, the selection formula of the cost function from the current node n to the target point is: , where n is the current node, is the total cost of the node, is the discount factor, is the target weight parameter, The target cost, is the yaw weight parameter, The deviation cost, is the path smoothing weight parameter, is the path smoothing cost, is the tree smoothing weight parameter, is the tree smoothing cost, and the weights of all the obstacle avoidance paths are calculated by a weight calculation formula, wherein the weight calculation formula is: , where g is the target position direction of the inspection UAV, and p is the candidate flight direction of the current inspection UAV. is the candidate flight direction of the inspection drone at the previous time. is the target cost coefficient, is the smoothing cost coefficient, For the target price, is the smoothing cost, is the total cost weight, is the yaw angle difference in two directions, is the upper deviation of the pitch angle, is the lower deviation of the pitch angle, and is the corresponding coefficient, and the final obstacle avoidance path is selected according to the path evaluation results and weights of all the obstacle avoidance paths.

2. The autonomous obstacle avoidance method for an inspection drone according to claim 1, characterized in that: Before the step of obtaining the flight parameters of the inspection drone and obtaining the binocular camera video frames on the preset route in real time through the binocular camera carried by the inspection drone, the method further includes: Setting the three-dimensional coordinates of the photographing point and the three-dimensional coordinates of the auxiliary point of the inspection drone; The preset route of the inspection drone is determined according to the three-dimensional coordinates of the photographing point and the three-dimensional coordinates of the auxiliary point.

3. The autonomous obstacle avoidance method for an inspection drone according to claim 1, characterized in that: The step of performing stitching processing and depth calculation processing on the binocular camera video frames to obtain a stitched video frame and a three-dimensional point cloud of the current field of view of the inspection drone includes: Performing registration processing on the key points of the binocular camera video frame based on the SIFT algorithm to obtain a registered video frame and establishing a spatial mapping relationship between pixels according to the registered video frame; Based on the spatial mapping relationship, the binocular camera video frames are stitched to obtain a stitched video frame of the current field of view of the inspection drone; Calculate the disparity of the binocular camera video frame based on the SGBM algorithm; The parallax is converted into depth information based on the triangulation principle, and the pixel points in the binocular camera video frame are converted into a three-dimensional point cloud based on the depth information.

4. The autonomous obstacle avoidance method for an inspection drone according to claim 1, characterized in that: The step of performing obstacle detection on the spliced ​​video frame based on a target detection algorithm to obtain obstacle information of obstacles in the spliced ​​video frame includes: Input the spliced ​​video frame into the feature extraction network of the YOLOv5 algorithm and output three feature layers; The three feature layers are fused to obtain a fused feature layer; Regression calculation and classification calculation are performed on the fused feature layer according to a preset anchor frame to obtain obstacle information of obstacles in the spliced ​​video frame.

5. The autonomous obstacle avoidance method for an inspection drone according to claim 1, characterized in that: The step of performing association processing on the obstacle information of the obstacle based on the target tracking algorithm to obtain the motion equation and motion trajectory of the obstacle in the spliced ​​video frame includes: At time t, the spliced ​​video frame Each detection result constructs a tracking instance, wherein the state of the tracking instance includes the spliced ​​video frame The position parameters of the detection results and the spliced ​​video frames Stitched video frames predicted by detection results Position parameters of the test results; At time t+1, the spliced ​​video frame is processed based on the Hungarian algorithm. The detection result and the spliced ​​video frame at time t Stitched video frames predicted by detection results The detection results are tracked and matched with the instances; Performing update processing, new addition processing, or deletion processing on the tracking instance according to the tracking instance matching result; The motion equation and motion trajectory of the obstacle in the spliced ​​video frame are obtained by combining the tracking examples of the spliced ​​video frames at different historical moments.

6. An autonomous obstacle avoidance system for an inspection drone, characterized in that: The system comprises: An acquisition module is used to obtain the flight parameters of the inspection UAV and obtain the binocular camera video frames on the preset route in real time through the binocular camera carried by the inspection UAV; A processing module, used for performing stitching processing and depth calculation processing on the binocular camera video frames to obtain a stitched video frame and a three-dimensional point cloud of the current field of view of the inspection drone; A detection module, configured to perform obstacle detection on the spliced ​​video frame based on a target detection algorithm to obtain obstacle information of the obstacle in the spliced ​​video frame; A tracking module, used for performing association processing on the obstacle information of the obstacle based on a target tracking algorithm to obtain a motion equation and a motion trajectory of the obstacle in the spliced ​​video frame; A judgment module, used to judge whether there is a collision risk between the inspection drone and the obstacle according to the flight parameters of the inspection drone and the motion equation of the obstacle in the spliced ​​video frame, construct the motion equation of the inspection drone according to the flight parameters of the inspection drone, and determine the relationship between the distance between the obstacle and the inspection drone over time according to the motion equation of the inspection drone and the motion equation of the obstacle. If the distance between the inspection drone and the obstacle is always greater than a preset safety obstacle avoidance threshold, the tracking instance corresponding to the obstacle is deleted; if the distance between the inspection drone and the obstacle is less than the preset safety obstacle avoidance threshold at any time, it is determined that there is a collision risk between the inspection drone and the obstacle. A calculation module is used for constructing a two-dimensional binary polar histogram of the inspection drone based on the three-dimensional point cloud if there is a risk of collision between the inspection drone and the obstacle, and calculating the obstacle avoidance path of the inspection drone based on the two-dimensional binary polar histogram and the motion trajectory of the obstacle, creating an octree environment map for the three-dimensional point cloud based on the octomap algorithm, constructing a two-dimensional polar coordinate histogram centered on the inspection drone through the octree environment map, binarizing the two-dimensional polar coordinate histogram according to a preset binarization threshold to obtain a two-dimensional binary polar histogram, introducing obstacle information of obstacles with a collision risk into the two-dimensional binary polar histogram according to the motion trajectory of the obstacle, scanning the two-dimensional binary polar histogram based on a sliding window, detecting and marking windows with all elements being 0, obtaining all obstacle avoidance paths of the inspection drone, and performing path evaluation processing on all the obstacle avoidance paths, wherein the cost formula of the path evaluation processing is: ,in, is the total cost of the current node n, is the cost from the current node n to the target point, is the cost from the starting point to the current node n. In the two-dimensional binary histogram, the selection formula of the cost function from the current node n to the target point is: , where n is the current node, is the total cost of the node, is the discount factor, is the target weight parameter, The target cost, is the yaw weight parameter, The deviation cost, is the path smoothing weight parameter, is the path smoothing cost, is the tree smoothing weight parameter, is the tree smoothing cost, and the weights of all the obstacle avoidance paths are calculated by a weight calculation formula, wherein the weight calculation formula is: , where g is the target position direction of the inspection UAV, and p is the candidate flight direction of the current inspection UAV. is the candidate flight direction of the inspection drone at the previous time. is the target cost coefficient, is the smoothing cost coefficient, For the target price, is the smoothing cost, is the total cost weight, is the yaw angle difference in two directions, is the upper deviation of the pitch angle, is the lower deviation of the pitch angle, and is the corresponding coefficient, and the final obstacle avoidance path is selected according to the path evaluation results and weights of all the obstacle avoidance paths.

7. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the autonomous obstacle avoidance method for an inspection drone described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the autonomous obstacle avoidance method for an inspection drone according to any one of claims 1 to 5 are implemented.

Images