Unmanned aerial vehicle obstacle avoidance method and device, unmanned aerial vehicle and storage medium
By combining real-time environmental images and reference images, obstacle avoidance strategies are generated to avoid obstacles, which solves the problem that four-rotor aircraft is difficult to avoid obstacles in dynamic environments, and improves the obstacle avoidance ability and flight safety of the drone .
Patent Information
- Application Number
- CN202410992280.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
AI Technical Summary
Quadrotor aircraft is difficult to avoid obstacles in a timely manner in dynamic and uncertain environments, resulting in accidents and damage.
By obtaining the real-time environmental images and obstacle distance information of the drone, estimating the movement path of the target object based on real-time and reference images, combining the obstacle distance information to generate obstacle avoidance strategies, and controlling the drone to avoid obstacles.
It improves the flexibility, autonomy and intelligence of the obstacle avoidance of drones, ensures flight safety, and achieves independent obstacle avoidance in complex environments.
Smart Images

Figure CN118938987A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) path planning, and in particular to an unmanned aerial vehicle (UAV) obstacle avoidance method, device, UAV and storage medium. Background Art
[0002] In the mountainous areas of southwest my country, due to complex geographical factors and other natural factors, it is often necessary to establish a large-scale and ultra-long power transportation network. The transmission lines in this part of the area are often affected by natural disasters. The main types of disasters are floods, landslides and mining collapses, among which landslides account for a relatively high proportion. In addition, in many special terrains with high vegetation coverage, it is difficult to use artificial means to check for landslide risks. Therefore, in the maintenance of transmission lines, quadcopters play a great role in the operation and maintenance of power systems.
[0003] In the face of dynamic and uncertain environments, quadcopters cannot avoid obstacles in time when encountering them, which may cause accidents and damage themselves. Therefore, how to enable quadcopters to accurately detect obstacles that affect their flight and respond to avoid obstacles is an urgent problem to be solved. Summary of the invention
[0004] The present application proposes a drone obstacle avoidance method, device, drone and storage medium to improve the drone's obstacle avoidance flexibility, autonomy and intelligence, and improve its flight safety.
[0005] In a first aspect, a method for avoiding obstacles of a drone is provided, comprising:
[0006] Acquire a real-time environment image and obstacle distance information of the UAV, wherein the real-time environment image includes at least one target object;
[0007] estimating a motion path of the target object based on the real-time environment image and a reference image, wherein the reference image is a real-time environment image of the UAV at a previous moment;
[0008] generating an obstacle avoidance strategy based on the motion path and the obstacle distance information;
[0009] The drone is controlled to avoid obstacles based on the obstacle avoidance strategy.
[0010] In a second aspect, a drone obstacle avoidance device is provided, comprising:
[0011] An acquisition module, used to acquire a real-time environment image and obstacle distance information of the UAV, wherein the real-time environment image includes at least one target object;
[0012] An estimation module, used for estimating the motion path of the target object based on the real-time environment image and a reference image, wherein the reference image is the real-time environment image of the UAV at the previous moment;
[0013] A first generating module, configured to generate an obstacle avoidance strategy based on the motion path and the obstacle distance information;
[0014] A control module is used to control the drone to avoid obstacles based on the obstacle avoidance strategy.
[0015] Optionally, in some embodiments of the present application, the estimation module includes:
[0016] A matching submodule, used for performing feature matching between the real-time environment image and the reference image;
[0017] A determination submodule, configured to determine a motion vector of the target object based on the real-time environment image and the reference image if feature matching is successful;
[0018] The estimation submodule is used to estimate the motion path of the target object based on the motion vector.
[0019] Optionally, in some embodiments of the present application, the matching submodule includes:
[0020] A first recognition unit, used for recognizing a first key feature point of the real-time environment image and extracting a first descriptor of the first key feature point;
[0021] A second recognition unit, used for recognizing a second key feature point of the reference image and extracting a second descriptor of the second key feature point;
[0022] A matching unit is used to perform feature matching on the first descriptor and the second descriptor.
[0023] Optionally, in some embodiments of the present application, the first generating module includes:
[0024] A first generating submodule, configured to generate the obstacle avoidance strategy based on the motion path if the distance in the obstacle distance information is greater than a preset distance threshold;
[0025] The second generating submodule is used to generate the obstacle avoidance strategy based on the distance if the distance in the obstacle distance information is less than or equal to a preset distance threshold.
[0026] Optionally, in some embodiments of the present application, the drone obstacle avoidance device further includes a preprocessing module, and the preprocessing module is specifically used to:
[0027] A preprocessing operation is performed on the real-time environment image.
[0028] Optionally, in some embodiments of the present application, the drone obstacle avoidance device further includes:
[0029] A second generating module, configured to generate the obstacle avoidance strategy based on the distance if it is detected that the target object does not exist in the real-time environment image and the distance in the obstacle distance information is less than or equal to a preset distance threshold;
[0030] The execution module is used to continue to acquire the real-time environment image of the drone if it is detected that the target object does not exist in the real-time environment image and the distance in the obstacle distance information is greater than the preset distance threshold.
[0031] Optionally, in some embodiments of the present application, the drone obstacle avoidance device further includes:
[0032] A detection module, used to perform feature point detection on the real-time environment image to obtain feature point data;
[0033] A classification module, used to classify the feature point data to obtain a grouping result corresponding to each target object;
[0034] A division module, used for performing spatial division on the real-time environment image to obtain a spatial division result;
[0035] A determination module, configured to determine the position information of the target object in the real-time environment image based on the grouping result and the space division result;
[0036] The third generating module is used to generate an obstacle avoidance strategy based on the position information of each target object.
[0037] In a third aspect, a drone is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned drone obstacle avoidance method when executing the computer program.
[0038] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned drone obstacle avoidance method are implemented.
[0039] The present application provides a method, device, drone and storage medium for avoiding obstacles of a drone, which obtains the real-time environment image and obstacle distance information of the drone, wherein the real-time environment image includes at least one target object; estimates the motion path of the target object based on the real-time environment image and the reference image, wherein the reference image is the real-time environment image of the drone at the previous moment; generates an obstacle avoidance strategy based on the motion path and the obstacle distance information; and controls the drone to avoid obstacles based on the obstacle avoidance strategy. In the obstacle avoidance scheme of the drone provided by the present application, the motion path of the target object is estimated by the real-time environment image and the reference image, and the obstacle avoidance strategy is generated by combining the obstacle distance information and the motion path, so as to control the drone to avoid obstacles. It can be seen that by combining the obstacle distance information and the motion path to generate the obstacle avoidance strategy, the environment and obstacle distance can be monitored in real time, so that the drone can fly purposefully to avoid obstacles, ensure the safety of itself and its environment, and is conducive to improving the flexibility, autonomy and intelligence of the drone to avoid obstacles, and improving its flight safety. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0041] Figure 1 A diagram of the application environment of the drone obstacle avoidance method provided in the embodiment of the present application;
[0042] Figure 2 A flowchart of a drone obstacle avoidance method provided in an embodiment of the present application;
[0043] Figure 3 A decision flow chart for obstacle avoidance of a drone provided in an embodiment of the present application;
[0044] Figure 4 A flowchart of the obstacle avoidance process for a drone provided in one embodiment of the present application;
[0045] Figure 5 A structural block diagram of a drone obstacle avoidance device provided in an embodiment of the present application;
[0046] Figure 6 A structural block diagram of the drone provided in an embodiment of the present application. DETAILED DESCRIPTION
[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0048] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.
[0049] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.
[0050] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.
[0051] The unmanned aerial vehicle obstacle avoidance method provided by the embodiment of the present invention can be applied in the following aspects: Figure 1In the application environment, the drone 110 communicates with the server 120 through the network 130. The server 120 can obtain the real-time environment image and obstacle distance information of the drone, wherein the real-time environment image includes at least one target object; estimate the motion path of the target object based on the real-time environment image and the reference image, wherein the reference image is the real-time environment image of the drone at the previous moment; generate an obstacle avoidance strategy based on the motion path and the obstacle distance information; and control the drone 110 to avoid obstacles based on the obstacle avoidance strategy. In the present invention, the motion path of the target object is estimated by the real-time environment image and the reference image, and the obstacle avoidance strategy is generated in combination with the obstacle distance information and the motion path, so as to control the drone to avoid obstacles. It can be seen that by generating an obstacle avoidance strategy in combination with the obstacle distance information and the motion path, the environment and obstacle distance can be monitored in real time, so that the drone can purposefully fly to avoid obstacles, ensure the safety of itself and its environment, and is conducive to improving the obstacle avoidance flexibility, autonomy and intelligence of the drone, and improving its flight safety. The present invention is described in detail below through specific embodiments.
[0052] See also Figure 2 As shown, Figure 2 The following is a flow chart of a method for avoiding obstacles on a drone provided by an embodiment of the present invention. The method can be applied to a server. This embodiment is described by taking the application to the server as an example. The method for avoiding obstacles on a drone includes the following steps:
[0053] 101: Obtain the real-time environment image and obstacle distance information of the drone.
[0054] The real-time environment image includes at least one target object, which may be an obstacle, including vegetation, power transmission lines, animals, natural terrain (such as hillsides, cliffs, valleys), and geological disasters (such as landslides and mud-rock flows).
[0055] During the flight of the drone, the distance between each obstacle and the drone can be measured by a sensor (such as an ultrasonic sensor) installed on the drone, thereby obtaining obstacle distance information. Specifically, the distance between each obstacle and the drone can be measured by the time from the sensor transmitting the signal to the signal receiving the signal.
[0056] The real-time environment image is an image of the surrounding environment captured by the drone in real time during flight. For example, the environment around the drone can be captured by a camera provided on the drone to obtain the real-time environment image.
[0057] 102: Estimate a motion path of the target object based on the real-time environment image and the reference image.
[0058] The reference image is a real-time environment image of the UAV at the last moment.
[0059] The motion path of the target object refers to the position sequence of the target object in space that changes over time. The motion path can be used to indicate the speed and direction of the obstacle and its position information at the next moment.
[0060] The motion path of the target object can be estimated by analyzing the pixel motion of the target object in the reference image and the real-time environment image.
[0061] Deep learning models, such as convolutional neural networks (CNNs), can be used to identify and predict the motion of target objects and obtain the motion path of the target objects.
[0062] 103: Generate an obstacle avoidance strategy based on the motion path and the obstacle distance information.
[0063] The obstacle avoidance strategy is a strategy used by the drone to identify and avoid obstacles during flight. The obstacle avoidance strategy may include changing the flight direction of the drone, raising or lowering the flight altitude, or changing the speed (such as deceleration) to avoid obstacles.
[0064] The obstacle distance information can be used to indicate whether the distance between each obstacle and the drone is a safe distance. When the distance between the obstacle and the drone is at a safe distance, the drone can be guaranteed to have enough reaction time to avoid the obstacle. Therefore, the flight parameters of the drone, such as the speed, direction, and altitude, can be determined based on the distance between the obstacle and the drone and the movement path of the obstacle, which includes the movement speed, movement direction, and position information of the obstacle at the next moment; the flight parameters of the drone, such as the speed, direction, and altitude, are used as an obstacle avoidance strategy. For example, compare the position information of the obstacle at the next moment with the position of the drone at the same moment in the flight path to see if there is a collision risk. If there is a collision risk, an obstacle avoidance strategy is generated based on the movement speed, movement direction, and position information of the obstacle at the next moment.
[0065] For example, the obstacle avoidance strategy can be that if the obstacle is close, the drone needs to slow down or change speed so that the drone has more time to respond to the obstacle avoidance; if the obstacle is far away, the drone may be able to maintain the current speed or adjust it according to the mission requirements; if the obstacle poses a threat in the vertical direction, calculate and adjust the flight altitude of the drone to avoid the obstacle; adjust the direction of the drone to avoid the predicted path of the obstacle. The vector dot product or cross product can be used to calculate the angle that the drone needs to turn to for the unit vector corresponding to the direction of the drone's movement and the direction vector determined by the obstacle's position at the next moment relative to the drone's position at the same moment, so as to adjust the direction of the drone based on the angle.
[0066] 104: Control the drone to avoid obstacles based on the obstacle avoidance strategy.
[0067] The flight altitude, flight speed and flight direction of the UAV can be determined according to the obstacle avoidance strategy, and the current flight path of the UAV can be updated based on the flight altitude, flight speed and flight direction. The UAV can be controlled to fly according to the updated flight path to achieve autonomous and safe obstacle avoidance.
[0068] The obstacle avoidance strategy can be converted into specific flight targets, such as flight path, altitude, speed, etc., and the proportional (P), integral (I), and differential (D) parameters of the PID controller can be configured to achieve fast response and stable control; the error between the flight target corresponding to the obstacle avoidance measurement and the current flight target of the drone is calculated, and the PID controller outputs a control signal proportional to the size of the error (proportional gain * error) to adjust the various actuators of the drone. For example, the PID controller is used to adjust the attitude (pitch angle, roll angle, yaw angle) of the drone to change the flight direction; the propeller speed of the drone is controlled to adjust the forward speed or ascent / descent speed of the drone.
[0069] In actual operation, according to the changes in the flying environment of the UAV, an obstacle avoidance strategy is generated in real time, and the PID parameters are dynamically adjusted based on the obstacle avoidance strategy to generate control instructions to adjust the various actuators of the UAV.
[0070] In this embodiment, the control system of the UAV can accurately execute obstacle avoidance strategies to achieve autonomous flight and safe obstacle avoidance in complex environments.
[0071] In one embodiment, estimating the motion path of the target object based on the real-time environment image and the reference image includes:
[0072] Performing feature matching on the real-time environment image and the reference image;
[0073] If the feature matching is successful, determining the motion vector of the target object based on the real-time environment image and the reference image;
[0074] A motion path of the target object is estimated based on the motion vector.
[0075] A neural network can be used to perform feature matching between the target object in the real-time environment image and the target object in the reference image. If the feature matching is successful, it means that the target object is moving. Therefore, the position difference of each pair of matching feature points in the real-time environment image and the reference image can be calculated, and the average or weighted average of each position difference can be calculated to obtain a motion vector to reduce the estimation error of the motion path of the target object. The motion vector includes the motion direction and speed of the target object. According to the motion characteristics of the target object, a suitable motion model is selected, such as a constant speed model (predicted position = current position + motion vector × Δt, where Δt is the time interval), a constant acceleration model, etc. The current position information and motion vector of the drone are used as the input of the motion model to predict the position information of the target object at a future time, thereby obtaining the motion path of the target object.
[0076] Image processing technology can be used to process the real-time environment image and the reference image to estimate the motion path of the target object. This process includes preprocessing the real-time environment image and the reference image (such as grayscale, filtering, sharpening, etc.); using feature detection methods, such as corner detection algorithms, to perform feature detection on the preprocessed real-time environment image and the reference image respectively, and matching the feature points (corner points or edges) found in the reference image with the feature points (corner points or edges) found in the real-time environment image. After the match is successful, the optical flow method, such as the Lucas-King method, is applied to each pair of matched features to obtain the motion vector of the target object. The motion vector is obtained by the position of the obstacle at the corner feature or edge feature; the motion vector of the target object can be calculated according to the following formula:
[0077]
[0078] Wherein, h represents the estimated optical flow vector and the direction and size of the feature point moving in the image sequence (such as the image sequence composed of the reference image and the real-time environment image) (i.e., the motion vector of the target object); x is the feature point identified in the image; F(x) represents the feature function in the real-time environment image and the position of the feature point in the real-time environment image; F′(x) is the gradient of the feature function, which represents the rate of change of the feature point in space, which is obtained by calculating the first-order derivative of the real-time environment image; G(x) is the feature function after motion estimation, which represents the position of the feature point in the real-time environment image at the next moment; It is the sum of the product of the gradient of the feature function and the difference between the real-time environment image of the feature point at the next moment and the current real-time environment image, which represents the error of the feature point movement; the denominator The sum of the squares of the feature function gradients is used to normalize the numerator to ensure that the calculated optical flow vector is not affected by the size of the feature gradients.
[0079] During the operation of the aircraft, it is possible that the feature points (corners or edges) corresponding to the target object produce unusual movements in a specific environment. The disadvantage of this method is that if there is a large undetected movement in the search area. In order to solve this problem, the Lucas-Kanade method is developed into a pyramid shape for compensation, that is, the real-time environment image and the reference image are sampled separately to obtain corresponding images of different resolutions, starting from the highest position of the pyramid (low-resolution image), and gradually moving to the lowest position of the pyramid (high-resolution image), the position information of the feature point is accurately determined, thereby obtaining the motion vector of the feature point.
[0080] In one embodiment, the performing feature matching on the real-time environment image and the reference image includes:
[0081] Identifying a first key feature point of the real-time environment image and extracting a first descriptor of the first key feature point;
[0082] Identifying a second key feature point of the reference image and extracting a second descriptor of the second key feature point;
[0083] Perform feature matching on the first descriptor and the second descriptor.
[0084] A corner detection algorithm is used to perform feature matching between the real-time environment image and the reference image. Specifically, a threshold value of the minimum quality level of corners and edges is determined, and the threshold value R is calculated according to the following formula:
[0085] R=min(λ1λ2)
[0086] Wherein, eigenvalue λ1 and eigenvalue λ2 represent the bidirectional axis values of the ellipse that determine the corners.
[0087] The bidirectional axis values of the corner ellipse can be the second-order moment (also called diffusion moment or moment) around the feature point, which can be used to describe the shape and size of the local area around the feature point. The local area around the feature point can be fitted into an ellipse, and the major and minor axes of the ellipse represent the scale (size) and direction of the area.
[0088] If there is at least one feature point in the real-time environment image whose feature value is greater than a threshold, it means that the at least one feature point is a corner point or an edge, and the at least one feature point is taken as a first key feature point, and a descriptor of the corner point or the edge is extracted from the first key feature point to form a feature vector to obtain a first descriptor; if there is at least one feature point in the reference image whose feature value is greater than a threshold, it means that the at least one feature point is a corner point or an edge, and the at least one feature point is taken as a second key feature point, and a descriptor of the corner point or the edge is extracted from the second key feature point to form a feature vector to obtain a second descriptor; calculate the similarity between the first descriptor and the second descriptor, if the similarity meets the preset similarity threshold, the feature matching is successful, otherwise the feature matching fails.
[0089] In one embodiment, the generating an obstacle avoidance strategy based on the motion path and the obstacle distance information includes:
[0090] If the distance in the obstacle distance information is greater than a preset distance threshold, generating the obstacle avoidance strategy based on the motion path;
[0091] If the distance in the obstacle distance information is less than or equal to a preset distance threshold, the obstacle avoidance strategy is generated based on the distance.
[0092] like Figure 3 As shown, the minimum distance between the UAV and the obstacle (i.e., the preset threshold) is set to 50 meters. During the image processing of the real-time environment image, if the image processing detects an obstacle, it is identified whether the distance between the UAV and the obstacle obtained by the ultrasonic sensor is 50 centimeters or less; if it is greater than 50 centimeters, the movement path of the obstacle will be used as the obstacle avoidance reference of the UAV to update the current flight path of the UAV (i.e., flight speed, flight altitude, flight direction, etc.); but if it is less than or equal to 50 centimeters, the distance between the UAV and the obstacle is used as the obstacle avoidance reference of the obstacle avoiding UAV to update the current flight path of the UAV (i.e., flight speed, flight altitude, flight direction, etc.).
[0093] In one embodiment, before estimating the motion path of the target object based on the real-time environment image and the reference image, the method further includes:
[0094] A preprocessing operation is performed on the real-time environment image.
[0095] The preprocessing operation includes filtering, that is, filtering the real-time environment image to reduce random noise in the image and improve the image quality.
[0096] In one embodiment, it further includes:
[0097] If it is detected that the target object does not exist in the real-time environment image, and the distance in the obstacle distance information is less than or equal to a preset distance threshold, generating the obstacle avoidance strategy based on the distance;
[0098] If it is detected that the target object does not exist in the real-time environment image and the distance in the obstacle distance information is greater than the preset distance threshold, the real-time environment image of the drone is acquired.
[0099] like Figure 3 As shown, the minimum distance between the drone and the obstacle (i.e., the preset threshold value) is set to 50 meters. During the image processing of the real-time environment image, if the image processing cannot detect the obstacle (such as the obstacle cannot be photographed in the dark), the ultrasonic sensor reads that the distance between the drone and the obstacle is still above 50 cm, then continue to obtain the real-time environment image of the drone for obstacle detection; if the image processing cannot detect the obstacle (such as the obstacle cannot be photographed in the dark), but the ultrasonic sensor reads that the distance between the drone and the obstacle is still less than or equal to 50 centimeters, then the distance between the drone and the obstacle is used as an obstacle avoidance reference for the obstacle-avoiding drone, so as to update the current flight path of the drone (i.e., flight speed, flight altitude, flight direction, etc.).
[0100] In one embodiment, it further includes:
[0101] Performing feature point detection on the real-time environment image to obtain feature point data;
[0102] Classifying the feature point data to obtain a grouping result corresponding to each target object;
[0103] Performing spatial division on the real-time environment image to obtain a spatial division result;
[0104] Determine the position information of the target object in the real-time environment image based on the grouping result and the space division result;
[0105] An obstacle avoidance strategy is generated based on the position information of each of the target objects.
[0106] A neural network or other method can be used to detect feature points of the real-time environment image to obtain feature point data, and a classification algorithm (such as a support vector machine or a decision tree) can be used to classify the feature point data to obtain a grouping result of the feature point data. Based on the grouping result, the number of target objects in the real-time environment image and the position information of each target object in the real-time environment image can be determined; the real-time environment image is divided into four different quadrants to obtain a spatial division result, and then an obstacle avoidance strategy is generated based on the position information of each target object, that is, the distribution of the position information of each target object in the real-time environment image in the real-time environment image is determined, the quadrant without the target object is determined, and the drone is controlled to move to the quadrant without the target object or the quadrant with the least number of target objects.
[0107] In one embodiment, the features in the real-time environment image can be grouped by the preset average value of each feature, and the real-time environment image can be divided into four different quadrants, so as to determine the distribution of obstacles in the real-time environment image. The obstacle avoidance strategy is to control the drone to move to the quadrant without obstacles.
[0108] In one embodiment, the drone is a quadcopter, which is equipped with a camera and an ultrasonic sensor. Figure 4 As shown, the obstacle avoidance method of the drone includes:
[0109] The original image (i.e., the real-time environment image) is obtained through the camera, and the real-time environment image is preprocessed. The preprocessing process includes filtering the real-time environment image, using the corner detection algorithm to perform feature detection and feature comparison of the real-time environment image and the reference image respectively. If the feature match is successful, the differential method of optical flow estimation is used to track the successfully matched feature pair, estimate the movement path of the obstacle, and obtain the obstacle avoidance strategy; the features in the real-time environment image can also be grouped by the preset average value of each feature, and the real-time environment image is divided into four different quadrants to determine the distribution of obstacles in the real-time environment image. The obstacle avoidance strategy is to control the drone to move to the quadrant without obstacles; at the same time, the ultrasonic sensor is used to read the distance between the drone and the obstacle to make a decision to determine the obstacle avoidance strategy executed by the drone, and then the PID controller is used to control the movement of the drone based on the obstacle avoidance strategy. Ultrasonic sensors are used as additional functions of drones to help drones determine whether there are obstacles. The principle is to detect the time required for ultrasonic waves to travel from the transmitter to the receiver. The detection of obstacles by ultrasonic sensors depends on many factors such as the direction, reflectivity, and curvature of the obstacle surface to the sensor, and also depends on the received echo detection threshold.
[0110] In this embodiment, an ultrasonic sensor is used to supplement the success rate of image processing to fully detect obstacles. Obstacle conditions are read by the cooperation of the camera and the ultrasonic sensor, and the obstacle avoidance strategy of the drone is determined based on the distance between the drone and the obstacle. The PID controller based on visual feedback control is used to control the movement of the drone to improve the obstacle avoidance flexibility, autonomy and safety of the drone. In addition, the function of reading the distance between the obstacle and the drone is provided by using an ultrasonic sensor to help the drone improve the accuracy of detecting obstacles. If an error occurs when the real-time environmental image is processed (such as an obstacle cannot be detected), the ultrasonic sensor can read the distance between the obstacle and the drone for reference by the drone to avoid obstacles. The purpose of allowing the quadcopter to perform automatic obstacle avoidance within a certain range is to enable the quadcopter to fly purposefully to avoid obstacles and ensure the safety of itself and its environment without human intervention. Therefore, this application has a strong practical significance for the research, promotion and development of autonomous obstacle avoidance of drones and the application of quadcopters.
[0111] The above is the obstacle avoidance process of the drone in this application.
[0112] As described above, the present application provides a method, device, drone and storage medium for avoiding obstacles of a drone, by acquiring the real-time environment image and obstacle distance information of the drone, wherein the real-time environment image includes at least one target object; estimating the motion path of the target object based on the real-time environment image and the reference image, wherein the reference image is the real-time environment image of the drone at the previous moment; generating an obstacle avoidance strategy based on the motion path and the obstacle distance information; and controlling the drone to avoid obstacles based on the obstacle avoidance strategy. In the obstacle avoidance scheme of the drone provided by the present application, the motion path of the target object is estimated by the real-time environment image and the reference image, and the obstacle avoidance strategy is generated by combining the obstacle distance information and the motion path, so as to control the drone to avoid obstacles. It can be seen that by generating an obstacle avoidance strategy by combining the obstacle distance information and the motion path, the environment and obstacle distance can be monitored in real time, so that the drone can fly purposefully to avoid obstacles, ensure the safety of itself and its environment, and is conducive to improving the flexibility, autonomy and intelligence of the drone to avoid obstacles, and improving its flight safety.
[0113] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.
[0114] In one embodiment, a drone obstacle avoidance device is provided, which corresponds one-to-one to the drone obstacle avoidance method in the above embodiment. Figure 5As shown, the unmanned aerial vehicle obstacle avoidance device includes an acquisition module 201, an estimation module 202, a first generation module 203 and a control module 204, and each functional module is described in detail as follows:
[0115] An acquisition module 201 is used to acquire a real-time environment image and obstacle distance information of the UAV, wherein the real-time environment image includes at least one target object;
[0116] An estimation module 202, configured to estimate a motion path of the target object based on the real-time environment image and a reference image, wherein the reference image is a real-time environment image of the UAV at a previous moment;
[0117] A first generating module 203, configured to generate an obstacle avoidance strategy based on the motion path and the obstacle distance information;
[0118] The control module 204 is used to control the UAV to avoid obstacles based on the obstacle avoidance strategy.
[0119] Optionally, in some embodiments of the present application, the estimation module 202 includes:
[0120] A matching submodule, used for performing feature matching between the real-time environment image and the reference image;
[0121] A determination submodule, configured to determine a motion vector of the target object based on the real-time environment image and the reference image if feature matching is successful;
[0122] The estimation submodule is used to estimate the motion path of the target object based on the motion vector.
[0123] Optionally, in some embodiments of the present application, the matching submodule includes:
[0124] A first recognition unit, used for recognizing a first key feature point of the real-time environment image and extracting a first descriptor of the first key feature point;
[0125] A second recognition unit, used for recognizing a second key feature point of the reference image and extracting a second descriptor of the second key feature point;
[0126] A matching unit is used to perform feature matching on the first descriptor and the second descriptor.
[0127] Optionally, in some embodiments of the present application, the first generating module 203 includes:
[0128] A first generating submodule, configured to generate the obstacle avoidance strategy based on the motion path if the distance in the obstacle distance information is greater than a preset distance threshold;
[0129] The second generating submodule is used to generate the obstacle avoidance strategy based on the distance if the distance in the obstacle distance information is less than or equal to a preset distance threshold.
[0130] Optionally, in some embodiments of the present application, the drone obstacle avoidance device further includes a preprocessing module, and the preprocessing module is specifically used to:
[0131] A preprocessing operation is performed on the real-time environment image.
[0132] Optionally, in some embodiments of the present application, the drone obstacle avoidance device further includes:
[0133] A second generating module, configured to generate the obstacle avoidance strategy based on the distance if it is detected that the target object does not exist in the real-time environment image and the distance in the obstacle distance information is less than or equal to a preset distance threshold;
[0134] The execution module is used to continue to acquire the real-time environment image of the drone if it is detected that the target object does not exist in the real-time environment image and the distance in the obstacle distance information is greater than the preset distance threshold.
[0135] Optionally, in some embodiments of the present application, the drone obstacle avoidance device further includes:
[0136] A detection module, used to perform feature point detection on the real-time environment image to obtain feature point data;
[0137] A classification module, used to classify the feature point data to obtain a grouping result corresponding to each target object;
[0138] A division module, used for performing spatial division on the real-time environment image to obtain a spatial division result;
[0139] A determination module, configured to determine the position information of the target object in the real-time environment image based on the grouping result and the space division result;
[0140] The third generating module is used to generate an obstacle avoidance strategy based on the position information of each target object.
[0141] In one embodiment, a drone is provided, the internal structure diagram of which can be as follows: Figure 6As shown. The drone includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the drone is used to provide computing and control capabilities. The memory of the drone includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the drone is used to communicate with an external server through a network connection. When the computer program is executed by the processor, the functions or steps of a drone obstacle avoidance method are implemented.
[0142] In one embodiment, a drone is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:
[0143] Acquire the real-time environment image and obstacle distance information of the UAV, where the real-time environment image includes at least one target object; estimate the motion path of the target object based on the real-time environment image and a reference image, where the reference image is the real-time environment image of the UAV at the previous moment; generate an obstacle avoidance strategy based on the motion path and obstacle distance information; and control the UAV to avoid obstacles based on the obstacle avoidance strategy.
[0144] This embodiment estimates the motion path of the target object through the real-time environment image and the reference image, generates an obstacle avoidance strategy by combining the obstacle distance information and the motion path, and thus controls the drone to avoid obstacles. It can be seen that by combining the obstacle distance information and the motion path to generate an obstacle avoidance strategy, the environment and obstacle distance can be monitored in real time, so that the drone can fly purposefully to avoid obstacles, ensure the safety of itself and its environment, and help improve the obstacle avoidance flexibility, autonomy and intelligence of the drone, and improve its flight safety.
[0145] In one embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:
[0146] Acquire the real-time environment image and obstacle distance information of the UAV, where the real-time environment image includes at least one target object; estimate the motion path of the target object based on the real-time environment image and a reference image, where the reference image is the real-time environment image of the UAV at the previous moment; generate an obstacle avoidance strategy based on the motion path and obstacle distance information; and control the UAV to avoid obstacles based on the obstacle avoidance strategy.
[0147] This embodiment estimates the motion path of the target object through the real-time environment image and the reference image, generates an obstacle avoidance strategy by combining the obstacle distance information and the motion path, and thus controls the drone to avoid obstacles. It can be seen that by combining the obstacle distance information and the motion path to generate an obstacle avoidance strategy, the environment and obstacle distance can be monitored in real time, so that the drone can fly purposefully to avoid obstacles, ensure the safety of itself and its environment, and help improve the obstacle avoidance flexibility, autonomy and intelligence of the drone, and improve its flight safety.
[0148] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or the drone can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.
[0149] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0150] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0151] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.
Claims
1. A method for avoiding obstacles in a drone, characterized in that: include: Acquire a real-time environment image and obstacle distance information of the UAV, wherein the real-time environment image includes at least one target object; estimating a motion path of the target object based on the real-time environment image and a reference image, wherein the reference image is a real-time environment image of the UAV at a previous moment; generating an obstacle avoidance strategy based on the motion path and the obstacle distance information; The drone is controlled to avoid obstacles based on the obstacle avoidance strategy.
2. The unmanned aerial vehicle obstacle avoidance method according to claim 1, characterized in that: The estimating the motion path of the target object based on the real-time environment image and the reference image comprises: Performing feature matching on the real-time environment image and the reference image; If the feature matching is successful, determining the motion vector of the target object based on the real-time environment image and the reference image; A motion path of the target object is estimated based on the motion vector.
3. The unmanned aerial vehicle obstacle avoidance method according to claim 2, characterized in that: The performing feature matching on the real-time environment image and the reference image comprises: Identifying a first key feature point of the real-time environment image and extracting a first descriptor of the first key feature point; Identifying a second key feature point of the reference image and extracting a second descriptor of the second key feature point; Perform feature matching on the first descriptor and the second descriptor.
4. The unmanned aerial vehicle obstacle avoidance method according to claim 1, characterized in that: The generating of the obstacle avoidance strategy based on the motion path and the obstacle distance information includes: If the distance in the obstacle distance information is greater than a preset distance threshold, generating the obstacle avoidance strategy based on the motion path; If the distance in the obstacle distance information is less than or equal to a preset distance threshold, the obstacle avoidance strategy is generated based on the distance.
5. The unmanned aerial vehicle obstacle avoidance method according to claim 1, characterized in that: Before estimating the motion path of the target object based on the real-time environment image and the reference image, the method further includes: A preprocessing operation is performed on the real-time environment image.
6. The unmanned aerial vehicle obstacle avoidance method according to claim 1, characterized in that: Also includes: If it is detected that the target object does not exist in the real-time environment image, and the distance in the obstacle distance information is less than or equal to a preset distance threshold, generating the obstacle avoidance strategy based on the distance; If it is detected that the target object does not exist in the real-time environment image and the distance in the obstacle distance information is greater than the preset distance threshold, the real-time environment image of the drone is acquired.
7. The unmanned aerial vehicle obstacle avoidance method according to claim 1, characterized in that: Also includes: Performing feature point detection on the real-time environment image to obtain feature point data; Classifying the feature point data to obtain a grouping result corresponding to each target object; Performing spatial division on the real-time environment image to obtain a spatial division result; Determine the position information of the target object in the real-time environment image based on the grouping result and the space division result; An obstacle avoidance strategy is generated based on the position information of each of the target objects.
8. An obstacle avoidance device for a drone, characterized in that: include: An acquisition module, used to acquire a real-time environment image and obstacle distance information of the UAV, wherein the real-time environment image includes at least one target object; An estimation module, used for estimating the motion path of the target object based on the real-time environment image and a reference image, wherein the reference image is the real-time environment image of the UAV at the previous moment; A first generating module, configured to generate an obstacle avoidance strategy based on the motion path and the obstacle distance information; A control module is used to control the drone to avoid obstacles based on the obstacle avoidance strategy.
9. A drone comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the drone obstacle avoidance method as described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the drone obstacle avoidance method as described in any one of claims 1 to 7 are implemented.
