UAV collision prediction and obstacle avoidance method and related equipment
Through optical flow method, the drone image information is analyzed, the collision risk is predicted and the attitude avoidance is adjusted, the data lag and safety hazards in the drone obstacle avoidance technology are solved, and the intelligent safety improvement of drone flight is achieved .
Patent Information
- Application Number
- CN202410663101.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-27
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-05-27
AI Technical Summary
The existing UAV obstacle avoidance technology has data transmission lag and wireless transmission link instability, resulting in high security risks and lack of the ability to intelligently identify and deal with obstacle collision risks.
The optical flow method is used to preprocess the real-time image information collected by the drone, analyze the relative distance and speed of the target object, predict the collision risk by calculating the optical flow trajectory, and adjust the drone's posture according to the optical flow vector information to perform obstacle avoidance actions. .
It effectively improves the safety of drone flight, realizes intelligent crisis response capabilities, and solves the problem of lack of intelligent identification and response to obstacle collision risks during drone flight.
Smart Images

Figure CN118466575B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of unmanned aerial vehicles, and in particular to a method for predicting and avoiding collisions of unmanned aerial vehicles and related equipment. Background Art
[0002] As an efficient and flexible line inspection tool, drones are widely used in commercial and civilian fields. In urban application scenarios, the number of drones and the presence of obstacles such as buildings, pedestrians and vehicles are related to safety, and the anti-collision technology of drones has become one of the research hotspots.
[0003] In traditional technology, the drone sends the acquired image information to the ground, and then the operator performs obstacle avoidance control based on the real-time image information received. On the one hand, this method is prone to lag in data transmission and flight control, making it impossible for users to receive real-time image information, or to control the flight action of the drone in real time. In addition, the instability of the wireless transmission link also has a great impact on the entire obstacle avoidance method. Therefore. How to solve the safety hazards of drone obstacle avoidance in the existing technology is very important. Summary of the invention
[0004] The main purpose of the present invention is to provide a UAV collision prediction and obstacle avoidance method and related equipment, which are used to solve the technical problem of potential safety hazards in UAV obstacle avoidance in the prior art.
[0005] In a first aspect, the present invention provides a UAV collision prediction method, which is applied to a UAV, comprising:
[0006] Acquire the real-time image information collected by the drone, pre-process the real-time image information, and generate the image information to be detected;
[0007] Analyze the target objects in the image information to be detected, and use the optical flow method to identify and track each target object;
[0008] Obtain the real-time relative distance information between each target object and the drone;
[0009] If the real-time relative distance of the target object is less than the first preset value, the relative speed information between the target object and the drone is obtained;
[0010] If the relative speed of the target object is less than a second preset value, obtaining the optical flow trajectory of the target object;
[0011] Based on each optical flow trajectory, predict whether the drone has a collision risk.
[0012] Preferably, if the real-time relative distance of the target object is less than the first preset value, the step of obtaining the relative speed information between the target object and the drone includes:
[0013] The formula for calculating relative speed information is:
[0014]
[0015] Among them, V t is the relative speed between the target object and the UAV, S t S is the first depth information of the target object calculated by the drone at the current moment; t-1 is the second depth information of the target object calculated by the drone at the previous moment, and Δt is the time difference between the current moment and the previous moment.
[0016] Preferably, if the relative speed of the target object is less than a second preset value, the step of acquiring the optical flow trajectory of the target object includes:
[0017] The formula for calculating the optical flow vector of the target object is:
[0018]
[0019] in, Represents the optical flow vector of the center point of the target object from time t-1 to time t, (x t ,y t ) is the pixel coordinate of the center point of the target object at the current moment, (x t-1 ,y t-1 ) is the pixel coordinate of the center point of the target object at the previous moment.
[0020] Preferably, the step of predicting whether the drone has a collision risk according to each optical flow trajectory includes:
[0021] By calculating the angle between the optical flow vector of the target object and the normal direction of the coordinate system quadrant where the center point of the target object is located at the current moment, it is determined whether the target object has a velocity component toward the direction of the drone, where the angle is far away from the origin of the coordinate axis.
[0022] Preferably, the step of determining whether the target object has a velocity component in the direction of the drone includes:
[0023] By calculating the optical flow vector of the target object, the angle between the optical flow vector and the normal direction of the coordinate system quadrant where the center point of the target object is located at the current moment is calculated as follows:
[0024]
[0025] in, is the optical flow vector of the target object, and the angle between it and the normal direction of the coordinate system quadrant where the center point of the target object is located at the current moment, ρ is the normal vector, is the optical flow vector of the target object at the current moment.
[0026] Preferably, the step of determining whether the target object has a velocity component in the direction of the drone includes:
[0027] when When it is less than 90°, it is considered that the target object has no relative velocity component toward the UAV;
[0028] or,
[0029] when When it is greater than 90°, it is considered that the target object has a relative velocity component toward the UAV.
[0030] Preferably, the steps of obtaining real-time image information collected by the drone and preprocessing the real-time image information to generate image information to be detected include:
[0031] Get camera video stream information;
[0032] Decode the original video into a single-frame image in a unified RGB format;
[0033] Perform color space conversion and image filtering and denoising on single-frame images to improve image quality.
[0034] In a second aspect, the present invention provides a method for avoiding obstacles of a UAV, applying the above-mentioned method for predicting collision of a UAV, comprising:
[0035] Obtain the ID information and optical flow vector information of the target object, and adjust the drone's posture according to the optical flow vector information.
[0036] Preferably, after the step of adjusting the posture of the drone according to the optical flow vector information, the following steps are included:
[0037] After the drone has finished adjusting its attitude, continue with the drone collision prediction;
[0038] Repeat the above steps until the drone stops working.
[0039] In a third aspect, the present invention provides a drone, which executes the above method, including:
[0040] Video acquisition module, used to collect video information obtained by drones in real time;
[0041] The target detection and tracking module is used to detect and track the target objects in the video;
[0042] The distance judgment module is used to judge the relative distance between each target object and the drone;
[0043] An optical flow trajectory judgment module is used to judge the optical flow trajectory of each target object;
[0044] The collision and obstacle avoidance module is used for the drone to perform obstacle avoidance actions.
[0045] In summary, the beneficial effects of the present invention are as follows: in a drone device, by calculating the distance between the target object and the drone and analyzing the optical flow trajectory of the target object, it is predicted whether the drone has a collision risk. When the drone has a collision risk, the obstacle target optical flow trajectory is inferred to guide the drone to avoid the obstacle. This can effectively improve the safety of drone flight and solve the problem of lack of intelligent identification and response to obstacle collision risks during drone flight. The intelligent crisis response capability of drone flight is realized. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 It is a schematic flow chart of a first embodiment of a method for predicting collision of a drone according to the present invention;
[0047] Figure 2 for Figure 1 Schematic diagram of the drone acquiring the target object;
[0048] Figure 3 for Figure 1 Schematic diagram of optical flow trajectory judgment;
[0049] Figure 4 A schematic diagram of an obstacle avoidance calculation process of a first embodiment of a method for avoiding obstacles in a drone according to the present invention;
[0050] Figure 5 It is a schematic diagram of the structure of the drone of the present invention;
[0051] Figure 6 The figure is a schematic diagram of the hardware structure of a UAV of the present invention.
[0052] Description of labels:
[0053] 1. Data acquisition module; 2. Data processing module; 3. Training module; 4. Prediction module;
[0054] 310, bus; 301, processor; 302, memory; 303, communication interface.
[0055] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0056] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0057] Combination Figure 1 and 2 The present invention provides a UAV collision prediction method, which is applied to a UAV, comprising:
[0058] S1: Obtain the real-time image information collected by the drone, pre-process the real-time image information, and generate the image information to be detected;
[0059] S2: Analyze the target objects in the image information to be detected, and use the optical flow method to identify and track each target object;
[0060] S3: Obtain the real-time relative distance information between each target object and the drone;
[0061] S4: If the real-time relative distance of the target object is less than the first preset value, the relative speed information between the target object and the drone is obtained;
[0062] S5: If the relative speed of the target object is less than a second preset value, obtaining the optical flow trajectory of the target object;
[0063] S6: Based on each optical flow trajectory, predict whether the drone has a collision risk.
[0064] In an embodiment of the present invention, the drone has edge computing capabilities and obtains real-time image information through image sensors, which include but are not limited to monocular cameras and binocular cameras. The drone preprocesses the real-time image information so that the image data is easier to use for detection and analysis, and generates image information to be detected after the processing is completed. The drone trains a high-precision high-altitude flight target monitoring model through the Yo lov5 algorithm, takes the center point of the camera screen as the origin of the pixel coordinate system, and outputs the coordinate information of each target frame, wherein each target frame locks at least one target object. According to the coordinates of the upper left corner and the lower right corner of each target frame, the coordinates of the center point of the target object are calculated. Since the ambient light brightness changes little during the flight of the drone, the optical flow method is used to track the center point of the target object with high accuracy, and the interference of the sky background can be eliminated, and each target object is given a unique identifier U ID until it leaves the field of view of the drone. The drone tracks the center point of each target object and continuously records the pixel coordinates of the center point of each target object. In an embodiment of the present invention, the drone uses a binocular camera for image acquisition, and calculates the depth information of the center point of the target object (i.e., the relative distance information between the target object and the drone) based on the binocular vision algorithm. When the relative distance between the target object and the drone is less than the warning distance, the drone triggers a collision warning and obtains the relative speed of the target object. When the relative speed is lower than the second preset value, it proves that the relative distance between the target object and the drone is shortening, and there is a possibility of collision. The drone obtains the UID of the target object and the pixel trajectory information of the center point of the target object to predict whether there is a possibility of collision. In summary, in the drone equipment, deep learning technology is used to process the binocular camera video stream in real time, and the distance between the target object and the drone is calculated, and the method of analyzing the optical flow trajectory of the target object is used to predict whether the drone has a collision risk. This can effectively improve the flight safety of the drone and solve the problem of lack of intelligent identification and response to obstacle collision risks during the flight of the drone. Realize the intelligent crisis response capability of drone flight.
[0065] Furthermore, if the real-time relative distance of the target object is less than the first preset value, the step S4 of obtaining the relative speed information between the target object and the drone includes:
[0066] The formula for calculating relative speed information is:
[0067]
[0068] Among them, V t is the relative speed between the target object and the UAV, S t S is the first depth information of the target object calculated by the drone at the current moment; t-1is the second depth information of the target object calculated by the drone at the previous moment, and Δt is the time difference between the current moment and the previous moment.
[0069] In the embodiment of the present invention, when V t When V > 0, it means the target object is moving away from the drone and the collision warning is canceled. t When <0, it means that the target object is moving towards the drone and there is a possibility of collision with the drone. The drone obtains the U ID of the target object and the pixel trajectory information of the center point of the target object to predict whether there is a possibility of collision.
[0070] according to Figure 3 If the relative speed of the target object is less than the second preset value, the step S5 of acquiring the optical flow trajectory of the target object comprises:
[0071] The formula for calculating the optical flow vector of the target object is:
[0072]
[0073] in, Represents the optical flow vector of the center point of the target object from time t-1 to time t, (x t ,y t ) is the pixel coordinate of the center point of the target object at the current moment, (x t-1 ,y t-1 ) is the pixel coordinate of the center point of the target object at the previous moment.
[0074] In an embodiment of the present invention, by calculating the angle between the optical flow vector of the target object and the normal direction of the coordinate system quadrant where the center point of the target object is located at the current moment, it is determined whether the target object has a velocity component toward the direction of the drone, wherein the angle is far away from the origin of the coordinate axis.
[0075] Specifically, step S51 of determining whether the target object has a velocity component in the direction of the drone includes:
[0076] By calculating the optical flow vector of the target object, the angle between the optical flow vector and the normal direction of the coordinate system quadrant where the center point of the target object is located at the current moment is calculated as follows:
[0077]
[0078] in, is the angle between the optical flow vector of the target object and the normal direction of the coordinate system quadrant where the center point of the target object is located at the current moment. is the normal vector, is the optical flow vector of the target object at the current moment.
[0079] In an embodiment of the present invention, The value is (1,1). Taking the value as a unit vector helps calculate the angle when When it is less than 90°, it is considered that the target object has no relative velocity component toward the drone. When it is greater than 90°, it is considered that the target object has a relative velocity component toward the UAV.
[0080] Furthermore, the step S1 of obtaining real-time image information collected by the drone and preprocessing the real-time image information to generate image information to be detected includes:
[0081] S11: Get camera video stream information;
[0082] S12: Decode the original video into a single frame image in a unified RGB format;
[0083] S13: Perform color space conversion and image filtering and denoising processing on the single frame image to improve the image quality.
[0084] In an embodiment of the present invention, a binocular camera is installed at the head of the drone, and the observation area is the forward direction of the drone. First, the drone accesses the camera video stream information, connects the camera and the airborne edge computing platform through the internal Ethernet network of the drone, and the edge computing platform obtains the real-time video stream information collected by the camera through the RTSP video stream address. Then, the drone decodes the original video into a single-frame image in a unified RGB format. Then, the single-frame image is converted in color space and image filtering and denoising is performed, so as to achieve the purpose of improving the image and facilitate the subsequent drone module to perform the next step of processing the image information.
[0085] Reference Figure 4 The present invention also provides a method for avoiding obstacles of unmanned aerial vehicles, which applies the above-mentioned method for predicting collision of unmanned aerial vehicles, and comprises:
[0086] S6: Obtain the ID information and optical flow vector information of the target object, and adjust the posture of the drone according to the optical flow vector information.
[0087] In the embodiment of the present invention, the drone obtains the ID information and optical flow vector information of the target object, and adjusts the posture of the drone according to the optical flow vector information. Figure 4 As shown, The moving vector of the center point of the drone camera screen, the direction of this vector and the optical flow vector of the target object Vertical. For example, let the optical flow vector is (1,2), then the moving vector of the center point of the drone camera image is = (2,1). The specific obstacle avoidance method is as follows: Calculate the moving vector of the center point of the drone camera image (x,y) guides the drone to adjust its attitude, where the x value corresponds to adjusting the drone's heading angle, and the y value corresponds to adjusting the drone's pitch angle. Here, each unit length of the pixel coordinate axis of the drone's onboard camera center point corresponds to 5° of the coordinate system in the inertial navigation system (this value can be adjusted according to actual conditions). The modulus of the target object to the drone's flight speed is used as the modulus of the drone's obstacle avoidance speed, and the direction of movement is the forward direction of the drone after adjusting its attitude. Then the drone moves a preset distance, which is one-fifth of the relative distance between the target object and the drone (the moving distance can be adjusted according to actual conditions).
[0088] Further, after step S6 of adjusting the posture of the drone according to the optical flow vector information, the following steps are included:
[0089] S61: After the UAV has finished adjusting its attitude, continue with the UAV collision prediction;
[0090] S62: The above steps are executed repeatedly until the drone stops working.
[0091] In an embodiment of the present invention, the drone continues to predict collisions and avoid obstacles during its movement until the drone stops working, thereby ensuring the safety of the drone and its surrounding environment.
[0092] Reference Figure 5 The present invention provides a drone for executing the above method, including:
[0093] Video acquisition module 1, used to collect video information obtained by the drone in real time;
[0094] Target detection and tracking module 2, used to detect and track target objects in the video;
[0095] The distance judgment module 3 is used to judge the relative distance between the target object and the drone;
[0096] An optical flow trajectory determination module 4 is used to determine the optical flow trajectory of each target object;
[0097] The collision and obstacle avoidance module 5 is used for the drone to perform obstacle avoidance actions.
[0098] In an embodiment of the present invention, the drone has edge computing capabilities, and the video acquisition module 1 obtains real-time image information through an image sensor, which includes but is not limited to a monocular camera and a binocular camera. The drone preprocesses the real-time image information so that the image data is easier to use for detection and analysis, and generates image information to be detected after the processing is completed. The drone uses the Yo l ov5 algorithm to train a high-precision high-altitude flight target monitoring model through the target detection and tracking module 2, takes the center point of the camera screen as the origin of the pixel coordinate system, and outputs the coordinate information of each target frame, wherein each target frame locks at least one target object. According to the coordinates of the upper left corner and the lower right corner of each target frame, the coordinates of the center point of the target object are calculated. Since the ambient light brightness changes little during the flight of the drone, the optical flow method is used to track the center point of the target object with high accuracy, and the interference of the sky background can be eliminated, and each target object is given a unique identifier UID until it leaves the field of view of the drone. The drone tracks the center point of each target object and continuously records the pixel coordinates of the center point of each target object. In an embodiment of the present invention, the drone uses a binocular camera for image acquisition, and uses a distance judgment module 3 to calculate the depth information of the center point of the target object (i.e., the relative distance information between the target object and the drone) based on a binocular vision algorithm. When the relative distance between the target object and the drone is less than the warning distance, the drone triggers a collision warning and obtains the relative speed of the target object. When the relative speed is lower than the second preset value, it proves that the relative distance between the target object and the drone is shortening, and there is a possibility of collision. The drone obtains the UI of the target object and the pixel trajectory information of the center point of the target object, and predicts whether there is a possibility of collision through the optical flow trajectory judgment module 4. If there is a possibility of collision, the collision avoidance module 5 is used to perform the obstacle avoidance action. In summary, in the drone device, deep learning technology is used to process the binocular camera video stream in real time, and the distance between the target object and the drone is calculated, and the method of analyzing the optical flow trajectory of the target object is used to predict whether the drone has a collision risk. When the drone is at risk of collision, the drone is guided to avoid obstacles by inferring the target optical flow trajectory of the obstacle. This can effectively improve the flight safety of the drone and solve the problem of lack of intelligent identification and response to the risk of obstacle collision during the flight of the drone. The intelligent crisis response capability of drone flight is realized.
[0099] In addition, combined Figure 6 The drone collision prediction and obstacle avoidance method described in the embodiment of the present invention can be implemented by a drone. Figure 6 A schematic diagram of the hardware structure of a drone provided in an embodiment of the present invention is shown.
[0100] The drone may include a processor and a memory storing computer program instructions.
[0101] Specifically, the processor may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present invention.
[0102] The memory 302 may include a large capacity memory for data or instructions. For example, but not limitation, the memory may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus 310 (USB) drive or a combination of two or more of these. In appropriate cases, the memory 302 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 302 may be inside or outside the data processing device. In a specific embodiment, the memory 302 is a non-volatile solid-state memory. In a specific embodiment, the memory 302 includes a read-only memory (ROM). In appropriate cases, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM) or a flash memory or a combination of two or more of these.
[0103] The processor 301 implements any one of the methods in the above embodiments by reading and executing computer program instructions stored in the memory.
[0104] In one example, the drone may further include a communication interface 303 and a bus 310. Figure 6 As shown, the processor, memory, and communication interface are connected via bus 310 and communicate with each other.
[0105] The communication interface is mainly used to implement communication between the modules, devices, units and / or equipment in the embodiments of the present invention.
[0106] The bus 310 includes hardware, software or both, coupling the components of the electronic device to each other. By way of example and not limitation, the bus 310 may include an accelerated graphics port (AGP) or other graphics bus 310, an enhanced industry standard architecture (EISA) bus 310, a front side bus 310 (FSB), a hypertransport (HT) interconnect, an industry standard architecture (ISA) bus 310, an infinite bandwidth interconnect, a low pin count (LPC) bus 310, a memory bus 310, a microchannel architecture (MCA) bus 310, a peripheral component interconnect (PCI) bus 310, a PCI-Express (PCI-X) bus 310, a serial advanced technology attachment (SATA) bus 310, a video electronics standard association local (VLB) bus 310 or other suitable bus 310 or a combination of two or more of these. Where appropriate, the bus 310 may include one or more buses 310. Although the embodiments of the present invention describe and illustrate a specific bus 310, the present invention contemplates any suitable bus 310 or interconnect.
[0107] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A UAV collision prediction method, applied to UAVs, characterized in that: include: Acquire real-time image information collected by the drone, and pre-process the real-time image information to generate image information to be detected; Analyze the target objects in the image information to be detected, and use the optical flow method to identify and track each of the target objects; Acquiring real-time relative distance information between each target object and the drone; If the real-time relative distance of the target object is less than a first preset value, the relative speed information of the target object and the drone is obtained; If the relative speed of the target object is less than a second preset value, obtaining the optical flow trajectory of the target object; According to each of the optical flow trajectories, predict whether the drone has a collision risk, Wherein, if the relative speed of the target object is less than a second preset value, the step of obtaining the optical flow trajectory of the target object includes: The formula for calculating the optical flow vector of the target object is: in, Represents the optical flow vector of the center point of the target object from time t-1 to time t, (x t ,y t ) is the pixel coordinate of the center point of the target object at the current moment, (x t-1 ,y t-1 ) is the pixel coordinate of the center point of the target object at the previous moment; The step of predicting whether the drone has a collision risk according to each of the optical flow trajectories includes: By calculating the angle between the optical flow vector of the target object and the normal direction of the coordinate system quadrant where the center point of the target object is located at the current moment, it is determined whether the target object has a velocity component toward the direction of the drone, wherein the angle is away from the origin of the coordinate axis; Wherein, the step of determining whether the target object has a velocity component in the direction of the drone includes: The calculation formula for calculating the angle between the optical flow vector of the target object and the normal direction of the coordinate system quadrant where the center point of the target object is located at the current moment is: in, is the angle between the optical flow vector of the target object and the normal direction of the coordinate system quadrant where the center point of the target object is located at the current moment, is the normal vector, is the optical flow vector of the target object at the current moment.
2. The UAV collision prediction method according to claim 1, characterized in that: If the real-time relative distance of the target object is less than a first preset value, the step of obtaining the relative speed information between the target object and the drone includes: The formula for calculating the relative speed information is: Among them, V t is the relative speed between the target object and the UAV, S t S is the first depth information of the target object calculated by the drone at the current moment; t-1 is the second depth information of the target object calculated by the drone at the previous moment, is the time difference between the current moment and the previous moment.
3. The UAV collision prediction method according to claim 1, characterized in that: The step of determining whether the target object has a velocity component in the direction of the drone comprises: when When it is less than 90°, it is considered that the target object has no relative velocity component toward the UAV; or, when When it is greater than 90°, it is considered that the target object has a relative velocity component toward the direction of the UAV.
4. The method for predicting a UAV collision according to claim 3, characterized in that: The step of acquiring real-time image information collected by the drone and preprocessing the real-time image information to generate image information to be detected includes: Get camera video stream information; Decode the original video into a single-frame image in a unified RGB format; The single-frame image is subjected to color space conversion and image filtering and denoising processing to improve image quality.
5. A drone obstacle avoidance method, using the drone collision prediction method according to any one of claims 1 to 4, characterized in that: include: The ID information and optical flow vector information of the target object are obtained, and the posture of the drone is adjusted according to the optical flow vector information.
6. The unmanned aerial vehicle obstacle avoidance method according to claim 5, characterized in that: After the step of adjusting the posture of the drone according to the optical flow vector information, the method further comprises: After the UAV has completed the posture adjustment, the UAV collision prediction is continued; The above steps are executed repeatedly until the drone stops working.
7. A drone, executing the method according to any one of claims 1 to 6, characterized in that: include: Video acquisition module, used to collect video information obtained by drones in real time; The target detection and tracking module is used to detect and track the target objects in the video; A distance determination module, used to determine the relative distance between each target object and the drone; An optical flow trajectory judgment module, used to judge the optical flow trajectory of each target object; The collision and obstacle avoidance module is used for the drone to perform obstacle avoidance actions.
