Image-based unmanned aerial vehicle autonomous line-imitating flight system and method
Through the image-based autonomous imitation line flight system of drone, using depth cameras and deep learning technology to identify and plan three-dimensional models of power lines in real time, it solves the difficulty of traditional drone flight operations to achieve high precision and high efficiency in complex environments. The problem of power line inspection achieves high accuracy and high efficiency real-time imitation line flight control.
Patent Information
- Application Number
- CN202411543252.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-31
- Publication Date
- 2025-02-14
AI Technical Summary
Traditional drone flight operations are difficult to achieve high-precision and efficient power line inspection in complex and changeable natural environments. Especially in complex terrain areas such as mountainous areas and forest areas, it is difficult to accurately fit the route with artificially set flight paths. Direction, and the line offset or damage caused by weather and other factors cannot be updated in real time.
Using an image-based drone autonomous imitation line flight system, the image data with depth information is collected through the depth camera, the CNN model and the fast Hough transformation algorithm are used to extract keyframes in real time, identify power line targets, generate a three-dimensional model of power line, and Perform imitation line path planning and control instructions to realize real-time imitation line flight control.
Real-time imitation line flight control based on image is realized, with high control accuracy, can quickly capture linear features, efficiently identify power lines, meet the needs of real-time imitation line control, and have high control efficiency.
Smart Images
Figure CN119440044A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of unmanned aerial vehicles, and in particular to an image-based unmanned aerial vehicle autonomous line-following flight system and method. Background Art
[0002] The rapid development of drone technology is leading the transformation and innovation of many industries. Especially in operating environments that require high precision, high efficiency, and high safety, drones have become an indispensable tool with their unique advantages, such as high flexibility, wide coverage, and low operating costs. However, traditional drone flight operations mostly rely on manual remote control or preset GPS paths, which expose many limitations when facing complex and changeable natural environments and dynamically changing mission requirements. In power line inspections, manually set flight paths are often difficult to accurately fit the direction of the line, especially in complex terrains such as mountainous areas and forest areas. The lines often pass through dense forests and cross rivers, making manual planning extremely difficult. In addition, power lines may also be offset or damaged due to weather, external forces, and the manual paths cannot be updated in real time, resulting in poor inspection results.
[0003] Image processing technology is the key to realize autonomous line patrol and line-following flight of UAVs. Traditional image processing technology mainly relies on feature extraction and matching algorithms, such as SIFT and SURF. These algorithms perform well when processing clear and stable images, but their stability and accuracy will be seriously affected in complex and changeable natural environments, such as illumination changes, shadow interference, and the presence of obstructions. Therefore, in recent years, with the rapid development of deep learning technology, image recognition and segmentation technology based on convolutional neural network (CNN) has gradually become mainstream. CNN can automatically learn features in images and accurately recognize and understand complex environments through a large amount of training data. In autonomous flight of UAVs, CNN is used to identify targets on the ground or in the air, such as wires, road edges, farmland boundaries, etc., to generate high-precision environmental maps. However, although CNN can accurately identify objects such as wires, its processing efficiency is low and its target tracking performance is poor. It cannot meet the requirements of autonomous line patrol.
[0004] In order to solve these problems, the industry has begun to explore image-based drone autonomous line patrol and autonomous line-following flight technologies, but has not yet achieved good results. Summary of the invention
[0005] The present invention aims to provide an image-based autonomous line-following flight system and method for unmanned aerial vehicles, which can realize image-based line-following flight control with high control efficiency and precise control.
[0006] To achieve the above objectives, the present invention provides the following basic solutions.
[0007] Solution 1
[0008] An image-based autonomous line-following flight system for unmanned aerial vehicles comprises an unmanned aerial vehicle, wherein the unmanned aerial vehicle is provided with a satellite positioning module and a communication module, wherein the satellite positioning module is used to capture the position information of the unmanned aerial vehicle in real time; the unmanned aerial vehicle is also equipped with a flight control module and an image module; the image module is provided with an image acquisition unit and an image processing unit; the image acquisition unit is used to acquire image data with depth information; the image processing unit is used to extract key frames from the image data, identify power line targets from the key frames, calculate the power line coordinates, and generate a three-dimensional model of the power line;
[0009] The flight control module includes a line-simulating path planning unit and a control unit; the line-simulating path planning unit is used to match the UAV's posture position information with the three-dimensional model of the power line and generate line-simulating planning waypoints; the control unit is used to compare the preset waypoints of the UAV route with the line-simulating planning waypoints and modify the UAV cruise action instructions.
[0010] Furthermore, the image acquisition unit adopts a depth camera module.
[0011] Furthermore, the key frame is an image frame containing power line related features.
[0012] Furthermore, an image matching model is preset in the image processing unit; the image matching model is a CNN model; when extracting key frames, the image matching model extracts based on a fuzzy matching principle.
[0013] Furthermore, when the image processing unit identifies the power line target from the key frame, it performs the identification based on the fast Hough transform algorithm; by performing Hough transform on the target object in the key frame, the fitting direction and target range of the individual linear target are calculated, and the power line target is fitted.
[0014] Furthermore, the line-simulating path planning unit first fits the current flight route of the UAV based on the posture and position information of the UAV; and fits the appearance boundary line of the UAV on the current flight route based on the appearance data of the UAV; and matches the current flight route and the appearance boundary line with the three-dimensional model of the power line to calculate the safety interval value between the current flight route, the appearance boundary line and the power line; and derives the line-simulating planning waypoints based on the safety interval value.
[0015] Furthermore, the drone cruise action instructions include hovering, moving forward, moving backward, moving left, moving right, rising, falling, and rotating.
[0016] Solution 2
[0017] An image-based UAV autonomous line-following flight method, using an image-based UAV autonomous line-following flight system as described in Scheme 1 to perform UAV autonomous line-following flight control; comprising the following steps:
[0018] The satellite positioning module captures the position information of the drone in real time; the image acquisition unit in the image module collects image data with depth information; the image processing unit in the image module extracts key frames from the image data, identifies the power line target from the key frames, calculates the power line coordinates, and generates a three-dimensional model of the power line;
[0019] The line-mimicking path planning unit in the flight control module matches the UAV's position information with the three-dimensional model of the power line and generates line-mimicking waypoints. The control unit in the flight control module compares the preset waypoints of the UAV route with the line-mimicking waypoints and modifies the UAV's cruise action instructions.
[0020] The working principle and advantages of the present invention are:
[0021] The present invention discloses an autonomous line-following flight system and method for unmanned aerial vehicles based on images. The system can recognize and analyze power lines by collecting image data with depth information in real time, extract key frames by seconds to generate a three-dimensional model of power lines, and instantly perform line-following waypoint planning and control instruction modification. The system can realize real-time line-following flight control based on images. Moreover, the system performs point-to-point line-following control based on waypoints in real time, and has high control accuracy. In addition, the system recognizes power lines based on key frames and fast Hough transform algorithms. Compared with traditional image recognition algorithms or point cloud recognition algorithms, the system has less calculation and more targeted detection. It can quickly capture linear features and efficiently identify power lines, meet the needs of real-time line-following control, and has high control efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 The present invention is a schematic structural diagram of a first embodiment of an image-based autonomous line-following flight system and method for unmanned aerial vehicles. DETAILED DESCRIPTION
[0023] The following is a further detailed description through specific implementation methods:
[0024] Embodiment 1
[0025] The embodiment is basically as shown in the attached Figure 1As shown: An image-based autonomous line-following flight system for unmanned aerial vehicles includes a unmanned aerial vehicle. The unmanned aerial vehicle is provided with a satellite positioning module and a communication module. The satellite positioning module is used to capture the position information of the unmanned aerial vehicle in real time. Here, the position information includes the flight speed, flight attitude information (including pitch angle, roll angle and heading angle) and other key parameters of the unmanned aerial vehicle. The communication module is used to receive and transmit data back.
[0026] The drone is also equipped with a flight control module and an imaging module.
[0027] Specifically, the image module is provided with an image acquisition unit and an image processing unit; the image acquisition unit is used to acquire image data with depth information; the image acquisition unit adopts a depth camera module.
[0028] The image processing unit is used to extract key frames from the image data, identify power line targets from the key frames, calculate the power line coordinates, and generate a three-dimensional model of the power line. The key frames are image frames containing power line related features. In addition, when extracting key frames, 10 frames are extracted as key frames per second, which can capture sufficient image information and the amount of data processed will not be too large.
[0029] Among them, the three-dimensional model of the power line can accurately reflect the position of the power line in the three-dimensional space, including the horizontal distance in the XOY direction and the vertical distance in the YOZ direction, as shown in the following formula:
[0030]
[0031] The image processing unit is preset with an image matching model; the image matching model is a CNN model; when extracting key frames, the image matching model extracts based on the fuzzy matching principle. After extracting the key frames, the image processing unit also performs pose estimation in combination with the drone's own IMU data to correct the geometric errors in the key frames. The CNN model is pre-trained using an existing image data set of power lines.
[0032] When the image processing unit identifies the power line target from the key frame, it performs the identification based on the fast Hough transform algorithm; by performing Hough transform on the target in the key frame, the fitting direction and target range of the individual linear target are calculated, and the power line target is fitted.
[0033] Before identifying the power line target from the key frame, the key frame is preprocessed, including grayscale, denoising and edge detection (in this embodiment, existing edge detection operators such as Canny, Sobel, Prewitt are selected to enhance the edge information in the image).
[0034] The flight control module includes a line-simulating path planning unit and a control unit; the line-simulating path planning unit is used to match the posture position information of the UAV with the three-dimensional model of the power line and generate line-simulating planning waypoints.
[0035] The line-simulating path planning unit first fits the current flight route of the UAV based on the UAV's posture and position information; and fits the UAV's appearance boundary line on the current flight route based on the UAV's appearance data; and based on the current flight route and the appearance boundary line, the three-dimensional model of the power line is matched to calculate the safety interval value between the current flight route, the appearance boundary line and the power line; and the line-simulating planning waypoints are derived based on the safety interval value.
[0036] When the three-dimensional model of the power line contains multiple power lines, the power line closest to the UAV is selected as the matching object.
[0037] The control unit is used to compare the preset waypoints of the drone route with the line-simulating planned waypoints, calculate the difference between the preset waypoints and the line-simulating planned waypoints, and optimize the path between the preset waypoints and the line-simulating planned waypoints. In this embodiment, the shortest path is taken as the final path obtained by optimization, and the drone cruise action instructions are modified according to the final path. The drone cruise action instructions include hovering, moving forward, moving backward, moving left, moving right, ascending, descending, and rotating.
[0038] Furthermore, when the control unit controls the UAV to execute the modified UAV cruise action command, it first controls to stop the previous UAV route planning, and adjusts the pitch, roll and other attitude information of the UAV to the default state, and then refreshes the cruise action command, imports the modified cruise action command and refreshes the cruise route, so that the UAV can fly along the new route (that is, fly along the route according to the waypoints planned by the route).
[0039] An image-based UAV autonomous line-following flight method uses the above-mentioned image-based UAV autonomous line-following flight system to perform UAV autonomous line-following flight control; comprising the following steps:
[0040] The satellite positioning module captures the position information of the drone in real time; the image acquisition unit in the image module collects image data with depth information; the image processing unit in the image module extracts key frames from the image data, identifies the power line target from the key frames, calculates the power line coordinates, and generates a three-dimensional model of the power line;
[0041] The line-mimicking path planning unit in the flight control module matches the UAV's position information with the three-dimensional model of the power line and generates line-mimicking waypoints. The control unit in the flight control module compares the preset waypoints of the UAV route with the line-mimicking waypoints and modifies the UAV's cruise action instructions.
[0042] The present embodiment provides an image-based autonomous line-following flight system and method for unmanned aerial vehicles, which can realize image-based line-following flight control with high control efficiency and precise control.
[0043] Embodiment 2
[0044] An image-based autonomous line-following flight system for unmanned aerial vehicles, based on the first embodiment, the unmanned aerial vehicle is further equipped with an environment perception module group; the environment perception module group includes a temperature and humidity sensor and a wind resistance sensor. The temperature and humidity sensor is used to sense temperature and humidity, and the wind resistance sensor is used to sense wind resistance value.
[0045] When modifying the cruise action instruction, the flight control module also adjusts the cruise action instruction of the drone according to the sensing data of the temperature and humidity sensor and the wind resistance sensor. Specifically, since the flight state of the drone is actually affected during flight (such as excessive wind resistance or in a rainy environment, the drone may not be able to stably maintain its flight attitude), the accuracy of the drone's line-flight is affected. Therefore, when performing flight control, the flight control module also keeps monitoring the sensing data. When there is a value in the sensing data that exceeds the corresponding preset threshold, the drone's posture error is checked according to the drone's posture position information. If the posture error is less than the corresponding threshold, the drone's cruise action instruction is not adjusted; otherwise, the parameters of the drone's cruise action instruction are corrected according to the posture difference every 2 seconds.
[0046] When calculating the posture stability rate, 2s is taken as a small cycle, and the posture data 2s ago is used as the reference posture. The difference between the current posture and the reference posture is calculated. 10s is taken as a large cycle, and the root mean square error of the posture change in the large cycle is calculated. The root mean square error is used as the posture error.
[0047] The present embodiment provides an image-based autonomous line-following flight system and method for UAVs. Compared with the first embodiment, the present embodiment additionally considers the impact of weather-related environmental factors on the flight attitude of the UAV and performs more detailed posture adjustment, thereby achieving higher line-following flight control accuracy.
[0048] The above is only an embodiment of the present invention. The common sense such as the known specific structure and characteristics in the scheme is not described in detail here. The ordinary technicians in the relevant field are aware of all the common technical knowledge in the technical field of the invention before the application date or priority date, can obtain all the existing technologies in the field, and have the ability to apply the conventional experimental means before that date. The ordinary technicians in the relevant field can improve and implement the scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for the ordinary technicians in the relevant field to implement this application. It should be pointed out that for the technicians in this field, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent.
Claims
1. An image-based UAV autonomous line-following flight system, comprising a UAV, wherein the UAV is provided with a satellite positioning module and a communication module, wherein: The satellite positioning module is used to capture the position information of the drone in real time; the drone is also equipped with a flight control module and an imaging module; the imaging module is provided with an image acquisition unit and an image processing unit; the image acquisition unit is used to acquire image data with depth information; the image processing unit is used to extract key frames from the image data, identify power line targets from the key frames, calculate the power line coordinates, and generate a three-dimensional model of the power line; The flight control module includes a line-simulating path planning unit and a control unit; the line-simulating path planning unit is used to match the UAV's posture position information with the three-dimensional model of the power line and generate line-simulating planning waypoints; the control unit is used to compare the preset waypoints of the UAV route with the line-simulating planning waypoints and modify the UAV cruise action instructions.
2. The image-based autonomous line-following flight system for unmanned aerial vehicles according to claim 1, characterized in that: The image acquisition unit adopts a depth camera module.
3. The image-based autonomous line-following flight system for unmanned aerial vehicles according to claim 1, characterized in that: The key frame is an image frame containing power line related features.
4. The image-based autonomous line-following flight system for unmanned aerial vehicles according to claim 3, characterized in that: An image matching model is preset in the image processing unit; the image matching model is a CNN model; when extracting key frames, the image matching model extracts based on a fuzzy matching principle.
5. The image-based autonomous line-following flight system for unmanned aerial vehicles according to claim 1, characterized in that: When the image processing unit identifies the power line target from the key frame, it performs the identification based on the fast Hough transform algorithm; by performing Hough transform on the target in the key frame, the fitting direction and target range of the individual linear target are calculated, and the power line target is fitted.
6. The image-based autonomous line-following flight system for unmanned aerial vehicles according to claim 1, characterized in that: The line-simulation path planning unit first obtains the current flight route of the UAV based on the position information of the UAV, and then obtains the outer shape boundary line of the UAV on the current flight route based on the outer shape data of the UAV. Based on the current flight route and the external boundary line, the three-dimensional model of the power line is matched to calculate the safety interval value between the current flight route, the external boundary line and the power line; based on the safety interval value, the line-simulation planning waypoints are derived.
7. The image-based autonomous line-following flight system for unmanned aerial vehicles according to claim 1, characterized in that: The UAV cruise action instructions include hovering, moving forward, moving backward, moving left, moving right, rising, falling, and rotating.
8. An image-based autonomous line-flying method for unmanned aerial vehicles, characterized in that: Using an image-based UAV autonomous line-following flight system as described in any one of claims 1 to 7 to perform autonomous line-following flight control of a UAV, comprising the following steps: The satellite positioning module captures the position information of the drone in real time; the image acquisition unit in the image module collects image data with depth information; the image processing unit in the image module extracts key frames from the image data, identifies the power line target from the key frames, calculates the power line coordinates, and generates a three-dimensional model of the power line; The line-mimicking path planning unit in the flight control module matches the UAV's position information with the three-dimensional model of the power line and generates line-mimicking waypoints. The control unit in the flight control module compares the preset waypoints of the UAV route with the line-mimicking waypoints and modifies the UAV's cruise action instructions.