Vision-based flight control method for low-altitude UAVs

Abstract

The present invention provides a vision-based flight control method for a low-altitude unmanned aerial vehicle (UAV). The specific steps are as follows: using deep learning to detect obstacles to be detected; collecting aerial image samples and establishing an improved single-stage target detection network model; inputting a test set into the optimized network to obtain the results of obstacles to be detected. Using image processing to detect the detection results, the obstacles to be detected are then detected: grayscale processing is performed on the input image to be detected; the edge information of the image is enhanced, and the image is smoothed and noise is removed; the results of detection based on the LSD line detection algorithm are screened; the screened results are grouped and fitted to obtain the results of obstacles to be detected. Finally, flight control instructions are issued based on the obstacle detection results. The present invention has a relatively fast detection speed and a high accuracy rate, and has the characteristics of high real-time performance and high robustness. It is of great significance for realizing automatic obstacle avoidance of UAVs and ensuring the low-altitude flight of UAVs.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous obstacle avoidance for unmanned aerial vehicles (UAVs), and in particular to a vision-based obstacle detection and intelligent flight control method for UAVs. Background Art

[0002] With the advancement of drone technology, the ability of drones to autonomously complete their missions is becoming increasingly important. Drones may encounter obstacles during their missions. To ensure that drones can better complete their missions and safely reach their destinations, they need to identify obstacles during flight. This allows the drone to enter obstacle avoidance mode and circumvent them. Power lines are considered one of the most threatening hazards and the most difficult obstacles to avoid during low-altitude drone flight. In addition to power lines, trees and utility poles also pose a threat to the safe flight of low-altitude drones.

[0003] In recent years, numerous vision-based powerline detection algorithms have been proposed to facilitate autonomous drones and automatic obstacle avoidance. These algorithms typically implement powerline detection through edge detection, Hough or Radon transforms, and line fitting based on powerline constraints. However, these powerline detection algorithms can only be applied in simple scenarios. In complex scenarios, detection is not only slow and inaccurate, but also requires extensive post-processing to distinguish power lines from pseudo-lines, making false detections and missed detections highly likely. Furthermore, other obstacles encountered by low-altitude drones include tree barriers and utility poles. Traditional target detection methods have low detection speed and accuracy for these types of obstacles, and are prone to missed detections and false detections in the presence of background interference, significantly reducing performance. Summary of the Invention

[0004] In response to the shortcomings of existing drone obstacle detection, the present invention provides a vision-based flight control method for low-altitude drones. First, for the detection of typical obstacles such as power lines, an image processing detection method is used to pre-enhance the image through the Hessian matrix method, which helps feature extraction to correctly identify power lines in the image; the LSD line detection algorithm is used to extract straight line edge information in the image, and the processing complexity is reduced, thereby improving detection speed and robustness; length plus angle screening is used and a tolerance threshold is set to eliminate interference in the image; hierarchical clustering and least squares method are used to fit the detection results, effectively improving detection accuracy. Secondly, for the detection of typical obstacles such as trees and utility poles, a deep learning network is used to automatically detect typical obstacles such as tree barriers and utility poles from aerial image samples. This is of great significance for realizing automatic obstacle avoidance of drones, ensuring the safe flight of drones at low altitudes, and ensuring the smooth completion of established tasks.

[0005] The present invention provides a vision-based low-altitude UAV flight control method, the specific implementation steps are as follows:

[0006] S1. Use deep learning to detect obstacles such as trees, utility poles, and power lines when the drone is flying at low altitude:

[0007] S11. Collect aerial image samples during the flight of the UAV and expand the number of samples through data enhancement methods;

[0008] S12, labeling typical targets in the aerial image samples in S11, generating a detection dataset of the typical targets, and dividing the detection dataset into a training set, a validation set, and a test set;

[0009] S13. Establish an improved single-stage target detection network model;

[0010] S14, inputting the training set and the test set into the improved single-stage target detection network model and setting the training parameters;

[0011] S15. Use the optimizer to train the improved single-stage target detection network model, and record and save the trained model and the corresponding training loss and validation loss in each round until the single-stage target detection network converges;

[0012] S16. Input the test set into the improved single-stage target detection network model trained in S15 for testing to obtain the detection results of the obstacles to be detected;

[0013] S2: Use image processing to detect obstacles based on the detection results obtained in S1:

[0014] S21, grayscale processing is performed on the detection result of the obstacle to be detected to obtain a grayscale image;

[0015] S22, enhancing edge information of the obstacle to be detected in the grayscale image obtained in S21 to obtain an enhanced image;

[0016] S23, performing Gaussian filtering and edge detection on the enhanced image obtained in S22;

[0017] S24, detecting the final pre-processed image DE (x, y) to obtain the detection result of the straight line in the obstacle to be detected;

[0018] S25, performing main direction screening and length screening on the image obtained in S24;

[0019] S26, grouping the line detection results obtained in S25 using an agglomerative hierarchical clustering method;

[0020] S27. For the grouping results obtained in S26, a least squares method is used to perform fitting to obtain a final detection result of the obstacle to be detected;

[0021] S3: Generate a decision frame based on the UAV flight conditions, combine S1 and S2 to obtain the detection results of the obstacles to be detected, and perform flight control and obstacle avoidance instructions to enable the UAV to avoid obstacles:

[0022] S31. Generate a square decision box with the center point of the drone's viewing angle as the center of the decision box. The side length R of the decision box is expressed as follows:

[0023]

[0024] Where K is the scaling factor, R A is the width of the drone’s viewing angle frame, and h is the height of the drone;

[0025] S32. Send corresponding flight control and obstacle avoidance instructions to the UAV based on the relative position of the detected obstacle and the decision frame.

[0026] Preferably, the specific implementation process of step S13 is as follows:

[0027] S131. Establish a backbone feature extraction network: Replace the backbone feature extraction network CSPdarknet53 with the MobileNetv3 network, and use the effective feature layer output by the MobileNetv3 network for feature extraction;

[0028] S132. Establish a feature fusion network: fuse the effective feature layers of different levels extracted in S131, and set attention mechanisms after the three effective feature layers output by the backbone feature network;

[0029] S133, establish a prediction network: connect the basic convolution block and the 1*1 convolution to obtain a prediction network. Preferably, in step S21, the specific expression of the grayscale processing is as follows:

[0030] Gray(x,y)=0.3R+0.6G+0.1B

[0031] Among them, Gray(x,y) is the grayscale image, R, G, and B are the pixel distribution functions of the three channels of the obstacle detection results.

[0032] Preferably, the specific process of enhancing the edge information of the obstacle to be detected in the grayscale image in step S22 is: calculating the Hessian matrix at each point of the image I (x, y) The eigenvalue λ of is expressed as follows:

[0033]

[0034] Among them, V(x,y) is the image enhancement result, R represents the clumping metric, S represents the smoothness metric, β is the adjustment parameter of R, and c is the adjustment parameter of S.

[0035] Preferably, the specific implementation process of step S23 is as follows:

[0036] S231, using a two-dimensional Gaussian function of a Gaussian filter to filter the binary image to remove noise in the binary image and obtain a smoothed image. The specific expression of the two-dimensional Gaussian function is:

[0037]

[0038] Among them, σ is used as the Gaussian filter parameter to control the smoothness of the image, x, y are the coordinates of the pixel point, and G(x, y) is the image after Gaussian filtering;

[0039] S232, using the Canny operator in the y direction to perform Canny edge detection on G(x,y) to obtain an edge image E(x,y);

[0040] S233 , performing a dilation operation on the edge image E(x, y) to remove noise and smooth the edges, thereby obtaining a final pre-processed image DE(x, y).

[0041] Preferably, the specific implementation process of step S25 is as follows:

[0042] S251, dividing the straight line detected in S24 into eight areas according to the inclination angle of the straight line;

[0043] S252, respectively calculating the average length of the straight lines in each of the eight regions, setting the region with the longest average length as the main direction, and defining the average angle of the region with the longest average length as the main direction angle;

[0044] S253, setting a threshold angle of the main direction angle to form a candidate area, eliminating all lines that are not in the candidate area, and only retaining the line detection results in the candidate area;

[0045] S254: Set a straight line length threshold, and remove all straight lines with lengths below the threshold from the straight line detection results obtained in S253.

[0046] Preferably, the specific implementation process of step S26 is as follows:

[0047] S261, forming an intercept list Blist with the intercept of each straight line on the image in the straight line detection result of S25;

[0048] S262. Calculate the absolute difference between each element Blist[i] and other elements in Blist, take the minimum value Dmin[i] of the absolute difference, and form a new spacing list Dlist;

[0049] S263, for the distance list Dlist, set a distance threshold, and cluster all elements in the distance list Dlist using a top-down agglomerative hierarchical clustering method to obtain a grouping result list LList;

[0050] S264. Determine the number of groups and the members of each group according to the classification index.

[0051] Preferably, the specific implementation process of step S32 is as follows:

[0052] S321. Send flight control and obstacle avoidance instructions based on the position of the power line relative to the decision frame:

[0053] S3211. If the power line is within the judgment frame, the flight control and obstacle avoidance command is sent as a deceleration command;

[0054] S3212: If the power line is in the upper part of the decision box, send a flight control and obstacle avoidance command to lower the altitude until the decision box and the power line have no intersection.

[0055] S3213. If the power line is in the lower part of the decision box, send a flight control and obstacle avoidance command as an altitude increase command until the decision box and the power line have no intersection;

[0056] S3214: If the power line is not within the judgment frame, no flight control and obstacle avoidance commands are sent;

[0057] S322. Calculate the intersection-over-union ratio of the detection frame and the decision frame of the tree or telephone pole to send flight control and obstacle avoidance instructions:

[0058] S3221. If the intersection-over-combination ratio is greater than 0, send flight control and obstacle avoidance instructions such as deceleration and ascent until the intersection-over-combination ratio is equal to 0.

[0059] S3222: If the intersection-to-combination ratio is 0, no flight control and obstacle avoidance commands are sent.

[0060] Compared with the prior art, the present invention has the following advantages:

[0061] 1. This paper adopts a deep learning-based target detection method for obstacles such as trees, and adopts a lightweight network MobileNetv3-YOLOv4 to address the limited resources of drone onboard computers, and adds an attention mechanism ECA, which has high detection accuracy, fast speed, good robustness and strong generalization ability.

[0062] 2. The power line detection method of the present invention is based on image processing. The Hessian matrix method enhances the main information of the image to a certain extent, retains or enhances the linear part, removes large smooth areas, and helps straight line detection and feature extraction to correctly identify the power lines in the image.

[0063] 3. The present invention performs pre-processing on the image, including Gaussian filtering smoothing, edge detection, and morphological processing, which not only makes the features of the image prominent, but also smoothes the image and eliminates most of the noise.

[0064] 4. The LSD algorithm of the present invention effectively overcomes the false detection and missed detection problems of the Hough transform algorithm, and the algorithm running time is also greatly improved compared with the Hough transform algorithm, with the characteristics of high real-time performance and high robustness.

[0065] 5. The present invention eliminates irrelevant interference in images with complex backgrounds and can completely detect power lines through main direction screening and length screening, a grouping method based on hierarchical clustering, and linear fitting based on the least squares method. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a flow chart of the vision-based low-altitude UAV flight control method of the present invention;

[0067] Figure 2 This is a diagram showing the detection results of trees and utility poles in the vision-based low-altitude UAV flight control method of the present invention;

[0068] Figure 3 This is a diagram showing the detection results of power lines in the vision-based low-altitude UAV flight control method of the present invention;

[0069] Figure 4a This is the original image from the perspective of a drone with a complex background containing power lines in the vision-based low-altitude drone flight control method of the present invention;

[0070] Figure 4b The present invention is based on the vision of the low altitude UAV flight control method Figure 4a The result of Sobel edge detection;

[0071] Figure 4c The present invention is based on the vision of the low altitude UAV flight control method Figure 4a The result of Canny edge detection in the y direction;

[0072] Figure 5a The present invention is based on the vision-based low-altitude UAV flight control method based on Hough transform Figure 4a Results of power line detection;

[0073] Figure 5b The present invention is based on the vision-based low-altitude UAV flight control method based on the LSD straight line detection algorithm Figure 4a Results of power line detection;

[0074] Figure 6a This is the original image of the drone's perspective used for testing the vision-based low-altitude drone flight control method of the present invention;

[0075] Figure 6b The present invention is based on the vision of the low altitude UAV flight control method Figure 6a The result graph after LSD line detection and screening without clustering and fitting;

[0076] Figure 6c The present invention is based on the vision of the low altitude UAV flight control method Figure 6b The result of hierarchical clustering plus fitting power lines based on the least squares method;

[0077] Figure 7 This is the network structure diagram of MobileNetv3-YOL Ov4 in the vision-based low-altitude UAV flight control method of the present invention;

[0078] Figure 8 This is a structural diagram of the efficient channel attention mechanism ECA in the vision-based low-altitude UAV flight control method of the present invention;

[0079] Figure 9 This is a diagram showing the dynamic changes in learning rate with the number of training rounds during the deep learning training process in the vision-based low-altitude UAV flight control method of the present invention;

[0080] Figure 10a Three diagrams of situations in which power line obstacles are located at different positions in the decision box in the vision-based low-altitude UAV flight control method of the present invention;

[0081] Figure 10b These are two diagrams showing two situations in which obstacles such as trees are located at different positions in the decision box in the vision-based low-altitude UAV flight control method of the present invention. DETAILED DESCRIPTION

[0082] To fully describe the technical content, objectives and effects of the present invention, the following will be described in detail with reference to the accompanying drawings.

[0083] The vision-based flight control method for low-altitude UAVs of the present invention is implemented as follows:

[0084] S1. Use deep learning to detect obstacles.

[0085] S2. Utilize image processing to detect obstacles based on the detection results obtained in S1.

[0086] S3. Generate a decision frame based on the detection results obtained in S1 and S2 in this embodiment, and send it to the drone through flight control and obstacle avoidance instructions to avoid obstacles.

[0087] In a preferred embodiment of the present invention, the flight control method of a low-altitude UAV based on vision of the present invention comprises the following steps: Figure 1 As shown:

[0088] S1. Use deep learning to detect obstacles such as trees, utility poles, and power lines during low-altitude flight of drones.

[0089] S11. Collect aerial image samples during the flight of the drone, adjust the resolution of the collected aerial image samples to 416×416 using a lossless image compression method to facilitate downsampling at different multiples, and expand the number of samples using a data augmentation method.

[0090] Specifically, the data augmentation method includes random cropping, horizontal flipping, color gamut adjustment, mosaic data augmentation, and the addition of Gaussian noise. The probability of performing random cropping and horizontal flipping is set to 0.5, the color gamut adjustment includes adjustments to saturation and brightness, and the probability of mosaic data augmentation is set to 0.5, and mosaic data augmentation is disabled for the last 30% of training rounds.

[0091] S12. Use LabelImg software to label typical targets in the aerial image samples in S11, generate a detection dataset of typical targets, and divide the detection dataset into a training set, a validation set, and a test set.

[0092] S13. Establish an improved single-stage target detection network MobileNet-YOLOv4 model, wherein the MobileNet-YOLOv4 model consists of a MobileNetv3 model with an added attention mechanism ECA, which includes a backbone feature extraction network, a feature fusion network, and a prediction network.

[0093] In a preferred embodiment of the present invention, the backbone feature network used is MobileNetv3.

[0094] S14. Input the training set and test set into the MobileNet-YOLOv4 model and set the training parameters such as batch size, initial learning rate, and number of training rounds.

[0095] S15. Use the Adam optimizer or the stochastic gradient descent method with momentum (SGD) to optimize the MobileNet-YOLOv4 model, perform repeated iterative training, and record and save the trained model and the corresponding training loss and validation loss for each round until the single-stage target detection network converges and the loss function of the training set no longer decreases. The learning rate of the training process continuously changes according to the number of current training rounds, and the change pattern is learning rate warmup plus cosine learning rate annealing.

[0096] Specifically, if the Adam optimizer is selected, the weight decay coefficient is set to 0 and the momentum parameter is set to 0.9; if the stochastic gradient descent method SGD with momentum is selected, the weight decay coefficient is set to 5e-4 and the momentum is set to 0.9.

[0097] The learning rate preheating plus cosine learning rate annealing, that is, initially using a larger learning rate to make the training process converge quickly, and then the learning rate changes in the form of a cosine function, and the rate of change of the learning rate gradually decreases, thereby achieving the purpose of dynamic adjustment.

[0098] S16. Input the test set into the MobileNet-YOLOv4 model optimized in S15 for testing. The test network selects the network with the smallest loss on the validation set to obtain the obstacle detection result.

[0099] S2. Utilize image processing to perform power line detection on the detection result obtained in S1.

[0100] S21. Grayscale the obstacle detection result to obtain a grayscale image of the detection result. The specific expression is as follows:

[0101] Gray(x,y)=0.3R+0.6G+0.1B

[0102] Among them, Gray(x,y) is the grayscale image, R, G, and B are the pixel distribution functions of the three channels of the obstacle detection results.

[0103] S22, using the image enhancement technology Sobel based on the Hessian matrix to enhance the edge information of the power lines in the grayscale image obtained in S21, by calculating the Hessian matrix at each point of the image I (x, y) The eigenvalue λ is obtained to obtain the enhanced image. The specific expression is as follows:

[0104]

[0105] Among them, V(x,y) is the image enhancement result, R represents the clumping metric, S represents the smoothness metric, β is the adjustment parameter of R, and c is the adjustment parameter of S.

[0106] Preferably, an image enhancement method based on the Hessian matrix is ​​used to enhance the linear part of the image, thereby enhancing the power line extraction capability of the algorithm.

[0107] S23. Perform Gaussian filtering on the enhanced image obtained in S22. First, perform smoothing to remove image noise. Then, perform Canny edge detection in the y direction to obtain edge detection results. Finally, use a morphological algorithm to process the edge points after edge detection to smooth the edges and filter out most of the noise.

[0108] S24. Use the LSD line detection algorithm to detect the final preprocessed image DE(x,y), obtain the line detection result in the image, and display the line detection result on the image DE(x,y).

[0109] Specifically, before using the LSD line detection algorithm, the image must be scaled to 0.8 times the original image in both length and width.

[0110] S25. Perform main direction screening and length screening on the image obtained in S24.

[0111] S26. Group the straight line detection results obtained in S25, i.e., the intercepts of the straight lines, by using an agglomerative hierarchical clustering method.

[0112] S27 . For the grouping result obtained in S264 , a least square method is used to perform fitting to obtain a final detection result of the power lines, and the final result is displayed on the image.

[0113] S3: Generate a decision frame based on factors such as the drone's flight altitude, and combine the detection results obtained in S1 and S2 to send flight control and obstacle avoidance instructions to enable the drone to avoid obstacles.

[0114] S31. Generate a decision box with the center point of the drone's view as the center of the decision box. The decision box is a square with a side length R as follows:

[0115]

[0116] Where K is the scaling factor, R A is the width of the drone’s viewing angle frame, and h is the height of the drone.

[0117] S32. Send corresponding flight control and obstacle avoidance instructions to the UAV based on the relative position of the obstacle and the decision frame.

[0118] In a preferred embodiment, the process of establishing the MobileNet-YOLOv4 model of the improved single-stage object detection network in step S13 includes:

[0119] S131. Establish a backbone feature extraction network: Replace the backbone feature extraction network CSPdarknet53 with the MobileNetv3 network, and use the effective feature layer output by the MobileNetv3 network for feature extraction.

[0120] S132. Establish a feature fusion network PANet: fuse the effective feature layers of different levels extracted by S131, and set the attention mechanism ECA after the three effective feature layers output by the backbone feature extraction network.

[0121] Specifically, the attention mechanism is set after the three effective feature layers output by the backbone feature extraction network and before the feature fusion network PANet.

[0122] S133. Establish a prediction network: connect the basic convolution block and the 1*1 convolution to obtain a prediction network.

[0123] Specifically, the basic convolution block consists of a 3*3 convolution, batch normalization, and a nonlinear activation function.

[0124] Furthermore, the specific process of performing Gaussian filtering and edge detection on the enhanced image in step S23 is as follows:

[0125] S231, using a two-dimensional Gaussian function of a Gaussian filter to filter the binary image to remove noise in the binary image and obtain a smoothed image. The specific expression of the two-dimensional Gaussian function is:

[0126]

[0127] Among them, σ is used as a Gaussian filter parameter to control the smoothness of the image, x and y are the coordinates of the pixel point, and G(x, y) is the image after Gaussian filtering.

[0128] S232. Use the Canny operator in the y direction to perform Canny edge detection on G(x,y) to obtain the edge image E(x,y). The edge detection result has problems such as non-smoothness and a large amount of noise.

[0129] S233 , performing a dilation operation on the edge image E(x, y) to remove noise and smooth the edges, thereby obtaining a final pre-processed image DE(x, y).

[0130] Furthermore, the specific process of performing main direction screening and length screening on the image in step S25 is as follows:

[0131] S251 , dividing the straight line detected in S24 into eight regions according to the inclination angle of the straight line, that is, dividing 180° into eight equal parts.

[0132] S252. Calculate the average length of the straight lines in each of the eight areas respectively, set the area with the longest average length as the main direction, and define the average angle of the area with the longest average length as the main direction angle. For example, if the main direction is 0 to 25.5 degrees, then the main direction angle is 11.25 degrees.

[0133] S253. Set the threshold angle θ of the main direction angle to 16° to form a candidate area. For example, if the main direction angle is 11.25°, then the candidate area is 3.25° to 19.25°. Eliminate all straight lines that are not in the candidate area and only retain the straight line detection results in the candidate area.

[0134] S254: Set a straight line length threshold, and remove all straight lines with lengths below the threshold from the straight line detection results obtained in S253, thereby completing length screening.

[0135] Specifically, the method for determining the candidate area is main direction angle screening and angle threshold setting. The main direction takes the angle average of the area with the largest average straight line length, and the threshold of the main direction angle is set to 16°; and length screening is performed after completing the main direction screening.

[0136] Furthermore, the specific process of grouping the line detection results obtained in S25 using the agglomerative hierarchical clustering method in step S26 includes:

[0137] S261 , forming an intercept list Blist with the intercept of each straight line on the image in the straight line detection result of S25 .

[0138] S262. Calculate the absolute difference between each element Blist[i] and other elements in Blist, take the minimum value Dmin[i] of the absolute difference, and form a new spacing list Dlist.

[0139] S263 , for the distance list Dlist, set a distance threshold, and cluster all elements in the distance list Dlist using a top-down agglomerative hierarchical clustering method to obtain a grouping result list LList.

[0140] S264. Determine the number of groups and the members of each group according to the classification index.

[0141] Furthermore, the specific process of sending corresponding flight control and obstacle avoidance instructions to the UAV according to the relative position of the obstacle and the decision frame in step S32 includes:

[0142] S321. Send flight control and obstacle avoidance instructions based on the position of the power line relative to the decision frame:

[0143] S3211. If the power line is within the judgment frame, the flight control and obstacle avoidance command is sent as a deceleration command;

[0144] S3212: If the power line is in the upper part of the decision box, send a flight control and obstacle avoidance command to lower the altitude until the decision box and the power line have no intersection.

[0145] S3213. If the power line is in the lower part of the decision box, send a flight control and obstacle avoidance command as an altitude increase command until the decision box and the power line have no intersection;

[0146] S3214: If the power line is not within the judgment frame, no flight control and obstacle avoidance commands are sent;

[0147] S322. Calculate the intersection-over-union ratio of the detection frame and the decision frame of the tree or telephone pole to send flight control and obstacle avoidance instructions:

[0148] S3221. If the intersection-over-combination ratio is greater than 0, send flight control and obstacle avoidance instructions such as deceleration and ascent until the intersection-over-combination ratio is equal to 0.

[0149] S3222: If the intersection-to-combination ratio is 0, no flight control and obstacle avoidance commands are sent.

[0150] The following is a further description of a vision-based low-altitude UAV flight control method of the present invention in conjunction with an embodiment:

[0151] In this embodiment, typical obstacles such as Figure 2 and Figure 3 In conjunction with FIG4 , the specific implementation process of the method of the present invention is as follows:

[0152] S1. Use deep learning to detect trees and utility poles.

[0153] S11. Collect aerial image samples during the flight of the drone, adjust the resolution of the collected aerial image samples to 416×416 using a lossless image compression method to facilitate downsampling at different multiples, and expand the number of samples using a data augmentation method.

[0154] S12. Use LabelImg software to label the trees and utility poles in the aerial image samples in S11, generate a detection dataset of trees and utility poles, and divide the detection dataset into a training set, a validation set, and a test set.

[0155] S13. Establish a MobileNet-YOLOv4 model that improves the single-stage target detection network.

[0156] like Figure 7As shown, the lightweight MobileNetv3 network is used in the backbone feature extraction network, and an efficient channel attention mechanism (ECA) is set up in the three effective feature layers of the output. In the feature fusion network PANet, channel-separable convolution is used instead of ordinary convolution, significantly reducing the number of network parameters and further accelerating the network's inference speed.

[0157] like Figure 8 As shown in the figure, the attention mechanism ECA removes the fully connected layer in the original SE module and directly learns through a 1D convolution on the features after global average pooling, using convolution to achieve good cross-channel information acquisition capabilities.

[0158] S14. Input the training set and test set into the MobileNet-YOLOv4 model and set the batch size, initial learning rate, and number of training rounds.

[0159] S15. Use the stochastic gradient descent method (SGD) with momentum to optimize the MobileNet-YOLOv4 model, perform repeated iterative training, and record and save the trained model and the corresponding training loss and validation loss for each round until the single-stage object detection network converges and the loss function of the training set no longer decreases.

[0160] Specifically, the weight decay coefficient of the stochastic gradient descent method SGD with momentum is set to 5e-4 and the momentum is set to 0.9.

[0161] like Figure 9 As shown, in this embodiment, the learning rate of the training process adopts the strategy of learning rate preheating plus cosine learning rate annealing. In the initial stage, the learning rate is continuously increased, so that the improved single-stage target detection network can be adjusted quickly. As the training progresses, the learning rate continues to decrease and changes more and more slowly, achieving the purpose of adaptive adjustment of the learning rate, which has the effect of making the improved single-stage target detection network converge faster and improving the accuracy of the improved single-stage target detection network.

[0162] S16. Input the test set into the MobileNet-YOLOv4 model optimized in S15 for testing. The test network selects the network with the smallest loss on the validation set to obtain the detection results of trees and telephone poles.

[0163] like Figure 2 The drone’s viewing angle is shown. The judgment frame is in the default position. Figure 2 The center position is not marked. The white dotted box is the selection box of obstacles such as trees or telephone poles. It can be seen that the proposed algorithm can quickly detect trees and telephone poles in a complex background as a basis for sending drone control commands.

[0164] S2. Detect power lines based on the obstacle detection results using image processing.

[0165] S21. Grayscale the detection results of trees and electric poles to obtain a grayscale image of the detection results. The specific expression is as follows:

[0166] Gray(x,y)=0.3R+0.6G+0.1B

[0167] Among them, Gray(x,y) is the grayscale image, and R, G, and B are the pixel distribution functions of the three channels of the detection results of trees and telephone poles respectively.

[0168] S22, using the image enhancement technology Sobel based on the Hessian matrix to enhance the edge information of the power lines in the grayscale image obtained in S21, by calculating the Hessian matrix at each point of the image I (x, y) The eigenvalue λ is obtained to obtain the enhanced image. The specific expression is as follows:

[0169]

[0170] Among them, V(x,y) is the image enhancement result, R represents the clumping metric, S represents the smoothness metric, β is the adjustment parameter of R, and c is the adjustment parameter of S.

[0171] S23. Perform Gaussian filtering on the enhanced image obtained in S22. First, perform smoothing to remove image noise. Then, perform Canny edge detection in the y direction to obtain edge detection results. Finally, use a morphological algorithm to process the edge points after edge detection to smooth the edges and filter out most of the noise.

[0172] As shown in Figure 4, Figure 4a The drone’s viewing angle frame, the judgment frame is in the default position Figure 4a The center position of the image is not marked. When the background where the power line is located is relatively complex, it will be difficult for the image enhancement technology Sobel based on the Hessian matrix to extract the power line features, such as Figure 4b As shown in the figure, interferences such as stones, grass and trees have generated many noise points, making it almost impossible to distinguish the edge of the power line; Figure 4c As shown, the Canny edge detection in the y direction can well detect the edge characteristics by utilizing the penetration characteristics of the power lines for subsequent processing.

[0173] S24. Use the LSD line detection algorithm to detect the final preprocessed image DE(x,y), obtain the line detection result in the image, and display the line detection result on the image DE(x,y).

[0174] As shown in Figure 5, Figure 5aThe drone’s viewing angle frame, the judgment frame is in the default position Figure 5a The center of the area is not marked. The white dotted line represents obstacles such as power lines. Figure 5a The power line detection result based on Hough change. Figure 5b The following are the power line detection results based on the LSD algorithm. As can be seen from the two figures, the traditional Hough transform method can effectively extract power lines in simple backgrounds. However, in complex backgrounds, the extracted lines are scattered, disordered, and fragmented, and the average length of the extracted lines is short, leading to errors in determining the main direction and completely incorrect power line identification. Furthermore, the Hough transform is computationally intensive and unsuitable for real-time detection of power line obstacles by drones. Using the LSD algorithm avoids these issues and can completely and accurately extract power lines.

[0175] S25. Perform main direction screening and length screening on the image obtained in S24.

[0176] Although the results obtained by S26 and S25 can detect power lines, there is a situation where a power line is composed of multiple discontinuous straight line segments. The detection results cannot fully represent the entire power line. The agglomerative hierarchical clustering method is used to group the straight line detection results obtained by S25, that is, the intercepts of the straight lines.

[0177] S261 , forming an intercept list Blist with the intercept of each straight line on the image in the straight line detection result of S25 .

[0178] S262. Calculate the absolute difference between each element Blist[i] and other elements in Blist, take the minimum value Dmin[i] of the absolute difference, and form a new spacing list Dlist.

[0179] S263. For the distance list Dlist, set a distance threshold, and cluster all elements in the distance list Dlist using a top-down agglomerative hierarchical clustering method to obtain a grouping result list LList = [1 4 1 2 5 3 5 4 45 1 2 3 2].

[0180] S264. Determine the number of groups and the members of each group based on the classification index: the same element value indicates that they can be classified into one category. For example, there are three elements 2 with indexes 3, 11 and 13, which means that in the straight line represented by Blist, the 3rd, 11th and 13th items are a group.

[0181] S27 . For the grouping result obtained in S264 , a least square method is used to perform fitting to obtain a final detection result of the power lines, and the final result is displayed on the image.

[0182] As shown in Figure 6, Figure 6aThe drone’s viewing angle frame, the judgment frame is in the default position Figure 3 The exact center of the Figure 6b Central, original Figure 6a Although the interference lines and noise are eliminated after the LSD algorithm is used for screening, there are still cases where a power line consists of multiple intermittent straight line segments. For example, some power lines have multiple straight line segments, and some straight line segments on power lines cannot completely cover the entire transmission line. Figure 6c , the agglomerative method of hierarchical clustering is used to group the straight line segments, and the clustering basis is the intercept of the straight line; after the straight line segments are grouped, all the endpoints in the same group of straight lines can be fitted with the least squares method, and finally the entire power line can be well fitted.

[0183] The detection results of power lines are as follows Figure 3 As shown, Figure 3 The drone’s viewing angle frame, the judgment frame is in the default position Figure 3 It can be seen from the figure that even when there are a lot of complex straight line interferences in the background, the proposed method can still accurately detect the power lines.

[0184] S3. Generate a decision frame based on factors such as the drone's flight altitude, and combine the detection results obtained in S1 and S2 in this embodiment to perform flight control and send obstacle avoidance instructions to enable the drone to avoid obstacles.

[0185] S31. The center point of the drone's viewing angle is used as the center of the decision box to generate a decision box. As shown in FIG10 , the shape of the decision box is a square, and the expression of the side length R is as follows:

[0186]

[0187] Where K is the scaling factor, R A is the width of the drone’s viewing angle frame, and h is the height of the drone.

[0188] S32, according to the relative position of the obstacle and the decision frame, send corresponding flight control and obstacle avoidance instructions to the drone, such as Figure 10a shown.

[0189] S321. Send flight control and obstacle avoidance instructions based on the position of the power line relative to the decision frame:

[0190] S3211. If the power line is within the judgment frame, the flight control and obstacle avoidance command is sent as a deceleration command;

[0191] S3212: If the power line is in the upper part of the decision box, send a flight control and obstacle avoidance command to lower the altitude until the decision box and the power line have no intersection.

[0192] S3213. If the power line is in the lower part of the decision box, send a flight control and obstacle avoidance command as an altitude increase command until the decision box and the power line have no intersection;

[0193] S3214: If the power line is not within the judgment frame, no flight control and obstacle avoidance commands are sent;

[0194] S322, calculate the intersection and union ratio of the detection frame of the tree or telephone pole and the decision frame to send flight control and obstacle avoidance instructions, such as Figure 10b As shown:

[0195] S3221. If the intersection-over-combination ratio is greater than 0, send flight control and obstacle avoidance instructions such as deceleration and ascent until the intersection-over-combination ratio is equal to 0.

[0196] S3222: If the intersection-to-combination ratio is 0, no flight control and obstacle avoidance commands are sent.

[0197] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A vision-based low-altitude UAV flight control method, characterized in that: The specific implementation steps are as follows: S1. Use deep learning to detect obstacles such as trees, utility poles, and power lines when the drone is flying at low altitude: S11. Collect aerial image samples during the flight of the UAV and expand the number of samples through data enhancement methods; S12, labeling typical targets in the aerial image samples in S11, generating a detection dataset of the typical targets, and dividing the detection dataset into a training set, a validation set, and a test set; S13. Establish an improved single-stage target detection network model; S14, inputting the training set and the test set into the improved single-stage target detection network model and setting the training parameters; S15. Use the optimizer to train the improved single-stage target detection network model, and record and save the trained model and the corresponding training loss and validation loss in each round until the single-stage target detection network converges; S16. Input the test set into the improved single-stage target detection network model trained in S15 for testing to obtain the detection results of the obstacles to be detected; S2: Use image processing to detect obstacles based on the detection results obtained in S1: S21, grayscale processing is performed on the detection result of the obstacle to be detected to obtain a grayscale image; S22, enhancing edge information of the obstacle to be detected in the grayscale image obtained in S21 to obtain an enhanced image; The image enhancement method based on Hessian matrix is ​​used to enhance the linear part of the image and improve the power line extraction capability of the algorithm. S23, performing Gaussian filtering and edge detection on the enhanced image obtained in S22; S24, detecting the final pre-processed image DE (x, y) to obtain the detection result of the straight line in the obstacle to be detected; S25, performing main direction screening and length screening on the image obtained in S24; the specific implementation process is as follows: S251, dividing the straight line detected in S24 into eight areas according to the inclination angle of the straight line; S252, respectively calculating the average length of the straight lines in each of the eight regions, setting the region with the longest average length as the main direction, and defining the average angle of the region with the longest average length as the main direction angle; S253, setting a threshold angle of the main direction angle to form a candidate area, eliminating all lines that are not in the candidate area, and only retaining the line detection results in the candidate area; S254, setting a straight line length threshold, and removing all straight lines whose lengths are less than the threshold from the straight line detection results obtained in S253; S26. Group the line detection results obtained in S25 using an agglomerative hierarchical clustering method. The specific implementation process is as follows: S261, forming an intercept list Blist with the intercept of each straight line on the image in the straight line detection result of S25; S262. Calculate the absolute difference between each element Blist[i] and other elements in Blist, take the minimum value Dmin[i] of the absolute difference, and form a new spacing list Dlist; S263, for the distance list Dlist, set a distance threshold, and cluster all elements in the distance list Dlist using a top-down agglomerative hierarchical clustering method to obtain a grouping result list LList; S264. Determine the number of groups and the members of each group according to the classification index; S27. For the grouping results obtained in S26, a least squares method is used to perform fitting to obtain a final detection result of the obstacle to be detected; S3: Generate a decision frame based on the UAV flight conditions, combine S1 and S2 to obtain the detection results of the obstacles to be detected, and perform flight control and obstacle avoidance instructions to enable the UAV to avoid obstacles: S31. Generate a square decision box with the center point of the drone's viewing angle as the center of the decision box. The side length R of the decision box is expressed as follows: Where K is the scaling factor, R A is the width of the drone’s viewing angle frame, and h is the height of the drone; S32. Send corresponding flight control and obstacle avoidance instructions to the UAV based on the relative position of the detected obstacle and the decision frame.

2. The vision-based low-altitude UAV flight control method according to claim 1, characterized in that: The specific implementation process of step S13 is as follows: S131. Establish a backbone feature extraction network: Replace the backbone feature extraction network CSPdarknet53 with the MobileNetv3 network, and use the effective feature layer output by the MobileNetv3 network for feature extraction; S132. Establish a feature fusion network: fuse the effective feature layers of different levels extracted in S131, and set attention mechanisms after the three effective feature layers output by the backbone feature network; S133. Establish a prediction network: connect the basic convolution block and the 1*1 convolution to obtain a prediction network.

3. The vision-based low-altitude UAV flight control method according to claim 1, characterized in that: In step S21, the specific expression of the grayscale processing is as follows: Gray(x,y)=0.3R+0.6G+0.1B Among them, Gray(x,y) is the grayscale image, R, G, and B are the pixel distribution functions of the three channels of the obstacle detection results.

4. The vision-based low-altitude UAV flight control method according to claim 1, characterized in that: The specific process of enhancing the edge information of the obstacle to be detected in the grayscale image in step S22 is as follows: Calculate the Hessian matrix at each point of the image I (x, y) The eigenvalue λ of is expressed as follows: Among them, V(x,y) is the image enhancement result, R represents the clumping metric, S represents the smoothness metric, β is the adjustment parameter of R, and c is the adjustment parameter of S.

5. The vision-based low-altitude UAV flight control method according to claim 1, characterized in that: The step S23 includes the following sub-steps: S231, using a two-dimensional Gaussian function of a Gaussian filter to filter the binary image to remove noise in the binary image and obtain a smoothed image. The specific expression of the two-dimensional Gaussian function is: Among them, σ is used as the Gaussian filter parameter to control the smoothness of the image, x, y are the coordinates of the pixel point, and G(x, y) is the image after Gaussian filtering; S232, using the Canny operator in the y direction to perform Canny edge detection on G(x,y) to obtain an edge image E(x,y); S233 , performing a dilation operation on the edge image E(x, y) to remove noise and smooth the edges, thereby obtaining a final pre-processed image DE(x, y).

6. The vision-based low-altitude UAV flight control method according to claim 1, characterized in that: The step S32 includes the following sub-steps: S321. Send flight control and obstacle avoidance instructions based on the position of the power line relative to the decision frame: S3211. If the power line is within the judgment frame, the flight control and obstacle avoidance command is sent as a deceleration command; S3212: If the power line is in the upper part of the decision box, send a flight control and obstacle avoidance command to lower the altitude until the decision box and the power line have no intersection. S3213. If the power line is in the lower part of the decision box, send a flight control and obstacle avoidance command as an altitude increase command until the decision box and the power line have no intersection; S3214: If the power line is not within the judgment frame, no flight control and obstacle avoidance commands are sent; S322. Calculate the intersection-over-union ratio of the detection frame and the decision frame of the tree or telephone pole to send flight control and obstacle avoidance instructions: S3221. If the intersection-over-combination ratio is greater than 0, send flight control and obstacle avoidance instructions such as deceleration and ascent until the intersection-over-combination ratio is equal to 0. S3222: If the intersection-to-combination ratio is 0, no flight control and obstacle avoidance commands are sent.

Images