Unmanned aerial vehicle local horizon positioning method

Abstract

This invention relates to the field of image processing technology, and more particularly to a method for local horizon positioning of unmanned aerial vehicles (UAVs). The method utilizes multiple miniature cameras mounted on the UAV to capture surrounding images and extracts a preliminary skyline from these images. Then, based on the preliminary skyline, an energy function of the surrounding images is calculated, and the preliminary skyline is optimized using this energy function to obtain a complete skyline. The surrounding images are then divided into sky and ground regions using the complete skyline. These two regions are input into a trained convolutional neural network model to obtain the UAV's position coordinates relative to the complete skyline. This invention extracts the skyline from the image by calculating the edge response of the image and combining this with an energy function optimization method. Then, a lightweight environment recognition neural network model is used to perform collaborative environment-level recognition of the local environment, avoiding the limitations of feature-level and target-level recognition methods that are confined to local image information. This achieves efficient positioning of UAVs in local environments.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a method for local horizon positioning of unmanned aerial vehicles (UAVs). Background Technology

[0002] Drones are now widely used in agriculture and industry. Navigation systems play a crucial role in drone flight. Classical drone navigation systems include Global Navigation Satellite Systems (GNSS) and Inertial Navigation Systems (INS). However, these systems have limitations. With technological advancements, Vision-Based Navigation (VBN) has gradually become an important type of autonomous navigation system. Through onboard visual sensors, drones can acquire real-time information about their surroundings, process and analyze it using visual image algorithms, and thus achieve precise positioning and navigation even without GPS signals.

[0003] Numerous universities and research institutes both domestically and internationally have conducted extensive research on UAV visual positioning and navigation systems. Existing research directions include navigation based on local depth maps, positioning and navigation based on target recognition, and positioning and navigation based on local feature database matching. From an image processing perspective, the main approach is to achieve relative estimation of the UAV's position by matching and searching for local image features or local image targets, thereby realizing vision-based UAV positioning within a certain range.

[0004] Methods based on local feature data, such as SIFT (Scale-invariant feature transform) and ORB (Oriented Fast and Rotated BRIEF), have the advantage of not needing to rely on other information, tracking solely through feature matching. The matching model is relatively simple and computationally inexpensive. However, because the local navigation environment is wide-ranging and often contains similar features, the probability of mismatches is also relatively high.

[0005] Target recognition-based positioning and navigation can reduce the probability of mismatches by identifying specific targets such as airports, airplanes, cars, and ships. However, to achieve better recognition results, the recognition models are quite complex, typically employing convolutional neural networks with hundreds or even thousands of layers, resulting in significant computational overhead and hindering implementation on edge devices like UAVs. Conversely, reducing the complexity of convolutional neural networks for edge implementation compromises the performance of the recognition model, thereby reducing positioning accuracy. The root cause of these two types of visual navigation problems lies in their over-reliance on local image information and under-reliance on global image information. Summary of the Invention

[0006] In view of this, the present invention aims to provide a local ground positioning method for unmanned aerial vehicles (UAVs). By combining multiple miniature cameras deployed on the UAV to capture surrounding images, and using a lightweight environmental recognition neural network model to perform collaborative environmental-level recognition of the local environment, the method avoids the problem that feature-level recognition and target-level recognition are limited to local image information, thereby achieving efficient positioning of UAVs in local environments.

[0007] To achieve the above objectives, the technical solution created by this invention is implemented as follows: A method for local horizon positioning of unmanned aerial vehicles (UAVs) includes: S1: Deploy multiple cameras around the drone and control the multiple cameras to simultaneously capture images of the drone in the local environment, obtaining multiple local images; S2: Extract edges from each local image obtained in step S1 and project the edges to obtain a preliminary skyline; divide each local image according to the preliminary skyline to obtain a preliminary sky region and a preliminary ground region; S3: Calculate the energy function of each local image based on the statistical information of the preliminary sky region and preliminary ground region obtained in step S2 in each local image; S4: Use the energy function obtained in step S3 to optimize the boundary of the preliminary skyline obtained in step S2, and obtain the complete skyline of the local image; S5: Based on the complete skyline obtained in step S4, divide each local image into a sky region and a ground region; input the corresponding sky region and ground region of all images into the trained convolutional neural network model to obtain the position coordinates of the drone relative to the complete skyline.

[0008] Furthermore, in step S2: gradient calculations are performed in the horizontal, vertical, and diagonal directions for each local image; edge extraction of the sky and ground in each local image is performed using the following formula: ; Where, d g (x,y) represents the edge where the sky and ground meet in each local image, and d x (x,y) represents the horizontal gradient of each local image, d y (x,y) represents the vertical gradient of each local image, d xy (x,y) represents the diagonal gradient of each local image. This represents the weights used to adjust the diagonal gradient; the edges are projected onto the x- or y-direction of each local image to obtain a preliminary skyline.

[0009] Furthermore, the gradient calculation for each local image in the three directions of horizontal, vertical, and diagonal includes: calculating the horizontal gradient, vertical gradient, and diagonal gradient for each image channel in each local image; taking the maximum value of all horizontal gradients as the horizontal gradient of each local image; taking the maximum value of all vertical gradients as the vertical gradient of each local image; and taking the maximum value of all diagonal gradients as the diagonal gradient of each local image.

[0010] Furthermore, a preliminary skyline is obtained by projecting the edges in the x-direction into each local image: B(x,b)=min{y|d g (x,y)≥b}; Where B(x,y) represents the preliminary skyline obtained by projection, and d g (x,y) represents the edge at the boundary between the sky and the ground in each local image, and b represents the projection threshold.

[0011] Furthermore, in step S3, the energy function of each local image is: ; Where E represents the energy function, α and β represent the weights of the initial sky region and the initial ground region, respectively, and σ s and σ g Let λ represent the values ​​of the determinants of the variance matrices of the pixel sets in the initial sky region and the initial ground region, respectively. s and λ g These represent the largest eigenvalues ​​of the two corresponding variance matrices.

[0012] Furthermore, the process of obtaining the complete skyline of the local image in step S4 includes: traversing the projection threshold within the set projection threshold range to find the preliminary skyline corresponding to the maximum energy function; fitting the obtained preliminary skyline to obtain the complete skyline.

[0013] Furthermore, in the convolutional neural network model in step S5: convolution is performed on the sky region and the ground region of all images respectively to capture local image features in the sky region and the ground region respectively; a fully connected operation is performed on the two local image features respectively to establish the relationship between the two local image features; a fully connected operation is performed on the two features obtained by concatenating them to integrate the sky and ground information in the two features to obtain the integrated feature; a fully connected operation is performed on the integrated feature to output the position coordinates of the UAV relative to the complete skyline.

[0014] Furthermore, the convolutional neural network model is trained using the following loss function: ; Where L represents the loss function, (zx ,z y () represents the drone's actual position coordinates relative to the complete skyline. This represents the predicted position coordinates output by the convolutional neural network model.

[0015] Furthermore, after the convolution and fully connected operations, the processed features are activated using the following formula: ; Among them, ι(x) in ) represents the input feature x in Activation operation.

[0016] Compared with the prior art, the present invention can achieve the following beneficial effects: This invention presents a localized UAV horizon positioning method. It combines multiple miniature cameras mounted on the UAV to capture surrounding images and utilizes a lightweight environmental recognition neural network model for collaborative environment-level recognition of the local environment. This avoids the limitations of feature-level and target-level recognition, which are confined to local image information, thus achieving efficient UAV positioning in localized environments. By calculating the edge response of the image and combining it with an optimized energy function calculation method, the skyline is extracted from the image, achieving accurate division between the sky and the ground. Using the skyline to divide the image into sky and ground parts and obtaining local features separately reflects the objective reality that the sky and ground differ significantly in scale, texture, and other local features, significantly improving the model's recognition performance. Attached Figure Description

[0017] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 A schematic flowchart of the UAV local horizon positioning method described in the embodiments of the present invention; Figure 2 A comparison diagram of the method provided by the present invention and existing methods, as described in the embodiments of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0019] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] like Figure 1 As shown, the UAV local horizon positioning method described in the embodiments of the present invention includes: S1: Multiple cameras are deployed around the drone, and the cameras are controlled to simultaneously capture images of the drone in a local environment, obtaining multiple partial images. In this embodiment of the invention, specifically, one camera is deployed around each of the four drones, and the four cameras are controlled to simultaneously capture images of the drone in a local environment, obtaining four RGB format partial images I1, I2, I3, and I4, that is, each partial image I includes an R channel I. r G channel I g and B channel I b .

[0021] S2: Extract edges from each local image obtained in step S1 and project the edges to obtain a preliminary skyline; divide each local image according to the preliminary skyline to obtain a preliminary sky region and a preliminary ground region.

[0022] In some embodiments, step S2 includes: S21: Calculate the gradients in the horizontal, vertical, and diagonal directions for each local image.

[0023] In some embodiments, the gradient calculation in the three directions includes: calculating the horizontal gradient, vertical gradient, and diagonal gradient for each image channel in each local image; taking the maximum value of all horizontal gradients as the horizontal gradient of each local image; taking the maximum value of all vertical gradients as the vertical gradient of each local image; and taking the maximum value of all diagonal gradients as the diagonal gradient of each local image.

[0024] In this embodiment of the invention, the calculation process of the gradients in the three directions of each local image is as follows: The R channel I in each local image is calculated using the following formula. r G channel I g and B channel I b Calculate the horizontal gradient using the following formulas respectively: ; in, , and Representing R channel I r G channel I g and B channel I b The horizontal gradient.

[0025] At this point, the horizontal gradient of each local image is: ; Where, dx (x,y) represents the horizontal gradient of each local image.

[0026] The R channel I in each local image is calculated using the following formula. r G channel I g and B channel I b Perform the following vertical gradient calculations: ; in, , and Representing R channel I r G channel I g and B channel I b The vertical gradient.

[0027] At this point, the vertical gradient of each local image is: ; Where, d y (x,y) represents the vertical gradient of each local image.

[0028] The R channel I in each local image is calculated using the following formula. r G channel I g and B channel I b Calculate the diagonal gradient of each of the following equations: ; in, , and Representing R channel I r G channel I g and B channel I b The diagonal gradient.

[0029] At this point, the diagonal gradient of each local image is: ; Where, d xy (x,y) represents the diagonal gradient of each local image.

[0030] S22: Edge extraction of the sky and ground in each local image is performed using the following formula: ; Where, d g (x,y) represents the edge where the sky and ground meet in each local image. This represents the weights used to adjust the diagonal gradient, which are preferably set in this embodiment of the invention. =1.6.

[0031] The edge response of an image reflects the possibility that a local part of the image is an edge. Since the boundary between heaven and earth generally produces local edges, the local contour of the complete skyline is obtained through steps S21 and S22. The subsequent skyline optimization based on the local contour can greatly reduce the size of the candidate set for searching the skyline and locate the skyline more efficiently.

[0032] S23: Project the edges onto the x-direction or y-direction of each local image to obtain a preliminary skyline. In some embodiments, a preliminary skyline is obtained by projecting the edges onto the x-direction of each local image: B(x,b)=min{y|d g (x,y)≥b}; Where B(x,b) represents the initial skyline obtained by projection, and b represents the projection threshold.

[0033] The above formula means that, given the position in the x-direction and the projection threshold b, the initial skyline B(x,b) takes the value such that d g The smallest position in the y-direction for (x,y)≥b to hold true (even if d) g The minimum value of y for (x,y)≥b. Given a projection threshold b, the initial skyline B(x,b) degenerates into a univariate function, called the edge d. g The projection in the x-direction. The projection B(x,b) should be a coordinate value in the y-direction. Based on this coordinate value, the sky boundary at x can be defined. That is, the sky part at x is the part with a y-direction coordinate less than the above projection value, and the ground part at x is the part with a y-direction coordinate less than the above projection value.

[0034] S3: Calculate the energy function of each local image based on the statistical information of the preliminary sky region and preliminary ground region obtained in step S2 in each local image.

[0035] In some embodiments, the energy function for each local image is: ; Where E represents the energy function, α and β represent the weights of the initial sky region and the initial ground region, respectively, and σ s and σ g Let λ represent the values ​​of the determinants of the variance matrices of the pixel sets in the initial sky region and the initial ground region, respectively. s and λ g These represent the eigenvalues ​​with the largest values ​​in the corresponding two variance matrices. In this embodiment of the invention, the weights α and β are set to 2 and 0.8, respectively.

[0036] Furthermore, in this embodiment of the invention, since each local image is an RGB image, meaning each pixel in the local image contains three values: red, green, and blue, the variance matrix of the pixel set in the initial sky region and the initial ground region is a 3×3 matrix. The variance matrix of the pixel set in the initial sky region and the initial ground region is obtained using the following formula: ; Where, Σ s and Σ g Let I represent the variance matrices of the preliminary sky region and the preliminary ground region, respectively. s (x,y) and I g (x, y) represent the pixel sets of the initial sky region and the initial ground region, respectively, μ s and μ g These represent the mean values ​​of the three color values ​​of pixels in the two sets (each is a 3×1 dimensional vector).

[0037] S4: Use the energy function obtained in step S3 to optimize the boundary of the preliminary skyline obtained in step S2, and obtain the complete skyline of the local image.

[0038] In some embodiments, the process of obtaining the complete skyline of a local image includes: Within the set projection threshold range, the projection threshold is iterated to find the preliminary skyline corresponding to the maximum energy function (i.e., the projection corresponding to the maximum energy function). The obtained preliminary skyline is fitted to obtain the complete skyline. In this embodiment of the invention, the obtained preliminary skyline is specifically fitted using the least squares method to obtain the complete skyline.

[0039] S5: Based on the complete skyline obtained in step S4, divide each local image into a sky region and a ground region; input the corresponding sky region and ground region of all images into the trained convolutional neural network model to obtain the position coordinates of the drone relative to the complete skyline.

[0040] In this embodiment of the invention, based on the complete skyline of the four local images, the four local images I1, I2, I3, and I4 are respectively divided into sky regions S1, S2, S3, and S4, and ground regions G1, G2, G3, and G4. The above eight region images are then normalized to equal-sized sub-images, which are used as input to the convolutional neural network model.

[0041] In some embodiments, the data processing procedure for a convolutional neural network model includes: Convolution is performed on the sky and ground regions of all images respectively to capture local image features in the sky and ground regions, as shown in the following embodiment of the invention: ; in, and Let ι represent local image features in the sky region and the ground region, respectively, and let ω represent the activation operation. s (u,v) and ω g (u,v) represent the convolution kernels for the sky and ground regions, respectively, and η s and η g S represents the biases for convolution on the sky and ground regions, respectively. i and G i Let represent the sky region and the ground region in the i-th local image, respectively.

[0042] In some embodiments, the activation operation is as follows: ; Where, x in This indicates the feature to be activated.

[0043] Fully connected operations are performed on the two local image features respectively to establish the relationship between the two local image features, realizing the mapping from image to location. In this embodiment of the invention, the following formula is used: ; in, and Let ρ represent the features after performing a fully connected operation on the two local image features, k represents the number of feature channels, and ρ represents the feature values. s (u,v) and ρ g (u,v) represent the fully connected linear coefficients when performing a fully connected operation on two local image features, γ s and γ g ... and The number of channels is 128, meaning that each of the two features is a 128-dimensional vector.

[0044] After concatenating the two obtained features, a fully connected operation is performed to integrate the sky and ground information from the two features, resulting in a comprehensive feature. In this embodiment of the invention, the following formula is used: ; Where F3(n) represents the composite feature, ρ3 represents the fully connected linear coefficients when performing a fully connected operation on the two concatenated features, τ represents the weight of the fully connected operation, and n represents the dimension of the composite feature.

[0045] By integrating image information from the sky and the ground using the above formula, and establishing a fully connected mapping layer to model the features of the integrated information, it is helpful to improve recognition performance.

[0046] Perform a fully connected operation on the comprehensive features to output the position coordinates of the UAV relative to the complete skyline.

[0047] In this embodiment of the invention, two parallel fully connected operations are performed on the F3(n) comprehensive features to output the abscissa z of the UAV relative to the complete skyline. x and the vertical coordinate z y As shown in the following formula: ; in, and Let represent the fully connected linear coefficients of the two fully connected operations, respectively. 1 and 2 represents the bias of the two full-join operations.

[0048] In some embodiments, the convolutional neural network model is trained using the following loss function: ; Where L represents the loss function, This represents the predicted position coordinates output by the convolutional neural network model. In this embodiment of the invention, the training hyperparameters and training method of a classic backpropagation (BP) neural network are used to train the convolutional neural network model.

[0049] To verify that the method provided by this invention can effectively detect the location of a drone, embodiments of this invention performed drone location detection based on existing methods under the same conditions. Specifically, embodiments of this invention compared the method provided by this invention with the SIFT algorithm (feature-level localization), AlexNet, and YOLO networks (object-level localization), and the comparison results are as follows. Figure 2 As shown, positioning error and positioning time are used for objective evaluation. A lower positioning error indicates higher positioning accuracy, and a shorter positioning time indicates a faster positioning response. From... Figure 2 It can be seen that the method provided by the present invention has good effects on positioning error and positioning time, which shows that the method provided by the present invention achieves efficient positioning of UAVs on edge platforms in local environments.

[0050] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0051] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for local horizontal positioning of unmanned aerial vehicles (UAVs), characterized in that, include: S1: Deploy multiple cameras around the drone and control the multiple cameras to simultaneously capture images of the drone in the local environment, obtaining multiple local images; S2: Extract edges from each local image obtained in step S1 and project the edges to obtain a preliminary skyline; divide each local image according to the preliminary skyline to obtain a preliminary sky region and a preliminary ground region; S3: Calculate the energy function of each local image based on the statistical information of the preliminary sky region and preliminary ground region obtained in step S2 in each local image; S4: Use the energy function obtained in step S3 to optimize the boundary of the preliminary skyline obtained in step S2, and obtain the complete skyline of the local image; S5: Based on the complete skyline obtained in step S4, divide each local image into a sky region and a ground region; input the corresponding sky region and ground region of all images into the trained convolutional neural network model to obtain the position coordinates of the drone relative to the complete skyline.

2. The UAV local horizon positioning method according to claim 1, characterized in that, In step S2: For each local image, gradient calculations are performed in the horizontal, vertical, and diagonal directions; Edge extraction for the sky and ground in each local image is performed using the following formula: ； Where, d g (x,y) represents the edge where the sky and ground meet in each local image, and d x (x,y) represents the horizontal gradient of each local image, d y (x,y) represents the vertical gradient of each local image, d xy (x,y) represents the diagonal gradient of each local image. This represents the weights used to adjust the diagonal gradient; By projecting the edges onto the x- or y-direction of each local image, a preliminary skyline is obtained.

3. The UAV local horizon positioning method according to claim 2, characterized in that, The gradient calculation for each local image in the horizontal, vertical, and diagonal directions includes: For each local image, calculate the horizontal gradient, vertical gradient, and diagonal gradient for each image channel. Take the maximum value of all horizontal gradients as the horizontal gradient of each local image; Take the maximum value of all vertical gradients as the vertical gradient of each local image; Take the maximum value of all diagonal gradients as the diagonal gradient of each local image.

4. The UAV local horizon positioning method according to claim 2, characterized in that, A preliminary skyline is obtained by projecting the edges in the x-direction into each local image: B(x,b)=min{y|d g (x,y)≥b}; Where B(x,y) represents the preliminary skyline obtained by projection, and d g (x,y) represents the edge at the boundary between the sky and the ground in each local image, and b represents the projection threshold.

5. The UAV local horizon positioning method according to claim 1, characterized in that, In step S3, the energy function of each local image is: ； Where E represents the energy function, α and β represent the weights of the initial sky region and the initial ground region, respectively, and σ s and σ g Let λ represent the values ​​of the determinants of the variance matrices of the pixel sets in the initial sky region and the initial ground region, respectively. s and λ g These represent the largest eigenvalues ​​of the two corresponding variance matrices.

6. The UAV local horizon positioning method according to claim 4, characterized in that, The process of obtaining the complete skyline of the local image in step S4 includes: Within the set projection threshold range, the projection thresholds are iterated to find the preliminary skyline corresponding to the maximum energy function. The initial skyline is fitted to obtain the complete skyline.

7. The UAV local horizon positioning method according to claim 1, characterized in that, In the convolutional neural network model in step S5: Convolution is performed on the sky and ground regions of all images respectively to capture local image features in the sky and ground regions respectively; Perform fully connected operations on the two local image features respectively to establish the relationship between the two local image features; After concatenating the two features, a fully connected operation is performed to integrate the sky and ground information from the two features, resulting in a comprehensive feature. Perform a fully connected operation on the comprehensive features to output the position coordinates of the UAV relative to the complete skyline.

8. The UAV local horizon positioning method according to claim 7, characterized in that, The convolutional neural network model is trained using the following loss function: ； Where L represents the loss function, (z x ,z y This represents the drone's actual position coordinates relative to the complete skyline. This represents the predicted position coordinates output by the convolutional neural network model.

9. The UAV local horizon positioning method according to claim 7, characterized in that, After the convolution and fully connected operations, the processed features are activated using the following formula: ； Among them, ι(x) in ) represents the input feature x in Activation operation.

Images