UAV positioning method, UAV and storage medium

Abstract

The present invention provides a method for positioning a drone, a drone, and a storage medium. The method includes: acquiring image data and inertial navigation data; acquiring optical flow data based on the image data; executing a first positioning algorithm when the number of optical flow tracking feature points is greater than a first preset threshold; executing a second positioning algorithm when the number of optical flow tracking feature points is less than or equal to the first preset threshold or every time a preset interval is reached. The second positioning algorithm is executed every preset interval, and the accumulated errors in the first positioning algorithm are corrected by scene matching to avoid inaccurate positioning, thereby improving the reliability of positioning. When the number of optical flow tracking feature points is abnormal, positioning is achieved through the second positioning algorithm to avoid positioning failure caused by abnormal visual inertial navigation intermediate frames. The common view area is obtained through coarse matching to narrow the range that requires accurate matching and reduce the amount of data processing. Positioning is achieved through scene matching through fine matching, thereby improving processing efficiency and alleviating the problem of positioning lag.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle (UAV) positioning technology, and in particular to a UAV positioning method, a UAV, and a storage medium. Background Art

[0002] Drones, with their high maneuverability, low-risk combat capabilities, and collaborative combat capabilities, have been widely used in various scenarios. They are currently the most widely used type of unmanned equipment, and their diverse applications rely heavily on their autonomous positioning capabilities. The most widely used positioning technologies currently include visual inertial navigation and scene matching positioning. However, these two technologies suffer from the following issues:

[0003] Visual inertial navigation algorithms primarily rely on sensors to perform recursive pose estimation to estimate the current position. During the recursive process, errors are inevitably present in each relative pose calculation. A common solution is to establish a feature point map and reduce errors through joint optimization of multiple key frames. However, after long-distance and long-term flight, the continuous accumulation of errors will eventually amplify the overall error. At the same time, if an abnormality occurs in a frame of data, the positioning system will produce large errors or even fail, resulting in weak robustness.

[0004] Scene matching positioning algorithms directly use traditional or network features for matching. The network-based feature matching calculation time is too long, resulting in a low frequency of positioning output and an inability to provide real-time and reliable positioning. For drones flying at faster speeds, excessive positioning lag will seriously affect autonomous flight missions.

[0005] In summary, how to overcome the problems of error accumulation, positioning failure caused by intermediate frame anomalies, and positioning lag is an urgent problem that needs to be solved in UAV positioning. Summary of the Invention

[0006] The present invention provides a UAV positioning method, a UAV and a storage medium, which are used to solve the defects of the prior art UAV positioning process, such as error accumulation, positioning failure caused by intermediate frame anomalies, and positioning lag.

[0007] The present invention provides a method for positioning a UAV, comprising:

[0008] Acquire image data and inertial navigation data;

[0009] Acquire optical flow data according to the image data, wherein the optical flow data includes optical flow tracking feature points in the image;

[0010] According to the optical flow data, when the number of optical flow tracking feature points is greater than a first preset threshold, executing a first positioning algorithm;

[0011] According to the optical flow data and the preset interval time, when the number of optical flow tracking feature points is less than or equal to the first preset threshold or every time the preset interval time is reached, executing a second positioning algorithm;

[0012] The first positioning algorithm includes:

[0013] Acquiring posture change information according to the optical flow data and the inertial navigation data;

[0014] Obtaining current positioning information based on the posture change information and the previous positioning information, wherein the previous positioning information represents the positioning information determined by the last positioning of the drone;

[0015] The second positioning algorithm includes:

[0016] Acquire map data, roughly match the image data with the map data, and acquire a common view area, where the common view area represents an area in the map that overlaps with the image captured by the drone;

[0017] Precisely matching the image data in the common view area with the map data to obtain a first homography matrix, where the first homography matrix represents a projection transformation relationship between the image data and the map data;

[0018] According to the first homography matrix, the center point coordinates of the image data are transformed to obtain current positioning information.

[0019] According to a method for positioning a drone provided by the present invention, the map data includes pre-processed map feature data; the acquiring of the map data, performing rough matching between the image data and the map data, and acquiring a common view area, includes:

[0020] Acquire the pre-processed map feature data from a map library, wherein the pre-processed map feature data is obtained by downsampling and feature extraction of a map image;

[0021] performing a rotation transformation on the image data according to the inertial navigation data to reduce an error between an image angle and a map angle of the image data;

[0022] Obtaining drone altitude information, performing multi-layer downsampling and feature extraction processing on the image data according to the drone altitude information to obtain multi-layer feature data, wherein the multi-layer downsampling represents reducing the resolution of the image data according to the drone altitude information to form multiple sampling layers with different resolutions;

[0023] Performing sparse feature matching on the multi-layer feature data and the pre-processed map feature data to obtain a coarse matching feature point set having a matching value greater than a second preset threshold;

[0024] Restoring the coarse matching feature point set to the image data of original resolution;

[0025] The image data and the pre-processed map feature data are respectively processed by a clustering algorithm to obtain an area where the coarse matching feature points are clustered to form the common view area.

[0026] According to a drone positioning method provided by the present invention, precisely matching the image data in the common view area with the map data to obtain a first homography matrix includes:

[0027] Performing dense feature matching on the image data corresponding to the common view area and the pre-processed map feature data to obtain a set of precise matching feature points having a matching value greater than a third preset threshold;

[0028] Merging the coarse matching feature point set and the fine matching feature point set to form a total matching feature point set;

[0029] Performing non-maximum suppression processing on the total matching feature point set to reduce the density of the feature points;

[0030] The first homography matrix is ​​calculated and obtained according to the total matching feature point set.

[0031] According to a drone positioning method provided by the present invention, the center point coordinates of the image data are transformed according to the first homography matrix to obtain current positioning information, including:

[0032] Performing a projection transformation on the center point coordinates of the image data based on the first homography matrix to obtain transformation position information;

[0033] According to the transformed position information, whether it is abnormal is determined. When it is abnormal, the drone state is marked as a lost state. When it is not abnormal, the transformed position information is used as the current positioning information.

[0034] According to a drone positioning method provided by the present invention, obtaining posture change information based on the optical flow data and the inertial navigation data includes:

[0035] Obtaining an essential matrix and a second homography matrix by calculating the optical flow data;

[0036] Calculating a first chi-square test score corresponding to the essential matrix, and calculating a second chi-square test score corresponding to the second homography matrix;

[0037] When the first chi-square test score and the second chi-square test score are both less than a fourth preset threshold, executing the second positioning algorithm;

[0038] When at least one of the first chi-square test score and the second chi-square test score is greater than the fourth preset threshold, selecting the matrix with the largest corresponding score between the essential matrix and the second homography matrix as the restoration matrix;

[0039] Obtaining the rotation amount and the translation amount by singular value decomposition according to the restoration matrix;

[0040] Calculating and obtaining a true rotation value according to the inertial navigation data and the rotation amount, and calculating and obtaining a true translation value according to the inertial navigation data and the translation amount;

[0041] The true rotation value and the true translation value form the posture change information.

[0042] According to a drone positioning method provided by the present invention, before obtaining optical flow data based on the image data, the method further includes:

[0043] Get the drone status;

[0044] When the drone is in a lost state, obtaining map data, wherein the lost state indicates that the drone lacks accurate positioning information from the last positioning;

[0045] matching a most similar map image from the map data as a matching map image according to the image data;

[0046] The center coordinates of the matching map image are obtained as the previous positioning information.

[0047] According to a drone positioning method provided by the present invention, matching the most similar map image from the map data as a matching map image based on the image data includes:

[0048] Acquiring local feature information according to the image data;

[0049] Performing clustering processing on the local feature information to obtain corresponding visual semantic information;

[0050] Calculating and obtaining a residual vector between the local feature information and the visual semantic information;

[0051] Aggregate all the residual vectors to obtain the global image vector;

[0052] Normalizing the global image vector so that a size corresponding to the global image vector is consistent with a size corresponding to the map data;

[0053] Obtaining a similarity value corresponding to the map image based on the global vector of the image and the global vector of the map image in the map data;

[0054] When the maximum similarity value is greater than a fifth preset threshold, the map image corresponding to the maximum similarity value is used as the matching map image, and the state of the drone is set to normal;

[0055] When the maximum similarity value is less than or equal to the fifth preset threshold, the drone state is maintained in a lost state, and the image data is updated until the matching map image is obtained.

[0056] According to a drone positioning method provided by the present invention, the obtaining of a similarity value corresponding to the map image based on the calculation of the global vector of the image and the global vector of the map image in the map data includes:

[0057] Determining whether there is historical location information or location-specifying information, wherein the location-specifying information represents the input location information;

[0058] When the historical location information or the location designation information exists, determining a map matching range based on the historical location information or the location designation information, calculating the image global vector with a global vector of a map image corresponding to the map matching range in the map data, and obtaining the similarity value;

[0059] When the historical position information and the position designation information do not exist, the image global vector is calculated with the global vectors of all map images in the map data to obtain the similarity value.

[0060] The present invention also provides a drone, comprising:

[0061] Input module, used to obtain image data and inertial navigation data;

[0062] A positioning and tracking module includes a first positioning module and a second positioning module, wherein the first positioning module is used to obtain posture change information and obtain current positioning information based on the posture change information and the previous positioning information, and the second positioning module is used to obtain a common view area through coarse matching and obtain current positioning information through fine matching;

[0063] The repositioning module is used to obtain the previous positioning information when the drone is in a lost state;

[0064] Map module, used to store map library data;

[0065] The input module, the positioning and tracking module, the repositioning module and the map module cooperate with each other to implement the above-mentioned drone positioning method.

[0066] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a drone positioning method as described in any one of the above.

[0067] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described methods for positioning a drone.

[0068] The present invention provides a method for positioning a drone, a drone, and a storage medium, each having at least the following beneficial effects: optical flow data is obtained based on image data captured by the drone, and whether the image data is abnormal can be determined based on whether the number of optical flow tracking feature points therein is greater than a first preset threshold. When normal, that is, the number of optical flow tracking feature points is greater than the first preset threshold, the drone's posture change information is obtained based on the optical flow data and inertial navigation data, and the current positioning information is obtained by combining the previous positioning information, that is, the previous positioning information, and positioning is achieved through visual inertial navigation. When abnormal, that is, the number of optical flow tracking feature points is less than or equal to the first preset threshold, or when a preset interval has passed, a coarse matching is first performed based on the image data and map data to obtain a common view area, and then a fine matching is performed based on the common view area. A first homography matrix is ​​obtained and projected to obtain the current positioning information, and positioning is achieved through scene matching. In this way, a second positioning algorithm is executed at each preset interval, and scene matching can correct the accumulated errors in the first positioning algorithm, avoiding inaccurate positioning caused by error accumulation and improving positioning reliability. At the same time, when the number of optical flow tracking feature points is abnormal, that is, when the intermediate frames are abnormal, the second positioning algorithm is used to achieve positioning. This can avoid positioning failure caused by visual inertial navigation intermediate frame anomalies, which helps improve robustness. In addition, when executing the second positioning algorithm, coarse matching is used to obtain the common view area, narrowing the range requiring accurate matching and reducing the amount of data processing. Then, fine matching is used to accurately match the scene to achieve positioning, which helps improve processing efficiency, alleviate positioning lag, and make positioning more real-time and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0070] Figure 1 This is one of the flow charts of a UAV positioning method provided by the present invention;

[0071] Figure 2 This is the second flow chart of a UAV positioning method provided by the present invention;

[0072] Figure 3 This is the third flow chart of a UAV positioning method provided by the present invention;

[0073] Figure 4 This is a fourth flow chart of a UAV positioning method provided by the present invention;

[0074] Figure 5 This is the fifth flow chart of a UAV positioning method provided by the present invention;

[0075] Figure 6 This is the sixth flow chart of a UAV positioning method provided by the present invention;

[0076] Figure 7 This is the seventh flow chart of a UAV positioning method provided by the present invention;

[0077] Figure 8 This is the eighth flow chart of a UAV positioning method provided by the present invention;

[0078] Figure 9 This is a schematic structural diagram of one embodiment of a drone provided by the present invention;

[0079] Figure 10 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0080] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0081] The following combination Figures 1-8 A method for positioning a drone according to the present invention is described, comprising:

[0082] S100: Acquire image data and inertial navigation data;

[0083] S200: Acquire optical flow data according to the image data, where the optical flow data includes optical flow tracking feature points in the image;

[0084] S300: executing a first positioning algorithm according to the optical flow data when the number of optical flow tracking feature points is greater than a first preset threshold;

[0085] S400: Executing a second positioning algorithm based on the optical flow data and a preset interval time when the number of optical flow tracking feature points is less than or equal to the first preset threshold or each time the preset interval time is reached;

[0086] The first positioning algorithm includes:

[0087] S310: Acquire posture change information according to the optical flow data and the inertial navigation data;

[0088] S320: Acquire current positioning information based on the posture change information and the previous positioning information, where the previous positioning information represents the positioning information determined by the last positioning of the drone;

[0089] The second positioning algorithm includes:

[0090] S410: Acquire map data, roughly match the image data with the map data, and acquire a common view area, where the common view area represents an area in the map that overlaps with the image captured by the drone;

[0091] S420: Performing precise matching of the image data in the common view area with the map data to obtain a first homography matrix, where the first homography matrix represents a projection transformation relationship between the image data and the map data;

[0092] S430: Transform the center point coordinates of the image data according to the first homography matrix to obtain current positioning information.

[0093] Based on the image data captured by the drone, optical flow data is obtained, and whether the image data is abnormal is determined by determining whether the number of optical flow tracking feature points therein is greater than a first preset threshold. When the image data is normal, meaning the number of optical flow tracking feature points is greater than the first preset threshold, the drone's position change information is obtained based on the optical flow data and inertial navigation data. This information is combined with the previous positioning information to obtain the current positioning information, achieving positioning through visual inertial navigation. When the image data is abnormal, meaning the number of optical flow tracking feature points is less than or equal to the first preset threshold, or when a preset interval has passed, a coarse matching is performed based on the image data and map data to obtain the common view area. A fine matching is then performed based on the common view area. A first homography matrix is ​​obtained and projected to obtain the current positioning information, achieving positioning through scene matching. In this way, a second positioning algorithm is executed at each preset interval. Scene matching can correct accumulated errors in the first positioning algorithm, preventing inaccurate positioning caused by accumulated errors and improving positioning reliability. Furthermore, when the number of optical flow tracking feature points is abnormal, meaning an abnormal intermediate frame, positioning is achieved through the second positioning algorithm. This can avoid positioning failure caused by abnormal intermediate frames of visual inertial navigation, thereby improving robustness. In addition, when executing the second positioning algorithm, the common view area is obtained through coarse matching, the range that requires accurate matching is narrowed, the amount of data processing is reduced, and then accurate scene matching is performed through fine matching to achieve positioning, which is conducive to improving processing efficiency, alleviating the problem of positioning lag, and making positioning more real-time and reliable.

[0094] Inertial navigation data refers to the raw measurement data obtained from inertial sensors (such as accelerometers and gyroscopes), which can characterize the acceleration, angular velocity, speed and other information of the drone.

[0095] Optical flow tracking of feature points assumes constant brightness within a pixel's local neighborhood and is based on the position of the feature point in the previous image frame in the next image frame. Typical feature points are highly distinguishable pixels extracted from an image using a feature extraction algorithm. Feature points typically consist of their location and a descriptor, which is used for feature matching. However, in this invention, optical flow tracking of feature points is used for matching and tracking, which only extracts the feature points without calculating their descriptors.

[0096] The optical flow data obtained from the image data is used to determine whether the image data is abnormal. From another perspective, this can be considered as whether the number of optical flow tracking feature points in the optical flow data meets the requirements of the visual inertial navigation in the first positioning algorithm. When it does not meet the requirements, it is abnormal. Because visual inertial navigation relies on the changing relationship between image frames, when the image frame is abnormal, the first positioning algorithm cannot accurately obtain the current positioning information. Therefore, the second positioning algorithm obtains the current positioning information based on scene matching, which can avoid the problem of positioning failure. In this way, the fusion of visual inertial navigation and scene matching can avoid the problem of positioning failure caused by abnormal intermediate frames. When visual inertial navigation fails, positioning is achieved through the second positioning algorithm, enhancing the reliability and robustness of positioning.

[0097] The image data is continuously updated. The current positioning information corresponding to the current image data becomes the previous positioning information from the perspective of the next updated image data.

[0098] The preset interval time can be set according to the needs of the actual application environment. The second positioning algorithm will be executed once every preset interval, that is, positioning is performed through scene matching, which can correct the error accumulation caused by visual inertial navigation and avoid error accumulation problems.

[0099] It's important to emphasize that while the terms "visual inertial navigation" and "scene matching" are used in the above description, the "visual inertial navigation" and "scene matching" processes used in this invention differ from existing visual inertial navigation and scene matching processes. The specific difference between "scene matching," or the second positioning algorithm, lies in that it first uses coarse matching to obtain a common view area, then restricts the time-consuming fine matching process to that area, eliminating the need for fine matching outside the common view area. This reduces the amount of data processed, improves processing efficiency, and effectively mitigates positioning lag. Furthermore, fine matching is effective for scenes with less texture, improving matching accuracy and enhancing robustness. The "visual inertial navigation" in the present invention, that is, the first positioning algorithm, has also been improved accordingly under the concept of integrating "visual inertial navigation" and "scene matching". The existing visual inertial navigation processing process includes two major processes: front-end tracking and back-end optimization. The back-end optimization is to reduce error accumulation. In the present invention, "scene matching" can solve the problem of error accumulation, and "visual inertial navigation" no longer needs to perform a back-end optimization process. At the same time, the "visual inertial navigation" in the present invention is also different from the existing scheme. The main feature is: the posture change information is obtained by preferentially recovering the essential matrix and the second homography matrix. The specific details will be explained later when the first positioning algorithm is introduced in detail.

[0100] refer to Figure 2 In some embodiments of the drone positioning method of the present invention, the map data includes pre-processed map feature data; and S410 includes:

[0101] S411: Acquire the pre-processed map feature data from a map library, wherein the pre-processed map feature data is obtained by downsampling and feature extraction of a map image;

[0102] S412: performing a rotation transformation on the image data according to the inertial navigation data to reduce an error between an image angle and a map angle of the image data;

[0103] S413: Acquire drone altitude information, perform multi-layer downsampling and feature extraction processing on the image data according to the drone altitude information, and obtain multi-layer feature data, wherein the multi-layer downsampling represents reducing the resolution of the image data according to the drone altitude information to form multiple sampling layers with different resolutions;

[0104] S414: performing sparse feature matching on the multi-layer feature data and the pre-processed map feature data to obtain a coarse matching feature point set having a matching value greater than a second preset threshold;

[0105] S415: Restoring the coarse matching feature point set to the image data of original resolution;

[0106] S416: Perform clustering algorithm processing on the image data and the pre-processed map feature data respectively, and obtain the area where the coarse matching feature points are clustered to form the common view area.

[0107] The map library stores preprocessed map feature data, which is obtained by pre-processing feature extraction from map images. This data does not occupy processing resources during real-time positioning, thereby reducing processing resource consumption during real-time positioning and improving processing efficiency. Since the image data captured by the drone has an angular deviation from the map, the image data is rotated according to the inertial navigation data to align the image data angle with the map angle, which helps to simplify and improve the accuracy of subsequent matching processing.

[0108] Because drones may be at different altitudes during flight, the coverage and actual unit area resolution of the captured image data will also vary, leading to matching stability issues when matching with a map captured at the same altitude. By performing multi-layer downsampling on the image data based on the drone's altitude information, the image data is reduced to multiple resolutions, forming multiple layers of sampling layers. Feature extraction is then performed on each sampling layer to obtain multiple layers of feature data. This multi-layer downsampling process can simulate images captured at different altitudes, improving matching stability. Sparse feature matching is performed on the multiple layers of feature data with preprocessed map feature data. Matching values ​​between feature points in the sampling layers and feature points in the map are calculated. Feature points in each sampling layer with matching values ​​greater than a second preset threshold are integrated to form a set of coarse matching feature points. The coarse matching feature point set is then restored to the original resolution image data. A clustering algorithm is then applied to the coarse matching feature point set in the image data to obtain a first region where the coarse matching feature points are clustered. Simultaneously, a clustering algorithm is applied to the corresponding coarse matching feature point set in the map to obtain a second region where the coarse matching feature points are clustered. The first and second regions constitute the common view area.

[0109] Sparse feature matching refers to extracting a small number of key points or feature points from an image and matching these key points. These feature points are extracted by a feature detector. In some embodiments of the present invention, the coarse matching process can be achieved by using SuperPoint combined with a feature matching algorithm with a detector such as LightGlue. SuperPoint is used for feature point extraction and descriptor generation, and LightGlue performs matching based on the descriptor, thereby achieving the purpose of coarse matching through sparse feature matching.

[0110] refer to Figure 3 In some embodiments of the drone positioning method of the present invention, the step S420 includes:

[0111] S421: performing dense feature matching on the image data corresponding to the common view area and the pre-processed map feature data to obtain a set of precise matching feature points whose matching values ​​are greater than a third preset threshold;

[0112] S422: merging the coarse matching feature point set and the fine matching feature point set to form a total matching feature point set;

[0113] S423: performing non-maximum suppression processing on the total matching feature point set to reduce the density of feature points;

[0114] S424: Calculate and obtain the first homography matrix according to the total matching feature point set.

[0115] In the common view area, dense feature matching is performed on the image data and the pre-processed map feature data to obtain a set of fine matching feature points with matching values ​​greater than a third preset threshold. This allows for in-depth and accurate acquisition of the matching feature points between the captured image and the map. The coarse matching feature point set and the fine matching feature point set are merged to form a total matching feature point set, which can more comprehensively characterize the corresponding feature points. Non-maximum suppression (NMS) processing is performed on the total matching feature point set to screen the most useful and significant feature points, reducing the density of feature points, which is beneficial for reducing the amount of data for subsequent processing and improving the efficiency of subsequent processing. Based on the total matching feature point set, a first homography matrix is ​​calculated, which can characterize the relationship between the plane of the drone image and the map plane. Drones do not necessarily shoot vertically downward when shooting images. Based on the drone's posture, there is a certain transformation relationship between the image plane and the map plane. Therefore, based on this plane transformation relationship, the drone's posture can also be inferred.

[0116] Dense feature matching involves extracting and matching a large number of feature points from an image. These feature points are typically evenly distributed across the image pixels, and dense feature matching typically uses a pixel-level similarity metric to calculate the matching relationship for each pixel in the image. Dense feature matching is also suitable for matching image features with less pronounced textures. It can also ensure matching accuracy in scenes with weaker textures, avoiding mismatches caused by weak textures.

[0117] In some embodiments of the present invention, during the precise matching process, a detector-free feature matching algorithm such as LoFTR (Detector-Free Local Feature Matching with Transformers) may be used to achieve precise matching through dense feature matching.

[0118] refer to Figure 4 In some embodiments of the drone positioning method of the present invention, the step S430 includes:

[0119] S431: Performing a projection transformation on the center point coordinates of the image data based on the first homography matrix to obtain transformation position information;

[0120] S432: Determine whether there is an abnormality based on the transformed position information. If there is an abnormality, mark the drone status as a lost state. If there is no abnormality, use the transformed position information as the current positioning information.

[0121] The center point coordinates of the image data are associated with the posture of the drone. The center point coordinates of the image data are projected and transformed based on the first homography matrix to obtain the transformed position information. It is judged whether the transformed position information is abnormal. It can be judged by whether the transformed position information is within a reasonable range of variation. When it is abnormal, it means that the transformed position information is not accurate positioning information. The drone status is marked as a lost state for subsequent repositioning. When it is not abnormal, that is, the transformed position information is accurate positioning information, the transformed position information is used as the current positioning information.

[0122] In this way, by judging the way of transforming the position information, it is possible to avoid using the abnormal position obtained after the conversion as the current positioning information, prevent the drone from making more serious errors in the future, and help improve the reliability of positioning.

[0123] refer to Figure 5 In some embodiments of the drone positioning method of the present invention, the step S310 includes:

[0124] S311: Obtaining an essential matrix and a second homography matrix by calculation according to the optical flow data;

[0125] S312: Calculate a first chi-square test score corresponding to the essential matrix, and calculate a second chi-square test score corresponding to the second homography matrix;

[0126] S313: When the first chi-square test score and the second chi-square test score are both less than a fourth preset threshold, executing the second positioning algorithm;

[0127] S314: When at least one of the first chi-square test score and the second chi-square test score is greater than the fourth preset threshold, selecting the matrix with the largest corresponding score between the essential matrix and the second homography matrix as the restoration matrix;

[0128] S315: Obtaining the rotation amount and the translation amount through singular value decomposition processing according to the restoration matrix;

[0129] S316: Calculate and obtain a true rotation value based on the inertial navigation data and the rotation amount, and calculate and obtain a true translation value based on the inertial navigation data and the translation amount;

[0130] S317: The true rotation value and the true translation value form the posture change information.

[0131] When the optical flow data contains a sufficient number of optical flow tracking feature points, the current positioning information is obtained using a first positioning algorithm. The optical flow data can characterize the pixel motion changes between the current image and the previous image. This can be used to calculate an intrinsic matrix, which represents the change in the pose of the camera that captured the current image compared to the pose of the camera that captured the previous image, i.e., the change in the current drone's pose compared to the previous drone's pose. A second homography matrix is ​​also calculated. The second homography matrix reflects the transformation relationship between the plane of the current image and the plane of the previous image, reflecting the change in the drone's pose from the perspective of plane transformation. Positioning accuracy can be assessed by calculating the first and second chi-square test scores corresponding to the intrinsic matrix and the second homography matrix. If both the first and second chi-square test scores are less than a fourth preset threshold, accurate positioning cannot be achieved using the first positioning algorithm, and the second positioning algorithm is therefore executed to obtain the current positioning information. If at least one of the first and second chi-square test scores is greater than the fourth preset threshold, the matrix corresponding to the larger of the two scores is used as the recovery matrix, facilitating more accurate positioning. According to the recovery matrix, the rotation and translation are obtained through singular value decomposition processing, which can characterize the rotation and translation changes of the drone from the previous moment to the current moment, that is, the posture change. However, the rotation and translation are not the rotation and translation in real space. It is necessary to combine the inertial navigation data to calculate the real rotation value and real translation value in real space size to form the posture change information.

[0132] In this way, the essential matrix and the homography matrix are restored preferentially to obtain the posture change information, which is conducive to making the final current positioning information more accurate and improving the accuracy of positioning. At the same time, the chi-square test score is used to evaluate whether the restored posture change is reliable. If it is unreliable, the second positioning algorithm is executed, which is conducive to improving the reliability of positioning.

[0133] In some embodiments of the present invention, the first chi-squared check score and the second chi-squared check score may be calculated using the following formula:

[0134]

[0135] Where s is the scaling factor of the image pyramid, n is the number of image pyramid layers, and e x and e y It is the pixel error calculated by projecting the feature points of the previous frame image to the current frame using the essential matrix or homography matrix, and th is the threshold selected based on the chi-square distribution table.

[0136] In some embodiments of the present invention, the rotation amount and the translation amount are obtained by singular value decomposition according to the restoration matrix, which can be calculated by the following formula:

[0137] E=UΣVT

[0138] Among them, E is the recovery matrix, U and V are orthogonal matrices. In the singular value decomposition, for any recovery matrix, there are two possible rotations R and translations t corresponding to it:

[0139]

[0140]

[0141] in is the matrix obtained by rotating 90 degrees along the Z axis.

[0142] In some embodiments of the present invention, the actual rotation value is calculated based on the inertial navigation data and the rotation amount, and the actual translation value is calculated based on the inertial navigation data and the translation amount, which can be calculated using the following formula:

[0143]

[0144]

[0145]

[0146] and is the inertial navigation state corresponding to the previous image frame, and is the inertial navigation state corresponding to the current image frame after integration, a is the acceleration and w is the angular velocity, and Δt is the time difference between the previous image frame and the current image frame.

[0147]

[0148]

[0149] and are the acceleration and angular velocity obtained by the inertial navigation measurement corresponding to the previous image frame, and is the acceleration and angular velocity obtained by the inertial navigation measurement corresponding to the current image frame, and is the offset of the angular velocity meter and accelerometer obtained by the inertial navigation measurement corresponding to the current image frame, g w is the gravitational acceleration, and the scale ratio is calculated by translation:

[0150]

[0151]

[0152] Among them, x, y, and z are the translation changes between the previous image frame and the current image frame.

[0153] Assuming that the rotation and translation of the previous image frame are (R1, t1), the true rotation value and true translation value (R2, t2) of the current frame are:

[0154] t2=t1+scale*(R1t)

[0155] R2=RR1

[0156] Among them, R and t are the rotation and translation amounts after the decomposition of the recovery matrix E, and based on this, the true rotation value R2 and the true translation value t2 are calculated.

[0157] refer to Figure 6 In some embodiments of the UAV positioning method of the present invention, before S200, the method further includes:

[0158] S110: Get drone status;

[0159] S120: When the drone is in a lost state, map data is acquired, wherein the lost state indicates that the drone lacks accurate positioning information from the last positioning;

[0160] S130: matching the most similar map image from the map data according to the image data as a matching map image;

[0161] S140: Acquire the center coordinates of the matching map image as the previous positioning information.

[0162] When a drone is just taking off, or if the previous image frame during flight cannot accurately locate the drone, it is in a lost state and cannot accurately determine its current location. When the drone is in a lost state, the currently captured image data is matched to the most similar map image from the map database. The position corresponding to this map image is used as a rough positioning, and the coordinates of the center point of this map image are used as the positioning information obtained based on the previous image frame. A secondary positioning algorithm is then executed to obtain accurate current positioning information. This allows repositioning even when the drone is lost, helping to adapt to various flight situations and improve positioning reliability.

[0163] refer to Figure 7 In some embodiments of the drone positioning method of the present invention, the step S130 includes:

[0164] S131: Acquire local feature information according to the image data;

[0165] S132: performing clustering processing on the local feature information to obtain corresponding visual semantic information;

[0166] S133: Calculate and obtain a residual vector between the local feature information and the visual semantic information;

[0167] S134: Aggregate all the residual vectors to obtain a global image vector;

[0168] S135: performing normalization processing on the global image vector so that a size corresponding to the global image vector is consistent with a size corresponding to the map data;

[0169] S136: Obtaining a similarity value corresponding to the map image based on the global vector of the image and the global vector of the map image in the map data;

[0170] S137: When the largest similarity value is greater than a fifth preset threshold, the map image corresponding to the largest similarity value is used as the matching map image, and the state of the drone is set to normal;

[0171] S138: When the maximum similarity value is less than or equal to the fifth preset threshold, the drone state is maintained in a lost state, and the image data is updated until the matching map image is obtained.

[0172] When matching the most similar map image from the map database based on drone-captured image data, the system first extracts local feature information from the image data and uses clustering to obtain the corresponding visual semantic information. The residual vectors between the local feature information and the visual semantic information are then calculated and aggregated to obtain a global image vector, which represents the content and semantics of the image data. Because the scale of drone-captured image data may not match the scale of the map data, the global image vector is normalized to align its dimensions with those of the map, ensuring more accurate and reliable matching.

[0173] A search calculation is performed in the map data based on the global vector of the image. When there is a similarity value greater than the fifth preset threshold, the map image corresponding to the maximum similarity value is used as the matching map image. When all similarity values ​​are less than the fifth preset threshold, it means that there is no map image in the map data that is similar enough to the required level. The image data is updated again for matching until the similarity value is greater than the fifth preset threshold, and the matching map image is obtained.

[0174] In some embodiments of the present invention, global feature information can be obtained from image data using the NetVLAD (Network Vector of Locally Aggregated Descriptors) algorithm, which uses a CNN to extract local feature information and then clusters and generates VLAD features as global feature information through neural network processing. Accordingly, the map data in the map library includes global feature information obtained by extracting and processing map images using the NetVLAD algorithm.

[0175] refer to Figure 8 In some embodiments of the drone positioning method of the present invention, the step S136 includes:

[0176] S1361: Determine whether there is historical location information or location-specified information, where the location-specified information represents the input location information;

[0177] S1362: When the historical location information or the location specification information exists, determining a map matching range based on the historical location information or the location specification information, calculating the image global vector with a global vector of a map image corresponding to the map matching range in the map data, and obtaining the similarity value;

[0178] S1363: When the historical location information and the location designation information do not exist, the image global vector is calculated with the global vectors of all map images in the map data to obtain the similarity value.

[0179] If a drone enters a lost state during flight, historical location information is available, which can be used to narrow the map range for matching. When the drone takes off, the user inputs location-specific information, which also narrows the map range for matching. If neither location-specific information is available during takeoff, the entire map library is used as the matching range for a comprehensive search for matching map images. Thus, determining the map matching range when either historical location information or location-specific information is available can reduce the amount of map data required for matching and improve matching efficiency.

[0180] refer to Figure 9 The present invention further provides a drone, comprising:

[0181] Input module 710, used to obtain image data and inertial navigation data;

[0182] The positioning and tracking module 720 includes a first positioning module and a second positioning module. The first positioning module is used to obtain posture change information and obtain current positioning information based on the posture change information and the previous positioning information. The second positioning module is used to obtain the common view area through coarse matching and obtain the current positioning information through fine matching.

[0183] The repositioning module 730 is used to obtain the previous positioning information when the drone is in a lost state;

[0184] Map module 740, used to store map library data;

[0185] The input module 710 , the positioning and tracking module 720 , the repositioning module 730 and the map module 740 cooperate with each other to implement the above-mentioned drone positioning method.

[0186] Drone data first enters input module 710. This data includes image data captured by the drone's camera and inertial navigation data acquired by sensors. The inertial navigation data is filtered and then enters positioning and tracking module 720. The purpose of the filtering is to filter out the high-frequency jitter noise caused by the drone's flight. The image data is distributed according to the drone's current state. If the drone is in a lost state (i.e., it has just started and has no absolute positioning or positioning and tracking fails), the image data first enters repositioning module 730 and then positioning and tracking module 720. If the drone is in a normal state without an aircraft, the image data enters positioning and tracking module 720 directly.

[0187] In the positioning and tracking module 720, the first positioning module executes a first positioning algorithm to obtain current positioning information through visual inertial navigation, and the second positioning module executes a second positioning algorithm to obtain current positioning information through mirror matching.

[0188] The goal of the relocalization module 730 is to find the map image that is most similar to the target image, i.e., the image data, from multiple map images to determine the rough positioning of the drone.

[0189] The map data in map module 740 consists of map images with longitude and latitude information captured by satellite or drone. This data is processed offline, eliminating the time and processing resources required for real-time operation. Map images are merged according to their longitude and latitude information, forming a complete map. The entire map is then re-segmented based on the drone's flight altitude range, using the visible area on the map at maximum altitude as a search window. This creates post-processed tiled map images, and feature information is extracted from each tile to form pre-processed map feature information.

[0190] In some embodiments of the present invention, the map module 740 may extract feature points from each tile map image using the SuperPoint algorithm and extract VLAD features using the NetVLAD algorithm as feature information. A feature tree and a VLAD tree are constructed based on the map using a KD-Tree data structure. The feature tree contains the SuperPoint feature points of all tile map images. After SuperPoint feature extraction is completed for each tile map image, the coordinates of its feature points are converted from the tile coordinate system to the full map coordinate system. A key-value pair is then constructed between the coordinates of each feature and its descriptor, and the feature tree is constructed using the coordinates based on the KD-Tree. The VLAD library in the map library contains the VLAD features of all tile maps. After VLAD features are extracted using NetVLAD, the coordinates of the center point of each tile map image in the full map coordinate system are used as a key to construct a key-value pair with the VLAD feature. These coordinates are then used to construct a VLAD tree based on the KD-Tree. Subsequent searches are performed using the full map pixel coordinates and the pixel window size as search criteria for SuperPoint or VLAD features.

[0191] Figure 10 An example of a physical structure diagram of an electronic device is shown below. Figure 10 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the above-mentioned drone positioning method.

[0192] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0193] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the above-mentioned drone positioning method provided by the above-mentioned methods.

[0194] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the above-mentioned drone positioning method provided by the above-mentioned methods.

[0195] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. In the description of the present invention, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0196] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0197] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for positioning a drone, characterized in that: include: Acquire image data and inertial navigation data; Acquire optical flow data according to the image data, wherein the optical flow data includes optical flow tracking feature points in the image; According to the optical flow data, when the number of optical flow tracking feature points is greater than a first preset threshold, executing a first positioning algorithm; According to the optical flow data and the preset interval time, when the number of optical flow tracking feature points is less than or equal to the first preset threshold or every time the preset interval time is reached, executing a second positioning algorithm; The first positioning algorithm includes: Acquiring posture change information according to the optical flow data and the inertial navigation data; Obtaining current positioning information based on the posture change information and the previous positioning information, wherein the previous positioning information represents the positioning information determined by the last positioning of the drone; The acquiring of posture change information according to the optical flow data and the inertial navigation data includes: Obtaining an essential matrix and a second homography matrix by calculating the optical flow data; Calculating a first chi-square test score corresponding to the essential matrix, and calculating a second chi-square test score corresponding to the second homography matrix; When the first chi-square test score and the second chi-square test score are both less than a fourth preset threshold, executing the second positioning algorithm; When at least one of the first chi-square test score and the second chi-square test score is greater than the fourth preset threshold, selecting the matrix with the largest corresponding score between the essential matrix and the second homography matrix as the restoration matrix; Obtaining the rotation amount and the translation amount by singular value decomposition according to the restoration matrix; Calculating and obtaining a true rotation value according to the inertial navigation data and the rotation amount, and calculating and obtaining a true translation value according to the inertial navigation data and the translation amount; The true rotation value and the true translation value form the posture change information; The second positioning algorithm includes: Acquire map data, roughly match the image data with the map data, and acquire a common view area, where the common view area represents an area in the map that overlaps with the image captured by the drone; Precisely matching the image data in the common view area with the map data to obtain a first homography matrix, where the first homography matrix represents a projection transformation relationship between the image data and the map data; According to the first homography matrix, the center point coordinates of the image data are transformed to obtain current positioning information.

2. A method for positioning a UAV according to claim 1, characterized in that: The map data includes pre-processed map feature data; The acquiring of map data, roughly matching the image data with the map data, and acquiring a common view area includes: Acquire the pre-processed map feature data from a map library, wherein the pre-processed map feature data is obtained by downsampling and feature extraction of a map image; performing a rotation transformation on the image data according to the inertial navigation data to reduce an error between an image angle and a map angle of the image data; Obtaining drone altitude information, performing multi-layer downsampling and feature extraction processing on the image data according to the drone altitude information to obtain multi-layer feature data, wherein the multi-layer downsampling represents reducing the resolution of the image data according to the drone altitude information to form multiple sampling layers with different resolutions; Performing sparse feature matching on the multi-layer feature data and the pre-processed map feature data to obtain a coarse matching feature point set having a matching value greater than a second preset threshold; Restoring the coarse matching feature point set to the image data of original resolution; The image data and the pre-processed map feature data are respectively processed by a clustering algorithm to obtain an area where the coarse matching feature points are clustered to form the common view area.

3. A method for positioning a UAV according to claim 2, characterized in that: The step of precisely matching the image data in the common view area with the map data to obtain a first homography matrix includes: Performing dense feature matching on the image data corresponding to the common view area and the pre-processed map feature data to obtain a set of precise matching feature points having a matching value greater than a third preset threshold; Merging the coarse matching feature point set and the fine matching feature point set to form a total matching feature point set; Performing non-maximum suppression processing on the total matching feature point set to reduce the density of the feature points; The first homography matrix is ​​calculated and obtained according to the total matching feature point set.

4. A method for positioning a UAV according to claim 3, characterized in that: The step of transforming the center point coordinates of the image data according to the first homography matrix to obtain current positioning information includes: Performing a projection transformation on the center point coordinates of the image data based on the first homography matrix to obtain transformation position information; According to the transformed position information, whether it is abnormal is determined. When it is abnormal, the drone state is marked as a lost state. When it is not abnormal, the transformed position information is used as the current positioning information.

5. The method for positioning a UAV according to claim 1, wherein: Before acquiring optical flow data according to the image data, the method further includes: Get the drone status; When the drone is in a lost state, obtaining map data, wherein the lost state indicates that the drone lacks accurate positioning information from the last positioning; matching a most similar map image from the map data as a matching map image according to the image data; The center coordinates of the matching map image are obtained as the previous positioning information.

6. A method for positioning a UAV according to claim 5, characterized in that: The matching of the most similar map image from the map data as a matching map image according to the image data includes: Acquiring local feature information according to the image data; Performing clustering processing on the local feature information to obtain corresponding visual semantic information; Calculating and obtaining a residual vector between the local feature information and the visual semantic information; Aggregate all the residual vectors to obtain the global image vector; Normalizing the global image vector so that a size corresponding to the global image vector is consistent with a size corresponding to the map data; Obtaining a similarity value corresponding to the map image based on the global vector of the image and the global vector of the map image in the map data; When the maximum similarity value is greater than a fifth preset threshold, the map image corresponding to the maximum similarity value is used as the matching map image, and the state of the drone is set to normal; When the maximum similarity value is less than or equal to the fifth preset threshold, the drone state is maintained in a lost state, and the image data is updated until the matching map image is obtained.

7. A method for positioning a UAV according to claim 6, characterized in that: The obtaining of a similarity value corresponding to the map image by calculating based on the global vector of the image and the global vector of the map image in the map data includes: Determining whether there is historical location information or location-specifying information, wherein the location-specifying information represents the input location information; When the historical location information or the location designation information exists, determining a map matching range based on the historical location information or the location designation information, calculating the image global vector with a global vector of a map image corresponding to the map matching range in the map data, and obtaining the similarity value; When the historical position information and the position designation information do not exist, the image global vector is calculated with the global vectors of all map images in the map data to obtain the similarity value.

8. A drone, characterized in that: include: Input module, used to obtain image data and inertial navigation data; A positioning and tracking module includes a first positioning module and a second positioning module, wherein the first positioning module is used to obtain posture change information and obtain current positioning information based on the posture change information and the previous positioning information, and the second positioning module is used to obtain a common view area through coarse matching and obtain current positioning information through fine matching; The repositioning module is used to obtain the previous positioning information when the drone is in a lost state; Map module, used to store map library data; The input module, the positioning and tracking module, the repositioning module and the map module cooperate with each other to implement the drone positioning method as described in any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for positioning a drone as claimed in any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for positioning a drone as claimed in any one of claims 1 to 7 is implemented.

Images